Loader for fetching files and emitting them as DataFrames.
Functions
file_to_df_loader(control_message, task) |
This function is used to load files containing data into a dataframe. |
- file_to_df_loader(control_message, task)[source]
This function is used to load files containing data into a dataframe. Dataframe is created by processing files either using a single thread, dask, or dask_thread. This function determines the download method to use, and if it starts with “dask,” it creates a dask client and uses it to process the files. Otherwise, it uses a single thread to process the files. This function then caches the resulting dataframe using a hash of the file paths. The dataframe is wrapped in a MessageMeta and then attached as a payload to a ControlMessage object and passed on to further stages.
- Parameters
- control_message
- task
The ControlMessage object containing the pipeline control message.
A dictionary representing the current task in the pipeline control message.
- Returns
- message
Updated message control object with payload as a MessageMeta.
- Raises
- RuntimeError:
If no files matched the input strings specified in the task, or if there was an error loading the data.