morpheus.loaders.file_to_df_loader
Functions
|
This function is used to load files containing data into a dataframe. |
|
|
|
- close_dask_cluster()[source]
- file_to_df_loader(control_message, task)[source]
This function is used to load files containing data into a dataframe. Dataframe is created by processing files either using a single thread, multiprocess, dask, or dask_thread. This function determines the download method to use, and if it starts with “dask,” it creates a dask client and uses it to process the files. Otherwise, it uses a single thread or multiprocess to process the files. This function then caches the resulting dataframe using a hash of the file paths. The dataframe is wrapped in a MessageMeta and then attached as a payload to a ControlMessage object and passed on to further stages.
- Parameters
- control_messageControlMessage
- tasktyping.Dict[any, any]
The ControlMessage object containing the pipeline control message.
A dictionary representing the current task in the pipeline control message.
- Returns
- messageControlMessage
Updated message control object with payload as a MessageMeta.
- Raises
- RuntimeError:
If no files matched the input strings specified in the task, or if there was an error loading the data.
- get_dask_client(dask_cluster)[source]
- get_dask_cluster(download_method)[source]