morpheus.loaders.file_to_df_loader

Loader for fetching files and emitting them as DataFrames.

Functions

file_to_df_loader(control_message, task) This function is used to load files containing data into a dataframe.
file_to_df_loader(control_message, task)[source]

This function is used to load files containing data into a dataframe. Dataframe is created by processing files either using a single thread, dask, or dask_thread. This function determines the download method to use, and if it starts with “dask,” it creates a dask client and uses it to process the files. Otherwise, it uses a single thread to process the files. This function then caches the resulting dataframe using a hash of the file paths. The dataframe is wrapped in a MessageMeta and then attached as a payload to a ControlMessage object and passed on to further stages.

Parameters
control_messageControlMessage

The ControlMessage object containing the pipeline control message.

tasktyping.Dict[any, any]

A dictionary representing the current task in the pipeline control message.

Returns
messageControlMessage

Updated message control object with payload as a MessageMeta.

Raises
RuntimeError:

If no files matched the input strings specified in the task, or if there was an error loading the data.

Previous morpheus.loaders
Next morpheus.loaders.fsspec_loader
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.