Morpheus pipeline module for fetching files and emitting them as DataFrames.
Functions
single_object_to_dataframe(file_object, ...) |
Converts a file object into a Pandas DataFrame with optional preprocessing. |
Classes
FileToDFController(schema, filter_null, ...) |
Controller class for converting file objects to Pandas DataFrames with optional preprocessing. |
- single_object_to_dataframe(file_object, schema, file_type, filter_null, parser_kwargs)[source]
Converts a file object into a Pandas DataFrame with optional preprocessing.
- Parameters
-
file_object :
fsspec.core.OpenFile -
schema :
morpheus.utils.column_info.DataFrameInputSchema -
file_type :
morpheus.common.FileTypes - filter_null
- parser_kwargs
A file object, typically from a remote storage system.
A schema defining how to process the data.
The type of the file being processed (e.g., CSV, Parquet).
Flag to indicate whether to filter out null values.
Additional keyword arguments to pass to the file parser.
-
file_object :
- Returns
- pd.DataFrame: The resulting Pandas DataFrame after processing and optional preprocessing.