Morpheus pipeline module for fetching files and emitting them as DataFrames.
Functions
file_to_df(builder) |
This module reads data from batched files into a DataFrame after receiving input from the "FileBatcher" module. |
- file_to_df(builder)[source]
This module reads data from batched files into a DataFrame after receiving input from the “FileBatcher” module. It can load file content from both local disk and S3 buckets.
- Parameters
- builder
mrc Builder object.
Notes
- Configurable parameters:
cache_dir (str): Directory to cache the rolling window data.
file_type (str): Type of the input file.
filter_null (bool): Whether to filter out null values.
parser_kwargs (dict): Keyword arguments to pass to the parser.
schema (dict): Schema of the input data.
timestamp_column_name (str): Name of the timestamp column.