morpheus.modules.file_to_df

Morpheus pipeline module for fetching files and emitting them as DataFrames.

Functions

file_to_df(builder)[source]

This module reads data from batched files into a DataFrame after receiving input from the “FileBatcher” module. It can load file content from both local disk and S3 buckets.

Parameters

builder

mrc Builder object.

Notes

Configurable parameters:

• cache_dir (str): Directory to cache the rolling window data.

• file_type (str): Type of the input file.

• filter_null (bool): Whether to filter out null values.

• parser_kwargs (dict): Keyword arguments to pass to the parser.

• schema (dict): Schema of the input data.

• timestamp_column_name (str): Name of the timestamp column.