morpheus.modules.file_to_df

Morpheus pipeline module for fetching files and emitting them as DataFrames.

Functions

file_to_df(builder) This module reads data from batched files into a DataFrame after receiving input from the "FileBatcher" module.
file_to_df(builder)[source]

This module reads data from batched files into a DataFrame after receiving input from the “FileBatcher” module. It can load file content from both local disk and S3 buckets.

Parameters
builder

mrc Builder object.

Notes

Configurable parameters:
  • cache_dir (str): Directory to cache the rolling window data.

  • file_type (str): Type of the input file.

  • filter_null (bool): Whether to filter out null values.

  • parser_kwargs (dict): Keyword arguments to pass to the parser.

  • schema (dict): Schema of the input data.

  • timestamp_column_name (str): Name of the timestamp column.

Previous morpheus.modules.file_batcher
Next morpheus.modules.filter_cm_failed
© Copyright 2024, NVIDIA. Last updated on Apr 25, 2024.