stages.text.io.reader.base#
Module Contents#
Classes#
Common base for tabular file readers. |
API#
- class stages.text.io.reader.base.BaseReader#
Bases:
nemo_curator.stages.base.ProcessingStage[nemo_curator.tasks.FileGroupTask,nemo_curator.tasks.DocumentBatch]Common base for tabular file readers.
Subclasses must implement the read_data method.
- fields: list[str] | None#
None
- inputs() tuple[list[str], list[str]]#
Define stage input requirements.
Returns (tuple[list[str], list[str]]): Tuple of (required_attributes, required_columns) where: - required_top_level_attributes: List of task attributes that must be present - required_data_attributes: List of attributes within the data that must be present
- outputs() tuple[list[str], list[str]]#
Define stage output specification.
Returns (tuple[list[str], list[str]]): Tuple of (output_attributes, output_columns) where: - output_top_level_attributes: List of task attributes this stage adds/modifies - output_data_attributes: List of attributes within the data that this stage adds/modifies
- process(
- task: nemo_curator.tasks.FileGroupTask,
Process a task and return the result. Args: task (X): Input task to process Returns (Y | list[Y]): - Single task: For 1-to-1 transformations - List of tasks: For 1-to-many transformations (e.g., readers) - None: If the task should be filtered out
- ray_stage_spec() dict[str, Any]#
Get Ray configuration for this stage. Note : This is only used for Ray Data which is an experimental backend. The keys are defined in RayStageSpecKeys in backends/experimental/ray_data/utils.py
Returns (dict[str, Any]): Dictionary containing Ray-specific configuration
- abstractmethod read_data(
- file_paths: list[str],
- read_kwargs: dict[str, Any] | None,
- fields: list[str] | None,
- read_kwargs: dict[str, Any]#
‘field(…)’
- setup(
- _: nemo_curator.backends.base.WorkerMetadata | None = None,
Setup method called once before processing begins. Override this method to perform any initialization that should happen once per worker. Args: worker_metadata (WorkerMetadata, optional): Information about the worker (provided by some backends)