stages.text.io.writer.base

Module Contents

Classes

BaseWriter

Base class for all writer stages.

API

class stages.text.io.writer.base.BaseWriter

Bases: nemo_curator.stages.base.ProcessingStage[nemo_curator.tasks.DocumentBatch, nemo_curator.tasks.FileGroupTask], abc.ABC

Base class for all writer stages.

This abstract base class provides common functionality for writing DocumentBatch tasks to files, including file naming, metadata handling, and filesystem operations.

append_mode_implemented: bool

False

fields: list[str] | None

None

file_extension: str

None

get_file_extension() → str

Return the file extension for this writer format.

inputs() → tuple[list[str], list[str]]

Define stage input requirements.

Returns (tuple[list[str], list[str]]): Tuple of (required_attributes, required_columns) where: - required_top_level_attributes: List of task attributes that must be present - required_data_attributes: List of attributes within the data that must be present

mode: Literal[ignore, overwrite, append, error]

‘ignore’

outputs() → tuple[list[str], list[str]]

Define stage output specification.

Returns (tuple[list[str], list[str]]): Tuple of (output_attributes, output_columns) where: - output_top_level_attributes: List of task attributes this stage adds/modifies - output_data_attributes: List of attributes within the data that this stage adds/modifies

path: str

None

process(

task: nemo_curator.tasks.DocumentBatch,

) → nemo_curator.tasks.FileGroupTask

Process a DocumentBatch and write to files.

Args: task (DocumentBatch): DocumentBatch containing data to write

Returns: FileGroupTask: Task containing paths to written files

abstractmethod write_data(

task: nemo_curator.tasks.DocumentBatch,

file_path: str,

) → None

Write data to file using format-specific implementation.

write_kwargs: dict[str, Any]

‘field(…)’