stages.text.modules.modifier#

Module Contents#

Classes#

Modify

Modify fields of dataset records.

API#

class stages.text.modules.modifier.Modify#

Bases: nemo_curator.stages.base.ProcessingStage[nemo_curator.tasks.DocumentBatch, nemo_curator.tasks.DocumentBatch]

Modify fields of dataset records.

You can provide:

  • a DocumentModifier instance; its modify_document will be used

  • a callable that takes a single value (single-input) or a dict[str, any] (multi-input)

  • a list mixing the above, applied in order

Input fields can be:

  • str (single field reused for each modifier)

  • list[str] (one field per modifier)

  • list[list[str]] (per-modifier multiple input fields)

Implicit behavior:

  • If output_field is None and each modifier has exactly one input field, results are written in-place to that input field.

  • If any modifier has multiple input fields, output_field is required (provide a single name to reuse for all, or one per modifier).

Args: modifier_fn (Callable | DocumentModifier | list[DocumentModifier | Callable]): Modifier or list of modifiers to apply. input_fields (str | list[str] | list[list[str]]): Input field(s); see above for accepted forms. output_fields (str | list[str] | None): Output field name(s). If None and all inputs are single-column, in-place update is performed.

input_fields: str | list[str] | list[list[str]]#

‘text’

inputs() → tuple[list[str], list[str]]#

Define stage input requirements.

Returns (tuple[list[str], list[str]]): Tuple of (required_attributes, required_columns) where: - required_top_level_attributes: List of task attributes that must be present - required_data_attributes: List of attributes within the data that must be present

modifier_fn: collections.abc.Callable | nemo_curator.stages.text.modifiers.doc_modifier.DocumentModifier | list[nemo_curator.stages.text.modifiers.doc_modifier.DocumentModifier | collections.abc.Callable]#

None

output_fields: str | list[str | None] | None#

None

outputs() → tuple[list[str], list[str]]#

Define stage output specification.

Returns (tuple[list[str], list[str]]): Tuple of (output_attributes, output_columns) where: - output_top_level_attributes: List of task attributes this stage adds/modifies - output_data_attributes: List of attributes within the data that this stage adds/modifies

process(
batch: nemo_curator.tasks.DocumentBatch,
) → nemo_curator.tasks.DocumentBatch | None#

Process a task and return the result. Args: task (X): Input task to process Returns (Y | list[Y]): - Single task: For 1-to-1 transformations - List of tasks: For 1-to-many transformations (e.g., readers) - None: If the task should be filtered out