modules.modify

Module Contents

Classes

Modify

Base class for all NeMo Curator modules.

API

class modules.modify.Modify(

modifier: nemo_curator.modifiers.DocumentModifier,

text_field: str = 'text',

)

Bases: nemo_curator.modules.base.BaseModule

Base class for all NeMo Curator modules.

Handles validating that data lives on the correct device for each module

Initialization

Constructs a Module

Args: input_backend (Literal[“pandas”, “cudf”, “any”]): The backend the input dataframe must be on for the module to work name (str, Optional): The name of the module. If None, defaults to self.class.name

call(

dataset: nemo_curator.datasets.DocumentDataset,

) → nemo_curator.datasets.DocumentDataset

Performs an arbitrary operation on a dataset

Args: dataset (DocumentDataset): The dataset to operate on