modules.to_backend

Module Contents

Classes

ToBackend

A module for moving dataframes between backends.

API

class modules.to_backend.ToBackend(backend: Literal[pandas, cudf])

Bases: nemo_curator.modules.base.BaseModule

A module for moving dataframes between backends.

Initialization

Constructs a ToBackend module

Args: backend (str): The backend to transfer the dataset to. Can be “pandas” or “cudf”

call(

dataset: nemo_curator.datasets.doc_dataset.DocumentDataset,

) → nemo_curator.datasets.doc_dataset.DocumentDataset

Performs an arbitrary operation on a dataset

Args: dataset (DocumentDataset): The dataset to operate on