For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DocumentationAPI Reference
DocumentationAPI Reference
  • API Reference
    • Overview
        • Nemo Curator
          • Backends
          • Config
          • Core
          • Metrics
          • Models
          • Package Info
          • Pipeline
          • Stages
          • Tasks
            • Audio Task
            • Document
            • File Group
            • Image
            • Interleaved
            • Tasks
            • Utils
            • Video
          • Utils
    • Pipeline
    • ProcessingStage
    • CompositeStage
    • Resources
NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoNeMo Curator
On this page
  • Module Contents
  • Classes
  • API
API ReferenceFull Library ReferenceNemo CuratorNemo CuratorTasks

nemo_curator.tasks.document

||View as Markdown|
Previous

nemo_curator.tasks.audio_task

Next

nemo_curator.tasks.file_group

Module Contents

Classes

NameDescription
DocumentBatchTask for processing batches of text documents.

API

class nemo_curator.tasks.document.DocumentBatch(
    task_id: str,
    dataset_name: str,
    data: pyarrow.Table | pandas.DataFrame = pa.Table(),
    _stage_perf: list[nemo_curator.utils.performance_utils.StagePerfStats] = list(),
    _metadata: dict[str, typing.Any] = dict()
)
Dataclass

Bases: Task[Table | DataFrame]

Task for processing batches of text documents. Documents are stored as a dataframe (PyArrow Table or Pandas DataFrame).

data
Table | DataFrame = field(default_factory=(pa.Table))
num_items
int

Get the number of documents in this batch.

nemo_curator.tasks.document.DocumentBatch.get_columns() -> list[str]

Get column names from the data.

nemo_curator.tasks.document.DocumentBatch.to_pandas() -> pandas.DataFrame

Convert data to Pandas DataFrame.

nemo_curator.tasks.document.DocumentBatch.to_pyarrow() -> pyarrow.Table

Convert data to PyArrow table.

nemo_curator.tasks.document.DocumentBatch.validate() -> bool

Validate the task data.