For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DocumentationAPI Reference
DocumentationAPI Reference
  • API Reference
    • Overview
        • Nemo Curator
          • Backends
            • Base
            • Internal
            • Ray Actor Pool
            • Ray Data
            • Utils
            • Xenna
          • Config
          • Core
          • Metrics
          • Models
          • Package Info
          • Pipeline
          • Stages
          • Tasks
          • Utils
    • Pipeline
    • ProcessingStage
    • CompositeStage
    • Resources
NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoNeMo Curator
On this page
  • Module Contents
  • Classes
  • API
API ReferenceFull Library ReferenceNemo CuratorNemo CuratorBackends

nemo_curator.backends.base

||View as Markdown|
Previous

nemo_curator.backends

Next

nemo_curator.backends.internal

Module Contents

Classes

NameDescription
BaseExecutorExecutor for a pipeline.
BaseStageAdapterAdapts ProcessingStage to an execution backend, if needed.
NodeInfoGeneric node information for setup_on_node calls across backends.
WorkerMetadataGeneric worker metadata for setup_on_node calls across backends.

API

class nemo_curator.backends.base.BaseExecutor(
    config: dict[str, typing.Any] | None = None,
    ignore_head_node: bool = False
)
Abstract

Executor for a pipeline.

config
= config or {}
ignore_head_node
= ignore_head_node or ignore_ray_head_node()
nemo_curator.backends.base.BaseExecutor.execute(
    stages: list[nemo_curator.stages.base.ProcessingStage],
    initial_tasks: list[nemo_curator.tasks.Task] | None = None
) -> None
abstract

Execute the pipeline.

class nemo_curator.backends.base.BaseStageAdapter(
    stage: nemo_curator.stages.base.ProcessingStage
)

Adapts ProcessingStage to an execution backend, if needed.

nemo_curator.backends.base.BaseStageAdapter.process_batch(
    tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]

Process a batch of tasks.

Parameters:

tasks
list[Task]

List of tasks to process

Returns: list[Task]

list[Task]: List of processed tasks

nemo_curator.backends.base.BaseStageAdapter.setup(
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None

Setup the stage once per actor.

Parameters:

worker_metadata
WorkerMetadataDefaults to None

Information about the worker

nemo_curator.backends.base.BaseStageAdapter.setup_on_node(
    node_info: nemo_curator.backends.base.NodeInfo | None = None,
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None

Setup the stage on a node.

Parameters:

node_info
NodeInfoDefaults to None

Information about the node

worker_metadata
WorkerMetadataDefaults to None

Information about the worker

nemo_curator.backends.base.BaseStageAdapter.teardown() -> None

Teardown the stage once per actor.

class nemo_curator.backends.base.NodeInfo(
    node_id: str = ''
)
Dataclass

Generic node information for setup_on_node calls across backends. Simplified to match Xenna’s structure.

node_id
str = ''
class nemo_curator.backends.base.WorkerMetadata(
    worker_id: str = '',
    allocation: typing.Any = None
)
Dataclass

Generic worker metadata for setup_on_node calls across backends. Simplified to match Xenna’s structure. The allocation field can contain backend-specific allocation information.

worker_id
str = ''