> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/curator/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.backends.base

## Module Contents

### Classes

| Name                                                               | Description                                                         |
| ------------------------------------------------------------------ | ------------------------------------------------------------------- |
| [`BaseExecutor`](#nemo_curator-backends-base-BaseExecutor)         | Executor for a pipeline.                                            |
| [`BaseStageAdapter`](#nemo_curator-backends-base-BaseStageAdapter) | Adapts ProcessingStage to an execution backend, if needed.          |
| [`NodeInfo`](#nemo_curator-backends-base-NodeInfo)                 | Generic node information for setup\_on\_node calls across backends. |
| [`WorkerMetadata`](#nemo_curator-backends-base-WorkerMetadata)     | Generic worker metadata for setup\_on\_node calls across backends.  |

### API

```python
class nemo_curator.backends.base.BaseExecutor(
    config: dict[str, typing.Any] | None = None,
    ignore_head_node: bool = False
)
```

Abstract

Executor for a pipeline.

```python
nemo_curator.backends.base.BaseExecutor.execute(
    stages: list[nemo_curator.stages.base.ProcessingStage],
    initial_tasks: list[nemo_curator.tasks.Task] | None = None
) -> None
```

abstract

Execute the pipeline.

```python
class nemo_curator.backends.base.BaseStageAdapter(
    stage: nemo_curator.stages.base.ProcessingStage
)
```

Adapts ProcessingStage to an execution backend, if needed.

```python
nemo_curator.backends.base.BaseStageAdapter.process_batch(
    tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]
```

Process a batch of tasks.

**Parameters:**

List of tasks to process

**Returns:** `list[Task]`

list\[Task]: List of processed tasks

```python
nemo_curator.backends.base.BaseStageAdapter.setup(
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Setup the stage once per actor.

**Parameters:**

Information about the worker

```python
nemo_curator.backends.base.BaseStageAdapter.setup_on_node(
    node_info: nemo_curator.backends.base.NodeInfo | None = None,
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Setup the stage on a node.

**Parameters:**

Information about the node

Information about the worker

```python
nemo_curator.backends.base.BaseStageAdapter.teardown() -> None
```

Teardown the stage once per actor.

```python
class nemo_curator.backends.base.NodeInfo(
    node_id: str = ''
)
```

Dataclass

Generic node information for setup\_on\_node calls across backends.
Simplified to match Xenna's structure.

```python
class nemo_curator.backends.base.WorkerMetadata(
    worker_id: str = '',
    allocation: typing.Any = None
)
```

Dataclass

Generic worker metadata for setup\_on\_node calls across backends.
Simplified to match Xenna's structure. The allocation field can contain
backend-specific allocation information.