> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.backends.base

## Module Contents

### Classes

| Name                                                               | Description                                                         |
| ------------------------------------------------------------------ | ------------------------------------------------------------------- |
| [`BaseExecutor`](#nemo_curator-backends-base-BaseExecutor)         | Executor for a pipeline.                                            |
| [`BaseStageAdapter`](#nemo_curator-backends-base-BaseStageAdapter) | Adapts ProcessingStage to an execution backend, if needed.          |
| [`NodeInfo`](#nemo_curator-backends-base-NodeInfo)                 | Generic node information for setup\_on\_node calls across backends. |
| [`WorkerMetadata`](#nemo_curator-backends-base-WorkerMetadata)     | Generic worker metadata for setup\_on\_node calls across backends.  |

### Functions

| Name                                                       | Description                                                                  |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------- |
| [`_is_sentinel`](#nemo_curator-backends-base-_is_sentinel) | A payload-less marker (NoneTask/FailedTask), stripped before the next stage. |

### API

```python
class nemo_curator.backends.base.BaseExecutor(
    config: dict[str, typing.Any] | None = None,
    ignore_head_node: bool = False
)
```

Abstract

Executor for a pipeline.

**`config`** `= config or {}`

---

**`ignore_head_node`** `= ignore_head_node or ignore_ray_head_node()`

---

```python
nemo_curator.backends.base.BaseExecutor.execute(
    stages: list[nemo_curator.stages.base.ProcessingStage],
    initial_tasks: list[nemo_curator.tasks.Task] | None = None
) -> None
```

abstract

Execute the pipeline.

```python
class nemo_curator.backends.base.BaseStageAdapter(
    stage: nemo_curator.stages.base.ProcessingStage
)
```

Adapts ProcessingStage to an execution backend, if needed.

```python
nemo_curator.backends.base.BaseStageAdapter._apply_resumability_counters(
    input_tasks: list[nemo_curator.tasks.Task],
    output_tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]
```

```python
nemo_curator.backends.base.BaseStageAdapter._post_process_task_ids(
    input_tasks: list[nemo_curator.tasks.Task],
    output_tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]
```

Assign a deterministic `task_id` (parent id + own segment) to every
emitted task. Runs once per stage on every backend, so `process` vs
`process_batch` makes no difference; ids are re-derived at each stage
boundary, so one object passing through N stages gets N ids.

* single input → fan-out: each output is `parent_<seg>`
* `len(output) == len(input)` → positional 1:1: `parent_i_<seg>`; a
  `NoneTask` slot means input `i` was filtered (kept for alignment, then
  dropped from the result)
* any other cardinality → a random `"r"`-prefixed uuid (non-deterministic,
  ancestry-not-tracked; see `Task.task_id`)

`seg` is the content id (`get_deterministic_id()`) for a source stage,
else the positional index. A stage that both filters and fans out in one
batch can't be mapped positionally and falls to the `"r"` case — return
one value (or `None`) per input to stay positional.

```python
nemo_curator.backends.base.BaseStageAdapter._source_counters(
    output_tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]
```

Source stage: each output is a source partition; its `_source_id` is
`Task.get_source_id()`. Drop already-completed sources; each survivor fires `+1`.

```python
nemo_curator.backends.base.BaseStageAdapter.process_batch(
    tasks: list[nemo_curator.tasks.Task]
) -> list[nemo_curator.tasks.Task]
```

Process a batch of tasks.

**Parameters:**

**`tasks`** `list[Task]`

List of tasks to process

---

**Returns:** `list[Task]`

list\[Task]: List of processed tasks

```python
nemo_curator.backends.base.BaseStageAdapter.setup(
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Setup the stage once per actor.

**Parameters:**

**`worker_metadata`** `WorkerMetadata` — default: None

Information about the worker

---

```python
nemo_curator.backends.base.BaseStageAdapter.setup_on_node(
    node_info: nemo_curator.backends.base.NodeInfo | None = None,
    worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Setup the stage on a node.

**Parameters:**

**`node_info`** `NodeInfo` — default: None

Information about the node

---

**`worker_metadata`** `WorkerMetadata` — default: None

Information about the worker

---

```python
nemo_curator.backends.base.BaseStageAdapter.teardown() -> None
```

Teardown the stage once per actor.

```python
class nemo_curator.backends.base.NodeInfo(
    node_id: str = ''
)
```

Dataclass

Generic node information for setup\_on\_node calls across backends.
Simplified to match Xenna's structure.

**`node_id`** `str = ''`

---

```python
class nemo_curator.backends.base.WorkerMetadata(
    worker_id: str = '',
    allocation: typing.Any = None
)
```

Dataclass

Generic worker metadata for setup\_on\_node calls across backends.
Simplified to match Xenna's structure. The allocation field can contain
backend-specific allocation information.

**`worker_id`** `str = ''`

---

```python
nemo_curator.backends.base._is_sentinel(
    task: nemo_curator.tasks.Task
) -> bool
```

A payload-less marker (NoneTask/FailedTask), stripped before the next stage.