***

layout: overview
slug: nemo-curator/nemo\_curator/backends/base
title: nemo\_curator.backends.base
----------------------------------

## Module Contents

### Classes

| Name                                                               | Description                                                         |
| ------------------------------------------------------------------ | ------------------------------------------------------------------- |
| [`BaseExecutor`](#nemo_curator-backends-base-BaseExecutor)         | Executor for a pipeline.                                            |
| [`BaseStageAdapter`](#nemo_curator-backends-base-BaseStageAdapter) | Adapts ProcessingStage to an execution backend, if needed.          |
| [`NodeInfo`](#nemo_curator-backends-base-NodeInfo)                 | Generic node information for setup\_on\_node calls across backends. |
| [`WorkerMetadata`](#nemo_curator-backends-base-WorkerMetadata)     | Generic worker metadata for setup\_on\_node calls across backends.  |

### API

<Anchor id="nemo_curator-backends-base-BaseExecutor">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_curator.backends.base.BaseExecutor(
        config: dict[str, typing.Any] | None = None,
        ignore_head_node: bool = False
    )
    ```
  </CodeBlock>
</Anchor>

<Indent>
  <Badge>
    Abstract
  </Badge>

  Executor for a pipeline.

  <ParamField path="config" type="= config or {}" />

  <Anchor id="nemo_curator-backends-base-BaseExecutor-execute">
    <CodeBlock links={{"nemo_curator.stages.base.ProcessingStage":"/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage","nemo_curator.tasks.Task":"/nemo-curator/nemo_curator/tasks/tasks#nemo_curator-tasks-tasks-Task"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_curator.backends.base.BaseExecutor.execute(
          stages: list[nemo_curator.stages.base.ProcessingStage],
          initial_tasks: list[nemo_curator.tasks.Task] | None = None
      ) -> None
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    <Badge>
      abstract
    </Badge>

    Execute the pipeline.
  </Indent>
</Indent>

<Anchor id="nemo_curator-backends-base-BaseStageAdapter">
  <CodeBlock links={{"nemo_curator.stages.base.ProcessingStage":"/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage"}} showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_curator.backends.base.BaseStageAdapter(
        stage: nemo_curator.stages.base.ProcessingStage
    )
    ```
  </CodeBlock>
</Anchor>

<Indent>
  Adapts ProcessingStage to an execution backend, if needed.

  <Anchor id="nemo_curator-backends-base-BaseStageAdapter-process_batch">
    <CodeBlock links={{"nemo_curator.tasks.Task":"/nemo-curator/nemo_curator/tasks/tasks#nemo_curator-tasks-tasks-Task"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_curator.backends.base.BaseStageAdapter.process_batch(
          tasks: list[nemo_curator.tasks.Task]
      ) -> list[nemo_curator.tasks.Task]
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    Process a batch of tasks.

    **Parameters:**

    <ParamField path="tasks" type="list[Task]">
      List of tasks to process
    </ParamField>

    **Returns:** `list[Task]`

    list\[Task]: List of processed tasks
  </Indent>

  <Anchor id="nemo_curator-backends-base-BaseStageAdapter-setup">
    <CodeBlock links={{"nemo_curator.backends.base.WorkerMetadata":"#nemo_curator-backends-base-WorkerMetadata"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_curator.backends.base.BaseStageAdapter.setup(
          worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
      ) -> None
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    Setup the stage once per actor.

    **Parameters:**

    <ParamField path="worker_metadata" type="WorkerMetadata" default="None">
      Information about the worker
    </ParamField>
  </Indent>

  <Anchor id="nemo_curator-backends-base-BaseStageAdapter-setup_on_node">
    <CodeBlock links={{"nemo_curator.backends.base.NodeInfo":"#nemo_curator-backends-base-NodeInfo","nemo_curator.backends.base.WorkerMetadata":"#nemo_curator-backends-base-WorkerMetadata"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_curator.backends.base.BaseStageAdapter.setup_on_node(
          node_info: nemo_curator.backends.base.NodeInfo | None = None,
          worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
      ) -> None
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    Setup the stage on a node.

    **Parameters:**

    <ParamField path="node_info" type="NodeInfo" default="None">
      Information about the node
    </ParamField>

    <ParamField path="worker_metadata" type="WorkerMetadata" default="None">
      Information about the worker
    </ParamField>
  </Indent>

  <Anchor id="nemo_curator-backends-base-BaseStageAdapter-teardown">
    <CodeBlock showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_curator.backends.base.BaseStageAdapter.teardown() -> None
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    Teardown the stage once per actor.
  </Indent>
</Indent>

<Anchor id="nemo_curator-backends-base-NodeInfo">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_curator.backends.base.NodeInfo(
        node_id: str = ''
    )
    ```
  </CodeBlock>
</Anchor>

<Indent>
  <Badge>
    Dataclass
  </Badge>

  Generic node information for setup\_on\_node calls across backends.
  Simplified to match Xenna's structure.

  <ParamField path="node_id" type="str = ''" />
</Indent>

<Anchor id="nemo_curator-backends-base-WorkerMetadata">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_curator.backends.base.WorkerMetadata(
        worker_id: str = '',
        allocation: typing.Any = None
    )
    ```
  </CodeBlock>
</Anchor>

<Indent>
  <Badge>
    Dataclass
  </Badge>

  Generic worker metadata for setup\_on\_node calls across backends.
  Simplified to match Xenna's structure. The allocation field can contain
  backend-specific allocation information.

  <ParamField path="worker_id" type="str = ''" />
</Indent>