> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.datasets.zephon.pipeline

Construct pipelines exclusively through Zephon's public API.

## Module Contents

### Classes

| Name                                                                                           | Description |
| ---------------------------------------------------------------------------------------------- | ----------- |
| [`_PrefetchOverrides`](#nemo_automodel-components-datasets-zephon-pipeline-_PrefetchOverrides) | -           |
| [`_SourceOverrides`](#nemo_automodel-components-datasets-zephon-pipeline-_SourceOverrides)     | -           |

### Functions

| Name                                                                                                   | Description                                                                 |
| ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
| [`create_source_pipeline`](#nemo_automodel-components-datasets-zephon-pipeline-create_source_pipeline) | Create the shared source/runtime backbone before objective-specific stages. |
| [`finalize_pipeline`](#nemo_automodel-components-datasets-zephon-pipeline-finalize_pipeline)           | Batch complete documents and configure/validate the shared runtime.         |

### Data

[`logger`](#nemo_automodel-components-datasets-zephon-pipeline-logger)

### API

```python
class nemo_automodel.components.datasets.zephon.pipeline._PrefetchOverrides
```

**Bases:** `typing.TypedDict`

**`buffer_size`** `int`

---

**`parallelism`** `int`

---

```python
class nemo_automodel.components.datasets.zephon.pipeline._SourceOverrides
```

**Bases:** `typing.TypedDict`

**`stop_after_passes`** `int`

---

```python
nemo_automodel.components.datasets.zephon.pipeline.create_source_pipeline(
    options: nemo_automodel.components.datasets.zephon.config.ZephonOptions,
    seed: int,
    dp_rank: int,
    dp_world_size: int,
    measure: collections.abc.Callable[[Any], int] | None = None
) -> zephon.Pipeline
```

Create the shared source/runtime backbone before objective-specific stages.

`measure` is an adapter-owned token counter for pretokenized or multimodal
documents. Raw text uses Zephon's built-in tokenizer counting.

```python
nemo_automodel.components.datasets.zephon.pipeline.finalize_pipeline(
    pipeline: zephon.Pipeline,
    options: nemo_automodel.components.datasets.zephon.config.ZephonOptions,
    batch_size: int,
    seed: int,
    dp_rank: int,
    dp_world_size: int
) -> None
```

Batch complete documents and configure/validate the shared runtime.

Call after objective-specific encoding and packing.
The pipeline is mutated in place; no iteration or checkpoint is performed.
The pipeline ends at batch grouping. The loader calls to_training and applies
model-input adaptations after receiving each original SampleBatch.

```python
nemo_automodel.components.datasets.zephon.pipeline.logger = logging.getLogger(__name__)
```