nemo_automodel.components.datasets.zephon.pipeline

Construct pipelines exclusively through Zephon’s public API.

Module Contents

Classes

Name	Description
`_PrefetchOverrides`	-
`_SourceOverrides`	-

Functions

Name	Description
`create_source_pipeline`	Create the shared source/runtime backbone before objective-specific stages.
`finalize_pipeline`	Batch complete documents and configure/validate the shared runtime.

Data

logger

API

class nemo_automodel.components.datasets.zephon.pipeline._PrefetchOverrides

Bases: typing.TypedDict

buffer_size

int

parallelism

int

class nemo_automodel.components.datasets.zephon.pipeline._SourceOverrides

Bases: typing.TypedDict

stop_after_passes

int

nemo_automodel.components.datasets.zephon.pipeline.create_source_pipeline(
    options: nemo_automodel.components.datasets.zephon.config.ZephonOptions,
    seed: int,
    dp_rank: int,
    dp_world_size: int,
    measure: collections.abc.Callable[[Any], int] | None = None
) -> zephon.Pipeline

Create the shared source/runtime backbone before objective-specific stages.

measure is an adapter-owned token counter for pretokenized or multimodal documents. Raw text uses Zephon’s built-in tokenizer counting.

nemo_automodel.components.datasets.zephon.pipeline.finalize_pipeline(
    pipeline: zephon.Pipeline,
    options: nemo_automodel.components.datasets.zephon.config.ZephonOptions,
    batch_size: int,
    seed: int,
    dp_rank: int,
    dp_world_size: int
) -> None

Batch complete documents and configure/validate the shared runtime.

Call after objective-specific encoding and packing. The pipeline is mutated in place; no iteration or checkpoint is performed. The pipeline ends at batch grouping. The loader calls to_training and applies model-input adaptations after receiving each original SampleBatch.

nemo_automodel.components.datasets.zephon.pipeline.logger = logging.getLogger(__name__)