nemo_curator.backends.xenna

Submodules

Package Contents

Classes

Name	Description
`XennaExecutor`	Executor that runs pipelines using Cosmos-Xenna.

API

class nemo_curator.backends.xenna.XennaExecutor(
    config: dict[str, typing.Any] | None = None,
    ignore_head_node: bool = False
)

Bases: BaseExecutor

Executor that runs pipelines using Cosmos-Xenna. This executor provides integration between the nemo-curator pipeline framework and the Cosmos-Xenna execution engine for distributed processing.

_default_pipeline_config

nemo_curator.backends.xenna.XennaExecutor._get_pipeline_config(
    key: str
) -> typing.Any

Get configuration value with fallback to defaults.

nemo_curator.backends.xenna.XennaExecutor.execute(
    stages: list[nemo_curator.stages.base.ProcessingStage],
    initial_tasks: list[nemo_curator.tasks.Task] | None = None
) -> list[nemo_curator.tasks.Task]

Execute the pipeline using Cosmos-Xenna.

Parameters:

stages

list[ProcessingStage]

list[ ProcessingStage ] — The stages to run

initial_tasks

list[Task]Defaults to None

list[ Task ] — The initial tasks to run. Empty list of Task is used if not provided.

Returns: list[ Task ]

list[Task]: List of output tasks from the pipeline