*** layout: overview slug: nemo-curator/nemo\_curator/pipeline/pipeline title: nemo\_curator.pipeline.pipeline -------------------------------------- ## Module Contents ### Classes | Name | Description | | ------------------------------------------------------ | ---------------------------------------------------------------- | | [`Pipeline`](#nemo_curator-pipeline-pipeline-Pipeline) | User-facing pipeline definition for composing processing stages. | ### API ```python class nemo_curator.pipeline.pipeline.Pipeline( name: str, description: str | None = None, stages: list[nemo_curator.stages.base.ProcessingStage] | None = None, config: dict[str, typing.Any] | None = None ) ``` User-facing pipeline definition for composing processing stages. ```python nemo_curator.pipeline.pipeline.Pipeline.__repr__() -> str ``` String representation of the pipeline. ```python nemo_curator.pipeline.pipeline.Pipeline._decompose_stages( stages: list[nemo_curator.stages.base.ProcessingStage | nemo_curator.stages.base.CompositeStage] ) -> tuple[list[nemo_curator.stages.base.ProcessingStage], dict[str, list[str]]] ``` Decompose composite stages into execution stages. **Parameters:** List of stages that may include composite stages **Returns:** `tuple[list[ProcessingStage], dict[str, list[str]]]` tuple\[list\[ProcessingStage], dict\[str, list\[str]]]: Tuple of (execution stages, decomposition info dict) **Raises:** * `TypeError`: If a composite stage is decomposed into another composite stage ```python nemo_curator.pipeline.pipeline.Pipeline.add_stage( stage: nemo_curator.stages.base.ProcessingStage ) -> nemo_curator.pipeline.pipeline.Pipeline ``` Add a stage to the pipeline. **Parameters:** Processing stage to add **Returns:** `Pipeline` Self (Pipeline) for method chaining ```python nemo_curator.pipeline.pipeline.Pipeline.build() -> None ``` Build an execution plan from the pipeline. **Raises:** * `ValueError`: If the pipeline has no stages ```python nemo_curator.pipeline.pipeline.Pipeline.describe() -> str ``` Get a detailed description of the pipeline stages and their requirements. ```python nemo_curator.pipeline.pipeline.Pipeline.run( executor: nemo_curator.backends.base.BaseExecutor | None = None, initial_tasks: list[nemo_curator.tasks.Task] | None = None ) -> list[nemo_curator.tasks.Task] | None ``` Run the pipeline. **Parameters:** Executor to use Initial tasks to start the pipeline with. Defaults to None. **Returns:** `list[Task] | None` list\[Task] | None: List of tasks