nemo_curator.config.run

Module Contents

Functions

Name	Description
`_instantiate_stage`	Instantiate a single stage from its Hydra config.
`create_executor_from_yaml`	Create the configured pipeline executor, if executor settings are present.
`create_pipeline_from_yaml`	-
`create_ray_client_from_yaml`	-
`main`	-

Data

_EXECUTOR_TARGETS

_XENNA_EXECUTION_MODES

API

nemo_curator.config.run._instantiate_stage(
    stage_cfg: omegaconf.DictConfig
) -> typing.Any

Instantiate a single stage from its Hydra config.

Extracts resources before calling hydra.utils.instantiate (it is applied via .with_(), not as a constructor argument) and re-applies it after construction. batch_size is left in the config dict so that stages declaring it as a dataclass field receive it during construction.

nemo_curator.config.run.create_executor_from_yaml(
    cfg: omegaconf.DictConfig
) -> nemo_curator.backends.base.BaseExecutor | None

Create the configured pipeline executor, if executor settings are present.

nemo_curator.config.run.create_pipeline_from_yaml(
    cfg: omegaconf.DictConfig,
    log_config: bool = True
) -> nemo_curator.pipeline.Pipeline | typing.Any

nemo_curator.config.run.create_ray_client_from_yaml(
    cfg: omegaconf.DictConfig
) -> nemo_curator.core.client.RayClient

nemo_curator.config.run.main(
    cfg: omegaconf.DictConfig
) -> None

nemo_curator.config.run._EXECUTOR_TARGETS = {'xenna': 'nemo_curator.backends.xenna.XennaExecutor', 'ray_data': 'nemo_curator...

nemo_curator.config.run._XENNA_EXECUTION_MODES = {'batch', 'streaming'}