nv_ingest.pipeline.config package#
Submodules#
nv_ingest.pipeline.config.loaders module#
Configuration loading and management functions for pipeline execution.
This module provides declarative functions for loading, validating, and applying runtime overrides to pipeline configurations, replacing imperative inline logic.
- nv_ingest.pipeline.config.loaders.apply_runtime_overrides(
- config: PipelineConfigSchema,
- overrides: PipelineRuntimeOverrides,
Apply runtime parameter overrides to a pipeline configuration.
This function creates a copy of the provided configuration and applies any non-None override values to the pipeline runtime settings.
- Parameters:
config (PipelineConfigSchema) – Base pipeline configuration to modify.
overrides (PipelineRuntimeOverrides) – Runtime overrides to apply. Only non-None values are applied.
- Returns:
Modified configuration with overrides applied.
- Return type:
- nv_ingest.pipeline.config.loaders.load_default_libmode_config() PipelineConfigSchema[source]#
Load and validate the default libmode pipeline configuration.
This function loads the embedded default libmode pipeline YAML, performs environment variable substitution, and returns a validated configuration object.
- Returns:
Validated default libmode pipeline configuration.
- Return type:
- Raises:
ValueError – If the default YAML cannot be parsed or validated.
- nv_ingest.pipeline.config.loaders.load_default_pipeline_config() PipelineConfigSchema[source]#
Load and validate the embedded default (non-libmode) pipeline configuration.
- Returns:
Validated default pipeline configuration.
- Return type:
- Raises:
ValueError – If the default YAML cannot be parsed or validated.
- nv_ingest.pipeline.config.loaders.load_pipeline_config(
- config_path: str,
Load a pipeline configuration file, substituting environment variables.
- Parameters:
config_path (str) – The path to the YAML configuration file.
- Returns:
A validated PipelineConfigSchema object.
- Return type:
- Raises:
ValueError – If the YAML file cannot be parsed after environment variable substitution.
- nv_ingest.pipeline.config.loaders.resolve_pipeline_config(
- provided_config: PipelineConfigSchema | None,
- libmode: bool,
Resolve the final pipeline configuration from inputs.
This function implements the configuration resolution logic: - If config provided: use it - If libmode=True and no config: load default libmode config - If libmode=False and no config: raise error
- Parameters:
provided_config (Optional[PipelineConfigSchema]) – User-provided pipeline configuration, or None.
libmode (bool) – Whether to allow loading default libmode configuration.
- Returns:
Resolved and validated pipeline configuration.
- Return type:
- Raises:
ValueError – If no config provided and libmode=False.
- nv_ingest.pipeline.config.loaders.validate_pipeline_config(
- config: PipelineConfigSchema | None,
Validate and ensure a pipeline configuration is available.
This function ensures that a valid pipeline configuration is available, either from the provided config or by loading the default libmode config.
- Parameters:
config (Optional[PipelineConfigSchema]) – Pipeline configuration to validate, or None to load default.
- Returns:
Validated pipeline configuration.
- Return type:
- Raises:
ValueError – If config is None and default config cannot be loaded.
nv_ingest.pipeline.config.replica_resolver module#
Runtime replica resolution for static scaling mode.
This module provides functionality to resolve replica counts for stages using non-static strategies when dynamic scaling is disabled, ensuring total memory consumption stays within the static_memory_threshold.
- nv_ingest.pipeline.config.replica_resolver.get_memory_intensive_stages(
- pipeline_config: PipelineConfigSchema,
Identify stages that are memory-intensive and may need special handling.
- Parameters:
pipeline_config (PipelineConfigSchema) – The pipeline configuration.
- Returns:
List of stage names that are memory-intensive.
- Return type:
List[str]
- nv_ingest.pipeline.config.replica_resolver.resolve_static_replicas(
- pipeline_config: PipelineConfigSchema,
Resolve static replica counts for all stages when dynamic scaling is disabled.
This function calculates the static replica counts for stages using non-static strategies, ensuring the total memory consumption stays within the configured static_memory_threshold. If the total exceeds the threshold, all non-static stages are scaled down proportionally (minimum 1 replica each).
- Parameters:
pipeline_config (PipelineConfigSchema) – The pipeline configuration with potentially unresolved replica strategies.
- Returns:
A new pipeline configuration with all static replica counts resolved.
- Return type: