`nemo_rl.algorithms.single_controller_utils.config`#

Module Contents#

Classes#

`NativeRolloutFTConfig`	Fault-tolerance knobs read only by `AsyncRolloutImpl` (the native GRPO path).
`NemoGymRolloutFTConfig`	Fault-tolerance knobs read only by `AsyncNemoGymRolloutImpl`.
`RolloutFailureConfig`	Fault tolerance for a rollout that fails.
`FleetHealthConfig`	Liveness tracking for the vLLM generation fleet.
`GenerationRouterConfig`	NeMo-RL-owned HTTP router placed in front of the vLLM fleet for NeMo-Gym.
`WatchdogConfig`	Last-resort detection for stalls that no other layer catches.
`AsyncRLConfig`
`MasterConfig`
`AdvantageConfig`	Internal DataPlane field mapping for advantage calculation.

Functions#

`validate_sampler_buffer_capacity`	Validate that backpressure cannot deadlock the selected sampler.
`_validate_failure_settings`	Check rollout_failure settings that cannot do what they were set for.
`validate_single_controller_config`	Validate cross-section SingleController constraints before setup.

Data#

_GYM_RETRY_STATUSES

API#

class nemo_rl.algorithms.single_controller_utils.config.NativeRolloutFTConfig#

Bases: pydantic.BaseModel

Fault-tolerance knobs read only by AsyncRolloutImpl (the native GRPO path).

Setting these on a NeMo-Gym run does nothing; validate_single_controller_config rejects that rather than letting it pass silently.

generation_timeout_s: Optional[pydantic.PositiveFloat]#: None

env_timeout_s: Optional[pydantic.PositiveFloat]#: None

class nemo_rl.algorithms.single_controller_utils.config.NemoGymRolloutFTConfig#

Bases: pydantic.BaseModel

Fault-tolerance knobs read only by AsyncNemoGymRolloutImpl.

Setting these on a native run does nothing; validate_single_controller_config rejects that rather than letting it pass silently.

rollout_timeout_s: Optional[pydantic.PositiveFloat]#: None

max_row_attempts: pydantic.PositiveInt#: 3

class nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#

Bases: pydantic.BaseModel

Fault tolerance for a rollout that fails.

The budgets at the top level are consumed by generate_and_push, which sits above the native/NeMo-Gym split, so they govern both paths. Everything path-specific lives in the native and nemo_gym sub-blocks, so the structure itself says which knob applies where – these used to dangle on async_rl among unrelated pump and buffer settings, where a native-path operator could set the most generic-sounding one (rollout_timeout_s) and silently get no deadline at all.

Infrastructure failures re-dispatch the prompt onto a different generation shard; data failures are deterministic, so their budget is small and exhausting it is reported rather than absorbed. Nothing here ever discards a prompt silently.

Each class also has a budget for how many prompts may be given up on entirely, and the two differ because the question they answer differs. Data exhaustion is a property of the dataset, so max_skipped_prompts counts them for the run’s lifetime. Infra exhaustion is a property of the fleet at a moment in time, so max_consecutive_dropped_prompts resets on every success: an outage that ends is absorbed, one that does not stops the run.

Once a prompt has been given up on, on_dropped_prompt decides what happens to the training step it was stamped for: train the step on fewer groups, or substitute a fresh prompt so the step keeps its configured batch size.

max_infra_attempts_per_prompt: pydantic.PositiveInt#: 5

max_data_attempts_per_prompt: pydantic.PositiveInt#: 2

backoff_base_s: pydantic.PositiveFloat#: 1.0

max_backoff_s: pydantic.PositiveFloat#: 30.0

max_skipped_prompts: pydantic.NonNegativeInt#: 0

max_consecutive_dropped_prompts: pydantic.NonNegativeInt#: 0

min_step_batch_fraction: Annotated[float, Field(gt=0.0, le=1.0)]#: 0.9

on_dropped_prompt: Literal[shrink, replace]#: ‘shrink’

max_replacement_attempts: pydantic.NonNegativeInt#: 1

replacement_reserve_prompts: pydantic.NonNegativeInt#: 1

native: nemo_rl.algorithms.single_controller_utils.config.NativeRolloutFTConfig#: ‘Field(…)’

nemo_gym: nemo_rl.algorithms.single_controller_utils.config.NemoGymRolloutFTConfig#: ‘Field(…)’

_check_consistent() → nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#

_reject_renamed_keys() → nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#

Fail loudly on the previous key names rather than ignoring them.

extra="allow" means an old key parses fine and then does nothing. For on_data_exhausted: skip that is a behaviour change – prompts that used to be skipped now fail the run – arriving with no diagnostic at all.

class nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#

Bases: pydantic.BaseModel

Liveness tracking for the vLLM generation fleet.

Only the knobs P1 actually consumes are declared. Recovery modes beyond fail_fast need the communicator rebuild that lands later, so the Literal rejects them rather than accepting a value that would silently do nothing.

enabled: bool#: False

probe_interval_s: pydantic.PositiveFloat#: 5.0

probe_timeout_s: pydantic.PositiveFloat#: 2.0

unhealthy_threshold: pydantic.PositiveInt#: 3

healthy_threshold: pydantic.PositiveInt#: 2

selection: Literal[least_outstanding]#: ‘least_outstanding’

on_dead_shard: Literal[fail_fast]#: ‘fail_fast’

max_restart_attempts_per_shard: pydantic.PositiveInt#: 5

min_healthy_shards: pydantic.PositiveInt#: 1

_check_consistent() → nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#

nemo_rl.algorithms.single_controller_utils.config._GYM_RETRY_STATUSES: frozenset[int]#: ‘frozenset(…)’

class nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#

Bases: pydantic.BaseModel

NeMo-RL-owned HTTP router placed in front of the vLLM fleet for NeMo-Gym.

Gym selects a policy endpoint by static round-robin over a list fixed at process start and never fails over. Handing it a single NeMo-RL-owned URL moves that decision to where fleet health already lives, without changing Gym.

enabled: bool#: False

port_range_low: pydantic.PositiveInt#: 6000

port_range_high: pydantic.PositiveInt#: 6099

backend_timeout_s: pydantic.PositiveFloat#: 600.0

connect_timeout_s: pydantic.PositiveFloat#: 5.0

no_healthy_backend_status: pydantic.PositiveInt#: 409

_check_port_range() → nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#

_check_connect_timeout_fits() → nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#

_check_status_is_not_retried_by_gym() → nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#

class nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#

Bases: pydantic.BaseModel

Last-resort detection for stalls that no other layer catches.

interval_s: pydantic.PositiveFloat#: 30.0

stall_timeout_s: pydantic.PositiveFloat#: 600.0

stall_action: Literal[warn, abort]#: ‘warn’

gym_subprocess_check: bool#: True

_check_consistent() → nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#

class nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#

Bases: pydantic.BaseModel

sampler: nemo_rl.algorithms.async_utils.staleness_sampler.SamplerConfig#: ‘Field(…)’

rollout_failure: nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#: ‘Field(…)’

stall_watchdog: nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#: ‘Field(…)’

generation_fleet_health: nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#: ‘Field(…)’

generation_router: nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#: ‘Field(…)’

recompute_kv_cache_after_weight_updates: bool#: False

min_groups_for_streaming_train: int#: 32

max_inflight_prompts: int#: 32

max_buffered_rollouts: int#: 64

diagnostics: bool#: False

_reject_renamed_blocks() → nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#

Fail loudly on the previous block names rather than ignoring them.

extra="allow" means an old key parses fine and then does nothing at all – so a config carrying watchdog: would silently lose its stall detection and run with the defaults, which is precisely the class of silent misconfiguration this work exists to remove. watchdog in particular shipped, so this is a migration path rather than a courtesy.

_check_router_deadline_fits_inside_the_rollout() → nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#

The router’s per-request deadline must not outlast the whole rollout’s.

backend_timeout_s bounds ONE HTTP call; rollout_timeout_s bounds the whole prompt-group stream, which is many of them. Set the inner one larger and it can never fire: the rollout deadline always expires first, so the timeout the router exists to add is dead config – the silent no-op shape this series exists to remove. It is also the wrong failure to surface, because the rollout layer reports the group while the router could have named the backend.

_check_stall_watchdog_outlasts_rollouts() → nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#

_reject_relocated_keys() → nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#

Fail loudly on keys that moved, instead of ignoring them.

These models are extra="allow", so a config written against the previous layout keeps parsing and its fault-tolerance settings simply stop taking effect. A silently ignored rollout_timeout_s: 900 is precisely the failure mode the restructure was meant to remove, so the move must not create one on its way out.

class nemo_rl.algorithms.single_controller_utils.config.MasterConfig#

Bases: pydantic.BaseModel

policy: nemo_rl.models.policy.PolicyConfig#: None

loss_fn: nemo_rl.algorithms.loss.ClippedPGLossConfig#: None

env: dict[str, Any]#: None

data: nemo_rl.data.DataConfig#: None

grpo: nemo_rl.algorithms.grpo.GRPOConfig#: None

logger: nemo_rl.algorithms.grpo.GRPOLoggerConfig#: None

cluster: nemo_rl.distributed.virtual_cluster.ClusterConfig#: None

checkpointing: nemo_rl.utils.checkpoint.CheckpointingConfig#: None

data_plane: nemo_rl.data_plane.interfaces.DataPlaneConfig#: None

async_rl: nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#: None

nemo_rl.algorithms.single_controller_utils.config.validate_sampler_buffer_capacity( async_config: nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig, *, required_capacity: Optional[int], sampler_name: str, ) → None#: Validate that backpressure cannot deadlock the selected sampler.

nemo_rl.algorithms.single_controller_utils.config._validate_failure_settings( async_config: nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig, num_prompts_per_step: int, ) → None#

Check rollout_failure settings that cannot do what they were set for.

RolloutFailureConfig._check_consistent rejects the two combinations that make on_dropped_prompt="replace" unable to ever produce a replacement. These are the combinations it cannot see, because each depends on a field outside that block.

Warnings rather than errors for most of them, deliberately: a strict “never train a short batch” run, one base YAML whose sampler is overridden per experiment, a deliberately deep spare pool are all coherent things to have typed, so rejecting them would forbid configurations somebody wants. What is not acceptable is finding out hours into a run, or not at all.

The exception, and the only hard error here, is a drop budget under a sampler that stamps no target step. That one is not a knob cancelling itself out; it converts a recoverable prompt failure into a stalled run, so there is no configuration it could be the intent of.

nemo_rl.algorithms.single_controller_utils.config.validate_single_controller_config( master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig, ) → None#: Validate cross-section SingleController constraints before setup.

class nemo_rl.algorithms.single_controller_utils.config.AdvantageConfig#

Internal DataPlane field mapping for advantage calculation.

output_field: str#: ‘advantages’

prompt_ids_field: str#: ‘prompt_ids_for_adv’

reward_field: str#: ‘total_reward’

token_mask_field: str#: ‘token_mask’

sample_mask_field: str#: ‘sample_mask’

repeated_batch_fields: list[str]#: ‘field(…)’

policy_logprobs_field: str#: ‘prev_logprobs’

generation_logprobs_field: str#: ‘generation_logprobs’

reference_logprobs_field: str#: ‘reference_policy_logprobs’

nemo_rl.algorithms.single_controller_utils.config#

Module Contents#

Classes#

Functions#

Data#

API#

`nemo_rl.algorithms.single_controller_utils.config`#