nemo_rl.algorithms.single_controller_utils.config#
Module Contents#
Classes#
Fault-tolerance knobs read only by |
|
Fault-tolerance knobs read only by |
|
Fault tolerance for a rollout that fails. |
|
Liveness tracking for the vLLM generation fleet. |
|
NeMo-RL-owned HTTP router placed in front of the vLLM fleet for NeMo-Gym. |
|
Last-resort detection for stalls that no other layer catches. |
|
Run the weight synchronizer in a separate CPU Ray actor (weight_sync/refit_actor.py). |
|
Synthetic first training pass, run while the trainer waits for generation. |
|
Ledger-authoritative token capture (token-in/token-out via NeMo-Gym). |
|
Resolved retry and restore policy for a prompt-group reservation. |
|
Retry and restore policy for unfinished token-capture prompt groups. |
|
Frequent rollout-state snapshots anchored to durable trainer state. |
|
Internal DataPlane field mapping for advantage calculation. |
Functions#
Keep the initial Forest objective identical at startup and before fanout. |
|
Return policy/reference forward requirements for both setup and execution. |
|
Common row divisibility for fixed or packed Megatron consumers. |
|
Whether this SingleController run trains a PPO critic alongside the policy. |
|
Whether the run saves and restores rollout snapshots. |
|
The active algorithm block: |
|
Validate that backpressure cannot deadlock the selected sampler. |
|
Validate the full-vocabulary MOPD block against the rest of the run. |
|
Check rollout_failure settings that cannot do what they were set for. |
|
Return the VLM structure that determines packed-media compatibility. |
|
Fingerprint the token-to-id contract consumed by the critic. |
|
Allow distinct checkpoints only when their media and token contracts match. |
|
Reject algorithm blocks the SingleController path cannot honour. |
|
Samples a dispatch must be a multiple of to split cleanly on this consumer. |
|
Validate PPO batch alignment and return sampler group and tail multiples. |
|
Reject a GRPO chunk cap whose samples cannot split across the policy. |
|
Validate cross-section SingleController constraints before setup. |
Data#
API#
- class nemo_rl.algorithms.single_controller_utils.config.NativeRolloutFTConfig#
Bases:
pydantic.BaseModelFault-tolerance knobs read only by
AsyncRolloutImpl(the native GRPO path).Setting these on a NeMo-Gym run does nothing;
validate_single_controller_configrejects that rather than letting it pass silently.- generation_timeout_s: Optional[pydantic.PositiveFloat]#
None
- env_timeout_s: Optional[pydantic.PositiveFloat]#
None
- class nemo_rl.algorithms.single_controller_utils.config.NemoGymRolloutFTConfig#
Bases:
pydantic.BaseModelFault-tolerance knobs read only by
AsyncNemoGymRolloutImpl.Setting these on a native run does nothing;
validate_single_controller_configrejects that rather than letting it pass silently.- rollout_timeout_s: Optional[pydantic.PositiveFloat]#
None
- max_row_attempts: pydantic.PositiveInt#
3
- class nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#
Bases:
pydantic.BaseModelFault tolerance for a rollout that fails.
The budgets at the top level are consumed by
generate_and_push, which sits above the native/NeMo-Gym split, so they govern both paths. Everything path-specific lives in thenativeandnemo_gymsub-blocks, so the structure itself says which knob applies where – these used to dangle onasync_rlamong unrelated pump and buffer settings, where a native-path operator could set the most generic-sounding one (rollout_timeout_s) and silently get no deadline at all.Infrastructure failures re-dispatch the prompt onto a different generation shard; data failures are deterministic, so their budget is small and exhausting it is reported rather than absorbed. Nothing here ever discards a prompt silently.
Each class also has a budget for how many prompts may be given up on entirely, and the two differ because the question they answer differs. Data exhaustion is a property of the dataset, so
max_skipped_promptscounts them for the run’s lifetime. Infra exhaustion is a property of the fleet at a moment in time, somax_consecutive_dropped_promptsresets on every success: an outage that ends is absorbed, one that does not stops the run.Once a prompt has been given up on,
on_dropped_promptdecides what happens to the training step it is charged to (the step it was stamped for, or the step being assembled when the sampler stamps none): train the step on fewer groups, or substitute a fresh prompt so the step keeps its configured batch size.- max_infra_attempts_per_prompt: pydantic.PositiveInt#
5
- max_data_attempts_per_prompt: pydantic.PositiveInt#
2
- backoff_base_s: pydantic.PositiveFloat#
1.0
- max_backoff_s: pydantic.PositiveFloat#
30.0
- max_skipped_prompts: pydantic.NonNegativeInt#
0
- max_consecutive_dropped_prompts: pydantic.NonNegativeInt#
0
- min_step_batch_fraction: Annotated[float, Field(gt=0.0, le=1.0)]#
0.9
- on_dropped_prompt: Literal[shrink, replace]#
‘shrink’
- max_replacement_attempts: pydantic.NonNegativeInt#
1
- replacement_reserve_prompts: pydantic.NonNegativeInt#
1
- native: nemo_rl.algorithms.single_controller_utils.config.NativeRolloutFTConfig#
‘Field(…)’
- nemo_gym: nemo_rl.algorithms.single_controller_utils.config.NemoGymRolloutFTConfig#
‘Field(…)’
- _check_consistent() nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#
- _reject_renamed_keys() nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#
Fail loudly on the previous key names rather than ignoring them.
extra="allow"means an old key parses fine and then does nothing. Foron_data_exhausted: skipthat is a behaviour change – prompts that used to be skipped now fail the run – arriving with no diagnostic at all.
- class nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#
Bases:
pydantic.BaseModelLiveness tracking for the vLLM generation fleet.
Only the knobs P1 actually consumes are declared. Recovery modes beyond
fail_fastneed the communicator rebuild that lands later, so the Literal rejects them rather than accepting a value that would silently do nothing.- enabled: bool#
False
- probe_interval_s: pydantic.PositiveFloat#
5.0
- probe_timeout_s: pydantic.PositiveFloat#
2.0
- unhealthy_threshold: pydantic.PositiveInt#
3
- healthy_threshold: pydantic.PositiveInt#
2
- selection: Literal[least_outstanding]#
‘least_outstanding’
- max_restart_attempts_per_shard: pydantic.PositiveInt#
5
- min_healthy_shards: pydantic.PositiveInt#
1
- refit_timeout_s: Optional[pydantic.PositiveFloat]#
300.0
- restart_dead_shards: bool#
False
- restart_timeout_s: pydantic.PositiveFloat#
1800.0
- restart_backoff_s: pydantic.PositiveFloat#
60.0
- _check_consistent() nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#
- nemo_rl.algorithms.single_controller_utils.config._GYM_RETRY_STATUSES: frozenset[int]#
‘frozenset(…)’
- class nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#
Bases:
pydantic.BaseModelNeMo-RL-owned HTTP router placed in front of the vLLM fleet for NeMo-Gym.
Gym selects a policy endpoint by static round-robin over a list fixed at process start and never fails over. Handing it a single NeMo-RL-owned URL moves that decision to where fleet health already lives, without changing Gym.
- enabled: bool#
False
- port_range_low: pydantic.PositiveInt#
None
- port_range_high: pydantic.PositiveInt#
None
- backend_timeout_s: pydantic.PositiveFloat#
600.0
- connect_timeout_s: pydantic.PositiveFloat#
5.0
- no_healthy_backend_status: pydantic.PositiveInt#
409
- _check_port_range() nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#
- _check_connect_timeout_fits() nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#
- _check_status_is_not_retried_by_gym() nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#
- class nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#
Bases:
pydantic.BaseModelLast-resort detection for stalls that no other layer catches.
- interval_s: pydantic.PositiveFloat#
30.0
- stall_timeout_s: pydantic.PositiveFloat#
600.0
- stall_action: Literal[warn, abort]#
‘warn’
- gym_subprocess_check: bool#
True
- _check_consistent() nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#
- class nemo_rl.algorithms.single_controller_utils.config.RefitActorConfig#
Bases:
pydantic.BaseModelRun the weight synchronizer in a separate CPU Ray actor (weight_sync/refit_actor.py).
- enabled: bool#
False
- class nemo_rl.algorithms.single_controller_utils.config.TrainerWarmupConfig#
Bases:
pydantic.BaseModelSynthetic first training pass, run while the trainer waits for generation.
- enabled: bool#
False
- seqlens: Optional[list[int]]#
None
- class nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
Bases:
pydantic.BaseModel- log_full_train_data: bool#
False
- debug_writer: nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriterConfig#
‘Field(…)’
- defer_async_checkpoint_finalize: bool#
False
- sampler: nemo_rl.algorithms.async_utils.staleness_sampler.SamplerConfig#
‘Field(…)’
- rollout_failure: nemo_rl.algorithms.single_controller_utils.config.RolloutFailureConfig#
‘Field(…)’
- stall_watchdog: nemo_rl.algorithms.single_controller_utils.config.WatchdogConfig#
‘Field(…)’
- generation_fleet_health: nemo_rl.algorithms.single_controller_utils.config.FleetHealthConfig#
‘Field(…)’
- generation_router: nemo_rl.algorithms.single_controller_utils.config.GenerationRouterConfig#
‘Field(…)’
- refit_actor: nemo_rl.algorithms.single_controller_utils.config.RefitActorConfig#
‘Field(…)’
- recompute_kv_cache_after_weight_updates: bool#
True
- min_groups_for_streaming_train: int#
32
- max_groups_for_streaming_train: Optional[int]#
None
- max_inflight_prompts: int#
32
- max_buffered_rollouts: int#
64
- diagnostics: bool#
False
- num_advantage_workers: pydantic.NonNegativeInt#
0
- early_trainer_build: bool#
False
- trainer_warmup: nemo_rl.algorithms.single_controller_utils.config.TrainerWarmupConfig#
‘Field(…)’
- _reject_renamed_blocks() nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
Fail loudly on the previous block names rather than ignoring them.
extra="allow"means an old key parses fine and then does nothing at all – so a config carryingwatchdog:would silently lose its stall detection and run with the defaults, which is precisely the class of silent misconfiguration this work exists to remove.watchdogin particular shipped, so this is a migration path rather than a courtesy.
- _check_streaming_chunk_bounds() nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
The chunk cap must leave room for the floor it is capping.
_train_pumpclamps the floor down to the cap so a short tail chunk can still close the step. Set the cap below the floor and that clamp fires on every chunk, so the configured floor never applies and the run streams at the cap instead – the floor becomes dead config rather than a bound.
- _check_router_deadline_fits_inside_the_rollout() nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
The router’s per-request deadline must not outlast the whole rollout’s.
backend_timeout_sbounds ONE HTTP call;rollout_timeout_sbounds the whole prompt-group stream, which is many of them. Set the inner one larger and it can never fire: the rollout deadline always expires first, so the timeout the router exists to add is dead config – the silent no-op shape this series exists to remove. It is also the wrong failure to surface, because the rollout layer reports the group while the router could have named the backend.
- _check_stall_watchdog_outlasts_rollouts() nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
- _reject_relocated_keys() nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig#
Fail loudly on keys that moved, instead of ignoring them.
These models are
extra="allow", so a config written against the previous layout keeps parsing and its fault-tolerance settings simply stop taking effect. A silently ignoredrollout_timeout_s: 900is precisely the failure mode the restructure was meant to remove, so the move must not create one on its way out.
- class nemo_rl.algorithms.single_controller_utils.config.TokenCaptureConfig#
Bases:
pydantic.BaseModelLedger-authoritative token capture (token-in/token-out via NeMo-Gym).
Dormant by default: with
enabled=Falseevery legacy codepath behaves exactly as before — no staging partition is registered, no ledger is installed, and rollouts ride the token-echo path.- enabled: bool#
False
- staging_partition: str#
‘rollout_staging’
- min_valid_fraction_per_group: Optional[float]#
None
- control_auth_token: Optional[str]#
None
- control_timeout_s: float#
60.0
- capture_dir: Optional[str]#
None
- generation_backend: Optional[Literal[vllm, megatron, dynamo]]#
None
- defer_routed_experts_to_policy: bool#
False
- num_reassembler_workers: pydantic.PositiveInt#
2
- forest_training: bool#
False
- selection_policy: str#
‘returned_history’
- validate_selection_policy() nemo_rl.algorithms.single_controller_utils.config.TokenCaptureConfig#
- class nemo_rl.algorithms.single_controller_utils.config.TaskSourceRecoveryGranularity#
Resolved retry and restore policy for a prompt-group reservation.
task_sourceis the materialized taskset or the flat Gym row’s task source. Materialized rows select a taskset override or the global default; flat rows select an agent override, a task-source override, or the global default.restore_levelrecords that target;granularityderives its retry unit.- task_source: Optional[str]#
None
- restore_level: nemo_rl.experience.rollout_recovery.RecoveryTargetLevel#
None
- property granularity: nemo_rl.experience.rollout_recovery.RecoveryGranularity#
- class nemo_rl.algorithms.single_controller_utils.config.RolloutRecoveryConfig#
Bases:
pydantic.BaseModelRetry and restore policy for unfinished token-capture prompt groups.
prefixrestores Gym participant state and decoded tokens from an active model call,turnrequests Gym participant recovery and restarts an active call from its preceding durable turn,siblingpreserves completed generations and restarts unfinished siblings, andprompt_grouprestarts every sibling in an unfinished group. Overrides may only make a particular taskset, task source, or agent coarser than the global target.- target_level: nemo_rl.experience.rollout_recovery.RecoveryTargetLevel#
None
- task_source_target_level_overrides: dict[str, nemo_rl.experience.rollout_recovery.RecoveryTargetLevel]#
‘Field(…)’
- agent_target_level_overrides: dict[str, nemo_rl.experience.rollout_recovery.RecoveryTargetLevel]#
‘Field(…)’
- gym_prepare_timeout_s: Annotated[float, Field(gt=0)]#
300.0
- allow_restart_only_gym_servers: bool#
False
- prefix_write_max_rows: Annotated[int, Field(gt=0)]#
256
- prefix_write_max_tokens: Annotated[int, Field(gt=0)]#
4194304
- prefix_read_max_rows: Annotated[int, Field(gt=0)]#
256
- _validate_restart_only_opt_in() nemo_rl.algorithms.single_controller_utils.config.RolloutRecoveryConfig#
Reject the restart-only opt-in where Gym does not checkpoint rollouts.
- _validate_target_overrides() nemo_rl.algorithms.single_controller_utils.config.RolloutRecoveryConfig#
Reject overrides finer than the global target.
- property turn_checkpointing_enabled: bool#
Whether Gym participant state is required for the requested target.
- property generation_prefix_cuts_enabled: bool#
Whether active model calls participate in Gym’s checkpoint cut.
- resolve_for_prompt(
- prompt: collections.abc.Mapping[str, Any],
Resolve by taskset for materialized tasks, or agent then source for flat rows.
- class nemo_rl.algorithms.single_controller_utils.config.RolloutCheckpointConfig#
Bases:
pydantic.BaseModelFrequent rollout-state snapshots anchored to durable trainer state.
Rollout snapshots are on when
snapshot_attempt_interval_sis set or with turn-level recovery; see docs/guides/single-controller.md for when they are saved.A snapshot taken before the first trainer checkpoint is anchored to the initial model and a rollout-semantic configuration fingerprint. Later snapshots require the durable trainer checkpoint for the controller’s current completed step; attempts are skipped until that exact anchor exists.
restore_mode="latest"selects the newest compatible periodic snapshot.trainer_checkpointignores newer periodic snapshots and restores the rollout state bundled with the durable trainer checkpoint. Restore selection never deletes checkpoint state. If no trainer checkpoint exists,trainer_checkpointrejects an occupied bootstrap namespace; uselatestor a new checkpoint directory instead.Bootstrap compatibility is fail-closed: every configuration value affects the fingerprint unless it is on the built-in operational denylist.
extra_fingerprint_excluded_pathslets integrations exclude additional runtime-only dotpaths.*matches one mapping or list level and**matches any number of levels.SingleController has no validation loop, so checkpoint selection must use
checkpointing.metric_name=Noneor atrain:<name>metric. Inheritedval:<name>settings are rejected during setup. Unknown keys are forbidden because a misspelled interval, retention, or restore option can silently disable the durability behavior the operator intended.telemetry_interval_s=Nonedisables the independent wall-clock sampler for rollout/checkpoint benchmark metrics. It does not enable checkpointing and may be configured withoutsnapshot_attempt_interval_s.max_consecutive_failurescontrols how many consecutive retryable periodic-checkpoint failures are tolerated before the controller aborts the run. A successful or skipped attempt resets the counter; checkpoint invariant failures still fail immediately.trace=Trueappends one JSON line per rollout event torollout_trace.jsonlin the log directory, to check what a stop and restore cost. Per-rollout events needrollout_recoverydispatch. It does not enable checkpointing. The events are listed innemo_rl/algorithms/rollout_checkpointing/trace.py.- snapshot_attempt_interval_s: Annotated[Optional[float], Field(gt=0)]#
None
- snapshot_on_trainer_checkpoint: bool#
True
- telemetry_interval_s: Annotated[Optional[float], Field(gt=0)]#
None
- max_consecutive_failures: Annotated[int, Field(ge=1)]#
3
- keep_latest_k: Annotated[int, Field(ge=1)]#
2
- restore_mode: Literal[latest, trainer_checkpoint]#
‘latest’
- extra_fingerprint_excluded_paths: list[str]#
‘Field(…)’
- trace: bool#
False
- validate_extra_fingerprint_excluded_paths() nemo_rl.algorithms.single_controller_utils.config.RolloutCheckpointConfig#
Reject ambiguous paths that could silently fail to exclude a value.
- class nemo_rl.algorithms.single_controller_utils.config.MasterConfig#
Bases:
pydantic.BaseModel- grpo: Optional[nemo_rl.algorithms.grpo.GRPOConfig]#
None
- ppo: Optional[nemo_rl.algorithms.ppo.PPOConfig]#
None
- policy: nemo_rl.models.policy.PolicyConfig#
None
- value: Optional[nemo_rl.models.value.ValueConfig]#
None
- loss_fn: nemo_rl.algorithms.loss.ClippedPGLossConfig#
None
- value_loss_fn: Optional[nemo_rl.algorithms.loss.loss_functions.MseValueLossConfig]#
None
- env: dict[str, Any]#
None
- data: nemo_rl.data.DataConfig#
None
- logger: nemo_rl.utils.logger.LoggerConfig#
None
- cluster: nemo_rl.distributed.virtual_cluster.ClusterConfig#
None
- checkpointing: nemo_rl.utils.checkpoint.CheckpointingConfig#
None
- reward_penalties: nemo_rl.algorithms.grpo.RewardPenaltyConfig#
‘Field(…)’
- data_plane: nemo_rl.data_plane.interfaces.DataPlaneConfig#
None
- rollout_recovery: nemo_rl.algorithms.single_controller_utils.config.RolloutRecoveryConfig#
‘Field(…)’
- rollout_checkpointing: nemo_rl.algorithms.single_controller_utils.config.RolloutCheckpointConfig#
‘Field(…)’
- on_policy_distillation: Optional[nemo_rl.algorithms.opd.OnPolicyDistillationConfig]#
None
- telemetry: Optional[nemo_rl.telemetry.config.TelemetryConfig]#
None
- token_capture: nemo_rl.algorithms.single_controller_utils.config.TokenCaptureConfig#
‘Field(…)’
- validate_algorithm_block() nemo_rl.algorithms.single_controller_utils.config.MasterConfig#
- nemo_rl.algorithms.single_controller_utils.config.validate_forest_objective(
- grpo: nemo_rl.algorithms.grpo.GRPOConfig,
- loss: nemo_rl.algorithms.loss.ClippedPGLossConfig,
Keep the initial Forest objective identical at startup and before fanout.
- nemo_rl.algorithms.single_controller_utils.config.logprob_passes_required(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
- algo_cfg: nemo_rl.algorithms.grpo.GRPOConfig | nemo_rl.algorithms.ppo.PPOConfig,
Return policy/reference forward requirements for both setup and execution.
- nemo_rl.algorithms.single_controller_utils.config.forest_execution_row_multiple(
- policy: nemo_rl.models.policy.PolicyConfig,
- *,
- dp_size: int,
- logprobs_required: bool,
Common row divisibility for fixed or packed Megatron consumers.
Policy and reference forwards share this trainer’s mesh and logprob MBS. Packed consumers form their own bins and only need complete DP shards. Non-interleaved PP admits any positive microbatch count; interleaved PP and dynamic batching require separate qualification.
- nemo_rl.algorithms.single_controller_utils.config.is_ppo_run(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
Whether this SingleController run trains a PPO critic alongside the policy.
Single source of truth for the flag: setup reads it to decide whether to build the value model, and the controller reads it to decide whether the train pump runs the critic stages.
model_constructskips defaults, so the attribute can genuinely be missing on a hand-built config.
- nemo_rl.algorithms.single_controller_utils.config.rollout_snapshots_enabled(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
Whether the run saves and restores rollout snapshots.
Periodic snapshots turn them on, and so does turn-level recovery, whose environment state only a rollout snapshot holds.
- nemo_rl.algorithms.single_controller_utils.config.dedicated_value_enabled(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
- nemo_rl.algorithms.single_controller_utils.config.algo_config(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
The active algorithm block:
ppoon a PPO run, elsegrpo.Exactly one of the two is set; MasterConfig.validate_algorithm_block checks that at construction.
- nemo_rl.algorithms.single_controller_utils.config.validate_sampler_buffer_capacity(
- async_config: nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig,
- *,
- required_capacity: Optional[int],
- sampler_name: str,
Validate that backpressure cannot deadlock the selected sampler.
- nemo_rl.algorithms.single_controller_utils.config._validate_opd_full_config(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
- opd_config: nemo_rl.algorithms.opd.OnPolicyDistillationConfig,
Validate the full-vocabulary MOPD block against the rest of the run.
- Parameters:
master_config – Full SingleController config, already known to have OPD on.
opd_config – The resolved
on_policy_distillationblock.
- Raises:
ValueError – If
opd_fullis enabled with an unsupported backend, an incompatible logprob path, a fused packing path that never reaches the opd_full branch, or a sampling temperature the hidden-state payload cannot honor.
- nemo_rl.algorithms.single_controller_utils.config._validate_failure_settings(
- async_config: nemo_rl.algorithms.single_controller_utils.config.AsyncRLConfig,
- num_prompts_per_step: int,
Check rollout_failure settings that cannot do what they were set for.
RolloutFailureConfig._check_consistentrejects the two combinations that makeon_dropped_prompt="replace"unable to ever produce a replacement. These are the combinations it cannot see, because each depends on a field outside that block.Most checks warn rather than raise: strict full batches, per-experiment sampler overrides, and deep spare pools can be intentional. A nonzero drop budget requires a boolean class declaration of the sampler’s single-weight requirement and rejects policies that need a complete single-weight batch.
- nemo_rl.algorithms.single_controller_utils.config._vlm_architecture_signature(
- model_name: str,
- hf_config_overrides: collections.abc.Mapping[str, Any] | None,
Return the VLM structure that determines packed-media compatibility.
- nemo_rl.algorithms.single_controller_utils.config._vlm_tokenizer_signature(
- tokenizer_config: collections.abc.Mapping[str, Any],
- fallback_name: str,
Fingerprint the token-to-id contract consumed by the critic.
- nemo_rl.algorithms.single_controller_utils.config._validate_vlm_critic_compatibility(
- policy_config: collections.abc.Mapping[str, Any],
- value_config: collections.abc.Mapping[str, Any],
Allow distinct checkpoints only when their media and token contracts match.
- nemo_rl.algorithms.single_controller_utils.config._validate_algo_settings(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
Reject algorithm blocks the SingleController path cannot honour.
Both directions on the critic: one the PPO path needs and does not have, and one a GRPO run carries and would never build, plus sampling and length-penalty settings the SC path does not implement.
- nemo_rl.algorithms.single_controller_utils.config.sample_multiple(
- config: nemo_rl.models.policy.PolicyConfig | nemo_rl.models.value.ValueConfig,
- dp_size: int,
Samples a dispatch must be a multiple of to split cleanly on this consumer.
A dispatch is split twice: once across the DP ranks, and then on the static batching path again inside each rank by the train and logprob microbatch sizes. Packed and dynamically batched consumers form their own microbatches, so for them only the DP split applies.
- Parameters:
config – Policy or value configuration of the consuming trainer.
dp_size – Data-parallel size of that trainer.
- Returns:
The sample count both splits divide.
- nemo_rl.algorithms.single_controller_utils.config.resolve_ppo_chunk_alignment(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
- policy_dp: int,
- value_dp: int,
Validate PPO batch alignment and return sampler group and tail multiples.
- nemo_rl.algorithms.single_controller_utils.config.validate_grpo_chunk_cap_alignment(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
- policy_dp: int,
Reject a GRPO chunk cap whose samples cannot split across the policy.
GRPO passes no
group_multipleto the sampler, so a chunk can be any size from the floor up to the cap, and the training dispatch shards it withoutallow_uneven_shards. The chunk therefore has to satisfy the samesample_multiplePPO aligns to: the DP split, and under static batching the microbatch split inside each rank as well. This guards the full-cap chunk only: a cap of 3 at 2 generations per prompt is 6 samples, which fails on a DP of 4. An unaligned chunk below the cap stays reachable, as it already is from an unalignedmin_groups_for_streaming_train, which no validator covers today.Only a cap below
num_prompts_per_stepcan bind, and Forest is exempt: its producer pads each group to the policy execution row multiple, so the logical group count is not the physical row count.- Parameters:
master_config – The run configuration.
policy_dp – Data-parallel size of the policy trainer.
- Raises:
ValueError – The capped chunk’s sample count is not a multiple of the policy’s
sample_multiple.
- nemo_rl.algorithms.single_controller_utils.config.validate_single_controller_config(
- master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig,
Validate cross-section SingleController constraints before setup.
- class nemo_rl.algorithms.single_controller_utils.config.AdvantageConfig#
Internal DataPlane field mapping for advantage calculation.
- output_field: str#
‘advantages’
- reward_field: str#
‘total_reward’
- token_mask_field: str#
‘token_mask’
- sample_mask_field: str#
‘sample_mask’
- invalid_tool_call_mask_field: str#
None
- malformed_thinking_mask_field: str#
None
- mask_sample_field: str#
‘mask_sample’
- truncated_field: str#
‘truncated’
- repeated_batch_fields: list[str]#
‘field(…)’
- policy_logprobs_field: str#
‘prev_logprobs’
- generation_logprobs_field: str#
‘generation_logprobs’
- reference_logprobs_field: str#
‘reference_policy_logprobs’
- teacher_logprobs_field: str#
‘teacher_reference_logprobs’
- values_field: str#
‘values’
- returns_field: str#
‘returns’
- prompt_ids_field: str#
‘prompt_ids_for_adv’