nemo_rl.algorithms.async_utils.staleness_sampler#
Prompt-group staleness policies over a TQReplayBuffer.
A staleness policy owns the whole off-policyness contract for the SC async loop in one object, injected into both pumps:
admit(rollout-pump side): block until the next prompt batch may dispatch, then return thetarget_stepstamp for that batch (Nonewhen the policy doesn’t stamp). Owning admission here is what lets the rollout pump follow whichever sampling algorithm is selected without a second, hand-kept copy of the gating logic.select/evict(train-pump side): pick / drop prompt groups.is_on_policy/required_buffer_capacity(derived facts): what the weight-sync and capacity-validation paths need without re-reading raw knobs, so those consumers can’t drift out of sync with the sampler.drop accounting (
charge_step_for_lost_groupplus theuses_pooled_replacements/closes_short_at_exhaustion/requires_shortfall_reconciliationflags): what losing a prompt group for good costs under this policy, i.e. which step it is charged to and whether a spare pool replaces it.
PromptGroupSampler is the interface; WindowedSampler /
ReadyFirstSampler / WeightFifoSampler / InOrderSampler are the
built-in policies, one per behavior, each owning only the args that apply to it.
create_sampler builds one from a discriminated-union config (or a
module:ClassName FQN for a policy defined outside this repo) — the config’s
name is the single source of truth for which behavior runs, so there are no
cross-field knob combinations to validate.
Module Contents#
Classes#
Staleness policy shared by the SC rollout and train pumps. |
|
Sampler whose blocking wait is separate from its cursor mutation. |
|
Shared machinery for the built-in policies. |
|
Over-sampled windowed selection. |
|
Base for policies that admit exactly one dispatch batch per trainer step. |
|
Gated admission with ready-first, mixed-version selection. |
|
Gated, strict weight-version FIFO. |
|
Gated, exact batch->step matching. |
|
Functions#
Return capacity for one live batch plus each lookahead batch. |
|
Return a sampler config’s required capacity without constructing it. |
|
Import and return a custom sampler class without constructing it. |
|
Return the sampler class selected by a built-in or custom config. |
|
Whether admission can wait outside mutations while the pump refills prompts. |
|
Read the resolved policy’s selection requirement without constructing it. |
|
Whether the configured sampler replaces lost groups from a spare pool. |
|
Return a sampler class’s static replay-checkpoint capability. |
|
Return whether selection transfers rows into training ownership. |
|
Reject a custom sampler that omits a drop-accounting class attribute. |
|
Build a sampler from its config (or import one by FQN). |
Data#
API#
- nemo_rl.algorithms.async_utils.staleness_sampler._GATE_POLL_SECONDS#
0.005
- class nemo_rl.algorithms.async_utils.staleness_sampler.PromptGroupSampler#
Bases:
typing.ProtocolStaleness policy shared by the SC rollout and train pumps.
Implement this (or subclass
BaseSampler) to add a custom sampling algorithm; pointasync_rl.sampleratmodule:ClassNameto load it. A custom sampler that supports replay recovery must explicitly declaresupports_buffer_checkpoint = True. It must additionally declaresupports_training_claims = Truebefore periodic rollout snapshots may be enabled; omitting that optional capability preserves the legacy remove-on-selection behavior.- async admit(
- *,
- trainer_version_fn: Callable[[], int],
Block until the next prompt batch may dispatch.
- Parameters:
trainer_version_fn – Zero-arg accessor for the live trainer version (polled, so a blocking policy sees updates while it waits).
- Returns:
The
target_stepto stamp on this batch’s slots, orNonewhen the policy does not stamp target steps.
- async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
Pick up to
max_prompt_groupseligible groups for training.Claim-aware samplers transfer the groups from ordinary replay-buffer selection into training ownership until the controller releases them. Legacy custom samplers may still remove selected groups immediately.
- Parameters:
current_train_weight – Trainer version used by the staleness policy.
min_prompt_groups – Minimum groups needed to return a selection.
max_prompt_groups – Maximum selection size. For chunked training, this is the groups remaining in the current batch, or
async_rl.max_groups_for_streaming_trainwhen that is lower.group_multiple – Align the capped selection to this many complete groups, leaving any unaligned ready groups unclaimed.
min_remaining_groups – Defer a nonfinal selection if it would leave fewer than this many groups in
max_prompt_groups. Under a chunk cap that window is the capped chunk rather than the step, so a floor close to the cap defers until the chunk can fill. A complete aligned remainder may be selected even when below this floor, provided it meetsmin_prompt_groups. Zero preserves greedy selection. Custom streaming PPO samplers must honor this bound.
- Returns:
Selected metadata and group count, or
(None, 0)when the eligible groups do not satisfy the selection bounds.
- async evict(*, current_train_weight: int) int#
Drop groups that can no longer be selected; clear their DP rows.
- should_abort_inflight(
- *,
- start_weight_version: int,
- current_train_weight: int,
Return whether an unfinished rollout can no longer be selected.
- property is_on_policy: bool#
True when the policy admits zero staleness (sync mode).
- supports_buffer_checkpoint: ClassVar[bool]#
None
Whether completed buffered groups can be restored safely.
- supports_training_claims: ClassVar[bool]#
None
Whether selected groups remain owned until the train step commits.
- uses_pooled_replacements: ClassVar[bool]#
None
Whether dropped groups are replaced from a dedicated spare pool.
- closes_short_at_exhaustion: ClassVar[bool]#
None
Whether the last step may close on the groups left once the data runs out.
- requires_shortfall_reconciliation: ClassVar[bool]#
None
Whether a loss charged during an awaited selection must be reconciled.
True for policies that charge against the assembling cursor (it can move while selection awaits its claim); False for policies that stamp a fixed target step.
- charge_step_for_lost_group(
- *,
- target_step: Optional[int],
- assembling_version: int,
Return the step a permanently lost group is charged to, or None.
- required_buffer_capacity(groups_per_step: int) Optional[int]#
Buffer-capacity the policy needs, or
Noneif unconstrained.
- property dispatch_index: int#
Last admitted dispatch batch index.
- set_dispatch_index(resume_from_trainer_version: int) None#
Seed the cursor for checkpoints that predate exact sampler state.
- restore_dispatch_index(dispatch_index: int) None#
Restore the exact dispatch cursor from controller state.
- class nemo_rl.algorithms.async_utils.staleness_sampler.TransactionalAdmissionSampler#
Bases:
typing.ProtocolSampler whose blocking wait is separate from its cursor mutation.
- async wait_until_admissible(
- *,
- trainer_version_fn: Callable[[], int],
Wait until one admission can commit without mutating sampler state.
- commit_admission( ) Optional[int]#
Advance the cursor under a live data-plane mutation cut.
- class nemo_rl.algorithms.async_utils.staleness_sampler.BaseSampler( )#
Bases:
abc.ABCShared machinery for the built-in policies.
Owns the monotonic dispatch counter (the batch index formerly tracked as
SingleControllerActor._max_rollout_version) and the common select-finalize / weight-window-evict helpers.Initialization
- supports_buffer_checkpoint: ClassVar[bool]#
False
- supports_training_claims: ClassVar[bool]#
True
- uses_pooled_replacements: ClassVar[bool]#
True
- closes_short_at_exhaustion: ClassVar[bool]#
False
- requires_shortfall_reconciliation: ClassVar[bool]#
True
- charge_step_for_lost_group(
- *,
- target_step: Optional[int],
- assembling_version: int,
Charge the group’s own step, or the one being assembled if it has none.
- property dispatch_index: int#
Return the last admitted dispatch batch index.
- set_dispatch_index(resume_from_trainer_version: int) None#
Seed the cursor for checkpoints that predate exact sampler state.
- Parameters:
resume_from_trainer_version – Trainer version from which the run resumes. The next admitted batch receives that version.
- restore_dispatch_index(dispatch_index: int) None#
Restore the exact dispatch cursor.
- Parameters:
dispatch_index – Last admitted batch index, or
-1when no batch has been admitted. Call before the firstadmit.
- abstract property requires_single_weight_batch: bool#
Whether selection needs a complete batch from one weight version.
Concrete policies declare a boolean class attribute so setup can reject incompatible drop policies before constructing the sampler.
- abstractmethod async admit(
- *,
- trainer_version_fn: Callable[[], int],
- abstractmethod async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
- async evict(*, current_train_weight: int) int#
Default: drop ready groups below the weight window.
Skips unready (reserved-but-uncommitted) slots so eviction can’t race a concurrent
committhat re-looks-up the slot after itsawait. Policies whoseselectkey isn’t the start weight (e.g.InOrderSampler) override this so evict and select agree.
- should_abort_inflight(
- *,
- start_weight_version: int,
- current_train_weight: int,
- property is_on_policy: bool#
- property eviction_window: int#
Weight versions a group stays trainable for; older groups are evicted.
- required_buffer_capacity(groups_per_step: int) Optional[int]#
- _eviction_window() int#
Weight-version span kept selectable; drives the default
evict.
- static _validate_group_bounds(
- min_prompt_groups: int,
- max_prompt_groups: int,
- *,
- min_remaining_groups: int = 0,
- async _finalize_selection(
- valid_idxs: list[int],
- min_prompt_groups: int,
- max_prompt_groups: int,
- *,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
Cap, align, and claim the selected groups for training.
Returns the largest aligned selection up to
max_prompt_groups, or(None, 0)belowmin_prompt_groups. Unaligned tail groups stay unclaimed, preserving their rows and readiness for a later chunk. A selection leaving a nonzero tail belowmin_remaining_groupsis also deferred before any groups are claimed.
- class nemo_rl.algorithms.async_utils.staleness_sampler.WindowedSampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- *,
- max_staleness_versions: int,
- sample_freshest_first: bool = False,
- min_groups_for_streaming_train: Optional[int] = None,
Bases:
nemo_rl.algorithms.async_utils.staleness_sampler.BaseSamplerOver-sampled windowed selection.
Rollout never gates on the trainer version — the pump keeps producing and samples aged past the window are evicted.
selecttakes any ready group within[train_weight - max_staleness_versions, train_weight], optionally freshest-first.Initialization
- supports_buffer_checkpoint: ClassVar[bool]#
True
- requires_single_weight_batch: ClassVar[bool]#
False
- uses_pooled_replacements: ClassVar[bool]#
False
- closes_short_at_exhaustion: ClassVar[bool]#
True
- _eviction_window() int#
- required_buffer_capacity(groups_per_step: int) Optional[int]#
- charge_step_for_lost_group(
- *,
- target_step: Optional[int],
- assembling_version: int,
Charge nothing: a step takes any ready groups, so none of them is missing.
- should_abort_inflight(
- *,
- start_weight_version: int,
- current_train_weight: int,
- async wait_until_admissible(
- *,
- trainer_version_fn: Callable[[], int],
Return immediately because buffer capacity is this policy’s gate.
- commit_admission( ) Optional[int]#
Return the unstamped admission result without changing a cursor.
- async admit(
- *,
- trainer_version_fn: Callable[[], int],
- async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
- nemo_rl.algorithms.async_utils.staleness_sampler._gated_required_buffer_capacity(
- groups_per_step: int,
- *,
- gate_window: int,
Return capacity for one live batch plus each lookahead batch.
- class nemo_rl.algorithms.async_utils.staleness_sampler._GatedSampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- *,
- gate_window: int,
Bases:
nemo_rl.algorithms.async_utils.staleness_sampler.BaseSamplerBase for policies that admit exactly one dispatch batch per trainer step.
The gate bounds how far generation may run ahead of the trainer (
gate_windowversions of lookahead).Initialization
- set_gate_window(gate_window: int) None#
- _eviction_window() int#
- required_buffer_capacity(groups_per_step: int) Optional[int]#
- can_admit(*, trainer_version: int) bool#
Whether the gate permits admission without advancing the cursor.
- async wait_until_admissible(
- *,
- trainer_version_fn: Callable[[], int],
Wait for the gate without advancing the durable dispatch cursor.
- commit_admission( ) Optional[int]#
Advance the cursor after the controller enters its mutation cut.
- _commit_admission() Optional[int]#
Advance admission for the legacy monolithic API.
- async admit(
- *,
- trainer_version_fn: Callable[[], int],
- _stamp() Optional[int]#
- class nemo_rl.algorithms.async_utils.staleness_sampler.ReadyFirstSampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- *,
- max_staleness_versions: int,
Bases:
nemo_rl.algorithms.async_utils.staleness_sampler._GatedSamplerGated admission with ready-first, mixed-version selection.
Admission limits generation to
max_staleness_versionsdispatch batches ahead of the current trainer version. Selection remains ready-first across weight versions instead of draining one version at a time: every ready group generated by a policy version no newer than the trainer stays selectable, including late stragglers outside the admission window, so no rollout is ever discarded.Initialization
- supports_buffer_checkpoint: ClassVar[bool]#
True
- requires_single_weight_batch: ClassVar[bool]#
False
- async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
- async evict(*, current_train_weight: int) int#
- class nemo_rl.algorithms.async_utils.staleness_sampler.WeightFifoSampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- *,
- max_staleness_versions: int,
Bases:
nemo_rl.algorithms.async_utils.staleness_sampler._GatedSamplerGated, strict weight-version FIFO.
selectdrains the oldest in-windowstart_weightfirst and waits for that weight’s batch to fill. Evict uses the weight window (default).Initialization
- supports_buffer_checkpoint: ClassVar[bool]#
True
- requires_single_weight_batch: ClassVar[bool]#
True
- uses_pooled_replacements: ClassVar[bool]#
False
- async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
- class nemo_rl.algorithms.async_utils.staleness_sampler.InOrderSampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- *,
- max_lookahead_versions: int,
- warmup_lookahead_versions: Optional[int] = None,
Bases:
nemo_rl.algorithms.async_utils.staleness_sampler._GatedSamplerGated, exact batch->step matching.
Each dispatched batch is stamped with its dispatch index as
target_step;selecttakes the batch whosetarget_stepequals the trainer version (the staleness window is not used for selection).evictis keyed ontarget_step— not the start weight — so a slot whose target step is still upcoming is never dropped early, and evict/select can’t disagree.warmup_lookahead_versions widens the gate while the PPO policy is frozen, so more batches stay in flight during critic warmup. The driver retunes the window every step and shrinks it back to max_lookahead_versions once the policy starts training, so the widened lookahead does not turn into permanent extra staleness. Buffer capacity is sized for the peak of the two.
Initialization
- supports_buffer_checkpoint: ClassVar[bool]#
True
- requires_single_weight_batch: ClassVar[bool]#
False
- uses_pooled_replacements: ClassVar[bool]#
True
- closes_short_at_exhaustion: ClassVar[bool]#
False
- requires_shortfall_reconciliation: ClassVar[bool]#
False
- charge_step_for_lost_group(
- *,
- target_step: Optional[int],
- assembling_version: int,
Charge the group’s own step: each batch is trained as exactly one step.
- required_buffer_capacity(groups_per_step: int) Optional[int]#
- _stamp() Optional[int]#
- async select(
- *,
- current_train_weight: int,
- min_prompt_groups: int,
- max_prompt_groups: int,
- group_multiple: int = 1,
- min_remaining_groups: int = 0,
- async evict(*, current_train_weight: int) int#
- class nemo_rl.algorithms.async_utils.staleness_sampler.WindowedSamplerConfig#
Bases:
pydantic.BaseModel- name: Literal[windowed]#
‘windowed’
- max_staleness_versions: pydantic.NonNegativeInt#
1
- sample_freshest_first: bool#
False
- class nemo_rl.algorithms.async_utils.staleness_sampler.ReadyFirstSamplerConfig#
Bases:
pydantic.BaseModel- name: Literal[ready_first]#
‘ready_first’
- max_staleness_versions: pydantic.NonNegativeInt#
1
- class nemo_rl.algorithms.async_utils.staleness_sampler.WeightFifoSamplerConfig#
Bases:
pydantic.BaseModel- name: Literal[weight_fifo]#
‘weight_fifo’
- max_staleness_versions: pydantic.NonNegativeInt#
1
- class nemo_rl.algorithms.async_utils.staleness_sampler.InOrderSamplerConfig#
Bases:
pydantic.BaseModel- name: Literal[in_order]#
‘in_order’
- max_lookahead_versions: pydantic.NonNegativeInt#
1
- warmup_lookahead_versions: Optional[pydantic.NonNegativeInt]#
None
- validate_warmup_lookahead() nemo_rl.algorithms.async_utils.staleness_sampler.InOrderSamplerConfig#
- property peak_lookahead_versions: int#
Widest window the run can reach; what buffer capacity must cover.
- class nemo_rl.algorithms.async_utils.staleness_sampler.CustomSamplerConfig#
Bases:
pydantic.BaseModel- name: Literal[custom]#
‘custom’
- target: str#
None
- nemo_rl.algorithms.async_utils.staleness_sampler.SamplerConfig#
None
- nemo_rl.algorithms.async_utils.staleness_sampler.required_buffer_capacity_for_config(
- cfg: nemo_rl.algorithms.async_utils.staleness_sampler.SamplerConfig,
- groups_per_step: int,
- *,
- min_groups_for_streaming_train: int,
Return a sampler config’s required capacity without constructing it.
- nemo_rl.algorithms.async_utils.staleness_sampler._custom_sampler_class( ) type#
Import and return a custom sampler class without constructing it.
- nemo_rl.algorithms.async_utils.staleness_sampler._sampler_class_for_config( ) type#
Return the sampler class selected by a built-in or custom config.
- nemo_rl.algorithms.async_utils.staleness_sampler.sampler_supports_transactional_admission( ) bool#
Whether admission can wait outside mutations while the pump refills prompts.
- nemo_rl.algorithms.async_utils.staleness_sampler.sampler_requires_single_weight_batch( ) bool#
Read the resolved policy’s selection requirement without constructing it.
Custom subclasses inherit their selection policy’s declaration. Missing or instance-only declarations fail before a run can enable filtering or drops.
- nemo_rl.algorithms.async_utils.staleness_sampler.sampler_uses_pooled_replacements( ) bool#
Whether the configured sampler replaces lost groups from a spare pool.
Reads the resolved class without constructing it. Every sampler declares the member: built-ins as a ClassVar, custom samplers enforced by
require_drop_accounting_membersduring config validation, which runs before this reader, so the attribute is always present.
- nemo_rl.algorithms.async_utils.staleness_sampler.sampler_supports_buffer_checkpoint( ) bool#
Return a sampler class’s static replay-checkpoint capability.
Custom classes are imported but not instantiated, allowing setup to fail before allocating cluster resources or triggering constructor side effects.
- nemo_rl.algorithms.async_utils.staleness_sampler.sampler_supports_training_claims( ) bool#
Return whether selection transfers rows into training ownership.
Built-in samplers use :meth:
TQReplayBuffer.claim_for_training. Custom samplers retain the legacy local-removal contract unless they explicitly opt in, so enabling periodic snapshots cannot silently assume ownership metadata that the sampler never created.
- nemo_rl.algorithms.async_utils.staleness_sampler.require_drop_accounting_members( ) None#
Reject a custom sampler that omits a drop-accounting class attribute.
Dynamic sampling (and nonzero drop budgets) read these members directly off the sampler. A custom class that leaves any out — or sets them only in
__init__, which is invisible to this pre-construction check — fails here rather than silently getting gated defaults. Built-in samplers are skipped.
- nemo_rl.algorithms.async_utils.staleness_sampler._DROP_ACCOUNTING_MEMBERS#
(‘uses_pooled_replacements’, ‘closes_short_at_exhaustion’, ‘requires_shortfall_reconciliation’, ‘cha…
- nemo_rl.algorithms.async_utils.staleness_sampler.create_sampler(
- buffer: nemo_rl.algorithms.async_utils.replay_buffer.TQReplayBuffer,
- cfg: nemo_rl.algorithms.async_utils.staleness_sampler.SamplerConfig,
- *,
- min_groups_for_streaming_train: Optional[int] = None,
Build a sampler from its config (or import one by FQN).
- Parameters:
buffer – Shared TQReplayBuffer holding the candidate slots.
cfg – Discriminated sampler config selecting the policy.
min_groups_for_streaming_train – Streaming floor used by the windowed sampler.