nemo_gym.token_id_capture.delivery

Build a token-bearing record from one finished rollout.

The finalizer freezes the rollout’s captured model calls. It rebuilds response.output from that frozen snapshot. It does not retire the snapshot. The caller may retire it only after durable handoff. Retirement uses the frozen snapshot_id and version.

The caller provides both the rollout record and its TokenSource. Gym resolves its source from Gym configuration. A training framework may provide a source from its own data plane. Training correlation must preserve /ng-rollout/<id>/training-token-capture.

Existing token ids are the policy’s sampled data. The finalizer leaves a rollout containing any token ids unchanged. Failed or masked builds retain their capture evidence.

Module Contents

Functions

Name	Description
`_unusable`	Mask a rollout that needed token ids and could not get them.
`capture_build_can_retire`	Whether a successful build consumed a frozen snapshot.
`finalize_rollout_token_capture`	Rebuild one finished rollout record’s `response.output` from its recorded token ids.
`retire_rollout_token_capture`	Retire a frozen snapshot after durable handoff.
`rollout_carries_token_ids`	Whether this rollout already holds what training needs.

Data

MASK_SAMPLE_KEY

TOKEN_CAPTURE_KEY

_REDUNDANT_CAPTURE_KEY

API

nemo_gym.token_id_capture.delivery._unusable(
    result: dict,
    error: str,
    message: str
) -> dict

Mask a rollout that needed token ids and could not get them.

An unmasked rollout would appear healthy until it reaches the trainer. The record retains the reason for aggregate reporting.

nemo_gym.token_id_capture.delivery.capture_build_can_retire(
    built: dict | None
) -> bool

Whether a successful build consumed a frozen snapshot.

nemo_gym.token_id_capture.delivery.finalize_rollout_token_capture(
    result: dict,
    source: nemo_gym.token_id_capture.protocols.TokenSource | None
) -> dict | None

async

Rebuild one finished rollout record’s response.output from its recorded token ids.

Call this after the harness and verifier finish the record. The function mutates result in place. It replaces only response.output. It preserves the reward and all other harness and verifier output.

The function freezes capture records through source. It rebuilds from that frozen snapshot. It never retires the snapshot. A None source means this caller does not rebuild. A rollout that already carries token ids is left unchanged. Its redundant frozen capture remains eligible for retirement after handoff.

The function never raises. Missing or ambiguous tokens cause masking. Failed or masked builds retain their frozen evidence.

Return the build with its rebuilt response, metrics, and optional error. Return None when no source exists. An unusable build has no rebuilt response and sets mask_sample.

nemo_gym.token_id_capture.delivery.retire_rollout_token_capture(
    rollout_id: str,
    source: nemo_gym.token_id_capture.protocols.TokenSource | None,
    built: dict | None
) -> bool

async

Retire a frozen snapshot after durable handoff.

The caller owns the durability boundary. Call this only after downstream acceptance or a local fsync. Retirement uses the frozen snapshot_id and version. Failed or masked builds remain as diagnostic evidence.

nemo_gym.token_id_capture.delivery.rollout_carries_token_ids(
    result: dict
) -> bool

Whether this rollout already holds what training needs.

Return true when any output item carries generated token ids. These ids are what the policy sampled. They take precedence over a reconstruction that may differ. Partial token coverage must remain visible instead of being overwritten.

nemo_gym.token_id_capture.delivery.MASK_SAMPLE_KEY = 'mask_sample'

nemo_gym.token_id_capture.delivery.TOKEN_CAPTURE_KEY = '_ng_token_capture'

nemo_gym.token_id_capture.delivery._REDUNDANT_CAPTURE_KEY = '_redundant_capture'