nemo_gym.token_id_capture.staging.digest

Encode staged records for integrity verification.

The staged-record digest covers the call metadata and token columns. Optional extras are encoded separately. Their digest is included in the staged-record digest. This lets one consumer verify token data without loading extras. A later consumer can verify extras against the same recorded digest.

All values use versioned, length-delimited encodings. Masks and log probabilities use IEEE-754 float32 bit patterns. Extras use a typed binary encoding instead of JSON serialization.

Module Contents

Functions

Name	Description
`_encode_bytes`	-
`_encode_extra`	-
`_encode_float32_values`	-
`_encode_optional_text`	-
`_encode_present_digest`	-
`_encode_text`	-
`_encode_uint`	-
`_validate_digest`	-
`build_staging_delta`	Slice a full prompt/generation pair into the next staged delta.
`compute_chain_hash`	Chain one staged delta onto its parent’s chain hash.
`compute_extras_digest`	Digest a normalized JSON-like extras envelope.
`compute_staging_digest`	Compute the v2 digest for one staged call delta.
`encode_token_ids`	Encode token IDs as a length followed by unsigned big-endian values.
`hash_token_ids`	Hash one exact cumulative token prefix.

Data

EMPTY_EXTRAS_DIGEST

EXTRAS_DIGEST_VERSION

STAGING_DIGEST_VERSION

STAGING_SCHEMA_VERSION

_CALL_DIGEST_DOMAIN

_CHAIN_DIGEST_DOMAIN

_EXTRAS_DIGEST_DOMAIN

_HEX_DIGEST_LENGTH

_TOKEN_DIGEST_DOMAIN

API

nemo_gym.token_id_capture.staging.digest._encode_bytes(
    value: bytes
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_extra(
    value: typing.Any
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_float32_values(
    values: collections.abc.Sequence[float],
    field: str
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_optional_text(
    value: str | None
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_present_digest(
    value: str,
    field: str
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_text(
    value: str
) -> bytes

nemo_gym.token_id_capture.staging.digest._encode_uint(
    value: int,
    field: str
) -> bytes

nemo_gym.token_id_capture.staging.digest._validate_digest(
    value: str,
    field: str
) -> None

nemo_gym.token_id_capture.staging.build_staging_delta(
    prompt_token_ids: list[int],
    generated_token_ids: list[int],
    generated_log_probs: list[float],
    prev_len: int
) -> tuple[list[int], list[float], list[float]]

Slice a full prompt/generation pair into the next staged delta.

nemo_gym.token_id_capture.staging.compute_chain_hash(
    parent_chain_hash: str | None,
    token_ids_delta: collections.abc.Sequence[int]
) -> str

Chain one staged delta onto its parent’s chain hash.

chain_hash_N = H(chain_hash_{N-1} || token_ids_delta_N) lets every consumer verify parent-child token continuity without materializing the cumulative sequence. A root delta chains from None. The marker byte keeps a root encoding from colliding with a child encoding.

Parameters:

parent_chain_hash

str | None

The parent call’s chain hash, or None for a root.

token_ids_delta

Sequence[int]

This call’s exact staged token delta.

Returns: str

The lowercase SHA-256 hex chain hash for this call.

nemo_gym.token_id_capture.staging.compute_extras_digest(
    extras: collections.abc.Mapping[str, typing.Any] | None
) -> str

Digest a normalized JSON-like extras envelope.

Supported values are None, exact bool/int/float/str scalars, lists, and string-keyed dictionaries. This deliberately rejects numpy scalars, tuples, bytes, NaN, and infinity so different runtimes cannot silently choose different encodings.

nemo_gym.token_id_capture.staging.compute_staging_digest(
    schema_version: int,
    digest_version: int,
    extras_digest_version: int,
    rollout_id: str,
    model_call_id: str,
    parent_call_id: str | None,
    mode: str,
    prev_len: int,
    delta_len: int,
    cum_len: int,
    weight_version: int,
    token_ids_delta: collections.abc.Sequence[int],
    token_mask_delta: collections.abc.Sequence[float],
    generation_log_probs_delta: collections.abc.Sequence[float],
    extras_digest: str,
    chain_hash: str,
    cumulative_hash: str
) -> str

Compute the v2 digest for one staged call delta.

nemo_gym.token_id_capture.staging.encode_token_ids(
    token_ids: collections.abc.Sequence[int]
) -> bytes

Encode token IDs as a length followed by unsigned big-endian values.

nemo_gym.token_id_capture.staging.hash_token_ids(
    token_ids: collections.abc.Sequence[int]
) -> str

Hash one exact cumulative token prefix.

nemo_gym.token_id_capture.staging.digest.EMPTY_EXTRAS_DIGEST = compute_extras_digest(None)

nemo_gym.token_id_capture.staging.digest.EXTRAS_DIGEST_VERSION = 1

nemo_gym.token_id_capture.staging.digest.STAGING_DIGEST_VERSION = 2

nemo_gym.token_id_capture.staging.digest.STAGING_SCHEMA_VERSION = 2

nemo_gym.token_id_capture.staging.digest._CALL_DIGEST_DOMAIN = b'nemo-gym-staging-call-v2'

nemo_gym.token_id_capture.staging.digest._CHAIN_DIGEST_DOMAIN = b'nemo-gym-staging-chain-v1'

nemo_gym.token_id_capture.staging.digest._EXTRAS_DIGEST_DOMAIN = b'nemo-gym-staging-extras-v1'

nemo_gym.token_id_capture.staging.digest._HEX_DIGEST_LENGTH = 64

nemo_gym.token_id_capture.staging.digest._TOKEN_DIGEST_DOMAIN = b'nemo-gym-staging-prefix-v2'