nemo_gym.models.token_capture.records

Bases: BaseModel

One model call the engine refused, so the rollout has no record for it.

A refusal is not a lost capture: the call never generated, and the client was told why. On the local capture path the refusal resolves the call’s pre-dispatch intent, so the rollout is not incomplete; the record is what tells a consumer where the engine refused. code is the error code the client received: CONTEXT_LENGTH_EXCEEDED_ERROR_CODE for a prompt that does not fit the context window, None when the envelope carried no code (an output-token parameter the engine rejected). parent_call_id is the recorded call the refused request continued, which places the refusal on one branch of the rollout.

Bases: enum.Enum

Describe whether a model call has a proven captured predecessor.

Bases: BaseModel

Store one model call’s content and token metadata.

The capture key identifies the training sample: one attempt of one rollout. The model call id joins evaluation context. output_items preserves assistant text and tool calls. Text-based penalties require that content. Token arrays are stored once at the top level. token_item_index identifies their original output item. A trajectory builder can restore chain-correct token fields there.

nemo_gym.models.token_capture.records.TokenEntry._read_version_1_key(
    data: typing.Any
) -> typing.Any

nemo_gym.models.token_capture.records.TokenEntry._refuse_a_newer_record() -> 'TokenEntry'

Put the capture’s truncation verdict on a harness response, under one precedence rule.

incomplete_reason is the incomplete_details.reason the capture derived for the delivered chain. On the local path it is incomplete_reason_for_finish_reason of the terminal call’s finish reason, or max_output_tokens when the engine refused the request that continued the terminal call for the context window (builder.project_main_chain_response). Under worker custody it is staging.rebuild.LinearizedRow.incomplete_reason, which carries the finish-reason mapping only: the ledger row of a refused call records no code and no lineage, so a rollout rebuilt from staged records reports no verdict for a refusal. None means the chain ended on its own and leaves the response alone. The verdict is written only when the harness reported no status or completed: a harness that reported incomplete or failed itself gave the more specific verdict and keeps its status and incomplete_details. Returns whether the response changed. Gym’s delivery and a framework that rebuilds from staged records both apply this precedence rule, so a verdict that comes from the finish reason is the same whichever placement captured it.

Digest of an exact token sequence.

The builder verifies a claimed parent by hashing the corresponding prompt prefix. A mismatch quarantines the call. This prevents stale or interleaved records from merging silently.

The full sequence a child of this call must start with.

Delta records require parent-chain reconstruction.

Stable, length-delimited, big-endian encoding of a token sequence.

Independent implementations can hash the same bytes. One vectorized pack replaces the per-token loop. The byte layout remains identical to the original encoding.

Pull the token-id fields off a served response, or None if absent.

Handle Responses output items and Chat Completions messages. Exactly one item may carry token metadata. Return None when no item carries token ids.

Return the incomplete_details.reason a response that stopped for finish_reason reports.

length becomes max_output_tokens and content_filter stays content_filter, the verdict the Responses converter gives a chat completion that ended the same way. A complete ending (stop, tool_calls), an unknown reason, and None return None.

Return why a served response stopped, in the Chat Completions vocabulary.

A chat payload states it directly as choices[0].finish_reason. An Anthropic Messages payload states stop_reason; map it back through the converter’s table. A Responses payload carries no finish reason. The converter maps length and content_filter to incomplete_details.reason; reverse that map. A completed Responses payload stopped at a tool call (tool_calls) or at the end of its text (stop). Return None when the payload is none of these shapes or is incomplete without a stated reason.

Normalize a served response to a list of content-bearing Responses output items.

Responses payloads already carry output. Chat payloads carry choices[*].message. Anthropic Messages payloads carry top-level assistant content. Wrap each assistant message as a Responses message item.

Fill token lineage and the request-time parent decision.

cum_len and digest describe the full sequence. A delta entry must pass cumulative explicitly. parent_resolution=None preserves records built by compatibility callers.

Drop the token arrays from output items, keeping the content.

Return the stripped items and the index of their token-bearing item. Capture requires exactly one token-bearing item. The arrays are held once on the entry. Storing them again per item would roughly double the record size.