> This page is for version 0.4.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.models.token_capture.records

Define training-token records extracted from served responses.

A `TokenEntry` contains one model call's training data.
It stores the exact prompt token ids.
It stores generated token ids and their log probabilities.
Evaluation uses a separate `ModelCallRecord`.
Evaluation records do not carry token arrays.
Both records share a `model_call_id`.

## Module Contents

### Classes

| Name                                                                                      | Description                                                             |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [`EngineRefusalRecord`](#nemo_gym-models-token_capture-records-EngineRefusalRecord)       | One model call the engine refused, so the rollout has no record for it. |
| [`ParentResolutionStatus`](#nemo_gym-models-token_capture-records-ParentResolutionStatus) | Describe whether a model call has a proven captured predecessor.        |
| [`TokenEntry`](#nemo_gym-models-token_capture-records-TokenEntry)                         | Store one model call's content and token metadata.                      |

### Functions

| Name                                                                                                                | Description                                                                                 |
| ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`apply_capture_incomplete_verdict`](#nemo_gym-models-token_capture-records-apply_capture_incomplete_verdict)       | Put the capture's truncation verdict on a harness response, under one precedence rule.      |
| [`compute_digest`](#nemo_gym-models-token_capture-records-compute_digest)                                           | Digest of an exact token sequence.                                                          |
| [`cumulative_tokens`](#nemo_gym-models-token_capture-records-cumulative_tokens)                                     | The full sequence a child of this call must start with.                                     |
| [`encode_token_ids`](#nemo_gym-models-token_capture-records-encode_token_ids)                                       | Stable, length-delimited, big-endian encoding of a token sequence.                          |
| [`extract_token_fields`](#nemo_gym-models-token_capture-records-extract_token_fields)                               | Pull the token-id fields off a served response, or `None` if absent.                        |
| [`incomplete_reason_for_finish_reason`](#nemo_gym-models-token_capture-records-incomplete_reason_for_finish_reason) | Return the `incomplete_details.reason` a response that stopped for `finish_reason` reports. |
| [`response_finish_reason`](#nemo_gym-models-token_capture-records-response_finish_reason)                           | Return why a served response stopped, in the Chat Completions vocabulary.                   |
| [`response_to_output_items`](#nemo_gym-models-token_capture-records-response_to_output_items)                       | Normalize a served response to a list of content-bearing Responses output items.            |
| [`stamp_lineage`](#nemo_gym-models-token_capture-records-stamp_lineage)                                             | Fill token lineage and the request-time parent decision.                                    |
| [`strip_token_fields`](#nemo_gym-models-token_capture-records-strip_token_fields)                                   | Drop the token arrays from output items, keeping the content.                               |

### Data

[`CONTEXT_LENGTH_EXCEEDED_ERROR_CODE`](#nemo_gym-models-token_capture-records-CONTEXT_LENGTH_EXCEEDED_ERROR_CODE)

[`DIGEST_VERSION`](#nemo_gym-models-token_capture-records-DIGEST_VERSION)

[`ENGINE_REFUSED_CALL_REASON`](#nemo_gym-models-token_capture-records-ENGINE_REFUSED_CALL_REASON)

[`FINISH_REASON_DISAGREEMENT_REASON`](#nemo_gym-models-token_capture-records-FINISH_REASON_DISAGREEMENT_REASON)

[`LEDGER_ROW_MISSING_CHAIN_HASH_REASON`](#nemo_gym-models-token_capture-records-LEDGER_ROW_MISSING_CHAIN_HASH_REASON)

[`LEDGER_ROW_MISSING_RESPONSE_ID_REASON`](#nemo_gym-models-token_capture-records-LEDGER_ROW_MISSING_RESPONSE_ID_REASON)

[`MAX_OUTPUT_TOKENS_INCOMPLETE_REASON`](#nemo_gym-models-token_capture-records-MAX_OUTPUT_TOKENS_INCOMPLETE_REASON)

[`NAMESPACE_TOOL_DELIMITER`](#nemo_gym-models-token_capture-records-NAMESPACE_TOOL_DELIMITER)

[`TOKEN_ENTRY_MIN_SCHEMA_VERSION`](#nemo_gym-models-token_capture-records-TOKEN_ENTRY_MIN_SCHEMA_VERSION)

[`TOKEN_ENTRY_RECORD_SCHEMA_VERSION`](#nemo_gym-models-token_capture-records-TOKEN_ENTRY_RECORD_SCHEMA_VERSION)

[`TOKEN_FIELDS`](#nemo_gym-models-token_capture-records-TOKEN_FIELDS)

[`UNCOMMITTED_CALL_REASON`](#nemo_gym-models-token_capture-records-UNCOMMITTED_CALL_REASON)

[`UNRESOLVED_PARENT_REASON`](#nemo_gym-models-token_capture-records-UNRESOLVED_PARENT_REASON)

[`UNSERVED_CALL_FAILURE_REASONS`](#nemo_gym-models-token_capture-records-UNSERVED_CALL_FAILURE_REASONS)

[`_DIGEST_DOMAIN`](#nemo_gym-models-token_capture-records-_DIGEST_DOMAIN)

[`_EMPTY_DIGEST`](#nemo_gym-models-token_capture-records-_EMPTY_DIGEST)

[`_FINISH_REASON_TO_INCOMPLETE_REASON`](#nemo_gym-models-token_capture-records-_FINISH_REASON_TO_INCOMPLETE_REASON)

[`_INCOMPLETE_REASON_TO_FINISH_REASON`](#nemo_gym-models-token_capture-records-_INCOMPLETE_REASON_TO_FINISH_REASON)

[`_STOP_REASON_TO_FINISH_REASON`](#nemo_gym-models-token_capture-records-_STOP_REASON_TO_FINISH_REASON)

[`_TOOL_CALL_ITEM_TYPES`](#nemo_gym-models-token_capture-records-_TOOL_CALL_ITEM_TYPES)

### API

```python
class nemo_gym.models.token_capture.records.EngineRefusalRecord()
```

**Bases:** `BaseModel`

One model call the engine refused, so the rollout has no record for it.

A refusal is not a lost capture: the call never generated, and the client was told why.
On the local capture path the refusal resolves the call's pre-dispatch intent, so the
rollout is not incomplete; the record is what tells a consumer where the engine refused.
`code` is the error code the client received: `CONTEXT_LENGTH_EXCEEDED_ERROR_CODE` for
a prompt that does not fit the context window, `None` when the envelope carried no code
(an output-token parameter the engine rejected). `parent_call_id` is the recorded call the
refused request continued, which places the refusal on one branch of the rollout.

**`code`** `str | None = None`

---

**`created_at`** `float = 0.0`

---

**`model_call_id`** `str`

---

**`model_config`** `= ConfigDict(extra='allow')`

---

**`parent_call_id`** `str | None = None`

---

**`parent_resolution`** `ParentResolutionStatus | None = None`

---

```python
class nemo_gym.models.token_capture.records.ParentResolutionStatus
```

**Bases:** `enum.Enum`

Describe whether a model call has a proven captured predecessor.

**`RESOLVED`** `= 'resolved'`

---

**`ROOT`** `= 'root'`

---

**`UNRESOLVED`** `= 'unresolved'`

---

```python
class nemo_gym.models.token_capture.records.TokenEntry()
```

**Bases:** `BaseModel`

Store one model call's content and token metadata.

The capture key identifies the training sample: one attempt of one rollout.
The model call id joins evaluation context.
`output_items` preserves assistant text and tool calls.
Text-based penalties require that content.
Token arrays are stored once at the top level.
`token_item_index` identifies their original output item.
A trajectory builder can restore chain-correct token fields there.

**`capture_key`** `str`

---

**`continuation_context_digest`** `str = ''`

---

**`continuation_context_len`** `int = 0`

---

**`continuation_fingerprint`** `str = ''`

---

**`created_at`** `float = 0.0`

---

**`cum_len`** `int | None = None`

---

**`digest`** `str | None = None`

---

**`fingerprint_version`** `int | None = None`

---

**`finish_reason`** `str | None = None`

---

**`generation_log_probs`** `list[float]`

---

**`generation_token_ids`** `list[int]`

---

**`model`** `str = ''`

---

**`model_call_id`** `str`

---

**`model_config`** `= ConfigDict(extra='allow')`

---

**`output_items`** `list[dict] = Field(default_factory=list)`

---

**`parent_call_id`** `str | None = None`

---

**`parent_resolution`** `ParentResolutionStatus | None = None`

---

**`parent_resolution_reason`** `str = ''`

---

**`prefix_requested`** `bool = False`

---

**`prefix_supplied`** `bool = False`

---

**`prompt_is_delta`** `bool = False`

---

**`prompt_token_ids`** `list[int]`

---

**`response_id`** `str | None = None`

---

**`routed_experts`** `Any | None = None`

---

**`schema_version`** `int = TOKEN_ENTRY_RECORD_SCHEMA_VERSION`

---

**`token_item_index`** `int | None = None`

---

```python
nemo_gym.models.token_capture.records.TokenEntry._read_version_1_key(
    data: typing.Any
) -> typing.Any
```

classmethod

Read the capture key of a version 1 row, which names it `rollout_id`.

```python
nemo_gym.models.token_capture.records.TokenEntry._refuse_a_newer_record() -> 'TokenEntry'
```

Accept older records and reject newer records.

Missing older fields use their defaults.
Unknown newer fields may change token semantics.
Rejecting them prevents silent training corruption.

```python
nemo_gym.models.token_capture.records.apply_capture_incomplete_verdict(
    response: dict,
    incomplete_reason: str | None
) -> bool
```

Put the capture's truncation verdict on a harness response, under one precedence rule.

`incomplete_reason` is the `incomplete_details.reason` the capture derived for the
delivered chain. On the local path it is `incomplete_reason_for_finish_reason` of the
terminal call's finish reason, or `max_output_tokens` when the engine refused the request
that continued the terminal call for the context window
(`builder.project_main_chain_response`). Under worker custody it is
`staging.rebuild.LinearizedRow.incomplete_reason`, which carries the finish-reason mapping
only: the ledger row of a refused call records no code and no lineage, so a rollout rebuilt
from staged records reports no verdict for a refusal.
`None` means the chain ended on its own and leaves the response alone. The verdict is
written only when the harness reported no status or `completed`: a harness that reported
`incomplete` or `failed` itself gave the more specific verdict and keeps its `status` and
`incomplete_details`. Returns whether the response changed. Gym's delivery and a framework
that rebuilds from staged records both apply this precedence rule, so a verdict that comes
from the finish reason is the same whichever placement captured it.

```python
nemo_gym.models.token_capture.records.compute_digest(
    token_ids: list[int]
) -> str
```

Digest of an exact token sequence.

The builder verifies a claimed parent by hashing the corresponding prompt prefix.
A mismatch quarantines the call.
This prevents stale or interleaved records from merging silently.

```python
nemo_gym.models.token_capture.records.cumulative_tokens(
    entry: nemo_gym.models.token_capture.records.TokenEntry
) -> list[int]
```

The full sequence a child of this call must start with.

Delta records require parent-chain reconstruction.

```python
nemo_gym.models.token_capture.records.encode_token_ids(
    token_ids: list[int]
) -> bytes
```

Stable, length-delimited, big-endian encoding of a token sequence.

Independent implementations can hash the same bytes.
One vectorized pack replaces the per-token loop.
The byte layout remains identical to the original encoding.

```python
nemo_gym.models.token_capture.records.extract_token_fields(
    response_json: dict
) -> dict | None
```

Pull the token-id fields off a served response, or `None` if absent.

Handle Responses output items and Chat Completions messages.
Exactly one item may carry token metadata.
Return `None` when no item carries token ids.

```python
nemo_gym.models.token_capture.records.incomplete_reason_for_finish_reason(
    finish_reason: str | None
) -> str | None
```

Return the `incomplete_details.reason` a response that stopped for `finish_reason` reports.

`length` becomes `max_output_tokens` and `content_filter` stays `content_filter`, the
verdict the Responses converter gives a chat completion that ended the same way. A complete
ending (`stop`, `tool_calls`), an unknown reason, and `None` return `None`.

```python
nemo_gym.models.token_capture.records.response_finish_reason(
    payload: dict
) -> str | None
```

Return why a served response stopped, in the Chat Completions vocabulary.

A chat payload states it directly as `choices[0].finish_reason`.
An Anthropic Messages payload states `stop_reason`; map it back through the converter's table.
A Responses payload carries no finish reason.
The converter maps `length` and `content_filter` to `incomplete_details.reason`; reverse that map.
A completed Responses payload stopped at a tool call (`tool_calls`) or at the end of its text (`stop`).
Return `None` when the payload is none of these shapes or is incomplete without a stated reason.

```python
nemo_gym.models.token_capture.records.response_to_output_items(
    payload: dict
) -> list[dict]
```

Normalize a served response to a list of content-bearing Responses output items.

Responses payloads already carry `output`.
Chat payloads carry `choices[*].message`.
Anthropic Messages payloads carry top-level assistant content.
Wrap each assistant message as a Responses `message` item.

```python
nemo_gym.models.token_capture.records.stamp_lineage(
    entry: nemo_gym.models.token_capture.records.TokenEntry,
    parent_call_id: str | None,
    parent_resolution: nemo_gym.models.token_capture.records.ParentResolutionStatus | None = None,
    cumulative: list[int] | None = None
) -> nemo_gym.models.token_capture.records.TokenEntry
```

Fill token lineage and the request-time parent decision.

`cum_len` and `digest` describe the full sequence.
A delta entry must pass `cumulative` explicitly.
`parent_resolution=None` preserves records built by compatibility callers.

```python
nemo_gym.models.token_capture.records.strip_token_fields(
    items: list[dict]
) -> tuple[list[dict], int | None]
```

Drop the token arrays from output items, keeping the content.

Return the stripped items and the index of their token-bearing item.
Capture requires exactly one token-bearing item.
The arrays are held once on the entry.
Storing them again per item would roughly double the record size.

```python
nemo_gym.models.token_capture.records.CONTEXT_LENGTH_EXCEEDED_ERROR_CODE = 'context_length_exceeded'
```

```python
nemo_gym.models.token_capture.records.DIGEST_VERSION = 1
```

```python
nemo_gym.models.token_capture.records.ENGINE_REFUSED_CALL_REASON = 'engine_refused_request'
```

```python
nemo_gym.models.token_capture.records.FINISH_REASON_DISAGREEMENT_REASON = 'finish_reason_disagreement'
```

```python
nemo_gym.models.token_capture.records.LEDGER_ROW_MISSING_CHAIN_HASH_REASON = 'ledger_row_missing_chain_hash'
```

```python
nemo_gym.models.token_capture.records.LEDGER_ROW_MISSING_RESPONSE_ID_REASON = 'ledger_row_missing_response_id'
```

```python
nemo_gym.models.token_capture.records.MAX_OUTPUT_TOKENS_INCOMPLETE_REASON = 'max_output_tokens'
```

```python
nemo_gym.models.token_capture.records.NAMESPACE_TOOL_DELIMITER = '__'
```

```python
nemo_gym.models.token_capture.records.TOKEN_ENTRY_MIN_SCHEMA_VERSION = 1
```

```python
nemo_gym.models.token_capture.records.TOKEN_ENTRY_RECORD_SCHEMA_VERSION = 2
```

```python
nemo_gym.models.token_capture.records.TOKEN_FIELDS = ('prompt_token_ids', 'generation_token_ids', 'generation_log_probs', 'routed_exp...
```

```python
nemo_gym.models.token_capture.records.UNCOMMITTED_CALL_REASON = 'request_finished_without_staged_coordinates'
```

```python
nemo_gym.models.token_capture.records.UNRESOLVED_PARENT_REASON = 'unresolved_parent'
```

```python
nemo_gym.models.token_capture.records.UNSERVED_CALL_FAILURE_REASONS = frozenset({UNCOMMITTED_CALL_REASON, ENGINE_REFUSED_CALL_REASON})
```

```python
nemo_gym.models.token_capture.records._DIGEST_DOMAIN = b'nemo-gym-tokens'
```

```python
nemo_gym.models.token_capture.records._EMPTY_DIGEST = hashlib.sha256(_DIGEST_DOMAIN).hexdigest()
```

```python
nemo_gym.models.token_capture.records._FINISH_REASON_TO_INCOMPLETE_REASON = {'length': MAX_OUTPUT_TOKENS_INCOMPLETE_REASON, 'content_filter': 'content_filte...
```

```python
nemo_gym.models.token_capture.records._INCOMPLETE_REASON_TO_FINISH_REASON = {MAX_OUTPUT_TOKENS_INCOMPLETE_REASON: 'length', 'content_filter': 'content_filte...
```

```python
nemo_gym.models.token_capture.records._STOP_REASON_TO_FINISH_REASON = {'end_turn': 'stop', 'stop_sequence': 'stop', 'tool_use': 'tool_calls', 'max_tok...
```

```python
nemo_gym.models.token_capture.records._TOOL_CALL_ITEM_TYPES = frozenset({'function_call', 'custom_tool_call'})
```