Model-Call Capture

Model-call capture records requests, responses, token usage, tool calls, reasoning, latency, and errors at a Gym model server. Capture does not modify model responses, agent responses, or rollout rewards.

gym eval submit enables capture by default under each benchmark’s job directory at model-calls/. Set driver.benchmarks.<name>.run.observability_enabled: false to opt out, or set driver.benchmarks.<name>.run.model_call_capture_dir to an absolute path to customize the directory. Other entry points, including local evaluations and training, remain opt-in.

Run an evaluation with capture

This example extends the quickstart MCQA evaluation. It assumes that your model credentials are already configured in env.yaml.

In the first terminal, choose an absolute capture directory and start the servers with capture enabled:

mkdir -p results/model-calls
capture_dir="$(pwd)/results/model-calls"
gym env start \
    --resources-server mcqa \
    --model-type openai_model \
    ++observability_enabled=true \
    ++model_call_capture_dir="${capture_dir}"

In a second terminal, activate the same environment and pass the same capture settings to rollout collection:

source .venv/bin/activate
capture_dir="$(pwd)/results/model-calls"
gym eval run --no-serve \
    --agent mcqa_simple_agent \
    --input resources_servers/mcqa/data/example.jsonl \
    --output results/mcqa_rollouts.jsonl \
    --limit 1 \
    --num-repeats 1 \
    ++observability_enabled=true \
    ++model_call_capture_dir="${capture_dir}"

Inspect the attachment on the first rollout and list the raw capture files:

head -n 1 results/mcqa_rollouts.jsonl | jq '.ng_model_call_capture'
ls -1 "${capture_dir}"

The rollout record is the usual starting point for analysis. Use its ng_model_call_capture field for normalized calls and aggregate metrics; use the matching file in CaptureStore when you need the original request and response payloads.

What capture contains

Buffered SSE with external token staging

With token_id_capture.external_staging: true, captured requests can use stream: true on Chat Completions, Responses (including the compaction adapter), and Anthropic Messages. Gym requests a complete, non-streaming answer from the inference worker. The worker stages training tokens before returning commit coordinates. Gym removes internal token fields, converts the answer to the client-visible API format, and serializes the SSE events before recording a successful capture and replaying the answer. Capture fingerprints preserve namespaced tool identity across the backend and client representations. Conversion or serialization failures leave no successful capture record, and capture failures are recorded before a terminal response.failed event is sent, including when observability is disabled. No content is delivered during generation, so buffered SSE does not reduce time to first content.

Worker streaming with external capture remains unsupported. Training reconstruction reads the external staged tokens, not the SSE text.

The staging ledger’s wire models (nemo_gym/token_id_capture/staging/records.py) forbid unknown fields, so the model server that stages a call and the framework that rebuilds the rollout from the staged rows must run the same Gym version. A framework whose Gym predates a column the model server writes, such as capture_key or finish_reason, rejects every manifest row as invalid_manifest_row and masks the rollout.

Rollout-record attachment

rollout_collection attaches available model-call data to each rollout. This representative example includes a gap to show the complete shape. The gaps field is omitted when there are no gaps.

{
  "ng_model_call_capture": {
    "rollout_id": "0-0",
    "metrics": {
      "tokens_in": 94,
      "tokens_out": 28,
      "tokens_reasoning": 12,
      "tokens_total": 122,
      "cached_tokens": 0,
      "latency_total_ms": 842.7,
      "num_calls": 1
    },
    "calls": [
      {
        "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629",
        "response_id": "resp_123",
        "call_index": 0,
        "model_ref": {
          "type": "responses_api_models",
          "name": "policy_model"
        },
        "model": "gpt-4.1-2025-04-14",
        "dialect": "responses",
        "status_code": 200,
        "response_status": "completed",
        "tokens_in": 94,
        "tokens_out": 28,
        "tokens_reasoning": 12,
        "tokens_total": 122,
        "latency_total_ms": 842.7
      }
    ],
    "gaps": [
      {
        "code": "model_call_capture_incomplete"
      }
    ]
  }
}

metrics.num_calls is the number of captured calls. Token and latency totals include only values reported by the provider; a null value means unknown, not zero. For example, Anthropic does not report reasoning-token usage, so tokens_reasoning is null for that dialect.

After successful ng_trajectory projection, attached calls omit request and response payloads; those remain in CaptureStore. If projection fails, payloads remain attached and gaps includes trajectory_projection_failed. LabBench rows with the multimodal_history_redacted gap omit payload copies from ng_trajectory.

A capture write started by a cancelled call runs detached, so a rollout’s outstanding writes are waited for before its record is snapshotted. If one has not landed by the drain timeout, gaps includes model_call_capture_pending.

The attachment is additive: it does not replace or rewrite the existing response, reward, NeMoGymResponse, token-id, or log-prob fields. Aggregate-metrics requests exclude it, and W&B rollout tables omit ng_trajectory and model-call request and response payloads.

Raw CaptureStore record

Each rollout has an append-only file named <rollout_id>.capture.jsonl. Each line contains one raw model-server exchange, including its request and response. The following synthetic record shows the exact persisted field shape. It is formatted across lines for readability but stored as one JSONL line.

{
  "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629",
  "dialect": "responses",
  "model_ref": {
    "type": "responses_api_models",
    "name": "policy_model"
  },
  "started_at": 1786032154.12,
  "completed_at": 1786032154.96,
  "latency_ms": 842.7,
  "latency_ttft_ms": 213.4,
  "status_code": 200,
  "error_category": null,
  "request": {
    "model": "gpt-4.1-2025-04-14",
    "input": [{"role": "user", "content": "Which option is correct?"}]
  },
  "response": {
    "id": "resp_123",
    "status": "completed",
    "usage": {
      "input_tokens": 94,
      "output_tokens": 28,
      "total_tokens": 122
    }
  }
}

Writes are flushed and fsynced. Capture failures are logged without failing model requests. Resume attempts use an -a<n> suffix so their data does not mix with an earlier attempt. Before dispatch, the collector clears any existing capture for the exact rollout-attempt id.

Read raw records programmatically when you need more than the rollout attachment:

from nemo_gym.base_responses_api_model import (
    CaptureStore,
    aggregate_model_call_metrics,
    read_model_call_records,
)
store = CaptureStore("/absolute/path/to/results/model-calls")
calls = read_model_call_records(store, rollout_id)
totals = aggregate_model_call_metrics(store, rollout_id)

ModelCallRecord includes the server-generated model_call_id, protocol response_id, typed model_ref, UTC started_at and completed_at timestamps, durable call_index, API dialect, token and cache usage, status, finish reason, latency, error details, tool calls, reasoning, and captured payloads. call_index and timestamps do not establish causal ordering between concurrent calls.

Streaming Responses, Chat Completions, and Anthropic Messages are reassembled for capture on a best-effort basis. If reassembly fails, request, status, correlation, and latency data remain available. A successful-status stream that closes without its dialect’s terminal event retains any partial reconstruction and is marked stream_truncated.

Configure capture

The CLI workflow above sets these top-level fields in Gym’s global config:

You can put the same settings in a YAML config instead:

observability_enabled: true
model_call_capture_dir: /absolute/path/to/model-calls
policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py

Model servers and rollout_collection must resolve model_call_capture_dir to the same location. For separate pods or nodes, mount that absolute path in every producer and the collector. Multiple model servers can append to one rollout file when the shared filesystem supports POSIX advisory file locking.

Rollout ids contain task, rollout, and retry-attempt indices, but not an evaluation-run id. Use a separate capture directory for concurrent collection jobs whose indices can overlap.

If you build a custom agent

Capture requires a caller-supplied rollout correlation id. Gym’s built-in compatible agents handle this routing; custom agent authors must preserve it on model calls.

Use /ng-rollout/<rollout_id> as the model-server URL prefix. SDKs append /v1/responses, /v1/chat/completions, or /v1/messages, and the model server removes the rollout prefix before routing. An unprefixed call is forwarded normally but is not captured. Calls sent directly to an external provider instead of through a Gym model server are also not captured.

Agents based on SimpleResponsesAPIAgent can add the prefix with these helpers:

The prefixed self-call route (/ng-rollout/{rollout_id}/v1/responses) is registered on every agent by default.

Add agent-side evidence

Model-call capture is one input to Gym’s broader rollout-evidence pipeline. Supported harnesses can also emit ng_agent_observations; rollout_collection exact-joins the available sources into ng_trajectory and derives ng_perf when sufficient evidence exists.

See Rollout evidence for the full pipeline, exact correlation rules, payload retention, agent observations, gaps, and performance summaries. Agent-side coverage varies by producer and execution path; the trajectory capability matrix shows measured harness checks.

Framework-owned media staging

Training frameworks using nemo_gym.token_id_capture.staging can hand a framework’s processed media (for example an inference worker’s pixel tensors) to RolloutTokenCapture.complete_call or complete_call_from_response through the keyword-only attachments mapping. Gym forwards the mapping unchanged to StagingSink.stage(record, attachments=...) so the media lands in the same durable write as the token record, and staged coordinates are returned only after that combined write succeeds. Attachments are opaque to Gym: they never enter StagedCallRecord, extras, or any digest, so digest, extras_digest, and CommitCoords are identical with or without them.

The keyword is passed only when the caller supplies a mapping. A legacy stage(record) sink therefore keeps working for text-only calls; if it is given attachments it raises TypeError, which the capture boundary reports as capture_failed. Gym never retries a write with the attachments dropped. A sink that accepts the keyword must store every supplied attachment before returning success or reject the call. Validation of the attached tensors, their storage layout, and read-back belong to the framework.

VLLMCaptureAdapter preserves optional routed_experts extras and passes through media_spans, small metadata that the RL vLLM worker uses to remap image/video placeholders when reusing a captured prefix. vLLM uses expanded token IDs and does not emit a separate compact token delta.

This replaces both the experimental tensor-attachment sink extension and the digest-covered media summary. Deploy the paired RL changes with this adapter; rows and checkpoints written in either older format need a separate migration. Supported processors and modalities are enforced by the framework’s inference worker.

Retire capture ledgers

With external staging, the model server keeps a capture ledger for each rollout: one <rollout_id>.lineage.jsonl file in the lineage store’s directory, recording the model calls the rollout made without their token arrays. The training framework reads it once, through the manifest route, when the rollout ends. Gym can’t tell when the framework is done with a rollout, so the framework removes each ledger itself, using the same two operations the complete-record token store has: retire and delete.

Retire a rollout the framework is done with

Retiring deletes the rollout’s ledger and leaves an empty <rollout_id>.lineage.retired file in its place. That file acts as a fence: if a model call for the rollout finishes later, Gym discards its row instead of starting a new ledger. Late calls are more common than they sound, because an agent the framework gave up on can keep running, and a client retry can run a finished rollout a second time. TokenSource.drop leaves the same kind of fence in the complete-record token store. After a rollout is retired, its manifest route returns HTTP 410 instead of an empty manifest, so a framework that reads it too late finds out rather than building an empty receipt.

A call that starts after its rollout was retired is not admitted for capture, so it stages nothing. A call that was already in flight, however, may have staged its tokens before Gym discarded its row, and no manifest names those tokens. Gym logs a warning with the call’s staging key, and the framework that staged the tokens has to remove them, for example by sweeping staged rows that no receipt references.

Retire every rollout the framework is done with, whether it finished or was abandoned. A finished rollout can be retired once its manifest has been read and its receipt built, unless a crash before the receipt is saved would make the framework rerun it under the same ID; in that case, wait until the receipt is saved. An abandoned rollout should be retired as soon as the framework gives up on it.

Delete fences once nothing can still write

Deleting removes a rollout’s ledger together with its fence, which is only safe once no call from that rollout can still arrive. In practice that means the end of the run, or a point where every Gym server sharing the lineage directory has restarted. Until then, the fences accumulate at one empty file per retired rollout. Also delete a rollout ID before reusing it, so that the new run starts from an empty ledger, as TokenCaptureStore.delete does for the complete-record store.

Send the requests

Send each request to a model server whose lineage directory holds the rollout’s ledger, either the server that served the rollout or any other server configured with the same directory. A model server only sees the ledgers in its own lineage directory and reports any other rollout as absent.

POST /training-token-capture/control/rollouts/retire
POST /training-token-capture/control/rollouts/delete
Authorization: Bearer <control auth token>
{"rollout_ids": ["<rollout_id>", ...]}

Each request takes at most 4096 rollout IDs, and RolloutControlClient.retire and RolloutControlClient.delete split longer lists into batches for you. The reply lists the rollouts whose ledger was removed under removed and the rest under absent. Because repeating either operation changes nothing, a failed batch can simply be sent again.

Neither operation touches staged token data. The framework that staged the data removes it, and it must keep any staged data that a checkpoint it may restore still refers to.

Limitations

The model-server boundary observes model HTTP requests and responses. It can record tool calls, reasoning, and tool results present in those payloads, but it does not observe tool execution, environment events, context compaction, semantic turns, or subagent structure. Those require instrumentation at the agent, tool, environment, or rollout layer.