Model-call capture

Model-call capture records requests, responses, token usage, tool calls, reasoning, latency, and errors at a Gym model server. Capture does not modify model responses, agent responses, or rollout rewards.

gym eval submit enables capture by default under each benchmark’s job directory at model-calls/. Set driver.benchmarks.<name>.run.observability_enabled: false to opt out, or set driver.benchmarks.<name>.run.model_call_capture_dir to an absolute path to customize the directory. Other entry points, including local evaluations and training, remain opt-in.

Run an evaluation with capture

This example extends the quickstart MCQA evaluation. It assumes that your model credentials are already configured in env.yaml.

In the first terminal, choose an absolute capture directory and start the servers with capture enabled:

mkdir -p results/model-calls
capture_dir="$(pwd)/results/model-calls"
gym env start \
    --resources-server mcqa \
    --model-type openai_model \
    ++observability_enabled=true \
    ++model_call_capture_dir="${capture_dir}"

In a second terminal, activate the same environment and pass the same capture settings to rollout collection:

source .venv/bin/activate
capture_dir="$(pwd)/results/model-calls"
gym eval run --no-serve \
    --agent mcqa_simple_agent \
    --input resources_servers/mcqa/data/example.jsonl \
    --output results/mcqa_rollouts.jsonl \
    --limit 1 \
    --num-repeats 1 \
    ++observability_enabled=true \
    ++model_call_capture_dir="${capture_dir}"

Inspect the attachment on the first rollout and list the raw capture files:

head -n 1 results/mcqa_rollouts.jsonl | jq '.ng_model_call_capture'
ls -1 "${capture_dir}"

The rollout record is the usual starting point for analysis. Use its ng_model_call_capture field for normalized calls and aggregate metrics; use the matching file in CaptureStore when you need the original request and response payloads.

What capture contains

Buffered SSE with external token staging

With token_id_capture.external_staging: true, captured requests can use stream: true on Chat Completions, Responses (including the compaction adapter), and Anthropic Messages. Gym requests a complete, non-streaming answer from the inference worker. The worker stages training tokens before returning commit coordinates. Gym removes internal token fields, converts the answer to the client-visible API format, and serializes the SSE events before recording a successful capture and replaying the answer. Capture fingerprints preserve namespaced tool identity across the backend and client representations. Conversion or serialization failures leave no successful capture record, and capture failures are recorded before a terminal response.failed event is sent, including when observability is disabled. No content is delivered during generation, so buffered SSE does not reduce time to first content.

Worker streaming with external capture remains unsupported. Training reconstruction reads the external staged tokens, not the SSE text.

Rollout-record attachment

rollout_collection attaches available model-call data to each rollout. This representative example includes a gap to show the complete shape. The gaps field is omitted when there are no gaps.

{
  "ng_model_call_capture": {
    "rollout_id": "0-0",
    "metrics": {
      "tokens_in": 94,
      "tokens_out": 28,
      "tokens_reasoning": 12,
      "tokens_total": 122,
      "cached_tokens": 0,
      "latency_total_ms": 842.7,
      "num_calls": 1
    },
    "calls": [
      {
        "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629",
        "response_id": "resp_123",
        "call_index": 0,
        "model_ref": {
          "type": "responses_api_models",
          "name": "policy_model"
        },
        "model": "gpt-4.1-2025-04-14",
        "dialect": "responses",
        "status_code": 200,
        "response_status": "completed",
        "tokens_in": 94,
        "tokens_out": 28,
        "tokens_reasoning": 12,
        "tokens_total": 122,
        "latency_total_ms": 842.7
      }
    ],
    "gaps": [
      {
        "code": "model_call_capture_incomplete"
      }
    ]
  }
}

metrics.num_calls is the number of captured calls. Token and latency totals include only values reported by the provider; a null value means unknown, not zero. For example, Anthropic does not report reasoning-token usage, so tokens_reasoning is null for that dialect.

After successful ng_trajectory projection, attached calls omit request and response payloads; those remain in CaptureStore. If projection fails, payloads remain attached and gaps includes trajectory_projection_failed. LabBench rows with the multimodal_history_redacted gap omit payload copies from ng_trajectory.

The attachment is additive: it does not replace or rewrite the existing response, reward, NeMoGymResponse, token-id, or log-prob fields. Aggregate-metrics requests exclude it, and W&B rollout tables omit ng_trajectory and model-call request and response payloads.

Raw CaptureStore record

Each rollout has an append-only file named <rollout_id>.capture.jsonl. Each line contains one raw model-server exchange, including its request and response. The following synthetic record shows the exact persisted field shape. It is formatted across lines for readability but stored as one JSONL line.

{
  "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629",
  "dialect": "responses",
  "model_ref": {
    "type": "responses_api_models",
    "name": "policy_model"
  },
  "started_at": 1786032154.12,
  "completed_at": 1786032154.96,
  "latency_ms": 842.7,
  "latency_ttft_ms": 213.4,
  "status_code": 200,
  "error_category": null,
  "request": {
    "model": "gpt-4.1-2025-04-14",
    "input": [{"role": "user", "content": "Which option is correct?"}]
  },
  "response": {
    "id": "resp_123",
    "status": "completed",
    "usage": {
      "input_tokens": 94,
      "output_tokens": 28,
      "total_tokens": 122
    }
  }
}

Writes are flushed and fsynced. Capture failures are logged without failing model requests. Resume attempts use an -a<n> suffix so their data does not mix with an earlier attempt. Before dispatch, the collector clears any existing capture for the exact rollout-attempt id.

Read raw records programmatically when you need more than the rollout attachment:

from nemo_gym.base_responses_api_model import (
    CaptureStore,
    aggregate_model_call_metrics,
    read_model_call_records,
)
store = CaptureStore("/absolute/path/to/results/model-calls")
calls = read_model_call_records(store, rollout_id)
totals = aggregate_model_call_metrics(store, rollout_id)

ModelCallRecord includes the server-generated model_call_id, protocol response_id, typed model_ref, UTC started_at and completed_at timestamps, durable call_index, API dialect, token and cache usage, status, finish reason, latency, error details, tool calls, reasoning, and captured payloads. call_index and timestamps do not establish causal ordering between concurrent calls.

Streaming Responses, Chat Completions, and Anthropic Messages are reassembled for capture on a best-effort basis. If reassembly fails, request, status, correlation, and latency data remain available. A successful-status stream that closes without its dialect’s terminal event retains any partial reconstruction and is marked stream_truncated.

Configure capture

The CLI workflow above sets these top-level global-config fields:

You can put the same settings in a YAML config instead:

observability_enabled: true
model_call_capture_dir: /absolute/path/to/model-calls
policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py

Model servers and rollout_collection must resolve model_call_capture_dir to the same location. For separate pods or nodes, mount that absolute path in every producer and the collector. Multiple model servers can append to one rollout file when the shared filesystem supports POSIX advisory file locking.

Rollout ids contain task, rollout, and retry-attempt indices, but not an evaluation-run id. Use a separate capture directory for concurrent collection jobs whose indices can overlap.

If you build a custom agent

Capture requires a caller-supplied rollout correlation id. Gym’s built-in compatible agents handle this routing; custom agent authors must preserve it on model calls.

Use /ng-rollout/<rollout_id> as the model-server URL prefix. SDKs append /v1/responses, /v1/chat/completions, or /v1/messages, and the model server removes the rollout prefix before routing. An unprefixed call is forwarded normally but is not captured. Calls sent directly to an external provider instead of through a Gym model server are also not captured.

Agents based on SimpleResponsesAPIAgent can use url_path_for_run(url_path, body) to prefix a downstream call from the run request’s task and rollout indices. Use url_path_for_request(url_path, request) to carry an inbound prefixed self-call through to the model call. The prefixed self-call route (/ng-rollout/{rollout_id}/v1/responses) is registered on every agent by default.

SDK harnesses that configure a client once can use base_url_for_run(base_url, body). Harnesses that must thread the id through multiple layers can use rollout_id_from_run(body) with resolve_model_base_url(name, rollout_id).

Add agent-side evidence

Model-call capture is one input to Gym’s broader rollout-evidence pipeline. Supported harnesses can also emit ng_agent_observations; rollout_collection exact-joins the available sources into ng_trajectory and derives ng_perf when sufficient evidence exists.

See Rollout evidence for the full pipeline, exact correlation rules, payload retention, agent observations, gaps, and performance summaries. Agent-side coverage varies by producer and execution path; the trajectory capability matrix records the evaluated coverage.

Framework-owned media staging

Training frameworks using nemo_gym.token_id_capture.staging can hand a framework’s processed media (for example an inference worker’s pixel tensors) to RolloutTokenCapture.complete_call or complete_call_from_response through the keyword-only attachments mapping. Gym forwards the mapping unchanged to StagingSink.stage(record, attachments=...) so the media lands in the same durable write as the token record, and staged coordinates are returned only after that combined write succeeds. Attachments are opaque to Gym: they never enter StagedCallRecord, extras, or any digest, so digest, extras_digest, and CommitCoords are identical with or without them.

The keyword is passed only when the caller supplies a mapping. A legacy stage(record) sink therefore keeps working for text-only calls; if it is given attachments it raises TypeError, which the capture boundary reports as capture_failed. Gym never retries a write with the attachments dropped. A sink that accepts the keyword must store every supplied attachment before returning success or reject the call. Validation of the attached tensors, their storage layout, and read-back belong to the framework.

VLLMCaptureAdapter preserves optional routed_experts extras and passes through media_spans, small metadata that the RL vLLM worker uses to remap image/video placeholders when reusing a captured prefix. vLLM uses expanded token IDs and does not emit a separate compact token delta.

This replaces both the experimental tensor-attachment sink extension and the digest-covered media summary. Deploy the paired RL changes with this adapter; rows and checkpoints written in either older format need a separate migration. Supported processors and modalities are enforced by the framework’s inference worker.

Limitations

The model-server boundary observes model HTTP requests and responses. It can record tool calls, reasoning, and tool results present in those payloads, but it does not observe tool execution, environment events, context compaction, semantic turns, or subagent structure. Those require instrumentation at the agent, tool, environment, or rollout layer.