> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Model-call capture > Optional per-rollout model-call evidence for evaluation and debugging Model-call capture records requests, responses, token usage, tool calls, reasoning, latency, and errors at a Gym model server. Capture does not modify model responses, agent responses, or rollout rewards. `gym eval submit` enables capture by default under each benchmark's job directory at `model-calls/`. Set `driver.benchmarks..run.observability_enabled: false` to opt out, or set `driver.benchmarks..run.model_call_capture_dir` to an absolute path to customize the directory. Other entry points, including local evaluations and training, remain opt-in. ## Run an evaluation with capture This example extends the [quickstart](/main/get-started/quickstart) MCQA evaluation. It assumes that your model credentials are already configured in `env.yaml`. In the first terminal, choose an absolute capture directory and start the servers with capture enabled: ```bash mkdir -p results/model-calls capture_dir="$(pwd)/results/model-calls" gym env start \ --resources-server mcqa \ --model-type openai_model \ ++observability_enabled=true \ ++model_call_capture_dir="${capture_dir}" ``` In a second terminal, activate the same environment and pass the same capture settings to rollout collection: ```bash source .venv/bin/activate capture_dir="$(pwd)/results/model-calls" gym eval run --no-serve \ --agent mcqa_simple_agent \ --input resources_servers/mcqa/data/example.jsonl \ --output results/mcqa_rollouts.jsonl \ --limit 1 \ --num-repeats 1 \ ++observability_enabled=true \ ++model_call_capture_dir="${capture_dir}" ``` Inspect the attachment on the first rollout and list the raw capture files: ```bash head -n 1 results/mcqa_rollouts.jsonl | jq '.ng_model_call_capture' ls -1 "${capture_dir}" ``` The rollout record is the usual starting point for analysis. Use its `ng_model_call_capture` field for normalized calls and aggregate metrics; use the matching file in `CaptureStore` when you need the original request and response payloads. ## What capture contains ### Buffered SSE with external token staging With `token_id_capture.external_staging: true`, captured requests can use `stream: true` on Chat Completions, Responses (including the compaction adapter), and Anthropic Messages. Gym requests a complete, non-streaming answer from the inference worker. The worker stages training tokens before returning commit coordinates. Gym removes internal token fields, converts the answer to the client-visible API format, and serializes the SSE events before recording a successful capture and replaying the answer. Capture fingerprints preserve namespaced tool identity across the backend and client representations. Conversion or serialization failures leave no successful capture record, and capture failures are recorded before a terminal `response.failed` event is sent, including when observability is disabled. No content is delivered during generation, so buffered SSE does not reduce time to first content. Worker streaming with external capture remains unsupported. Training reconstruction reads the external staged tokens, not the SSE text. ### Rollout-record attachment `rollout_collection` attaches available model-call data to each rollout. This representative example includes a gap to show the complete shape. The `gaps` field is omitted when there are no gaps. ```json { "ng_model_call_capture": { "rollout_id": "0-0", "metrics": { "tokens_in": 94, "tokens_out": 28, "tokens_reasoning": 12, "tokens_total": 122, "cached_tokens": 0, "latency_total_ms": 842.7, "num_calls": 1 }, "calls": [ { "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629", "response_id": "resp_123", "call_index": 0, "model_ref": { "type": "responses_api_models", "name": "policy_model" }, "model": "gpt-4.1-2025-04-14", "dialect": "responses", "status_code": 200, "response_status": "completed", "tokens_in": 94, "tokens_out": 28, "tokens_reasoning": 12, "tokens_total": 122, "latency_total_ms": 842.7 } ], "gaps": [ { "code": "model_call_capture_incomplete" } ] } } ``` `metrics.num_calls` is the number of captured calls. Token and latency totals include only values reported by the provider; a `null` value means unknown, not zero. For example, Anthropic does not report reasoning-token usage, so `tokens_reasoning` is `null` for that dialect. After successful [`ng_trajectory`](/main/reference/trajectory-capabilities) projection, attached calls omit request and response payloads; those remain in `CaptureStore`. If projection fails, payloads remain attached and `gaps` includes `trajectory_projection_failed`. LabBench rows with the `multimodal_history_redacted` gap omit payload copies from `ng_trajectory`. The attachment is additive: it does not replace or rewrite the existing response, reward, `NeMoGymResponse`, token-id, or log-prob fields. Aggregate-metrics requests exclude it, and W\&B rollout tables omit `ng_trajectory` and model-call request and response payloads. ### Raw `CaptureStore` record Each rollout has an append-only file named `.capture.jsonl`. Each line contains one raw model-server exchange, including its request and response. The following synthetic record shows the exact persisted field shape. It is formatted across lines for readability but stored as one JSONL line. ```json { "model_call_id": "863410a8d52e4be6a31efc5c6e9d4629", "dialect": "responses", "model_ref": { "type": "responses_api_models", "name": "policy_model" }, "started_at": 1786032154.12, "completed_at": 1786032154.96, "latency_ms": 842.7, "latency_ttft_ms": 213.4, "status_code": 200, "error_category": null, "request": { "model": "gpt-4.1-2025-04-14", "input": [{"role": "user", "content": "Which option is correct?"}] }, "response": { "id": "resp_123", "status": "completed", "usage": { "input_tokens": 94, "output_tokens": 28, "total_tokens": 122 } } } ``` Writes are flushed and fsynced. Capture failures are logged without failing model requests. Resume attempts use an `-a` suffix so their data does not mix with an earlier attempt. Before dispatch, the collector clears any existing capture for the exact rollout-attempt id. Read raw records programmatically when you need more than the rollout attachment: ```python from nemo_gym.base_responses_api_model import ( CaptureStore, aggregate_model_call_metrics, read_model_call_records, ) store = CaptureStore("/absolute/path/to/results/model-calls") calls = read_model_call_records(store, rollout_id) totals = aggregate_model_call_metrics(store, rollout_id) ``` `ModelCallRecord` includes the server-generated `model_call_id`, protocol `response_id`, typed `model_ref`, UTC `started_at` and `completed_at` timestamps, durable `call_index`, API dialect, token and cache usage, status, finish reason, latency, error details, tool calls, reasoning, and captured payloads. `call_index` and timestamps do not establish causal ordering between concurrent calls. Streaming Responses, Chat Completions, and Anthropic Messages are reassembled for capture on a best-effort basis. If reassembly fails, request, status, correlation, and latency data remain available. A successful-status stream that closes without its dialect's terminal event retains any partial reconstruction and is marked `stream_truncated`. ## Configure capture The CLI workflow above sets these top-level global-config fields: | Field | Type | Default | Description | | ------------------------ | ------ | --------------------- | ------------------------------------------------------------------------------- | | `observability_enabled` | `bool` | `false` | Enables model-call capture for the run. | | `model_call_capture_dir` | `str` | required when enabled | Absolute shared capture directory used by model servers and rollout collection. | You can put the same settings in a YAML config instead: ```yaml observability_enabled: true model_call_capture_dir: /absolute/path/to/model-calls policy_model: responses_api_models: vllm_model: entrypoint: app.py ``` Model servers and `rollout_collection` must resolve `model_call_capture_dir` to the same location. For separate pods or nodes, mount that absolute path in every producer and the collector. Multiple model servers can append to one rollout file when the shared filesystem supports POSIX advisory file locking. Rollout ids contain task, rollout, and retry-attempt indices, but not an evaluation-run id. Use a separate capture directory for concurrent collection jobs whose indices can overlap. ## If you build a custom agent Capture requires a caller-supplied rollout correlation id. Gym's built-in compatible agents handle this routing; custom agent authors must preserve it on model calls. Use `/ng-rollout/` as the model-server URL prefix. SDKs append `/v1/responses`, `/v1/chat/completions`, or `/v1/messages`, and the model server removes the rollout prefix before routing. An unprefixed call is forwarded normally but is not captured. Calls sent directly to an external provider instead of through a Gym model server are also not captured. Agents based on `SimpleResponsesAPIAgent` can use `url_path_for_run(url_path, body)` to prefix a downstream call from the run request's task and rollout indices. Use `url_path_for_request(url_path, request)` to carry an inbound prefixed self-call through to the model call. The prefixed self-call route (`/ng-rollout/{rollout_id}/v1/responses`) is registered on every agent by default. SDK harnesses that configure a client once can use `base_url_for_run(base_url, body)`. Harnesses that must thread the id through multiple layers can use `rollout_id_from_run(body)` with `resolve_model_base_url(name, rollout_id)`. ## Add agent-side evidence Model-call capture is one input to Gym's broader rollout-evidence pipeline. Supported harnesses can also emit `ng_agent_observations`; `rollout_collection` exact-joins the available sources into `ng_trajectory` and derives `ng_perf` when sufficient evidence exists. See [Rollout evidence](/main/observability/rollout-evidence) for the full pipeline, exact correlation rules, payload retention, agent observations, gaps, and performance summaries. Agent-side coverage varies by producer and execution path; the [trajectory capability matrix](/main/reference/trajectory-capabilities) records the evaluated coverage. ## Framework-owned media staging Training frameworks using `nemo_gym.token_id_capture.staging` can hand a framework's processed media (for example an inference worker's pixel tensors) to `RolloutTokenCapture.complete_call` or `complete_call_from_response` through the keyword-only `attachments` mapping. Gym forwards the mapping unchanged to `StagingSink.stage(record, attachments=...)` so the media lands in the same durable write as the token record, and `staged` coordinates are returned only after that combined write succeeds. Attachments are opaque to Gym: they never enter `StagedCallRecord`, `extras`, or any digest, so `digest`, `extras_digest`, and `CommitCoords` are identical with or without them. The keyword is passed only when the caller supplies a mapping. A legacy `stage(record)` sink therefore keeps working for text-only calls; if it is given attachments it raises `TypeError`, which the capture boundary reports as `capture_failed`. Gym never retries a write with the attachments dropped. A sink that accepts the keyword must store every supplied attachment before returning success or reject the call. Validation of the attached tensors, their storage layout, and read-back belong to the framework. `VLLMCaptureAdapter` preserves optional `routed_experts` extras and passes through `media_spans`, small metadata that the RL vLLM worker uses to remap image/video placeholders when reusing a captured prefix. vLLM uses expanded token IDs and does not emit a separate compact token delta. This replaces both the experimental tensor-attachment sink extension and the digest-covered `media` summary. Deploy the paired RL changes with this adapter; rows and checkpoints written in either older format need a separate migration. Supported processors and modalities are enforced by the framework's inference worker. ## Limitations The model-server boundary observes model HTTP requests and responses. It can record tool calls, reasoning, and tool results present in those payloads, but it does not observe tool execution, environment events, context compaction, semantic turns, or subagent structure. Those require instrumentation at the agent, tool, environment, or rollout layer.