Rollout Evidence

Model-call evidence records exchanges with a model. Agent observations record harness activity, including conversations and tool execution. Rollout collection joins these sources into a trajectory for analysis.

Enable Collection

Set these top-level fields in the shared global configuration used by the agent servers, model servers, and rollout collector, not inside an individual server’s configuration:

observability_enabled: true
model_call_capture_dir: /absolute/shared/path/model-calls

Agent processes also need observability_enabled: true to propagate rollout prefixes and collect supported agent-side evidence. The model servers and rollout collector must use the same shared capture directory. Compatible agents route model calls through /ng-rollout/<rollout_id> on the Gym Model Server. Unprefixed and direct-provider calls bypass capture. Use a separate directory for each evaluation job: rollout IDs do not contain a run ID.

Follow Model-Call Capture for a runnable evaluation example. Agent-side evidence depends on the harness and execution path. See the harness conformance results for measured harness checks.

Record From a Training Framework

A framework that dispatches through RolloutCollectionHelper.run_examples instead of gym eval run gets the same evidence with two opt-in options:

• enrich_results=True attaches what collection records to each result before it is returned: the row’s rollout identity, the captured model calls (ng_model_call_capture, with request and response bodies unless include_model_call_payloads=False), the standardized ng_trajectory and ng_perf. Model calls are captured only when the global config enables observability_enabled. A failed /run under route_failures_to_sidecar=True comes back as the typed RolloutFailureRecord. delete_model_call_capture=True removes a rollout’s capture file once its calls are on the record, and needs the payloads kept.
• record_output_dir with record_shard also writes each rollout to that folder as collection lays out a run: rollouts-<shard>.jsonl, its failures sidecar with the typed failure rows collection writes, the rows as the shard’s materialized inputs, the shard’s run manifest, and its saved progress (rollouts-<shard>_batch_status.json, read by gym eval status). The caller picks the shard label (for example a training step and worker) and Gym owns the file names, so gym eval aggregate +input_glob='run/rollouts-*.jsonl', the run health check and the Store read a folder of shards like an eval run. Reopening a shard replaces it, so a replayed step does not duplicate rows. Rows need their task and rollout indices; record_omitted_rollouts=True also keeps an omitted-rollout row for an interrupted or never-started try. A shard dispatches each row under the row’s own rollout id (its _ng_rollout_id, or <task>-<rollout>) prefixed with the shard’s run id, so its model calls are captured under keys of its own even when a later shard dispatches the same task and rollout; read a capture’s key from the returned row. A shard whose rollouts did not all succeed resumes like a run: gym eval run +input_jsonl_fpath=runs/policy/rollouts-<shard>_materialized_inputs.jsonl +output_jsonl_fpath=runs/policy/rollouts-<shard>.jsonl +resume_from_cache=true, with the same servers, runs only what the shard still lacks.

from nemo_gym.rollouts import RolloutCollectionHelper
helper = RolloutCollectionHelper()
completions = helper.run_examples(
    rows, route_failures_to_sidecar=True, record_output_dir="runs/policy", record_shard="step_000007-w0"
)
for completion in completions:
    row, result = await completion  # the record the shard holds, or the typed failure record

Which Artifact to Read

After each attempt, collection joins call references and projects the available evidence into ng_trajectory. Successful projection removes duplicate request and response payloads from ng_model_call_capture; originals remain in CaptureStore. If projection fails, attached payloads remain and a trajectory_projection_failed gap is recorded. LabBench’s multimodal_history_redacted gap indicates that trajectory payload copies were omitted.

The trajectory does not include the full compaction and sandbox records. Read ng_agent_observations for those details.

Correlation and Gaps

Collection matches calls by model_call_id, or by the exact pair of model_ref and response_id. A match must be unique and consistent with any other supplied identifiers. Timestamps and list order do not establish ownership, including for concurrent calls.

Missing evidence makes dependent health checks unobserved. Unmatched, ambiguous, or conflicting references produce a capture-mismatch finding. A gap alone does not mean the task failed.

Joined Example

This synthetic rollout contains one model call requesting a tool, its observed execution, and the resulting trajectory. Optional fields are omitted. The harness provides no semantic turns, so collection records turns_unavailable instead of inferring a turn from the model call.

{
  "ng_model_call_capture": {
    "rollout_id": "0-0",
    "metrics": {"num_calls": 1, "tokens_in": 12, "tokens_out": 8},
    "calls": [{"model_call_id": "call-1", "tokens_in": 12, "tokens_out": 8}]
  },
  "ng_agent_observations": {
    "source": "example_agent",
    "records": [
      {
        "kind": "agent_invocation",
        "invocation_id": "root",
        "model_calls": [{"model_call_id": "call-1"}],
        "conversation": [
          {"type": "function_call", "call_id": "tool-1", "name": "lookup", "arguments": "{}"},
          {"type": "function_call_output", "call_id": "tool-1", "output": "42"}
        ]
      },
      {
        "kind": "tool_call", "invocation_id": "root", "tool_call_id": "tool-1",
        "tool_name": "lookup", "duration_ms": 200, "timing_source": "harness",
        "status": "completed"
      }
    ],
    "gaps": []
  },
  "ng_trajectory": {
    "schema_version": "1.0",
    "task_id": "task-7",
    "rollout_id": "0-0",
    "invocations": [{
      "kind": "agent_invocation",
      "invocation_id": "root",
      "model_calls": [{"model_call_id": "call-1"}],
      "conversation": [
        {"type": "function_call", "call_id": "tool-1", "name": "lookup", "arguments": "{}"},
        {"type": "function_call_output", "call_id": "tool-1", "output": "42"}
      ]
    }],
    "turns": [],
    "model_calls": [{
      "model_call_id": "call-1",
      "request": {"input": "Look up the answer"},
      "response": {"output": [
        {"type": "function_call", "call_id": "tool-1", "name": "lookup", "arguments": "{}"}
      ]},
      "token_stats": {"prompt_tokens": 12, "completion_tokens": 8}
    }],
    "tool_calls": [{
      "kind": "tool_call", "invocation_id": "root", "tool_call_id": "tool-1",
      "tool_name": "lookup", "duration_ms": 200, "timing_source": "harness",
      "status": "completed", "output": "42"
    }],
    "gaps": [{"code": "turns_unavailable"}]
  },
  "ng_perf": {
    "num_turns": 1, "num_tool_calls": 1, "token_observability_coverage": 1.0,
    "prompt_tokens": 12, "completion_tokens": 8, "total_latency_ms": 910
  }
}

The invocation references call-1. Tool timing joins the conversation output through (invocation_id, tool_call_id), giving the trajectory tool record its output of 42.

Add Evidence to a Custom Agent

Materialized task rows use the scalar task_id.task_id for trajectory and turn task IDs. A valid explicit row trajectory_identity overrides both task and rollout IDs. Otherwise, rollout IDs retain the capture identity and retry attempt suffix. Match these IDs when supplying producer ng_trajectory records and their turns.

Preserve the rollout URL prefix and return an AgentObservationBundle as ng_agent_observations:

• Use stable invocation IDs and exact model-call references.
• Put model-visible tool calls and results in the invocation’s conversation; put measured execution intervals in ToolCallObservation. Join them by invocation and tool-call ID.
• Record missing evidence as gaps. Sandbox usage must be measured, not copied from configured limits or assigned to individual overlapping tools.
• Report the agent’s response usage as the sum over model responses that finished. A call that failed without a finished response adds nothing, so report zero tokens when no call finished, for example when every request was rejected. Omit usage only when a finished response’s usage was lost; rollout health then leaves the token comparison unobserved.
• Supply explicit TrajectoryTurn records in a producer ng_trajectory when semantic turns are observable. Collection preserves them; it does not derive them from an invocation’s conversation.

See the observation schemas for field definitions.

Health and Performance

Rollout health checks read ng_trajectory without reopening capture files. They check evidence integrity; the verifier determines task success.

ng_perf summarizes invocation-owned calls and tools. Its turn count uses explicit turns, then owned model-call counts, then one per invocation. This fallback explains num_turns: 1 in the example even though semantic turns are unavailable. Token totals include only observed values; token_observability_coverage reports matched calls divided by counted turns, capped at one.

Collection adds measured wall-clock latency when available. ng_perf is absent when observability is disabled or no valid trajectory invocation exists. Aggregate metrics include perf_summary when rollouts contain ng_perf.