Rollout Health Checks

Rollout health checks verify the integrity and observability of completed evaluation artifacts. They detect unreadable records, missing agent activity, failed or incomplete policy-model calls, inconsistent token counts, and failures repeated across every attempt at a task.

Health checks do not evaluate whether an answer is correct. The verifier and reward determine task success; rollout health determines whether the saved evidence is complete and internally consistent enough to trust and investigate the result.

This page follows Gym’s definitions of task, rollout or trajectory, agent harness, and policy model. Health-specific terms such as finding, verdict, and bound policy-model call are defined below where they are used.

When Health Checks Run

gym eval run runs health checks after rollout collection and aggregate-metrics computation. gym eval aggregate runs health checks after aggregating the selected shards. Both commands print a short report:

Rollout health: 100 checked, 96 healthy, 3 unhealthy, 1 unobserved
Quality summary: results/quality_summary.json

If gym eval run uses --disable-aggregation, it also defers health verification. Run gym eval aggregate over the completed shards to produce the combined health report.

To check an existing run directly, use:

$
gym eval health-check results/my-run

The standalone command reads results/my-run/rollouts.jsonl by default. Select a different file explicitly when the rollout filename is nonstandard:

$
gym eval health-check results/my-run \
>
    --rollouts-file evaluator_rollouts.jsonl \
>
    --workers 8

A relative --rollouts-file path resolves under the run directory. An absolute path is used as written. Reports from the standalone command are always written in the run directory.

Refer to the gym eval CLI reference for all flags.

Required Evidence

Health checks read each rollout record and its standard ng_trajectory attachment. ng_trajectory combines the agent turns, model-call evidence, explicit model-call references, and observation gaps produced during collection. Refer to the trajectory capability matrix for producer coverage and model-call capture for collection configuration.

The health checks runner does not reopen raw model-call capture files, inspect ng_model_call_capture, or reconstruct turns from an agent-specific response. A custom rollout driver receives the same behavior as a built-in driver: checks run when it writes a valid ng_trajectory, and checks that need missing evidence are unobserved.

Findings and Verdicts

A finding is evidence emitted by one check. Checks do not emit verdicts. After all enabled rollout-level checks run, the runner derives one verdict for each rollout using this priority:

healthy is therefore a strong claim about all enabled checks, not merely the checks that happened to be computable. A rollout can be correct according to its reward and still be unhealthy, or incorrect according to its reward and healthy.

Task-level checks add flags to the task summary after rollout verdicts are derived. Tasks do not receive a separate verdict.

Check Catalog

Check IDs identify both the evidence being evaluated and the condition detected. Rollout-level checks create findings in rollout_verdicts.jsonl; task-level checks create flags in quality_summary.json.

What Counts as Agent Activity

The turn checks read ng_trajectory.turns. Non-empty answer text, reasoning content, or a tool call counts as agent activity. Reasoning-only turns are not hollow. A model-call reference also establishes activity for rollout_missing_agent_turns, although agent_turn_hollow still requires content or a tool call in the turn itself.

Agent logic that advances a rollout without calling the policy model has no bound model call and is outside model-call checks. The runner does not infer a special dispatch marker from sampling parameters or request metadata.

Binding Turns to Policy-Model Calls

An evaluation can contain calls to the policy model and to auxiliary models such as a judge or user simulator. Model-call checks must therefore establish which calls belong to policy-model turns before evaluating them.

The runner binds TrajectoryTurn.model_calls to TrajectoryRecord.model_calls using only one of these explicit identities:

• model_call_id
• the exact pair of model_ref and response_id

It does not match calls by list position, timestamp, model name, or payload similarity. A reference that resolves exactly once becomes a bound policy-model call. Unreferenced model calls are not assigned to the policy model because they can belong to an auxiliary model.

Length-Limited Responses

model_call_runaway_generation needs one provider-independent definition of an empty saved response. The check recognizes non-empty text, content, output text, answer, encrypted reasoning content, reasoning, or reasoning summaries in OpenAI Responses, Chat Completions, and Messages-style responses. If any supported content is present, a length-limited response is not classified as an empty runaway generation.

How Health Checks Interpret Existing Observation Gaps

During rollout collection, Gym’s observability layer writes an ObservationGap to ng_trajectory.gaps when it cannot collect, normalize, or correlate evidence exactly. The health runner does not create these gaps. It maps each persisted gap code to health behavior according to the evidence required by a check. A recorded gap is not automatically a health finding:

Other gap codes affect health only when a check defines an explicit rule for them. The original gap details remain in ng_trajectory.gaps; rollout_verdicts.jsonl records only the IDs of checks that were unobserved.

This distinction keeps absent evidence separate from contradictory evidence: missing input reduces coverage, while an explicit conflict produces a finding.

Task-Level Reduction

task_consistently_unhealthy considers a repeat computable when its rollout verdict is healthy or unhealthy. It requires at least two computable repeats and flags the task when all of them are unhealthy. An additional unobserved repeat does not erase the repeated observed failures.

task_no_successful_model_calls requires complete policy-model-call bindings for every repeat. If any repeat lacks complete bindings, the task check is unobserved. This prevents the runner from claiming that a task had no successful policy-model call when an unobserved repeat might contain one.

Duplicate persisted records are kept in the per-rollout report and counted at run level, but records with the same task and rollout indices count as one repeat during task-level reduction. This prevents copied records from appearing to be independent attempts.

Report Files

Health verification writes two files without changing the selected rollout JSONL or aggregate-metrics file.

quality_summary.json

The summary contains run-level verdict counts, findings, check coverage, raw artifact statistics, and per-task counts and flags. This abridged example omits the other check-coverage entries and zero-valued issue entries:

1
{
2
  "run": {
3
    "ignored_checks": [],
4
    "artifacts": {
5
      "records": 100,
6
      "captures": 98,
7
      "coverage": {
8
        "agent_turn_hollow": {
9
          "evaluated": 98,
10
          "unobserved": 2,
11
          "ignored": 0
12
        }
13
      }
14
    },
15
    "verdicts": {
16
      "healthy": 96,
17
      "unhealthy": 3,
18
      "unobserved": 1
19
    },
20
    "issues": {
21
      "agent_turn_hollow": 2,
22
      "model_call_failed": 1
23
    },
24
    "stats": {
25
      "model_call_errors": {
26
        "total": 1,
27
        "by_status": {"408": 1},
28
        "rollouts_affected": 1,
29
        "ended_on_error": 1
30
      },
31
      "duplicated_calls": {
32
        "replayed": 0,
33
        "rollouts": 0
34
      },
35
      "tokens": {
36
        "prompt": 42000,
37
        "completion": 9000,
38
        "capture_prompt": 42000,
39
        "capture_completion": 9000
40
      }
41
    }
42
  },
43
  "tasks": {
44
    "0": {
45
      "repeats": 4,
46
      "healthy": 3,
47
      "unhealthy": 1,
48
      "unobserved": 0,
49
      "flags": []
50
    }
51
  }
52
}

run.issues counts individual findings, not distinct affected rollouts or tasks. It contains every registered check ID, including IDs whose count is zero. run.artifacts.coverage reports how often each check was evaluated, unobserved, or explicitly ignored.

run.artifacts.captures counts rollout trajectories that contain model-call evidence; health does not count or reopen raw capture files. Raw statistics under run.stats describe all model calls stored in ng_trajectory, including unreferenced auxiliary calls. Checks with a policy-model subject evaluate only exactly bound calls.

rollout_verdicts.jsonl

The per-rollout report contains one row for every non-empty input line and is sorted by _ng_task_index and _ng_rollout_index:

1
{
2
  "_ng_task_index": 12,
3
  "_ng_rollout_index": 0,
4
  "rollout_id": "12-0",
5
  "verdict": "unhealthy",
6
  "findings": [
7
    {
8
      "check": "model_call_failed",
9
      "locator": {"call_id": "call-7"},
10
      "detail": {
11
        "status": 408,
12
        "error_category": "timeout",
13
        "terminal": true
14
      }
15
    }
16
  ],
17
  "unobserved": []
18
}

Each finding contains a stable check ID, an optional locator for the affected turn or model call, and check-specific detail. The file stores each displayed object on one JSONL line. unobserved lists enabled rollout-level checks that lacked required evidence. Ignored checks are recorded only in quality_summary.json, so consumers must read the summary before interpreting verdicts produced with a reduced check set.

If a rollout line is unreadable, its real identity is unavailable. The report assigns a synthetic _ng_task_index such as __unreadable_record__:input-0:line-42, uses _ng_rollout_index: 0, and records the source file and physical line in the finding locator. Other checks are unobserved for that line.

Selecting or Disabling Checks

All registered checks are enabled by default. To skip health verification entirely after a run or aggregation:

$
gym eval run ... --no-health-check
$
gym eval aggregate ... --no-health-check

To exclude specific checks from one automatic run:

$
gym eval run ... \
>
    --health-check-ignore model_call_missing_token_counts,model_call_zero_completion_tokens

For the standalone command, use --ignore-checks or its --ignore alias:

$
gym eval health-check results/my-run \
>
    --ignore-checks model_call_missing_token_counts,model_call_zero_completion_tokens

An ignored check does not execute and does not affect findings, unobserved states, rollout verdicts, or task flags. Unknown check IDs are rejected. The summary records ignored IDs and coverage so a reduced check set is explicit.

Execution and Failure Handling

The runner indexes non-empty JSONL lines by byte offset, then processes each line independently in a process pool. It uses at most eight workers by default, bounded by the available CPU count. --workers and --health-check-workers set an explicit limit. No Ray runtime is required.

If the platform cannot create or execute the process pool, the runner warns and evaluates the same records serially. If one check raises unexpectedly, check_execution_error records the defect, the affected check becomes unobserved, and other checks continue. An unreadable record also receives a report row instead of aborting the run.

These choices favor completing verification and preserving evidence over failing after rollout collection and aggregate metrics have already completed. The report makes degraded execution and missing evidence visible rather than silently treating them as healthy.

Implementation Reference

• TrajectoryRecord, TrajectoryTurn, ObservationGap, and ModelCallRef define the standard evidence consumed by health checks.
• CheckSpec, Finding, and RolloutDigest define the health data contracts.
• The check registry defines the stable check IDs and per-rollout rules.
• The rollout-health runner derives verdicts and writes reports.