Reverify Rollouts

gym eval reverify replays stored rollouts through a resources server’s /verify endpoint to produce new rewards. Use it when you want to change a verifier parameter — a grading mode, a threshold, a judge prompt — and measure the effect on the same trajectories you already collected, without paying for model inference again.

When to use

Rollout collection fuses two steps that have very different costs: model inference (expensive) and reward computation (cheap). Both outputs land in rollouts.jsonl at collection time, and the reward is frozen there.

gym eval reverify is the right tool when:

• You changed a verifier hyperparameter and want to recompute rewards on existing rollouts.
• You want to compare two verifier configurations on the same rollouts, without sampling noise from different trajectories.
• You iterated on a judge prompt and want to rescore a rollout set without re-running the policy.

Prerequisites

You need the two artifact files that gym eval run writes alongside the main rollouts output:

Both files are required. rollouts.jsonl alone is not sufficient because many resources servers drop verifier-specific fields from the verify response; those fields are preserved in *_materialized_inputs.jsonl.

Step 1 — Run reverification

gym eval reverify starts the resources server automatically from the config you provide. It does not need a model server or an agent — pass a resources-server-only config to avoid starting inference infrastructure you don’t need.

1
# my_resources_server.yaml
2
mcqa:
3
  resources_servers:
4
    mcqa:
5
      entrypoint: app.py
6
      domain: knowledge
7
      grading_mode: lenient_boxed   # the updated hyperparameter

Then run:

$
gym eval reverify \
>
  --config my_resources_server.yaml \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts_lenient_reverified.jsonl

The command:

1. Starts the resources server from the provided config.
2. Loads both input files and joins rows on (_ng_task_index, _ng_rollout_index).
3. Checks that the resources server supports reverification (see ReverifyMode below).
4. POSTs each joined row to /verify — no model calls.
5. Writes recomputed rollouts to --output and aggregate metrics to <output>_aggregate_metrics.json.

Reverifying against a benchmark config

You don’t have to write a resources-server-only config. You can point --config at an existing benchmark or agent config and override verifier parameters inline with ++ — convenient when you’d rather not maintain a separate file just to tweak a hyperparameter e.g.:

$
gym eval reverify \
>
  --config benchmarks/birdbench/config.yaml \
>
  --model-type vllm_model \
>
  "++birdbench_bird_sql_resources_server.resources_servers.bird_sql.sql_execution_timeout_s=60" \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts_reverified.jsonl

Output files

Options

Hydra overrides work the same as in other gym eval commands:

$
gym eval reverify \
>
  --config my_resources_server.yaml \
>
  "++mcqa_resources_server.resources_servers.mcqa.grading_mode=strict_single_letter_boxed" \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts_strict_reverified.jsonl

ReverifyMode

Before starting, gym eval reverify calls GET /reverify_mode on each resources server it will use. Servers report one of three modes:

Session-stateful resources servers (for example, those that use cookie-keyed state seeded by /seed_session) should report unsupported. Resources servers that have not yet been audited for reverification safety report unknown (the default).

By default, gym eval reverify refuses to proceed against any server reporting unsupported or unknown, and prints a clear error.

If you understand the risk and want to run anyway — for example, to get an approximate lower-bound estimate — pass --force. The output file will be prefixed with unsafe_ to mark it as potentially incorrect:

$
gym eval reverify \
>
  --config my_stateful_server.yaml \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts_reverified.jsonl \
>
  --force
$
# Output written to: results/unsafe_rollouts_reverified.jsonl

Recover judge failures

When a run uses an LLM-as-judge verifier, the judge is a second model call made after the expensive policy inference. If that judge call fails (bad key, unreachable or rate-limited endpoint), the sample fails even though the policy response is already computed and stored in <rollouts>_failures.jsonl. --judge-failed-only re-runs the judge on only those rollouts, reusing the stored responses — no inference — and merges the recovered rows back with the ones that already scored, so the metrics match a clean run.

• --rollouts is the run’s successful rollouts (rollouts.jsonl); they are carried through unchanged.
• The failed subset is read automatically from the sidecar next to it (rollouts_failures.jsonl), filtered to the rows tagged as judge failures.
• The reverify-mode guard is skipped, so --force is not needed and the output is not prefixed with unsafe_.

$
gym eval reverify --judge-failed-only \
>
  --config my_resources_server.yaml \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts_recovered.jsonl

The output holds the successes plus the recovered rows. Re-running is idempotent, and --resume picks up any rows a fixed-but-still-flaky judge left behind.

To add the recovered rows onto a file that already holds the successes — for example, to complete the run’s own rollouts.jsonl in place — use --append instead of writing a fresh output:

$
gym eval reverify --judge-failed-only --append \
>
  --config my_resources_server.yaml \
>
  --inputs results/rollouts_materialized_inputs.jsonl \
>
  --rollouts results/rollouts.jsonl \
>
  --output results/rollouts.jsonl

Sharding

For large rollout sets you can shard reverification across multiple jobs by splitting the input files, passing --disable-aggregation on each shard, and then combining with gym eval aggregate:

$
# Shard 0
$
gym eval reverify \
>
  --config my_resources_server.yaml \
>
  --inputs shards/inputs_0.jsonl \
>
  --rollouts shards/rollouts_0.jsonl \
>
  --output shards/reverified_0.jsonl \
>
  --disable-aggregation
$
$
# Shard 1
$
gym eval reverify \
>
  --config my_resources_server.yaml \
>
  --inputs shards/inputs_1.jsonl \
>
  --rollouts shards/rollouts_1.jsonl \
>
  --output shards/reverified_1.jsonl \
>
  --disable-aggregation
$
$
# Merge and compute aggregate metrics
$
gym eval aggregate \
>
  --config my_resources_server.yaml \
>
  --input-glob 'shards/reverified_*.jsonl' \
>
  --output results/reverified.jsonl