> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Reverify Rollouts

> Recompute rewards from existing rollouts without re-running model inference.

`gym eval reverify` replays stored rollouts through a resources server's `/verify` endpoint to produce new rewards. Use it when you want to change a verifier parameter — a grading mode, a threshold, a judge prompt — and measure the effect on the **same trajectories** you already collected, without paying for model inference again.

## When to use

Rollout collection fuses two steps that have very different costs: model inference (expensive) and reward computation (cheap). Both outputs land in `rollouts.jsonl` at collection time, and the reward is frozen there.

`gym eval reverify` is the right tool when:

* You changed a verifier hyperparameter and want to recompute rewards on existing rollouts.
* You want to compare two verifier configurations on the **same** rollouts, without sampling noise from different trajectories.
* You iterated on a judge prompt and want to rescore a rollout set without re-running the policy.

LLM-as-judge verifiers will re-invoke the judge model during reverification. This is still far cheaper than re-running the full policy.

## Prerequisites

You need the two artifact files that `gym eval run` writes alongside the main rollouts output:

| File                                 | Contents                                                                                                                                                                            |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rollouts.jsonl`                     | One row per rollout — includes the model `response` (the expensive artifact)                                                                                                        |
| `rollouts_materialized_inputs.jsonl` | One row per rollout — includes per-task verifier inputs (for example, `options` and `expected_answer` in the `mcqa` resources server) that are not echoed back into the rollout row |

Both files are required. `rollouts.jsonl` alone is not sufficient because many resources servers drop verifier-specific fields from the verify response; those fields are preserved in `*_materialized_inputs.jsonl`.

## Step 1 — Run reverification

`gym eval reverify` starts the resources server automatically from the config you provide. It does **not** need a model server or an agent — pass a resources-server-only config to avoid starting inference infrastructure you don't need.

```yaml
# my_resources_server.yaml
mcqa:
  resources_servers:
    mcqa:
      entrypoint: app.py
      domain: knowledge
      grading_mode: lenient_boxed   # the updated hyperparameter
```

Then run:

```bash
gym eval reverify \
  --config my_resources_server.yaml \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts_lenient_reverified.jsonl
```

The command:

1. Starts the resources server from the provided config.
2. Loads both input files and joins rows on `(_ng_task_index, _ng_rollout_index)`.
3. Checks that the resources server supports reverification (see [ReverifyMode](#reverifymode) below).
4. POSTs each joined row to `/verify` — no model calls.
5. Writes recomputed rollouts to `--output` and aggregate metrics to `<output>_aggregate_metrics.json`.

### Reverifying against a benchmark config

You don't have to write a resources-server-only config. You can point `--config` at an existing benchmark or agent config and override verifier parameters inline with `++` — convenient when you'd rather not maintain a separate file just to tweak a hyperparameter e.g.:

```bash
gym eval reverify \
  --config benchmarks/birdbench/config.yaml \
  --model-type vllm_model \
  "++birdbench_bird_sql_resources_server.resources_servers.bird_sql.sql_execution_timeout_s=60" \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts_reverified.jsonl
```

A benchmark/agent config's agent references a model server, so a model config must also be present for that reference to resolve — pass `--model-type` (or the model config you used at collection time). The trade-off of this convenience: reverify then starts **all** the servers in the config (agent and model included), even though it only calls the resources server's `/verify`.

## Output files

| File                              | Contents                                                                   |
| --------------------------------- | -------------------------------------------------------------------------- |
| `<output>.jsonl`                  | Recomputed rollouts with updated `reward` and any verifier-specific fields |
| `<output>_failures.jsonl`         | Sidecar with rows that encountered a non-terminal verifier failure         |
| `<output>_aggregate_metrics.json` | Aggregate metrics computed from the recomputed rollouts                    |

## Options

| Option                  | Description                                                                                                                                         |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--config PATH`         | Resources-server config YAML. Repeatable.                                                                                                           |
| `--inputs PATH`         | `*_materialized_inputs.jsonl` from `gym eval run`.                                                                                                  |
| `--rollouts PATH`       | `rollouts.jsonl` from `gym eval run`.                                                                                                               |
| `--output PATH`, `-o`   | Output JSONL for recomputed rollouts.                                                                                                               |
| `--force`               | Override the `UNSUPPORTED` or `UNKNOWN` reverify mode guard (see below). Output is prefixed with `unsafe_`.                                         |
| `--overwrite`           | Delete an existing output file instead of raising an error.                                                                                         |
| `--resume`              | Resume a partial run: skip rows already in the output (and its failures sidecar) and re-verify only the rest.                                       |
| `--judge-failed-only`   | Recover only the rollouts whose judge call failed in the original run (see [Recover judge failures](#recover-judge-failures)).                      |
| `--append`              | With `--judge-failed-only`: append recovered rows to an existing `--output` instead of writing a fresh file. Mutually exclusive with `--overwrite`. |
| `--disable-aggregation` | Skip aggregate-metrics computation. Use when sharding (see [Sharding](#sharding)).                                                                  |

Hydra overrides work the same as in other `gym eval` commands:

```bash
gym eval reverify \
  --config my_resources_server.yaml \
  "++mcqa_resources_server.resources_servers.mcqa.grading_mode=strict_single_letter_boxed" \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts_strict_reverified.jsonl
```

## ReverifyMode

Before starting, `gym eval reverify` calls `GET /reverify_mode` on each resources server it will use. Servers report one of three modes:

| Mode          | Meaning                                                                                                                                                                                               |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stateless`   | Reverification is safe. The verifier is a pure function of `(request body, server config)`. Set `REVERIFY_MODE = ReverifyMode.STATELESS` on your resources server config once you have verified this. |
| `unsupported` | Reverification is not safe. The verifier reads per-rollout session state that was seeded during the original run and is no longer present. Results would be silently wrong.                           |
| `unknown`     | The resources server has not declared its reverification behaviour. This is the **default** for all resources servers until explicitly set. Treat it as potentially unsafe.                           |

Session-stateful resources servers (for example, those that use cookie-keyed state seeded by `/seed_session`) should report `unsupported`. Resources servers that have not yet been audited for reverification safety report `unknown` (the default).

By default, `gym eval reverify` refuses to proceed against any server reporting `unsupported` or `unknown`, and prints a clear error.

If you understand the risk and want to run anyway — for example, to get an approximate lower-bound estimate — pass `--force`. The output file will be prefixed with `unsafe_` to mark it as potentially incorrect:

```bash
gym eval reverify \
  --config my_stateful_server.yaml \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts_reverified.jsonl \
  --force
# Output written to: results/unsafe_rollouts_reverified.jsonl
```

## Recover judge failures

When a run uses an LLM-as-judge verifier, the judge is a second model call made *after* the expensive policy inference. If that judge call fails (bad key, unreachable or rate-limited endpoint), the sample fails even though the policy `response` is already computed and stored in `<rollouts>_failures.jsonl`. `--judge-failed-only` re-runs the judge on **only** those rollouts, reusing the stored responses — no inference — and merges the recovered rows back with the ones that already scored, so the metrics match a clean run.

* `--rollouts` is the run's **successful** rollouts (`rollouts.jsonl`); they are carried through unchanged.
* The failed subset is read automatically from the sidecar next to it (`rollouts_failures.jsonl`), filtered to the rows tagged as judge failures.
* The reverify-mode guard is skipped, so `--force` is not needed and the output is not prefixed with `unsafe_`.

```bash
gym eval reverify --judge-failed-only \
  --config my_resources_server.yaml \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts_recovered.jsonl
```

The output holds the successes plus the recovered rows. Re-running is idempotent, and `--resume` picks up any rows a fixed-but-still-flaky judge left behind.

The judge model must be healthy (and configured the same as the original run) when you recover. Since the recovered rewards are computed now while the successes were scored during the original run, keep the verifier/judge config identical between the two so the merged metrics stay consistent.

To add the recovered rows onto a file that already holds the successes — for example, to complete the run's own `rollouts.jsonl` in place — use `--append` instead of writing a fresh output:

```bash
gym eval reverify --judge-failed-only --append \
  --config my_resources_server.yaml \
  --inputs results/rollouts_materialized_inputs.jsonl \
  --rollouts results/rollouts.jsonl \
  --output results/rollouts.jsonl
```

## Sharding

For large rollout sets you can shard reverification across multiple jobs by splitting the input files, passing `--disable-aggregation` on each shard, and then combining with `gym eval aggregate`:

```bash
# Shard 0
gym eval reverify \
  --config my_resources_server.yaml \
  --inputs shards/inputs_0.jsonl \
  --rollouts shards/rollouts_0.jsonl \
  --output shards/reverified_0.jsonl \
  --disable-aggregation

# Shard 1
gym eval reverify \
  --config my_resources_server.yaml \
  --inputs shards/inputs_1.jsonl \
  --rollouts shards/rollouts_1.jsonl \
  --output shards/reverified_1.jsonl \
  --disable-aggregation

# Merge and compute aggregate metrics
gym eval aggregate \
  --config my_resources_server.yaml \
  --input-glob 'shards/reverified_*.jsonl' \
  --output results/reverified.jsonl
```