> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# Plan a Comparison Budget

> Choose benchmark and suite repeat counts, inspect a complete planning report, and carry the planned decision into a Gym comparison.

Suppose you want to compare a quantized checkpoint with a baseline. You have historical repeat-level standard deviations and want to know how many complete benchmark passes to collect before deciding whether the candidate's quality loss is acceptable.

This walkthrough uses `gym eval plan-repeats` to plan one benchmark, reuse a baseline across several candidates, and plan a suite with an existing baseline. It also shows how a benchmark with two metrics keeps the same weight as a benchmark with one. All SDs here are illustrative; the commands perform CPU-only statistical calculations and do not start model inference.

## Prerequisites

Complete [Installation](/nemo/gym/main/get-started/installation) and run the commands from a NeMo Gym checkout containing `examples/repeat-planning/`.

One repeat is one complete pass over a fixed benchmark task population. Supply the standard deviation across those complete-pass scores, on the normalized 0–100 scale, with higher scores better. An SD across individual tasks or the standard error of an existing mean is a different quantity. See [Plan Repeats](/nemo/gym/main/evaluation/plan-repeats) for the complete input contract.

## 1. Define the decision before collecting results

Start with one model and one benchmark. Assume a complete-pass SD of one score point for both checkpoints. The goal is to establish non-inferiority: the candidate's quality loss is smaller than the amount you agreed to tolerate. Choose these inputs before running the experiment:

| Planning input                                                       | Example value | Meaning in this example                                                                                                                                                                           |
| -------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Acceptable quality loss (margin)                                     | 1 point       | Agree that a true loss of 1 point or more is unacceptable. Confirm acceptable quality only when the confidence interval rules out that loss.                                                      |
| Assumed true score difference                                        | 0 points      | Plan repeats assuming the candidate and baseline truly have the same average score. The difference is candidate minus baseline.                                                                   |
| Chance of confirming acceptable quality (target analytic power)      | 80%           | If the checkpoints truly have equal average scores, aim for about 80 out of 100 experiments to confirm that the candidate loses less than 1 point.                                                |
| False acceptance at the loss boundary (one-sided significance level) | 5%            | Now assume the candidate is truly 1 point worse than baseline. At the nominal test level, about 5 out of 100 experiments could mistakenly confirm acceptable quality because of random variation. |
| Number of candidate checkpoints                                      | 1             | Plan one candidate-versus-baseline comparison. The planner chooses how many complete passes to collect for each checkpoint.                                                                       |

Power and false acceptance describe repeated experiments under different assumed true differences, using the supplied SDs. They are planning probabilities, not the probability that a particular candidate is good. For this one-sided 5% test, confirm acceptable quality when the lower endpoint of the two-sided 90% Welch interval is strictly above −1 point.

Copy the complete single-benchmark example:

```bash
mkdir -p results/repeat-planning-walkthrough
cp examples/repeat-planning/benchmark.yaml \
    results/repeat-planning-walkthrough/benchmark.yaml
```

The copied file contains:

```yaml
schema_version: 1
aggregation_level: benchmark
baseline:
  mode: optimize
candidate_experiments: 1
margin_points: 1.0
target_power: 0.80
alpha: 0.05
expected_delta_points: 0.0
cells:
  - model: model-a
    benchmark: example-benchmark
    baseline_std_points: 1.0
    candidate_std_points: 1.0
```

`aggregation_level: benchmark` makes the quality claim about this one model × benchmark entry. `baseline.mode: optimize` asks for new baseline and candidate counts. With no `repeat_step` or `repeat_cost`, all integer counts of at least two are allowed and each pass has unit cost.

## 2. Run the planner and read the report

```bash
gym eval plan-repeats \
    --plan results/repeat-planning-walkthrough/benchmark.yaml \
    --output-dir results/repeat-planning-walkthrough/benchmark
```

The command writes `repeat_planning_report.md` and `repeat_planning_report.json`. The example produces:

| Report item                           |        Result |
| ------------------------------------- | ------------: |
| New baseline passes                   |            15 |
| New candidate passes                  |            12 |
| Total new passes                      |            27 |
| Approximate planning power            |        80.58% |
| Simulated Welch decision power        |        80.66% |
| Simulation's 95% Monte Carlo interval | 79.87%–81.42% |

These simulation values use the default 10,000 trials and seed `20261109`. The Markdown report rounds values; JSON retains full precision.

The allocation meets the **analytic** target. Its simulation checks the actual Welch decision with newly estimated sample variances and degrees of freedom. The two power values need not be identical. The Monte Carlo interval measures simulation uncertainty; it is not a confidence interval for the candidate's eventual score difference and does not measure uncertainty in the supplied historical SDs.

One candidate does not require equal arm counts. The exact integer search finds a 27-pass allocation for this input, while constraining both arms to equal counts would require 28 passes. The report identifies the search method so you can distinguish a scalar minimum from a suite's feasible local allocation.

## 3. Reuse a planned baseline across 48 candidates

Suppose the baseline will serve 48 distinct candidate checkpoints, all with the same candidate SD forecast. Copy the previous input:

```bash
cp results/repeat-planning-walkthrough/benchmark.yaml \
    results/repeat-planning-walkthrough/shared-baseline.yaml
```

In `shared-baseline.yaml`, change only `candidate_experiments` from `1` to `48`, then run:

```bash
gym eval plan-repeats \
    --plan results/repeat-planning-walkthrough/shared-baseline.yaml \
    --output-dir results/repeat-planning-walkthrough/shared-baseline
```

| Report item                             | One candidate | 48 candidates |
| --------------------------------------- | ------------: | ------------: |
| Baseline passes collected once          |            15 |            30 |
| Passes per candidate checkpoint         |            12 |             9 |
| Candidate passes across all checkpoints |            12 |           432 |
| Total new passes                        |            27 |           462 |
| Approximate power per comparison        |        80.58% |        80.17% |

The larger baseline investment reduces each candidate's planned repeat count. The count of 48 changes the total cost objective; it does not turn 48 checkpoints into repeat observations for one comparison. Power remains marginal for one candidate-versus-baseline decision, rather than a guarantee that all 48 decisions succeed.

## 4. Plan a suite against an existing baseline

Next, suppose baseline passes are already available: 16 for GPQA and 32 for AA-LCR. The checked-in suite example supplies SDs of two and three points, respectively, for both arms. Its repeat increments are eight and sixteen passes, matching fixed prepared dataset copies.

```bash
cp examples/repeat-planning/suite.yaml \
    results/repeat-planning-walkthrough/suite.yaml
gym eval plan-repeats \
    --plan results/repeat-planning-walkthrough/suite.yaml \
    --output-dir results/repeat-planning-walkthrough/suite
```

This file uses `aggregation_level: suite` and `baseline.mode: existing`. Its report shows:

| Benchmark | Suite weight | Existing baseline passes | New candidate passes | Candidate runtime multiplier |
| --------- | -----------: | -----------------------: | -------------------: | ---------------------------: |
| GPQA      |          0.5 |                       16 |                   96 |                           12 |
| AA-LCR    |          0.5 |                       32 |                  160 |                           10 |

The existing 48 baseline passes contribute uncertainty but no new collection cost. The plan requests 256 new candidate passes in total and has approximate power **80.16%** for the equal-weight suite average. The search reports a feasible local allocation, without claiming a global integer minimum.

The runtime multiplier column shows how to apply the counts when prepared copies stay fixed. The planner reports counts; you apply them to the evaluation's launch configuration. It does not edit that configuration or submit a run.

This suite plan supports an average-quality claim. It does not independently power GPQA and AA-LCR at 80%. To plan a separate benchmark claim, make a separate file with `aggregation_level: benchmark` and only that benchmark's entry. Use `aggregation_level: model` when averaging selected benchmarks of one model; each file still describes one aggregate.

## 5. Prepare one score per benchmark

The planner accepts one score per model × benchmark entry, even when that score combines several metrics. Normalize each selected metric to 0–100 using its theoretical bounds and direction, then equally average the normalized metrics **within each complete repeat**. Calculate sample SD across those averages. Use the same score definition for both checkpoints; estimate their SDs separately, or omit `candidate_std_points` to reuse the baseline SD forecast.

The following three-repeat calculation illustrates this preparation for accuracy and a lower-is-better hallucination rate. It is not measured AA Omniscience data or a definition of its published index. Use the exact metric definitions, including their denominators, selected in your final comparison.

| Complete repeat | Accuracy (0–1) | Hallucination rate (%) | Normalized accuracy | Normalized non-hallucination | Equal average |
| --------------- | -------------: | ---------------------: | ------------------: | ---------------------------: | ------------: |
| 1               |           0.80 |                   21.8 |                  80 |                         78.2 |          79.1 |
| 2               |           0.82 |                   22.0 |                  82 |                         78.0 |          80.0 |
| 3               |           0.84 |                   22.2 |                  84 |                         77.8 |          80.9 |

Accuracy uses bounds 0–1 with higher scores better. The illustrative rate uses bounds 0–100 with lower values better, so its normalized score is `100 - rate`. Python's `statistics.stdev` calculates sample SD with divisor `number_of_repeats - 1`:

```python
from statistics import stdev

accuracy = [0.80, 0.82, 0.84]
hallucination_percent = [21.8, 22.0, 22.2]
repeat_scores = [
    (100 * correct + (100 - hallucinated)) / 2
    for correct, hallucinated in zip(accuracy, hallucination_percent, strict=True)
]
baseline_std_points = stdev(repeat_scores)
print(f"{baseline_std_points:.4f}")  # 0.9000
```

Run the complete suite example, which supplies one scalar SD for GPQA and one for an illustrative two-metric Omniscience score:

```bash
cp examples/repeat-planning/multiple-metrics.yaml \
    results/repeat-planning-walkthrough/multiple-metrics.yaml
gym eval plan-repeats \
    --plan results/repeat-planning-walkthrough/multiple-metrics.yaml \
    --output-dir results/repeat-planning-walkthrough/multiple-metrics
```

Its existing baseline counts and repeat increments are eight for GPQA and ten for Omniscience. Each entry receives weight `1 / N`, where `N = 2` is the number of model × benchmark entries. Within an average of `M` metrics, each metric has weight `1 / M`:

| Benchmark score                  | Selected metrics, M | Within-repeat metric weight | Suite entry weight | Supplied baseline / candidate SD |
| -------------------------------- | ------------------: | --------------------------: | -----------------: | -------------------------------: |
| GPQA                             |                   1 |                           1 |                0.5 |                      1 / 1 point |
| Illustrative Omniscience average |                   2 |                         0.5 |                0.5 |                 0.9 / 0.9 points |

Omniscience's two metrics form one benchmark score with one repeat count and one repeat cost. For this illustrative plan, the candidate uses the same **0.9-point SD forecast** as baseline. The planning file and report contain only the scalar benchmark score SDs; the final comparison manifest records the selected metrics, normalization bounds, and directions.

The minimum allowed new counts already meet the target: **eight GPQA passes and ten Omniscience passes per candidate**, 18 new passes in total, with approximate suite power **91.87%**. These results use this example's SDs and existing counts; they are not measured AA Omniscience or GPQA budgets.

## 6. Collect the planned passes and compare once

Keep the input YAML and report with the experiment. Collect the reported baseline and per-candidate counts using the evaluation workflow appropriate to your benchmark, keeping tasks, scoring, sampling, and other comparison settings fixed. The commands above only plan the budget.

If your benchmark uses a single higher-is-better reward metric on a 0–1 scale, use the same decision level:

```bash
gym eval compare \
    --baseline results/baseline/rollouts.jsonl \
    --candidates results/candidate/rollouts.jsonl \
    --confidence-level 0.90 \
    --output-dir results/final-comparison
```

For multi-metric benchmarks, metrics with other bounds or directions, model averages, and suites, configure the collected runs in a comparison manifest with the same metric selection, normalization bounds, and directions used to estimate the planning SDs. Use a one-cell manifest for a benchmark claim, the per-model aggregate for a model claim, or the overall aggregate over the same entries for a suite claim:

```bash
gym eval compare-suite \
    --suite collected-suite.yaml \
    --confidence-level 0.90 \
    --output-dir results/final-suite-comparison
```

These final commands require the collected artifacts; the planning walkthrough does not create them. A comparison manifest names run paths and metrics, while a planning YAML contains SD inputs and design settings. See [Compare Results](/nemo/gym/main/evaluation/compare-results) for the comparison manifest and report walkthrough.

For the one-point margin used here, establish non-inferiority when the resulting delta interval's lower endpoint is above −1 normalized point. For a single higher-is-better 0–1 reward, the threshold is −0.01 in `compare`'s raw units. `compare` does not produce the multi-metric composite interval; use the matching `compare-suite` aggregate for that claim. An interval crossing the threshold is inconclusive; it does not establish that the candidate is too much worse. Keep that outcome rather than automatically extending the experiment after inspecting the result.