Plan a Comparison Budget
Suppose you want to compare a quantized checkpoint with a baseline. You have historical repeat-level standard deviations and want to know how many complete benchmark passes to collect before deciding whether the candidate’s quality loss is acceptable.
This walkthrough uses gym eval plan-repeats to plan one benchmark, reuse a baseline across several candidates, and plan a suite with an existing baseline. It also shows how a benchmark with two metrics keeps the same weight as a benchmark with one. All SDs here are illustrative; the commands perform CPU-only statistical calculations and do not start model inference.
Prerequisites
Complete Installation and run the commands from a NeMo Gym checkout containing examples/repeat-planning/.
One repeat is one complete pass over a fixed benchmark task population. Supply the standard deviation across those complete-pass scores, on the normalized 0–100 scale, with higher scores better. An SD across individual tasks or the standard error of an existing mean is a different quantity. See Plan Repeats for the complete input contract.
1. Define the decision before collecting results
Start with one model and one benchmark. Assume a complete-pass SD of one score point for both checkpoints. The goal is to establish non-inferiority: the candidate’s quality loss is smaller than the amount you agreed to tolerate. Choose these inputs before running the experiment:
Power and false acceptance describe repeated experiments under different assumed true differences, using the supplied SDs. They are planning probabilities, not the probability that a particular candidate is good. For this one-sided 5% test, confirm acceptable quality when the lower endpoint of the two-sided 90% Welch interval is strictly above −1 point.
Copy the complete single-benchmark example:
The copied file contains:
aggregation_level: benchmark makes the quality claim about this one model × benchmark entry. baseline.mode: optimize asks for new baseline and candidate counts. With no repeat_step or repeat_cost, all integer counts of at least two are allowed and each pass has unit cost.
2. Run the planner and read the report
The command writes repeat_planning_report.md and repeat_planning_report.json. The example produces:
These simulation values use the default 10,000 trials and seed 20261109. The Markdown report rounds values; JSON retains full precision.
The allocation meets the analytic target. Its simulation checks the actual Welch decision with newly estimated sample variances and degrees of freedom. The two power values need not be identical. The Monte Carlo interval measures simulation uncertainty; it is not a confidence interval for the candidate’s eventual score difference and does not measure uncertainty in the supplied historical SDs.
One candidate does not require equal arm counts. The exact integer search finds a 27-pass allocation for this input, while constraining both arms to equal counts would require 28 passes. The report identifies the search method so you can distinguish a scalar minimum from a suite’s feasible local allocation.
3. Reuse a planned baseline across 48 candidates
Suppose the baseline will serve 48 distinct candidate checkpoints, all with the same candidate SD forecast. Copy the previous input:
In shared-baseline.yaml, change only candidate_experiments from 1 to 48, then run:
The larger baseline investment reduces each candidate’s planned repeat count. The count of 48 changes the total cost objective; it does not turn 48 checkpoints into repeat observations for one comparison. Power remains marginal for one candidate-versus-baseline decision, rather than a guarantee that all 48 decisions succeed.
4. Plan a suite against an existing baseline
Next, suppose baseline passes are already available: 16 for GPQA and 32 for AA-LCR. The checked-in suite example supplies SDs of two and three points, respectively, for both arms. Its repeat increments are eight and sixteen passes, matching fixed prepared dataset copies.
This file uses aggregation_level: suite and baseline.mode: existing. Its report shows:
The existing 48 baseline passes contribute uncertainty but no new collection cost. The plan requests 256 new candidate passes in total and has approximate power 80.16% for the equal-weight suite average. The search reports a feasible local allocation, without claiming a global integer minimum.
The runtime multiplier column shows how to apply the counts when prepared copies stay fixed. The planner reports counts; you apply them to the evaluation’s launch configuration. It does not edit that configuration or submit a run.
This suite plan supports an average-quality claim. It does not independently power GPQA and AA-LCR at 80%. To plan a separate benchmark claim, make a separate file with aggregation_level: benchmark and only that benchmark’s entry. Use aggregation_level: model when averaging selected benchmarks of one model; each file still describes one aggregate.
5. Prepare one score per benchmark
The planner accepts one score per model × benchmark entry, even when that score combines several metrics. Normalize each selected metric to 0–100 using its theoretical bounds and direction, then equally average the normalized metrics within each complete repeat. Calculate sample SD across those averages. Use the same score definition for both checkpoints; estimate their SDs separately, or omit candidate_std_points to reuse the baseline SD forecast.
The following three-repeat calculation illustrates this preparation for accuracy and a lower-is-better hallucination rate. It is not measured AA Omniscience data or a definition of its published index. Use the exact metric definitions, including their denominators, selected in your final comparison.
Accuracy uses bounds 0–1 with higher scores better. The illustrative rate uses bounds 0–100 with lower values better, so its normalized score is 100 - rate. Python’s statistics.stdev calculates sample SD with divisor number_of_repeats - 1:
Run the complete suite example, which supplies one scalar SD for GPQA and one for an illustrative two-metric Omniscience score:
Its existing baseline counts and repeat increments are eight for GPQA and ten for Omniscience. Each entry receives weight 1 / N, where N = 2 is the number of model × benchmark entries. Within an average of M metrics, each metric has weight 1 / M:
Omniscience’s two metrics form one benchmark score with one repeat count and one repeat cost. For this illustrative plan, the candidate uses the same 0.9-point SD forecast as baseline. The planning file and report contain only the scalar benchmark score SDs; the final comparison manifest records the selected metrics, normalization bounds, and directions.
The minimum allowed new counts already meet the target: eight GPQA passes and ten Omniscience passes per candidate, 18 new passes in total, with approximate suite power 91.87%. These results use this example’s SDs and existing counts; they are not measured AA Omniscience or GPQA budgets.
6. Collect the planned passes and compare once
Keep the input YAML and report with the experiment. Collect the reported baseline and per-candidate counts using the evaluation workflow appropriate to your benchmark, keeping tasks, scoring, sampling, and other comparison settings fixed. The commands above only plan the budget.
If your benchmark uses a single higher-is-better reward metric on a 0–1 scale, use the same decision level:
For multi-metric benchmarks, metrics with other bounds or directions, model averages, and suites, configure the collected runs in a comparison manifest with the same metric selection, normalization bounds, and directions used to estimate the planning SDs. Use a one-cell manifest for a benchmark claim, the per-model aggregate for a model claim, or the overall aggregate over the same entries for a suite claim:
These final commands require the collected artifacts; the planning walkthrough does not create them. A comparison manifest names run paths and metrics, while a planning YAML contains SD inputs and design settings. See Compare Results for the comparison manifest and report walkthrough.
For the one-point margin used here, establish non-inferiority when the resulting delta interval’s lower endpoint is above −1 normalized point. For a single higher-is-better 0–1 reward, the threshold is −0.01 in compare’s raw units. compare does not produce the multi-metric composite interval; use the matching compare-suite aggregate for that claim. An interval crossing the threshold is inconclusive; it does not establish that the candidate is too much worse. Keep that outcome rather than automatically extending the experiment after inspecting the result.