Plan Repeats
Use gym eval plan-repeats before running evaluation benchmarks with NeMo Gym. Its goal is to choose appropriate repeat counts for your statistical requirements, such as acceptable loss and target power. Historical standard deviations can inform this planning. Plan one benchmark, a model’s selected benchmarks, or a suite average. The command reads a small YAML file; it does not start evaluations or access a cluster.
For a step-by-step example with expected counts, reports, baseline reuse, and multiple metrics, follow Plan a Comparison Budget.
One repeat means one complete pass over the fixed benchmark task population. Supply the standard deviation of those complete-pass scores, rather than the standard deviation across tasks or the standard error of an existing average. Scores and standard deviations use the same normalized 0–100 scale as suite comparison, with all selected metrics oriented so higher scores are better.
Each model × benchmark entry represents one score. Supply its baseline_std_points; omit candidate_std_points to reuse that baseline SD forecast. If the score averages several metrics, construct one score within each repeat before estimating its SD. The planning YAML contains the resulting scalar SDs; keep metric definitions and extraction evidence with the source data.
Plan against an existing baseline
Save this example as planning.yaml. The SDs are illustrative.
Runnable examples are checked in under examples/repeat-planning/: a single benchmark, the existing-baseline suite above, and a suite whose Omniscience entry supplies the SD of an illustrative two-metric score. Their SDs are illustrative rather than measurements.
baseline.mode: existing keeps each cell’s available baseline count fixed and plans additional candidate repeats. The baseline’s uncertainty stays in the calculation, while its collection cost is already paid. Existing counts need not be multiples of the current repeat_step.
repeat_step is the increment for newly collected complete passes. Eight fixed prepared dataset copies permit 8, 16, 24, … planned passes, corresponding to runtime multipliers 1, 2, 3, …. Apply the reported count by adjusting the runtime multiplier while keeping prepared copies fixed; the planner does not modify the launch configuration. The default step is one, and new counts are always at least two.
Choose the quality claim
Each file produces one aggregate plan over its listed model × benchmark entries:
All entries receive equal weight. A sparse suite therefore gives models with more listed benchmarks more total weight. A suite-average claim does not establish each benchmark’s or model’s quality. Use separate files for separate benchmark claims.
margin_points: 1 permits a loss smaller than one normalized score point; it is not a 1% relative loss. expected_delta_points is the forecast true candidate-minus-baseline change, with zero forecasting equal quality. Set both before seeing the comparison’s results.
alpha: 0.05 selects a one-sided 5% non-inferiority test. This uses the lower endpoint of a two-sided 90% Welch interval: the candidate establishes non-inferiority when that endpoint is above the negative margin. With either compare or compare-suite, set --confidence-level 0.90 to use that decision. A two-sided 95% interval instead gives a one-sided 2.5% test.
Use compare-suite with the same selected metrics, normalization bounds, and directions to evaluate the planned score. A benchmark claim uses a one-cell comparison manifest; a model claim uses that model’s aggregate, and a suite claim uses the overall aggregate over the same entries. gym eval compare reports separate intervals in raw metric units. It matches the walkthrough’s one-point decision directly for a single higher-is-better metric on a 0–1 scale, where the threshold is −0.01.
Plan baseline investment across candidate checkpoints
To plan a baseline and 48 distinct candidate checkpoints, change the baseline mode, set the candidate count, and remove existing_baseline_repeats from every entry:
The planner pays for the baseline once and for each candidate’s planned passes 48 times. All candidates share the supplied candidate SD forecast and receive the same per-candidate allocation. candidate_experiments defaults to one. It counts distinct candidate checkpoints, not repeat observations. One candidate does not require equal baseline and candidate repeat counts.
repeat_cost optionally supplies a positive cost for one complete pass in a cell, using common units across all cells and the same per-pass cost in both arms. It defaults to one, which minimizes new pass count. With measured GPU-hours per pass, the result is projected additional GPU-hours. Candidate experiment count affects cost, not the variance or marginal power of one candidate comparison. Shared-baseline comparisons do not provide a simultaneous all-candidates guarantee.
Prepare one score from several metrics
Prepare the same benchmark score that the final comparison will evaluate:
- Normalize each selected metric to 0–100 using its theoretical bounds, reversing lower-is-better metrics so higher scores are better. Use the same definitions, bounds, and directions across models and checkpoints.
- Within each complete repeat, average the
Mselected normalized metrics with weight1 / Meach. - Calculate sample SD across those repeat averages, using divisor
number_of_repeats - 1. Supply this one scalar asbaseline_std_points. Estimate the candidate SD separately when data are available, or omitcandidate_std_pointsto forecast the same SD as baseline.
The planning YAML does not accept a metrics list. Supply the scalar SD of the prepared benchmark score. For example, this entry supplies an illustrative 0.9-point SD:
Every listed model × benchmark entry receives weight 1 / N in a plan with N entries, regardless of how many metrics formed its score. All selected metrics share that benchmark’s repeat count and cost. Define metric selection in the final compare-suite manifest; planning inputs contain one SD per benchmark score. The walkthrough shows a complete numerical example of preparing this SD.
Inspect the result
The command writes repeat_planning_report.md and repeat_planning_report.json. The Markdown report shows benchmark score inputs, equal entry weights, baseline and per-candidate counts, additional work and cost, analytic power, and validation results. JSON retains full precision. Neither report includes worked arithmetic.
The analytic power calculation uses forecast variances, Welch–Satterthwaite degrees of freedom, and a noncentral-t tail. Its numerical implementation shares the uncertainty calculation with NeMo Gym’s existing Welch comparison. This prospective power is approximate: the actual future comparison recalculates variances and degrees of freedom from collected scores.
The default validation uses 10,000 normal-model experiments and reports power and false-success rate with 95% Wilson Monte Carlo intervals. validation_trials and seed are configurable in YAML; set validation_trials: 0 to skip this diagnostic. Simulation holds the supplied variance forecasts fixed, so its intervals measure simulation precision rather than uncertainty in historical SD estimates. A disagreement with the analytic target remains visible; simulation does not change the chosen allocation.
This diagnostic repeats hypothetical experiments by redrawing sample means and sample variances in both arms, including when baseline counts are existing. It is not a conditional power estimate given the already collected baseline observations.
Single-benchmark searches check legal integer allocations and identify an exact minimum within max_repeats (default 20,000 newly planned repeats per arm). Existing baseline counts may exceed that limit. Suite searches report a feasible local allocation rather than a globally optimal integer solution. Exhausting a bound means that no feasible plan was found within that search, not that the target is impossible. A zero total forecast variance cannot support a Welch power calculation.
If the final planned experiment is inconclusive, retain that result. Adding repeats after inspecting a confidence interval requires a separately designed sequential procedure; this command plans fixed counts in advance.
Use saved cluster results
On a cluster, obtain the SD across complete-repeat scores from the selected run’s saved aggregate metrics. For a multi-metric benchmark, normalize and equally average the selected metrics within each repeat first, then calculate the SD across those averaged scores. For multiple independent source runs, concatenate their complete-repeat scores and recompute SD; do not average source SDs or use standard error as SD. Normalize before constructing the direct-input YAML.
The same YAML runs on a laptop or on the cluster once NeMo Gym is installed. It contains only numerical inputs and benchmark labels, so no rollout text, prompts, cluster connection, or inference service is required. Keep the source run identities and SD-extraction evidence with the experiment’s validation artifacts.
See Compare Results for the completed experiment and CLI reference for command options.