nemo_gym.eval.comparison.statistics.repeat_planning
nemo_gym.eval.comparison.statistics.repeat_planning
Plan complete benchmark passes with prospective unpaired Welch power.
Inputs are one-pass variances in normalized score points squared. The noncentral-t power approximation holds those variances fixed. Simulations separately evaluate the actual Welch decision with newly estimated variances; they never change an allocation. This module performs no artifact I/O.
Module Contents
Classes
Functions
API
One model/benchmark’s pass variances, legal increment and per-pass cost.
Baseline and candidate costs share the same arbitrary positive cost unit. Existing baseline passes are already available and are a sunk cost.
Fixed, pre-run non-inferiority design settings.
expected_delta_points is candidate minus baseline. Success means a
one-sided lower bound exceeds -margin_points. alpha=.05 uses the
lower endpoint of a two-sided 90% interval. Candidate experiments share
one baseline and each receive the returned candidate counts; power remains
marginal for one comparison. max_repeats bounds newly planned counts.
Prospective Welch uncertainty and approximate noncentral-t power.
Counts in input-cell order, with baseline paid once and candidates K times.
A simulated event frequency and its pointwise 95% Wilson interval.
Actual-Welch diagnostics at fixed supplied population variances.
Return scalar bounds for newly collected complete-repeat counts.
Evaluate one-sided non-inferiority power for one design or a batch.
For cell i and arm a (baseline or candidate), let v_ia be its forecast one-pass variance, n_ia its complete-repeat count, and w_i its cell weight. The code evaluates the following equations, summing over both indices:
Delta means candidate minus baseline; the null boundary is -margin. The df expression equals V2 / sum(q_ia2 / (n_ia - 1)), using fractions for numerical stability. This is a prospective NCT approximation with fixed forecast variances; actual Welch decisions re-estimate them and df.
Formula references:
- NCSS, pp. 522-2 and 522-4: the null boundary and one-benchmark NCT formula. https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Two-Sample_T-Tests_for_Non-Inferiority_Allowing_Unequal_Variance.pdf#page=4
- NIST: the weighted Welch-Satterthwaite generalization used across cells. https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/welchsat.htm
- SciPy: the noncentral t parameterization and survival function (sf). https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.nct.html
Minimize new cost for a fixed pre-run comparison design.
One-cell searches enumerate legal integer counts and certify the returned analytic-model minimum. Suites use deterministic cost-aware local search. Existing baseline observations contribute uncertainty but no new cost.
Evaluate fixed counts using Gym’s Welch uncertainty and NCT power.
Normalized one-pass variances are treated as fixed design inputs. The noncentrality is (expected delta + margin) / standard error. The NCT distribution approximates the future Welch statistic, whose estimated degrees of freedom vary between experiments.
Evaluate actual Gym Welch decisions under independent normal pass scores.
Draw sufficient statistics: normal sample means and independent chi-square
sample variances. Each trial calls aggregate_welch with its estimated
variances. Power uses the expected delta; false positives use the negative
margin boundary. Wilson intervals quantify simulation sampling error only.
Historical variance-estimation uncertainty is not resampled.
Both arms’ sample means and variances are redrawn in hypothetical experiments, including existing-baseline mode. This is an unconditional repeated-experiment diagnostic, not a calculation conditional on the already collected baseline observations.