Skip to navigation

nemo_gym.eval.comparison.statistics.repeat_planning

View as MarkdownOpen in Claude

Plan complete benchmark passes with prospective unpaired Welch power.

Inputs are one-pass variances in normalized score points squared. The noncentral-t power approximation holds those variances fixed. Simulations separately evaluate the actual Welch decision with newly estimated variances; they never change an allocation. This module performs no artifact I/O.

Module Contents

Classes

NameDescription
PlanningCellOne model/benchmark’s pass variances, legal increment and per-pass cost.
PlanningParametersFixed, pre-run non-inferiority design settings.
PlanningStatisticsProspective Welch uncertainty and approximate noncentral-t power.
RepeatAllocationCounts in input-cell order, with baseline paid once and candidates K times.
SimulationRateA simulated event frequency and its pointwise 95% Wilson interval.
SimulationResultActual-Welch diagnostics at fixed supplied population variances.

Functions

NameDescription
_counts-
_inputs-
_integer-
_limitsReturn scalar bounds for newly collected complete-repeat counts.
_normal_start-
_powerEvaluate one-sided non-inferiority power for one design or a batch.
_rate-
_real-
_scalar_counts-
_suite_counts-
plan_repeatsMinimize new cost for a fixed pre-run comparison design.
power_for_countsEvaluate fixed counts using Gym’s Welch uncertainty and NCT power.
simulate_planEvaluate actual Gym Welch decisions under independent normal pass scores.

API

class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell(
model: str,
benchmark: str,
baseline_variance: float,
candidate_variance: float,
repeat_step: int = 1,
repeat_cost: float = 1.0,
existing_baseline_repeats: int | None = None
)
Dataclass

One model/benchmark’s pass variances, legal increment and per-pass cost.

Baseline and candidate costs share the same arbitrary positive cost unit. Existing baseline passes are already available and are a sunk cost.

baseline_variance
float
benchmark
str
candidate_variance
float
existing_baseline_repeats
int | None = None
model
str
repeat_cost
float = 1.0
repeat_step
int = 1
nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell.__post_init__() -> None
class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters(
baseline_mode: typing.Literal['optimize', 'existing'] = 'optimize',
candidate_experiments: int = 1,
margin_points: float = 1.0,
target_power: float = 0.8,
alpha: float = 0.05,
expected_delta_points: float = 0.0,
max_repeats: int = 20000
)
Dataclass

Fixed, pre-run non-inferiority design settings.

expected_delta_points is candidate minus baseline. Success means a one-sided lower bound exceeds -margin_points. alpha=.05 uses the lower endpoint of a two-sided 90% interval. Candidate experiments share one baseline and each receive the returned candidate counts; power remains marginal for one comparison. max_repeats bounds newly planned counts.

alpha
float = 0.05
baseline_mode
Literal['optimize', 'existing'] = 'optimize'
candidate_experiments
int = 1
expected_delta_points
float = 0.0
margin_points
float = 1.0
max_repeats
int = 20000
target_power
float = 0.8
nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters.__post_init__() -> None
class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningStatistics(
variance: float,
standard_error: float,
degrees_of_freedom: float,
critical_value: float,
half_width: float,
power: float
)
Dataclass

Prospective Welch uncertainty and approximate noncentral-t power.

critical_value
float
degrees_of_freedom
float
half_width
float
power
float
standard_error
float
variance
float
class nemo_gym.eval.comparison.statistics.repeat_planning.RepeatAllocation(
baseline_counts: tuple[int, ...],
candidate_counts: tuple[int, ...],
additional_baseline_counts: tuple[int, ...],
total_additional_passes: int,
total_cost: float,
method: str
)
Dataclass

Counts in input-cell order, with baseline paid once and candidates K times.

additional_baseline_counts
tuple[int, ...]
baseline_counts
tuple[int, ...]
candidate_counts
tuple[int, ...]
method
str
statistics
PlanningStatistics
total_additional_passes
int
total_cost
float
class nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate(
successes: int,
trials: int,
estimate: float,
ci_low: float,
ci_high: float
)
Dataclass

A simulated event frequency and its pointwise 95% Wilson interval.

ci_high
float
ci_low
float
estimate
float
successes
int
trials
int
class nemo_gym.eval.comparison.statistics.repeat_planning.SimulationResult(
trials: int,
seed: int,
)
Dataclass

Actual-Welch diagnostics at fixed supplied population variances.

coverage
SimulationRate
false_positive
SimulationRate
power
SimulationRate
seed
int
trials
int
nemo_gym.eval.comparison.statistics.repeat_planning._counts(
baseline_counts: collections.abc.Sequence[int],
candidate_counts: collections.abc.Sequence[int],
) -> tuple[tuple[int, ...], tuple[int, ...]]
nemo_gym.eval.comparison.statistics.repeat_planning._inputs(
weights: collections.abc.Sequence[float] | None
) -> tuple[float, ...]
nemo_gym.eval.comparison.statistics.repeat_planning._integer(
value: object,
name: str,
minimum: int = 1
) -> None
nemo_gym.eval.comparison.statistics.repeat_planning._limits(
) -> tuple[int, int]

Return scalar bounds for newly collected complete-repeat counts.

nemo_gym.eval.comparison.statistics.repeat_planning._normal_start(
variance_coefficients: numpy.typing.NDArray[numpy.float64],
minimum_counts: numpy.typing.NDArray[numpy.float64],
maximum_counts: numpy.typing.NDArray[numpy.float64],
repeat_steps: numpy.typing.NDArray[numpy.float64],
repeat_costs: numpy.typing.NDArray[numpy.float64]
) -> numpy.typing.NDArray[numpy.float64]
nemo_gym.eval.comparison.statistics.repeat_planning._power(
repeat_counts: numpy.typing.NDArray[numpy.float64],
weights: tuple[float, ...],
) -> numpy.typing.NDArray[numpy.float64]

Evaluate one-sided non-inferiority power for one design or a batch.

For cell i and arm a (baseline or candidate), let v_ia be its forecast one-pass variance, n_ia its complete-repeat count, and w_i its cell weight. The code evaluates the following equations, summing over both indices:

q_ia = w_i**2 * v_ia / n_ia
V = sum(q_ia)
f_ia = q_ia / V
df = 1 / sum(f_ia**2 / (n_ia - 1))
noncentrality = (expected_delta_points + margin_points) / sqrt(V)
critical = t.isf(alpha, df) # t quantile at 1 - alpha
power = nct.sf(critical, df, noncentrality) # 1 - F_nct(critical; df, noncentrality)

Delta means candidate minus baseline; the null boundary is -margin. The df expression equals V2 / sum(q_ia2 / (n_ia - 1)), using fractions for numerical stability. This is a prospective NCT approximation with fixed forecast variances; actual Welch decisions re-estimate them and df.

Formula references:

nemo_gym.eval.comparison.statistics.repeat_planning._rate(
successes: int,
trials: int
nemo_gym.eval.comparison.statistics.repeat_planning._real(
value: object,
name: str,
positive: bool = False
) -> None
nemo_gym.eval.comparison.statistics.repeat_planning._scalar_counts(
weights: tuple[float, ...]
) -> tuple[numpy.typing.NDArray[numpy.float64], str]
nemo_gym.eval.comparison.statistics.repeat_planning._suite_counts(
weights: tuple[float, ...]
) -> tuple[numpy.typing.NDArray[numpy.float64], str]
nemo_gym.eval.comparison.statistics.repeat_planning.plan_repeats(
weights: collections.abc.Sequence[float] | None = None

Minimize new cost for a fixed pre-run comparison design.

One-cell searches enumerate legal integer counts and certify the returned analytic-model minimum. Suites use deterministic cost-aware local search. Existing baseline observations contribute uncertainty but no new cost.

nemo_gym.eval.comparison.statistics.repeat_planning.power_for_counts(
baseline_counts: collections.abc.Sequence[int],
candidate_counts: collections.abc.Sequence[int],
weights: collections.abc.Sequence[float] | None = None

Evaluate fixed counts using Gym’s Welch uncertainty and NCT power.

Normalized one-pass variances are treated as fixed design inputs. The noncentrality is (expected delta + margin) / standard error. The NCT distribution approximates the future Welch statistic, whose estimated degrees of freedom vary between experiments.

nemo_gym.eval.comparison.statistics.repeat_planning.simulate_plan(
weights: collections.abc.Sequence[float] | None = None,
trials: int = 10000,
seed: int = 20261109

Evaluate actual Gym Welch decisions under independent normal pass scores.

Draw sufficient statistics: normal sample means and independent chi-square sample variances. Each trial calls aggregate_welch with its estimated variances. Power uses the expected delta; false positives use the negative margin boundary. Wilson intervals quantify simulation sampling error only. Historical variance-estimation uncertainty is not resampled.

Both arms’ sample means and variances are redrawn in hypothetical experiments, including existing-baseline mode. This is an unconditional repeated-experiment diagnostic, not a calculation conditional on the already collected baseline observations.