> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.comparison.statistics.repeat_planning

Plan complete benchmark passes with prospective unpaired Welch power.

Inputs are one-pass variances in normalized score points squared. The
noncentral-t power approximation holds those variances fixed. Simulations
separately evaluate the actual Welch decision with newly estimated variances;
they never change an allocation. This module performs no artifact I/O.

## Module Contents

### Classes

| Name                                                                                            | Description                                                                 |
| ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [`PlanningCell`](#nemo_gym-eval-comparison-statistics-repeat_planning-PlanningCell)             | One model/benchmark's pass variances, legal increment and per-pass cost.    |
| [`PlanningParameters`](#nemo_gym-eval-comparison-statistics-repeat_planning-PlanningParameters) | Fixed, pre-run non-inferiority design settings.                             |
| [`PlanningStatistics`](#nemo_gym-eval-comparison-statistics-repeat_planning-PlanningStatistics) | Prospective Welch uncertainty and approximate noncentral-t power.           |
| [`RepeatAllocation`](#nemo_gym-eval-comparison-statistics-repeat_planning-RepeatAllocation)     | Counts in input-cell order, with baseline paid once and candidates K times. |
| [`SimulationRate`](#nemo_gym-eval-comparison-statistics-repeat_planning-SimulationRate)         | A simulated event frequency and its pointwise 95% Wilson interval.          |
| [`SimulationResult`](#nemo_gym-eval-comparison-statistics-repeat_planning-SimulationResult)     | Actual-Welch diagnostics at fixed supplied population variances.            |

### Functions

| Name                                                                                        | Description                                                               |
| ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| [`_counts`](#nemo_gym-eval-comparison-statistics-repeat_planning-_counts)                   | -                                                                         |
| [`_inputs`](#nemo_gym-eval-comparison-statistics-repeat_planning-_inputs)                   | -                                                                         |
| [`_integer`](#nemo_gym-eval-comparison-statistics-repeat_planning-_integer)                 | -                                                                         |
| [`_limits`](#nemo_gym-eval-comparison-statistics-repeat_planning-_limits)                   | Return scalar bounds for newly collected complete-repeat counts.          |
| [`_normal_start`](#nemo_gym-eval-comparison-statistics-repeat_planning-_normal_start)       | -                                                                         |
| [`_power`](#nemo_gym-eval-comparison-statistics-repeat_planning-_power)                     | Evaluate one-sided non-inferiority power for one design or a batch.       |
| [`_rate`](#nemo_gym-eval-comparison-statistics-repeat_planning-_rate)                       | -                                                                         |
| [`_real`](#nemo_gym-eval-comparison-statistics-repeat_planning-_real)                       | -                                                                         |
| [`_scalar_counts`](#nemo_gym-eval-comparison-statistics-repeat_planning-_scalar_counts)     | -                                                                         |
| [`_suite_counts`](#nemo_gym-eval-comparison-statistics-repeat_planning-_suite_counts)       | -                                                                         |
| [`plan_repeats`](#nemo_gym-eval-comparison-statistics-repeat_planning-plan_repeats)         | Minimize new cost for a fixed pre-run comparison design.                  |
| [`power_for_counts`](#nemo_gym-eval-comparison-statistics-repeat_planning-power_for_counts) | Evaluate fixed counts using Gym's Welch uncertainty and NCT power.        |
| [`simulate_plan`](#nemo_gym-eval-comparison-statistics-repeat_planning-simulate_plan)       | Evaluate actual Gym Welch decisions under independent normal pass scores. |

### API

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell(
    model: str,
    benchmark: str,
    baseline_variance: float,
    candidate_variance: float,
    repeat_step: int = 1,
    repeat_cost: float = 1.0,
    existing_baseline_repeats: int | None = None
)
```

Dataclass

One model/benchmark's pass variances, legal increment and per-pass cost.

Baseline and candidate costs share the same arbitrary positive cost unit.
Existing baseline passes are already available and are a sunk cost.

**`baseline_variance`** `float`

---

**`benchmark`** `str`

---

**`candidate_variance`** `float`

---

**`existing_baseline_repeats`** `int | None = None`

---

**`model`** `str`

---

**`repeat_cost`** `float = 1.0`

---

**`repeat_step`** `int = 1`

---

```python
nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell.__post_init__() -> None
```

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters(
    baseline_mode: typing.Literal['optimize', 'existing'] = 'optimize',
    candidate_experiments: int = 1,
    margin_points: float = 1.0,
    target_power: float = 0.8,
    alpha: float = 0.05,
    expected_delta_points: float = 0.0,
    max_repeats: int = 20000
)
```

Dataclass

Fixed, pre-run non-inferiority design settings.

`expected_delta_points` is candidate minus baseline. Success means a
one-sided lower bound exceeds `-margin_points`. `alpha=.05` uses the
lower endpoint of a two-sided 90% interval. Candidate experiments share
one baseline and each receive the returned candidate counts; power remains
marginal for one comparison. `max_repeats` bounds newly planned counts.

**`alpha`** `float = 0.05`

---

**`baseline_mode`** `Literal['optimize', 'existing'] = 'optimize'`

---

**`candidate_experiments`** `int = 1`

---

**`expected_delta_points`** `float = 0.0`

---

**`margin_points`** `float = 1.0`

---

**`max_repeats`** `int = 20000`

---

**`target_power`** `float = 0.8`

---

```python
nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters.__post_init__() -> None
```

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.PlanningStatistics(
    variance: float,
    standard_error: float,
    degrees_of_freedom: float,
    critical_value: float,
    half_width: float,
    power: float
)
```

Dataclass

Prospective Welch uncertainty and approximate noncentral-t power.

**`critical_value`** `float`

---

**`degrees_of_freedom`** `float`

---

**`half_width`** `float`

---

**`power`** `float`

---

**`standard_error`** `float`

---

**`variance`** `float`

---

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.RepeatAllocation(
    baseline_counts: tuple[int, ...],
    candidate_counts: tuple[int, ...],
    statistics: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningStatistics,
    additional_baseline_counts: tuple[int, ...],
    total_additional_passes: int,
    total_cost: float,
    method: str
)
```

Dataclass

Counts in input-cell order, with baseline paid once and candidates K times.

**`additional_baseline_counts`** `tuple[int, ...]`

---

**`baseline_counts`** `tuple[int, ...]`

---

**`candidate_counts`** `tuple[int, ...]`

---

**`method`** `str`

---

**`statistics`** `PlanningStatistics`

---

**`total_additional_passes`** `int`

---

**`total_cost`** `float`

---

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate(
    successes: int,
    trials: int,
    estimate: float,
    ci_low: float,
    ci_high: float
)
```

Dataclass

A simulated event frequency and its pointwise 95% Wilson interval.

**`ci_high`** `float`

---

**`ci_low`** `float`

---

**`estimate`** `float`

---

**`successes`** `int`

---

**`trials`** `int`

---

```python
class nemo_gym.eval.comparison.statistics.repeat_planning.SimulationResult(
    trials: int,
    seed: int,
    power: nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate,
    false_positive: nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate,
    coverage: nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate
)
```

Dataclass

Actual-Welch diagnostics at fixed supplied population variances.

**`coverage`** `SimulationRate`

---

**`false_positive`** `SimulationRate`

---

**`power`** `SimulationRate`

---

**`seed`** `int`

---

**`trials`** `int`

---

```python
nemo_gym.eval.comparison.statistics.repeat_planning._counts(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    baseline_counts: collections.abc.Sequence[int],
    candidate_counts: collections.abc.Sequence[int],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters
) -> tuple[tuple[int, ...], tuple[int, ...]]
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._inputs(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    weights: collections.abc.Sequence[float] | None
) -> tuple[float, ...]
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._integer(
    value: object,
    name: str,
    minimum: int = 1
) -> None
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._limits(
    cell: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell,
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters
) -> tuple[int, int]
```

Return scalar bounds for newly collected complete-repeat counts.

```python
nemo_gym.eval.comparison.statistics.repeat_planning._normal_start(
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    variance_coefficients: numpy.typing.NDArray[numpy.float64],
    minimum_counts: numpy.typing.NDArray[numpy.float64],
    maximum_counts: numpy.typing.NDArray[numpy.float64],
    repeat_steps: numpy.typing.NDArray[numpy.float64],
    repeat_costs: numpy.typing.NDArray[numpy.float64]
) -> numpy.typing.NDArray[numpy.float64]
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._power(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    repeat_counts: numpy.typing.NDArray[numpy.float64],
    weights: tuple[float, ...],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters
) -> numpy.typing.NDArray[numpy.float64]
```

Evaluate one-sided non-inferiority power for one design or a batch.

For cell i and arm a (baseline or candidate), let v_ia be its forecast
one-pass variance, n_ia its complete-repeat count, and w_i its cell weight.
The code evaluates the following equations, summing over both indices:

```python
q_ia = w_i**2 * v_ia / n_ia
V = sum(q_ia)
f_ia = q_ia / V
df = 1 / sum(f_ia**2 / (n_ia - 1))
noncentrality = (expected_delta_points + margin_points) / sqrt(V)
critical = t.isf(alpha, df)  # t quantile at 1 - alpha
power = nct.sf(critical, df, noncentrality)  # 1 - F_nct(critical; df, noncentrality)
```

Delta means candidate minus baseline; the null boundary is -margin.
The df expression equals V**2 / sum(q_ia**2 / (n_ia - 1)), using fractions
for numerical stability. This is a prospective NCT approximation with
fixed forecast variances; actual Welch decisions re-estimate them and df.

Formula references:

* NCSS, pp. 522-2 and 522-4: the null boundary and one-benchmark NCT formula.
  [https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Two-Sample_T-Tests_for_Non-Inferiority_Allowing_Unequal_Variance.pdf#page=4](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Two-Sample_T-Tests_for_Non-Inferiority_Allowing_Unequal_Variance.pdf#page=4)
* NIST: the weighted Welch-Satterthwaite generalization used across cells.
  [https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/welchsat.htm](https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/welchsat.htm)
* SciPy: the noncentral t parameterization and survival function (sf).
  [https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.nct.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.nct.html)

```python
nemo_gym.eval.comparison.statistics.repeat_planning._rate(
    successes: int,
    trials: int
) -> nemo_gym.eval.comparison.statistics.repeat_planning.SimulationRate
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._real(
    value: object,
    name: str,
    positive: bool = False
) -> None
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._scalar_counts(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    weights: tuple[float, ...]
) -> tuple[numpy.typing.NDArray[numpy.float64], str]
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning._suite_counts(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    weights: tuple[float, ...]
) -> tuple[numpy.typing.NDArray[numpy.float64], str]
```

```python
nemo_gym.eval.comparison.statistics.repeat_planning.plan_repeats(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    weights: collections.abc.Sequence[float] | None = None
) -> nemo_gym.eval.comparison.statistics.repeat_planning.RepeatAllocation
```

Minimize new cost for a fixed pre-run comparison design.

One-cell searches enumerate legal integer counts and certify the returned
analytic-model minimum. Suites use deterministic cost-aware local search.
Existing baseline observations contribute uncertainty but no new cost.

```python
nemo_gym.eval.comparison.statistics.repeat_planning.power_for_counts(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    baseline_counts: collections.abc.Sequence[int],
    candidate_counts: collections.abc.Sequence[int],
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    weights: collections.abc.Sequence[float] | None = None
) -> nemo_gym.eval.comparison.statistics.repeat_planning.PlanningStatistics
```

Evaluate fixed counts using Gym's Welch uncertainty and NCT power.

Normalized one-pass variances are treated as fixed design inputs. The
noncentrality is (expected delta + margin) / standard error. The NCT
distribution approximates the future Welch statistic, whose estimated
degrees of freedom vary between experiments.

```python
nemo_gym.eval.comparison.statistics.repeat_planning.simulate_plan(
    cells: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.repeat_planning.PlanningCell],
    allocation: nemo_gym.eval.comparison.statistics.repeat_planning.RepeatAllocation,
    parameters: nemo_gym.eval.comparison.statistics.repeat_planning.PlanningParameters,
    weights: collections.abc.Sequence[float] | None = None,
    trials: int = 10000,
    seed: int = 20261109
) -> nemo_gym.eval.comparison.statistics.repeat_planning.SimulationResult
```

Evaluate actual Gym Welch decisions under independent normal pass scores.

Draw sufficient statistics: normal sample means and independent chi-square
sample variances. Each trial calls `aggregate_welch` with its estimated
variances. Power uses the expected delta; false positives use the negative
margin boundary. Wilson intervals quantify simulation sampling error only.
Historical variance-estimation uncertainty is not resampled.

Both arms' sample means and variances are redrawn in hypothetical
experiments, including existing-baseline mode. This is an unconditional
repeated-experiment diagnostic, not a calculation conditional on the
already collected baseline observations.