> This page is for version 0.6.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite

Normalize repeat scores and estimate fixed-suite unpaired Welch intervals.

Baseline and candidate repeats are independent samples. Benchmarks are a fixed
suite, not a sample of possible benchmarks. Multiple metrics within a benchmark
are averaged *within each repeat* before estimating sample variance, preserving
their covariance. The weighted aggregate uses independent variance contributions
and Welch–Satterthwaite effective degrees of freedom.

This module contains numerical inputs and outputs only; it neither loads Gym
artifacts nor decides which benchmarks are eligible for comparison.

## Module Contents

### Classes

| Name                                                                                                           | Description                                                                      |
| -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| [`BenchmarkSamples`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-BenchmarkSamples)         | Independent baseline and candidate repeat summaries for one benchmark cell.      |
| [`MetricScale`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-MetricScale)                   | Theoretical metric bounds and direction for conversion to a 0–100 score.         |
| [`SampleSummary`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-SampleSummary)               | Mean and unbiased sample variance of independent repeat scores.                  |
| [`VarianceContribution`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-VarianceContribution) | One cell's weighted mean-variance contributions, each equal to w²s²/n.           |
| [`WelchResult`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-WelchResult)                   | Weighted scores, candidate-minus-baseline estimate, and unpaired Welch interval. |

### Functions

| Name                                                                                                                                                   | Description                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| [`aggregate_welch`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-aggregate_welch)                                                   | Estimate a weighted candidate-minus-baseline delta with Welch–Satterthwaite uncertainty. |
| [`normalize_and_average_metrics_per_repeat`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-normalize_and_average_metrics_per_repeat) | Average equally weighted normalized metrics within each repeat.                          |
| [`normalize_score`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-normalize_score)                                                   | Scale a finite metric to percentage points, reversing lower-is-better metrics.           |
| [`summarize_sample`](#nemo_gym-eval-comparison-statistics-unpaired_welch_fixed_suite-summarize_sample)                                                 | Summarize finite repeat scores using sample variance with divisor n − 1.                 |

### API

```python
class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.BenchmarkSamples(
    baseline: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary,
    candidate: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary
)
```

Dataclass

Independent baseline and candidate repeat summaries for one benchmark cell.

**`baseline`** `SampleSummary`

---

**`candidate`** `SampleSummary`

---

```python
class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale(
    low: float,
    high: float,
    higher_is_better: bool = True
)
```

Dataclass

Theoretical metric bounds and direction for conversion to a 0–100 score.

**`high`** `float`

---

**`higher_is_better`** `bool = True`

---

**`low`** `float`

---

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale.__post_init__() -> None
```

```python
class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary(
    count: int,
    mean: float,
    sample_variance: float
)
```

Dataclass

Mean and unbiased sample variance of independent repeat scores.

**`count`** `int`

---

**`mean`** `float`

---

**`sample_variance`** `float`

---

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary.__post_init__() -> None
```

```python
class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.VarianceContribution(
    weight: float,
    baseline: float,
    candidate: float
)
```

Dataclass

One cell's weighted mean-variance contributions, each equal to w²s²/n.

**`baseline`** `float`

---

**`candidate`** `float`

---

**`weight`** `float`

---

```python
class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.WelchResult(
    baseline_mean: float,
    candidate_mean: float,
    delta: float,
    variance: float,
    standard_error: float,
    degrees_of_freedom: float | None,
    ci_low: float | None,
    ci_high: float | None,
    confidence_level: float,
    contributions: tuple[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.VarianceContribution, ...]
)
```

Dataclass

Weighted scores, candidate-minus-baseline estimate, and unpaired Welch interval.

**`baseline_mean`** `float`

---

**`candidate_mean`** `float`

---

**`ci_high`** `float | None`

---

**`ci_low`** `float | None`

---

**`confidence_level`** `float`

---

**`contributions`** `tuple[VarianceContribution, ...]`

---

**`degrees_of_freedom`** `float | None`

---

**`delta`** `float`

---

**`standard_error`** `float`

---

**`variance`** `float`

---

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.aggregate_welch(
    comparisons: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.BenchmarkSamples],
    weights: collections.abc.Sequence[float] | None = None,
    confidence_level: float = 0.9
) -> nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.WelchResult
```

Estimate a weighted candidate-minus-baseline delta with Welch–Satterthwaite uncertainty.

For each cell, `a = w²s_C²/n_C` and `b = w²s_B²/n_B`. Total
variance is `V = sum(a + b)`, and effective degrees of freedom are
`V² / sum(a²/(n_C−1) + b²/(n_B−1))`. The two-sided interval uses
Student-t quantile `(1 + confidence_level) / 2`.

Weights default to equal values and must sum to one when provided. Constant
cells retain their weights. If all contributions are zero, the estimate and
zero standard error are returned with no interval or degrees of freedom.

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.normalize_and_average_metrics_per_repeat(
    repeat_values: collections.abc.Sequence[collections.abc.Sequence[float]],
    scales: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale]
) -> tuple[float, ...]
```

Average equally weighted normalized metrics within each repeat.

Rows represent repeats; columns correspond to `scales`. Averaging before
summarizing keeps covariance between metrics from the same repeat. A cell's
benchmark weight is independent of its number of selected metrics.

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.normalize_score(
    value: float,
    scale: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale
) -> float
```

Scale a finite metric to percentage points, reversing lower-is-better metrics.

Bounds describe the metric, not observed extrema. Scores outside these
theoretical bounds are rejected; confidence intervals remain unclipped.

```python
nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.summarize_sample(
    values: collections.abc.Sequence[float]
) -> nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary
```

Summarize finite repeat scores using sample variance with divisor n − 1.