nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite

Normalize repeat scores and estimate fixed-suite unpaired Welch intervals.

Baseline and candidate repeats are independent samples. Benchmarks are a fixed suite, not a sample of possible benchmarks. Multiple metrics within a benchmark are averaged within each repeat before estimating sample variance, preserving their covariance. The weighted aggregate uses independent variance contributions and Welch–Satterthwaite effective degrees of freedom.

This module contains numerical inputs and outputs only; it neither loads Gym artifacts nor decides which benchmarks are eligible for comparison.

Module Contents

Classes

Name	Description
`BenchmarkSamples`	Independent baseline and candidate repeat summaries for one benchmark cell.
`MetricScale`	Theoretical metric bounds and direction for conversion to a 0–100 score.
`SampleSummary`	Mean and unbiased sample variance of independent repeat scores.
`VarianceContribution`	One cell’s weighted mean-variance contributions, each equal to w²s²/n.
`WelchResult`	Weighted scores, candidate-minus-baseline estimate, and unpaired Welch interval.

Functions

Name	Description
`aggregate_welch`	Estimate a weighted candidate-minus-baseline delta with Welch–Satterthwaite uncertainty.
`normalize_and_average_metrics_per_repeat`	Average equally weighted normalized metrics within each repeat.
`normalize_score`	Scale a finite metric to percentage points, reversing lower-is-better metrics.
`summarize_sample`	Summarize finite repeat scores using sample variance with divisor n − 1.

API

class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.BenchmarkSamples(
    baseline: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary,
    candidate: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary
)

Dataclass

Independent baseline and candidate repeat summaries for one benchmark cell.

baseline

SampleSummary

candidate

SampleSummary

class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale(
    low: float,
    high: float,
    higher_is_better: bool = True
)

Dataclass

Theoretical metric bounds and direction for conversion to a 0–100 score.

high

float

higher_is_better

bool = True

low

float

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale.__post_init__() -> None

class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary(
    count: int,
    mean: float,
    sample_variance: float
)

Dataclass

Mean and unbiased sample variance of independent repeat scores.

count

int

mean

float

sample_variance

float

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary.__post_init__() -> None

class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.VarianceContribution(
    weight: float,
    baseline: float,
    candidate: float
)

Dataclass

One cell’s weighted mean-variance contributions, each equal to w²s²/n.

baseline

float

candidate

float

weight

float

class nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.WelchResult(
    baseline_mean: float,
    candidate_mean: float,
    delta: float,
    variance: float,
    standard_error: float,
    degrees_of_freedom: float | None,
    ci_low: float | None,
    ci_high: float | None,
    confidence_level: float,
    contributions: tuple[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.VarianceContribution, ...]
)

Dataclass

Weighted scores, candidate-minus-baseline estimate, and unpaired Welch interval.

baseline_mean

float

candidate_mean

float

ci_high

float | None

ci_low

float | None

confidence_level

float

contributions

tuple[VarianceContribution, ...]

degrees_of_freedom

float | None

delta

float

standard_error

float

variance

float

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.aggregate_welch(
    comparisons: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.BenchmarkSamples],
    weights: collections.abc.Sequence[float] | None = None,
    confidence_level: float = 0.9
) -> nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.WelchResult

Estimate a weighted candidate-minus-baseline delta with Welch–Satterthwaite uncertainty.

For each cell, a = w²s_C²/n_C and b = w²s_B²/n_B. Total variance is V = sum(a + b), and effective degrees of freedom are V² / sum(a²/(n_C−1) + b²/(n_B−1)). The two-sided interval uses Student-t quantile (1 + confidence_level) / 2.

Weights default to equal values and must sum to one when provided. Constant cells retain their weights. If all contributions are zero, the estimate and zero standard error are returned with no interval or degrees of freedom.

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.normalize_and_average_metrics_per_repeat(
    repeat_values: collections.abc.Sequence[collections.abc.Sequence[float]],
    scales: collections.abc.Sequence[nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale]
) -> tuple[float, ...]

Average equally weighted normalized metrics within each repeat.

Rows represent repeats; columns correspond to scales. Averaging before summarizing keeps covariance between metrics from the same repeat. A cell’s benchmark weight is independent of its number of selected metrics.

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.normalize_score(
    value: float,
    scale: nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.MetricScale
) -> float

Scale a finite metric to percentage points, reversing lower-is-better metrics.

Bounds describe the metric, not observed extrema. Scores outside these theoretical bounds are rejected; confidence intervals remain unclipped.

nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.summarize_sample(
    values: collections.abc.Sequence[float]
) -> nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.SampleSummary

Summarize finite repeat scores using sample variance with divisor n − 1.