nemo_gym.eval.comparison.suite.processing

Validate coverage and assemble statistical inputs from stored suite artifacts.

Module Contents

Functions

Name	Description
`_collection_summary`	-
`_metric_values`	-
`_run_summary`	-
`_samples`	-
`_select_metric_repeats`	Omit token-only rows lacking selected estimates; score rows need not have sample_count.
`_statistics`	-
`aggregate_suite_cells`	Aggregate eligible cells equally, retaining declared exclusions in the cell report.
`compare_suite_cell`	Compare concatenated independent repeat samples, retaining source-level evidence.

API

nemo_gym.eval.comparison.suite.processing._collection_summary(
    runs: tuple[nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun, ...],
    coverages: list[nemo_gym.eval.comparison.suite.schema.SuiteCoverage],
    scores: tuple[float, ...],
    collection_notes: list[str]
) -> nemo_gym.eval.comparison.suite.schema.SuiteRunSummary

nemo_gym.eval.comparison.suite.processing._metric_values(
    run: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun,
    metrics: list[nemo_gym.eval.comparison.suite.schema.SuiteMetricResult]
) -> list[list[float]]

nemo_gym.eval.comparison.suite.processing._run_summary(
    run: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun,
    coverage: nemo_gym.eval.comparison.suite.schema.SuiteCoverage,
    scores: tuple[float, ...]
) -> nemo_gym.eval.comparison.suite.schema.SuiteRunSummary

nemo_gym.eval.comparison.suite.processing._samples(
    cell: nemo_gym.eval.comparison.suite.schema.SuiteCellResult
) -> nemo_gym.eval.comparison.statistics.unpaired_welch_fixed_suite.BenchmarkSamples

nemo_gym.eval.comparison.suite.processing._select_metric_repeats(
    run: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun,
    metrics: list[nemo_gym.eval.comparison.suite.schema.SuiteMetricResult]
) -> nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun

Omit token-only rows lacking selected estimates; score rows need not have sample_count.

nemo_gym.eval.comparison.suite.processing._statistics(
    cells: list[nemo_gym.eval.comparison.suite.schema.SuiteCellResult],
    confidence_level: float
) -> nemo_gym.eval.comparison.suite.schema.SuiteStatistics

nemo_gym.eval.comparison.suite.processing.aggregate_suite_cells(
    name: str,
    cells: list[nemo_gym.eval.comparison.suite.schema.SuiteCellResult],
    confidence_level: float
) -> nemo_gym.eval.comparison.suite.schema.SuiteAggregateResult

Aggregate eligible cells equally, retaining declared exclusions in the cell report.

nemo_gym.eval.comparison.suite.processing.compare_suite_cell(
    model: str,
    benchmark: str,
    baseline: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun | tuple[nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun, ...],
    candidate: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun | tuple[nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun, ...],
    metrics: list[nemo_gym.eval.comparison.suite.schema.SuiteMetricResult],
    confidence_level: float
) -> nemo_gym.eval.comparison.suite.schema.SuiteCellResult

Compare concatenated independent repeat samples, retaining source-level evidence.