nemo_gym.eval.comparison.pooling

Pool independent repeat observations without averaging run-level estimates.

Module Contents

Functions

Name	Description
`_finite_number`	-
`_pooled_groups`	Recompute per-task rewards from observations; never average task summaries.
`_pooled_metric_samples`	Require complete quality samples and retain explicitly available token accounting.
`_quality_rows`	Keep scored metric estimates while recognizing accounting-only producer rows.
`_validated_rows`	Validate the observation structure before collecting any metric samples.
`pool_loaded_runs`	Concatenate complete repeat samples and recompute point estimates and 95% intervals.

API

nemo_gym.eval.comparison.pooling._finite_number(
    value: object
) -> typing.TypeGuard[int | float]

nemo_gym.eval.comparison.pooling._pooled_groups(
    runs: typing.Sequence[nemo_gym.eval.comparison.loading.LoadedRun]
) -> typing.Tuple[typing.List[typing.Dict[str, typing.Any]], typing.Optional[str]]

Recompute per-task rewards from observations; never average task summaries.

nemo_gym.eval.comparison.pooling._pooled_metric_samples(
    name: str,
    rows_by_source: typing.Sequence[typing.List[typing.Dict[str, typing.Any]]],
    source_labels: typing.Sequence[str]
) -> typing.Tuple[typing.List[float], typing.List[str]]

Require complete quality samples and retain explicitly available token accounting.

nemo_gym.eval.comparison.pooling._quality_rows(
    run: nemo_gym.eval.comparison.loading.LoadedRun,
    rows: typing.List[typing.Dict[str, typing.Any]],
    label: str
) -> typing.List[typing.Dict[str, typing.Any]]

Keep scored metric estimates while recognizing accounting-only producer rows.

nemo_gym.eval.comparison.pooling._validated_rows(
    run: nemo_gym.eval.comparison.loading.LoadedRun,
    label: str
) -> typing.List[typing.Dict[str, typing.Any]]

Validate the observation structure before collecting any metric samples.

nemo_gym.eval.comparison.pooling.pool_loaded_runs(
    runs: typing.Sequence[nemo_gym.eval.comparison.loading.LoadedRun],
    source_labels: typing.Sequence[str]
) -> nemo_gym.eval.comparison.loading.LoadedRun

Concatenate complete repeat samples and recompute point estimates and 95% intervals.

Each repeat has equal weight, even when sources have different repeat counts. Metrics available only as whole-run aggregates (including pass@k summaries) are omitted. A metric other than token accounting must be finite in every scored row of every source. Token metrics use the observations that record them, including accounting-only rows from unscored repeats, without inventing a zero quality or token estimate.