nemo_gym.comparison.diff

Diffing two loaded runs: metric rows, difference confidence intervals, and per-task sample flips.

Module Contents

Functions

Name	Description
`_candidate_metric_value`	-
`_cap_reward_lists`	Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so
`_comparison_value`	-
`_flip_direction`	-
`_groups_by_task`	-
`_is_number`	-
`_looks_binary`	Whether every observed reward on both sides is exactly 0 or 1.
`_metric_value`	-
`_numeric`	-
`_ordered_metric_names`	Baseline order first (it is the reference), then anything only the candidates reported.
`_per_repeat_rewards`	-
`_repeat_metric_values`	Finite numeric values for one metric, with one observation per repeat.
`_warn_if_repeat_samples_differ`	Warn when repeat estimates cover incomplete or unequal task samples.
`_welch_delta_confidence_interval`	Two-sided 95% Welch interval for candidate minus baseline.
`build_flip_summary`	Per-task movement between the two runs, joined on task index.
`build_metric_rows`	One row per metric reported by any side, key metrics flagged.
`compare_runs`	Build one agent’s comparison block: metric rows, flips, and anything worth flagging.

Data

API

nemo_gym.comparison.diff._candidate_metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str,
    baseline_value: typing.Optional[float],
    delta_ci: typing.Tuple[typing.Optional[float], typing.Optional[float]]
) -> typing.Optional[nemo_gym.comparison.schema.CandidateMetricValue]

nemo_gym.comparison.diff._cap_reward_lists(
    flips: typing.List[nemo_gym.comparison.schema.TaskFlip]
) -> typing.List[nemo_gym.comparison.schema.TaskFlip]

Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so compare_report.json doesn’t scale O(tasks x repeats) when most/all tasks move (continuous mode).

nemo_gym.comparison.diff._comparison_value(
    metric: nemo_gym.comparison.schema.MetricValue
) -> float

nemo_gym.comparison.diff._flip_direction(
    baseline_score: float,
    candidate_score: float
) -> typing.Optional[str]

nemo_gym.comparison.diff._groups_by_task(
    run: nemo_gym.comparison.loading.LoadedRun
) -> typing.Dict[int, typing.Dict[str, typing.Any]]

nemo_gym.comparison.diff._is_number(
    value: typing.Any
) -> bool

nemo_gym.comparison.diff._looks_binary(
    common: typing.Sequence[int],
    baseline_groups: typing.Dict[int, typing.Dict[str, typing.Any]],
    candidate_groups: typing.Dict[int, typing.Dict[str, typing.Any]]
) -> bool

Whether every observed reward on both sides is exactly 0 or 1.

Uses each task’s recorded min/max where available so a task whose repeats disagree (mean 0.5) is still recognised as binary. Falls back to mean only when a task has neither min nor max recorded.

nemo_gym.comparison.diff._metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str
) -> typing.Optional[nemo_gym.comparison.schema.MetricValue]

nemo_gym.comparison.diff._numeric(
    value: typing.Any
) -> typing.Optional[float]

nemo_gym.comparison.diff._ordered_metric_names(
    baseline: typing.Dict[str, typing.Any],
    candidates: typing.Sequence[typing.Dict[str, typing.Any]]
) -> typing.List[str]

Baseline order first (it is the reference), then anything only the candidates reported.

nemo_gym.comparison.diff._per_repeat_rewards(
    group: typing.Dict[str, typing.Any]
) -> typing.Optional[typing.List[float]]

nemo_gym.comparison.diff._repeat_metric_values(
    run: nemo_gym.comparison.loading.LoadedRun,
    name: str
) -> typing.List[float]

Finite numeric values for one metric, with one observation per repeat.

nemo_gym.comparison.diff._warn_if_repeat_samples_differ(
    run: nemo_gym.comparison.loading.LoadedRun,
    label: str
) -> None

Warn when repeat estimates cover incomplete or unequal task samples.

nemo_gym.comparison.diff._welch_delta_confidence_interval(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    name: str
) -> typing.Tuple[typing.Optional[float], typing.Optional[float]]

Two-sided 95% Welch interval for candidate minus baseline.

nemo_gym.comparison.diff.build_flip_summary(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    candidate_index: int = 0
) -> nemo_gym.comparison.schema.FlipSummary

Per-task movement between the two runs, joined on task index.

*_aggregate_metrics.json carries no task identity, so tasks are matched by _ng_task_index alone — which assumes both runs used the same dataset, split, limit and ordering.

nemo_gym.comparison.diff.build_metric_rows(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> typing.List[nemo_gym.comparison.schema.MetricRow]

One row per metric reported by any side, key metrics flagged.

key_metrics can rename or synthesize names that never appear in agent_metrics (e.g. an ASR server’s corpus_wer@k=N -> wer), so rows are built from the union of both, taking the value from key_metrics only when agent_metrics doesn’t already carry that name.

nemo_gym.comparison.diff.compare_runs(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> nemo_gym.comparison.schema.AgentComparison

Build one agent’s comparison block: metric rows, flips, and anything worth flagging.

nemo_gym.comparison.diff.FLIP_FIELD = REWARD_KEY_NAME

nemo_gym.comparison.diff.PASS_THRESHOLD = 0.5

nemo_gym.comparison.diff.TASK_MAX_KEY = f'{Stat.MAX.prefix}{FLIP_FIELD}'

nemo_gym.comparison.diff.TASK_MEAN_KEY = f'{Stat.MEAN.prefix}{FLIP_FIELD}'

nemo_gym.comparison.diff.TASK_MIN_KEY = f'{Stat.MIN.prefix}{FLIP_FIELD}'