nemo_gym.comparison.diff

Diffing two loaded runs: metric rows and per-task sample flips.

Pure computation — no filesystem access, no statistics. Confidence intervals are read verbatim from what the runs already recorded; for now nothing here estimates, tests, or judges.

Module Contents

Functions

Name	Description
`_candidate_metric_value`	-
`_cap_reward_lists`	Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so
`_flip_direction`	-
`_groups_by_task`	-
`_is_number`	-
`_looks_binary`	Whether every observed reward on both sides is exactly 0 or 1.
`_metric_value`	-
`_numeric`	-
`_ordered_metric_names`	Baseline order first (it is the reference), then anything only the candidates reported.
`_per_repeat_rewards`	-
`build_flip_summary`	Per-task movement between the two runs, joined on task index.
`build_metric_rows`	One row per metric reported by any side, key metrics flagged.
`compare_runs`	Build one agent’s comparison block: metric rows, flips, and anything worth flagging.
`is_comparable_metric`	Whether an `agent_metrics` key earns its own row in the all-metrics table.

Data

DISPERSION_PREFIXES

API

nemo_gym.comparison.diff._candidate_metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str,
    baseline_value: typing.Optional[float]
) -> typing.Optional[nemo_gym.comparison.schema.CandidateMetricValue]

nemo_gym.comparison.diff._cap_reward_lists(
    flips: typing.List[nemo_gym.comparison.schema.TaskFlip]
) -> typing.List[nemo_gym.comparison.schema.TaskFlip]

Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so compare_report.json doesn’t scale O(tasks x repeats) when most/all tasks move (continuous mode).

nemo_gym.comparison.diff._flip_direction(
    baseline_score: float,
    candidate_score: float
) -> typing.Optional[str]

nemo_gym.comparison.diff._groups_by_task(
    run: nemo_gym.comparison.loading.LoadedRun
) -> typing.Dict[int, typing.Dict[str, typing.Any]]

nemo_gym.comparison.diff._is_number(
    value: typing.Any
) -> bool

nemo_gym.comparison.diff._looks_binary(
    common: typing.Sequence[int],
    baseline_groups: typing.Dict[int, typing.Dict[str, typing.Any]],
    candidate_groups: typing.Dict[int, typing.Dict[str, typing.Any]]
) -> bool

Whether every observed reward on both sides is exactly 0 or 1.

Uses each task’s recorded min/max where available so a task whose repeats disagree (mean 0.5) is still recognised as binary. Falls back to mean only when a task has neither min nor max recorded.

nemo_gym.comparison.diff._metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str
) -> typing.Optional[nemo_gym.comparison.schema.MetricValue]

nemo_gym.comparison.diff._numeric(
    value: typing.Any
) -> typing.Optional[float]

nemo_gym.comparison.diff._ordered_metric_names(
    baseline: typing.Dict[str, typing.Any],
    candidates: typing.Sequence[typing.Dict[str, typing.Any]]
) -> typing.List[str]

Baseline order first (it is the reference), then anything only the candidates reported.

nemo_gym.comparison.diff._per_repeat_rewards(
    group: typing.Dict[str, typing.Any]
) -> typing.Optional[typing.List[float]]

nemo_gym.comparison.diff.build_flip_summary(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    candidate_index: int = 0
) -> nemo_gym.comparison.schema.FlipSummary

Per-task movement between the two runs, joined on task index.

*_aggregate_metrics.json carries no task identity, so tasks are matched by _ng_task_index alone — which assumes both runs used the same dataset, split, limit and ordering.

nemo_gym.comparison.diff.build_metric_rows(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> typing.List[nemo_gym.comparison.schema.MetricRow]

One row per metric reported by any side, key metrics flagged.

key_metrics can rename or synthesize names that never appear in agent_metrics (e.g. an ASR server’s corpus_wer@k=N -> wer), so rows are built from the union of both, taking the value from key_metrics only when agent_metrics doesn’t already carry that name.

nemo_gym.comparison.diff.compare_runs(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> nemo_gym.comparison.schema.AgentComparison

Build one agent’s comparison block: metric rows, flips, and anything worth flagging.

nemo_gym.comparison.diff.is_comparable_metric(
    name: str
) -> bool

Whether an agent_metrics key earns its own row in the all-metrics table.

nemo_gym.comparison.diff.DISPERSION_PREFIXES = (MEDIAN_PREFIX, STD_PREFIX, MIN_PREFIX, MAX_PREFIX, P25_PREFIX, P75_PREFIX, SEM_...

nemo_gym.comparison.diff.FLIP_FIELD = REWARD_KEY_NAME

nemo_gym.comparison.diff.PASS_THRESHOLD = 0.5

nemo_gym.comparison.diff.STAT_SUFFIXES = (STD_DEV_ACROSS_RUNS_SUFFIX, STD_ERR_ACROSS_RUNS_SUFFIX, AVG_SAMPLE_STD_DEV_SUFF...

nemo_gym.comparison.diff.TASK_MAX_KEY = f'{MAX_PREFIX}{FLIP_FIELD}'

nemo_gym.comparison.diff.TASK_MEAN_KEY = f'{MEAN_PREFIX}{FLIP_FIELD}'

nemo_gym.comparison.diff.TASK_MIN_KEY = f'{MIN_PREFIX}{FLIP_FIELD}'