> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.comparison.diff

Diffing two loaded runs: metric rows, difference confidence intervals, and per-task sample flips.

## Module Contents

### Functions

| Name                                                                                             | Description                                                                                   |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| [`_candidate_metric_value`](#nemo_gym-comparison-diff-_candidate_metric_value)                   | -                                                                                             |
| [`_cap_reward_lists`](#nemo_gym-comparison-diff-_cap_reward_lists)                               | Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so |
| [`_comparison_value`](#nemo_gym-comparison-diff-_comparison_value)                               | -                                                                                             |
| [`_flip_direction`](#nemo_gym-comparison-diff-_flip_direction)                                   | -                                                                                             |
| [`_groups_by_task`](#nemo_gym-comparison-diff-_groups_by_task)                                   | -                                                                                             |
| [`_is_number`](#nemo_gym-comparison-diff-_is_number)                                             | -                                                                                             |
| [`_looks_binary`](#nemo_gym-comparison-diff-_looks_binary)                                       | Whether every observed reward on both sides is exactly 0 or 1.                                |
| [`_metric_value`](#nemo_gym-comparison-diff-_metric_value)                                       | -                                                                                             |
| [`_numeric`](#nemo_gym-comparison-diff-_numeric)                                                 | -                                                                                             |
| [`_ordered_metric_names`](#nemo_gym-comparison-diff-_ordered_metric_names)                       | Baseline order first (it is the reference), then anything only the candidates reported.       |
| [`_per_repeat_rewards`](#nemo_gym-comparison-diff-_per_repeat_rewards)                           | -                                                                                             |
| [`_repeat_metric_values`](#nemo_gym-comparison-diff-_repeat_metric_values)                       | Finite numeric values for one metric, with one observation per repeat.                        |
| [`_warn_if_repeat_samples_differ`](#nemo_gym-comparison-diff-_warn_if_repeat_samples_differ)     | Warn when repeat estimates cover incomplete or unequal task samples.                          |
| [`_welch_delta_confidence_interval`](#nemo_gym-comparison-diff-_welch_delta_confidence_interval) | Two-sided 95% Welch interval for candidate minus baseline.                                    |
| [`build_flip_summary`](#nemo_gym-comparison-diff-build_flip_summary)                             | Per-task movement between the two runs, joined on task index.                                 |
| [`build_metric_rows`](#nemo_gym-comparison-diff-build_metric_rows)                               | One row per metric reported by any side, key metrics flagged.                                 |
| [`compare_runs`](#nemo_gym-comparison-diff-compare_runs)                                         | Build one agent's comparison block: metric rows, flips, and anything worth flagging.          |

### Data

[`FLIP_FIELD`](#nemo_gym-comparison-diff-FLIP_FIELD)

[`PASS_THRESHOLD`](#nemo_gym-comparison-diff-PASS_THRESHOLD)

[`TASK_MAX_KEY`](#nemo_gym-comparison-diff-TASK_MAX_KEY)

[`TASK_MEAN_KEY`](#nemo_gym-comparison-diff-TASK_MEAN_KEY)

[`TASK_MIN_KEY`](#nemo_gym-comparison-diff-TASK_MIN_KEY)

### API

```python
nemo_gym.comparison.diff._candidate_metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str,
    baseline_value: typing.Optional[float],
    delta_ci: typing.Tuple[typing.Optional[float], typing.Optional[float]]
) -> typing.Optional[nemo_gym.comparison.schema.CandidateMetricValue]
```

```python
nemo_gym.comparison.diff._cap_reward_lists(
    flips: typing.List[nemo_gym.comparison.schema.TaskFlip]
) -> typing.List[nemo_gym.comparison.schema.TaskFlip]
```

Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so
`compare_report.json` doesn't scale O(tasks x repeats) when most/all tasks move (continuous
mode).

```python
nemo_gym.comparison.diff._comparison_value(
    metric: nemo_gym.comparison.schema.MetricValue
) -> float
```

```python
nemo_gym.comparison.diff._flip_direction(
    baseline_score: float,
    candidate_score: float
) -> typing.Optional[str]
```

```python
nemo_gym.comparison.diff._groups_by_task(
    run: nemo_gym.comparison.loading.LoadedRun
) -> typing.Dict[int, typing.Dict[str, typing.Any]]
```

```python
nemo_gym.comparison.diff._is_number(
    value: typing.Any
) -> bool
```

```python
nemo_gym.comparison.diff._looks_binary(
    common: typing.Sequence[int],
    baseline_groups: typing.Dict[int, typing.Dict[str, typing.Any]],
    candidate_groups: typing.Dict[int, typing.Dict[str, typing.Any]]
) -> bool
```

Whether every observed reward on both sides is exactly 0 or 1.

Uses each task's recorded min/max where available so a task whose repeats disagree (mean 0.5)
is still recognised as binary. Falls back to mean only when a task has neither min nor max
recorded.

```python
nemo_gym.comparison.diff._metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str
) -> typing.Optional[nemo_gym.comparison.schema.MetricValue]
```

```python
nemo_gym.comparison.diff._numeric(
    value: typing.Any
) -> typing.Optional[float]
```

```python
nemo_gym.comparison.diff._ordered_metric_names(
    baseline: typing.Dict[str, typing.Any],
    candidates: typing.Sequence[typing.Dict[str, typing.Any]]
) -> typing.List[str]
```

Baseline order first (it is the reference), then anything only the candidates reported.

```python
nemo_gym.comparison.diff._per_repeat_rewards(
    group: typing.Dict[str, typing.Any]
) -> typing.Optional[typing.List[float]]
```

```python
nemo_gym.comparison.diff._repeat_metric_values(
    run: nemo_gym.comparison.loading.LoadedRun,
    name: str
) -> typing.List[float]
```

Finite numeric values for one metric, with one observation per repeat.

```python
nemo_gym.comparison.diff._warn_if_repeat_samples_differ(
    run: nemo_gym.comparison.loading.LoadedRun,
    label: str
) -> None
```

Warn when repeat estimates cover incomplete or unequal task samples.

```python
nemo_gym.comparison.diff._welch_delta_confidence_interval(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    name: str
) -> typing.Tuple[typing.Optional[float], typing.Optional[float]]
```

Two-sided 95% Welch interval for candidate minus baseline.

```python
nemo_gym.comparison.diff.build_flip_summary(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    candidate_index: int = 0
) -> nemo_gym.comparison.schema.FlipSummary
```

Per-task movement between the two runs, joined on task index.

`*_aggregate_metrics.json` carries no task identity, so tasks are matched by
`_ng_task_index` alone -- which assumes both runs used the same dataset, split, limit and
ordering.

```python
nemo_gym.comparison.diff.build_metric_rows(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> typing.List[nemo_gym.comparison.schema.MetricRow]
```

One row per metric reported by any side, key metrics flagged.

`key_metrics` can rename or synthesize names that never appear in `agent_metrics` (e.g. an ASR
server's `corpus_wer@k=N` -> `wer`), so rows are built from the union of both, taking the value
from `key_metrics` only when `agent_metrics` doesn't already carry that name.

```python
nemo_gym.comparison.diff.compare_runs(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> nemo_gym.comparison.schema.AgentComparison
```

Build one agent's comparison block: metric rows, flips, and anything worth flagging.

```python
nemo_gym.comparison.diff.FLIP_FIELD = REWARD_KEY_NAME
```

```python
nemo_gym.comparison.diff.PASS_THRESHOLD = 0.5
```

```python
nemo_gym.comparison.diff.TASK_MAX_KEY = f'{Stat.MAX.prefix}{FLIP_FIELD}'
```

```python
nemo_gym.comparison.diff.TASK_MEAN_KEY = f'{Stat.MEAN.prefix}{FLIP_FIELD}'
```

```python
nemo_gym.comparison.diff.TASK_MIN_KEY = f'{Stat.MIN.prefix}{FLIP_FIELD}'
```