> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.comparison.diff

Diffing two loaded runs: metric rows and per-task sample flips.

Pure computation -- no filesystem access, no statistics. Confidence intervals are read verbatim
from what the runs already recorded; for now nothing here estimates, tests, or judges.

## Module Contents

### Functions

| Name                                                                           | Description                                                                                   |
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| [`_candidate_metric_value`](#nemo_gym-comparison-diff-_candidate_metric_value) | -                                                                                             |
| [`_cap_reward_lists`](#nemo_gym-comparison-diff-_cap_reward_lists)             | Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so |
| [`_flip_direction`](#nemo_gym-comparison-diff-_flip_direction)                 | -                                                                                             |
| [`_groups_by_task`](#nemo_gym-comparison-diff-_groups_by_task)                 | -                                                                                             |
| [`_is_number`](#nemo_gym-comparison-diff-_is_number)                           | -                                                                                             |
| [`_looks_binary`](#nemo_gym-comparison-diff-_looks_binary)                     | Whether every observed reward on both sides is exactly 0 or 1.                                |
| [`_metric_value`](#nemo_gym-comparison-diff-_metric_value)                     | -                                                                                             |
| [`_numeric`](#nemo_gym-comparison-diff-_numeric)                               | -                                                                                             |
| [`_ordered_metric_names`](#nemo_gym-comparison-diff-_ordered_metric_names)     | Baseline order first (it is the reference), then anything only the candidates reported.       |
| [`_per_repeat_rewards`](#nemo_gym-comparison-diff-_per_repeat_rewards)         | -                                                                                             |
| [`build_flip_summary`](#nemo_gym-comparison-diff-build_flip_summary)           | Per-task movement between the two runs, joined on task index.                                 |
| [`build_metric_rows`](#nemo_gym-comparison-diff-build_metric_rows)             | One row per metric reported by any side, key metrics flagged.                                 |
| [`compare_runs`](#nemo_gym-comparison-diff-compare_runs)                       | Build one agent's comparison block: metric rows, flips, and anything worth flagging.          |
| [`is_comparable_metric`](#nemo_gym-comparison-diff-is_comparable_metric)       | Whether an `agent_metrics` key earns its own row in the all-metrics table.                    |

### Data

[`DISPERSION_PREFIXES`](#nemo_gym-comparison-diff-DISPERSION_PREFIXES)

[`FLIP_FIELD`](#nemo_gym-comparison-diff-FLIP_FIELD)

[`PASS_THRESHOLD`](#nemo_gym-comparison-diff-PASS_THRESHOLD)

[`STAT_SUFFIXES`](#nemo_gym-comparison-diff-STAT_SUFFIXES)

[`TASK_MAX_KEY`](#nemo_gym-comparison-diff-TASK_MAX_KEY)

[`TASK_MEAN_KEY`](#nemo_gym-comparison-diff-TASK_MEAN_KEY)

[`TASK_MIN_KEY`](#nemo_gym-comparison-diff-TASK_MIN_KEY)

### API

```python
nemo_gym.comparison.diff._candidate_metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str,
    baseline_value: typing.Optional[float]
) -> typing.Optional[nemo_gym.comparison.schema.CandidateMetricValue]
```

```python
nemo_gym.comparison.diff._cap_reward_lists(
    flips: typing.List[nemo_gym.comparison.schema.TaskFlip]
) -> typing.List[nemo_gym.comparison.schema.TaskFlip]
```

Null out per-repeat reward lists on flips beyond the per-direction cap the markdown shows, so
`compare_report.json` doesn't scale O(tasks x repeats) when most/all tasks move (continuous
mode).

```python
nemo_gym.comparison.diff._flip_direction(
    baseline_score: float,
    candidate_score: float
) -> typing.Optional[str]
```

```python
nemo_gym.comparison.diff._groups_by_task(
    run: nemo_gym.comparison.loading.LoadedRun
) -> typing.Dict[int, typing.Dict[str, typing.Any]]
```

```python
nemo_gym.comparison.diff._is_number(
    value: typing.Any
) -> bool
```

```python
nemo_gym.comparison.diff._looks_binary(
    common: typing.Sequence[int],
    baseline_groups: typing.Dict[int, typing.Dict[str, typing.Any]],
    candidate_groups: typing.Dict[int, typing.Dict[str, typing.Any]]
) -> bool
```

Whether every observed reward on both sides is exactly 0 or 1.

Uses each task's recorded min/max where available so a task whose repeats disagree (mean 0.5)
is still recognised as binary. Falls back to mean only when a task has neither min nor max
recorded.

```python
nemo_gym.comparison.diff._metric_value(
    metrics: typing.Dict[str, typing.Any],
    name: str
) -> typing.Optional[nemo_gym.comparison.schema.MetricValue]
```

```python
nemo_gym.comparison.diff._numeric(
    value: typing.Any
) -> typing.Optional[float]
```

```python
nemo_gym.comparison.diff._ordered_metric_names(
    baseline: typing.Dict[str, typing.Any],
    candidates: typing.Sequence[typing.Dict[str, typing.Any]]
) -> typing.List[str]
```

Baseline order first (it is the reference), then anything only the candidates reported.

```python
nemo_gym.comparison.diff._per_repeat_rewards(
    group: typing.Dict[str, typing.Any]
) -> typing.Optional[typing.List[float]]
```

```python
nemo_gym.comparison.diff.build_flip_summary(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidate: nemo_gym.comparison.loading.LoadedRun,
    candidate_index: int = 0
) -> nemo_gym.comparison.schema.FlipSummary
```

Per-task movement between the two runs, joined on task index.

`*_aggregate_metrics.json` carries no task identity, so tasks are matched by
`_ng_task_index` alone -- which assumes both runs used the same dataset, split, limit and
ordering.

```python
nemo_gym.comparison.diff.build_metric_rows(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> typing.List[nemo_gym.comparison.schema.MetricRow]
```

One row per metric reported by any side, key metrics flagged.

`key_metrics` can rename or synthesize names that never appear in `agent_metrics` (e.g. an ASR
server's `corpus_wer@k=N` -> `wer`), so rows are built from the union of both, taking the value
from `key_metrics` only when `agent_metrics` doesn't already carry that name.

```python
nemo_gym.comparison.diff.compare_runs(
    baseline: nemo_gym.comparison.loading.LoadedRun,
    candidates: typing.Sequence[nemo_gym.comparison.loading.LoadedRun]
) -> nemo_gym.comparison.schema.AgentComparison
```

Build one agent's comparison block: metric rows, flips, and anything worth flagging.

```python
nemo_gym.comparison.diff.is_comparable_metric(
    name: str
) -> bool
```

Whether an `agent_metrics` key earns its own row in the all-metrics table.

```python
nemo_gym.comparison.diff.DISPERSION_PREFIXES = (MEDIAN_PREFIX, STD_PREFIX, MIN_PREFIX, MAX_PREFIX, P25_PREFIX, P75_PREFIX, SEM_...
```

```python
nemo_gym.comparison.diff.FLIP_FIELD = REWARD_KEY_NAME
```

```python
nemo_gym.comparison.diff.PASS_THRESHOLD = 0.5
```

```python
nemo_gym.comparison.diff.STAT_SUFFIXES = (STD_DEV_ACROSS_RUNS_SUFFIX, STD_ERR_ACROSS_RUNS_SUFFIX, AVG_SAMPLE_STD_DEV_SUFF...
```

```python
nemo_gym.comparison.diff.TASK_MAX_KEY = f'{MAX_PREFIX}{FLIP_FIELD}'
```

```python
nemo_gym.comparison.diff.TASK_MEAN_KEY = f'{MEAN_PREFIX}{FLIP_FIELD}'
```

```python
nemo_gym.comparison.diff.TASK_MIN_KEY = f'{MIN_PREFIX}{FLIP_FIELD}'
```