> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.eval.comparison.suite.schema

Manifest, configuration, and report contracts for benchmark-suite comparisons.

## Module Contents

### Classes

| Name                                                                                            | Description                                                                     |
| ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`SuiteAggregateResult`](#nemo_gym-eval-comparison-suite-schema-SuiteAggregateResult)           | -                                                                               |
| [`SuiteBenchmarkDefinition`](#nemo_gym-eval-comparison-suite-schema-SuiteBenchmarkDefinition)   | Metrics averaged within each repeat of one benchmark.                           |
| [`SuiteCellResult`](#nemo_gym-eval-comparison-suite-schema-SuiteCellResult)                     | -                                                                               |
| [`SuiteComparisonConfig`](#nemo_gym-eval-comparison-suite-schema-SuiteComparisonConfig)         | Compare baseline and candidate runs across a suite of models and benchmarks.    |
| [`SuiteComparisonResult`](#nemo_gym-eval-comparison-suite-schema-SuiteComparisonResult)         | Separate schema for suite_compare_report.json; run reports remain unchanged.    |
| [`SuiteCoverage`](#nemo_gym-eval-comparison-suite-schema-SuiteCoverage)                         | Recorded and planned task-coverage evidence for one selected agent.             |
| [`SuiteFailureCounts`](#nemo_gym-eval-comparison-suite-schema-SuiteFailureCounts)               | Saved selected failure outcomes for the entire rollout file.                    |
| [`SuiteFailureSummary`](#nemo_gym-eval-comparison-suite-schema-SuiteFailureSummary)             | Precomputed accounting, without rescanning or certifying the source artifacts.  |
| [`SuiteManifest`](#nemo_gym-eval-comparison-suite-schema-SuiteManifest)                         | Explicit benchmark definitions and a possibly sparse model-by-benchmark matrix. |
| [`SuiteMetricDefinition`](#nemo_gym-eval-comparison-suite-schema-SuiteMetricDefinition)         | A selected repeat metric and an optional complete theoretical scale override.   |
| [`SuiteMetricResult`](#nemo_gym-eval-comparison-suite-schema-SuiteMetricResult)                 | -                                                                               |
| [`SuiteRunPair`](#nemo_gym-eval-comparison-suite-schema-SuiteRunPair)                           | Nonempty source collections, accepting a single mapping as shorthand in YAML.   |
| [`SuiteRunSource`](#nemo_gym-eval-comparison-suite-schema-SuiteRunSource)                       | One run with companion artifacts discovered from its rollout path.              |
| [`SuiteRunSummary`](#nemo_gym-eval-comparison-suite-schema-SuiteRunSummary)                     | One source or pooled repeats, with full per-source provenance under `sources`.  |
| [`SuiteSampleSummary`](#nemo_gym-eval-comparison-suite-schema-SuiteSampleSummary)               | -                                                                               |
| [`SuiteStatistics`](#nemo_gym-eval-comparison-suite-schema-SuiteStatistics)                     | Completed numerical results; presentation only formats these fields.            |
| [`SuiteVarianceContribution`](#nemo_gym-eval-comparison-suite-schema-SuiteVarianceContribution) | -                                                                               |
| [`_SuiteModel`](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)                             | -                                                                               |

### Functions

| Name                                                                        | Description                                                                      |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| [`summary_sources`](#nemo_gym-eval-comparison-suite-schema-summary_sources) | Return constituent provenance, preserving a standalone source's report contract. |

### Data

[`CoverageStatus`](#nemo_gym-eval-comparison-suite-schema-CoverageStatus)

### API

```python
class nemo_gym.eval.comparison.suite.schema.SuiteAggregateResult()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

**`cell_ids`** `list[str]`

---

**`name`** `str`

---

**`statistics`** `SuiteStatistics | None`

---

**`weights`** `list[float]`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Metrics averaged within each repeat of one benchmark.

**`metrics`** `list[SuiteMetricDefinition] = Field(min_length=1)`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition.validate_unique_metrics() -> nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition
```

```python
class nemo_gym.eval.comparison.suite.schema.SuiteCellResult()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

**`baseline`** `SuiteRunSummary`

---

**`benchmark`** `str`

---

**`candidate`** `SuiteRunSummary`

---

**`eligible`** `bool`

---

**`exclusion_reason`** `str | None = None`

---

**`metrics`** `list[SuiteMetricResult]`

---

**`model`** `str`

---

**`model_weight`** `float = 0.0`

---

**`overall_weight`** `float = 0.0`

---

**`statistics`** `SuiteStatistics | None = None`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteComparisonConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config/config_types#nemo_gym-config-config_types-BaseNeMoGymCLIConfig)

Compare baseline and candidate runs across a suite of models and benchmarks.

Run paths in the suite YAML resolve relative to that file. Reports give
`delta = candidate - baseline` on a 0-100 normalized scale with unpaired Welch intervals.

**`confidence_level`** `float`

---

**`output_dirpath`** `str | None`

---

**`report_format`** `ReportFormat`

---

**`suite_fpath`** `str`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteComparisonResult()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Separate schema for suite_compare_report.json; run reports remain unchanged.

**`assumptions`** `list[str]`

---

**`cells`** `list[SuiteCellResult]`

---

**`command`** `str`

---

**`confidence_level`** `float`

---

**`generated_at`** `str`

---

**`nemo_gym_version`** `str`

---

**`overall`** `SuiteAggregateResult`

---

**`per_model`** `list[SuiteAggregateResult]`

---

**`schema_version`** `Literal['1'] = '1'`

---

**`suite_fpath`** `Path`

---

**`warnings`** `list[str] = Field(default_factory=list)`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteCoverage()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Recorded and planned task-coverage evidence for one selected agent.

**`notes`** `list[str] = Field(default_factory=list)`

---

**`planned_task_count`** `int | None = None`

---

**`recorded_task_count`** `int`

---

**`status`** `CoverageStatus`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteFailureCounts()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Saved selected failure outcomes for the entire rollout file.

**`by_class`** `dict[str, Annotated[int, Field(strict=True, ge=0)]]`

---

**`counted_as_zero`** `int | None = Field(default=None, strict=True, ge=0)`

---

**`final`** `int = Field(strict=True, ge=0)`

---

**`final_zeros`** `int | None = Field(default=None, strict=True, ge=0)`

---

**`not_final`** `int = Field(strict=True, ge=0)`

---

**`total`** `int = Field(strict=True, ge=0)`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteFailureCounts.validate_totals() -> nemo_gym.eval.comparison.suite.schema.SuiteFailureCounts
```

```python
class nemo_gym.eval.comparison.suite.schema.SuiteFailureSummary()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Precomputed accounting, without rescanning or certifying the source artifacts.

**`counts`** `SuiteFailureCounts | None = None`

---

**`scope`** `Literal['rollout_file'] = 'rollout_file'`

---

**`source`** `Path`

---

**`unavailable_reason`** `str | None = None`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteFailureSummary.validate_outcome() -> nemo_gym.eval.comparison.suite.schema.SuiteFailureSummary
```

```python
class nemo_gym.eval.comparison.suite.schema.SuiteManifest()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Explicit benchmark definitions and a possibly sparse model-by-benchmark matrix.

**`benchmarks`** `dict[str, SuiteBenchmarkDefinition] = Field(min_length=1)`

---

**`models`** `dict[str, dict[str, SuiteRunPair]] = Field(min_length=1)`

---

**`schema_version`** `Literal[1] = 1`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteManifest.validate_cells() -> nemo_gym.eval.comparison.suite.schema.SuiteManifest
```

```python
class nemo_gym.eval.comparison.suite.schema.SuiteMetricDefinition()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

A selected repeat metric and an optional complete theoretical scale override.

**`high`** `float | None = None`

---

**`higher_is_better`** `bool | None = None`

---

**`low`** `float | None = None`

---

**`name`** `str = Field(min_length=1)`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteMetricDefinition.validate_complete_override() -> nemo_gym.eval.comparison.suite.schema.SuiteMetricDefinition
```

```python
class nemo_gym.eval.comparison.suite.schema.SuiteMetricResult()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

**`baseline_raw`** `list[float] = Field(default_factory=list)`

---

**`baseline_scaled`** `list[float] = Field(default_factory=list)`

---

**`candidate_raw`** `list[float] = Field(default_factory=list)`

---

**`candidate_scaled`** `list[float] = Field(default_factory=list)`

---

**`definition_source`** `str`

---

**`high`** `float`

---

**`higher_is_better`** `bool`

---

**`low`** `float`

---

**`name`** `str`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteRunPair()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Nonempty source collections, accepting a single mapping as shorthand in YAML.

**`baseline`** `list[SuiteRunSource] = Field(min_length=1)`

---

**`candidate`** `list[SuiteRunSource] = Field(min_length=1)`

---

```python
nemo_gym.eval.comparison.suite.schema.SuiteRunPair.normalize_sources(
    value: object
) -> object
```

classmethod

Choose the source shape before validation so errors describe only that shape.

```python
class nemo_gym.eval.comparison.suite.schema.SuiteRunSource()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

One run with companion artifacts discovered from its rollout path.

**`agent`** `str | None = None`

---

**`environment_server`** `str | None = None`

---

**`rollouts`** `Path`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteRunSummary()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

One source or pooled repeats, with full per-source provenance under `sources`.

For pooled repeats, path and selector fields identify the first source for
compatibility. Local repeat identities and saved failure accounting live under `sources`.

**`agent`** `str`

---

**`aggregate_metrics`** `Path`

---

**`coverage`** `SuiteCoverage`

---

**`environment_server`** `str | None = None`

---

**`failures`** `SuiteFailureSummary | None = None`

---

**`materialized_inputs`** `Path | None = None`

---

**`repeat_count`** `int | None`

---

**`repeat_indices`** `list[int] = Field(default_factory=list)`

---

**`repeat_scores`** `list[float] = Field(default_factory=list)`

---

**`rollouts`** `Path`

---

**`sources`** `list[SuiteRunSummary] = Field(default_factory=list)`

---

**`summary`** `SuiteSampleSummary | None = None`

---

**`unscored_repeat_indices`** `list[int] = Field(default_factory=list)`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteSampleSummary()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

**`count`** `int`

---

**`mean`** `float`

---

**`sample_variance`** `float`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteStatistics()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

Completed numerical results; presentation only formats these fields.

**`baseline_mean`** `float`

---

**`candidate_mean`** `float`

---

**`ci_high`** `float | None`

---

**`ci_low`** `float | None`

---

**`confidence_level`** `float`

---

**`contributions`** `list[SuiteVarianceContribution]`

---

**`degrees_of_freedom`** `float | None`

---

**`delta`** `float`

---

**`interval_unavailable_reason`** `str | None = None`

---

**`standard_error`** `float`

---

**`variance`** `float`

---

```python
class nemo_gym.eval.comparison.suite.schema.SuiteVarianceContribution()
```

**Bases:** [\_SuiteModel](#nemo_gym-eval-comparison-suite-schema-_SuiteModel)

**`baseline`** `float`

---

**`candidate`** `float`

---

**`weight`** `float`

---

```python
class nemo_gym.eval.comparison.suite.schema._SuiteModel()
```

**Bases:** `BaseModel`

**`model_config`** `= ConfigDict(extra='forbid')`

---

```python
nemo_gym.eval.comparison.suite.schema.summary_sources(
    summary: nemo_gym.eval.comparison.suite.schema.SuiteRunSummary
) -> tuple[nemo_gym.eval.comparison.suite.schema.SuiteRunSummary, ...]
```

Return constituent provenance, preserving a standalone source's report contract.

```python
nemo_gym.eval.comparison.suite.schema.CoverageStatus = Literal['verified', 'unverified', 'incomplete']
```