> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.eval.comparison.suite.loading

Read suite manifests and recorded artifacts without reconstructing repeat metrics.

## Module Contents

### Classes

| Name                                                                       | Description                                                            |
| -------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| [`LoadedSuiteRun`](#nemo_gym-eval-comparison-suite-loading-LoadedSuiteRun) | One selected aggregate entry and its optional planned task identities. |

### Functions

| Name                                                                                               | Description                                                                                 |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`_index`](#nemo_gym-eval-comparison-suite-loading-_index)                                         | -                                                                                           |
| [`_object_rows`](#nemo_gym-eval-comparison-suite-loading-_object_rows)                             | -                                                                                           |
| [`_read_planned_tasks`](#nemo_gym-eval-comparison-suite-loading-_read_planned_tasks)               | Select the plan using the same server identity as aggregation, with unstamped-row fallback. |
| [`_repeat_count`](#nemo_gym-eval-comparison-suite-loading-_repeat_count)                           | Retain verified zero/single-repeat counts when no repeat estimates are stored.              |
| [`_select_entry`](#nemo_gym-eval-comparison-suite-loading-_select_entry)                           | Resolve optional selectors independently for one source's aggregate file.                   |
| [`load_suite_manifest`](#nemo_gym-eval-comparison-suite-loading-load_suite_manifest)               | Read and validate the explicit suite input contract.                                        |
| [`load_suite_run`](#nemo_gym-eval-comparison-suite-loading-load_suite_run)                         | Select one aggregate entry by agent and/or server, retaining its recorded repeat rows.      |
| [`resolve_metric_definitions`](#nemo_gym-eval-comparison-suite-loading-resolve_metric_definitions) | Validate explicit metric scales, using supplied catalog metadata for mean/reward.           |

### API

```python
class nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun(
    rollouts: pathlib.Path,
    aggregate_metrics: pathlib.Path,
    materialized_inputs: pathlib.Path | None,
    agent: str,
    environment_server: str | None,
    repeat_rows: tuple[typing.Mapping[str, object], ...],
    groups: tuple[typing.Mapping[str, object], ...],
    agent_metrics: typing.Mapping[str, object],
    repeat_count: int,
    planned_tasks_by_repeat: typing.Mapping[int, frozenset[int]] | None,
    unscored_repeat_indices: tuple[int, ...] = ()
)
```

Dataclass

One selected aggregate entry and its optional planned task identities.

**`agent`** `str`

---

**`agent_metrics`** `Mapping[str, object]`

---

**`aggregate_metrics`** `Path`

---

**`environment_server`** `str | None`

---

**`groups`** `tuple[Mapping[str, object], ...]`

---

**`materialized_inputs`** `Path | None`

---

**`planned_tasks_by_repeat`** `Mapping[int, frozenset[int]] | None`

---

**`repeat_count`** `int`

---

**`repeat_rows`** `tuple[Mapping[str, object], ...]`

---

**`rollouts`** `Path`

---

**`unscored_repeat_indices`** `tuple[int, ...] = ()`

---

```python
nemo_gym.eval.comparison.suite.loading._index(
    value: object,
    context: str
) -> int
```

```python
nemo_gym.eval.comparison.suite.loading._object_rows(
    value: object,
    field: str,
    path: pathlib.Path
) -> tuple[typing.Mapping[str, object], ...]
```

```python
nemo_gym.eval.comparison.suite.loading._read_planned_tasks(
    path: pathlib.Path,
    agent: str,
    environment_server: str | None
) -> dict[int, frozenset[int]]
```

Select the plan using the same server identity as aggregation, with unstamped-row fallback.

```python
nemo_gym.eval.comparison.suite.loading._repeat_count(
    rows: tuple[typing.Mapping[str, object], ...],
    groups: tuple[typing.Mapping[str, object], ...],
    agent_metrics: typing.Mapping[str, object],
    path: pathlib.Path,
    agent: str
) -> int
```

Retain verified zero/single-repeat counts when no repeat estimates are stored.

```python
nemo_gym.eval.comparison.suite.loading._select_entry(
    entries: typing.Mapping[str, dict[str, object]],
    source: nemo_gym.eval.comparison.suite.schema.SuiteRunSource,
    path: pathlib.Path
) -> tuple[str, dict[str, object]]
```

Resolve optional selectors independently for one source's aggregate file.

```python
nemo_gym.eval.comparison.suite.loading.load_suite_manifest(
    path: pathlib.Path
) -> nemo_gym.eval.comparison.suite.schema.SuiteManifest
```

Read and validate the explicit suite input contract.

```python
nemo_gym.eval.comparison.suite.loading.load_suite_run(
    source: nemo_gym.eval.comparison.suite.schema.SuiteRunSource,
    directory: pathlib.Path
) -> nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun
```

Select one aggregate entry by agent and/or server, retaining its recorded repeat rows.

```python
nemo_gym.eval.comparison.suite.loading.resolve_metric_definitions(
    benchmark: str,
    definition: nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition,
    reward_metric_resolver: collections.abc.Callable[[str], nemo_gym.eval.comparison.suite.schema.SuiteMetricResult] | None = None
) -> list[nemo_gym.eval.comparison.suite.schema.SuiteMetricResult]
```

Validate explicit metric scales, using supplied catalog metadata for mean/reward.