> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.cli.eval

## Module Contents

### Classes

| Name                                                                  | Description                                                                                            |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| [`PlanBenchmarkConfig`](#nemo_gym-cli-eval-PlanBenchmarkConfig)       | Prepare one benchmark and write its plan: every task once, with a stable id and a digest of its input. |
| [`PrepareBenchmarkConfig`](#nemo_gym-cli-eval-PrepareBenchmarkConfig) | Prepare benchmark data by running the benchmark's prepare.py script.                                   |

### Functions

| Name                                                                                              | Description                                                                                                 |
| ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [`_datasets_to_prepare`](#nemo_gym-cli-eval-_datasets_to_prepare)                                 | Every dataset the resolved config prepares: benchmark datasets and other datasets with a prepare\_script.   |
| [`_gym_relative_path`](#nemo_gym-cli-eval-_gym_relative_path)                                     | A config path relative to the Gym checkout when it lies inside it, so plans match across machines.          |
| [`_inspect_benchmark`](#nemo_gym-cli-eval-_inspect_benchmark)                                     | Render the `gym list benchmarks <name>` inspect view for one benchmark.                               |
| [`_install_prepare_dependencies`](#nemo_gym-cli-eval-_install_prepare_dependencies)               | Install what a benchmark's prepare script imports, before importing it.                                     |
| [`_multiprocess_benchmark_prepare_fn`](#nemo_gym-cli-eval-_multiprocess_benchmark_prepare_fn)     | -                                                                                                           |
| [`_prepare_datasets`](#nemo_gym-cli-eval-_prepare_datasets)                                       | Run each dataset's prepare script, after validating all of them.                                            |
| [`_validate_prepared_split_file_exists`](#nemo_gym-cli-eval-_validate_prepared_split_file_exists) | Explicit check (not an assert: user-facing, and must survive `python -O`).                                  |
| [`_validate_split_datasets_declared`](#nemo_gym-cli-eval-_validate_split_datasets_declared)       | Fail fast when no config declares a dataset of the requested split's type.                                  |
| [`aggregate_rollouts`](#nemo_gym-cli-eval-aggregate_rollouts)                                     | -                                                                                                           |
| [`collect_rollouts`](#nemo_gym-cli-eval-collect_rollouts)                                         | -                                                                                                           |
| [`compare`](#nemo_gym-cli-eval-compare)                                                           | -                                                                                                           |
| [`e2e_rollout_collection`](#nemo_gym-cli-eval-e2e_rollout_collection)                             | -                                                                                                           |
| [`export_rollouts_as_atif`](#nemo_gym-cli-eval-export_rollouts_as_atif)                           | -                                                                                                           |
| [`health_check_rollouts`](#nemo_gym-cli-eval-health_check_rollouts)                               | Run rollout quality verification for an existing run directory.                                             |
| [`list_benchmarks`](#nemo_gym-cli-eval-list_benchmarks)                                           | List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered |
| [`plan_benchmark`](#nemo_gym-cli-eval-plan_benchmark)                                             | CLI command: write a benchmark's evaluation plan.                                                           |
| [`prepare_benchmark`](#nemo_gym-cli-eval-prepare_benchmark)                                       | CLI command: prepare benchmark data.                                                                        |
| [`reverify_rollouts`](#nemo_gym-cli-eval-reverify_rollouts)                                       | -                                                                                                           |
| [`reward_profile`](#nemo_gym-cli-eval-reward_profile)                                             | -                                                                                                           |

### Data

[`logger`](#nemo_gym-cli-eval-logger)

### API

```python
class nemo_gym.cli.eval.PlanBenchmarkConfig()
```

**Bases:** [PrepareBenchmarkConfig](#nemo_gym-cli-eval-PrepareBenchmarkConfig)

Prepare one benchmark and write its plan: every task once, with a stable id and a digest of its input.

The plan file's format is versioned and printed by `--schema`.
Benchmarks whose rows have no `task_id`/`problem_id`/`instance_id`, and benchmarks with a prompt template,
are refused.

Examples:

```python
gym eval plan --benchmark terminal_bench_2_1/terminus_2 --out plan.json
gym eval plan --schema
```

**`plan_output_fpath`** `Path | None`

---

**`plan_task_ids`** `List[str] | None`

---

**`print_plan_schema`** `bool`

---

```python
class nemo_gym.cli.eval.PrepareBenchmarkConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

Prepare benchmark data by running the benchmark's prepare.py script.

The benchmark is identified from a config\_paths entry pointing to a
benchmarks/\*/config.yaml file.

With `use_cached_prepared_benchmarks=true`, an existing prepared file is reused. A prepare.py whose output
depends on its own code (for example, settings written into each row) can define
`is_prepared_data_current(fpath: Path) -> bool`; when it returns False, the cached file is prepared again.

Examples:

```python
gym eval prepare --benchmark aime24
```

**`num_prepare_benchmark_processes`** `int`

---

**`prepare_script_args`** `Dict[str, Any]`

---

**`use_cached_prepared_benchmarks`** `bool`

---

```python
nemo_gym.cli.eval._datasets_to_prepare(
    global_config_dict: omegaconf.DictConfig
) -> typing.List[nemo_gym.benchmarks.BenchmarkConfig]
```

Every dataset the resolved config prepares: benchmark datasets and other datasets with a prepare\_script.

```python
nemo_gym.cli.eval._gym_relative_path(
    path: str
) -> str
```

A config path relative to the Gym checkout when it lies inside it, so plans match across machines.

```python
nemo_gym.cli.eval._inspect_benchmark(
    name: str,
    benchmarks: dict,
    global_config_dict
) -> None
```

Render the `gym list benchmarks <name>` inspect view for one benchmark.

```python
nemo_gym.cli.eval._install_prepare_dependencies(
    benchmark_config: nemo_gym.benchmarks.BenchmarkConfig
) -> None
```

Install what a benchmark's prepare script imports, before importing it.

Gym cannot depend on every benchmark's data-prep requirements, so a benchmark
needing something extra had to shell out to pip from inside the prepare script
itself. Declaring it on the dataset puts it in the config instead.

```python
nemo_gym.cli.eval._multiprocess_benchmark_prepare_fn(
    args
)
```

```python
nemo_gym.cli.eval._prepare_datasets(
    datasets: collections.abc.Sequence[nemo_gym.benchmarks.BenchmarkConfig],
    prepare_benchmark_config: nemo_gym.cli.eval.PrepareBenchmarkConfig
) -> None
```

Run each dataset's prepare script, after validating all of them.

```python
nemo_gym.cli.eval._validate_prepared_split_file_exists(
    input_jsonl_fpath: pathlib.Path,
    split: str,
    output_dirpath: pathlib.Path
) -> None
```

Explicit check (not an assert: user-facing, and must survive `python -O`).

```python
nemo_gym.cli.eval._validate_split_datasets_declared(
    split: str,
    server_instance_configs: collections.abc.Sequence[nemo_gym.config_types.ServerInstanceConfig]
) -> None
```

Fail fast when no config declares a dataset of the requested split's type.

Data preparation silently produces nothing for such a split, so without this check the run
walks the entire preparation sequence (including its success banners) and only dies later
trying to read the collated split file.

```python
nemo_gym.cli.eval.aggregate_rollouts()
```

```python
nemo_gym.cli.eval.collect_rollouts()
```

```python
nemo_gym.cli.eval.compare() -> None
```

```python
nemo_gym.cli.eval.e2e_rollout_collection()
```

```python
nemo_gym.cli.eval.export_rollouts_as_atif() -> None
```

```python
nemo_gym.cli.eval.health_check_rollouts(
    run_dir: str | pathlib.Path,
    rollout_file: str | pathlib.Path | None = None,
    workers: int | None = None,
    ignored_checks: collections.abc.Sequence[str] = (),
    json_output: bool = False
)
```

Run rollout quality verification for an existing run directory.

```python
nemo_gym.cli.eval.list_benchmarks() -> None
```

List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered
by a `query` (the `gym search` entry point).

A benchmark is a specific kind of environment, so it shares `gym list environments`' columns (name,
domain, description) and reads them through the same `read_config_metadata` helper. `--search-dir`
adds extra roots to scan on top of the cwd and built-ins.

```python
nemo_gym.cli.eval.plan_benchmark() -> None
```

CLI command: write a benchmark's evaluation plan.

```python
nemo_gym.cli.eval.prepare_benchmark() -> None
```

CLI command: prepare benchmark data.

```python
nemo_gym.cli.eval.reverify_rollouts()
```

```python
nemo_gym.cli.eval.reward_profile()
```

```python
nemo_gym.cli.eval.logger = logging.getLogger(__name__)
```