> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.cli.eval

## Module Contents

### Classes

| Name                                                                  | Description                                                          |
| --------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`PrepareBenchmarkConfig`](#nemo_gym-cli-eval-PrepareBenchmarkConfig) | Prepare benchmark data by running the benchmark's prepare.py script. |

### Functions

| Name                                                                                              | Description                                                                                                 |
| ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [`_inspect_benchmark`](#nemo_gym-cli-eval-_inspect_benchmark)                                     | Render the `gym list benchmarks <name>` inspect view for one benchmark.                               |
| [`_install_prepare_dependencies`](#nemo_gym-cli-eval-_install_prepare_dependencies)               | Install what a benchmark's prepare script imports, before importing it.                                     |
| [`_multiprocess_benchmark_prepare_fn`](#nemo_gym-cli-eval-_multiprocess_benchmark_prepare_fn)     | -                                                                                                           |
| [`_validate_prepared_split_file_exists`](#nemo_gym-cli-eval-_validate_prepared_split_file_exists) | Explicit check (not an assert: user-facing, and must survive `python -O`).                                  |
| [`_validate_split_datasets_declared`](#nemo_gym-cli-eval-_validate_split_datasets_declared)       | Fail fast when no config declares a dataset of the requested split's type.                                  |
| [`aggregate_rollouts`](#nemo_gym-cli-eval-aggregate_rollouts)                                     | -                                                                                                           |
| [`collect_rollouts`](#nemo_gym-cli-eval-collect_rollouts)                                         | -                                                                                                           |
| [`compare`](#nemo_gym-cli-eval-compare)                                                           | -                                                                                                           |
| [`e2e_rollout_collection`](#nemo_gym-cli-eval-e2e_rollout_collection)                             | -                                                                                                           |
| [`export_rollouts_as_atif`](#nemo_gym-cli-eval-export_rollouts_as_atif)                           | -                                                                                                           |
| [`health_check_rollouts`](#nemo_gym-cli-eval-health_check_rollouts)                               | Run rollout quality verification for an existing run directory.                                             |
| [`list_benchmarks`](#nemo_gym-cli-eval-list_benchmarks)                                           | List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered |
| [`prepare_benchmark`](#nemo_gym-cli-eval-prepare_benchmark)                                       | CLI command: prepare benchmark data.                                                                        |
| [`reverify_rollouts`](#nemo_gym-cli-eval-reverify_rollouts)                                       | -                                                                                                           |
| [`reward_profile`](#nemo_gym-cli-eval-reward_profile)                                             | -                                                                                                           |

### Data

[`logger`](#nemo_gym-cli-eval-logger)

### API

```python
class nemo_gym.cli.eval.PrepareBenchmarkConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

Prepare benchmark data by running the benchmark's prepare.py script.

The benchmark is identified from a config\_paths entry pointing to a
benchmarks/\*/config.yaml file.

Examples:

```python
gym eval prepare --benchmark aime24
```

**`num_prepare_benchmark_processes`** `int`

---

**`prepare_script_args`** `Dict[str, Any]`

---

**`use_cached_prepared_benchmarks`** `bool`

---

```python
nemo_gym.cli.eval._inspect_benchmark(
    name: str,
    benchmarks: dict,
    global_config_dict
) -> None
```

Render the `gym list benchmarks <name>` inspect view for one benchmark.

```python
nemo_gym.cli.eval._install_prepare_dependencies(
    benchmark_config: nemo_gym.benchmarks.BenchmarkConfig
) -> None
```

Install what a benchmark's prepare script imports, before importing it.

Gym cannot depend on every benchmark's data-prep requirements, so a benchmark
needing something extra had to shell out to pip from inside the prepare script
itself. Declaring it on the dataset puts it in the config instead.

```python
nemo_gym.cli.eval._multiprocess_benchmark_prepare_fn(
    args
)
```

```python
nemo_gym.cli.eval._validate_prepared_split_file_exists(
    input_jsonl_fpath: pathlib.Path,
    split: str,
    output_dirpath: pathlib.Path
) -> None
```

Explicit check (not an assert: user-facing, and must survive `python -O`).

```python
nemo_gym.cli.eval._validate_split_datasets_declared(
    split: str,
    server_instance_configs: collections.abc.Sequence[nemo_gym.config_types.ServerInstanceConfig]
) -> None
```

Fail fast when no config declares a dataset of the requested split's type.

Data preparation silently produces nothing for such a split, so without this check the run
walks the entire preparation sequence (including its success banners) and only dies later
trying to read the collated split file.

```python
nemo_gym.cli.eval.aggregate_rollouts()
```

```python
nemo_gym.cli.eval.collect_rollouts()
```

```python
nemo_gym.cli.eval.compare() -> None
```

```python
nemo_gym.cli.eval.e2e_rollout_collection()
```

```python
nemo_gym.cli.eval.export_rollouts_as_atif() -> None
```

```python
nemo_gym.cli.eval.health_check_rollouts(
    run_dir: str | pathlib.Path,
    rollout_file: str | pathlib.Path | None = None,
    workers: int | None = None,
    ignored_checks: collections.abc.Sequence[str] = (),
    json_output: bool = False
)
```

Run rollout quality verification for an existing run directory.

```python
nemo_gym.cli.eval.list_benchmarks() -> None
```

List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered
by a `query` (the `gym search` entry point).

A benchmark is a specific kind of environment, so it shares `gym list environments`' columns (name,
domain, description) and reads them through the same `read_config_metadata` helper. `--search-dir`
adds extra roots to scan on top of the cwd and built-ins.

```python
nemo_gym.cli.eval.prepare_benchmark() -> None
```

CLI command: prepare benchmark data.

```python
nemo_gym.cli.eval.reverify_rollouts()
```

```python
nemo_gym.cli.eval.reward_profile()
```

```python
nemo_gym.cli.eval.logger = logging.getLogger(__name__)
```