> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.cli.eval

## Module Contents

### Classes

| Name                                                                  | Description                                                          |
| --------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`PrepareBenchmarkConfig`](#nemo_gym-cli-eval-PrepareBenchmarkConfig) | Prepare benchmark data by running the benchmark's prepare.py script. |

### Functions

| Name                                                                                          | Description                                                                                              |
| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [`_benchmark_extras`](#nemo_gym-cli-eval-_benchmark_extras)                                   | Resolve a benchmark's config to its `(domain, extra search terms)`.                                      |
| [`_fuzzy_matches`](#nemo_gym-cli-eval-_fuzzy_matches)                                         | Whether `query` fuzzily matches any of `fields`: a substring or a close difflib match (token-aware).     |
| [`_multiprocess_benchmark_prepare_fn`](#nemo_gym-cli-eval-_multiprocess_benchmark_prepare_fn) | -                                                                                                        |
| [`aggregate_rollouts`](#nemo_gym-cli-eval-aggregate_rollouts)                                 | -                                                                                                        |
| [`collect_rollouts`](#nemo_gym-cli-eval-collect_rollouts)                                     | -                                                                                                        |
| [`e2e_rollout_collection`](#nemo_gym-cli-eval-e2e_rollout_collection)                         | -                                                                                                        |
| [`list_benchmarks`](#nemo_gym-cli-eval-list_benchmarks)                                       | CLI command: list available benchmarks, optionally filtered by a `query` (the `gym search` entry point). |
| [`prepare_benchmark`](#nemo_gym-cli-eval-prepare_benchmark)                                   | CLI command: prepare benchmark data.                                                                     |
| [`reward_profile`](#nemo_gym-cli-eval-reward_profile)                                         | -                                                                                                        |

### API

```python
class nemo_gym.cli.eval.PrepareBenchmarkConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

Prepare benchmark data by running the benchmark's prepare.py script.

The benchmark is identified from a config\_paths entry pointing to a
benchmarks/\*/config.yaml file.

Examples:

```python
gym eval prepare --benchmark aime24
```

```python
nemo_gym.cli.eval._benchmark_extras(
    bench: nemo_gym.benchmarks.BenchmarkConfig
) -> tuple[str, list[str]]
```

Resolve a benchmark's config to its `(domain, extra search terms)`.

`BenchmarkConfig` flattens away the resources server name, the resources server `domain`, and the
dataset names. We re-resolve the config with the same parser `BenchmarkConfig` uses (so chained
`config_paths` / `_inherit_from` are applied) and read those fields back out for the domain column
and richer `gym search` matching.

```python
nemo_gym.cli.eval._fuzzy_matches(
    query: str,
    fields: str = ()
) -> bool
```

Whether `query` fuzzily matches any of `fields`: a substring or a close difflib match (token-aware).

```python
nemo_gym.cli.eval._multiprocess_benchmark_prepare_fn(
    args
)
```

```python
nemo_gym.cli.eval.aggregate_rollouts()
```

```python
nemo_gym.cli.eval.collect_rollouts()
```

```python
nemo_gym.cli.eval.e2e_rollout_collection()
```

```python
nemo_gym.cli.eval.list_benchmarks() -> None
```

CLI command: list available benchmarks, optionally filtered by a `query` (the `gym search` entry point).

```python
nemo_gym.cli.eval.prepare_benchmark() -> None
```

CLI command: prepare benchmark data.

```python
nemo_gym.cli.eval.reward_profile()
```