> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.eval_plan

Build a benchmark's evaluation plan: every task once, with a stable id and a content digest.

The plan file format is versioned by its schema label and only changes additively within a version.

## Module Contents

### Classes

| Name                                                 | Description                                                            |
| ---------------------------------------------------- | ---------------------------------------------------------------------- |
| [`EvalPlan`](#nemo_gym-eval_plan-EvalPlan)           | A benchmark's tasks, each listed once, sorted by task id.              |
| [`PlanBenchmark`](#nemo_gym-eval_plan-PlanBenchmark) | The benchmark the plan was made for.                                   |
| [`PlanGym`](#nemo_gym-eval_plan-PlanGym)             | The Gym that produced the plan.                                        |
| [`PlanTask`](#nemo_gym-eval_plan-PlanTask)           | One task: its identity and a digest of the exact input a run receives. |

### Functions

| Name                                                                 | Description                                                                    |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`_row_task_id`](#nemo_gym-eval_plan-_row_task_id)                   | -                                                                              |
| [`build_eval_plan`](#nemo_gym-eval_plan-build_eval_plan)             | Turn prepared benchmark rows into a plan.                                      |
| [`canonical_json`](#nemo_gym-eval_plan-canonical_json)               | -                                                                              |
| [`eval_plan_json`](#nemo_gym-eval_plan-eval_plan_json)               | Serialize a plan deterministically: the same plan always gives the same bytes. |
| [`eval_plan_json_schema`](#nemo_gym-eval_plan-eval_plan_json_schema) | -                                                                              |
| [`write_eval_plan`](#nemo_gym-eval_plan-write_eval_plan)             | -                                                                              |

### Data

[`EVAL_PLAN_SCHEMA`](#nemo_gym-eval_plan-EVAL_PLAN_SCHEMA)

### API

```python
class nemo_gym.eval_plan.EvalPlan()
```

**Bases:** `BaseModel`

A benchmark's tasks, each listed once, sorted by task id.

**`benchmark`** `PlanBenchmark`

---

**`gym`** `PlanGym`

---

**`model_config`** `= ConfigDict(extra='forbid', populate_by_name=True)`

---

**`schema_name`** `Literal['nemo-gym.eval-plan.v1'] = Field(default=EVAL_PLAN_SCHEMA, alias='schema')`

---

**`selected_task_ids`** `list[str] | None`

---

**`tasks`** `list[PlanTask]`

---

```python
class nemo_gym.eval_plan.PlanBenchmark()
```

**Bases:** `BaseModel`

The benchmark the plan was made for.

**`config_paths`** `list[str]`

---

**`model_config`** `= ConfigDict(extra='forbid')`

---

**`name`** `str = Field(description='The benchmark dataset name.')`

---

```python
class nemo_gym.eval_plan.PlanGym()
```

**Bases:** `BaseModel`

The Gym that produced the plan.

**`commit`** `str | None`

---

**`model_config`** `= ConfigDict(extra='forbid')`

---

**`version`** `str`

---

```python
class nemo_gym.eval_plan.PlanTask()
```

**Bases:** `BaseModel`

One task: its identity and a digest of the exact input a run receives.

**`digest`** `str`

---

**`model_config`** `= ConfigDict(extra='forbid')`

---

**`task_id`** `TaskId`

---

```python
nemo_gym.eval_plan._row_task_id(
    row: collections.abc.Mapping[str, pydantic.JsonValue]
) -> str | None
```

```python
nemo_gym.eval_plan.build_eval_plan(
    rows: collections.abc.Sequence[collections.abc.Mapping[str, pydantic.JsonValue]],
    benchmark: nemo_gym.eval_plan.PlanBenchmark,
    taskset: str,
    gym: nemo_gym.eval_plan.PlanGym,
    selected_task_ids: collections.abc.Sequence[str] | None = None
) -> nemo_gym.eval_plan.EvalPlan
```

Turn prepared benchmark rows into a plan.

Every row must carry its own id in one of `TASK_ID_FIELDS`. Unlike collation, there is no fallback to
the row's position, because a position-based id names a different task once rows are reordered.
`taskset` is the dataset's declared taskset, or the benchmark name when it declares none.

```python
nemo_gym.eval_plan.canonical_json(
    value: pydantic.JsonValue
) -> bytes
```

```python
nemo_gym.eval_plan.eval_plan_json(
    plan: nemo_gym.eval_plan.EvalPlan
) -> str
```

Serialize a plan deterministically: the same plan always gives the same bytes.

```python
nemo_gym.eval_plan.eval_plan_json_schema() -> str
```

```python
nemo_gym.eval_plan.write_eval_plan(
    plan: nemo_gym.eval_plan.EvalPlan,
    path: pathlib.Path
) -> None
```

```python
nemo_gym.eval_plan.EVAL_PLAN_SCHEMA = 'nemo-gym.eval-plan.v1'
```