nemo_gym.eval_plan

Build a benchmark’s evaluation plan: every task once, with a stable id and a content digest.

The plan file format is versioned by its schema label and only changes additively within a version.

Module Contents

Classes

Name	Description
`EvalPlan`	A benchmark’s tasks, each listed once, sorted by task id.
`PlanBenchmark`	The benchmark the plan was made for.
`PlanGym`	The Gym that produced the plan.
`PlanTask`	One task: its identity and a digest of the exact input a run receives.

Functions

Name	Description
`_row_task_id`	-
`build_eval_plan`	Turn prepared benchmark rows into a plan.
`canonical_json`	-
`eval_plan_json`	Serialize a plan deterministically: the same plan always gives the same bytes.
`eval_plan_json_schema`	-
`write_eval_plan`	-

Data

EVAL_PLAN_SCHEMA

API

class nemo_gym.eval_plan.EvalPlan()

Bases: BaseModel

A benchmark’s tasks, each listed once, sorted by task id.

benchmark

PlanBenchmark

gym

PlanGym

model_config

= ConfigDict(extra='forbid', populate_by_name=True)

schema_name

Literal['nemo-gym.eval-plan.v1'] = Field(default=EVAL_PLAN_SCHEMA, alias='schema')

selected_task_ids

list[str] | None

tasks

list[PlanTask]

class nemo_gym.eval_plan.PlanBenchmark()

Bases: BaseModel

The benchmark the plan was made for.

config_paths

list[str]

model_config

= ConfigDict(extra='forbid')

name

str = Field(description='The benchmark dataset name.')

class nemo_gym.eval_plan.PlanGym()

Bases: BaseModel

The Gym that produced the plan.

commit

str | None

model_config

= ConfigDict(extra='forbid')

version

str

class nemo_gym.eval_plan.PlanTask()

Bases: BaseModel

One task: its identity and a digest of the exact input a run receives.

digest

str

model_config

= ConfigDict(extra='forbid')

task_id

TaskId

nemo_gym.eval_plan._row_task_id(
    row: collections.abc.Mapping[str, pydantic.JsonValue]
) -> str | None

nemo_gym.eval_plan.build_eval_plan(
    rows: collections.abc.Sequence[collections.abc.Mapping[str, pydantic.JsonValue]],
    benchmark: nemo_gym.eval_plan.PlanBenchmark,
    taskset: str,
    gym: nemo_gym.eval_plan.PlanGym,
    selected_task_ids: collections.abc.Sequence[str] | None = None
) -> nemo_gym.eval_plan.EvalPlan

Turn prepared benchmark rows into a plan.

Every row must carry its own id in one of TASK_ID_FIELDS. Unlike collation, there is no fallback to the row’s position, because a position-based id names a different task once rows are reordered. taskset is the dataset’s declared taskset, or the benchmark name when it declares none.

nemo_gym.eval_plan.canonical_json(
    value: pydantic.JsonValue
) -> bytes

nemo_gym.eval_plan.eval_plan_json(
    plan: nemo_gym.eval_plan.EvalPlan
) -> str

Serialize a plan deterministically: the same plan always gives the same bytes.

nemo_gym.eval_plan.eval_plan_json_schema() -> str

nemo_gym.eval_plan.write_eval_plan(
    plan: nemo_gym.eval_plan.EvalPlan,
    path: pathlib.Path
) -> None

nemo_gym.eval_plan.EVAL_PLAN_SCHEMA = 'nemo-gym.eval-plan.v1'