nemo_gym.benchmarks

Benchmark discovery and preparation utilities.

Module Contents

Classes

BenchmarkConfig	Represents a discovered benchmark’s configuration.
PrepareBenchmarkConfig	Prepare benchmark data by running the benchmark’s prepare.py script.

Functions

discover_benchmarks	Scan the benchmarks/ directory for subdirectories containing config.yaml.
get_benchmark	Get a specific benchmark by name. Raises ValueError if not found.
list_benchmarks	CLI command: list available benchmarks.
prepare_benchmark	CLI command: prepare benchmark data.

Data

BENCHMARKS_DIR

API

nemo_gym.benchmarks.BENCHMARKS_DIR

None

class nemo_gym.benchmarks.BenchmarkConfig(name: str, path: pathlib.Path, config_dict: dict)

Represents a discovered benchmark’s configuration.

Initialization

property agent_name: Optional[str]

property num_repeats: Optional[int]

nemo_gym.benchmarks.discover_benchmarks() → Dict[str, nemo_gym.benchmarks.BenchmarkConfig]

Scan the benchmarks/ directory for subdirectories containing config.yaml.

nemo_gym.benchmarks.get_benchmark(name: str) → nemo_gym.benchmarks.BenchmarkConfig

Get a specific benchmark by name. Raises ValueError if not found.

nemo_gym.benchmarks.list_benchmarks() → None

CLI command: list available benchmarks.

class nemo_gym.benchmarks.PrepareBenchmarkConfig(/, **data: typing.Any)

Bases: nemo_gym.config_types.BaseNeMoGymCLIConfig

Prepare benchmark data by running the benchmark’s prepare.py script.

Examples:

ng_prepare_benchmark +benchmark=aime24

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

benchmark: str

‘Field(…)’

nemo_gym.benchmarks.prepare_benchmark() → None

CLI command: prepare benchmark data.