nemo_gym.benchmarks

Benchmark discovery and preparation utilities.

Module Contents

Classes

Name	Description
`BenchmarkConfig`	-
`PrepareBenchmarkConfig`	Prepare benchmark data by running the benchmark’s prepare.py script.

Functions

Name	Description
`_load_benchmarks_from_config_paths`	-
`_multiprocess_benchmark_prepare_fn`	-
`list_benchmarks`	CLI command: list available benchmarks.
`prepare_benchmark`	CLI command: prepare benchmark data.

Data

BENCHMARKS_DIR

API

class nemo_gym.benchmarks.BenchmarkConfig()

Bases: BaseModel

agent_name

str

dataset

BenchmarkDatasetConfig

name

str

num_repeats

int

path

Path

nemo_gym.benchmarks.BenchmarkConfig.from_config_path(
    config_path: pathlib.Path
) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]

classmethod

nemo_gym.benchmarks.BenchmarkConfig.from_initial_config_dict(
    path: pathlib.Path,
    initial_config_dict: omegaconf.DictConfig
) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]

classmethod

class nemo_gym.benchmarks.PrepareBenchmarkConfig()

Bases: BaseNeMoGymCLIConfig

Prepare benchmark data by running the benchmark’s prepare.py script.

The benchmark is identified from a config_paths entry pointing to a benchmarks/*/config.yaml file.

Examples:

ng_prepare_benchmark "+config_paths=[benchmarks/aime24/config.yaml]"

num_prepare_benchmark_processes

int

use_cached_prepared_benchmarks

bool

nemo_gym.benchmarks._load_benchmarks_from_config_paths(
    config_paths: typing.List[pathlib.Path]
) -> typing.Dict[str, nemo_gym.benchmarks.BenchmarkConfig]

nemo_gym.benchmarks._multiprocess_benchmark_prepare_fn(
    args
)

nemo_gym.benchmarks.list_benchmarks() -> None

CLI command: list available benchmarks.

nemo_gym.benchmarks.prepare_benchmark() -> None

CLI command: prepare benchmark data.

nemo_gym.benchmarks.BENCHMARKS_DIR = PARENT_DIR / 'benchmarks'