nemo_gym.benchmarks

Benchmark discovery and preparation utilities.

Module Contents

Classes

Name	Description
`BenchmarkConfig`	-

Functions

Name	Description
`_load_benchmarks_from_config_paths`	-
`_parse_no_environment_tolerating_unset_values`	`parse_no_environment` for listing: fill unset `???` values and undefined `${...}` interpolations

Data

BENCHMARKS_DIR

_UNSET_VALUE_PLACEHOLDER

__getattr__

API

class nemo_gym.benchmarks.BenchmarkConfig()

Bases: BaseModel

agent_name

str

dataset

BenchmarkDatasetConfig

name

str

num_repeats

int

path

Path

nemo_gym.benchmarks.BenchmarkConfig.from_config_path(
    config_path: pathlib.Path,
    strict: bool = True
) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]

classmethod

nemo_gym.benchmarks.BenchmarkConfig.from_initial_config_dict(
    path: pathlib.Path,
    initial_config_dict: omegaconf.DictConfig,
    strict: bool = True
) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]

classmethod

nemo_gym.benchmarks._load_benchmarks_from_config_paths(
    config_paths: typing.List[pathlib.Path]
) -> typing.Dict[str, nemo_gym.benchmarks.BenchmarkConfig]

nemo_gym.benchmarks._parse_no_environment_tolerating_unset_values(
    initial_config_dict: omegaconf.DictConfig
) -> omegaconf.DictConfig

parse_no_environment for listing: fill unset ??? values and undefined ${...} interpolations with a placeholder so a benchmark referencing runtime-only values can still be identified. Never mutates the caller’s config; errors other than those two propagate.

??? is filled anywhere; ${...} only where parse forces resolution (top-level and server sections), not inside arbitrary non-server nested dicts — fine for listing, whose interpolations live in servers.

nemo_gym.benchmarks.BENCHMARKS_DIR = PARENT_DIR / 'benchmarks'

nemo_gym.benchmarks._UNSET_VALUE_PLACEHOLDER = '__unset_for_listing__'

nemo_gym.benchmarks.__getattr__ = moved_attr_getter(__name__, {'list_benchmarks': 'nemo_gym.cli.eval', 'PrepareBen...