nemo_gym.cli.eval

Module Contents

Classes

Name	Description
`PrepareBenchmarkConfig`	Prepare benchmark data by running the benchmark’s prepare.py script.

Functions

Name	Description
`_inspect_benchmark`	Render the `gym list benchmarks <name>` inspect view for one benchmark.
`_multiprocess_benchmark_prepare_fn`	-
`_validate_prepared_split_file_exists`	Explicit check (not an assert: user-facing, and must survive `python -O`).
`_validate_split_datasets_declared`	Fail fast when no config declares a dataset of the requested split’s type.
`aggregate_rollouts`	-
`collect_rollouts`	-
`compare`	-
`e2e_rollout_collection`	-
`export_rollouts_as_atif`	-
`health_check_rollouts`	Run rollout quality verification for an existing run directory.
`list_benchmarks`	List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered
`prepare_benchmark`	CLI command: prepare benchmark data.
`reverify_rollouts`	-
`reward_profile`	-

Data

logger

API

class nemo_gym.cli.eval.PrepareBenchmarkConfig()

Bases: BaseNeMoGymCLIConfig

Prepare benchmark data by running the benchmark’s prepare.py script.

The benchmark is identified from a config_paths entry pointing to a benchmarks/*/config.yaml file.

Examples:

gym eval prepare --benchmark aime24

num_prepare_benchmark_processes

int

prepare_script_args

Dict[str, Any]

use_cached_prepared_benchmarks

bool

nemo_gym.cli.eval._inspect_benchmark(
    name: str,
    benchmarks: dict,
    global_config_dict
) -> None

Render the gym list benchmarks <name> inspect view for one benchmark.

nemo_gym.cli.eval._multiprocess_benchmark_prepare_fn(
    args
)

nemo_gym.cli.eval._validate_prepared_split_file_exists(
    input_jsonl_fpath: pathlib.Path,
    split: str,
    output_dirpath: pathlib.Path
) -> None

Explicit check (not an assert: user-facing, and must survive python -O).

nemo_gym.cli.eval._validate_split_datasets_declared(
    split: str,
    server_instance_configs: collections.abc.Sequence[nemo_gym.config_types.ServerInstanceConfig]
) -> None

Fail fast when no config declares a dataset of the requested split’s type.

Data preparation silently produces nothing for such a split, so without this check the run walks the entire preparation sequence (including its success banners) and only dies later trying to read the collated split file.

nemo_gym.cli.eval.aggregate_rollouts()

nemo_gym.cli.eval.collect_rollouts()

nemo_gym.cli.eval.compare() -> None

nemo_gym.cli.eval.e2e_rollout_collection()

nemo_gym.cli.eval.export_rollouts_as_atif() -> None

nemo_gym.cli.eval.health_check_rollouts(
    run_dir: str | pathlib.Path,
    rollout_file: str | pathlib.Path | None = None,
    workers: int | None = None,
    ignored_checks: collections.abc.Sequence[str] = (),
    json_output: bool = False
)

Run rollout quality verification for an existing run directory.

nemo_gym.cli.eval.list_benchmarks() -> None

List available benchmarks, or inspect one by name (gym list benchmarks <name>). Optionally filtered by a query (the gym search entry point).

A benchmark is a specific kind of environment, so it shares gym list environments’ columns (name, domain, description) and reads them through the same read_config_metadata helper. --search-dir adds extra roots to scan on top of the cwd and built-ins.

nemo_gym.cli.eval.prepare_benchmark() -> None

CLI command: prepare benchmark data.

nemo_gym.cli.eval.reverify_rollouts()

nemo_gym.cli.eval.reward_profile()

nemo_gym.cli.eval.logger = logging.getLogger(__name__)