nemo_gym.cli.eval

Module Contents

Classes

Name	Description
`PrepareBenchmarkConfig`	Prepare benchmark data by running the benchmark’s prepare.py script.

Functions

Name	Description
`_inspect_benchmark`	Render the `gym list benchmarks <name>` inspect view for one benchmark.
`_multiprocess_benchmark_prepare_fn`	-
`aggregate_rollouts`	-
`collect_rollouts`	-
`e2e_rollout_collection`	-
`list_benchmarks`	List available benchmarks, or inspect one by name (`gym list benchmarks <name>`). Optionally filtered
`prepare_benchmark`	CLI command: prepare benchmark data.
`reverify_rollouts`	-
`reward_profile`	-

API

class nemo_gym.cli.eval.PrepareBenchmarkConfig()

Bases: BaseNeMoGymCLIConfig

Prepare benchmark data by running the benchmark’s prepare.py script.

The benchmark is identified from a config_paths entry pointing to a benchmarks/*/config.yaml file.

Examples:

gym eval prepare --benchmark aime24

num_prepare_benchmark_processes

int

prepare_script_args

Dict[str, Any]

use_cached_prepared_benchmarks

bool

nemo_gym.cli.eval._inspect_benchmark(
    name: str,
    benchmarks: dict,
    global_config_dict
) -> None

Render the gym list benchmarks <name> inspect view for one benchmark.

nemo_gym.cli.eval._multiprocess_benchmark_prepare_fn(
    args
)

nemo_gym.cli.eval.aggregate_rollouts()

nemo_gym.cli.eval.collect_rollouts()

nemo_gym.cli.eval.e2e_rollout_collection()

nemo_gym.cli.eval.list_benchmarks() -> None

List available benchmarks, or inspect one by name (gym list benchmarks <name>). Optionally filtered by a query (the gym search entry point).

A benchmark is a specific kind of environment, so it shares gym list environments’ columns (name, domain, description) and reads them through the same read_config_metadata helper. --search-dir adds extra roots to scan on top of the cwd and built-ins.

nemo_gym.cli.eval.prepare_benchmark() -> None

CLI command: prepare benchmark data.

nemo_gym.cli.eval.reverify_rollouts()

nemo_gym.cli.eval.reward_profile()