> This page is for version 0.6.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # CLI Commands > Reference for the unified gym CLI: command groups, flags, and migration from the legacy ng_* commands. This page documents the NeMo Gym command-line interface. All functionality is exposed through a single `gym` entry point, organized into command groups (`gym `). `ng` is a drop-in alias for `gym`. Every group and command supports `-h`/`--help`. > **Note** > > The legacy `ng_*` / `nemo_gym_*` commands (such as `ng_run` or `nemo_gym_collect_rollouts`) still work but are deprecated. Each one prints a notice pointing at its `gym` replacement and then runs it. See [Migrating from the legacy commands](#migrating-from-the-legacy-commands) at the bottom of this page. ## Quick Reference ```bash # General gym --help # list all command groups gym --version [--json] # print version and system info ng ... # 'ng' is an alias for 'gym' # Discover gym list benchmarks [] [--json] # list benchmarks (or inspect one), by their --benchmark value gym list environments [] [--json] # list the unified environment and benchmark catalog gym list agents [] [--json] # list agent harnesses and how each composes gym list models [] [--json] # list --model-type values (or inspect a model) gym list resources-servers [] [--json] # list --resources-server values (or inspect one) gym search [] [--json] # search the unified environment catalog, or one component type # Datasets gym dataset upload # upload a prepared dataset to HF (default) or GitLab gym dataset download # download a dataset from HF (default) or GitLab gym dataset rm # delete a dataset from GitLab gym dataset migrate # move a dataset from GitLab to HF gym dataset render # generate a dataset preview (materialize prompts) gym dataset collate # validate and collate a dataset # Environments gym env init # scaffold a resources server, environment, or benchmark gym env resolve # resolve and print the final merged config gym env validate # validate a manifest-backed workload or legacy config without services gym env packages # list packages in a server's virtual environment gym env test # run resources-server tests gym env publish # finalize readiness and confirm registry discovery gym env start # start the servers gym env status # show running servers # Evaluation gym eval prepare # prepare benchmark data and dump it to disk gym eval run # collate data, start servers, and collect rollouts gym eval aggregate # merge sharded rollout results gym eval health-check # verify rollout artifact quality for an existing run gym eval profile # compute a reward profile from rollouts gym eval reverify # recompute rewards from existing rollouts without re-running inference # Contributor helpers gym dev test # run NeMo Gym's unit tests ``` ## Common Options These options are shared across many commands. | Option | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--config PATH` | Load a Gym config YAML. Repeatable. Maps to `+config_paths=[...]`. | | `--benchmark NAME` | Select a registered benchmark by name instead of a config path. | | `--resources-server NAME` | Select a registered resources server by name. | | `--model-type NAME` | Select a registered model server type by name (such as `openai_model` or `vllm_model`). | | `--search-dir DIR` | Extra root directory to search for components and configs. Repeatable. Applies everywhere Gym resolves paths — discovery, the `--` selectors, `config_paths`, prompt configs, and data files — and is inherited by spawned servers. Equivalent to the `NEMO_GYM_EXTRA_ROOTS` environment variable; see [Configuration](/reference/configuration). | | `--json` | Emit machine-readable JSON instead of human-readable output (reporting commands only). Accepted before or after the subcommand; commands that emit no JSON reject it. | | `-v`, `--verbose` | Set the logging level to DEBUG. Flows through to spun-up servers. | | `-h`, `--help` | Show help for any group or command. | > **Tip** > > The `--benchmark`, `--resources-server`, and `--model-type` selectors resolve a component name to its config file for you, so you do not need to know the project's directory layout. If you mistype a name, the CLI suggests the closest match. To point at a config file directly, use `--config ` instead. ### Selecting the model server Commands that need a model (`gym env start`, `gym eval run`) configure it through four flags: | Flag | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `--model-type NAME` | The model server type to load (such as `openai_model`, `vllm_model`, or `local_vllm_model`). | | `--model`, `-m` | The served model identifier: an API model name, an HF id, or a local checkpoint path. Interpreted per `--model-type`. Maps to `policy_model_name`. | | `--model-url` | Base URL of an existing model server endpoint. Maps to `policy_base_url`. | | `--model-api-key` | API key for the model server. Maps to `policy_api_key`. | ### Hydra overrides (escape hatch) The `gym` CLI is a thin wrapper over Gym's Hydra config system. Standard flags (`--flag value`) cover the common inputs. Anything not covered by a flag can still be passed as a raw Hydra override using `+key=value` (add a new key) or `++key=value` (add or override an existing key). Unknown overrides are forwarded to Hydra untouched, so advanced config composition is fully functional. When a command needs overrides for keys that have no dedicated flag, you can keep the whole command in Hydra form (config paths included) or mix `--flag` and `+key=value` styles in one invocation: ```bash gym eval run \ --benchmark aime24 \ --model-type openai_model \ ++responses_create_params.reasoning.effort=low \ +wandb_project=gym-dev ``` --- ## General ### `gym --help` List all command groups and their descriptions. ```bash gym --help ``` ### `gym --version` Print the NeMo Gym version along with Python, key dependency, and system information. | Option | Description | | -------- | ----------------------------------- | | `--json` | Output version information as JSON. | ```bash gym --version # Output as JSON gym --version --json ``` --- ## Discovery Commands for discovering benchmarks, environments, model servers, agent harnesses, and resources servers. `gym list ` lists all components of that type; `gym list ` inspects that one. All commands accept: * `--json` which prints machine-readable output instead of a human-readable table. * `--search-dir` which allows you to surface your own Gym-compatible components alongside the built-ins. ### `gym list benchmarks` List the benchmarks available in NeMo Gym, with their kind, manifest status, domain, and description. The value in the **Name** column is the benchmark identifier and can be passed to `--benchmark`. To see which agent harness a benchmark composes with, inspect it with `gym list benchmarks `. | Option | Description | | ---------- | -------------------------------------------------------------------------------------------------- | | `[]` | Inspect a single benchmark (config path, manifest status, agent, datasets) instead of listing all. | | `--json` | Output the benchmark list as JSON. | ```bash gym list benchmarks gym list benchmarks aime24 # inspect one benchmark # Machine-readable output for scripting gym list benchmarks --json | jq '.[].name' ``` ### `gym list environments` List the manifest-first catalog unioned with legacy environment and benchmark configs. Manifest-backed entries expose their declared metadata; unmigrated entries remain visible with `no-manifest` status. | Option | Description | | --------------------------------------- | --------------------------------------------- | | `[]` | Inspect one environment or benchmark. | | `--kind environment\|benchmark` | Filter the list or resolve an ambiguous name. | | `--domain`, `--modality`, `--licensing` | Filter by catalog metadata. | | `--status experimental\|no-manifest` | Filter by manifest coverage status. | | `--lifecycle active\|deprecated` | Filter by lifecycle. | | `--json` | Output catalog entries as JSON. | ```bash gym list environments gym list environments calendar # inspect one environment gym list environments --kind benchmark --domain math gym list environments --json | jq '.[].name' ``` ### `gym list agents` List the agent harnesses under `responses_api_agents/`, with each one's composition pattern — **composable** (Pattern A: references a separate resources server, so it can be wired into a matching environment) vs **self-contained** (Pattern B: ships its own framework/environment) — plus its config variants. | Option | Description | | ---------- | ------------------------------------------------------------------- | | `[]` | Inspect a single agent (composition pattern and config variants). | | `--json` | Output the agent list as JSON (includes the `self_contained` flag). | ```bash gym list agents gym list agents simple_agent # inspect one agent gym list agents --json | jq '.[] | {name, self_contained}' ``` ### `gym list models` List the model servers under `responses_api_models/`. The value in the **Model** column corresponds to a config flavor and can be passed as the `--model-type` flag to other commands. | Option | Description | | ---------- | ---------------------------------------------- | | `[]` | Inspect a single model instead of listing all. | | `--json` | Output the rows as JSON. | ```bash gym list models gym list models vllm_model # inspect one model gym list models --json | jq '.[].model' ``` ### `gym list resources-servers` List the resources servers under `resources_servers/`. The value in the **Name** column corresponds to the config flavor and can be passed as `--resources-server`. | Option | Description | | ---------- | --------------------------------------------------------------------- | | `[]` | Inspect a single resources server (config path, domain, description). | | `--json` | Output the list as JSON. | ```bash gym list resources-servers gym list resources-servers mcqa # inspect one resources server gym list resources-servers --json | jq '.[].name' ``` ### `gym search` Filter a listing to fuzzy matches on a query (against each component's name/value, description, domain, and type-specific fields). Without a type, the command searches the unified environment and benchmark catalog. | Option | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `[]` | Optional component type: `benchmarks`, `environments`, `agents`, `models`, or `resources-servers`. | | `QUERY` | Text matched (substring or fuzzy) against a component's name, description, and key metadata (positional, required). | | `--json` | Output matches as JSON. | ```bash gym search math # environments and benchmarks matching "math" gym search benchmarks math # only benchmarks matching "math" gym search environments calendar # environments matching "calendar" gym search models vllm --json ``` --- ## Datasets Commands for preparing, previewing, and managing datasets. By default dataset transfer commands use HuggingFace; pass `--storage gitlab` to target the GitLab Registry. ### `gym dataset upload` Upload a prepared local JSONL dataset to HuggingFace (default) or GitLab. | Option | Description | | ----------------------- | ------------------------------------------------ | | `--storage {hf,gitlab}` | Storage backend. Default: `hf`. | | `--input`, `-i` | Local JSONL file to upload. | | `--name` | Dataset name. | | `--revision` | Dataset revision (version). | | `--split` | Dataset split (HF only). | | `--create-pr` | Open a pull request with your changes (HF only). | ```bash # Upload to HuggingFace gym dataset upload \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 # Upload to GitLab gym dataset upload \ --storage gitlab \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 ``` ### `gym dataset download` Download a dataset from HuggingFace (default) or GitLab. | Option | Description | | ----------------------- | -------------------------------------------------------------------------- | | `--storage {hf,gitlab}` | Storage backend. Default: `hf`. | | `--repo-id` | HF repo id, such as `org/dataset` (HF only). | | `--name` | Dataset name (GitLab only). | | `--revision` | Dataset version (GitLab only). | | `--artifact` | Remote file to fetch (GitLab: required; HF: optional raw file). | | `--output`, `-o` | Local destination file. | | `--output-dir` | Local destination directory; needed when downloading all splits (HF only). | | `--split` | Dataset split (HF only). | ```bash # Download a single file from HuggingFace gym dataset download --repo-id NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \ --artifact train.jsonl \ --output data/train.jsonl # Download from GitLab gym dataset download \ --storage gitlab \ --name example_multi_step \ --revision 0.0.1 \ --artifact train.jsonl \ --output data/train.jsonl ``` ### `gym dataset rm` Delete a dataset from the GitLab Registry. Prompts for confirmation. | Option | Description | | -------- | ------------------------------ | | `--name` | Name of the dataset to delete. | ```bash gym dataset rm --name old_dataset ``` ### `gym dataset migrate` Migrate a JSONL dataset to HuggingFace from GitLab. Use `gym dataset upload` if you do not want automatic GitLab deletion. | Option | Description | | --------------- | ---------------------------------------------------- | | `--input`, `-i` | Local JSONL file to upload to HF. | | `--name` | Dataset name. | | `--revision` | Dataset revision (HF). | | `--split` | Dataset split. | | `--create-pr` | Open a pull request to HF dataset with your changes. | ```bash gym dataset migrate \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 ``` ### `gym dataset collate` Validate and collate a dataset, generating metrics and statistics. | Option | Description | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--resources-server NAME` | Load the named resources server config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--mode {train_preparation,example_validation}` | Use `train_preparation` to prepare train/validation datasets, or `example_validation` to validate example data. | | `--output-dir` | Output directory for the prepared data. | | `--download` | Download source datasets before collating. | ```bash gym dataset collate \ --resources-server example_multi_step \ --output-dir data/example_multi_step \ --mode example_validation ``` ### `gym dataset render` Generate a dataset preview by materializing prompts from a raw input file and a prompt template, producing JSONL with populated `responses_create_params.input` for RL training. Each input row must **not** already have a populated `responses_create_params.input`; the command applies the prompt template from `--prompt-config` to each row, fills in the input, and preserves the row's other fields. | Option | Description | | ------------------ | -------------------------------------------------------------------- | | `--input`, `-i` | Raw input JSONL file (rows without `responses_create_params.input`). | | `--prompt-config` | Prompt template YAML to apply. | | `--output`, `-o` | Output JSONL file. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym dataset render \ --input raw.jsonl \ --prompt-config prompt.yaml \ --output preview.jsonl ``` > **Note** > > **Which data-preparation command should I use?** > > * **`gym dataset render`** — a focused, standalone step that applies a prompt template to raw rows to populate `responses_create_params.input`. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows. > * **`gym dataset collate`** — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission. --- ## Environments Commands for developing, running, and inspecting environments. An environment defines tasks, verification, and its interaction surface; Gym config composes it with agent, model, and runtime components. ### `gym env init` Scaffold a resources server, or create a manifest-backed environment or benchmark skeleton. Workload scaffolds include a manifest, Gym config, sample data, and the extension points selected by the integration profile. | Option | Description | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `--resources-server NAME` | Name of the resources server to create. | | `--environment NAME` / `--benchmark NAME` | Create a manifest-backed environment or benchmark. | | `--profile PROFILE` | Integration profile: `custom-gym-verifier`, `custom-gym-agent-loop`, `external-agent-loop`, or `external-rollout-driver`. | | `--reuse-verifier NAME` | Reuse an existing resources server that exports `VERIFIER_FIXTURE`. | | `--reward-range LOW HIGH` | Reward endpoints required when reusing a verifier. | | `--higher-is-better` / `--lower-is-better` | Reward direction required when reusing a verifier. | ```bash gym env init --resources-server my_server gym env init --environment my_eval --profile custom-gym-verifier gym env init --benchmark my_benchmark --reuse-verifier existing_scorer \ --reward-range 0 1 --higher-is-better ``` ### `gym env resolve` Resolve the configs, flags, and overrides into a final merged config and print it. Useful for debugging configuration. Secrets are hidden. Unlike `gym env validate`, `resolve` substitutes no dummy model, so a model config's `policy_*` values must be supplied for its interpolations to resolve. | Option | Description | | ------------------ | ---------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash # Merge a resources-server config with a model config, apply an override, and print the result gym env resolve \ --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml \ --config responses_api_models/openai_model/configs/openai_model.yaml \ ++responses_create_params.temperature=0.6 \ ++policy_base_url=https://api.openai.com/v1 \ ++policy_api_key=sk-example \ ++policy_model_name=gpt-4o-mini ``` ### `gym env validate` Validate a manifest-backed workload or legacy config without starting Ray or server subprocesses. Manifest validation resolves the Gym config offline, checks composition mirrors and datasets, and reports the resolved components. Legacy validation retains the existing config pre-flight behavior. Exits `0` when valid, or `1` with a clean message when not. By default, values a run needs but the config leaves undefined (such as `policy_base_url` from `env.yaml`) are filled with placeholders so the rest can still be checked, and reported as a warning. Pass `--strict` to fail on them instead. | Option | Description | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `NAME` | Manifest-backed catalog entry to validate. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--manifest PATH` | Validate a manifest directly instead of selecting by name. | | `--sync` | Atomically synchronize config-owned composition mirrors after all checks pass. | | `--strict` | Resolve the config exactly as a run would, with no placeholder model values. An undefined value fails instead of warning. | | `--json` | Output the manifest validation report as JSON. | | `--config PATH` | Config file to load. Repeatable. | | `--environment NAME` / `--benchmark NAME` | Validate a named environment / benchmark config. | | `--resources-server NAME` | Validate a named resources server config. | | `--model-type NAME` | Also load a named model config (otherwise a dummy `policy_model` is used). | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--model` / `--model-url` / `--model-api-key` | Override model name, base URL, and API key. | ```bash gym env validate my_eval gym env validate --sync my_eval gym env validate --environment workplace_assistant gym env validate --benchmark gsm8k # or explicit config path(s) gym env validate --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml # fail unless everything a run needs is defined, credentials included gym env validate --resources-server example_single_tool_call --model-type openai_model --strict ``` ### `gym env packages` Each server has its own isolated virtual environment. List the packages installed in a server's environment. | Option | Description | | ------------------------- | ---------------------------------------------------------------- | | `--resources-server NAME` | Name of the resources server. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--outdated` | List only outdated packages. | | `--json` | Output the package list as JSON. | ```bash gym env packages --resources-server example_single_tool_call # Check for outdated packages gym env packages \ --resources-server example_single_tool_call \ --outdated ``` ### `gym env test` Exercise a manifest-backed workload's verifier fixture in its resources-server environment, or run the existing pytest suite for one or all resources servers. | Option | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------- | | `NAME` | Manifest-backed catalog entry whose verifier fixture should run. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--update-expected` | Atomically update fixture rewards after every behavioral check passes. | | `--json` | Output the verifier report as JSON. | | `--resources-server NAME` | Resources server to test. | | `--all` | Test every resources server. Slow: builds one venv per server, so it must be requested explicitly. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash # Exercise a manifest-backed workload verifier gym env test my_eval # Test a single server gym env test --resources-server example_single_tool_call # Test all servers. Slow: builds one venv per server, so it needs an explicit opt-in. gym env test --all ``` ### `gym env publish` Run the local publication check for a manifest-backed workload. The command runs static validation and the verifier fixture, rejects manifest metadata placeholders, and confirms that the exact manifest is visible as `experimental`. It does not assess implementation quality, require a full evaluation or training run, commit, or push changes. | Option | Description | | ------------------------------- | ----------------------------------------- | | `NAME` | Manifest-backed catalog entry to publish. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--json` | Output the publication result as JSON. | ```bash gym env publish my_eval gym env publish my_benchmark --kind benchmark --json ``` ### `gym env start` Start the NeMo Gym servers (agents, models, resources) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment. | Option | Description | | ------------------------- | --------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config (start its servers). | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model server type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--model`, `-m` | Served model identifier. See [Selecting the model server](#selecting-the-model-server). | | `--model-url` | Model server base URL. | | `--model-api-key` | Model server API key. | ```bash gym env start \ --resources-server example_single_tool_call \ --model-type openai_model # Start a benchmark's servers gym env start \ --benchmark gpqa \ --model-type vllm_model ``` ### `gym env status` Show all currently running NeMo Gym servers and their health. | Option | Description | | -------- | ------------------------------- | | `--json` | Output the server list as JSON. | ```bash gym env status ``` ``` NeMo Gym Server Status: [1] ✓ example_single_tool_call (resources_servers/example_single_tool_call) { 'server_type': 'resources_servers', 'name': 'example_single_tool_call', 'port': 58117, 'pid': 89904, 'uptime_seconds': '0d 0h 0m 41.5s', } ... 3 servers found (3 healthy, 0 unhealthy) ``` --- ## Evaluation Commands for running evaluations end to end: prepare data, collect rollouts, aggregate sharded runs, verify rollout health, and profile results. ### `gym eval prepare` Prepare a benchmark's data by running its `prepare.py` script and dump the result to disk. | Option | Description | | ------------------ | ---------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym eval prepare --benchmark aime24 ``` ### `gym eval run` Collate data, start the servers, and collect rollouts. This is the main evaluation command. By default it spins up all required servers. Pass `--no-serve` to collect against servers you already started with `gym env start`. | Option | Description | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model server type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--no-serve` | Collect against already-running servers instead of starting them. | | `--resume` | Resume from cached rollouts instead of recollecting. Maps to legacy `+resume_from_cache=true`. Refer to [Resume interrupted runs](#resume-interrupted-runs). | | `--agent`, `-a` | Agent to collect rollouts with. | | `--input`, `-i` | Input tasks JSONL file. | | `--output`, `-o` | Output rollouts JSONL file. | | `--limit` | Maximum number of tasks to run. | | `--num-repeats` | Rollouts per task (for mean\@k metrics). Pass an int to apply to every task, or a dict keyed by `agent_ref.name` for per-agent counts (e.g. `'{simple_agent: 32, swe_agent: 1}'`) when one input file mixes agents. In dict form, the special key `_default` is the fallback for agents not explicitly listed; without it, any unlisted row's agent raises a single consolidated error. | | `--prompt-config` | Prompt template YAML to apply. | | `--concurrency` | Maximum number of concurrent samples. | | `--split` | Dataset split to use (`train`, `validation`, or `benchmark`). A dataset of the matching `type` must be declared in the loaded configs. `example` datasets are smoke-test samples, not a runnable split — run them with `--no-serve` and `--input` as shown in the [Quickstart](/get-started/quickstart). | | `--model`, `-m` | Served model identifier. | | `--model-url` | Model server base URL. | | `--model-api-key` | Model server API key. | | `--temperature` | Sampling temperature. | | `--top-p` | Nucleus sampling top-p. | | `--max-output-tokens` | Maximum output tokens. | | `--disable-aggregation` | Skip aggregate metrics and the automatic health check. Use for shards that will be combined with `gym eval aggregate`. | | `--no-health-check` | Skip the automatic post-run health check. | | `--health-check-workers` | Number of rollout-health worker processes. Defaults to the smaller of the CPU count and 8. | | `--health-check-ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | ```bash # End-to-end: spin up servers, then collect rollouts for a benchmark gym eval run --benchmark aime24 \ --model-type openai_model \ --output results/aime24.jsonl \ --split validation \ --concurrency 10 # Against an already-running server, with a remote vLLM endpoint gym eval run --no-serve \ --model-type openai_model \ --resources-server math_with_judge \ --output results/test_001.jsonl \ --split validation \ --model openai/gpt-oss-120b \ --model-url http://0.0.0.0:10240/v1 \ --model-api-key dummy_key \ --temperature 1.0 \ --top-p 1.0 # Per-agent repeats: one input file pins different agents per row via agent_ref.name gym eval run --no-serve \ --model-type openai_model \ --input mixed_agents.jsonl \ --output results/mixed_rollouts.jsonl \ --num-repeats '{agent_alpha: 4, agent_beta: 1, _default: 1}' ``` #### Generation parameters The most common sampling parameters have dedicated flags on `gym eval run`: `--temperature`, `--top-p`, and `--max-output-tokens`. These map onto `responses_create_params.temperature`, `responses_create_params.top_p`, and `responses_create_params.max_output_tokens`. Any other `responses_create_params` field that has no dedicated flag can be set with a raw Hydra override using the `++responses_create_params.` syntax. Overrides are merged into each input row's existing `responses_create_params` with a **shallow** merge (top-level keys only): ```bash gym eval run --no-serve \ --agent example_single_tool_call_simple_agent \ --input weather_query.jsonl \ --output weather_rollouts.jsonl \ --temperature 1.0 \ --top-p 1.0 \ --max-output-tokens 4096 \ ++responses_create_params.reasoning.effort=low ``` > **Tip** > > Because the merge is shallow, setting a field inside a nested object, such as `++responses_create_params.reasoning.effort=low`, replaces the row's entire nested dictionary at that key. Other fields under the same nested object are not preserved. #### Resume interrupted runs Pass `--resume` to restart the **same command** after a crash or interruption and pick up only the rows that have not finished yet. How it works: * **Materialized inputs.** On the first run, the fully expanded input rows (after `--num-repeats`, `--limit`, `--prompt-config`, and any overrides) are written to a sidecar file next to your output. The path is derived from `--output` by appending `_materialized_inputs` to the stem — so `rollouts.jsonl` produces `rollouts_materialized_inputs.jsonl`. * **Incremental output.** Successful rollouts are flushed to the main output JSONL after each completion; retriable failures go to a `_failures.jsonl` sidecar, so partial progress survives a crash. * **Failed agent calls.** An agent `/run` call that returns an error, or never returns a readable result, is recorded in the same sidecar as one attempt, with no reward. The class says whether the rollout ran: `agent_run_error` when the agent itself answered, since a NeMo Gym agent returns 500 when its handler raises, and `agent_request_failed` for a gateway status (429, 502, 503, 504) or no usable reply, neither of which says anything about the rollout. Other rollouts keep running, the score is computed over completed rollouts only, and resume re-dispatches the row up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. A run in which nothing produced a result fails instead of reporting an empty score. * **Counting failures in the score.** Metrics are computed over completed rollouts only. `+count_failure_classes_as_zero=[]` includes the named sidecar rows in `/aggregate_metrics` so they land in the denominator: `[agent_run_error]` counts the rollouts that reached the agent and broke, which is what a model emitting unparsable tool calls looks like from here. A row that carries no reward is scored zero for the metric input only, so neither the sidecar nor the rollouts JSONL records a verdict that no verifier gave. It works the same in `gym eval run` and `gym eval aggregate`. * **Matching.** On resume, completed work is matched by `(task_index, rollout_index)` against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run. * **Fallback.** If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. Without `--resume`, existing output is cleared before the run. > **Warning** > > If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the `*_materialized_inputs.jsonl` file (and the output file) to start fresh. ### `gym eval aggregate` Merge sharded rollout results into a single rollouts file with aggregate metrics. Reads every JSONL file matching `--input-glob`, recomputes aggregate metrics over the global union of records, and writes a `_aggregate_metrics.json` next to the merged rollouts. Use this to combine shards produced by `gym eval run --no-serve --disable-aggregation`. | Option | Description | | ------------------------ | ------------------------------------------------------------------------------------------ | | `--config PATH` | Config file to load. Repeatable. | | `--input-glob`, `-i` | Glob (or comma-separated globs) matching the rollout shards to aggregate. | | `--output`, `-o` | Path for the merged rollouts and aggregate-metrics file. | | `--no-health-check` | Skip the automatic post-aggregation health check. | | `--health-check-workers` | Number of rollout-health worker processes. Defaults to the smaller of the CPU count and 8. | | `--health-check-ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | ```bash gym eval aggregate \ --config benchmarks/aime24/config.yaml \ --config responses_api_models/vllm_model/configs/vllm_model.yaml \ --input-glob 'results/rollouts-rs*-chunk*.jsonl' \ --output results/rollouts.jsonl ``` ### `gym eval health-check` Verify rollout quality for an existing run directory. The command reads `/rollouts.jsonl` by default and writes `quality_summary.json` and `rollout_verdicts.jsonl` in the run directory. | Option | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `RUN_DIR` | Directory in which to write the health reports. | | `--rollouts-file PATH` | Rollout JSONL path. Relative paths resolve under `RUN_DIR`; absolute paths are used as written. Defaults to `rollouts.jsonl`. | | `--workers` | Number of worker processes. Defaults to the smaller of the CPU count and 8. | | `--ignore-checks`, `--ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | ```bash gym eval health-check results/my-run # Select a nonstandard rollout filename and exclude one known check gym eval health-check results/my-run \ --rollouts-file evaluator_rollouts.jsonl \ --ignore-checks model_call_missing_token_counts ``` Refer to [Rollout Health Checks](/main/evaluation/rollout-health) for verdict semantics, evidence requirements, the complete check catalog, and report schemas. ### `gym eval profile` Compute a reward profile from collected rollouts. Outputs per-task statistics such as average reward, standard deviation, min/max, and pass rate, useful for filtering tasks before training by difficulty or variance. Requires rollouts collected with `--num-repeats` greater than 1. | Option | Description | | ------------ | ---------------------------------------------------- | | `--inputs` | Materialized inputs JSONL fed to rollout collection. | | `--rollouts` | Rollouts JSONL produced by collection. | ```bash gym eval profile \ --inputs materialized_inputs.jsonl \ --rollouts rollouts.jsonl ``` Writes three files next to the rollouts, named from its stem: | File | Contents | | -------------------------------------- | --------------------- | | `_reward_profiling.jsonl` | One row per task. | | `_agent_metrics.json` | One entry per agent. | | `_repeat_level_metrics.json` | One entry per repeat. | See [Repeat-Level Metrics](/evaluation/aggregate-metrics#repeat-level-metrics) for the full field list and how to read the variability statistics. ### `gym eval reverify` Recompute rewards from existing rollouts by replaying them through a resources server's `/verify` endpoint — without re-running model inference. Starts the resources server automatically from the provided config. Requires the `*_materialized_inputs.jsonl` and `rollouts.jsonl` artifacts produced by `gym eval run`. Before starting, the command checks each resources server's `GET /reverify_mode` response. Resources servers report one of three modes: | Mode | Meaning | | ------------- | ----------------------------------------------------------------------------------------------------------------- | | `stateless` | Reverification is safe. The verifier is a pure function of `(request body, server config)`. | | `unsupported` | Reverification is not safe. The verifier reads per-rollout session state that is no longer present. | | `unknown` | The server has not declared its reverification behaviour. **This is the default.** Treated as potentially unsafe. | Servers that report `unsupported` or `unknown` cause the command to abort unless `--force` is passed. | Option | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file for the resources server. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--resources-server NAME` | Load the named resources server config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--inputs PATH` | `*_materialized_inputs.jsonl` from `gym eval run`. | | `--rollouts PATH` | `rollouts.jsonl` from `gym eval run`. | | `--output PATH`, `-o` | Output JSONL for recomputed rollouts. | | `--force` | Override the `UNSUPPORTED` or `UNKNOWN` reverify mode guard; output is prefixed with `unsafe_`. | | `--overwrite` | Delete an existing output file instead of raising an error. | | `--resume` | Resume a partial run: skip rows already in the output and re-verify only the rest. | | `--judge-failed-only` | Recover only the rollouts whose judge call failed in the original run (from `_failures.jsonl`), reusing the stored responses. Skips the reverify-mode guard. | | `--append` | With `--judge-failed-only`: append recovered rows to an existing `--output` instead of a fresh file. Mutually exclusive with `--overwrite`. | | `--disable-aggregation` | Skip aggregate-metrics computation (use when combining shards with `gym eval aggregate` afterward). | ```bash # Recompute rewards with an updated verifier config (no model server needed) gym eval reverify \ --config my_resources_server.yaml \ "++head_server.host=127.0.0.1" \ "++head_server.port=9500" \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_reverified.jsonl # Override a verifier hyperparameter inline gym eval reverify \ --config my_resources_server.yaml \ "++mcqa.resources_servers.mcqa.grading_mode=lenient_boxed" \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_lenient.jsonl # Re-run the judge on only the rollouts whose judge failed (reuses stored responses; no inference) gym eval reverify --judge-failed-only \ --config my_resources_server.yaml \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_recovered.jsonl # Force through an UNSUPPORTED or UNKNOWN server (output prefixed with unsafe_) gym eval reverify \ --config my_stateful_server.yaml \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_reverified.jsonl \ --force ``` See [Reverify Rollouts](/tutorials/evaluation-tutorials/reverify-rollouts) for a full walkthrough. --- ## Contributor Helpers ### `gym dev test` Run NeMo Gym's core unit tests with coverage reporting. ```bash gym dev test ``` --- ## Migrating from the legacy commands The legacy `ng_*` and `nemo_gym_*` are deprecated and will be removed in the future release. Use the tables below to find their `gym` replacement and update your scripts and workflows. ### Command mapping | Legacy command | New command | | --------------------------------- | --------------------------------------- | | `ng_help` | `gym --help` | | `ng_version` | `gym --version` | | `ng_list_benchmarks` | `gym list benchmarks` | | `ng_run` | `gym env start` | | `ng_status` | `gym env status` | | `ng_dump_config` | `gym env resolve` | | `ng_pip_list` | `gym env packages` | | `ng_init_resources_server` | `gym env init` | | `ng_test` | `gym env test` | | `ng_test_all` | `gym env test --all` | | `ng_prepare_benchmark` | `gym eval prepare` | | `ng_e2e_collect_rollouts` | `gym eval run` | | `ng_collect_rollouts` | `gym eval run --no-serve` | | `ng_aggregate_rollouts` | `gym eval aggregate` | | `ng_reward_profile` | `gym eval profile` | | `ng_prepare_data` | `gym dataset collate` | | `ng_materialize_prompts` | `gym dataset render` | | `ng_upload_dataset_to_hf` | `gym dataset upload` | | `ng_upload_dataset_to_gitlab` | `gym dataset upload --storage gitlab` | | `ng_download_dataset_from_hf` | `gym dataset download` | | `ng_download_dataset_from_gitlab` | `gym dataset download --storage gitlab` | | `ng_gitlab_to_hf_dataset` | `gym dataset migrate` | | `ng_delete_dataset_from_gitlab` | `gym dataset rm` | | `ng_dev_test` | `gym dev test` | | `ng_reinstall` | `uv sync --extra dev` | ### Replacing Hydra overrides with flags The most common Hydra overrides now have dedicated flags: | Legacy Hydra override | New flag | | ------------------------------------------------- | --------------------------------- | | `"+config_paths=[a.yaml,b.yaml]"` | `--config a.yaml --config b.yaml` | | `+agent_name=...` | `--agent` | | `+input_jsonl_fpath=...` | `--input` | | `+input_glob=...` | `--input-glob` | | `+output_jsonl_fpath=...` | `--output` | | `+limit=...` | `--limit` | | `+num_repeats=...` | `--num-repeats` | | `+num_samples_in_parallel=...` | `--concurrency` | | `++split=...` | `--split` | | `++policy_model_name=...` | `--model` | | `++policy_base_url=...` | `--model-url` | | `++policy_api_key=...` | `--model-api-key` | | `++responses_create_params.temperature=...` | `--temperature` | | `++responses_create_params.top_p=...` | `--top-p` | | `++responses_create_params.max_output_tokens=...` | `--max-output-tokens` | | `+mode=...` | `--mode` | | `+output_dirpath=...` | `--output-dir` | | `+should_download=true` | `--download` | | `+prompt_config=...` | `--prompt-config` | | `+resume_from_cache=true` | `--resume` | | `+dataset_name=...` | `--name` | | `+repo_id=...` | `--repo-id` | | `+revision=...` / `+version=...` | `--revision` | | `+artifact_fpath=...` | `--artifact` | | `+output_fpath=...` | `--output` | | `+create_pr=true` | `--create-pr` | | `+outdated=true` | `--outdated` | ### Common workflows, before and after ```bash # Start servers # Before: ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" # After: gym env start \ --resources-server example_single_tool_call \ --model-type openai_model # End-to-end rollout collection # Before: config_paths="responses_api_models/openai_model/configs/openai_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml" ng_e2e_collect_rollouts "+config_paths=[${config_paths}]" \ ++output_jsonl_fpath=results/aime24.jsonl \ ++split=validation # After: gym eval run \ --model-type openai_model \ --resources-server math_with_judge \ --output results/aime24.jsonl \ --split validation # Collect against already-running servers # Before: ng_collect_rollouts +agent_name=example_single_tool_call_simple_agent \ +input_jsonl_fpath=weather_query.jsonl \ +output_jsonl_fpath=weather_rollouts.jsonl \ +num_repeats=4 +num_samples_in_parallel=10 # After: gym eval run --no-serve \ --agent example_single_tool_call_simple_agent \ --input weather_query.jsonl \ --output weather_rollouts.jsonl \ --num-repeats 4 \ --concurrency 10 # Reward profiling # Before: ng_reward_profile +input_jsonl_fpath=materialized_inputs.jsonl +rollouts_jsonl_fpath=rollouts.jsonl # After: gym eval profile \ --inputs materialized_inputs.jsonl \ --rollouts rollouts.jsonl ``` > **Tip** > > Run any command with `--help` to see its full set of flags, and `gym --help` to list every group.