> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # CLI Commands > Reference for the unified gym CLI: command groups, flags, and migration from the legacy ng_* commands. This page documents the NeMo Gym command-line interface. All functionality is exposed through a single `gym` entry point, organized into command groups (`gym `). `ng` is a drop-in alias for `gym`. Every group and command supports `-h`/`--help`. > **Note** > > The legacy `ng_*` / `nemo_gym_*` commands (such as `ng_run` or `nemo_gym_collect_rollouts`) still work but are deprecated. Each one prints a notice pointing at its `gym` replacement and then runs it. See [Migrating from the legacy commands](#migrating-from-the-legacy-commands) at the bottom of this page. ## Quick Reference ```bash # General gym --help # list all command groups gym --version [--json] # print version and system info ng ... # 'ng' is an alias for 'gym' # Discover gym list benchmarks [] [--json] # list benchmarks (or inspect one), by their --benchmark value gym list environments [] [--json] # list the unified environment and benchmark catalog gym list agents [] [--json] # list agent harnesses and how each composes gym list models [] [--json] # list --model-type values (or inspect a model) gym list resources-servers [] [--json] # list --resources-server values (or inspect one) gym search [] [--json] # search the unified environment catalog, or one component type # Datasets gym dataset upload # upload a prepared dataset to HF (default) or GitLab gym dataset download # download a dataset from HF (default) or GitLab gym dataset rm # delete a dataset from GitLab gym dataset migrate # move a dataset from GitLab to HF gym dataset render # generate a dataset preview (materialize prompts) gym dataset collate # validate and collate a dataset # Environments gym env init # scaffold a resources server, environment, or benchmark gym env resolve # resolve and print the final merged config gym env validate # validate a manifest-backed workload or legacy config without services gym env packages # list packages in a server's virtual environment gym env test # run resources-server tests gym env publish # finalize readiness and confirm registry discovery gym env prefetch # pre-warm per-server venvs without starting servers gym env start # start the servers gym env status # show running servers gym env schema # print a resources server's task-data schema as JSON Schema # Evaluation gym eval prepare # prepare benchmark data and dump it to disk gym eval run # collate data, start servers, and collect rollouts gym eval submit # submit a benchmark job to Slurm gym eval aggregate # merge sharded rollout results gym eval export # export supported Gym trajectories as ATIF gym eval health-check # verify rollout artifact quality for an existing run gym eval profile # compute a reward profile from rollouts gym eval reverify # recompute rewards from existing rollouts without re-running inference gym eval compare # compare a baseline eval run against a candidate run # Contributor helpers gym dev test # run NeMo Gym's unit tests ``` ## Common Options These options are shared across many commands. | Option | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--config PATH` | Load a Gym config YAML. Repeatable. Maps to `+config_paths=[...]`. | | `--benchmark NAME` | Select a registered benchmark by name instead of a config path. | | `--resources-server NAME` | Select a registered resources server by name. | | `--model-type NAME` | Select a registered model server type by name (such as `openai_model` or `vllm_model`). | | `--agent-type NAME[/FLAVOR]` | Run the environment with a different agent harness. Refer to [Swapping the agent](#swapping-the-agent). | | `--search-dir DIR` | Extra root directory to search for components and configs. Repeatable. Applies everywhere Gym resolves paths — discovery, the `--` selectors, `config_paths`, prompt configs, and data files — and is inherited by spawned servers. Equivalent to the `NEMO_GYM_EXTRA_ROOTS` environment variable; see [Configuration](/reference/configuration). | | `--json` | Emit machine-readable JSON instead of human-readable output (reporting commands only). Accepted before or after the subcommand; commands that emit no JSON reject it. | | `-v`, `--verbose` | Set the logging level to DEBUG. Flows through to spun-up servers. | | `-h`, `--help` | Show help for any group or command. | > **Tip** > > The `--benchmark`, `--resources-server`, `--model-type`, and `--agent-type` selectors resolve a component name to its config file for you, so you do not need to know the project's directory layout. If you mistype a name, the CLI suggests the closest match. To point at a config file directly, use `--config ` instead. ### Selecting the model server Commands that need a model (`gym env start`, `gym eval run`) configure it through four flags: | Flag | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `--model-type NAME` | The model server type to load (such as `openai_model`, `vllm_model`, or `local_vllm_model`). | | `--model`, `-m` | The served model identifier: an API model name, an HF id, or a local checkpoint path. Interpreted per `--model-type`. Maps to `policy_model_name`. | | `--model-url` | Base URL of an existing model server endpoint. Maps to `policy_base_url`. | | `--model-api-key` | API key for the model server. Maps to `policy_api_key`. | > **Warning** > > `--model-api-key` passed on the command line appears in shell history. Prefer setting `policy_api_key` in your `env.yaml` instead. ### Swapping the agent `--agent-type NAME` runs an environment or benchmark with a different agent harness, without editing any config. It resolves to `responses_api_agents/NAME/configs/NAME.yaml`; use `NAME/FLAVOR` to pick a non-default config in that directory. ```bash # Run Terminal-Bench 2.1 under Terminus 2 instead of the OpenCode harness its config names gym eval run \ --benchmark terminal_bench_2_1/opencode \ --agent-type terminus_2_sandboxed_agent \ --model-type vllm_model ``` Available on `gym env start`, `gym env validate`, `gym env prefetch`, and `gym eval run`. Like the other selectors it is pure shorthand, so `--agent-type terminus_2_sandboxed_agent` and `--config ` behave identically. Run `gym list agents` to see the harnesses you can pass. **The composed server instance is renamed after the agent that runs it**, so metrics and `gym env status` report the harness actually used: composing `terminus_2_sandboxed_agent` onto `terminal_bench_2_1_opencode_sandboxed_agent` produces `terminal_bench_2_1_terminus_2_sandboxed_agent`. `gym env start` prints the instance list, which matters for the two-step flow: ```bash gym env start \ --benchmark terminal_bench_2_1/opencode \ --agent-type terminus_2_sandboxed_agent \ --model-type vllm_model gym eval run --no-serve --agent terminal_bench_2_1_terminus_2_sandboxed_agent --input rows.jsonl ``` > **Note** > > `--agent-type` chooses the harness to compose; `--agent` names the running instance to collect rollouts > from. Because the harness is settled once the servers are up, `--agent-type` is rejected with > `--no-serve`. Anything that routes rows — `--agent`, `+agent_map` values, `+fan_out` entries — must name the composed instance, and Gym says so rather than failing later: ```text Routing names agent instances that no longer exist once the agent is swapped: - agent_name: 'terminal_bench_2_1_opencode_sandboxed_agent' is now 'terminal_bench_2_1_terminus_2_sandboxed_agent' Use the name the composed config reports. ``` Rows collected before the swap are re-routed for you, so existing rollout files stay usable. > **Warning** > > Some tasks only work with the harness they were built for. A resources server may pin the harnesses > required for the task with [`allowed_agents`](/reference/configuration#resources-server-fields), and > Gym refuses a pairing it does not list, before any server starts: > > ```text > 'simple_agent' is not declared compatible with 1 of the agent instance(s) it would replace, > so it cannot be scored correctly: > - terminal_bench_2_1_opencode_sandboxed_agent uses terminal_bench_2_1_opencode_resources_server > and accepts opencode_sandboxed_agent, terminus_2_sandboxed_agent > > Select one of: opencode_sandboxed_agent, terminus_2_sandboxed_agent. Or pass > --allow-unsupported-pairing (or set NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING=1) to bypass the check. > ``` > > Results from an undeclared pairing might not be valid. ### Hydra overrides (escape hatch) The `gym` CLI is a thin wrapper over Gym's Hydra config system. Standard flags (`--flag value`) cover the common inputs. Anything not covered by a flag can still be passed as a raw Hydra override using `+key=value` (add a new key) or `++key=value` (add or override an existing key). Unknown overrides are forwarded to Hydra untouched, so advanced config composition is fully functional. When a command needs overrides for keys that have no dedicated flag, you can keep the whole command in Hydra form (config paths included) or mix `--flag` and `+key=value` styles in one invocation: ```bash gym eval run \ --benchmark aime24 \ --model-type openai_model \ ++responses_create_params.reasoning.effort=low \ +wandb_project=gym-dev ``` --- ## General ### `gym --help` List all command groups and their descriptions. ```bash gym --help ``` ### `gym --version` Print the NeMo Gym version along with Python, key dependency, and system information. | Option | Description | | -------- | ----------------------------------- | | `--json` | Output version information as JSON. | ```bash gym --version # Output as JSON gym --version --json ``` --- ## Discovery Commands for discovering benchmarks, environments, model servers, agent harnesses, and resources servers. `gym list ` lists all components of that type; `gym list ` inspects that one. All commands accept: * `--json` which prints machine-readable output instead of a human-readable table. * `--search-dir` which allows you to surface your own Gym-compatible components alongside the built-ins. ### `gym list benchmarks` List the benchmarks available in NeMo Gym, with their kind, manifest status, domain, and description. The value in the **Name** column is the benchmark identifier and can be passed to `--benchmark`. To see which agent harness a benchmark composes with, inspect it with `gym list benchmarks `. | Option | Description | | ---------- | -------------------------------------------------------------------------------------------------- | | `[]` | Inspect a single benchmark (config path, manifest status, agent, datasets) instead of listing all. | | `--json` | Output the benchmark list as JSON. | ```bash gym list benchmarks gym list benchmarks aime24 # inspect one benchmark # Machine-readable output for scripting gym list benchmarks --json | jq '.[].name' ``` ### `gym list environments` List the manifest-first catalog unioned with legacy environment and benchmark configs. Manifest-backed entries expose their declared metadata; unmigrated entries remain visible with `no-manifest` status. | Option | Description | | --------------------------------------- | ---------------------------------------------------------- | | `[]` | Inspect one environment or benchmark. | | `--kind environment\|benchmark` | Filter the list or resolve an ambiguous name. | | `--domain`, `--modality`, `--licensing` | Filter by catalog metadata. | | `--status experimental\|no-manifest` | Filter by the experimental annotation or missing manifest. | | `--lifecycle active\|deprecated` | Filter by lifecycle. | | `--json` | Output catalog entries as JSON. | ```bash gym list environments gym list environments calendar # inspect one environment gym list environments --kind benchmark --domain math gym list environments --json | jq '.[].name' ``` ### `gym list agents` List the agent harnesses under `responses_api_agents/`, with each one's composition pattern — **composable** (Pattern A: references a separate resources server, so it can be wired into a matching environment) vs **self-contained** (Pattern B: ships its own framework/environment) — plus its config variants. | Option | Description | | ---------- | ------------------------------------------------------------------- | | `[]` | Inspect a single agent (composition pattern and config variants). | | `--json` | Output the agent list as JSON (includes the `self_contained` flag). | ```bash gym list agents gym list agents simple_agent # inspect one agent gym list agents --json | jq '.[] | {name, self_contained}' ``` ### `gym list models` List the model servers under `responses_api_models/`. The value in the **Model** column corresponds to a config flavor and can be passed as the `--model-type` flag to other commands. | Option | Description | | ---------- | ---------------------------------------------- | | `[]` | Inspect a single model instead of listing all. | | `--json` | Output the rows as JSON. | ```bash gym list models gym list models vllm_model # inspect one model gym list models --json | jq '.[].model' ``` ### `gym list resources-servers` List the resources servers under `resources_servers/`. The value in the **Name** column corresponds to the config flavor and can be passed as `--resources-server`. | Option | Description | | ---------- | --------------------------------------------------------------------- | | `[]` | Inspect a single resources server (config path, domain, description). | | `--json` | Output the list as JSON. | ```bash gym list resources-servers gym list resources-servers mcqa # inspect one resources server gym list resources-servers --json | jq '.[].name' ``` ### `gym search` Filter a listing to fuzzy matches on a query (against each component's name/value, description, domain, and type-specific fields). Without a type, the command searches the unified environment and benchmark catalog. | Option | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `[]` | Optional component type: `benchmarks`, `environments`, `agents`, `models`, or `resources-servers`. | | `QUERY` | Text matched (substring or fuzzy) against a component's name, description, and key metadata (positional, required). | | `--json` | Output matches as JSON. | ```bash gym search math # environments and benchmarks matching "math" gym search benchmarks math # only benchmarks matching "math" gym search environments calendar # environments matching "calendar" gym search models vllm --json ``` --- ## Datasets Commands for preparing, previewing, and managing datasets. By default dataset transfer commands use HuggingFace; pass `--storage gitlab` to target the GitLab Registry. > **Note** > > HuggingFace commands require an `HF_TOKEN` environment variable (or `huggingface-cli login`). GitLab commands authenticate via `mlflow_tracking_token` in your YAML config (or the `MLFLOW_TRACKING_TOKEN` environment variable). See [Downloading from HuggingFace](/data/download-huggingface) for HF authentication details. ### `gym dataset upload` Upload a prepared local JSONL dataset to HuggingFace (default) or GitLab. | Option | Description | | ----------------------- | ------------------------------------------------ | | `--storage {hf,gitlab}` | Storage backend. Default: `hf`. | | `--input`, `-i` | Local JSONL file to upload. | | `--name` | Dataset name. | | `--revision` | Dataset revision (version). | | `--split` | Dataset split (HF only). | | `--create-pr` | Open a pull request with your changes (HF only). | ```bash # Upload to HuggingFace gym dataset upload \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 # Upload to GitLab gym dataset upload \ --storage gitlab \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 ``` ### `gym dataset download` Download a dataset from HuggingFace (default) or GitLab. | Option | Description | | ----------------------- | -------------------------------------------------------------------------- | | `--storage {hf,gitlab}` | Storage backend. Default: `hf`. | | `--repo-id` | HF repo id, such as `org/dataset` (HF only). | | `--name` | Dataset name (GitLab only). | | `--revision` | Dataset version (GitLab only). | | `--artifact` | Remote file to fetch (GitLab: required; HF: optional raw file). | | `--output`, `-o` | Local destination file. | | `--output-dir` | Local destination directory; needed when downloading all splits (HF only). | | `--split` | Dataset split (HF only). | ```bash # Download a single file from HuggingFace gym dataset download --repo-id NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \ --artifact train.jsonl \ --output data/train.jsonl # Download from GitLab gym dataset download \ --storage gitlab \ --name example_multi_step \ --revision 0.0.1 \ --artifact train.jsonl \ --output data/train.jsonl ``` ### `gym dataset rm` Delete a dataset from the GitLab Registry. Prompts for confirmation. | Option | Description | | -------- | ------------------------------ | | `--name` | Name of the dataset to delete. | ```bash gym dataset rm --name old_dataset ``` ### `gym dataset migrate` Migrate a JSONL dataset to HuggingFace from GitLab. Use `gym dataset upload` if you do not want automatic GitLab deletion. | Option | Description | | --------------- | ---------------------------------------------------- | | `--input`, `-i` | Local JSONL file to upload to HF. | | `--name` | Dataset name. | | `--revision` | Dataset revision (HF). | | `--split` | Dataset split. | | `--create-pr` | Open a pull request to HF dataset with your changes. | ```bash gym dataset migrate \ --name my_dataset \ --input data/train.jsonl \ --revision 0.0.1 ``` ### `gym dataset collate` Validate and collate a dataset, generating metrics and statistics. | Option | Description | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model-type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--mode {train_preparation,example_validation}` | Use `train_preparation` to prepare train/validation datasets, or `example_validation` to validate example data. | | `--output-dir` | Output directory for the prepared data. | | `--download` | Download source datasets before collating. | ```bash gym dataset collate \ --resources-server example_multi_step \ --output-dir data/example_multi_step \ --mode example_validation ``` ### `gym dataset render` Generate a dataset preview by materializing prompts from a raw input file and a prompt template, producing JSONL with populated `responses_create_params.input` for RL training. Each input row must **not** already have a populated `responses_create_params.input`; the command applies the prompt template from `--prompt-config` to each row, fills in the input, and preserves the row's other fields. | Option | Description | | ------------------ | -------------------------------------------------------------------- | | `--input`, `-i` | Raw input JSONL file (rows without `responses_create_params.input`). | | `--prompt-config` | Prompt template YAML to apply. | | `--output`, `-o` | Output JSONL file. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym dataset render \ --input raw.jsonl \ --prompt-config prompt.yaml \ --output preview.jsonl ``` > **Note** > > **Which data-preparation command should I use?** > > * **`gym dataset render`** — a focused, standalone step that applies a prompt template to raw rows to populate `responses_create_params.input`. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows. > * **`gym dataset collate`** — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission. --- ## Environments Commands for developing, running, and inspecting environments. An environment defines tasks, verification, and its interaction surface; Gym config composes it with agent, model, and runtime components. ### `gym env init` Scaffold a resources server, or create a manifest-backed environment or benchmark skeleton. Workload scaffolds include a manifest, Gym config, sample data, and the extension points selected by the integration profile. | Option | Description | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `--resources-server NAME` | Name of the resources server to create. | | `--environment NAME` / `--benchmark NAME` | Create a manifest-backed environment or benchmark. | | `--profile PROFILE` | Integration profile: `custom-gym-verifier`, `custom-gym-agent-loop`, `external-agent-loop`, or `external-rollout-driver`. | | `--reuse-verifier NAME` | Reuse an existing resources server that exports `VERIFIER_FIXTURE`. | | `--reward-range LOW HIGH` | Reward endpoints required when reusing a verifier. | | `--higher-is-better` / `--lower-is-better` | Reward direction required when reusing a verifier. | ```bash gym env init --resources-server my_server gym env init --environment my_eval --profile custom-gym-verifier gym env init --benchmark my_benchmark --reuse-verifier existing_scorer \ --reward-range 0 1 --higher-is-better ``` ### `gym env resolve` Resolve the configs, flags, and overrides into a final merged config and print it. Useful for debugging configuration. Secrets are hidden. Unlike `gym env validate`, `resolve` substitutes no dummy model, so a model config's `policy_*` values must be supplied for its interpolations to resolve. | Option | Description | | ------------------ | ---------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash # Merge a resources-server config with a model config, apply an override, and print the result gym env resolve \ --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml \ --config responses_api_models/openai_model/configs/openai_model.yaml \ ++responses_create_params.temperature=0.6 \ ++policy_base_url=https://api.openai.com/v1 \ ++policy_api_key=sk-example \ ++policy_model_name=gpt-4o-mini ``` ### `gym env validate` Validate a manifest-backed workload or legacy config without starting Ray or server subprocesses. Manifest validation resolves the Gym config offline, checks composition mirrors and datasets, and reports the resolved components. Legacy validation retains the existing config pre-flight behavior. Exits `0` when valid, or `1` with a clean message when not. By default, values a run needs but the config leaves undefined (such as `policy_base_url` from `env.yaml`) are filled with placeholders so the rest can still be checked, and reported as a warning. Pass `--strict` to fail on them instead. | Option | Description | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `NAME` | Manifest-backed catalog entry to validate. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--manifest PATH` | Validate a manifest directly instead of selecting by name. | | `--sync` | Atomically synchronize config-owned composition mirrors after all checks pass. | | `--strict` | Resolve the config exactly as a run would, with no placeholder model values. An undefined value fails instead of warning. | | `--json` | Output the manifest validation report as JSON. | | `--config PATH` | Config file to load. Repeatable. | | `--environment NAME` / `--benchmark NAME` | Validate a named environment / benchmark config. | | `--resources-server NAME` | Validate a named resources server config. | | `--model-type NAME` | Also load a named model config (otherwise a dummy `policy_model` is used). | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Refer to [Swapping the agent](#swapping-the-agent). | | `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. | | `--model` / `--model-url` / `--model-api-key` | Override model name, base URL, and API key. | ```bash gym env validate my_eval gym env validate --sync my_eval gym env validate --environment workplace_assistant gym env validate --benchmark gsm8k # or explicit config path(s) gym env validate --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml # fail unless everything a run needs is defined, credentials included gym env validate --resources-server example_single_tool_call --model-type openai_model --strict ``` ### `gym env packages` Each server has its own isolated virtual environment. List the packages installed in a server's environment. | Option | Description | | ------------------------- | ---------------------------------------------------------------- | | `--resources-server NAME` | Name of the resources server. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--outdated` | List only outdated packages. | | `--json` | Output the package list as JSON. | ```bash gym env packages --resources-server example_single_tool_call # Check for outdated packages gym env packages \ --resources-server example_single_tool_call \ --outdated ``` ### `gym env test` Exercise a manifest-backed workload's verifier fixture in its resources-server environment, or run the existing pytest suite for one or all resources servers. | Option | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------- | | `NAME` | Manifest-backed catalog entry whose verifier fixture should run. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--update-expected` | Atomically update fixture rewards after every behavioral check passes. | | `--json` | Output the verifier report as JSON. | | `--resources-server NAME` | Resources server to test. | | `--all` | Test every resources server. Slow: builds one venv per server, so it must be requested explicitly. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash # Exercise a manifest-backed workload verifier gym env test my_eval # Test a single server gym env test --resources-server example_single_tool_call # Test all servers. Slow: builds one venv per server, so it needs an explicit opt-in. gym env test --all ``` ### `gym env publish` Run the local publication check for a manifest-backed workload. The command runs static validation and the verifier fixture, rejects manifest metadata placeholders, and confirms that the exact manifest is discoverable. It includes an `experimental` annotation only when the manifest's flag is `true`. It does not certify the workload, assess implementation quality, require a full evaluation or training run, commit, or push changes. | Option | Description | | ------------------------------- | ---------------------------------------------------------------- | | `NAME` | Manifest-backed catalog entry to publish. | | `--kind environment\|benchmark` | Resolve an ambiguous catalog name. | | `--json` | Output the publication result as JSON. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym env publish my_eval gym env publish my_benchmark --kind benchmark --json ``` ### `gym env start` Start the NeMo Gym servers (agents, models, resources, and environment servers) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment. Rollout collection dispatches `/run` through environment servers. | Option | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config (start its servers). | | `--environment NAME` | Load the named environment config. Repeatable. | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model-type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Refer to [Swapping the agent](#swapping-the-agent). | | `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. | | `--model`, `-m` | Served model identifier. See [Selecting the model server](#selecting-the-model-server). | | `--model-url` | Model server base URL. | | `--model-api-key` | Model server API key. | ```bash gym env start \ --resources-server example_single_tool_call \ --model-type openai_model # Start a benchmark's servers gym env start \ --benchmark gpqa \ --model-type vllm_model ``` ### `gym env status` Show all currently running NeMo Gym servers and their health. | Option | Description | | -------- | ------------------------------- | | `--json` | Output the server list as JSON. | ```bash gym env status ``` ``` NeMo Gym Server Status: [1] ✓ example_single_tool_call (resources_servers/example_single_tool_call) { 'server_type': 'resources_servers', 'name': 'example_single_tool_call', 'port': 58117, 'pid': 89904, 'uptime_seconds': '0d 0h 0m 41.5s', } ... 3 servers found (3 healthy, 0 unhealthy) ``` ### `gym env prefetch` Pre-warm per-server venvs without starting any servers. Accepts the same config flags as `gym env start`. For each server in the config, creates its isolated venv and installs dependencies serially. Intended for use in Dockerfile builds so venvs are ready at runtime without network access. | Option | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. Repeatable. | | `--environment NAME` | Load the named environment config. Repeatable. | | `--resources-server NAME` | Load the named resources server config. Repeatable. | | `--model-type NAME` | Load the named model-type config. | | `--agent-type NAME[/FLAVOR]` | Compose with the named agent harness. | | `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash config_paths="responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,\ resources_servers/math/configs/math.yaml" gym env prefetch "+config_paths=[${config_paths}]" ``` ### `gym env schema` Print a resources server's `TaskData` schema as JSON Schema. Exits non-zero if the server does not export a `task_data.py` schema. | Option | Description | | ------------------------- | ---------------------------------------------------------------- | | `--resources-server NAME` | Resources server to inspect. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym env schema --resources-server example_multi_step ``` --- ## Evaluation Commands for running evaluations end to end: prepare data, collect rollouts, aggregate sharded runs, verify rollout health, and profile results. ### `gym eval prepare` Prepare a benchmark's data by running its `prepare.py` script and dump the result to disk. | Option | Description | | ------------------ | ---------------------------------------------------------------- | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | ```bash gym eval prepare --benchmark aime24 ``` ### `gym eval run` Collate data, start the servers, and collect rollouts. This is the main evaluation command. By default it spins up all required servers. Pass `--no-serve` to collect against servers you already started with `gym env start`. | Option | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--config PATH` | Config file to load. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--environment NAME` | Load the named environment config. Repeatable. | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model-type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--no-serve` | Collect against already-running servers instead of starting them. | | `--resume` | Resume from cached rollouts instead of recollecting. Maps to legacy `+resume_from_cache=true`. Refer to [Resume interrupted runs](#resume-interrupted-runs). | | `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Rejected with `--no-serve`. Refer to [Swapping the agent](#swapping-the-agent). | | `--agent`, `-a` | Agent instance to collect rollouts with. | | `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. | | `--input`, `-i` | Input tasks JSONL file. | | `--output`, `-o` | Output rollouts JSONL file. | | `--limit` | Maximum number of tasks to run. | | `--num-repeats` | Rollouts per task (for mean\@k metrics). Pass an int to apply to every task, or a dict keyed by `agent_ref.name` for per-agent counts (e.g. `'{simple_agent: 32, swe_agent: 1}'`) when one input file mixes agents. In dict form, the special key `_default` is the fallback for agents not explicitly listed; without it, any unlisted row's agent raises a single consolidated error. | | `--interleave-repeats` | Start repeats round by round (every task's first rollout, then every task's second) rather than each task's back to back. Use it when a task is heavy on the machine running it, for example when its sandbox loads large data, to spread its repeats over the run. This is best effort: repeats still overlap when the concurrency is high or rollouts are long. GDPVal's multistage orchestrator rebuilds its stage rows per task and does not keep this order. Maps to legacy `+interleave_repeats=true`. | | `--prompt-config` | Prompt template YAML to apply. | | `--concurrency` | Maximum number of concurrent samples. Must be positive when set. | | `--split` | Dataset split to use (`train`, `validation`, or `benchmark`). A dataset of the matching `type` must be declared in the loaded configs. `example` datasets are smoke-test samples, not a runnable split — run them with `--no-serve` and `--input` as shown in the [Quickstart](/get-started/quickstart). | | `--model`, `-m` | Served model identifier. | | `--model-url` | Model server base URL. | | `--model-api-key` | Model server API key. | | `--temperature` | Sampling temperature. | | `--top-p` | Nucleus sampling top-p. | | `--max-output-tokens` | Maximum output tokens. | | `--disable-aggregation` | Skip aggregate metrics and the automatic health check. Use for shards that will be combined with `gym eval aggregate`. | | `--no-health-check` | Skip the automatic post-run health check. | | `--health-check-workers` | Number of rollout-health worker processes. Defaults to the smaller of the CPU count and 8. | | `--health-check-ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | ```bash # End-to-end: spin up servers, then collect rollouts for a benchmark gym eval run --benchmark aime24 \ --model-type openai_model \ --output results/aime24.jsonl \ --split validation \ --concurrency 10 # Two-terminal sequence for --no-serve: model settings go to env start, not eval run. # Model URL and credentials configure servers at start time; passing them during # collection has no effect on already-running servers. # Terminal 1 — start servers with the model endpoint configured: gym env start \ --resources-server math_with_judge \ --model-type openai_model \ --model openai/gpt-oss-120b \ --model-url http://127.0.0.1:10240/v1 \ --model-api-key dummy_key # Terminal 2 — collect rollouts against the running servers: gym eval run --no-serve \ --resources-server math_with_judge \ --output results/test_001.jsonl \ --split validation \ --temperature 1.0 \ --top-p 1.0 # Per-agent repeats: one input file pins different agents per row via agent_ref.name gym eval run --no-serve \ --input mixed_agents.jsonl \ --output results/mixed_rollouts.jsonl \ --num-repeats '{agent_alpha: 4, agent_beta: 1, _default: 1}' ``` #### Generation parameters The most common sampling parameters have dedicated flags on `gym eval run`: `--temperature`, `--top-p`, and `--max-output-tokens`. These map onto `responses_create_params.temperature`, `responses_create_params.top_p`, and `responses_create_params.max_output_tokens`. Any other `responses_create_params` field that has no dedicated flag can be set with a raw Hydra override using the `++responses_create_params.` syntax. Overrides are merged into each input row's existing `responses_create_params` with a **shallow** merge (top-level keys only): ```bash gym eval run --no-serve \ --agent example_single_tool_call_simple_agent \ --input weather_query.jsonl \ --output weather_rollouts.jsonl \ --temperature 1.0 \ --top-p 1.0 \ --max-output-tokens 4096 \ ++responses_create_params.reasoning.effort=low ``` > **Tip** > > Because the merge is shallow, setting a field inside a nested object, such as `++responses_create_params.reasoning.effort=low`, replaces the row's entire nested dictionary at that key. Other fields under the same nested object are not preserved. #### Resume interrupted runs Pass `--resume` to restart the **same command** after a crash or interruption and pick up only the rows that have not finished yet. How it works: * **Materialized inputs.** On the first run, the fully expanded input rows (after `--num-repeats`, `--limit`, `--prompt-config`, and any overrides) are written to a sidecar file next to your output. The path is derived from `--output` by appending `_materialized_inputs` to the stem — so `rollouts.jsonl` produces `rollouts_materialized_inputs.jsonl`. * **Incremental output.** Successful rollouts are flushed to the main output JSONL after each completion; retriable failures go to a `_failures.jsonl` sidecar, so partial progress survives a crash. * **Failed agent calls.** By default a failed agent `/run` ends the run, because the alternative is a score reported over fewer rollouts than were asked for. Pass `+route_failures_to_sidecar=true` to let the run continue instead: the call is then recorded in the same sidecar as one attempt, with no reward. Every rollout that leaves the score is logged as it happens, whichever layer classified it, and the totals appear with the closing artifact paths. The class says whether the rollout ran: `agent_run_error` when the agent itself answered, since a NeMo Gym agent returns 500 when its handler raises, and `agent_request_failed` for a gateway status (429, 502, 503, 504) or no usable reply, neither of which says anything about the rollout. Other rollouts keep running, the score is computed over completed rollouts only, and resume re-dispatches the row up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. A run in which nothing produced a result fails instead of reporting an empty score, unless an option below counts its rollouts as zeros. * **Counting failures in the score.** Metrics are computed over completed rollouts only. `+count_failure_classes_as_zero=[]` includes the named sidecar rows in `/aggregate_metrics` so they land in the denominator: `[agent_run_error]` counts the rollouts that reached the agent and broke, which is what a model emitting unparsable tool calls looks like from here, and needs `+route_failures_to_sidecar=true` to have anything to count. A row that carries no reward is scored zero for the metric input only, so neither the sidecar nor the rollouts JSONL records a verdict that no verifier gave. It works the same in `gym eval run` and `gym eval aggregate`. * **Counting lost rollouts.** `+count_missing_rollouts_as_zero=true` scores as zero a rollout that is in the materialized inputs but left no row in the output JSONL or the sidecar, such as one killed mid-flight or one `dispatch_budget_s` never started. The zero goes into the metric input only, so `--resume` still runs the rollout and the resumed run scores it. A rollout recorded in the sidecar is never counted here, so this does not widen `+count_failure_classes_as_zero`. The count is reported as `coverage/imputed`. With `--disable-aggregation` nothing is scored, so a run with no row still fails. `gym eval aggregate` checks each shard against its own rows, materialized inputs and sidecar. * **Matching.** On resume, completed work is matched by `(task_index, rollout_index)` against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run. * **Terminal failures.** A sidecar row stamped `_ng_failure_terminal` is never retried. `+retry_terminal_timeouts=true` retries rows of class `timeout_exceeded` even when they are stamped terminal, and does the same for the repairable environment faults `reference_missing`, `eval_missing` and `transport_ineligible`; `skipped` rows are never retried. Retries still stop at `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. Off by default, because some agents stamp their timeouts terminal on purpose. * **Invalid judge responses.** `+retry_invalid_judge_responses=true` moves rows flagged `invalid_judge_response` out of the output JSONL on resume and into the sidecar as retryable `judge_invalid` attempts, so they leave the score and are judged again. A row whose deliverable is still on disk is marked `reuse_cached_deliverable`, so the agent can judge it again without rerunning the policy. Off by default, because some verifiers score an invalid judge response as zero on purpose. * **Dispatch budget.** `+dispatch_budget_s=` stops starting new rollouts that many seconds after collection starts; set it to the usable walltime of the allocation. It requires `--concurrency`. Rows that are not started write nothing, so the next `--resume` runs them. `+drain_margin_s=` also stops starting rows when less than that much time is left; unset, the margin is the run's own p75 rollout duration once 5 rollouts have finished. Every run ends with a per-rollout latency summary (median, p90, p99, max) and the number of rows not started. * **Longest first.** `+dispatch_longest_first=true` starts the rows whose earlier failed attempts ran longest first on resume, using the `elapsed_seconds` those attempts recorded. Rows with no recorded timing keep their input order and start after them. Without it, rows start in input order. * **Fallback.** If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. Without `--resume`, the existing output, its `_failures.jsonl` sidecar and its `_aggregate_metrics.json` are deleted before the run. > **Warning** > > If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the `*_materialized_inputs.jsonl` file (and the output file) to start fresh. ### `gym eval submit` Submit a benchmark evaluation job to Slurm via a submit config YAML file. Renders job scripts and calls `sbatch` for each benchmark defined in the config. Use `--dry-run` to preview the generated scripts without submitting. | Option | Description | | -------------------------- | -------------------------------------------------------------------------------------------------------- | | `--config PATH`, `-c PATH` | Submit config YAML file. Required. | | `--dry-run` | Print generated job scripts without submitting. | | `--resolve-only` | Resolve and validate the config, print it, and stop before rendering job scripts or submitting anything. | | `--json` | Emit the submission record (or, with `--resolve-only`, the resolved config) as JSON. | ```bash gym eval submit --config submit.yaml gym eval submit --config submit.yaml --dry-run gym eval submit --config submit.yaml --resolve-only --json ``` ### `gym eval aggregate` Merge sharded rollout results into a single rollouts file with aggregate metrics. Reads every JSONL file matching `--input-glob`, recomputes aggregate metrics over the global union of records, and writes a `_aggregate_metrics.json` next to the merged rollouts. Use this to combine shards produced by `gym eval run --no-serve --disable-aggregation`. | Option | Description | | ------------------------ | ------------------------------------------------------------------------------------------ | | `--config PATH` | Config file to load. Repeatable. | | `--input-glob`, `-i` | Glob (or comma-separated globs) matching the rollout shards to aggregate. | | `--output`, `-o` | Path for the merged rollouts and aggregate-metrics file. | | `--no-health-check` | Skip the automatic post-aggregation health check. | | `--health-check-workers` | Number of rollout-health worker processes. Defaults to the smaller of the CPU count and 8. | | `--health-check-ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | ```bash gym eval aggregate \ --config benchmarks/aime24/config.yaml \ --config responses_api_models/vllm_model/configs/vllm_model.yaml \ --input-glob 'results/rollouts-rs*-chunk*.jsonl' \ --output results/rollouts.jsonl ``` ### `gym eval export` Export supported `ng_trajectory` attachments from a Gym rollouts JSONL file as ATIF v1.7. The command writes one ATIF trajectory per rollout plus a manifest that preserves Gym's task and rollout identity. It is a strict offline conversion: if Gym cannot represent a source trajectory completely in the supported ATIF subset, the export fails instead of silently omitting data. All invalid JSONL rows are reported together, and no destination is published if any row fails. Captured model attempts not selected by a turn, such as failed retries, remain in `extra.nemo_gym.surplus_model_calls`; they do not become canonical ATIF steps or contribute to final usage totals. The destination directory must not already exist. This command does not parse ATIF and does not load NeMo Relay. Each ATIF trajectory takes its agent name from the source row's `agent_ref.name`; `--agent-version` records the version of that agent implementation. | Option | Description | | ------------------------- | ------------------------------------------------------------------------ | | `--format atif` | Output format. ATIF is the only format supported initially. | | `--rollouts PATH` | Gym rollouts JSONL containing `ng_trajectory` version `1.0` attachments. | | `--output-dir DIR` | New directory for the ATIF files and manifest. | | `--session-id ID` | Stable identifier for the source evaluation run. | | `--agent-version VERSION` | Version of the agent implementation that produced the rollouts. | ```bash gym eval export \ --format atif \ --rollouts results/rollouts.jsonl \ --output-dir results/atif \ --session-id eval-2026-08-25 \ --agent-version 1.2.3 ``` See the [trajectory capability matrix](/reference/trajectory-capabilities#atif-export) for the initial conversion boundary. ### `gym eval health-check` Verify rollout quality for an existing run directory. The command reads `/rollouts.jsonl` by default and writes `quality_summary.json` and `rollout_verdicts.jsonl` in the run directory. | Option | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `RUN_DIR` | Directory in which to write the health reports. | | `--rollouts-file PATH` | Rollout JSONL path. Relative paths resolve under `RUN_DIR`; absolute paths are used as written. Defaults to `rollouts.jsonl`. | | `--workers` | Number of worker processes. Defaults to the smaller of the CPU count and 8. | | `--ignore-checks`, `--ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. | | `--json` | Output health-check results as JSON. | ```bash gym eval health-check results/my-run # Select a nonstandard rollout filename and exclude one known check gym eval health-check results/my-run \ --rollouts-file evaluator_rollouts.jsonl \ --ignore-checks model_call_missing_token_counts ``` Refer to [Rollout Health Checks](/main/evaluation/rollout-health) for verdict semantics, evidence requirements, the complete check catalog, and report schemas. ### `gym eval profile` Compute a reward profile from collected rollouts. Outputs per-task statistics such as average reward, standard deviation, min/max, and pass rate, useful for filtering tasks before training by difficulty or variance. Requires rollouts collected with `--num-repeats` greater than 1. | Option | Description | | ------------ | ---------------------------------------------------- | | `--inputs` | Materialized inputs JSONL fed to rollout collection. | | `--rollouts` | Rollouts JSONL produced by collection. | ```bash gym eval profile \ --inputs materialized_inputs.jsonl \ --rollouts rollouts.jsonl ``` Writes three files next to the rollouts, named from its stem: | File | Contents | | -------------------------------------- | --------------------- | | `_reward_profiling.jsonl` | One row per task. | | `_agent_metrics.json` | One entry per agent. | | `_repeat_level_metrics.json` | One entry per repeat. | See [Repeat-Level Metrics](/evaluation/aggregate-metrics#repeat-level-metrics) for the full field list and how to read the variability statistics. ### `gym eval compare` Compare a baseline eval run against a candidate run and write a report. The report contains a table of point-estimate metrics (the change, plus each side's value and confidence interval), split into key metrics and all other metrics, and a per-task sample-flips table. Existing statistics such as percentile and confidence interval fields do not get separate rows. Confidence intervals are read from what each run recorded; this command does not yet perform statistical tests or issue a pass/fail verdict. | Option | Description | | ----------------------------------------- | ------------------------------------------------------------------------------------------------- | | `--baseline PATH` | Baseline run's rollouts JSONL. Its `*_aggregate_metrics.json` sibling is what gets read. | | `--candidates PATH[,PATH...]` | Candidate run's rollouts JSONL. Comma-separated; one candidate is supported today. | | `--baseline-agg-metrics PATH` | Baseline's aggregate-metrics JSON, when it is not the sibling of `--baseline`. | | `--candidates-agg-metrics PATH[,PATH...]` | Candidates' aggregate-metrics JSON, in `--candidates` order, when not siblings of `--candidates`. | | `--agent NAME` | Agent to compare on both sides. Defaults to every agent present in both runs. | | `--baseline-agent NAME` | Agent to read from the baseline. Takes precedence over `--agent`. | | `--candidate-agents NAME[,NAME...]` | Agent to read from each candidate, in `--candidates` order. Takes precedence over `--agent`. | | `--output-dir DIR`, `-o` | Where to write the report. Defaults to the directory containing the candidate rollouts JSONL. | | `--report-format` | `md`, `json`, or `both` (default). | ```bash gym eval compare \ --baseline runs/baseline/rollouts.jsonl \ --candidates runs/candidate/rollouts.jsonl ``` Writes into the output directory: | File | Contents | | --------------------- | ------------------------------------------------- | | `compare_report.md` | Human-readable report. | | `compare_report.json` | Machine-readable result (`schema_version` `"1"`). | If a run was collected with `--disable-aggregation` it has no `*_aggregate_metrics.json`; run [`gym eval aggregate`](#gym-eval-aggregate) first, or point at the file directly with `--baseline-agg-metrics` / `--candidates-agg-metrics`. ### `gym eval reverify` Recompute rewards by replaying a stored response through a resources server's `/verify` endpoint — without re-running model inference. The command accepts native Gym rollouts or the initial completed, text-only Relay ATIF v1.7 profile. It starts the resources server automatically from the provided config. Before starting, the command checks each resources server's `GET /reverify_mode` response. Resources servers report one of three modes: | Mode | Meaning | | ------------- | ----------------------------------------------------------------------------------------------------------------- | | `stateless` | Reverification is safe. The verifier is a pure function of `(request body, server config)`. | | `unsupported` | Reverification is not safe. The verifier reads per-rollout session state that is no longer present. | | `unknown` | The server has not declared its reverification behaviour. **This is the default.** Treated as potentially unsafe. | Servers that report `unsupported` or `unknown` cause the command to abort unless `--force` is passed. | Option | Description | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--config PATH` | Config file for the resources server. Repeatable. | | `--benchmark NAME` | Load the named benchmark config. | | `--environment NAME` | Load the named environment config. Repeatable. | | `--resources-server NAME` | Load the named resources server config. | | `--model-type NAME` | Load the named model-type config. | | `--search-dir DIR` | Extra root directory to search for named components. Repeatable. | | `--input-format gym\|atif` | Select native Gym rollout input or the bounded Relay ATIF v1.7 profile. Defaults to `gym`. | | `--inputs PATH` | `*_materialized_inputs.jsonl` from `gym eval run`. | | `--rollouts PATH` | Native Gym `rollouts.jsonl`. Required when `--input-format gym`. | | `--atif-manifest PATH` | JSONL manifest mapping each Relay ATIF trajectory to `_ng_task_index` and `_ng_rollout_index`, with an optional source SHA-256. Missing hashes produce a warning. Required when `--input-format atif`. | | `--output PATH`, `-o` | Output JSONL for recomputed rollouts. | | `--concurrency` | Maximum number of concurrent samples. | | `--limit` | Maximum number of examples to re-verify. | | `--force` | Override the `UNSUPPORTED` or `UNKNOWN` reverify mode guard; output is prefixed with `unsafe_`. | | `--overwrite` | Delete an existing output file instead of raising an error. | | `--resume` | Resume a partial run: skip rows already in the output and re-verify only the rest. | | `--judge-failed-only` | Recover only the rollouts whose judge call failed or returned an invalid verdict in the original run (`judge_failed` and `judge_invalid` rows in `_failures.jsonl`), reusing the stored responses. Skips the reverify-mode guard. Rejects multi-stage rows (`stage_index`) before writing anything; resume the multi-stage collection instead. | | `--append` | With `--judge-failed-only`: append recovered rows to an existing `--output` instead of a fresh file. Mutually exclusive with `--overwrite`. | | `--disable-aggregation` | Skip aggregate-metrics computation (use when combining shards with `gym eval aggregate` afterward). | Native Gym input also accepts the rollout collection options `+retry_terminal_timeouts=true` (with `--resume`) and `+retry_invalid_judge_responses=true`. With the second, a verifier result flagged `invalid_judge_response` goes to the output's failures sidecar as `judge_invalid` (`permanent` and terminal when it also sets `invalid_judge_retryable=false`) instead of being scored, and `--judge-failed-only` first moves such rows out of `--rollouts` into its sidecar, rewriting `--rollouts`. When a rollout row carries `deliverables_dir` or `reference_ids`, both are sent to `/verify` with the stored response. ATIF input requires a `stateless` resources server and rejects `--force`, `--resume`, `--judge-failed-only`, and `--append`. It fails closed on unsupported or lossy canonical trajectory shapes, including tool-bearing trajectories routed to a resources server with `expose_tools_over_mcp: true`, and preserves source identity under `_ng_atif_provenance` without sending that metadata to the verifier. Provider-native payloads and producer-private status records in optional ATIF `extra` metadata are not interpreted; their producer owns conversion into canonical ATIF fields. A terminal tool step is supported when every call has a correlated observation result. ```bash # Recompute rewards with an updated verifier config (no model server needed) gym eval reverify \ --config my_resources_server.yaml \ "++head_server.host=127.0.0.1" \ "++head_server.port=9500" \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_reverified.jsonl # Override a verifier hyperparameter inline gym eval reverify \ --config my_resources_server.yaml \ "++mcqa.resources_servers.mcqa.grading_mode=lenient_boxed" \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_lenient.jsonl # Score a Relay ATIF v1.7 trajectory against a materialized Gym task gym eval reverify \ --config my_resources_server.yaml \ --input-format atif \ --inputs results/materialized_inputs.jsonl \ --atif-manifest relay-atif/manifest.jsonl \ --output results/relay_atif_reverified.jsonl # Re-run the judge on only the rollouts whose judge failed (reuses stored responses; no inference) gym eval reverify --judge-failed-only \ --config my_resources_server.yaml \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_recovered.jsonl # Force through an UNSUPPORTED or UNKNOWN server (output prefixed with unsafe_) gym eval reverify \ --config my_stateful_server.yaml \ --inputs results/rollouts_materialized_inputs.jsonl \ --rollouts results/rollouts.jsonl \ --output results/rollouts_reverified.jsonl \ --force ``` See [Reverify Rollouts](/tutorials/evaluation-tutorials/reverify-rollouts) for a full walkthrough. --- ## Contributor Helpers ### `gym dev test` Run NeMo Gym's core unit tests with coverage reporting. ```bash gym dev test ``` --- ## Migrating from the legacy commands The legacy `ng_*` and `nemo_gym_*` are deprecated and will be removed in the future release. Use the tables below to find their `gym` replacement and update your scripts and workflows. ### Command mapping | Legacy command | New command | | --------------------------------- | --------------------------------------- | | `ng_help` | `gym --help` | | `ng_version` | `gym --version` | | `ng_list_benchmarks` | `gym list benchmarks` | | `ng_run` | `gym env start` | | `ng_status` | `gym env status` | | `ng_dump_config` | `gym env resolve` | | `ng_pip_list` | `gym env packages` | | `ng_init_resources_server` | `gym env init` | | `ng_test` | `gym env test` | | `ng_test_all` | `gym env test --all` | | `ng_prepare_benchmark` | `gym eval prepare` | | `ng_e2e_collect_rollouts` | `gym eval run` | | `ng_collect_rollouts` | `gym eval run --no-serve` | | `ng_aggregate_rollouts` | `gym eval aggregate` | | `ng_reward_profile` | `gym eval profile` | | `ng_prepare_data` | `gym dataset collate` | | `ng_materialize_prompts` | `gym dataset render` | | `ng_upload_dataset_to_hf` | `gym dataset upload` | | `ng_upload_dataset_to_gitlab` | `gym dataset upload --storage gitlab` | | `ng_download_dataset_from_hf` | `gym dataset download` | | `ng_download_dataset_from_gitlab` | `gym dataset download --storage gitlab` | | `ng_gitlab_to_hf_dataset` | `gym dataset migrate` | | `ng_delete_dataset_from_gitlab` | `gym dataset rm` | | `ng_dev_test` | `gym dev test` | | `ng_reinstall` | `uv sync --extra dev` | ### Replacing Hydra overrides with flags The most common Hydra overrides now have dedicated flags: | Legacy Hydra override | New flag | | ------------------------------------------------- | --------------------------------- | | `"+config_paths=[a.yaml,b.yaml]"` | `--config a.yaml --config b.yaml` | | `+agent_name=...` | `--agent` | | `+input_jsonl_fpath=...` | `--input` | | `+input_glob=...` | `--input-glob` | | `+output_jsonl_fpath=...` | `--output` | | `+limit=...` | `--limit` | | `+num_repeats=...` | `--num-repeats` | | `+num_samples_in_parallel=...` | `--concurrency` | | `++split=...` | `--split` | | `++policy_model_name=...` | `--model` | | `++policy_base_url=...` | `--model-url` | | `++policy_api_key=...` | `--model-api-key` | | `++responses_create_params.temperature=...` | `--temperature` | | `++responses_create_params.top_p=...` | `--top-p` | | `++responses_create_params.max_output_tokens=...` | `--max-output-tokens` | | `+mode=...` | `--mode` | | `+output_dirpath=...` | `--output-dir` | | `+should_download=true` | `--download` | | `+prompt_config=...` | `--prompt-config` | | `+resume_from_cache=true` | `--resume` | | `+dataset_name=...` | `--name` | | `+repo_id=...` | `--repo-id` | | `+revision=...` / `+version=...` | `--revision` | | `+artifact_fpath=...` | `--artifact` | | `+output_fpath=...` | `--output` | | `+create_pr=true` | `--create-pr` | | `+outdated=true` | `--outdated` | For standalone rollout collection, `num_samples_in_parallel` limits active requests, while `max_resident_rollout_tasks` limits how many rollout tasks the driver creates at once. When both are set, active requests cannot exceed the lower limit. Both settings are optional; leaving the task window unset preserves the previous admission behavior. Set `retain_results_in_memory=false` for artifact-oriented runs: results are appended and flushed as they complete, and `run_from_config()` returns `[]` instead of the ordered full result list. Aggregation and upload may still load the persisted results into memory. These settings are available through the collection config or Hydra overrides such as `+max_resident_rollout_tasks=64` and `+retain_results_in_memory=false`. ### Common workflows, before and after ```bash # Start servers # Before: ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" # After: gym env start \ --resources-server example_single_tool_call \ --model-type openai_model # End-to-end rollout collection # Before: config_paths="responses_api_models/openai_model/configs/openai_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml" ng_e2e_collect_rollouts "+config_paths=[${config_paths}]" \ ++output_jsonl_fpath=results/aime24.jsonl \ ++split=validation # After: gym eval run \ --model-type openai_model \ --resources-server math_with_judge \ --output results/aime24.jsonl \ --split validation # Collect against already-running servers # Before: ng_collect_rollouts +agent_name=example_single_tool_call_simple_agent \ +input_jsonl_fpath=weather_query.jsonl \ +output_jsonl_fpath=weather_rollouts.jsonl \ +num_repeats=4 +num_samples_in_parallel=10 # After: gym eval run --no-serve \ --agent example_single_tool_call_simple_agent \ --input weather_query.jsonl \ --output weather_rollouts.jsonl \ --num-repeats 4 \ --concurrency 10 # Reward profiling # Before: ng_reward_profile +input_jsonl_fpath=materialized_inputs.jsonl +rollouts_jsonl_fpath=rollouts.jsonl # After: gym eval profile \ --inputs materialized_inputs.jsonl \ --rollouts rollouts.jsonl ``` > **Tip** > > Run any command with `--help` to see its full set of flags, and `gym --help` to list every group.