CLI Commands

This page documents the NeMo Gym command-line interface.

All functionality is exposed through a single gym entry point, organized into command groups (gym <group> <command>). ng is a drop-in alias for gym. Every group and command supports -h/--help.

Quick Reference

# General
gym --help                   # list all command groups
gym --version [--json]        # print version and system info
ng ...                        # 'ng' is an alias for 'gym'
# Discover
gym list benchmarks [<name>] [--json]         # list benchmarks (or inspect one), by their --benchmark value
gym list environments [<name>] [--json]       # list the unified environment and benchmark catalog
gym list agents [<name>] [--json]             # list agent harnesses and how each composes
gym list models [<name>] [--json]             # list --model-type values (or inspect a model)
gym list resources-servers [<name>] [--json]  # list --resources-server values (or inspect one)
gym search [<type>] <query> [--json]          # search the unified environment catalog, or one component type
# Datasets
gym dataset upload            # upload a prepared dataset to HF (default) or GitLab
gym dataset download          # download a dataset from HF (default) or GitLab
gym dataset rm                # delete a dataset from GitLab
gym dataset migrate           # move a dataset from GitLab to HF
gym dataset render            # generate a dataset preview (materialize prompts)
gym dataset collate           # validate and collate a dataset
gym dataset init <name>       # scaffold a Harbor-format dataset (dataset.toml plus one task; --parameterized for a template)
gym dataset publish --harbor <target> --output <dir>  # write a Gym-authored dataset as plain Harbor task folders
gym dataset pull <target>     # fetch a Harbor dataset (hub reference or committed entry) into environments/harbor/
gym dataset validate <target> # run each Harbor task's reference solution through its verifier
# Environments
gym env init                  # scaffold a resources server, environment, or benchmark
gym env resolve               # resolve and print the final merged config
gym env validate              # validate a manifest-backed workload or legacy config without services
gym env packages              # list packages in a server's virtual environment
gym env test                  # run resources-server tests
gym env publish               # finalize readiness and confirm registry discovery
gym env prefetch              # pre-warm per-server venvs without starting servers
gym env start                 # start the servers
gym env status                # show running servers
gym env schema                # print a resources server's task-data schema as JSON Schema
# Evaluation
gym eval prepare              # prepare benchmark data and dump it to disk
gym eval plan                 # write one benchmark's task list (stable ids, digests) to a plan file
gym eval run                  # collate data, start servers, and collect rollouts
gym eval submit               # submit a benchmark job to Slurm
gym eval resubmit             # queue a submitted benchmark again, or recover it from its saved evidence
gym eval aggregate            # merge sharded rollout results
gym eval export               # export a finished run (default exporter: atif)
gym eval status               # show saved run progress
gym eval health-check         # verify rollout artifact quality for an existing run
gym eval profile              # compute a reward profile from rollouts
gym eval reverify             # recompute rewards from existing rollouts without re-running inference
gym eval compare              # compare a baseline eval run against a candidate run
gym eval compare-suite        # compare normalized scores across benchmarks and models
# Contributor helpers
gym dev test                  # run NeMo Gym's unit tests
gym dev compare OLD NEW       # per-task parity report between two rollout files (migration aid)

Common Options

These options are shared across many commands.

Selecting the model server

Commands that need a model (gym env start, gym eval run) configure it through four flags:

Swapping the agent

--agent-type NAME runs an environment or benchmark with a different agent harness, without editing any config. It resolves to responses_api_agents/NAME/configs/NAME.yaml; use NAME/FLAVOR to pick a non-default config in that directory.

# Run Terminal-Bench 2.1 under Terminus 2 instead of the OpenCode harness its config names
gym eval run \
    --benchmark terminal_bench_2_1/opencode \
    --agent-type terminus_2_sandboxed_agent \
    --model-type vllm_model

Available on gym env start, gym env validate, gym env prefetch, and gym eval run. Like the other selectors it is pure shorthand, so --agent-type terminus_2_sandboxed_agent and --config <that file> behave identically.

Run gym list agents to see the harnesses you can pass.

The composed server instance is renamed after the agent that runs it, so metrics and gym env status report the harness actually used: composing terminus_2_sandboxed_agent onto terminal_bench_2_1_opencode_sandboxed_agent produces terminal_bench_2_1_terminus_2_sandboxed_agent. gym env start prints the instance list, which matters for the two-step flow:

gym env start \
    --benchmark terminal_bench_2_1/opencode \
    --agent-type terminus_2_sandboxed_agent \
    --model-type vllm_model
gym eval run --no-serve --agent terminal_bench_2_1_terminus_2_sandboxed_agent --input rows.jsonl

Anything that routes rows — --agent, +agent_map values, +fan_out entries — must name the composed instance, and Gym says so rather than failing later:

Routing names agent instances that no longer exist once the agent is swapped:
  - agent_name: 'terminal_bench_2_1_opencode_sandboxed_agent' is now
    'terminal_bench_2_1_terminus_2_sandboxed_agent'
Use the name the composed config reports.

Rows collected before the swap are re-routed for you, so existing rollout files stay usable.

'simple_agent' is not declared compatible with 1 of the agent instance(s) it would replace,
so it cannot be scored correctly:
  - terminal_bench_2_1_opencode_sandboxed_agent uses terminal_bench_2_1_opencode_resources_server
    and accepts opencode_sandboxed_agent, terminus_2_sandboxed_agent
Select one of: opencode_sandboxed_agent, terminus_2_sandboxed_agent. Or pass
--allow-unsupported-pairing (or set NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING=1) to bypass the check.

Hydra overrides (escape hatch)

The gym CLI is a thin wrapper over Gym’s Hydra config system. Standard flags (--flag value) cover the common inputs. Anything not covered by a flag can still be passed as a raw Hydra override using +key=value (add a new key) or ++key=value (add or override an existing key). Unknown overrides are forwarded to Hydra untouched, so advanced config composition is fully functional.

When a command needs overrides for keys that have no dedicated flag, you can keep the whole command in Hydra form (config paths included) or mix --flag and +key=value styles in one invocation:

gym eval run \
    --benchmark aime24 \
    --model-type openai_model \
    ++responses_create_params.reasoning.effort=low \
    +wandb_project=gym-dev

General

gym --help

List all command groups and their descriptions.

gym --help

gym --version

Print the NeMo Gym version along with Python, key dependency, and system information.

gym --version
# Output as JSON
gym --version --json

Discovery

Commands for discovering benchmarks, environments, model servers, agent harnesses, and resources servers.

gym list <type> lists all components of that type; gym list <type> <name> inspects that one. All commands accept:

• --json which prints machine-readable output instead of a human-readable table.
• --search-dir which allows you to surface your own Gym-compatible components alongside the built-ins.

gym list benchmarks

List the benchmarks available in NeMo Gym, with their kind, manifest status, domain, and description. The value in the Name column is the benchmark identifier and can be passed to --benchmark. To see which agent harness a benchmark composes with, inspect it with gym list benchmarks <name>.

gym list benchmarks
gym list benchmarks aime24            # inspect one benchmark
# Machine-readable output for scripting
gym list benchmarks --json | jq '.[].name'

gym list environments

List the manifest-first catalog unioned with legacy environment and benchmark configs. Manifest-backed entries expose their declared metadata; unmigrated entries remain visible with no-manifest status.

gym list environments
gym list environments calendar        # inspect one environment
gym list environments --kind benchmark --domain math
gym list environments --json | jq '.[].name'

The JSON output includes three path fields:

• path: the entry’s directory. Multiple entries can share this directory, so it is not a unique identifier.
• config_path: the entry’s config file. Use this field to locate the config; it is not necessarily inside path.
• manifest_path: the entry’s manifest.yaml, or null when the entry has no manifest.

All non-null paths are absolute with symlinks resolved. A path reached through a symlink therefore uses its target’s location in the output. For example, the config for aalcr/config_v1_1 is benchmarks/aalcr/config_v1_1.yaml, beside its entry directory benchmarks/aalcr/config_v1_1/.

gym list agents

List the agent harnesses under responses_api_agents/, with each one’s composition pattern — composable (Pattern A: references a separate resources server, so it can be wired into a matching environment) vs self-contained (Pattern B: ships its own framework/environment) — plus its config variants.

gym list agents
gym list agents simple_agent          # inspect one agent
gym list agents --json | jq '.[] | {name, self_contained}'

gym list models

List the model servers under responses_api_models/. The value in the Model column corresponds to a config flavor and can be passed as the --model-type flag to other commands.

gym list models
gym list models vllm_model            # inspect one model
gym list models --json | jq '.[].model'

gym list resources-servers

List the resources servers under resources_servers/. The value in the Name column corresponds to the config flavor and can be passed as --resources-server.

gym list resources-servers
gym list resources-servers mcqa       # inspect one resources server
gym list resources-servers --json | jq '.[].name'

gym search

Filter a listing to fuzzy matches on a query (against each component’s name/value, description, domain, and type-specific fields). Without a type, the command searches the unified environment and benchmark catalog.

gym search math                       # environments and benchmarks matching "math"
gym search benchmarks math            # only benchmarks matching "math"
gym search environments calendar      # environments matching "calendar"
gym search models vllm --json

Datasets

Commands for preparing, previewing, and managing datasets. By default dataset transfer commands use HuggingFace; pass --storage gitlab to target the GitLab Registry.

gym dataset init

Scaffold a dataset in the Harbor task format: dataset.toml with a [gym] table and one task with an instruction, a verifier and a reference solution. Refer to Harbor Tasks.

gym dataset init my-dataset --image ubuntu:24.04   # dataset.toml names it local/my-dataset

gym dataset pull

Fetch a Harbor dataset into environments/harbor/ of the Gym checkout, or under the working directory when Gym is installed as a package ($NEMO_GYM_DATASETS_DIR when set), and print where it landed, its task count and its dataset digest. Nothing is prepared, built or run. The fetch is the one gym eval run and gym dataset validate make for a harbor: target, so a folder that was edited by hand, left by an interrupted fetch of another version, or written for another package is refused in the same words; a second pull of an unchanged dataset downloads nothing and says so. Refer to Harbor Tasks.

gym dataset pull terminal_bench_4                                  # the entry's source, into environments/harbor/terminal_bench_4/
gym dataset pull harbor:terminal-bench/terminal-bench@4.0.0        # the same folder: the entry declares this reference
gym dataset pull harbor:terminal-bench/terminal-bench-2-1          # no entry declares `latest`: environments/harbor/terminal-bench-2-1/

gym dataset validate

Run each Harbor task’s reference solution (solution/solve.sh) through its verifier with the oracle harness and print one line per task with its reward, or why there is none. gym eval run TARGET --agent harbor_oracle is the same run. Results go under results/harbor/validation/<taskset>/ (rollouts.jsonl unless --output names another file, the generated run config and skipped.json), apart from any evaluation of the same target, so an evaluation never resumes from the oracle’s rewards. The exit code is 0 when every selected solution scored 1, and 1 when a solution scored below 1, was masked or failed, when a task folder did not load, or when no selected task had a solution to run. Refer to Harbor Tasks.

gym dataset validate my-dataset --sandbox opensandbox
gym dataset validate harbor:terminal-bench/terminal-bench@4.0.0 --sandbox opensandbox --only-tasks 'hello-*'
gym dataset validate terminal_bench_4 --sandbox opensandbox      # the committed entry, after `gym dataset pull terminal_bench_4`

gym dataset upload

Upload a prepared local JSONL dataset to HuggingFace (default) or GitLab.

# Upload to HuggingFace
gym dataset upload \
    --name my_dataset \
    --input data/train.jsonl \
    --revision 0.0.1
# Upload to GitLab
gym dataset upload \
    --storage gitlab \
    --name my_dataset \
    --input data/train.jsonl \
    --revision 0.0.1

gym dataset download

Download a dataset from HuggingFace (default) or GitLab.

# Download a single file from HuggingFace
gym dataset download --repo-id NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \
    --artifact train.jsonl \
    --output data/train.jsonl
# Download from GitLab
gym dataset download \
    --storage gitlab \
    --name example_multi_step \
    --revision 0.0.1 \
    --artifact train.jsonl \
    --output data/train.jsonl

gym dataset rm

Delete a dataset from the GitLab Registry. Prompts for confirmation.

gym dataset rm --name old_dataset

gym dataset migrate

Migrate a JSONL dataset to HuggingFace from GitLab. Use gym dataset upload if you do not want automatic GitLab deletion.

gym dataset migrate \
    --name my_dataset \
    --input data/train.jsonl \
    --revision 0.0.1

gym dataset collate

Validate and collate a dataset, generating metrics and statistics.

gym dataset collate \
    --resources-server example_multi_step \
    --output-dir data/example_multi_step \
    --mode example_validation

gym dataset render

Generate a dataset preview by materializing prompts from a raw input file and a prompt template, producing JSONL with populated responses_create_params.input for RL training.

Each input row must not already have a populated responses_create_params.input; the command applies the prompt template from --prompt-config to each row, fills in the input, and preserves the row’s other fields.

gym dataset render \
    --input raw.jsonl \
    --prompt-config prompt.yaml \
    --output preview.jsonl

Environments

Commands for developing, running, and inspecting environments. An environment defines tasks, verification, and its interaction surface; Gym config composes it with agent, model, and runtime components.

gym env init

Scaffold a resources server, or create a manifest-backed environment or benchmark skeleton. Workload scaffolds include a manifest, Gym config, sample data, and the extension points selected by the integration profile.

gym env init --resources-server my_server
gym env init --environment my_eval --profile custom-gym-verifier
gym env init --benchmark my_benchmark --reuse-verifier existing_scorer \
    --reward-range 0 1 --higher-is-better

gym env resolve

Resolve the configs, flags, and overrides into a final merged config and print it. Useful for debugging configuration. Secrets are hidden.

Unlike gym env validate, resolve substitutes no dummy model, so a model config’s policy_* values must be supplied for its interpolations to resolve.

When an environment server block doesn’t match its server’s config class, resolve still prints the config and lists the fields to fix as a warning on stderr.

# Merge a resources-server config with a model config, apply an override, and print the result
gym env resolve \
    --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml \
    --config responses_api_models/openai_model/configs/openai_model.yaml \
    ++responses_create_params.temperature=0.6 \
    ++policy_base_url=https://api.openai.com/v1 \
    ++policy_api_key=sk-example \
    ++policy_model_name=gpt-4o-mini

gym env validate

Validate a manifest-backed workload or legacy config without starting Ray or server subprocesses. Manifest validation resolves the Gym config offline, checks composition mirrors and datasets, and reports the resolved components. Legacy validation retains the existing config pre-flight behavior, and also checks each environment server block against the config class of the server it runs, which gym env start otherwise applies only when that server launches. Exits 0 when valid, or 1 with a clean message when not.

By default, values a run needs but the config leaves undefined (such as policy_base_url from env.yaml) are filled with placeholders so the rest can still be checked, and reported as a warning. Pass --strict to fail on them instead.

gym env validate my_eval
gym env validate --sync my_eval
gym env validate --environment workplace_assistant
gym env validate --benchmark gsm8k
# or explicit config path(s)
gym env validate --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml
# fail unless everything a run needs is defined, credentials included
gym env validate --resources-server example_single_tool_call --model-type openai_model --strict

gym env packages

Each server has its own isolated virtual environment. List the packages installed in a server’s environment.

gym env packages --resources-server example_single_tool_call
# Check for outdated packages
gym env packages \
    --resources-server example_single_tool_call \
    --outdated

gym env test

Exercise a manifest-backed workload’s verifier fixture in its resources-server environment, or run the existing pytest suite for one or all resources servers.

# Exercise a manifest-backed workload verifier
gym env test my_eval
# Test a single server
gym env test --resources-server example_single_tool_call
# Test all servers. Slow: builds one venv per server, so it needs an explicit opt-in.
gym env test --all

gym env publish

Run the local publication check for a manifest-backed workload. The command runs static validation and the verifier fixture, rejects manifest metadata placeholders, and confirms that the exact manifest is discoverable. It includes an experimental annotation only when the manifest’s flag is true. It does not certify the workload, assess implementation quality, require a full evaluation or training run, commit, or push changes.

gym env publish my_eval
gym env publish my_benchmark --kind benchmark --json

gym env start

Start the NeMo Gym servers (agents, models, resources, and environment servers) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment. Rollout collection dispatches /run through environment servers.

gym env start \
    --resources-server example_single_tool_call \
    --model-type openai_model
# Start a benchmark's servers
gym env start \
    --benchmark gpqa \
    --model-type vllm_model

gym env status

Show all currently running NeMo Gym servers and their health.

gym env status

NeMo Gym Server Status:
[1] ✓ example_single_tool_call (resources_servers/example_single_tool_call)
{
    'server_type': 'resources_servers',
    'name': 'example_single_tool_call',
    'port': 58117,
    'pid': 89904,
    'uptime_seconds': '0d 0h 0m 41.5s',
}
...
3 servers found (3 healthy, 0 unhealthy)

gym env prefetch

Pre-warm per-server venvs without starting any servers. Accepts the same config flags as gym env start. For each server in the config, creates its isolated venv and installs dependencies serially. Intended for use in Dockerfile builds so venvs are ready at runtime without network access.

config_paths="responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,\
resources_servers/math_with_judge/configs/math_with_judge.yaml"
gym env prefetch "+config_paths=[${config_paths}]"

gym env schema

Print a resources server’s TaskData schema as JSON Schema. Exits non-zero if the server does not export a task_data.py schema.

gym env schema --resources-server example_multi_step

Evaluation

Commands for running evaluations end to end: prepare data, collect rollouts, aggregate sharded runs, verify rollout health, and profile results.

gym eval prepare

Prepare a benchmark’s data by running its prepare.py script and dump the result to disk.

gym eval prepare --benchmark aime24

gym eval plan

Prepare one benchmark and write its plan: every task once, sorted by task id, with a digest of its prepared row. The plan stores ids and digests, not the inputs themselves. The format is versioned ("schema": "nemo-gym.eval-plan.v1"), only changes additively within a version, and is printed as JSON Schema by --schema.

Each task’s id comes from its row’s task_id, problem_id or instance_id, and its taskset is the dataset’s declared taskset (the benchmark name when none is declared). Unlike gym eval run, there is no fallback to the row’s position, so benchmarks without such a field are refused. The digest covers the prepared row after excluding runtime routing and bookkeeping fields (agent_ref, task_source, skills_ref, and keys beginning with _ng_). A prompt template (prompt_config) is applied when a run starts, so editing the template does not change the digest.

gym eval plan --benchmark swebench/verified/opencode --out plan.json
gym eval plan --benchmark swebench/verified/opencode --task-ids astropy__astropy-12907 --out plan.json
gym eval plan --schema

gym eval run

Collate data, start the servers, and collect rollouts. This is the main evaluation command. By default it spins up all required servers. Pass --no-serve to collect against servers you already started with gym env start.

In place of --benchmark, a positional TARGET runs a dataset in the Harbor task format: a task folder, a folder of task folders, a harbor:<dataset>[@<version>] hub reference, or the name of a committed entry under environments/harbor/ (for example terminal_bench_4) after gym dataset pull. Refer to Harbor Tasks for the layout and the --sandbox, --only-tasks and --exclude-tasks options that apply to such a target.

# End-to-end: spin up servers, then collect rollouts for a benchmark
gym eval run --benchmark aime24 \
    --model-type openai_model \
    --output results/aime24.jsonl \
    --split validation \
    --concurrency 10
# Two-terminal sequence for --no-serve: model settings go to env start, not eval run.
# Model URL and credentials configure servers at start time; passing them during
# collection has no effect on already-running servers.
# Terminal 1 — start servers with the model endpoint configured:
gym env start \
    --resources-server math_with_judge \
    --model-type openai_model \
    --model openai/gpt-oss-120b \
    --model-url http://127.0.0.1:10240/v1 \
    --model-api-key dummy_key
# Terminal 2 — collect rollouts against the running servers:
gym eval run --no-serve \
    --resources-server math_with_judge \
    --output results/test_001.jsonl \
    --split validation \
    --temperature 1.0 \
    --top-p 1.0
# Per-agent repeats: one input file pins different agents per row via agent_ref.name
gym eval run --no-serve \
    --input mixed_agents.jsonl \
    --output results/mixed_rollouts.jsonl \
    --num-repeats '{agent_alpha: 4, agent_beta: 1, _default: 1}'

Generation parameters

The most common sampling parameters have dedicated flags on gym eval run: --temperature, --top-p, and --max-output-tokens. These map onto responses_create_params.temperature, responses_create_params.top_p, and responses_create_params.max_output_tokens.

Any other responses_create_params field that has no dedicated flag can be set with a raw Hydra override using the ++responses_create_params.<field> syntax. Overrides are merged into each input row’s existing responses_create_params with a shallow merge (top-level keys only):

gym eval run --no-serve \
    --agent example_single_tool_call_simple_agent \
    --input weather_query.jsonl \
    --output weather_rollouts.jsonl \
    --temperature 1.0 \
    --top-p 1.0 \
    --max-output-tokens 4096 \
    ++responses_create_params.reasoning.effort=low

Output files

Next to --output (<stem> is its name without .jsonl), the run writes:

Resume interrupted runs

Pass --resume to restart the same command after a crash or interruption and pick up only the rows that have not finished yet.

How it works:

• Materialized inputs. On the first run, the fully expanded input rows (after --num-repeats, --limit, --prompt-config, and any overrides) are written to a sidecar file next to your output. The path is derived from --output by appending _materialized_inputs to the stem — so rollouts.jsonl produces rollouts_materialized_inputs.jsonl.
• Incremental output. Successful rollouts are flushed to the main output JSONL after each completion; retryable and terminal failures, plus intentional omissions (_ng_omitted), are saved in a <stem>_failures.jsonl sidecar, so partial progress survives a crash.
• Failed agent calls. By default a failed agent /run ends the run, because the alternative is a score reported over fewer rollouts than were asked for. Before it ends, the run writes one <stem>_failures.jsonl row for that rollout with its identity, failure class, failure report and _ng_failure_message, marked _ng_omitted_rollout: true and _ng_unrouted_run_error: true. That row is evidence of the lost reply, not an attempt: no reader scores or counts it, readers skip it before they take a rollout’s last attempt, so an older failure of the rollout still counts. An uncertain or possibly delivered request blocks ordinary resume; proven single-attempt not_sent failures remain eligible within the saved budget. Pass +route_failures_to_sidecar=true to let the run continue instead: the call is then recorded in the same sidecar as one attempt, with no reward. Every rollout that leaves the score is logged as it happens, whichever layer classified it, and the totals appear with the closing artifact paths. The class says whether the rollout ran: agent_run_error when the agent itself answered, since a NeMo Gym agent returns 500 when its handler raises, and agent_request_failed for a gateway status (429, 502, 503, 504) or no usable reply, neither of which says anything about the rollout. Other rollouts keep running, the score is computed over completed rollouts only, and resume re-dispatches only eligible, classified pre-answer failures up to NEMO_GYM_MAX_ROLLOUT_ATTEMPTS. Saved judge answers and unclassified execution are not regenerated. A run in which nothing produced a result fails instead of reporting an empty score, unless an option below counts its rollouts as zeros.
• Counting failures in the score. Metrics are computed over completed rollouts only. +count_failure_classes_as_zero=[<class>] includes the named sidecar rows in /aggregate_metrics so they land in the denominator: [agent_run_error] counts the rollouts that reached the agent and broke, which is what a model emitting unparsable tool calls looks like from here, and needs +route_failures_to_sidecar=true to have anything to count. A row that carries no reward is scored zero for the metric input only, so neither the sidecar nor the rollouts JSONL records a verdict that no verifier gave. It works the same in gym eval run and gym eval aggregate. A sidecar row whose producer marked it _ng_failure_counts_as_zero: true is counted the same way whatever its class, when it is the rollout’s latest attempt: Simple Agent’s outcome_policy.by_failure_kind.model_policy_response_timeout=count_as_zero opts in only when every observed attempt of the final operation timed out waiting for a response after a confirmed completed upload; final_on_model_timeout=true is a compatibility alias. Missing upload or attempt evidence cannot activate this rule.
• Counting lost rollouts. +count_missing_rollouts_as_zero=true scores as zero a rollout that is in the materialized inputs but left no row in the output JSONL or the sidecar, such as one killed mid-flight or one dispatch_budget_s never started. A sidecar row written by +record_omitted_rollouts (see Rollouts with no result) records no result, so such a rollout is counted here. The zero goes into the metric input only, so it does not itself make an interrupted rollout safe to replay. --resume runs only eligible work; saved answers and uncertain execution still block regeneration. A rollout whose newest attempt has a known failure is never counted here, so this does not widen +count_failure_classes_as_zero. The count is reported as coverage/imputed. With --disable-aggregation nothing is scored, so a run with no row still fails. gym eval aggregate checks each shard against its own rows, materialized inputs and sidecar.
• Matching. Resume validates the saved manifest against the current dataset, materialized inputs, and task configuration. A configuration that differs only in operational settings (timeouts, retries, ports, concurrency, health limits, paths, tracking) resumes with a warning naming them and a line in <stem>_manifest.resume-drift.jsonl; a difference in a semantic setting (data and limits, repeats, agent and model, sampling, judge and grading) is refused, naming each setting. See the resume guide for the full split. The run manifest reserves execution identities and selects each logical rollout’s newest attempt. Completed results are reused, including zero rewards; eligible failed or interrupted work receives a new attempt identity.
• Terminal failures. A sidecar row stamped _ng_failure_terminal is never retried. +retry_terminal_timeouts=true retries rows of class timeout_exceeded even when they are stamped terminal. Retries still stop at NEMO_GYM_MAX_ROLLOUT_ATTEMPTS. Off by default, because some agents stamp their timeouts terminal on purpose.
• Dispatch budget. +dispatch_budget_s=<seconds> stops starting new rollouts that many seconds after collection starts; set it to the usable walltime of the allocation. It requires --concurrency or +max_resident_rollout_tasks; without either limit, the whole queue could be sent before the budget is checked. Rows that are not started are not attempts, so the next --resume runs them. +drain_margin_s=<seconds> also stops starting rows when less than that much time is left; unset, the margin is the p75 duration of the run’s recorded attempts once there are at least 5. An attempt whose agent /run returns a usable reply is always recorded, even when the agent reports the rollout itself as failed or timed out; an attempt whose /run fails (an error status, or no usable reply) is recorded only if it ran at least as long as the median so far, because a quick failure says nothing about how long rollouts take. Every run ends with a latency summary over the same attempts (median, p90, p99, max) and the number of rows not started.
• Rollouts with no result. An attempt the agent reports as kill_shaped (Slurm SIGTERM, a dead Ray actor, an OOM outside the rollout) is always saved in <stem>_failures.jsonl as an interruption: it is the rollout’s newest attempt, spends no retry, and blocks automatic replay when its execution cause is unknown. An unused failure budget does not establish that replay is safe. A row dispatch_budget_s never started is not an attempt and by default leaves no row; run coverage counts it as not_started. +record_omitted_rollouts=true writes one sidecar row for each such row, so the absence has a record. The row keeps the rollout’s task_index and rollout_index, and carries _ng_no_persist: true, _ng_omitted_rollout: true, _ng_dispatch_drained: true, the class cancelled and a reason in _ng_failure_message. Such a row is evidence, not an attempt or a failure: the run history skips it, and so do scoring, +count_failure_classes_as_zero, aggregate and exporter metrics, the run health summary, gym dev compare, rollout uploads and gym eval reverify --judge-failed-only. A sidecar written without a run manifest can also hold kill_shaped rows from this option; there an interrupted row is the rollout’s last attempt, so an older failure is not counted again and the rollout is missing. Rows carrying only _ng_no_persist, such as the kill_shaped rows ATIF reverification writes, are read as before. Off by default while readers of the sidecar outside NeMo Gym learn to treat these rows as not run rather than failed. A collector that is itself killed before it sees an outcome leaves no row, and its reserved attempt is counted as missing. A custom rollout_collection_driver such as GDPVal’s multi-stage one keeps its own persistence and ignores the option.
• Longest first. +dispatch_longest_first=true starts the rows whose earlier failed attempts ran longest first on resume, using the elapsed_seconds those attempts recorded. Rows with no recorded timing keep their input order and start after them. Without it, rows start in input order.

When require_complete=true (the default for gym eval submit), exit status 75 means incomplete with eligible retries and progress: new successes, omissions, terminal failures or counted failures. Admission timeouts and nonterminal kill_shaped interruptions are saved but do not establish progress. Retryable work without progress exits 1 and spends the Slurm infrastructure retry limit. Terminal failures or exhausted budgets with nothing eligible to retry exit 76; the Slurm chain stops. If at least one task completed and the only gaps are intentional omissions, collection exits 0 with a count warning, without treating omissions as completed or scored tasks. A run where every task is omitted exits 76 when completion is required. Local gym eval run and standalone collection default to require_complete=false and can return 0 with partial results. Use +require_complete=true to require completion. Explicit Ctrl-C also retains the existing partial-result behavior when completion is not required; inspect coverage to determine what finished.

See Resume and recover evaluations for artifact requirements, attempt selection, and the current limits on judge-only recovery.

gym eval submit

Submit a benchmark evaluation job to Slurm via a submit config YAML file. Renders job scripts and calls sbatch for each benchmark defined in the config. Use --dry-run to preview the generated scripts without submitting.

gym eval submit --config submit.yaml
gym eval submit --config submit.yaml --dry-run
gym eval submit --config submit.yaml --resolve-only --json

gym eval resubmit

Queue a benchmark of an earlier gym eval submit again. Without --recover the job starts with a fresh retry, walltime and per-rollout attempt allowance. With --recover it first decides one action from the run’s saved evidence (resubmit, re-judge saved answers, nothing, or a typed stop) and keeps the spent budgets; see Recover from saved evidence. With --recover it prints one JSON line, and the exit status is 0 only when gate_task.status is SUCCESS, so --inspect exits 1 while work is planned.

gym eval resubmit GYM_JOB_ID BENCHMARK [options]

gym eval resubmit "$GYM_JOB_ID" gpqa --recover --inspect \
  --score-agent simple_agent --score-key '["agent_metrics", "mean/reward"]'
gym eval resubmit "$GYM_JOB_ID" gpqa --recover --max-walltime 24:00:00 --walltime-reason "segments too short for the task"

gym eval aggregate

Merge sharded rollout results into a single rollouts file with aggregate metrics. Reads every JSONL file matching --input-glob, recomputes aggregate metrics over the global union of records, and writes a <output stem>_aggregate_metrics.json next to the merged rollouts. It also lists the shards in <output stem>_sources.json, which gym eval health-check reads for a merged rollouts.jsonl. The output may be a shard only when it is the only one, reached through its own directory rather than a file link from elsewhere, and then nothing is listed. Use this to combine shards produced by gym eval run --no-serve --disable-aggregation.

gym eval aggregate \
    --config benchmarks/aime24/config.yaml \
    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
    --input-glob 'results/rollouts-rs*-chunk*.jsonl' \
    --output results/rollouts.jsonl

gym eval export

Export a finished run with an exporter, chosen by the optional EXPORTER argument. atif is the default; mlflow and wandb send a finished run to those tracking backends, and evalrdl uploads it to EvalRDL. Exporter settings are Hydra +key=value overrides; the option flags below are equivalent forms for atif.

atif exports supported ng_trajectory attachments from a Gym rollouts JSONL file as ATIF v1.7. The command writes one ATIF trajectory per rollout plus a manifest that preserves Gym’s task and rollout identity. It is a strict offline conversion: if Gym cannot represent a source trajectory completely in the supported ATIF subset, the export fails instead of silently omitting data. All invalid JSONL rows are reported together, and no destination is published if any row fails. Captured model attempts not selected by a turn, such as failed retries, remain in extra.nemo_gym.surplus_model_calls; they do not become canonical ATIF steps or contribute to final usage totals. The destination directory must not already exist.

This command does not parse ATIF and does not load NeMo Relay. Each ATIF trajectory takes its agent name from the source row’s agent_ref.name; --agent-version records the version of that agent implementation.

gym eval export atif \
    +rollouts_jsonl_fpath=results/rollouts.jsonl \
    +output_dirpath=results/atif \
    +session_id=eval-2026-08-25 \
    +agent_version=1.2.3

Exporting to MLflow or W&B

mlflow and wandb log what a live run would have logged: the run’s saved config (<stem>_resolved_config.yaml), its aggregate metrics, and its rollouts. They need <stem>_aggregate_metrics.json (newer than the rollouts file) and the saved config next to the rollouts, so they cannot export a run that predates the saved config. Backend settings and credentials (mlflow_tracking_uri, wandb_api_key, …) come from the export command itself (arguments or env.yaml), never from the run’s saved config, which masks them. A backend SDK’s own environment variables, such as WANDB_API_KEY, are also used when the command sets no key. Rollouts are uploaded unless the run’s saved upload_rollouts is false; pass +upload_rollouts=true|false to override. The rollouts file itself is only needed when the rollouts are uploaded, so +upload_rollouts=false also exports a run whose rollouts file is gone.

Once a run is exported, <stem>_<exporter>_export_done is written next to the rollouts: a version-1 receipt that names the backend run, the sha256 of each run file it sent, whether the rollouts were uploaded and the rollout projection version. A later export of the same run is skipped without contacting the backend while the receipt matches the current files. A receipt that no longer matches them, or a legacy or malformed marker, fails the export and sends nothing; it is left in place for you to remove deliberately before exporting again. A failed or partial export leaves no receipt, so it can be retried. A run logged live gets its receipt at live completion when every upload landed and the artifacts match (an existing receipt is never replaced); its <stem>_<exporter>_run_id alone shows only that the live run opened, not that every upload landed, so without a receipt the post-run export sends the run again as a separate run and reports the live run id.

gym eval export mlflow \
    +rollouts_jsonl_fpath=results/rollouts.jsonl \
    +mlflow_tracking_uri=https://mlflow.example \
    +mlflow_experiment_name=gym \
    +mlflow_run_name=nightly

Exporting to EvalRDL

evalrdl uploads one run: the whole directory the rollouts file is in. Everything there is part of the upload (the rollouts and their sidecars, health reports, model-call captures and any benchmark-specific files), and nothing is copied or picked out, so keep each run in a directory of its own (the export is refused when the directory holds another rollouts file, because EvalRDL would read both as one run). The run still needs <stem>_aggregate_metrics.json and <stem>_resolved_config.yaml there, like the mlflow and wandb exporters. The evalrdl SDK is installed into the exporter’s own virtual environment, so it never enters Gym’s environment. The SDK is served by the EvalRDL service’s own package index, <service URL>/v1/client/simple/; evalrdl is not on PyPI. Configure that index through uv’s own index settings (for example UV_EXTRA_INDEX_URL) when you run the command, and do not point uv at an index that could carry another package named evalrdl, because the SDK runs with your token in its environment. The SDK reads the token from $EVALRDL_TOKEN, $GITLAB_TOKEN or $GITLAB_PRIVATE_TOKEN, then from glab, which cluster nodes usually lack, so set one of the variables there. The environment is built once and the SDK is not upgraded afterwards; when the service stops accepting an old client, the export fails and names the environment to delete, or pin a newer version with +requirement=evalrdl==VERSION.

The result is succeeded when EvalRDL stored the run as new, and skipped when it already held the same content. With +wait=false the run counts as succeeded once EvalRDL has stored it, and the upload token is in the result. When waiting, an upload EvalRDL stored without a verdict before the timeout fails with exit code 3, and one it refused fails with exit code 4; both name the reason. A worker still running after +timeout plus an hour for the upload itself is killed and the export fails.

gym eval export evalrdl \
    +rollouts_jsonl_fpath=results/rollouts.jsonl \
    +label=nightly '+tags=[a,b]' +meta.model=qwen

See ATIF Export for supported trajectories, preserved evidence, and conversion limitations.

gym eval status

Read saved progress for a run directory or a Gym job ID from gym eval submit:

gym eval status results/my-run
gym eval status gym-job-20261016T120000Z-abcdef --json

Collection without a batch manifest writes <output-stem>_batch_status.json next to its output JSONL (for example, gsm8k_batch_status.json), and a shard recorded through run_examples writes rollouts-<shard>_batch_status.json. Manifest-backed runs retain batch_status.json. A status without a manifest carries the count, failure and aggregation keys only; the manifest-only keys (recipe, dataset and metric expectations, repeat policy) belong to batch_status.json. The directory command lists every stem-derived status file and the legacy batch_status.json when present. Progress is written at initialization, on the first durable result, at most every five seconds thereafter, and at completion. Counts report completed and remaining rollouts and the latest outstanding failures by class and by failure kind, plus the model-final results by kind (an answered agent deadline, or a failure its producer settled as a scored zero): these are reported as facts and never counted as completed. A successful retry clears its earlier failure from these counts. Failed rollouts remain in the remaining count because they have no successful result. Status starts from what the run’s Store selected (the current attempt of every rollout), never from raw rollouts.jsonl, so an older success is not counted after a newer retry.

--json prints a mapping from each status-file path or submitted benchmark name to its saved status, plus an export entry: for each exporter that left a receipt (<stem>_<exporter>_export_done) or a live run id (<stem>_<exporter>_run_id) beside the output, whether the export completed and the run it logged to. The human-readable output prints the same as export receipt: mlflow present (run id ...), or missing. Human-readable output includes the expected count, aggregation status, update time, and tracking state. File targets or path-like targets that are not directories produce a run-directory error rather than a job-ID lookup. Gym job IDs use the local submission index; submitted artifacts must be mounted or copied to the paths recorded in the submission. Missing submitted artifacts are shown as unavailable. The command reads files without contacting a scheduler or remote host.

For runs without a manifest, status is best-effort: inconsistent resume identities or native result identities warn once and disable status reporting while collection continues. Gym tries once to write tracking: "disabled", the exception type as reason, and updated_at, replacing unreliable counts. Fresh runs clear their output stem’s previous status before initializing the tracker. Manifest-backed runs retain strict validation. Resume loading retains compact identities rather than full input payloads, and general-run progress counters update incrementally as results arrive.

gym eval health-check

Verify rollout quality for an existing run directory. The command reads <run-dir>/rollouts.jsonl by default, or the shards gym eval aggregate merged into it, and writes quality_summary.json and rollout_verdicts.jsonl in the run directory.

gym eval health-check results/my-run
# Select a nonstandard rollout filename and exclude one known check
gym eval health-check results/my-run \
    --rollouts-file evaluator_rollouts.jsonl \
    --ignore-checks model_call_missing_token_counts

Refer to Rollout Health Checks for verdict semantics, evidence requirements, the complete check catalog, and report schemas.

gym eval profile

Compute a reward profile from collected rollouts. Outputs per-task statistics such as average reward, standard deviation, min/max, and pass rate, useful for filtering tasks before training by difficulty or variance. Requires rollouts collected with --num-repeats greater than 1.

gym eval profile \
    --inputs materialized_inputs.jsonl \
    --rollouts rollouts.jsonl

Writes three files next to the rollouts, named from its stem:

See Repeat-Level Metrics for the full field list and how to read the variability statistics.

gym eval compare

Compare a baseline eval run against a candidate run and write a report.

The report contains a table of point-estimate metrics (the change, plus each side’s value and confidence interval), split into key metrics and all other metrics, and a per-task sample-flips table. Existing statistics such as percentile and confidence interval fields do not get separate rows. Each side’s separate intervals stay at 95%: recorded intervals for a single run, recomputed intervals for pooled repeats. For eligible metrics with at least two finite repeat-level values on each side and cross-repeat means, the command also computes a two-sided unpaired Welch confidence interval for candidate minus baseline, at 95% by default. Use --confidence-level to change the delta confidence level; the JSON report records it in the top-level confidence_level field. The command does not issue a pass/fail verdict.

See Compare Results for the evaluation workflow and interpretation of the confidence interval.

gym eval compare \
    --baseline runs/baseline/rollouts.jsonl \
    --candidates runs/candidate/rollouts.jsonl

Writes into the output directory:

To pool runs, use --baseline-runs b1/rollouts.jsonl,b2/rollouts.jsonl --candidate-runs c1/rollouts.jsonl,c2/rollouts.jsonl,c3/rollouts.jsonl --output-dir results/pooled-comparison. Each repeat contributes one observation to a single Welch comparison; source run means and confidence interval endpoints are not averaged. Either side can use its single-run option. Quality metrics require complete scored repeat estimates in every source; completion-token metrics retain their available observations and report missing accounting. Current single-repeat aggregates have no quality repeat estimates, so collect at least two scored repeats per source before pooling. Whole-run-only metrics cannot be pooled this way. See Pool independent runs for assumptions and examples.

For pooled JSON sides, the ordered sources list contains the complete provenance. Top-level rollout and aggregate paths identify the first source for compatibility. Reports in an existing output directory are replaced.

If a run was collected with --disable-aggregation it has no *_aggregate_metrics.json; run gym eval aggregate first, or supply aggregate files with --baseline-agg-metrics for --baseline, --baseline-runs-agg-metrics for --baseline-runs, and --candidates-agg-metrics for either candidate option.

gym eval compare-suite

Compare candidate and baseline runs across several benchmarks for each model, and across a declared set of model–benchmark pairs. This command normalizes selected metrics, weights benchmarks equally, and reports delta = candidate - baseline with unpaired Welch confidence intervals at a default confidence level of 90%.

gym eval compare-suite \
    --suite suite.yaml \
    --confidence-level 0.90 \
    --output-dir results/suite

Run paths and optional agent or environment-server selectors come from the suite YAML. See Compare Results for a complete multi-model suite YAML, source options, theoretical metric bounds, normalization, equal weighting, statistical assumptions, and coverage/exclusion behavior.

Writes suite_compare_report.md and/or suite_compare_report.json. The suite JSON format is versioned independently with "schema_version": "1" and has no level field. A pooled side’s path and identity fields represent its first source; its ordered sources list contains complete provenance. The separate gym eval compare command keeps its run-level reports and defaults to 95% difference intervals.

gym eval reverify

Recompute rewards by replaying a stored response through a resources server’s /verify endpoint — without re-running model inference. The command accepts native Gym rollouts or the initial completed, text-only Relay ATIF v1.7 profile. It starts the resources server automatically from the provided config.

Before starting, the command checks each resources server’s GET /reverify_mode response. Resources servers report one of three modes:

Servers that report unsupported or unknown cause the command to abort unless --force is passed.

Native Gym input also accepts the rollout collection option +retry_terminal_timeouts=true (with --resume). When a rollout row carries deliverables_dir or reference_ids, both are sent to /verify with the stored response.

ATIF input requires a stateless resources server and rejects --force, --resume, --judge-failed-only, and --append. It fails closed on unsupported or lossy canonical trajectory shapes, including tool-bearing trajectories routed to a resources server with expose_tools_over_mcp: true, and preserves source identity under _ng_atif_provenance without sending that metadata to the verifier. Provider-native payloads and producer-private status records in optional ATIF extra metadata are not interpreted; their producer owns conversion into canonical ATIF fields. A terminal tool step is supported when every call has a correlated observation result.

# Recompute rewards with an updated verifier config (no model server needed)
gym eval reverify \
    --config my_resources_server.yaml \
    "++head_server.host=127.0.0.1" \
    "++head_server.port=9500" \
    --inputs results/rollouts_materialized_inputs.jsonl \
    --rollouts results/rollouts.jsonl \
    --output results/rollouts_reverified.jsonl
# Override a verifier hyperparameter inline
gym eval reverify \
    --config my_resources_server.yaml \
    "++mcqa.resources_servers.mcqa.grading_mode=lenient_boxed" \
    --inputs results/rollouts_materialized_inputs.jsonl \
    --rollouts results/rollouts.jsonl \
    --output results/rollouts_lenient.jsonl
# Score a Relay ATIF v1.7 trajectory against a materialized Gym task
gym eval reverify \
    --config my_resources_server.yaml \
    --input-format atif \
    --inputs results/materialized_inputs.jsonl \
    --atif-manifest relay-atif/manifest.jsonl \
    --output results/relay_atif_reverified.jsonl
# Re-run the judge on only the rollouts whose judge failed (reuses stored responses; no inference)
gym eval reverify --judge-failed-only \
    --config my_resources_server.yaml \
    --inputs results/rollouts_materialized_inputs.jsonl \
    --rollouts results/rollouts.jsonl \
    --output results/rollouts_recovered.jsonl
# Force through an UNSUPPORTED or UNKNOWN server (output prefixed with unsafe_)
gym eval reverify \
    --config my_stateful_server.yaml \
    --inputs results/rollouts_materialized_inputs.jsonl \
    --rollouts results/rollouts.jsonl \
    --output results/rollouts_reverified.jsonl \
    --force

See Reverify Rollouts for a full walkthrough.

Contributor Helpers

gym dev test

Run NeMo Gym’s core unit tests with coverage reporting.

gym dev test

gym dev compare

Compare two rollout JSONL files task by task. A temporary migration aid: before an old resources server is deleted in favour of its Harbor-path replacement, run both on the same tasks and use this report as the gate.

gym dev compare old/rollouts.jsonl new/rollouts.jsonl --logs-root environment_servers/harbor --json compare.json

Rows are joined by task id (_ng_task_id on the Harbor path; task_name or a verifier_metadata id on legacy servers; override with --key FIELD). When both sides span several tasksets, tasks are keyed and listed as taskset/task_id. When only one side does, the other side keeps its plain task ids and each is matched to the task id that appears in exactly one taskset (or in that side’s own taskset); an id that recurs across tasksets is reported as ambiguous and listed as missing. A masked-failures sidecar (<name>_failures.jsonl) is folded back in, keeping only the last record per rollout and none for a rollout that was later scored; a record that names no task is listed as masked under unknown (<side>) #<n>. Each task is classified as identical, flipped (split into old-win and new-win), masked, or missing (only one side has it). A task is masked when either side has mask_sample true, a reward that is absent or null, or a NaN or infinite reward (reported with the raw value). A failure_kind on a row with mask_sample false (such as the Harbor server’s scored zero for a verifier timeout) does not mask: the reward counts, and a flip it causes is listed as a flip with the failure_kind printed beside the side that carries it. A flip where one side also reports an agent error is listed separately as a rerun candidate. A string or boolean reward is an input error that names the file, line and task. When a side has several rows for a task, the mean reward is compared. For every flip the report prints both rewards and the last --tail lines (default 15) of the losing side’s verifier output, read from verifier_logs_dir (resolved against the losing side’s own file directory, then the current directory, then --logs-root) or from the row’s embedded test_output; a log that cannot be read is reported as unavailable with the reason. gym --json dev compare OLD NEW prints the JSON summary on stdout instead of the report (the same object --json PATH writes). The exit code is 0 for a report and 1 when an input cannot be read or the --json PATH file cannot be written.

Migrating from the legacy commands

The legacy ng_* and nemo_gym_* are deprecated and will be removed in the future release. Use the tables below to find their gym replacement and update your scripts and workflows.

Command mapping

Replacing Hydra overrides with flags

The most common Hydra overrides now have dedicated flags:

For standalone rollout collection, num_samples_in_parallel limits active requests, while max_resident_rollout_tasks limits how many rollout tasks the driver creates at once. When both are set, active requests cannot exceed the lower limit. Both settings are optional; leaving the task window unset preserves the previous admission behavior. Set retain_results_in_memory=false for artifact-oriented runs: results are appended and flushed as they complete, and run_from_config() returns [] instead of the ordered full result list. Aggregation and upload may still load the persisted results into memory. These settings are available through the collection config or Hydra overrides such as +max_resident_rollout_tasks=64 and +retain_results_in_memory=false.

Common workflows, before and after

# Start servers
# Before:
ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"
# After:
gym env start \
    --resources-server example_single_tool_call \
    --model-type openai_model
# End-to-end rollout collection
# Before:
config_paths="responses_api_models/openai_model/configs/openai_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml"
ng_e2e_collect_rollouts "+config_paths=[${config_paths}]" \
    ++output_jsonl_fpath=results/aime24.jsonl \
    ++split=validation
# After:
gym eval run \
    --model-type openai_model \
    --resources-server math_with_judge \
    --output results/aime24.jsonl \
    --split validation
# Collect against already-running servers
# Before:
ng_collect_rollouts +agent_name=example_single_tool_call_simple_agent \
    +input_jsonl_fpath=weather_query.jsonl \
    +output_jsonl_fpath=weather_rollouts.jsonl \
    +num_repeats=4 +num_samples_in_parallel=10
# After:
gym eval run --no-serve \
    --agent example_single_tool_call_simple_agent \
    --input weather_query.jsonl \
    --output weather_rollouts.jsonl \
    --num-repeats 4 \
    --concurrency 10
# Reward profiling
# Before:
ng_reward_profile +input_jsonl_fpath=materialized_inputs.jsonl +rollouts_jsonl_fpath=rollouts.jsonl
# After:
gym eval profile \
    --inputs materialized_inputs.jsonl \
    --rollouts rollouts.jsonl