Reference

CLI Commands

View as Markdown

This page documents the NeMo Gym command-line interface.

All functionality is exposed through a single gym entry point, organized into command groups (gym <group> <command>). ng is a drop-in alias for gym. Every group and command supports -h/--help.

The legacy ng_* / nemo_gym_* commands (such as ng_run or nemo_gym_collect_rollouts) still work but are deprecated. Each one prints a notice pointing at its gym replacement and then runs it. See Migrating from the legacy commands at the bottom of this page.

Quick Reference

$# General
$gym --help                   # list all command groups
$gym --version [--json]        # print version and system info
$ng ...                        # 'ng' is an alias for 'gym'
$
$# Discover
$gym list benchmarks [<name>] [--json]         # list benchmarks (or inspect one), by their --benchmark value
$gym list environments [<name>] [--json]       # list the unified environment and benchmark catalog
$gym list agents [<name>] [--json]             # list agent harnesses and how each composes
$gym list models [<name>] [--json]             # list --model-type values (or inspect a model)
$gym list resources-servers [<name>] [--json]  # list --resources-server values (or inspect one)
$gym search [<type>] <query> [--json]          # search the unified environment catalog, or one component type
$
$# Datasets
$gym dataset upload            # upload a prepared dataset to HF (default) or GitLab
$gym dataset download          # download a dataset from HF (default) or GitLab
$gym dataset rm                # delete a dataset from GitLab
$gym dataset migrate           # move a dataset from GitLab to HF
$gym dataset render            # generate a dataset preview (materialize prompts)
$gym dataset collate           # validate and collate a dataset
$
$# Environments
$gym env init                  # scaffold a resources server, environment, or benchmark
$gym env resolve               # resolve and print the final merged config
$gym env validate              # validate a manifest-backed workload or legacy config without services
$gym env packages              # list packages in a server's virtual environment
>gym env test                  # exercise a workload verifier or run resources-server tests
>gym env publish               # finalize readiness and confirm registry discovery
>gym env start                 # start the servers
>gym env status                # show running servers
>
># Evaluation
>gym eval prepare              # prepare benchmark data and dump it to disk
>gym eval run                  # collate data, start servers, and collect rollouts
>gym eval aggregate            # merge sharded rollout results
>gym eval profile              # compute a reward profile from rollouts
>gym eval reverify             # recompute rewards from existing rollouts without re-running inference
>
># Contributor helpers
>gym dev test                  # run NeMo Gym's unit tests

Common Options

These options are shared across many commands.

OptionDescription
--config PATHLoad a Gym config YAML. Repeatable. Maps to +config_paths=[...].
--benchmark NAMESelect a registered benchmark by name instead of a config path.
--resources-server NAMESelect a registered resources server by name.
--model-type NAMESelect a registered model server type by name (such as openai_model or vllm_model).
--search-dir DIRExtra root directory to search for components and configs. Repeatable. Applies everywhere Gym resolves paths — discovery, the --<component> selectors, config_paths, prompt configs, and data files — and is inherited by spawned servers. Equivalent to the NEMO_GYM_EXTRA_ROOTS environment variable; see Configuration.
--jsonEmit machine-readable JSON instead of human-readable output (reporting commands only).
-v, --verboseSet the logging level to DEBUG. Flows through to spun-up servers.
-h, --helpShow help for any group or command.

The --benchmark, --resources-server, and --model-type selectors resolve a component name to its config file for you, so you do not need to know the project’s directory layout. If you mistype a name, the CLI suggests the closest match. To point at a config file directly, use --config <path> instead.

Selecting the model server

Commands that need a model (gym env start, gym eval run) configure it through four flags:

FlagDescription
--model-type NAMEThe model server type to load (such as openai_model, vllm_model, or local_vllm_model).
--model, -mThe served model identifier: an API model name, an HF id, or a local checkpoint path. Interpreted per --model-type. Maps to policy_model_name.
--model-urlBase URL of an existing model server endpoint. Maps to policy_base_url.
--model-api-keyAPI key for the model server. Maps to policy_api_key.

Hydra overrides (escape hatch)

The gym CLI is a thin wrapper over Gym’s Hydra config system. Standard flags (--flag value) cover the common inputs. Anything not covered by a flag can still be passed as a raw Hydra override using +key=value (add a new key) or ++key=value (add or override an existing key). Unknown overrides are forwarded to Hydra untouched, so advanced config composition is fully functional.

When a command needs overrides for keys that have no dedicated flag, you can keep the whole command in Hydra form (config paths included) or mix --flag and +key=value styles in one invocation:

$gym eval run \
>    --benchmark aime24 \
>    --model-type openai_model \
>    ++responses_create_params.reasoning.effort=low \
>    +wandb_project=gym-dev

General

gym --help

List all command groups and their descriptions.

$gym --help

gym --version

Print the NeMo Gym version along with Python, key dependency, and system information.

OptionDescription
--jsonOutput version information as JSON.
$gym --version
$
$# Output as JSON
$gym --version --json

Discovery

Commands for discovering benchmarks, environments, model servers, agent harnesses, and resources servers.

gym list <type> lists all components of that type; gym list <type> <name> inspects that one. All commands accept:

  • --json which prints machine-readable output instead of a human-readable table.
  • --search-dir which allows you to surface your own Gym-compatible components alongside the built-ins.

gym list benchmarks

List the benchmarks available in NeMo Gym, with their domain, agent, and configured number of repeats. The value in the Name column is the benchmark identifier and can be passed to --benchmark.

OptionDescription
[<name>]Inspect a single benchmark (config path, dataset, prepare script) instead of listing all.
--jsonOutput the benchmark list as JSON.
$gym list benchmarks
$gym list benchmarks aime24            # inspect one benchmark
$
$# Machine-readable output for scripting
$gym list benchmarks --json | jq '.[].name'

gym list environments

List the manifest-first catalog unioned with legacy environment and benchmark configs. Manifest-backed entries expose their declared metadata; unmigrated entries remain visible with no-manifest status.

OptionDescription
[<name>]Inspect one environment or benchmark.
--kind environment|benchmarkFilter the list or resolve an ambiguous name.
--domain, --modality, --licensingFilter by catalog metadata.
--status experimental|no-manifestFilter by manifest coverage status.
--lifecycle active|deprecatedFilter by lifecycle.
--jsonOutput catalog entries as JSON.
$gym list environments
$gym list environments calendar        # inspect one environment
$gym list environments --kind benchmark --domain math
$gym list environments --json | jq '.[].name'

gym list agents

List the agent harnesses under responses_api_agents/, with each one’s composition pattern — composable (Pattern A: references a separate resources server, so it can be wired into a matching environment) vs self-contained (Pattern B: ships its own framework/environment) — plus its config variants.

OptionDescription
[<name>]Inspect a single agent (composition pattern and config variants).
--jsonOutput the agent list as JSON (includes the self_contained flag).
$gym list agents
$gym list agents simple_agent          # inspect one agent
$gym list agents --json | jq '.[] | {name, self_contained}'

gym list models

List the model servers under responses_api_models/. The value in the Model column corresponds to a config flavor and can be passed as the --model-type flag to other commands.

OptionDescription
[<name>]Inspect a single model instead of listing all.
--jsonOutput the rows as JSON.
$gym list models
$gym list models vllm_model            # inspect one model
$gym list models --json | jq '.[].model'

gym list resources-servers

List the resources servers under resources_servers/. The value in the Name column corresponds to the config flavor and can be passed as --resources-server.

OptionDescription
[<name>]Inspect a single resources server (config path, domain, description).
--jsonOutput the list as JSON.
$gym list resources-servers
$gym list resources-servers mcqa       # inspect one resources server
$gym list resources-servers --json | jq '.[].name'

Filter a listing to fuzzy matches on a query (against each component’s name/value, description, domain, and type-specific fields). Without a type, the command searches the unified environment and benchmark catalog.

OptionDescription
[<type>]Optional component type: benchmarks, environments, agents, models, or resources-servers.
QUERYText matched (substring or fuzzy) against a component’s name, description, and key metadata (positional, required).
--jsonOutput matches as JSON.
$gym search math                       # environments and benchmarks matching "math"
$gym search benchmarks math            # only benchmarks matching "math"
$gym search environments calendar      # environments matching "calendar"
$gym search models vllm --json

Datasets

Commands for preparing, previewing, and managing datasets. By default dataset transfer commands use HuggingFace; pass --storage gitlab to target the GitLab Registry.

gym dataset upload

Upload a prepared local JSONL dataset to HuggingFace (default) or GitLab.

OptionDescription
--storage {hf,gitlab}Storage backend. Default: hf.
--input, -iLocal JSONL file to upload.
--nameDataset name.
--revisionDataset revision (version).
--splitDataset split (HF only).
--create-prOpen a pull request with your changes (HF only).
$# Upload to HuggingFace
$gym dataset upload \
>    --name my_dataset \
>    --input data/train.jsonl \
>    --revision 0.0.1
$
$# Upload to GitLab
$gym dataset upload \
>    --storage gitlab \
>    --name my_dataset \
>    --input data/train.jsonl \
>    --revision 0.0.1

gym dataset download

Download a dataset from HuggingFace (default) or GitLab.

OptionDescription
--storage {hf,gitlab}Storage backend. Default: hf.
--repo-idHF repo id, such as org/dataset (HF only).
--nameDataset name (GitLab only).
--revisionDataset version (GitLab only).
--artifactRemote file to fetch (GitLab: required; HF: optional raw file).
--output, -oLocal destination file.
--output-dirLocal destination directory; needed when downloading all splits (HF only).
--splitDataset split (HF only).
$# Download a single file from HuggingFace
$gym dataset download --repo-id NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \
>    --artifact train.jsonl \
>    --output data/train.jsonl
$
$# Download from GitLab
$gym dataset download \
>    --storage gitlab \
>    --name example_multi_step \
>    --revision 0.0.1 \
>    --artifact train.jsonl \
>    --output data/train.jsonl

gym dataset rm

Delete a dataset from the GitLab Registry. Prompts for confirmation.

OptionDescription
--nameName of the dataset to delete.
$gym dataset rm --name old_dataset

gym dataset migrate

Migrate a JSONL dataset to HuggingFace from GitLab. Use gym dataset upload if you do not want automatic GitLab deletion.

OptionDescription
--input, -iLocal JSONL file to upload to HF.
--nameDataset name.
--revisionDataset revision (HF).
--splitDataset split.
--create-prOpen a pull request to HF dataset with your changes.
$gym dataset migrate \
>    --name my_dataset \
>    --input data/train.jsonl \
>    --revision 0.0.1

gym dataset collate

Validate and collate a dataset, generating metrics and statistics.

OptionDescription
--config PATHConfig file to load. Repeatable.
--resources-server NAMELoad the named resources server config.
--search-dir DIRExtra root directory to search for named components. Repeatable.
--mode {train_preparation,example_validation}Use train_preparation to prepare train/validation datasets, or example_validation to validate example data.
--output-dirOutput directory for the prepared data.
--downloadDownload source datasets before collating.
$gym dataset collate \
>    --resources-server example_multi_step \
>    --output-dir data/example_multi_step \
>    --mode example_validation

gym dataset render

Generate a dataset preview by materializing prompts from a raw input file and a prompt template, producing JSONL with populated responses_create_params.input for RL training.

Each input row must not already have a populated responses_create_params.input; the command applies the prompt template from --prompt-config to each row, fills in the input, and preserves the row’s other fields.

OptionDescription
--input, -iRaw input JSONL file (rows without responses_create_params.input).
--prompt-configPrompt template YAML to apply.
--output, -oOutput JSONL file.
--search-dir DIRExtra root directory to search for named components. Repeatable.
$gym dataset render \
>    --input raw.jsonl \
>    --prompt-config prompt.yaml \
>    --output preview.jsonl

Which data-preparation command should I use?

  • gym dataset render — a focused, standalone step that applies a prompt template to raw rows to populate responses_create_params.input. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows.
  • gym dataset collate — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission.

Environments

Commands for developing, running, and inspecting environments. An environment defines tasks, verification, and its interaction surface; Gym config composes it with agent, model, and runtime components.

gym env init

Scaffold a resources server, or create a manifest-backed environment or benchmark skeleton. Workload scaffolds include a manifest, Gym config, sample data, and the extension points selected by the integration profile.

OptionDescription
--resources-server NAMEName of the resources server to create.
--environment NAME / --benchmark NAMECreate a manifest-backed environment or benchmark.
--profile PROFILEIntegration profile: custom-gym-verifier, custom-gym-agent-loop, external-agent-loop, or external-rollout-driver.
--reuse-verifier NAMEReuse an existing resources server that exports VERIFIER_FIXTURE.
--reward-range LOW HIGHReward endpoints required when reusing a verifier.
--higher-is-better / --lower-is-betterReward direction required when reusing a verifier.
$gym env init --resources-server my_server
$gym env init --environment my_eval --profile custom-gym-verifier
$gym env init --benchmark my_benchmark --reuse-verifier existing_scorer \
>    --reward-range 0 1 --higher-is-better

gym env resolve

Resolve the configs, flags, and overrides into a final merged config and print it. Useful for debugging configuration. Secrets are hidden.

Unlike gym env validate, resolve substitutes no dummy model, so a model config’s policy_* values must be supplied for its interpolations to resolve.

OptionDescription
--config PATHConfig file to load. Repeatable.
--search-dir DIRExtra root directory to search for named components. Repeatable.
$# Merge a resources-server config with a model config, apply an override, and print the result
$gym env resolve \
>    --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml \
>    --config responses_api_models/openai_model/configs/openai_model.yaml \
>    ++responses_create_params.temperature=0.6 \
>    ++policy_base_url=https://api.openai.com/v1 \
>    ++policy_api_key=sk-example \
>    ++policy_model_name=gpt-4o-mini

gym env validate

Validate a manifest-backed workload or legacy config without starting Ray or server subprocesses. Manifest validation resolves the Gym config offline, checks composition mirrors and datasets, and reports the resolved components. Legacy validation retains the existing config pre-flight behavior. Exits 0 when valid, or 1 with a clean message when not.

OptionDescription
NAMEManifest-backed catalog entry to validate.
--kind environment|benchmarkResolve an ambiguous catalog name.
--manifest PATHValidate a manifest directly instead of selecting by name.
--syncAtomically synchronize config-owned composition mirrors after all checks pass.
--jsonOutput the manifest validation report as JSON.
--config PATHConfig file to load. Repeatable.
--environment NAME / --benchmark NAMEValidate a named environment / benchmark config.
--resources-server NAMEValidate a named resources server config.
--model-type NAMEAlso load a named model config (otherwise a dummy policy_model is used).
--search-dir DIRExtra root directory to search for named components. Repeatable.
--model / --model-url / --model-api-keyOverride model name, base URL, and API key.
$gym env validate my_eval
$gym env validate --sync my_eval
$gym env validate --environment workplace_assistant
$gym env validate --benchmark gsm8k
$
$# or explicit config path(s)
$gym env validate --config resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml

gym env packages

Each server has its own isolated virtual environment. List the packages installed in a server’s environment.

OptionDescription
--resources-server NAMEName of the resources server.
--search-dir DIRExtra root directory to search for named components. Repeatable.
--outdatedList only outdated packages.
--jsonOutput the package list as JSON.
$gym env packages --resources-server example_single_tool_call
$
$# Check for outdated packages
$gym env packages \
>    --resources-server example_single_tool_call \
>    --outdated

gym env test

Exercise a manifest-backed workload’s verifier fixture in its resources-server environment, or run the existing pytest suite for one or all resources servers.

OptionDescription
NAMEManifest-backed catalog entry whose verifier fixture should run.
--kind environment|benchmarkResolve an ambiguous catalog name.
--update-expectedAtomically update fixture rewards after every behavioral check passes.
--jsonOutput the verifier report as JSON.
--resources-server NAMEResources server to test. Omit to test all servers.
--search-dir DIRExtra root directory to search for named components. Repeatable.
$# Exercise a manifest-backed workload verifier
$gym env test my_eval
$
$# Test a single server
$gym env test --resources-server example_single_tool_call
$
$# Test all servers
$gym env test

gym env publish

Run the local publication check for a manifest-backed workload. The command runs static validation and the verifier fixture, rejects manifest metadata placeholders, and confirms that the exact manifest is visible as experimental. It does not assess implementation quality, require a full evaluation or training run, commit, or push changes.

OptionDescription
NAMEManifest-backed catalog entry to publish.
--kind environment|benchmarkResolve an ambiguous catalog name.
--jsonOutput the publication result as JSON.
$gym env publish my_eval
$gym env publish my_benchmark --kind benchmark --json

gym env start

Start the NeMo Gym servers (agents, models, resources) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment.

OptionDescription
--config PATHConfig file to load. Repeatable.
--benchmark NAMELoad the named benchmark config (start its servers).
--resources-server NAMELoad the named resources server config.
--model-type NAMELoad the named model server type config.
--search-dir DIRExtra root directory to search for named components. Repeatable.
--model, -mServed model identifier. See Selecting the model server.
--model-urlModel server base URL.
--model-api-keyModel server API key.
$gym env start \
>    --resources-server example_single_tool_call \
>    --model-type openai_model
$
$# Start a benchmark's servers
>gym env start \
>    --benchmark gpqa \
>    --model-type vllm_model

gym env status

Show all currently running NeMo Gym servers and their health.

OptionDescription
--jsonOutput the server list as JSON.
$gym env status
NeMo Gym Server Status:
[1] ✓ example_single_tool_call (resources_servers/example_single_tool_call)
{
    'server_type': 'resources_servers',
    'name': 'example_single_tool_call',
    'port': 58117,
    'pid': 89904,
    'uptime_seconds': '0d 0h 0m 41.5s',
}
...
3 servers found (3 healthy, 0 unhealthy)

Evaluation

Commands for running evaluations end to end: prepare data, collect rollouts, aggregate for sharded runs, and profile.

gym eval prepare

Prepare a benchmark’s data by running its prepare.py script and dump the result to disk.

OptionDescription
--config PATHConfig file to load. Repeatable.
--benchmark NAMELoad the named benchmark config.
--search-dir DIRExtra root directory to search for named components. Repeatable.
$gym eval prepare --benchmark aime24

gym eval run

Collate data, start the servers, and collect rollouts. This is the main evaluation command. By default it spins up all required servers. Pass --no-serve to collect against servers you already started with gym env start.

OptionDescription
--config PATHConfig file to load. Repeatable.
--benchmark NAMELoad the named benchmark config.
--resources-server NAMELoad the named resources server config.
--model-type NAMELoad the named model server type config.
--search-dir DIRExtra root directory to search for named components. Repeatable.
--no-serveCollect against already-running servers instead of starting them.
--resumeResume from cached rollouts instead of recollecting. Maps to legacy +resume_from_cache=true. Refer to Resume interrupted runs.
--agent, -aAgent to collect rollouts with.
--input, -iInput tasks JSONL file.
--output, -oOutput rollouts JSONL file.
--limitMaximum number of tasks to run.
--num-repeatsRollouts per task (for mean@k metrics). Pass an int to apply to every task, or a dict keyed by agent_ref.name for per-agent counts (e.g. '{simple_agent: 32, swe_agent: 1}') when one input file mixes agents. In dict form, the special key _default is the fallback for agents not explicitly listed; without it, any unlisted row’s agent raises a single consolidated error.
--prompt-configPrompt template YAML to apply.
--concurrencyMaximum number of concurrent samples.
--splitDataset split to use (train, validation, or benchmark).
--model, -mServed model identifier.
--model-urlModel server base URL.
--model-api-keyModel server API key.
--temperatureSampling temperature.
--top-pNucleus sampling top-p.
--max-output-tokensMaximum output tokens.
$# End-to-end: spin up servers, then collect rollouts for a benchmark
$gym eval run --benchmark aime24 \
>    --model-type openai_model \
>    --output results/aime24.jsonl \
>    --split validation \
>    --concurrency 10
$
$# Against an already-running server, with a remote vLLM endpoint
$gym eval run --no-serve \
>    --model-type openai_model \
>    --resources-server math_with_judge \
>    --output results/test_001.jsonl \
>    --split validation \
>    --model openai/gpt-oss-120b \
>    --model-url http://0.0.0.0:10240/v1 \
>    --model-api-key dummy_key \
>    --temperature 1.0 \
>    --top-p 1.0
$
$# Per-agent repeats: one input file pins different agents per row via agent_ref.name
$gym eval run --no-serve \
>    --model-type openai_model \
>    --input mixed_agents.jsonl \
>    --output results/mixed_rollouts.jsonl \
>    --num-repeats '{agent_alpha: 4, agent_beta: 1, _default: 1}'

Generation parameters

The most common sampling parameters have dedicated flags on gym eval run: --temperature, --top-p, and --max-output-tokens. These map onto responses_create_params.temperature, responses_create_params.top_p, and responses_create_params.max_output_tokens.

Any other responses_create_params field that has no dedicated flag can be set with a raw Hydra override using the ++responses_create_params.<field> syntax. Overrides are merged into each input row’s existing responses_create_params with a shallow merge (top-level keys only):

$gym eval run --no-serve \
>    --agent example_single_tool_call_simple_agent \
>    --input weather_query.jsonl \
>    --output weather_rollouts.jsonl \
>    --temperature 1.0 \
>    --top-p 1.0 \
>    --max-output-tokens 4096 \
>    ++responses_create_params.reasoning.effort=low

Because the merge is shallow, setting a field inside a nested object, such as ++responses_create_params.reasoning.effort=low, replaces the row’s entire nested dictionary at that key. Other fields under the same nested object are not preserved.

Resume interrupted runs

Pass --resume to restart the same command after a crash or interruption and pick up only the rows that have not finished yet.

How it works:

  • Materialized inputs. On the first run, the fully expanded input rows (after --num-repeats, --limit, --prompt-config, and any overrides) are written to a sidecar file next to your output. The path is derived from --output by appending _materialized_inputs to the stem — so rollouts.jsonl produces rollouts_materialized_inputs.jsonl.
  • Incremental output. Successful rollouts are flushed to the main output JSONL after each completion; retriable failures go to a <stem>_failures.jsonl sidecar, so partial progress survives a crash.
  • Matching. On resume, completed work is matched by (task_index, rollout_index) against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run.
  • Fallback. If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. Without --resume, existing output is cleared before the run.

If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the *_materialized_inputs.jsonl file (and the output file) to start fresh.

gym eval aggregate

Merge sharded rollout results into a single rollouts file with aggregate metrics. Reads every JSONL file matching --input-glob, recomputes aggregate metrics over the global union of records, and writes a <output stem>_aggregate_metrics.json next to the merged rollouts. Use this to combine shards produced by gym eval run --no-serve +disable_aggregation=true.

OptionDescription
--config PATHConfig file to load. Repeatable.
--input-glob, -iGlob (or comma-separated globs) matching the rollout shards to aggregate.
--output, -oPath for the merged rollouts and aggregate-metrics file.
$gym eval aggregate \
>    --config benchmarks/aime24/config.yaml \
>    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
>    --input-glob 'results/rollouts-rs*-chunk*.jsonl' \
>    --output results/rollouts.jsonl

gym eval profile

Compute a reward profile from collected rollouts. Outputs per-task statistics such as average reward, standard deviation, min/max, and pass rate, useful for filtering tasks before training by difficulty or variance. Requires rollouts collected with --num-repeats greater than 1.

OptionDescription
--inputsMaterialized inputs JSONL fed to rollout collection.
--rolloutsRollouts JSONL produced by collection.
$gym eval profile \
>    --inputs materialized_inputs.jsonl \
>    --rollouts rollouts.jsonl

gym eval reverify

Recompute rewards from existing rollouts by replaying them through a resources server’s /verify endpoint — without re-running model inference. Starts the resources server automatically from the provided config. Requires the *_materialized_inputs.jsonl and rollouts.jsonl artifacts produced by gym eval run.

Before starting, the command checks each resources server’s GET /reverify_mode response. Resources servers report one of three modes:

ModeMeaning
statelessReverification is safe. The verifier is a pure function of (request body, server config).
unsupportedReverification is not safe. The verifier reads per-rollout session state that is no longer present.
unknownThe server has not declared its reverification behaviour. This is the default. Treated as potentially unsafe.

Servers that report unsupported or unknown cause the command to abort unless --force is passed.

OptionDescription
--config PATHConfig file for the resources server. Repeatable.
--benchmark NAMELoad the named benchmark config.
--resources-server NAMELoad the named resources server config.
--search-dir DIRExtra root directory to search for named components. Repeatable.
--inputs PATH*_materialized_inputs.jsonl from gym eval run.
--rollouts PATHrollouts.jsonl from gym eval run.
--output PATH, -oOutput JSONL for recomputed rollouts.
--forceOverride the UNSUPPORTED or UNKNOWN reverify mode guard; output is prefixed with unsafe_.
--overwriteDelete an existing output file instead of raising an error.
--resumeResume a partial run: skip rows already in the output and re-verify only the rest.
--judge-failed-onlyRecover only the rollouts whose judge call failed in the original run (from <rollouts>_failures.jsonl), reusing the stored responses. Skips the reverify-mode guard.
--appendWith --judge-failed-only: append recovered rows to an existing --output instead of a fresh file. Mutually exclusive with --overwrite.
--disable-aggregationSkip aggregate-metrics computation (use when combining shards with gym eval aggregate afterward).
$# Recompute rewards with an updated verifier config (no model server needed)
$gym eval reverify \
>    --config my_resources_server.yaml \
>    "++head_server.host=127.0.0.1" \
>    "++head_server.port=9500" \
>    --inputs results/rollouts_materialized_inputs.jsonl \
>    --rollouts results/rollouts.jsonl \
>    --output results/rollouts_reverified.jsonl
$
$# Override a verifier hyperparameter inline
$gym eval reverify \
>    --config my_resources_server.yaml \
>    "++mcqa.resources_servers.mcqa.grading_mode=lenient_boxed" \
>    --inputs results/rollouts_materialized_inputs.jsonl \
>    --rollouts results/rollouts.jsonl \
>    --output results/rollouts_lenient.jsonl
$
$# Re-run the judge on only the rollouts whose judge failed (reuses stored responses; no inference)
$gym eval reverify --judge-failed-only \
>    --config my_resources_server.yaml \
>    --inputs results/rollouts_materialized_inputs.jsonl \
>    --rollouts results/rollouts.jsonl \
>    --output results/rollouts_recovered.jsonl
$
$# Force through an UNSUPPORTED or UNKNOWN server (output prefixed with unsafe_)
$gym eval reverify \
>    --config my_stateful_server.yaml \
>    --inputs results/rollouts_materialized_inputs.jsonl \
>    --rollouts results/rollouts.jsonl \
>    --output results/rollouts_reverified.jsonl \
>    --force

See Reverify Rollouts for a full walkthrough.

Contributor Helpers

gym dev test

Run NeMo Gym’s core unit tests with coverage reporting.

$gym dev test

Migrating from the legacy commands

The legacy ng_* and nemo_gym_* are deprecated and will be removed in the future release. Use the tables below to find their gym replacement and update your scripts and workflows.

Command mapping

Legacy commandNew command
ng_helpgym --help
ng_versiongym --version
ng_list_benchmarksgym list benchmarks
ng_rungym env start
ng_statusgym env status
ng_dump_configgym env resolve
ng_pip_listgym env packages
ng_init_resources_servergym env init
ng_testgym env test
ng_test_allgym env test (no --resources-server)
ng_prepare_benchmarkgym eval prepare
ng_e2e_collect_rolloutsgym eval run
ng_collect_rolloutsgym eval run --no-serve
ng_aggregate_rolloutsgym eval aggregate
ng_reward_profilegym eval profile
ng_prepare_datagym dataset collate
ng_materialize_promptsgym dataset render
ng_upload_dataset_to_hfgym dataset upload
ng_upload_dataset_to_gitlabgym dataset upload --storage gitlab
ng_download_dataset_from_hfgym dataset download
ng_download_dataset_from_gitlabgym dataset download --storage gitlab
ng_gitlab_to_hf_datasetgym dataset migrate
ng_delete_dataset_from_gitlabgym dataset rm
ng_dev_testgym dev test
ng_reinstalluv sync --extra dev

Replacing Hydra overrides with flags

The most common Hydra overrides now have dedicated flags:

Legacy Hydra overrideNew flag
"+config_paths=[a.yaml,b.yaml]"--config a.yaml --config b.yaml
+agent_name=...--agent
+input_jsonl_fpath=...--input
+input_glob=...--input-glob
+output_jsonl_fpath=...--output
+limit=...--limit
+num_repeats=...--num-repeats
+num_samples_in_parallel=...--concurrency
++split=...--split
++policy_model_name=...--model
++policy_base_url=...--model-url
++policy_api_key=...--model-api-key
++responses_create_params.temperature=...--temperature
++responses_create_params.top_p=...--top-p
++responses_create_params.max_output_tokens=...--max-output-tokens
+mode=...--mode
+output_dirpath=...--output-dir
+should_download=true--download
+prompt_config=...--prompt-config
+resume_from_cache=true--resume
+dataset_name=...--name
+repo_id=...--repo-id
+revision=... / +version=...--revision
+artifact_fpath=...--artifact
+output_fpath=...--output
+create_pr=true--create-pr
+outdated=true--outdated

Common workflows, before and after

$# Start servers
$# Before:
$ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"
$# After:
$gym env start \
>    --resources-server example_single_tool_call \
>    --model-type openai_model
$
$# End-to-end rollout collection
$# Before:
$config_paths="responses_api_models/openai_model/configs/openai_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml"
$ng_e2e_collect_rollouts "+config_paths=[${config_paths}]" \
>    ++output_jsonl_fpath=results/aime24.jsonl \
>    ++split=validation
$# After:
$gym eval run \
>    --model-type openai_model \
>    --resources-server math_with_judge \
>    --output results/aime24.jsonl \
>    --split validation
$
$# Collect against already-running servers
$# Before:
$ng_collect_rollouts +agent_name=example_single_tool_call_simple_agent \
>    +input_jsonl_fpath=weather_query.jsonl \
>    +output_jsonl_fpath=weather_rollouts.jsonl \
>    +num_repeats=4 +num_samples_in_parallel=10
$# After:
$gym eval run --no-serve \
>    --agent example_single_tool_call_simple_agent \
>    --input weather_query.jsonl \
>    --output weather_rollouts.jsonl \
>    --num-repeats 4 \
>    --concurrency 10
$
$# Reward profiling
$# Before:
$ng_reward_profile +input_jsonl_fpath=materialized_inputs.jsonl +rollouts_jsonl_fpath=rollouts.jsonl
$# After:
$gym eval profile \
>    --inputs materialized_inputs.jsonl \
>    --rollouts rollouts.jsonl

Run any command with --help to see its full set of flags, and gym --help to list every group.