CLI Commands | NeMo Gym

This page documents all available NeMo Gym CLI commands.

Each command has both a short form (such as ng_run) and a full form (such as nemo_gym_run). They are functionally identical.

Quick Reference

$ # Display help
$ ng_help
$ 
$ # Get detailed help for any command
$ ng_run +help=true
$ ng_test +h=true

Server Management

Commands for running, testing, and managing NeMo Gym servers.

`ng_run` / `nemo_gym_run`

Start NeMo Gym servers for agents, models, and resources.

This command reads configuration from YAML files specified via +config_paths and starts all configured servers. The configuration files should define server instances with their entrypoints and settings.

Configuration Parameter

Parameter	Type	Description
`config_paths`	List[str]	Paths to YAML configuration files. Specify using Hydra: `+config_paths="[file1.yaml,file2.yaml]"`

Example

$ # Start servers with specific configs
$ config_paths="resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,\
> responses_api_models/openai_model/configs/openai_model.yaml"
$ ng_run "+config_paths=[${config_paths}]"

`ng_test` / `nemo_gym_test`

Test a specific server module by running its pytest suite and optionally validating example data.

Parameters

Parameter	Type	Description
`entrypoint`	str	Entrypoint for this command. Must be a relative path with two parts (such as `responses_api_agents/simple_agent`).
`should_validate_data`	bool	Whether to validate the example data (examples, metrics, rollouts, and so on) for this server. Default: `False`.

Example

$ ng_test +entrypoint=resources_servers/example_single_tool_call

`ng_test_all` / `nemo_gym_test_all`

Run tests for all server modules in the project.

Parameters

Parameter	Type	Description
`fail_on_total_and_test_mismatch`	bool	Fail if the number of server modules does not match the number with tests. Default: `False`.

Example

$ ng_test_all

`ng_dev_test` / `nemo_gym_dev_test`

Run core NeMo Gym tests with coverage reporting. Runs pytest with the --cov flag.

Example

$ ng_dev_test

`ng_init_resources_server` / `nemo_gym_init_resources_server`

Initialize a new resources server with template files and directory structure.

Example

$ ng_init_resources_server +entrypoint=resources_servers/my_server

Data Collection

Commands for collecting verified rollouts for RL training.

`ng_collect_rollouts` / `nemo_gym_collect_rollouts`

Perform a batch of rollout collection.

Parameters

Parameter	Type	Description
`agent_name`	str	The agent to collect rollouts from.
`input_jsonl_fpath`	str	The input data source to use to collect rollouts, in the form of a file path to a JSONL file.
`output_jsonl_fpath`	str	The output data JSONL file path.
`limit`	Optional[int]	Maximum number of examples to load and take from the input dataset.
`num_repeats`	Optional[int]	The number of times to repeat each example to run. Useful if you want to calculate mean@k, such as mean@4 or mean@16.
`num_repeats_add_seed`	bool	When num_repeats >1, add a “seed” parameter on the Responses create params.
`num_samples_in_parallel`	Optional[int]	Limit the number of concurrent samples running at once.
`responses_create_params`	Dict	Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to Generation Parameters.
`resume_from_cache`	bool	Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to Resume Interrupted Runs.

Example

$ ng_collect_rollouts \
>     +agent_name=example_single_tool_call_simple_agent \
>     +input_jsonl_fpath=weather_query.jsonl \
>     +output_jsonl_fpath=weather_rollouts.jsonl \
>     +limit=100 \
>     +num_repeats=4 \
>     +num_samples_in_parallel=10

Generation Parameters

Sampling parameters such as temperature, max_output_tokens, and top_p are not standalone CLI flags — they are passed as overrides inside responses_create_params using Hydra’s nested dot syntax. Overrides are merged into each input row’s existing responses_create_params with a shallow merge (top-level keys only):

$ ng_collect_rollouts \
>     +agent_name=example_single_tool_call_simple_agent \
>     +input_jsonl_fpath=weather_query.jsonl \
>     +output_jsonl_fpath=weather_rollouts.jsonl \
>     ++responses_create_params.temperature=1.0 \
>     ++responses_create_params.top_p=1.0 \
>     ++responses_create_params.max_output_tokens=4096

The same syntax works for ng_e2e_collect_rollouts. Top-level fields such as temperature and max_output_tokens are straightforward. For nested objects (for example, ++responses_create_params.reasoning.effort=low), the entire nested dict replaces the row’s existing value at that key — other fields under the same nested object are not preserved.

Resume Interrupted Runs

Setting +resume_from_cache=true lets you restart the same command after a crash or interruption and pick up only the rows that have not finished yet. It works for both ng_collect_rollouts and ng_e2e_collect_rollouts, across any environment.

How it works:

Materialized inputs. On the first run, the fully expanded input rows (after num_repeats, limit, prompt_config, and any overrides) are written to a sidecar file next to your output. The path is derived from output_jsonl_fpath by appending _materialized_inputs to the stem — so rollouts.jsonl produces rollouts_materialized_inputs.jsonl.
Incremental output. Results are flushed to output_jsonl_fpath after each completed rollout, so partial output survives a crash.
Matching. On resume, completed work is matched by (task_index, rollout_index) against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run.
Fallback. If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. With the default resume_from_cache=False, existing output is cleared before the run.

If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the *_materialized_inputs.jsonl file (and the output file) to start fresh.

`ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts`

Spin up all necessary servers and perform a batch of rollout collection using each dataset inside the provided configs.

Parameters

Parameter	Type	Description
`output_jsonl_fpath`	str	The output data JSONL file path.
`num_samples_in_parallel`	Optional[int]	Limit the number of concurrent samples running at once.
`responses_create_params`	Dict	Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to Generation Parameters.
`resume_from_cache`	bool	Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to Resume Interrupted Runs.

Examples

$ ng_e2e_collect_rollouts \
>     +output_jsonl_fpath=weather_rollouts.jsonl \
>     +num_samples_in_parallel=10

$ config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
> resources_servers/math_with_judge/configs/math_with_judge.yaml"
$ ng_e2e_collect_rollouts \
>     "+config_paths=[${config_paths}]" \
>     ++wandb_project= \
>     ++wandb_name= \
>     ++wandb_dir= \
>     ++output_jsonl_fpath=results/test_e2e_rollout_collection/aime24.jsonl \
>     ++split=validation

Example using GPT-OSS 120B remote vLLM endpoint

$ experiment_name=rollouts/test_001
$ config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
> resources_servers/math_with_judge/configs/math_with_judge.yaml"
$ ng_e2e_collect_rollouts \
>     "+config_paths=[${config_paths}]" \
>     +skip_venv_if_present=true \
>     +wandb_project=gym-dev \
>     +wandb_name=$(date +%Y%m%d)/$experiment_name \
>     ++output_jsonl_fpath=results/$experiment_name.jsonl \
>     ++overwrite_metrics_conflicts=true \
>     ++split=validation \
>     ++policy_model_name=openai/gpt-oss-120b \
>     ++policy_api_key=dummy_key \
>     ++policy_base_url=http://0.0.0.0:10240/v1 \
>     ++responses_create_params.reasoning.effort=low \
>     ++responses_create_params.temperature=1.0 \
>     ++responses_create_params.top_p=1.0 &> eval_gptoss120b.log &

`ng_reward_profile` / `nemo_gym_reward_profile`

Computes statistics on rewards and task difficulty for rollouts collected with ng_collect_rollouts with num_repeats >1. This outputs a new “reward profiled” dataset, where each task in the dataset has metrics like the average reward, standard deviation, min/max, and pass rate. This is useful in filtering tasks before training for difficulty, variance, or creating a curriculum.

Parameters

Parameter	Type	Description
`input_jsonl_fpath`	str	Path to the original task dataset JSONL file.
`rollouts_jsonl_fpath`	str	Path to the rollouts file from `ng_collect_rollouts` (must have been run with `num_repeats` >1).
`output_jsonl_fpath`	str	Output file path for the reward profiled dataset.
`pass_threshold`	Optional[float]	Reward threshold for computing pass rate. If not specified, pass rate metrics are not included.

Output Fields

Each output row contains all original task fields plus:

avg_reward: Average reward across all rollouts
std_reward: Standard deviation of rewards
min_reward: Minimum reward observed
max_reward: Maximum reward observed
total_samples: Number of rollout samples
pass_rate, pass_rate_total, pass_rate_passed, pass_threshold: (Only if pass_threshold is specified)

Example

$ ng_reward_profile \
>     +input_jsonl_fpath=tasks.jsonl \
>     +rollouts_jsonl_fpath=rollouts.jsonl \
>     +output_jsonl_fpath=profiled_tasks.jsonl \
>     +pass_threshold=1.0

Data Management

Commands for preparing and viewing training data.

`ng_prepare_data` / `nemo_gym_prepare_data`

Prepare and validate training data, generating metrics and statistics for datasets.

Parameters

Parameter	Type	Description
`output_dirpath`	str	Directory path where processed datasets and metrics will be saved.
`mode`	Literal[“train_preparation”, “example_validation”]	Processing mode. Use `train_preparation` to prepare train and validation datasets for training, or `example_validation` to validate example data for PR submission.
`should_download`	bool	Whether to automatically download missing datasets from remote registries. Default: `False`.
`overwrite_metrics_conflicts`	bool	Whether or not to overwrite metrics conflicts. Default: `False`.

Example

$ config_paths="resources_servers/example_multi_step/configs/example_multi_step.yaml,\
> responses_api_models/openai_model/configs/openai_model.yaml"
$ ng_prepare_data "+config_paths=[${config_paths}]" \
>     +output_dirpath=data/example_multi_step \
>     +mode=example_validation

`ng_materialize_prompts` / `nemo_gym_materialize_prompts`

Apply a prompt template to raw JSONL data, producing materialized JSONL with populated responses_create_params.input for RL training.

Each input row must not already have a populated responses_create_params.input; the command applies the prompt template from prompt_config to each row, fills in the input, and preserves the row’s other fields.

Parameters

Parameter	Type	Description
`input_jsonl_fpath`	str	Raw JSONL data (rows without `responses_create_params.input`).
`prompt_config`	str	Path to the prompt YAML file to apply.
`output_jsonl_fpath`	str	Output path for the materialized JSONL with populated prompts.

Example

$ ng_materialize_prompts \
>     +input_jsonl_fpath=data/my_dataset.jsonl \
>     +prompt_config=/path/to/my_prompt.yaml \
>     +output_jsonl_fpath=my_dataset_materialized.jsonl

Which data-preparation command should I use?

ng_materialize_prompts — a focused, standalone step that applies a prompt template to raw rows to populate responses_create_params.input. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows.
ng_prepare_data — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission.

Dataset Registry - GitLab

Commands for uploading, downloading, and managing datasets in GitLab Model Registry.

`ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab`

Upload a local JSONL dataset artifact to GitLab.

Parameters

Parameter	Type	Description
`dataset_name`	str	The dataset name.
`version`	str	The version of this dataset. Must be in the format `x.x.x`.
`input_jsonl_fpath`	str	Path to the JSONL file to upload.

Example

$ ng_upload_dataset_to_gitlab \
>     +dataset_name=example_multi_step \
>     +version=0.0.1 \
>     +input_jsonl_fpath=data/train.jsonl

`ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab`

Download a JSONL dataset from GitLab Model Registry.

Parameters

Parameter	Type	Description
`dataset_name`	str	The dataset name.
`version`	str	The version of this dataset. Must be in the format `x.x.x`.
`artifact_fpath`	str	The filepath to the artifact to download.
`output_fpath`	str	Path where the downloaded dataset will be saved.

Example

$ ng_download_dataset_from_gitlab \
>     +dataset_name=example_multi_step \
>     +version=0.0.1 \
>     +artifact_fpath=train.jsonl \
>     +output_fpath=data/train.jsonl

`ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab`

Delete a dataset from GitLab Model Registry. Prompts for confirmation.

Parameters

Parameter	Type	Description
`dataset_name`	str	Name of the dataset to delete from GitLab.

Example

$ ng_delete_dataset_from_gitlab +dataset_name=old_dataset

Dataset Registry - HuggingFace

Commands for uploading and downloading datasets to/from HuggingFace Hub.

`ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf`

Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.

Parameters

Parameter	Type	Description
`hf_token`	str	HuggingFace API token for authentication.
`hf_organization`	str	HuggingFace organization name where the dataset will be uploaded.
`hf_collection_name`	str	HuggingFace collection name for organizing datasets.
`hf_collection_slug`	str	Alphanumeric collection slug found at the end of the collection URI.
`dataset_name`	str	Name of the dataset. Will be combined with domain and resources server name.
`input_jsonl_fpath`	str	Path to the local JSONL file to upload.
`resource_config_path`	str	Path to resources server config file. Used to extract domain for naming convention.
`hf_dataset_prefix`	str	Prefix prepended to dataset name. Default: `NeMo-Gym`.
`delete_from_gitlab`	Optional[bool]	Delete the dataset from GitLab after successful upload to HuggingFace. Default: `False`.

Example

$ resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
$ ng_upload_dataset_to_hf \
>     +dataset_name=my_dataset \
>     +input_jsonl_fpath=data/train.jsonl \
>     +resource_config_path=${resource_config_path} \
>     +delete_from_gitlab=true

`ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf`

Download a JSONL dataset from HuggingFace Hub to local filesystem.

Parameters

Parameter	Type	Description
`output_fpath`	str	Local file path where the downloaded dataset will be saved.
`hf_token`	str	HuggingFace API token for authentication.
`artifact_fpath`	str	Name of the artifact file to download from the repository.
`repo_id`	str	HuggingFace repository ID in format `organization/dataset-name`.

Example

$ ng_download_dataset_from_hf \
>     +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \
>     +artifact_fpath=train.jsonl \
>     +output_fpath=data/train.jsonl

`ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset`

Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.

This command always deletes the dataset from GitLab after uploading to HuggingFace. Use ng_upload_dataset_to_hf if you want optional deletion control.

Parameters

Same as ng_upload_dataset_to_hf but delete_from_gitlab is not available. This command always deletes.

Example

$ resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
$ ng_gitlab_to_hf_dataset \
>     +dataset_name=my_dataset \
>     +input_jsonl_fpath=data/train.jsonl \
>     +resource_config_path=${resource_config_path}

Configuration & Help

Commands for debugging configuration and getting help.

`ng_dump_config` / `nemo_gym_dump_config`

Display the resolved Hydra configuration for debugging purposes.

Example

$ ng_dump_config "+config_paths=[<config1>,<config2>]"

`ng_help` / `nemo_gym_help`

Display a list of available NeMo Gym CLI commands.

Example

$ ng_help

`ng_version` / `nemo_gym_version`

Display NeMo Gym version and system information.

Parameters

Parameter	Type	Description
`json_format`	bool	Output in JSON format for programmatic use. Default: `False`. Can be specified with `+json=true`.

Example

$ # Display version information
$ ng_version
$ 
$ # Output as JSON
$ ng_version +json=true

`ng_pip_list` / `nemo_gym_pip_list`

Each server has its own isolated virtual environment. To inspect the packages:

Parameters

Parameter	Type	Description
`entrypoint`	str	The relative entrypoint path to the server directory
`format`	Optional[str]	Output format for pip list. Options: ‘columns’ (default), ‘freeze’, ‘json’. Default: `None`.
`outdated`	bool	List outdated packages. Default: `False`.

Examples

$ # List all packages
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call
$ 
$ # Output as JSON
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call +format=json
$ 
$ # Check for outdated packages
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call +outdated=true

`ng_status` / `nemo_gym_status`

View all currently running NeMo Gym servers and their health status.

Example

$ ng_status
$ 
$ NeMo Gym Server Status:
$ 
$ [1] ✓ example_single_tool_call (resources_servers/example_single_tool_call)
$ {
>     'server_type': 'resources_servers',
>     'name': 'example_single_tool_call',
>     'port': 58117,
>     'pid': 89904,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ [2] ✓ example_single_tool_call_simple_agent (responses_api_agents/simple_agent)
$ {
>     'server_type': 'responses_api_agents',
>     'name': 'simple_agent',
>     'port': 58118,
>     'pid': 89905,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ [3] ✓ policy_model (responses_api_models/openai_model)
$ {
>     'server_type': 'responses_api_models',
>     'name': 'openai_model',
>     'port': 58119,
>     'pid': 89907,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ 
$ 3 servers found (3 healthy, 0 unhealthy)

Getting Help

For detailed help on any command, run it with +help=true or +h=true:

$ ng_run +help=true
$ ng_collect_rollouts +h=true

This will display all available configuration parameters and their descriptions.

Re-install Gym and dependencies

$ ng_reinstall

This will re-install Gym and its dependencies into the currently activated Python virtual environment.

Quick Reference

Server Management

ng_run / nemo_gym_run

ng_test / nemo_gym_test

ng_test_all / nemo_gym_test_all

ng_dev_test / nemo_gym_dev_test

ng_init_resources_server / nemo_gym_init_resources_server

Data Collection

ng_collect_rollouts / nemo_gym_collect_rollouts

Generation Parameters

Resume Interrupted Runs

ng_e2e_collect_rollouts / nemo_gym_e2e_collect_rollouts

ng_reward_profile / nemo_gym_reward_profile

Data Management

ng_prepare_data / nemo_gym_prepare_data

ng_materialize_prompts / nemo_gym_materialize_prompts

Dataset Registry - GitLab

ng_upload_dataset_to_gitlab / nemo_gym_upload_dataset_to_gitlab

ng_download_dataset_from_gitlab / nemo_gym_download_dataset_from_gitlab

ng_delete_dataset_from_gitlab / nemo_gym_delete_dataset_from_gitlab

Dataset Registry - HuggingFace

ng_upload_dataset_to_hf / nemo_gym_upload_dataset_to_hf

ng_download_dataset_from_hf / nemo_gym_download_dataset_from_hf

ng_gitlab_to_hf_dataset / nemo_gym_gitlab_to_hf_dataset

Configuration & Help

ng_dump_config / nemo_gym_dump_config

ng_help / nemo_gym_help

ng_version / nemo_gym_version

ng_pip_list / nemo_gym_pip_list

ng_status / nemo_gym_status

Getting Help

Re-install Gym and dependencies

`ng_run` / `nemo_gym_run`

`ng_test` / `nemo_gym_test`

`ng_test_all` / `nemo_gym_test_all`

`ng_dev_test` / `nemo_gym_dev_test`

`ng_init_resources_server` / `nemo_gym_init_resources_server`

`ng_collect_rollouts` / `nemo_gym_collect_rollouts`

`ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts`

`ng_reward_profile` / `nemo_gym_reward_profile`

`ng_prepare_data` / `nemo_gym_prepare_data`

`ng_materialize_prompts` / `nemo_gym_materialize_prompts`

`ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab`

`ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab`

`ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab`

`ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf`

`ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf`

`ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset`

`ng_dump_config` / `nemo_gym_dump_config`

`ng_help` / `nemo_gym_help`

`ng_version` / `nemo_gym_version`

`ng_pip_list` / `nemo_gym_pip_list`

`ng_status` / `nemo_gym_status`