CLI Commands | NeMo Gym

This page documents all available NeMo Gym CLI commands.

Each command has both a short form (such as ng_run) and a full form (such as nemo_gym_run). They are functionally identical.

Quick Reference

$ # Display help
$ ng_help
$ 
$ # Get detailed help for any command
$ ng_run +help=true
$ ng_test +h=true

Server Management

Commands for running, testing, and managing NeMo Gym servers.

`ng_run` / `nemo_gym_run`

Start NeMo Gym servers for agents, models, and resources.

This command reads configuration from YAML files specified via +config_paths and starts all configured servers. The configuration files should define server instances with their entrypoints and settings.

Configuration Parameter

Parameter	Type	Description
`config_paths`	List[str]	Paths to YAML configuration files. Specify using Hydra: `+config_paths="[file1.yaml,file2.yaml]"`

Example

$ # Start servers with specific configs
$ config_paths="resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,\
> responses_api_models/openai_model/configs/openai_model.yaml"
$ ng_run "+config_paths=[${config_paths}]"

`ng_test` / `nemo_gym_test`

Test a specific server module by running its pytest suite and optionally validating example data.

Parameters

Parameter	Type	Description
`entrypoint`	str	Entrypoint for this command. Must be a relative path with two parts (such as `responses_api_agents/simple_agent`).
`should_validate_data`	bool	Whether to validate the example data (examples, metrics, rollouts, and so on) for this server. Default: `False`.

Example

$ ng_test +entrypoint=resources_servers/example_single_tool_call

`ng_test_all` / `nemo_gym_test_all`

Run tests for all server modules in the project.

Parameters

Parameter	Type	Description
`fail_on_total_and_test_mismatch`	bool	Fail if the number of server modules does not match the number with tests. Default: `False`.

Example

$ ng_test_all

`ng_dev_test` / `nemo_gym_dev_test`

Run core NeMo Gym tests with coverage reporting. Runs pytest with the --cov flag.

Example

$ ng_dev_test

`ng_init_resources_server` / `nemo_gym_init_resources_server`

Initialize a new resources server with template files and directory structure.

Example

$ ng_init_resources_server +entrypoint=resources_servers/my_server

Data Collection

Commands for collecting verified rollouts for RL training.

`ng_collect_rollouts` / `nemo_gym_collect_rollouts`

Perform a batch of rollout collection.

Parameters

Parameter	Type	Description
`agent_name`	str	The agent to collect rollouts from.
`input_jsonl_fpath`	str	The input data source to use to collect rollouts, in the form of a file path to a JSONL file.
`output_jsonl_fpath`	str	The output data JSONL file path.
`limit`	Optional[int]	Maximum number of examples to load and take from the input dataset.
`num_repeats`	Optional[int]	The number of times to repeat each example to run. Useful if you want to calculate mean@k, such as mean@4 or mean@16.
`num_repeats_add_seed`	bool	When num_repeats >1, add a “seed” parameter on the Responses create params.
`num_samples_in_parallel`	Optional[int]	Limit the number of concurrent samples running at once.
`responses_create_params`	Dict	Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`.

Example

$ ng_collect_rollouts \
>     +agent_name=example_single_tool_call_simple_agent \
>     +input_jsonl_fpath=weather_query.jsonl \
>     +output_jsonl_fpath=weather_rollouts.jsonl \
>     +limit=100 \
>     +num_repeats=4 \
>     +num_samples_in_parallel=10

`ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts`

Spin up all necessary servers and perform a batch of rollout collection using each dataset inside the provided configs.

Parameters

Parameter	Type	Description
`output_jsonl_fpath`	str	The output data JSONL file path.
`num_samples_in_parallel`	Optional[int]	Limit the number of concurrent samples running at once.
`responses_create_params`	Dict	Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`.

Examples

$ ng_e2e_collect_rollouts \
>     +output_jsonl_fpath=weather_rollouts.jsonl \
>     +num_samples_in_parallel=10

$ config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
> resources_servers/math_with_judge/configs/math_with_judge.yaml"
$ ng_e2e_collect_rollouts \
>     "+config_paths=[${config_paths}]" \
>     ++wandb_project= \
>     ++wandb_name= \
>     ++wandb_dir= \
>     ++output_jsonl_fpath=results/test_e2e_rollout_collection/aime24.jsonl \
>     ++split=validation

Example using GPT-OSS 120B remote vLLM endpoint

$ experiment_name=rollouts/test_001
$ config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
> resources_servers/math_with_judge/configs/math_with_judge.yaml"
$ ng_e2e_collect_rollouts \
>     "+config_paths=[${config_paths}]" \
>     +skip_venv_if_present=true \
>     +wandb_project=gym-dev \
>     +wandb_name=$(date +%Y%m%d)/$experiment_name \
>     ++output_jsonl_fpath=results/$experiment_name.jsonl \
>     ++overwrite_metrics_conflicts=true \
>     ++split=validation \
>     ++policy_model_name=openai/gpt-oss-120b \
>     ++policy_api_key=dummy_key \
>     ++policy_base_url=http://0.0.0.0:10240/v1 \
>     ++responses_create_params.reasoning.effort=low \
>     ++responses_create_params.temperature=1.0 \
>     ++responses_create_params.top_p=1.0 &> eval_gptoss120b.log &

`ng_reward_profile` / `nemo_gym_reward_profile`

Computes statistics on rewards and task difficulty for rollouts collected with ng_collect_rollouts with num_repeats >1. This outputs a new “reward profiled” dataset, where each task in the dataset has metrics like the average reward, standard deviation, min/max, and pass rate. This is useful in filtering tasks before training for difficulty, variance, or creating a curriculum.

Parameters

Parameter	Type	Description
`input_jsonl_fpath`	str	Path to the original task dataset JSONL file.
`rollouts_jsonl_fpath`	str	Path to the rollouts file from `ng_collect_rollouts` (must have been run with `num_repeats` >1).
`output_jsonl_fpath`	str	Output file path for the reward profiled dataset.
`pass_threshold`	Optional[float]	Reward threshold for computing pass rate. If not specified, pass rate metrics are not included.

Output Fields

Each output row contains all original task fields plus:

avg_reward: Average reward across all rollouts
std_reward: Standard deviation of rewards
min_reward: Minimum reward observed
max_reward: Maximum reward observed
total_samples: Number of rollout samples
pass_rate, pass_rate_total, pass_rate_passed, pass_threshold: (Only if pass_threshold is specified)

Example

$ ng_reward_profile \
>     +input_jsonl_fpath=tasks.jsonl \
>     +rollouts_jsonl_fpath=rollouts.jsonl \
>     +output_jsonl_fpath=profiled_tasks.jsonl \
>     +pass_threshold=1.0

Data Management

Commands for preparing and viewing training data.

`ng_prepare_data` / `nemo_gym_prepare_data`

Prepare and validate training data, generating metrics and statistics for datasets.

Parameters

Parameter	Type	Description
`output_dirpath`	str	Directory path where processed datasets and metrics will be saved.
`mode`	Literal[“train_preparation”, “example_validation”]	Processing mode. Use `train_preparation` to prepare train and validation datasets for training, or `example_validation` to validate example data for PR submission.
`should_download`	bool	Whether to automatically download missing datasets from remote registries. Default: `False`.
`overwrite_metrics_conflicts`	bool	Whether or not to overwrite metrics conflicts. Default: `False`.

Example

$ config_paths="resources_servers/example_multi_step/configs/example_multi_step.yaml,\
> responses_api_models/openai_model/configs/openai_model.yaml"
$ ng_prepare_data "+config_paths=[${config_paths}]" \
>     +output_dirpath=data/example_multi_step \
>     +mode=example_validation

Dataset Registry - GitLab

Commands for uploading, downloading, and managing datasets in GitLab Model Registry.

`ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab`

Upload a local JSONL dataset artifact to GitLab.

Parameters

Parameter	Type	Description
`dataset_name`	str	The dataset name.
`version`	str	The version of this dataset. Must be in the format `x.x.x`.
`input_jsonl_fpath`	str	Path to the JSONL file to upload.

Example

$ ng_upload_dataset_to_gitlab \
>     +dataset_name=example_multi_step \
>     +version=0.0.1 \
>     +input_jsonl_fpath=data/train.jsonl

`ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab`

Download a JSONL dataset from GitLab Model Registry.

Parameters

Parameter	Type	Description
`dataset_name`	str	The dataset name.
`version`	str	The version of this dataset. Must be in the format `x.x.x`.
`artifact_fpath`	str	The filepath to the artifact to download.
`output_fpath`	str	Path where the downloaded dataset will be saved.

Example

$ ng_download_dataset_from_gitlab \
>     +dataset_name=example_multi_step \
>     +version=0.0.1 \
>     +artifact_fpath=train.jsonl \
>     +output_fpath=data/train.jsonl

`ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab`

Delete a dataset from GitLab Model Registry. Prompts for confirmation.

Parameters

Parameter	Type	Description
`dataset_name`	str	Name of the dataset to delete from GitLab.

Example

$ ng_delete_dataset_from_gitlab +dataset_name=old_dataset

Dataset Registry - HuggingFace

Commands for uploading and downloading datasets to/from HuggingFace Hub.

`ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf`

Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.

Parameters

Parameter	Type	Description
`hf_token`	str	HuggingFace API token for authentication.
`hf_organization`	str	HuggingFace organization name where the dataset will be uploaded.
`hf_collection_name`	str	HuggingFace collection name for organizing datasets.
`hf_collection_slug`	str	Alphanumeric collection slug found at the end of the collection URI.
`dataset_name`	str	Name of the dataset. Will be combined with domain and resources server name.
`input_jsonl_fpath`	str	Path to the local JSONL file to upload.
`resource_config_path`	str	Path to resources server config file. Used to extract domain for naming convention.
`hf_dataset_prefix`	str	Prefix prepended to dataset name. Default: `NeMo-Gym`.
`delete_from_gitlab`	Optional[bool]	Delete the dataset from GitLab after successful upload to HuggingFace. Default: `False`.

Example

$ resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
$ ng_upload_dataset_to_hf \
>     +dataset_name=my_dataset \
>     +input_jsonl_fpath=data/train.jsonl \
>     +resource_config_path=${resource_config_path} \
>     +delete_from_gitlab=true

`ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf`

Download a JSONL dataset from HuggingFace Hub to local filesystem.

Parameters

Parameter	Type	Description
`output_fpath`	str	Local file path where the downloaded dataset will be saved.
`hf_token`	str	HuggingFace API token for authentication.
`artifact_fpath`	str	Name of the artifact file to download from the repository.
`repo_id`	str	HuggingFace repository ID in format `organization/dataset-name`.

Example

$ ng_download_dataset_from_hf \
>     +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \
>     +artifact_fpath=train.jsonl \
>     +output_fpath=data/train.jsonl

`ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset`

Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.

This command always deletes the dataset from GitLab after uploading to HuggingFace. Use ng_upload_dataset_to_hf if you want optional deletion control.

Parameters

Same as ng_upload_dataset_to_hf but delete_from_gitlab is not available. This command always deletes.

Example

$ resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
$ ng_gitlab_to_hf_dataset \
>     +dataset_name=my_dataset \
>     +input_jsonl_fpath=data/train.jsonl \
>     +resource_config_path=${resource_config_path}

Configuration & Help

Commands for debugging configuration and getting help.

`ng_dump_config` / `nemo_gym_dump_config`

Display the resolved Hydra configuration for debugging purposes.

Example

$ ng_dump_config "+config_paths=[<config1>,<config2>]"

`ng_help` / `nemo_gym_help`

Display a list of available NeMo Gym CLI commands.

Example

$ ng_help

`ng_version` / `nemo_gym_version`

Display NeMo Gym version and system information.

Parameters

Parameter	Type	Description
`json_format`	bool	Output in JSON format for programmatic use. Default: `False`. Can be specified with `+json=true`.

Example

$ # Display version information
$ ng_version
$ 
$ # Output as JSON
$ ng_version +json=true

`ng_pip_list` / `nemo_gym_pip_list`

Each server has its own isolated virtual environment. To inspect the packages:

Parameters

Parameter	Type	Description
`entrypoint`	str	The relative entrypoint path to the server directory
`format`	Optional[str]	Output format for pip list. Options: ‘columns’ (default), ‘freeze’, ‘json’. Default: `None`.
`outdated`	bool	List outdated packages. Default: `False`.

Examples

$ # List all packages
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call
$ 
$ # Output as JSON
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call +format=json
$ 
$ # Check for outdated packages
$ ng_pip_list +entrypoint=resources_servers/example_single_tool_call +outdated=true

`ng_status` / `nemo_gym_status`

View all currently running NeMo Gym servers and their health status.

Example

$ ng_status
$ 
$ NeMo Gym Server Status:
$ 
$ [1] ✓ example_single_tool_call (resources_servers/example_single_tool_call)
$ {
>     'server_type': 'resources_servers',
>     'name': 'example_single_tool_call',
>     'port': 58117,
>     'pid': 89904,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ [2] ✓ example_single_tool_call_simple_agent (responses_api_agents/simple_agent)
$ {
>     'server_type': 'responses_api_agents',
>     'name': 'simple_agent',
>     'port': 58118,
>     'pid': 89905,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ [3] ✓ policy_model (responses_api_models/openai_model)
$ {
>     'server_type': 'responses_api_models',
>     'name': 'openai_model',
>     'port': 58119,
>     'pid': 89907,
>     'uptime_seconds': '0d 0h 0m 41.5s',
> }
$ 
$ 3 servers found (3 healthy, 0 unhealthy)

Getting Help

For detailed help on any command, run it with +help=true or +h=true:

$ ng_run +help=true
$ ng_collect_rollouts +h=true

This will display all available configuration parameters and their descriptions.

Re-install Gym and dependencies

$ ng_reinstall

This will re-install Gym and its dependencies into the currently activated Python virtual environment.

Quick Reference

Server Management

ng_run / nemo_gym_run

ng_test / nemo_gym_test

ng_test_all / nemo_gym_test_all

ng_dev_test / nemo_gym_dev_test

ng_init_resources_server / nemo_gym_init_resources_server

Data Collection

ng_collect_rollouts / nemo_gym_collect_rollouts

ng_e2e_collect_rollouts / nemo_gym_e2e_collect_rollouts

ng_reward_profile / nemo_gym_reward_profile

Data Management

ng_prepare_data / nemo_gym_prepare_data

Dataset Registry - GitLab

ng_upload_dataset_to_gitlab / nemo_gym_upload_dataset_to_gitlab

ng_download_dataset_from_gitlab / nemo_gym_download_dataset_from_gitlab

ng_delete_dataset_from_gitlab / nemo_gym_delete_dataset_from_gitlab

Dataset Registry - HuggingFace

ng_upload_dataset_to_hf / nemo_gym_upload_dataset_to_hf

ng_download_dataset_from_hf / nemo_gym_download_dataset_from_hf

ng_gitlab_to_hf_dataset / nemo_gym_gitlab_to_hf_dataset

Configuration & Help

ng_dump_config / nemo_gym_dump_config

ng_help / nemo_gym_help

ng_version / nemo_gym_version

ng_pip_list / nemo_gym_pip_list

ng_status / nemo_gym_status

Getting Help

Re-install Gym and dependencies

`ng_run` / `nemo_gym_run`

`ng_test` / `nemo_gym_test`

`ng_test_all` / `nemo_gym_test_all`

`ng_dev_test` / `nemo_gym_dev_test`

`ng_init_resources_server` / `nemo_gym_init_resources_server`

`ng_collect_rollouts` / `nemo_gym_collect_rollouts`

`ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts`

`ng_reward_profile` / `nemo_gym_reward_profile`

`ng_prepare_data` / `nemo_gym_prepare_data`

`ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab`

`ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab`

`ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab`

`ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf`

`ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf`

`ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset`

`ng_dump_config` / `nemo_gym_dump_config`

`ng_help` / `nemo_gym_help`

`ng_version` / `nemo_gym_version`

`ng_pip_list` / `nemo_gym_pip_list`

`ng_status` / `nemo_gym_status`