> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # CLI Commands This page documents all available NeMo Gym CLI commands. Each command has both a short form (such as `ng_run`) and a full form (such as `nemo_gym_run`). They are functionally identical. ## Quick Reference ```bash # Display help ng_help # Get detailed help for any command ng_run +help=true ng_test +h=true ``` *** ## Server Management Commands for running, testing, and managing NeMo Gym servers. ### `ng_run` / `nemo_gym_run` Start NeMo Gym servers for agents, models, and resources. This command reads configuration from YAML files specified via `+config_paths` and starts all configured servers. The configuration files should define server instances with their entrypoints and settings. **Configuration Parameter** | Parameter | Type | Description | | -------------- | ---------- | ------------------------------------------------------------------------------------------------- | | `config_paths` | List\[str] | Paths to YAML configuration files. Specify using Hydra: `+config_paths="[file1.yaml,file2.yaml]"` | **Example** ```bash # Start servers with specific configs config_paths="resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,\ responses_api_models/openai_model/configs/openai_model.yaml" ng_run "+config_paths=[${config_paths}]" ``` *** ### `ng_test` / `nemo_gym_test` Test a specific server module by running its pytest suite and optionally validating example data. **Parameters** | Parameter | Type | Description | | ---------------------- | ---- | ------------------------------------------------------------------------------------------------------------------ | | `entrypoint` | str | Entrypoint for this command. Must be a relative path with two parts (such as `responses_api_agents/simple_agent`). | | `should_validate_data` | bool | Whether to validate the example data (examples, metrics, rollouts, and so on) for this server. Default: `False`. | **Example** ```bash ng_test +entrypoint=resources_servers/example_single_tool_call ``` *** ### `ng_test_all` / `nemo_gym_test_all` Run tests for all server modules in the project. **Parameters** | Parameter | Type | Description | | --------------------------------- | ---- | -------------------------------------------------------------------------------------------- | | `fail_on_total_and_test_mismatch` | bool | Fail if the number of server modules does not match the number with tests. Default: `False`. | **Example** ```bash ng_test_all ``` *** ### `ng_dev_test` / `nemo_gym_dev_test` Run core NeMo Gym tests with coverage reporting. Runs pytest with the `--cov` flag. **Example** ```bash ng_dev_test ``` *** ### `ng_init_resources_server` / `nemo_gym_init_resources_server` Initialize a new resources server with template files and directory structure. **Example** ```bash ng_init_resources_server +entrypoint=resources_servers/my_server ``` *** ## Data Collection Commands for collecting verified rollouts for RL training. ### `ng_collect_rollouts` / `nemo_gym_collect_rollouts` Perform a batch of rollout collection. **Parameters** | Parameter | Type | Description | | ------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `agent_name` | str | The agent to collect rollouts from. | | `input_jsonl_fpath` | str | The input data source to use to collect rollouts, in the form of a file path to a JSONL file. | | `output_jsonl_fpath` | str | The output data JSONL file path. | | `limit` | Optional\[int] | Maximum number of examples to load and take from the input dataset. | | `num_repeats` | Optional\[int] | The number of times to repeat each example to run. Useful if you want to calculate mean\@k, such as mean\@4 or mean\@16. | | `num_repeats_add_seed` | bool | When num\_repeats >1, add a "seed" parameter on the Responses create params. | | `num_samples_in_parallel` | Optional\[int] | Limit the number of concurrent samples running at once. | | `responses_create_params` | Dict | Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to [Generation Parameters](#generation-parameters). | | `resume_from_cache` | bool | Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to [Resume Interrupted Runs](#resume-interrupted-runs). | **Example** ```bash ng_collect_rollouts \ +agent_name=example_single_tool_call_simple_agent \ +input_jsonl_fpath=weather_query.jsonl \ +output_jsonl_fpath=weather_rollouts.jsonl \ +limit=100 \ +num_repeats=4 \ +num_samples_in_parallel=10 ``` #### Generation Parameters Sampling parameters such as `temperature`, `max_output_tokens`, and `top_p` are not standalone CLI flags — they are passed as overrides inside `responses_create_params` using Hydra's nested dot syntax. Overrides are merged into each input row's existing `responses_create_params` with a **shallow** merge (top-level keys only): ```bash ng_collect_rollouts \ +agent_name=example_single_tool_call_simple_agent \ +input_jsonl_fpath=weather_query.jsonl \ +output_jsonl_fpath=weather_rollouts.jsonl \ ++responses_create_params.temperature=1.0 \ ++responses_create_params.top_p=1.0 \ ++responses_create_params.max_output_tokens=4096 ``` The same syntax works for `ng_e2e_collect_rollouts`. Top-level fields such as `temperature` and `max_output_tokens` are straightforward. For nested objects (for example, `++responses_create_params.reasoning.effort=low`), the entire nested dict replaces the row's existing value at that key — other fields under the same nested object are not preserved. #### Resume Interrupted Runs Setting `+resume_from_cache=true` lets you restart the **same command** after a crash or interruption and pick up only the rows that have not finished yet. It works for both `ng_collect_rollouts` and `ng_e2e_collect_rollouts`, across any environment. How it works: * **Materialized inputs.** On the first run, the fully expanded input rows (after `num_repeats`, `limit`, `prompt_config`, and any overrides) are written to a sidecar file next to your output. The path is derived from `output_jsonl_fpath` by appending `_materialized_inputs` to the stem — so `rollouts.jsonl` produces `rollouts_materialized_inputs.jsonl`. * **Incremental output.** Results are flushed to `output_jsonl_fpath` after each completed rollout, so partial output survives a crash. * **Matching.** On resume, completed work is matched by `(task_index, rollout_index)` against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run. * **Fallback.** If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. With the default `resume_from_cache=False`, existing output is cleared before the run. If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the `*_materialized_inputs.jsonl` file (and the output file) to start fresh. ### `ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts` Spin up all necessary servers and perform a batch of rollout collection using each dataset inside the provided configs. **Parameters** | Parameter | Type | Description | | ------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `output_jsonl_fpath` | str | The output data JSONL file path. | | `num_samples_in_parallel` | Optional\[int] | Limit the number of concurrent samples running at once. | | `responses_create_params` | Dict | Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to [Generation Parameters](#generation-parameters). | | `resume_from_cache` | bool | Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to [Resume Interrupted Runs](#resume-interrupted-runs). | **Examples** ```bash ng_e2e_collect_rollouts \ +output_jsonl_fpath=weather_rollouts.jsonl \ +num_samples_in_parallel=10 ``` ```bash config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\ resources_servers/math_with_judge/configs/math_with_judge.yaml" ng_e2e_collect_rollouts \ "+config_paths=[${config_paths}]" \ ++wandb_project= \ ++wandb_name= \ ++wandb_dir= \ ++output_jsonl_fpath=results/test_e2e_rollout_collection/aime24.jsonl \ ++split=validation ``` Example using GPT-OSS 120B remote vLLM endpoint ```bash experiment_name=rollouts/test_001 config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\ resources_servers/math_with_judge/configs/math_with_judge.yaml" ng_e2e_collect_rollouts \ "+config_paths=[${config_paths}]" \ +skip_venv_if_present=true \ +wandb_project=gym-dev \ +wandb_name=$(date +%Y%m%d)/$experiment_name \ ++output_jsonl_fpath=results/$experiment_name.jsonl \ ++overwrite_metrics_conflicts=true \ ++split=validation \ ++policy_model_name=openai/gpt-oss-120b \ ++policy_api_key=dummy_key \ ++policy_base_url=http://0.0.0.0:10240/v1 \ ++responses_create_params.reasoning.effort=low \ ++responses_create_params.temperature=1.0 \ ++responses_create_params.top_p=1.0 &> eval_gptoss120b.log & ``` *** ### `ng_reward_profile` / `nemo_gym_reward_profile` Computes statistics on rewards and task difficulty for rollouts collected with `ng_collect_rollouts` with `num_repeats` >1. This outputs a new "reward profiled" dataset, where each task in the dataset has metrics like the average reward, standard deviation, min/max, and pass rate. This is useful in filtering tasks before training for difficulty, variance, or creating a curriculum. **Parameters** | Parameter | Type | Description | | ---------------------- | ---------------- | ------------------------------------------------------------------------------------------------ | | `input_jsonl_fpath` | str | Path to the original task dataset JSONL file. | | `rollouts_jsonl_fpath` | str | Path to the rollouts file from `ng_collect_rollouts` (must have been run with `num_repeats` >1). | | `output_jsonl_fpath` | str | Output file path for the reward profiled dataset. | | `pass_threshold` | Optional\[float] | Reward threshold for computing pass rate. If not specified, pass rate metrics are not included. | **Output Fields** Each output row contains all original task fields plus: * `avg_reward`: Average reward across all rollouts * `std_reward`: Standard deviation of rewards * `min_reward`: Minimum reward observed * `max_reward`: Maximum reward observed * `total_samples`: Number of rollout samples * `pass_rate`, `pass_rate_total`, `pass_rate_passed`, `pass_threshold`: (Only if `pass_threshold` is specified) **Example** ```bash ng_reward_profile \ +input_jsonl_fpath=tasks.jsonl \ +rollouts_jsonl_fpath=rollouts.jsonl \ +output_jsonl_fpath=profiled_tasks.jsonl \ +pass_threshold=1.0 ``` *** ## Data Management Commands for preparing and viewing training data. ### `ng_prepare_data` / `nemo_gym_prepare_data` Prepare and validate training data, generating metrics and statistics for datasets. **Parameters** | Parameter | Type | Description | | ----------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `output_dirpath` | str | Directory path where processed datasets and metrics will be saved. | | `mode` | Literal\["train\_preparation", "example\_validation"] | Processing mode. Use `train_preparation` to prepare train and validation datasets for training, or `example_validation` to validate example data for PR submission. | | `should_download` | bool | Whether to automatically download missing datasets from remote registries. Default: `False`. | | `overwrite_metrics_conflicts` | bool | Whether or not to overwrite metrics conflicts. Default: `False`. | **Example** ```bash config_paths="resources_servers/example_multi_step/configs/example_multi_step.yaml,\ responses_api_models/openai_model/configs/openai_model.yaml" ng_prepare_data "+config_paths=[${config_paths}]" \ +output_dirpath=data/example_multi_step \ +mode=example_validation ``` *** ### `ng_materialize_prompts` / `nemo_gym_materialize_prompts` Apply a prompt template to raw JSONL data, producing materialized JSONL with populated `responses_create_params.input` for RL training. Each input row must **not** already have a populated `responses_create_params.input`; the command applies the prompt template from `prompt_config` to each row, fills in the input, and preserves the row's other fields. **Parameters** | Parameter | Type | Description | | -------------------- | ---- | -------------------------------------------------------------- | | `input_jsonl_fpath` | str | Raw JSONL data (rows without `responses_create_params.input`). | | `prompt_config` | str | Path to the prompt YAML file to apply. | | `output_jsonl_fpath` | str | Output path for the materialized JSONL with populated prompts. | **Example** ```bash ng_materialize_prompts \ +input_jsonl_fpath=data/my_dataset.jsonl \ +prompt_config=/path/to/my_prompt.yaml \ +output_jsonl_fpath=my_dataset_materialized.jsonl ``` **Which data-preparation command should I use?** * **`ng_materialize_prompts`** — a focused, standalone step that applies a prompt template to raw rows to populate `responses_create_params.input`. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows. * **`ng_prepare_data`** — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission. *** ## Dataset Registry - GitLab Commands for uploading, downloading, and managing datasets in GitLab Model Registry. ### `ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab` Upload a local JSONL dataset artifact to GitLab. **Parameters** | Parameter | Type | Description | | ------------------- | ---- | ----------------------------------------------------------- | | `dataset_name` | str | The dataset name. | | `version` | str | The version of this dataset. Must be in the format `x.x.x`. | | `input_jsonl_fpath` | str | Path to the JSONL file to upload. | **Example** ```bash ng_upload_dataset_to_gitlab \ +dataset_name=example_multi_step \ +version=0.0.1 \ +input_jsonl_fpath=data/train.jsonl ``` *** ### `ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab` Download a JSONL dataset from GitLab Model Registry. **Parameters** | Parameter | Type | Description | | ---------------- | ---- | ----------------------------------------------------------- | | `dataset_name` | str | The dataset name. | | `version` | str | The version of this dataset. Must be in the format `x.x.x`. | | `artifact_fpath` | str | The filepath to the artifact to download. | | `output_fpath` | str | Path where the downloaded dataset will be saved. | **Example** ```bash ng_download_dataset_from_gitlab \ +dataset_name=example_multi_step \ +version=0.0.1 \ +artifact_fpath=train.jsonl \ +output_fpath=data/train.jsonl ``` *** ### `ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab` Delete a dataset from GitLab Model Registry. Prompts for confirmation. **Parameters** | Parameter | Type | Description | | -------------- | ---- | ------------------------------------------ | | `dataset_name` | str | Name of the dataset to delete from GitLab. | **Example** ```bash ng_delete_dataset_from_gitlab +dataset_name=old_dataset ``` *** ## Dataset Registry - HuggingFace Commands for uploading and downloading datasets to/from HuggingFace Hub. ### `ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf` Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload. **Parameters** | Parameter | Type | Description | | ---------------------- | --------------- | ---------------------------------------------------------------------------------------- | | `hf_token` | str | HuggingFace API token for authentication. | | `hf_organization` | str | HuggingFace organization name where the dataset will be uploaded. | | `hf_collection_name` | str | HuggingFace collection name for organizing datasets. | | `hf_collection_slug` | str | Alphanumeric collection slug found at the end of the collection URI. | | `dataset_name` | str | Name of the dataset. Will be combined with domain and resources server name. | | `input_jsonl_fpath` | str | Path to the local JSONL file to upload. | | `resource_config_path` | str | Path to resources server config file. Used to extract domain for naming convention. | | `hf_dataset_prefix` | str | Prefix prepended to dataset name. Default: `NeMo-Gym`. | | `delete_from_gitlab` | Optional\[bool] | Delete the dataset from GitLab after successful upload to HuggingFace. Default: `False`. | **Example** ```bash resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml" ng_upload_dataset_to_hf \ +dataset_name=my_dataset \ +input_jsonl_fpath=data/train.jsonl \ +resource_config_path=${resource_config_path} \ +delete_from_gitlab=true ``` *** ### `ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf` Download a JSONL dataset from HuggingFace Hub to local filesystem. **Parameters** | Parameter | Type | Description | | ---------------- | ---- | ---------------------------------------------------------------- | | `output_fpath` | str | Local file path where the downloaded dataset will be saved. | | `hf_token` | str | HuggingFace API token for authentication. | | `artifact_fpath` | str | Name of the artifact file to download from the repository. | | `repo_id` | str | HuggingFace repository ID in format `organization/dataset-name`. | **Example** ```bash ng_download_dataset_from_hf \ +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \ +artifact_fpath=train.jsonl \ +output_fpath=data/train.jsonl ``` *** ### `ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset` Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload. This command always deletes the dataset from GitLab after uploading to HuggingFace. Use `ng_upload_dataset_to_hf` if you want optional deletion control. **Parameters** Same as `ng_upload_dataset_to_hf` but `delete_from_gitlab` is not available. This command always deletes. **Example** ```bash resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml" ng_gitlab_to_hf_dataset \ +dataset_name=my_dataset \ +input_jsonl_fpath=data/train.jsonl \ +resource_config_path=${resource_config_path} ``` *** ## Configuration & Help Commands for debugging configuration and getting help. ### `ng_dump_config` / `nemo_gym_dump_config` Display the resolved Hydra configuration for debugging purposes. **Example** ```bash ng_dump_config "+config_paths=[,]" ``` *** ### `ng_help` / `nemo_gym_help` Display a list of available NeMo Gym CLI commands. **Example** ```bash ng_help ``` *** ### `ng_version` / `nemo_gym_version` Display NeMo Gym version and system information. **Parameters** | Parameter | Type | Description | | ------------- | ---- | ------------------------------------------------------------------------------------------------- | | `json_format` | bool | Output in JSON format for programmatic use. Default: `False`. Can be specified with `+json=true`. | **Example** ```bash # Display version information ng_version # Output as JSON ng_version +json=true ``` *** ### `ng_pip_list` / `nemo_gym_pip_list` Each server has its own isolated virtual environment. To inspect the packages: **Parameters** | Parameter | Type | Description | | ------------ | -------------- | -------------------------------------------------------------------------------------------- | | `entrypoint` | str | The relative entrypoint path to the server directory | | `format` | Optional\[str] | Output format for pip list. Options: 'columns' (default), 'freeze', 'json'. Default: `None`. | | `outdated` | bool | List outdated packages. Default: `False`. | **Examples** ```bash # List all packages ng_pip_list +entrypoint=resources_servers/example_single_tool_call # Output as JSON ng_pip_list +entrypoint=resources_servers/example_single_tool_call +format=json # Check for outdated packages ng_pip_list +entrypoint=resources_servers/example_single_tool_call +outdated=true ``` *** ### `ng_status` / `nemo_gym_status` View all currently running NeMo Gym servers and their health status. **Example** ```bash ng_status NeMo Gym Server Status: [1] ✓ example_single_tool_call (resources_servers/example_single_tool_call) { 'server_type': 'resources_servers', 'name': 'example_single_tool_call', 'port': 58117, 'pid': 89904, 'uptime_seconds': '0d 0h 0m 41.5s', } [2] ✓ example_single_tool_call_simple_agent (responses_api_agents/simple_agent) { 'server_type': 'responses_api_agents', 'name': 'simple_agent', 'port': 58118, 'pid': 89905, 'uptime_seconds': '0d 0h 0m 41.5s', } [3] ✓ policy_model (responses_api_models/openai_model) { 'server_type': 'responses_api_models', 'name': 'openai_model', 'port': 58119, 'pid': 89907, 'uptime_seconds': '0d 0h 0m 41.5s', } 3 servers found (3 healthy, 0 unhealthy) ``` *** ## Getting Help For detailed help on any command, run it with `+help=true` or `+h=true`: ```bash ng_run +help=true ng_collect_rollouts +h=true ``` This will display all available configuration parameters and their descriptions. *** ## Re-install Gym and dependencies ```bash ng_reinstall ``` This will re-install Gym and its dependencies into the currently activated Python virtual environment.