TutorialsTraining TutorialsNeMo RL

Gym Configuration

View as Markdown

Before running GRPO training, you need to configure how NeMo RL connects to NeMo Gym. The training config file contains Gym-specific parameters that control data loading, environment interaction, and validation.

Goal: Understand the Gym configuration parameters for RL training.

Time: ~10 minutes (read)

In this section, you will learn:

  1. How to configure data paths for training and validation
  2. How to enable and configure NeMo Gym in NeMo RL
← Previous: About Workplace Assistant

Prerequisites

Configuration File Location

The full training configuration file lives in the NeMo RL repository, not in NeMo Gym:

examples/nemo_gym/grpo_workplace_assistant_nemotron_nano_v2_9b.yaml

Paths in that file are relative to the NeMo RL repo root. After Setup, NeMo Gym is checked out inside that repo at 3rdparty/Gym-workspace/Gym/, which is why the data paths below point there.

Gym Configuration Sections

There are two Gym-specific sections in the NeMo RL training config: data and env.

Data Section

1data:
2  train:
3    data_path: 3rdparty/Gym-workspace/Gym/data/workplace_assistant/train.jsonl
4  validation:
5    data_path: 3rdparty/Gym-workspace/Gym/data/workplace_assistant/validation.jsonl
6  default:
7    dataset_name: NemoGymDataset
8    env_name: "nemo_gym"
9    processor: "nemo_gym_data_processor"
ParameterDescription
train.data_pathPath to training dataset (created later in Setup)
validation.data_pathPath to validation dataset
default.dataset_nameMust be NemoGymDataset so NeMo RL reads Gym-format JSONL
default.env_nameMust be nemo_gym so batches are routed to the Gym environment
default.processorMust be nemo_gym_data_processor

Environment Section

1env:
2  should_use_nemo_gym: true
3  should_log_nemo_gym_responses: true
4  should_mask_flagged_samples: true
5  nemo_gym:  # This is passed into NeMo Gym as the initial_global_config_dict
6    port_range_low: 5000
7    port_range_high: 5999
8    is_trajectory_collection: false
9    config_paths:
10    - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml
11    - resources_servers/workplace_assistant/configs/workplace_assistant.yaml
ParameterDescription
should_use_nemo_gymSet to true to enable Gym
should_log_nemo_gym_responsestrue skips writing the large per-step train_data_step*.jsonl files; prefer true if disk is tight
should_mask_flagged_samplesfalse ignores environment mask_sample flags so the loss trains on every sample
nemo_gymEverything under this key is passed to Gym as its global config
nemo_gym.port_range_low / port_range_highPort range for Gym HTTP servers, kept clear of NeMo RL (3000-4999) and vLLM (7000-8999)
nemo_gym.is_trajectory_collectionSet to true to collect trajectories without training
nemo_gym.config_pathsGym config files, relative to the Gym checkout: vLLM model config and Workplace Assistant agent/resources config

The vllm_model_for_training.yaml config is required for NeMo RL training integration.

Agent behavior such as max_steps is not set in the NeMo RL config. It comes from the Gym-side config listed in config_paths — for this tutorial, max_steps: 6 under workplace_assistant_simple_agent.responses_api_agents.simple_agent in resources_servers/workplace_assistant/configs/workplace_assistant.yaml.

To override a Gym config value from NeMo RL, nest it under env.nemo_gym using the same key path as the Gym config. The upstream example does this for the policy model:

1env:
2  nemo_gym:
3    policy_model:
4      responses_api_models:
5        vllm_model:
6          uses_reasoning_parser: false
7          extra_body:
8            chat_template_kwargs:
9              enable_thinking: false

Next Steps

With the Gym configuration understood, learn about the GRPO training parameters:

Continue to NeMo RL Configuration →