> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Configuration Complete syntax and field specifications for NeMo Gym configuration files. ## File Locations | File | Location | Version Control | | -------------- | ----------------------------------------------- | --------------------------- | | Server configs | `//configs/*.yaml` | ✅ Committed | | env.yaml | Repository root (`./env.yaml`) | ❌ Gitignored (user creates) | ### Artifact Roots (`results_dir`, `cache_dir`) Two **opt-in** top-level config keys that servers can consult for state that outlives a request: | Key | Default | Read today by | | ------------- | ----------------------- | -------------------------------------------------------------------- | | `results_dir` | `/results` | the `swe_agents` server (per-run results) and the W\&B run directory | | `cache_dir` | `/cache` | the `swe_agents` server (setup trees) and the `uv_cache_dir` default | These are opt-in settings, not a global redirect: components that don't consult them (profiling output, rollout capture, logs, Hugging Face caches, other agents) keep their own locations, and the module-level `RESULTS_DIR`/`CACHE_DIR` constants remain unchanged defaults for such consumers. New artifact producers are encouraged to read these keys. Both are independently overridable (`results_dir=/shared/results cache_dir=/shared/cache`), and relative values are normalized to absolute paths at parse time so all server processes agree. `uv_venv_dir` keeps its own key and default. On multi-node deployments, the `swe_agents` server embeds these paths in commands executed from **other nodes**, so both roots must resolve to the same content at the same path on every node that runs rollout workers: use a shared filesystem, or pre-stage identical trees at the same path on each node (for example, baked into the container image). A cache that exists only on the head node breaks remote rollouts. Servers that pre-staged setup trees next to the package (the layout before `cache_dir` existed, for example baked container images) keep using those trees **only while `cache_dir` is left at its default** — explicitly configuring `cache_dir` opts out, and removing the pre-staged trees migrates a default-config deployment. Contents accumulate across runs; both directories are safe to delete when no run is active. ### Server Virtual Environments `gym env start` sets up one uv virtual environment per server and starts the server with that environment's own interpreter (`/bin/python`), then logs which venv each server runs from. A venv that was copied or moved after it was built still runs its own interpreter, but its `bin/activate` still puts the original location first on `PATH`, so commands a server runs by name (such as `python` or `uv`) are looked up there first. Rebuild a moved venv, or create it with `uv venv --relocatable`, if its servers start such subprocesses. | Key | Default | Description | | ------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `uv_lock_timeout_seconds` | `UV_LOCK_TIMEOUT` if already exported, else `1800` | Seconds a server install waits for uv's cache lock. Servers start together and share one `uv_cache_dir`, so the rest wait while one of them does a cold install; uv's own default of 300 can be too short for that. Exported as `UV_LOCK_TIMEOUT` to the Gym process and to every server, test, and venv-setup subprocess it launches. Set it to `null` to leave `UV_LOCK_TIMEOUT` as inherited. | --- ## Path Resolution and External Roots Gym resolves every relative path — `config_paths`, `env.yaml`, prompt configs, dataset files, the `--` selectors (`--benchmark`, `--environment`, `--model-type`, `--resources-server`), and server directories (used by `gym env test`) — against an ordered list of **roots**, returning the first one where the path exists: 1. **Extra roots** from `NEMO_GYM_EXTRA_ROOTS` (or `--search-dir`), in the order listed. 2. **The current working directory** — your project. 3. **The Gym install root**, where the built-in components live (in both editable and wheel installs). Earlier roots win, so a component you provide shadows a same-named built-in. Absolute paths are used unchanged. ### External roots (`NEMO_GYM_EXTRA_ROOTS`) Point Gym at one or more extra roots so your own benchmarks, environments, resources servers, agents, models, configs, prompts, and data resolve by name — without forking Gym or copying files into the install tree. Each root uses the same layout as the Gym repo: ``` / ├── benchmarks// ├── environments// ├── resources_servers// ├── responses_api_agents// └── responses_api_models// ``` Set it as an `os.pathsep`-separated list (`:` on Linux/macOS): ```bash export NEMO_GYM_EXTRA_ROOTS=~/my-plugins:~/team-benchmarks gym list benchmarks # your benchmarks appear alongside the built-ins gym eval run --benchmark my_bench --model-type vllm_model ``` The variable is inherited by the servers Gym spawns, so plugin components resolve inside them too. ### `--search-dir` `--search-dir DIR` (repeatable) is the per-invocation equivalent: Gym sets `NEMO_GYM_EXTRA_ROOTS` to its value for the duration of that command, then restores it. Use it for one-off runs instead of exporting the variable. ```bash gym list benchmarks --search-dir ~/my-plugins gym eval run --benchmark my_bench --model-type vllm_model --search-dir ~/my-plugins ``` > **Tip** > > Prefer `NEMO_GYM_EXTRA_ROOTS` when the same plugin roots apply to every command in a shell session; reach for `--search-dir` for a single invocation. If both are set, `--search-dir` takes precedence for that command. --- ## Server Configuration All servers share this structure: ```yaml server_id: # Your unique name for this server server_type: # responses_api_models | resources_servers | responses_api_agents implementation: # Directory name inside the server type directory entrypoint: app.py # Python file to run # ... additional fields vary by server type ``` ### HTTP Connection Pool Gym uses one shared aiohttp connection pool per process. Each top-level limit is a budget for one server process group (or the rollout CLI), divided across that server's FastAPI workers. It is not a deployment-wide budget: every agent, model, and resources server gets its own pool. ```yaml global_aiohttp_connector_limit: 102400 global_aiohttp_connector_limit_per_host: 1024 # Optional sizing checks. Set these when the expected outbound concurrency is known. global_aiohttp_intended_concurrency: 4096 global_aiohttp_intended_concurrency_per_host: 1024 ``` At startup, Gym logs the configured and effective per-worker limits once per server. An explicit connector limit of `0` keeps aiohttp's existing meaning of unlimited. Gym rejects a positive limit that becomes zero after worker division, because that would accidentally turn a bounded pool into an unlimited one. The optional intended-concurrency values also apply separately to each server or CLI pool. When intended demand is larger than effective capacity or can exhaust the file-descriptor budget, Gym logs an actionable warning. The connector limit and file-descriptor soft limit remain in the startup information line, but the connector ceiling alone does not trigger a file-descriptor warning because it is not an allocation. The total aiohttp limit is a scheduling limit, not a strict open-socket cap across multiple hosts. Idle-connection reuse and aiohttp's waiter wake-up order can exceed the total limit for as long as requests to more than one host are in flight. Pool waits are also not first come, first served: a new request that can reuse an idle pooled connection skips requests that are already waiting, and aiohttp wakes waiting hosts in random order, so a few requests can wait much longer than the backend latency. When sizing pools, keep file-descriptor, ephemeral-port, and downstream connection budgets in mind. Pool metrics require an active metrics exporter before the shared client is constructed. Gym server processes and the orchestrator used by `gym eval run` initialize telemetry automatically. When Gym telemetry is enabled and another library, such as NeMo RL, already set up nemo-lens in the same process, Gym reuses its providers, and pool metrics follow that library's metrics setting. Standalone `gym eval run --no-serve` and library calls to `setup_server_client()` do not initialize telemetry; callers must initialize it explicitly before creating the client. Initializing telemetry later does not replace an existing connector. Missing metrics from these standalone processes do not mean that no requests queued. When this instrumentation is installed, Gym records one histogram sample per queued aiohttp connection acquisition. This metric is independent of the `http_client` tracing span group: | Metric or attribute | Meaning | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gym.http.connection_pool.queue_duration_ms` | Histogram of wait time for queued connects; immediate connects do not record a sample | | `gym.http.connection_pool.connect_total` | Observable counter of all connection attempts (`connect()` calls), including attempts that later fail or are abandoned; use it as the denominator for the queued fraction | | `nemo.gym.http.connection_pool.queue_constraint` | Binding limit: `total`, `per_host`, or `unknown` | | `nemo.gym.http.connection_pool.queue_outcome` | Queue outcome: `ok`, or `abandoned` when the wait ends before a slot is available, including timeout and cancellation | | `nemo.gym.http.destination.server.name` | Configured destination server name for `ServerClient` calls, `remote_agent_service` for the remote agent's external service, or `external` for unlabeled direct calls | The destination attribute identifies the configured or fixed request destination, including across redirects. The resource attribute `nemo.gym.server.name` identifies the Gym process recording the metric; they are not the same. When metrics export is active at client construction, Gym uses `QueueTimedTCPConnector` to count connection attempts and measure queue waits. Otherwise, Gym uses aiohttp's `TCPConnector` directly without collecting these metrics. ### Reverse Proxy Headers Gym disables Uvicorn proxy-header processing by default. Direct Gym traffic does not pass through a reverse proxy, so the network peer address and request scheme are authoritative without interpreting `X-Forwarded-For` or `X-Forwarded-Proto`. Deployments behind a reverse proxy must opt in with top-level configuration and list the proxy addresses or CIDRs that are allowed to supply forwarded headers: ```yaml uvicorn_proxy_headers: true uvicorn_forwarded_allow_ips: - 10.0.0.5 - 10.20.0.0/24 ``` Both settings apply to the Gym head server and the Agent, Model, and Resources servers. Enabling proxy headers without a non-empty allowlist fails at startup. Wildcards and all-address CIDRs such as `0.0.0.0/0` and `::/0` are rejected. > **Warning** > > Only list reverse proxies you control. Trusting a loopback address such as `127.0.0.1` trusts every local process, not > only a proxy running on that address. Deployments that previously relied on Uvicorn's implicit loopback trust must set > these options explicitly. ### Model Server Fields ```yaml policy_model: # Server ID (use "policy_model" — agent configs expect this name) responses_api_models: # Server type (must be "responses_api_models" for model servers) openai_model: # Implementation (use "openai_model", "vllm_model", or "azure_openai_model") entrypoint: app.py # Python file to run openai_base_url: ${policy_base_url} # API endpoint URL openai_api_key: ${policy_api_key} # Authentication key openai_model: ${policy_model_name} # Model identifier ``` > **Tip** > > Keep the server ID as `policy_model` — agent configs reference this name by default. The `${policy_base_url}`, `${policy_api_key}`, and `${policy_model_name}` placeholders should be defined in `env.yaml` at the repository root, allowing you to change model settings in one place. ### Resources Server Fields ```yaml my_resource: # Server ID (your choice — agents reference this name) resources_servers: # Server type (must be "resources_servers" for resources servers) example_single_tool_call: # Implementation (must match a directory in resources_servers/) entrypoint: app.py # Python file to run domain: agent # Server category (see values below) verified: false # Passed reward profiling and training checks (default: false) description: "Short description" # Server description value: "What this improves" # Training value provided allowed_agents: [my_agent] # Optional: agents that score this task correctly ``` **`allowed_agents`** pins the agent harnesses that verify this server's tasks correctly. Omit it (the default) when any harness will do. When it is set, [`--agent-type`](/reference/cli-commands#swapping-the-agent) refuses to substitute an agent that is not listed. Declare it on every config that writes this server's block — server configs are self-contained, so a config written from scratch does not inherit it. **Domain values:** `math`, `coding`, `agent`, `knowledge`, `instruction_following`, `long_context`, `safety`, `games`, `translation`, `e2e`, `rlhf`, `other` (see `Domain`) ### Agent Server Fields Agent servers must include both a `resources_server` and `model_server` block to specify which servers to use. ```yaml my_agent: # Server ID (your choice — used in API requests) responses_api_agents: # Server type (must be "responses_api_agents" for agent servers) simple_agent: # Implementation (must match a directory in responses_api_agents/) entrypoint: app.py # Python file to run resources_server: # Specifies which resources server to use type: resources_servers # Always "resources_servers" name: my_resource # Server ID of the resources server model_server: # Specifies which model server to use type: responses_api_models # Always "responses_api_models" name: policy_model # Server ID of the model server datasets: # Optional: define for training workflows - name: train # Dataset identifier type: train # example | train | validation jsonl_fpath: path/to/data.jsonl # Path to data file license: Apache 2.0 # Required for train/validation ``` #### Dataset Configuration Define datasets associated with agent servers for training and evaluation. ```yaml datasets: - name: my_dataset type: train jsonl_fpath: path/to/data.jsonl license: Apache 2.0 num_repeats: 1 ``` | Field | Required | Description | | ------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Dataset identifier | | `type` | Yes | `example`, `train`, or `validation` | | `jsonl_fpath` | Yes | Path to data file | | `license` | For train/validation | License identifier (see values below) | | `source` | No | Where to fetch the data from when it's missing locally. A `source:` block with `type: gitlab` (`dataset_name`, `version`, `artifact_fpath`) or `type: huggingface` (`repo_id`, optional `artifact_fpath`). Replaces the deprecated `gitlab_identifier:` / `huggingface_identifier:` blocks. | | `num_repeats` | No | Repeat dataset n times (default: `1`) | **Dataset types:** * `example` — For testing and development * `train` — Training data (requires `license`) * `validation` — Evaluation data (requires `license`) **License values:** `Apache 2.0`, `MIT`, `Creative Commons Attribution 4.0 International`, `Creative Commons Attribution-ShareAlike 4.0 International`, `CC BY-SA 4.0`, `CC BY-NC 3.0`, `TBD` (see `license`) --- ## Local Configuration (env.yaml) Store secrets and local settings at the repository root. This file is gitignored. ```yaml # Policy model (required for most setups) # Reference these variables in server configs using `${variable_name}` syntax (e.g., `${policy_base_url}`) policy_base_url: https://api.openai.com/v1 policy_api_key: sk-your-api-key policy_model_name: gpt-4o-2024-11-20 # Optional: store config paths for reuse my_config_paths: - responses_api_models/openai_model/configs/openai_model.yaml - resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml # Optional: multi-node setups use_absolute_ip: true # Bind servers to the host's IP instead of 127.0.0.1 (default: false) # Optional: validation behavior error_on_almost_servers: true # Exit on invalid configs (default: true) # Optional: experiment tracking (see Experiment Tracking below) wandb_project: gym-dev wandb_name: my-run wandb_api_key: your-wandb-key ``` ### Multi-Node Configuration **`use_absolute_ip`** — Controls the default host servers bind to. * **Default:** `false` — servers use `127.0.0.1` (localhost). * **When to use:** Set to `true` for multi-node setups (e.g. multi-node Ray clusters) where servers must communicate across machines. * **Effect:** Resolves and uses the host's IP address (`gethostbyname(gethostname())`) instead of localhost. --- ## Experiment Tracking Gym exports the resolved config, aggregate metrics, and rollouts to any tracking backend you configure. Weights & Biases and MLflow are both supported, and both can be on at once. A backend is used only when every key it needs is set; otherwise it is silently skipped. ### Weights & Biases | Key | Required | Description | | --------------- | -------- | ----------------------- | | `wandb_project` | yes | W\&B project to log to. | | `wandb_name` | yes | Run name. | | `wandb_api_key` | yes | W\&B API key. | ### MLflow | Key | Required | Description | | ------------------------ | -------- | --------------------------------------------------- | | `mlflow_tracking_uri` | yes | Tracking server URI. | | `mlflow_experiment_name` | yes | Experiment to log to. Created if it does not exist. | | `mlflow_run_name` | yes | Run name. | | `mlflow_tracking_token` | no | Bearer token. Omit for unauthenticated servers. | ```bash gym eval run \ --benchmark gpqa \ --model-type openai_model \ +mlflow_tracking_uri=https://mlflow.example.com/ \ +mlflow_experiment_name=my-experiment \ +mlflow_run_name=gpqa ``` > **Note** > > `mlflow_tracking_uri` and `mlflow_tracking_token` are shared with the GitLab model registry used by > `gym dataset upload|download`. Setting only those two does not enable the exporter — it also needs > an experiment and run name. ### Rollout upload Rollouts are uploaded to every configured backend by default. Turn this off when they are large: ```bash gym eval run ... +upload_rollouts=false ``` Metrics and config are still exported. The old name `upload_rollouts_to_wandb` is deprecated and will be removed in a future release. > **Note** > > Exporting is best-effort. Errors that happen during export (e.g., unavailable server) are reported as warnings and the run continues. --- ## Command Line Usage To run servers, use `gym env start`. NeMo Gym uses [Hydra](https://hydra.cc/) for configuration management. ### Loading Configs ```bash # Load one or more config files gym env start \ --config config1.yaml \ --config config2.yaml # Use paths stored in env.yaml gym env start "+config_paths=${my_config_paths}" ``` ### Overriding Values ```bash # Override nested values (use dot notation after server ID) gym env start \ --config config.yaml \ +my_server.resources_servers.my_impl.domain=coding # Override policy model gym env start --config config.yaml \ --model gpt-4o-mini # Disable strict validation gym env start \ --config config.yaml \ +error_on_almost_servers=false ``` --- ## Troubleshooting > **Note** > > [Configuration](/troubleshooting/configuration) for common configuration errors and solutions.