Configuration

Complete syntax and field specifications for NeMo Gym configuration files.

File Locations

Artifact Roots (results_dir, cache_dir)

Two opt-in top-level config keys that servers can consult for state that outlives a request:

These are opt-in settings, not a global redirect: components that don’t consult them (profiling output, rollout capture, logs, Hugging Face caches, other agents) keep their own locations, and the module-level RESULTS_DIR/CACHE_DIR constants remain unchanged defaults for such consumers. New artifact producers are encouraged to read these keys.

Both are independently overridable (results_dir=/shared/results cache_dir=/shared/cache), and relative values are normalized to absolute paths at parse time so all server processes agree. uv_venv_dir keeps its own key and default.

On multi-node deployments, the swe_agents server embeds these paths in commands executed from other nodes, so both roots must resolve to the same content at the same path on every node that runs rollout workers: use a shared filesystem, or pre-stage identical trees at the same path on each node (for example, baked into the container image). A cache that exists only on the head node breaks remote rollouts.

Servers that pre-staged setup trees next to the package (the layout before cache_dir existed, for example baked container images) keep using those trees only while cache_dir is left at its default — explicitly configuring cache_dir opts out, and removing the pre-staged trees migrates a default-config deployment. Contents accumulate across runs; both directories are safe to delete when no run is active.

Server Virtual Environments

gym env start sets up one uv virtual environment per server and starts the server with that environment’s own interpreter (<venv>/bin/python), then logs which venv each server runs from. A venv that was copied or moved after it was built still runs its own interpreter, but its bin/activate still puts the original location first on PATH, so commands a server runs by name (such as python or uv) are looked up there first. Rebuild a moved venv, or create it with uv venv --relocatable, if its servers start such subprocesses.

Per-server runtime files

A server directory can carry two standard uv files:

• .python-version selects the Python for that server’s venv. If the request admits the Python that runs gym (for example >=3.13.14,<3.14), the venv uses exactly that version, because Ray refuses to connect processes on different patch releases. A request that excludes it (for example 3.12) is passed to uv unchanged; a server built that way cannot join the parent’s Ray cluster.
• uv.toml is passed to uv venv and uv pip install with --config-file. Its resolver settings (for example [pip] torch-backend = "cpu") apply to that venv only, and an enclosing uv workspace contributes none of its own.

Every server venv also installs the packages the parent pins, such as its openai version (unless allow_openai_version_skew is set) and any telemetry or plugin packages, as ordinary requirements. A server that pins a conflicting version therefore fails setup rather than running with a mismatched client, unless the server’s own overrides.txt replaces that package on purpose. The parent’s Ray version is applied as both a constraint and an override: a server gets Ray only if its dependencies need it, and then at exactly the parent’s version, even when a dependency pins another one.

Restricting server locations (NEMO_GYM_ALLOWED_COMPONENT_ROOTS)

Set NEMO_GYM_ALLOWED_COMPONENT_ROOTS to one or more directories, separated by :, to make Gym refuse to start any server whose directory is outside them. A scheduler that stages a pinned Gym checkout can set it so that a server resolved from another copy, such as one baked into the container image, fails at startup instead of running stale code. When the variable is unset, servers start from wherever they resolve.

Path Resolution and External Roots

Gym resolves every relative path — config_paths, env.yaml, prompt configs, dataset files, the --<component> selectors (--benchmark, --environment, --model-type, --resources-server), and server directories (used by gym env test) — against an ordered list of roots, returning the first one where the path exists:

1. Extra roots from NEMO_GYM_EXTRA_ROOTS (or --search-dir), in the order listed.
2. The current working directory — your project.
3. The Gym install root, where the built-in components live (in both editable and wheel installs).

Earlier roots win, so a component you provide shadows a same-named built-in. Absolute paths are used unchanged.

In non-editable installs (including wheels), Python keeps its existing import-path order, so a cwd copy of a built-in can override its files while the installed module still wins. To override both files and modules, explicitly add your project with --search-dir . (for example, gym eval prepare --benchmark aime24 --search-dir .) or NEMO_GYM_EXTRA_ROOTS.

External roots (NEMO_GYM_EXTRA_ROOTS)

Point Gym at one or more extra roots so your own benchmarks, environments, resources servers, agents, models, configs, prompts, and data resolve by name — without forking Gym or copying files into the install tree. Each root uses the same layout as the Gym repo:

<root>/
├── benchmarks/<name>/
├── environments/<name>/
├── resources_servers/<name>/
├── responses_api_agents/<name>/
└── responses_api_models/<name>/

Set it as an os.pathsep-separated list (: on Linux/macOS):

export NEMO_GYM_EXTRA_ROOTS=~/my-plugins:~/team-benchmarks
gym list benchmarks                 # your benchmarks appear alongside the built-ins
gym eval run --benchmark my_bench --model-type vllm_model

The variable is inherited by the servers Gym spawns, so plugin components resolve inside them too.

--search-dir

--search-dir DIR (repeatable) is the per-invocation equivalent: Gym sets NEMO_GYM_EXTRA_ROOTS to its value for the duration of that command, then restores it. Use it for one-off runs instead of exporting the variable.

gym list benchmarks --search-dir ~/my-plugins
gym eval run --benchmark my_bench --model-type vllm_model --search-dir ~/my-plugins

Server Configuration

All servers share this structure:

server_id:                    # Your unique name for this server
  server_type:                # responses_api_models | resources_servers | responses_api_agents
    implementation:           # Directory name inside the server type directory
      entrypoint: app.py      # Python file to run
      # ... additional fields vary by server type

HTTP Connection Pool

Gym uses one shared aiohttp connection pool per process. Each top-level limit is a budget for one server process group (or the rollout CLI), divided across that server’s FastAPI workers. It is not a deployment-wide budget: every agent, model, and resources server gets its own pool.

global_aiohttp_connector_limit: 102400
global_aiohttp_connector_limit_per_host: 1024
# Optional sizing checks. Set these when the expected outbound concurrency is known.
global_aiohttp_intended_concurrency: 4096
global_aiohttp_intended_concurrency_per_host: 1024

At startup, Gym logs the configured and effective per-worker limits once per server. An explicit connector limit of 0 keeps aiohttp’s existing meaning of unlimited. Gym rejects a positive limit that becomes zero after worker division, because that would accidentally turn a bounded pool into an unlimited one. The optional intended-concurrency values also apply separately to each server or CLI pool. When intended demand is larger than effective capacity or can exhaust the file-descriptor budget, Gym logs an actionable warning. The connector limit and file-descriptor soft limit remain in the startup information line, but the connector ceiling alone does not trigger a file-descriptor warning because it is not an allocation.

The total aiohttp limit is a scheduling limit, not a strict open-socket cap across multiple hosts. Idle-connection reuse and aiohttp’s waiter wake-up order can exceed the total limit for as long as requests to more than one host are in flight. Pool waits are also not first come, first served: a new request that can reuse an idle pooled connection skips requests that are already waiting, and aiohttp wakes waiting hosts in random order, so a few requests can wait much longer than the backend latency. When sizing pools, keep file-descriptor, ephemeral-port, and downstream connection budgets in mind.

Pool metrics require an active metrics exporter before the shared client is constructed. Gym server processes and the supervisor used by gym eval run initialize telemetry automatically. When Gym telemetry is enabled and another library, such as NeMo RL, already set up nemo-lens in the same process, Gym reuses its providers, and pool metrics follow that library’s metrics setting. Standalone gym eval run --no-serve and library calls to setup_server_client() do not initialize telemetry; callers must initialize it explicitly before creating the client. Initializing telemetry later does not replace an existing connector. Missing metrics from these standalone processes do not mean that no requests queued.

When this instrumentation is installed, Gym records one histogram sample per queued aiohttp connection acquisition. This metric is independent of the http_client tracing span group:

The destination attribute identifies the configured or fixed request destination, including across redirects. The resource attribute nemo.gym.server.name identifies the Gym process recording the metric; they are not the same.

When metrics export is active at client construction, Gym uses QueueTimedTCPConnector to count connection attempts and measure queue waits. Otherwise, Gym uses aiohttp’s TCPConnector directly without collecting these metrics.

HTTP Timeouts and Retries Between Servers

Two settings bound how long one call from a Gym server to another waits on an unreachable peer.

global_aiohttp_sock_connect_timeout_seconds: 60
global_aiohttp_internal_retry_budget_seconds: 600

global_aiohttp_sock_connect_timeout_seconds (default 60) limits one TCP handshake. It does not count the wait for a free connection-pool slot, so a request queued behind a busy pool is not cut short. Total and read time stay unlimited unless the caller passes its own timeout, because a model generation or a /run episode can run for hours. A handshake that times out counts as a connection that was never made: the request cannot have been delivered, so it is retried like a refused connection, within the caller’s limits and even where a dropped connection is not resent.

global_aiohttp_internal_retry_budget_seconds (default 600) is the retry budget of a call between Gym servers whose caller sets none of _max_num_tries, _max_connection_retries or a connection retry window. Such a call retries quietly through a peer that is restarting, and past the budget it raises the last error. Two clocks make up the budget:

• Connection errors. The window opens at the first connection error and closes when the server answers, so it bounds an unbroken run of failures. After the first connection error, the window also bounds each attempt’s wait for a free pool slot, not only the handshake.
• Other errors, such as a timeout or a DNS failure. The cap is the time since the call started.

A call that sets its own limit uses that limit and not the budget. Set the budget to 0 to retry without a time limit, as before. Model connection retries are separate: connection_retry_timeout_s on a model client stays unset by default.

Listen Backlog

uvicorn_backlog sets how many new connections the kernel queues for each Gym server before the server accepts them (for each worker, on a server that uses uvicorn_reuse_port). It applies to the head server, every agent, model, environment, and resources server, and the private sockets that the workers of a multi-worker server use to forward requests to each other. The default is 16384:

uvicorn_backlog: 16384

When the queue is full, the kernel drops new connection attempts, and each client waits on TCP retransmission backoff (1, 3, 7, 15, and 31 seconds) before its request reaches the server. Size the backlog for the largest burst of new connections a server receives, which can approach the callers’ global_aiohttp_connector_limit_per_host.

The kernel caps the backlog at net.core.somaxconn, so raise that setting on hosts with a lower value.

Reverse Proxy Headers

Gym disables Uvicorn proxy-header processing by default. Direct Gym traffic does not pass through a reverse proxy, so the network peer address and request scheme are authoritative without interpreting X-Forwarded-For or X-Forwarded-Proto.

Deployments behind a reverse proxy must opt in with top-level configuration and list the proxy addresses or CIDRs that are allowed to supply forwarded headers:

uvicorn_proxy_headers: true
uvicorn_forwarded_allow_ips:
  - 10.0.0.5
  - 10.20.0.0/24

Both settings apply to the Gym head server and the Agent, Model, and Resources servers. Enabling proxy headers without a non-empty allowlist fails at startup. Wildcards and all-address CIDRs such as 0.0.0.0/0 and ::/0 are rejected.

Per-Worker Listening Sockets

A server with num_workers above 1 runs that many Uvicorn worker processes. By default they share one listening socket, and whichever worker wakes first accepts the next connection. Under load this spreads connections unevenly, and because clients keep connections alive, a few workers can end up serving most of the traffic while the rest idle; on a busy model server the overloaded workers then delay every call they hold.

On Linux, top-level uvicorn_reuse_port gives every worker its own listening socket on the server’s port (SO_REUSEPORT), and the kernel spreads new connections evenly across the workers:

uvicorn_reuse_port: true

The setting applies to every multi-worker server and has no effect on single-worker servers. A multi-worker server fails at startup on platforms other than Linux. To turn it on for one server only, or to exempt one server, set uvicorn_reuse_port on that server; an explicit true or false there wins over the top-level key:

policy_model:
  responses_api_models:
    vllm_model:
      num_workers: 32
      uvicorn_reuse_port: true

The kernel balances connections, not requests, so a few long-lived keep-alive connections can still load one worker more than the others. Each worker queues up to uvicorn_backlog connections on its own socket. When a worker exits, the connections waiting in its queue are reset rather than handed to another worker, unless net.ipv4.tcp_migrate_req=1 is set in the server’s network namespace (Linux 5.14 or later; it covers TCP over IPv6 too). In a container with its own network namespace, such as a Kubernetes pod, the host’s value does not apply. A worker that stops responding keeps receiving its share of new connections until Uvicorn’s health check replaces it after uvicorn_timeout_worker_healthcheck seconds, and a worker whose event loop is blocked while its process still answers the health check is not replaced. While every worker is down, new connections are refused instead of queued. The server holds its address from start-up, so within about a minute of a server that ran on the same port without uvicorn_reuse_port, while that server’s closed connections are in TIME_WAIT, the start fails with “Address already in use”.

With checkpointing on, /run is never resent, so a reset connection in a dead worker’s queue fails that episode attempt. Prefer enabling the setting only on the servers that need it, such as the model server.

Model Server Fields

policy_model:                                 # Server ID (use "policy_model" — agent configs expect this name)
  responses_api_models:                       # Server type (must be "responses_api_models" for model servers)
    openai_model:                             # Implementation (use "openai_model", "vllm_model", or "azure_openai_model")
      entrypoint: app.py                      # Python file to run
      openai_base_url: ${policy_base_url}     # API endpoint URL
      openai_api_key: ${policy_api_key}       # Authentication key
      openai_model: ${policy_model_name}      # Model identifier

Resources Server Fields

my_resource:                                  # Server ID (your choice — agents reference this name)
  resources_servers:                          # Server type (must be "resources_servers" for resources servers)
    example_single_tool_call:                   # Implementation (must match a directory in resources_servers/)
      entrypoint: app.py                      # Python file to run
      domain: agent                           # Server category (see values below)
      verified: false                         # Passed reward profiling and training checks (default: false)
      description: "Short description"        # Server description
      value: "What this improves"             # Training value provided
      allowed_agents: [my_agent]              # Optional: agents that score this task correctly

allowed_agents pins the agent harnesses that verify this server’s tasks correctly. Omit it (the default) when any harness will do. When it is set, --agent-type refuses to substitute an agent that is not listed. Declare it on every config that writes this server’s block — server configs are self-contained, so a config written from scratch does not inherit it.

Domain values: math, coding, agent, knowledge, instruction_following, long_context, safety, games, translation, e2e, rlhf, other (see Domain)

Agent Server Fields

Agent servers must include both a resources_server and model_server block to specify which servers to use.

my_agent:                                     # Server ID (your choice — used in API requests)
  responses_api_agents:                       # Server type (must be "responses_api_agents" for agent servers)
    simple_agent:                             # Implementation (must match a directory in responses_api_agents/)
      entrypoint: app.py                      # Python file to run
      resources_server:                       # Specifies which resources server to use
        type: resources_servers               # Always "resources_servers"
        name: my_resource                     # Server ID of the resources server
      model_server:                           # Specifies which model server to use
        type: responses_api_models            # Always "responses_api_models"
        name: policy_model                    # Server ID of the model server
      datasets:                               # Optional: define for training workflows
        - name: train                         # Dataset identifier
          type: train                         # example | train | validation
          jsonl_fpath: path/to/data.jsonl     # Path to data file
          license: Apache 2.0                 # Required for train/validation

Dataset Configuration

Define datasets associated with agent servers for training and evaluation.

datasets:
  - name: my_dataset
    type: train
    jsonl_fpath: path/to/data.jsonl
    license: Apache 2.0
    num_repeats: 1

Dataset types:

• example — For testing and development
• train — Training data (requires license)
• validation — Evaluation data (requires license)

License values: Apache 2.0, MIT, Creative Commons Attribution 4.0 International, Creative Commons Attribution-ShareAlike 4.0 International, CC BY-SA 4.0, CC BY-NC 3.0, TBD (see license)

Local Configuration (env.yaml)

Store secrets and local settings at the repository root. This file is gitignored.

# Policy model (required for most setups)
# Reference these variables in server configs using `${variable_name}` syntax (e.g., `${policy_base_url}`)
policy_base_url: https://api.openai.com/v1
policy_api_key: sk-your-api-key
policy_model_name: gpt-4o-2024-11-20
# Optional: store config paths for reuse
my_config_paths:
  - responses_api_models/openai_model/configs/openai_model.yaml
  - resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml
# Optional: multi-node setups
use_absolute_ip: true           # Bind servers to the host's IP instead of 127.0.0.1 (default: false)
# Optional: validation behavior
error_on_almost_servers: true   # Exit on invalid configs (default: true)
# Optional: experiment tracking (see Experiment Tracking below)
wandb_project: gym-dev
wandb_name: my-run
wandb_api_key: your-wandb-key

Multi-Node Configuration

use_absolute_ip — Controls the default host servers bind to.

• Default: false — servers use 127.0.0.1 (localhost).
• When to use: Set to true for multi-node setups (e.g. multi-node Ray clusters) where servers must communicate across machines.
• Effect: When default_host is unset, resolves and uses the host’s IP address (gethostbyname(gethostname())) instead of localhost. If lookup fails or returns a loopback or unspecified address, configuration fails with guidance to set a reachable address.
• Explicit address: default_host takes precedence over automatic lookup, whether use_absolute_ip is true or false. For example, +default_host=192.0.2.20 selects that address without consulting hostname lookup. A server’s own host setting still takes precedence over this shared default.
• Scoping: absolute_ip_server_types (a list of server types: responses_api_models, responses_api_agents, resources_servers, environment_servers) limits the resolved address to those types. Every other server, and the head server whose /global_config_dict_yaml endpoint returns the merged configuration, keeps 127.0.0.1; a server’s own host still wins. Unset or null (the default) moves every server, the head included, as before. The list only scopes the resolved address: an explicit default_host applies to every server regardless.
• Harbor runs: gym eval run <target> --agent <agent> sets use_absolute_ip: true and absolute_ip_server_types: [responses_api_models] by default when the harness declares that it calls the model from inside the sandbox (calls_model_from_sandbox = True on its agent class, read from the agent’s entrypoint source) or its configuration sets sandbox_model_base_url: the model server advertises the node’s address, the head server, the agent server and the environment server stay on loopback. An explicit config file or command-line setting can override either default (++absolute_ip_server_types=null exposes every server). Oracle validation and host-side harnesses do not enable it.

sidecar — Optional block that starts a local HTTP/2 PING proxy so requests to NVCF endpoints survive the 340 second idle limit.

• Default: absent or enabled: false — nothing is started and no URL changes.
• instances: required when enabled. Each entry (name, upstream, listen) gets its own proxy, started on the node that starts the Gym servers.
• Effect: model-server *base_url values whose host matches an instance’s upstream are rewritten to http://127.0.0.1:<port>; top-level keys such as policy_base_url are left alone.
• See: NVCF Long Requests for every key.

Experiment Tracking

Gym exports the resolved config, aggregate metrics, and rollouts to any tracking backend you configure. Weights & Biases and MLflow are both supported, and both can be on at once. A backend is used only when every key it needs is set; otherwise it is silently skipped.

Both SDKs are optional extras, not part of the core install.

uv sync --extra wandb   # Weights & Biases
uv sync --extra mlflow  # MLflow

If a backend is configured but its extra isn’t installed, Gym logs an error naming the missing extra and continues the run without that backend.

Each backend writes its run id to <output stem>_<backend>_run_id next to output_jsonl_fpath, for example rollouts_mlflow_run_id or rollouts_wandb_run_id. A resumed or resubmitted run with the same output logs to that run. Delete the file to start a new one.

A finished run can also be sent afterwards with gym eval export mlflow or gym eval export wandb, which log the run’s saved config, aggregate metrics and rollouts through the same backend. They keep their own <output stem>_<backend>_export_done marker, so exporting never changes which run a resumed run logs to. A run that was logged live is exported again as a separate run, because the run id file shows only that the live run opened, not that every upload landed. See gym eval export.

Weights & Biases

MLflow

gym eval run \
    --benchmark gpqa \
    --model-type openai_model \
    +mlflow_tracking_uri=https://mlflow.example.com/ \
    +mlflow_experiment_name=my-experiment \
    +mlflow_run_name=gpqa

Rollout upload

Rollouts are uploaded to every configured backend by default. Turn this off when they are large:

gym eval run ... +upload_rollouts=false

Metrics and config are still exported. The old name upload_rollouts_to_wandb is deprecated and will be removed in a future release.

Harbor image builds

A task folder whose Dockerfile needs a build (see Dockerfile modes) is built once at prepare, inside gym eval run and gym dataset validate. The top-level builds: block configures those builds. Set it in env.yaml, a --config file or with ++builds.<key>=<value>:

builds:
  tool: buildah
  repository: registry.example.com:5005/my-team/harbor-builds
  platform: linux/amd64
  enabled: true
  force: false

The hash is Harbor’s hash of the task’s environment/ folder (of tests/ for a separate verifier), so identical folders share one image. The tool pulls base images and pushes built ones with its own login (buildah login, podman login or docker login).

• buildah runs as the user who runs Gym, with chroot isolation and the vfs storage driver, under /raid/scratch/<user>/nemo-gym-buildah when /raid/scratch exists, else in the system temp folder. It reads each build context (environment/, or a Compose service’s build.context) in place inside the dataset folder. Its heredoc support fails on Lustre, so keep a dataset whose Dockerfiles use heredocs on local disk while it builds.
• buildah and podman run with Gym’s own containers.conf, layered over the host’s, and registries.conf, which replaces the host’s main one, so registry mirrors set there no longer apply unless you export your own CONTAINERS_REGISTRIES_CONF. They build Docker-format images. Dockerfile heredocs need a tool built with openshift/imagebuilder 1.2.6 or later (upstream buildah 1.33.0 or later); Ubuntu 24.04’s packaged buildah 1.33.7 and podman 4.9.3 do not qualify. Prepare checks the tool before the first heredoc build. If the check fails, tasks whose Dockerfiles use heredocs are left out, with the remedy as their reason in skipped.json, and gym eval run stops unless you pass --skip-unloadable.
• docker builds through docker compose build, so it needs the Compose v2 plugin (docker-compose-plugin).
• Proxies. A proxy the build tool inherits can answer 403 to the registry, which fails the base-image pull or the push. Add the registry host to no_proxy and NO_PROXY, including in the tool’s own proxy configuration.

Harbor environment variables

Command Line Usage

To run servers, use gym env start. NeMo Gym uses Hydra for configuration management.

Loading Configs

# Load one or more config files
gym env start \
    --config config1.yaml \
    --config config2.yaml
# Use paths stored in env.yaml
gym env start "+config_paths=${my_config_paths}"

Overriding Values

# Override nested values (use dot notation after server ID)
gym env start \
    --config config.yaml \
    +my_server.resources_servers.my_impl.domain=coding
# Override policy model
gym env start --config config.yaml \
    --model gpt-4o-mini
# Disable strict validation
gym env start \
    --config config.yaml \
    +error_on_almost_servers=false

Troubleshooting