Configuration Errors

These errors appear when running gym env start or gym eval run --no-serve. NeMo Gym validates configuration at startup before launching servers, and checks the model endpoints the config names once its own servers are up.

Startup Errors

Errors that prevent servers from starting.

Unset Required Config Value(s)

nemo_gym.config.config_types.ConfigMissingValuesError: 1 required config value(s) are unset (still '???') after merging:
  - policy_api_key
Provide each value via a CLI override, in env.yaml, or in a config you pass via config_paths.
For example, on the command line:
  ++policy_api_key=<value>

When: One or more required variables (usually in env.yaml) are left unset (???) after all config sources are merged. Every unset value is listed together in this single error.

Fix: Add the missing value to env.yaml:

policy_api_key: sk-your-api-key

Or pass via command line:

gym env start \
    --config config.yaml \
    --model-api-key sk-your-api-key

Server Reference Not Found

nemo_gym.config.config_types.ServerRefNotFoundError: In server instance 'my_agent', field 'resources_server' references resources_servers/'typo_weather', which is not defined in the merged config.
Did you mean: 'weather'?

When: A server config references another server that doesn’t exist or isn’t loaded. The error names the referencing instance/field and then either suggests a similarly named server (Did you mean: …) or, when nothing is close, lists the available servers of that type (Available resources_servers: …).

Common causes:

• Typo in the server name
• Referenced server’s config file not included in the --config flags
• Server defined in a different config file that wasn’t loaded

Fix:

1. Check server name spelling in your config
2. Ensure all required config files are passed with --config:

gym env start \
    --config model.yaml \
    --config resource.yaml \
    --config agent.yaml

Config Path Not Found

nemo_gym.config.config_types.ConfigPathNotFoundError: config_paths entry 'resources_servers/weather/config.yaml' was not found. Looked in:
  - /path/to/cwd/resources_servers/weather/config.yaml
  - /path/to/gym/resources_servers/weather/config.yaml
Check the path is spelled correctly and is relative to your working directory or the Gym install root.

When: A config_paths entry (in a config file or a --config flag) points at a file that doesn’t exist under any search root. The error lists every location that was searched.

Fix: Correct the path. A relative entry is resolved against your working directory and then the Gym install root, so running from the repository root usually fixes it; an absolute path also works.

Malformed config_paths

nemo_gym.config.config_types.MalformedConfigPathsError: 'config_paths' must be a list of paths. Got: 'resources_servers/weather/config.yaml'.
Pass each config with --config (it builds the list for you), e.g.:
  gym env start --config resources_servers/<env>/configs/<env>.yaml

When: config_paths is set to a scalar or mapping instead of a list of path strings.

Fix: Make it a YAML list, or pass each config with --config:

config_paths:
  - resources_servers/weather/config.yaml

Malformed head_server Values

nemo_gym.config.config_types.HeadServerConfigMalformedError: 'head_server' is invalid: head_server.port (Input should be a valid integer, unable to parse string as an integer).
`head_server.host` must be a hostname or IP; `head_server.port` and, if set, `head_server.num_workers` must be integers. Set them in a config or as overrides, e.g.:
  ++head_server.host=127.0.0.1 ++head_server.port=11000

When: A head_server field has the wrong type — typically a ++head_server.port= override that is not an integer, or an explicit host: null — or the whole block was given as something other than a mapping (++head_server=foo, ++head_server=false, '++head_server=[]'), which reports 'head_server' must be a mapping ... instead. Either key may be omitted (Gym fills the default), as may the whole block, but a value that is present must be well-typed.

Fix: Give host a hostname or IP and port an integer, in the config or on the command line:

head_server:
  host: 127.0.0.1
  port: 11000

No Server Instances Configured

nemo_gym.config.config_types.NoServerInstancesError: No server instances are configured, so there is nothing to run. Pass one or more configs, e.g.:
  gym env start --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml

When: The merged config defines no server instances — usually a wrong/empty --config or a config_paths list that resolved to nothing.

Fix: Pass the configs that define your servers, or check the config_paths entries actually load.

Validation Errors

Errors where config structure is correct but values are invalid.

Almost-Servers Detected

Configuration Warnings: Almost-Servers Detected
  Almost-Server Detected: 'example_simple_agent'
  This server configuration failed validation:
- ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: 
  Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', 
  'long_context', 'safety', 'games', 'translation', 'e2e', 'rlhf' or 'other'

When: Config has the right structure (server type, entrypoint) but contains invalid field values.

Common causes:

• Invalid domain value for resources servers
• Invalid license value in dataset configs
• Missing required fields for the server type

Fix: Check the validation error path (for example, resources_servers -> example_server -> domain) and update the field with a valid value. Refer to Configuration for valid field values.

Bypass Strict Validation

To continue with invalid configs (invalid servers will be skipped):

# In env.yaml
error_on_almost_servers: false

# Or via command line
gym env start \
    --config config.yaml \
    +error_on_almost_servers=false

Agent Without an Environment Server

DEPRECATED: agents without an environment server: `my_simple_agent`.
Rollout collection reaches an agent only through an environment server.
This run generated a legacy_agent environment server for each agent above.
Each one relays /run to its agent unchanged, so results are not affected.
A future release will reject these configs with AgentWithoutEnvironmentServerError.

When: An agent instance has no environment server naming it in agent_server. Rollout collection reaches agents only through an environment server. Gym therefore generates a legacy_agent relay in front of the agent and logs this warning. Your config keeps running for now, but a future release will make this an error, so migrate it.

Fix: Declare the environment servers in your config. From a NeMo Gym checkout, run the migration script on your config files or directories:

# Report what would change
python scripts/add_legacy_agent_environment_servers.py path/to/configs/ --check
# Add the blocks
python scripts/add_legacy_agent_environment_servers.py path/to/configs/

The script appends a block to each config that needs one and leaves the existing content unchanged, comments included. It skips agents that an environment server already names. It also skips agents that only override fields of an agent defined elsewhere. Run without paths, it updates this repository’s configs. It exits with 1 under --check when something needs adding, and with 2 when a file cannot be read.

You can also add the block by hand. The warning prints the exact block for each agent; add it at the same level as the agents:

my_environment_server:
  environment_servers:
    legacy_agent:
      entrypoint: app.py
      agent_server:
        type: responses_api_agents
        name: my_simple_agent

For a top-level _inherit_from rename with one destination that still defines exactly one agent, Gym automatically points environment server references at the new name. The server keeps its name, so environment_server_routes still resolve. The migration script leaves these renames alone. If several destinations inherit the same source and remain agents, and an environment server still references the old name, Gym reports AmbiguousAgentRenameError with the destinations and the field to update. Give each new agent its own environment server by inheriting the original server and naming that agent, for example:

variant_a_environment_server:
  _inherit_from: my_environment_server
  environment_servers:
    legacy_agent:
      agent_server:
        name: variant_a

Apply this pattern to each variant so every agent has a declared environment server, including when error_on_agent_without_environment_server is enabled. Alternatively, point the existing server’s field at one agent and ensure the other agents also have environment servers. Do not add a second server in front of an agent, because that makes agent-routed rows ambiguous.

Move a simple_agent Off the Relay

A legacy_agent relay sends the whole episode to the agent’s /run as one request, so a crash repeats the whole episode. single_agent_turn_legacy runs the same episode in steps (seed, agent turn, agent close, verify), so a crash repeats only the current step, and simple_agent resumes inside its turn. To replace the relays in front of simple_agent in your configs:

# Report what would switch and what stays on the relay
python scripts/add_legacy_agent_environment_servers.py --switch-simple-agent path/to/configs/ --check
# Switch them
python scripts/add_legacy_agent_environment_servers.py --switch-simple-agent path/to/configs/

Pass the configs that declare the relays yours inherit as well, such as the Resources Server configs a benchmark includes; run without paths to switch this repository’s configs.

Each relay is replaced by a single_agent_turn_legacy server of the same name. The migration copies the agent’s Resources Server reference into the Environment Server’s resources_server field and grants the agent direct HTTP access to its tools, as the relay’s /run used them. Inherited Environment Servers also receive their own explicit resources binding.

The Environment Server owns this binding at runtime. Both single_agent_turn and single_agent_turn_legacy require resources_server; leaving it unset or setting it to null no longer reads the agent’s configuration. Configs migrated by an earlier version of the script must add the reference to their Environment Server, including child configs that change the resources binding.

Configuration resolution follows a top-level _inherit_from rename when the original Resources Server is removed and exactly one resources instance replaces it. For example, the shared math Environment Server follows math_with_judge to indic_math_500_math_with_judge_resources_server in indic/math_500. If several resources instances inherit the same source, a reference to the removed source is ambiguous and resolution reports the field to update.

A _copy keeps the original resources instance, so its references stay unchanged. Set the child Environment Server’s resources_server explicitly when it should use the copy, as indic/ifeval does. Likewise, changing an agent’s resources setting does not change the Environment Server’s binding. The migration script refuses a switch when an included config changes that pairing without a matching Environment Server override.

A relay stays, and is listed with its reason, when:

• the agent binds no Resources Server, or declares a required MCP tool access or another direct HTTP tool access, which simple_agent sessions reject;
• the agent’s Resources Server has a seed_session that accepts only the legacy body rather than the ResourcesSeedSessionRequest an Environment Server sends, has request models that read the row’s _ng_* rollout coordinates, which an Environment Server does not forward, or has no source under resources_servers/ to check;
• the relay sets keys the switched server would not keep, such as port;
• another config merges it with a legacy_agent server of the same name, such as a benchmark variant that points it at another harness;
• the run cannot load one of its included configs.

A relay that inherits another relay keeps its _inherit_from, and the two must switch in the same run; the script exits with 2 and names the relays when only one would switch. It also exits with 2 when another config merges an inheriting relay with a legacy_agent server while the relay it inherits switches. That config should keep the server’s name, so rows keep their routing, and replace its definition, as benchmarks/hle/opencode_legacy_agent.yaml does: _delete_key: _inherit_from detaches it from the switched relay, _delete_key: single_agent_turn_legacy under its environment_servers removes the migrated type, and a complete legacy_agent server fronts its harness. Set the inherited relay to null when nothing else in the run uses it. The script writes nothing when it exits with 2. Compare the reward profile of a switched config with the relay’s before relying on it.

Require Declared Environment Servers

To make a missing environment server an error instead (AgentWithoutEnvironmentServerError):

# In env.yaml
error_on_agent_without_environment_server: true

# Or via command line
gym env start \
    --config config.yaml \
    +error_on_agent_without_environment_server=true

Environment Server Config Does Not Match Its Server

Error: Environment Server config is invalid, so `gym env start` would fail when the server launches.
'my_environment_server' (environment_servers/single_agent_turn_legacy/app.py) does not match SingleAgentTurnEnvironmentServerConfig:
  my_environment_server.environment_servers.single_agent_turn_legacy.cleanup_timeout_seconds: Field required
  my_environment_server.environment_servers.single_agent_turn_legacy.resources_tool_transport: Extra inputs are not permitted

When: An environment server block names its server type and entrypoint, but its fields don’t match the config class of the server that entrypoint runs. gym env validate reports this before anything starts. gym env start reports it only when that server launches, after the other servers are up. gym env resolve still prints the config and reports the same fields as a warning.

Common causes:

• Switching a block to another server type (for example, legacy_agent to single_agent_turn_legacy) without adding the fields the new type requires
• A misspelled field name

Fix: Set or correct each field at the path listed. The server’s configs/ directory under environment_servers/ shows the fields it accepts.

If the server’s entrypoint can’t be loaded in the environment gym runs in, for example because it needs a package installed only in the server’s own virtual environment, gym env validate warns that it could not check that server and leaves the check to its launch.

Connection Errors

Errors where the config is valid but an endpoint it names cannot be reached.

Head Server Unreachable

Error: Could not connect to the head server at http://127.0.0.1:11000. Is the head server running?
Start it with: `gym env start`. If it is already running on a different host or port, pass
`++head_server.host=<host>` / `++head_server.port=<port>` so this command can find it.

When: gym eval run --no-serve collects against servers that are already running, and found nothing listening at the head server address in its config. Either no gym env start is running, or it was started with a different head server address than this command is using. That address is head_server.host / head_server.port (default 127.0.0.1:11000). When head_server.host is not set, an explicit default_host takes precedence. If neither host setting is supplied, use_absolute_ip selects the machine’s resolved IP address when true, or 127.0.0.1 when false. A gym env start run with different host settings can therefore listen on a different address than the collecting command expects.

Fix: Check which head server address gym env start used (from its config and overrides), then run gym env status with matching overrides before concluding that nothing is running. gym env status only queries the head server address in its own config, so servers running on another host or port also show No NeMo Gym servers found running.

gym env status "++head_server.port=<port>"

If status lists your servers, pass the same overrides to the collecting command:

gym eval run --no-serve "++head_server.port=<port>" ...

If status finds no servers at the matching address either, start them with gym env start (in another terminal or the background) and retry.

Gym Server Startup Timeout

Gym stops waiting when its managed servers have not become ready within server_spinup_timeout_seconds (default 600). The error names the server instances still waiting. This bound also catches a multi-worker Uvicorn manager that remains alive while repeatedly replacing workers that fail during application construction.

The budget starts when Gym begins polling its servers for readiness, which happens after it has launched them and its head server is answering. Head-server startup does not count against it. The budget covers whatever each server still has to do before it opens its port. That includes installing the server’s dependencies and, for a locally launched model such as local_vllm_model, downloading and loading the model weights: that server waits for its inference engine before the Gym port opens. model_endpoint_readiness_timeout_seconds is a separate, later check of the model endpoints named in the config, so raising it does not help at this stage.

Raise the value for servers with legitimately long initialization with ++server_spinup_timeout_seconds=<seconds>, or set it to 0 to preserve an unbounded wait while debugging. The value must be a number of seconds; anything else is reported as a configuration error before any server starts. A deterministic startup exception should be fixed rather than hidden behind a larger timeout.

Server Shutdown

Managed servers run in separate process sessions. On shutdown, Gym sends each server group one SIGINT and waits within one shared server_shutdown_timeout_seconds grace period (default 1). Set it to finite nonnegative seconds; numeric strings are accepted, and 0 requests immediate escalation. Invalid values fail before any server starts. After the grace expires, Gym sends SIGKILL to the remaining groups and uses one additional bounded reap interval. Descendants remain part of the owned group even if the launching shell exits first. The HTTP sidecar stays alive until the servers stop, with its own shutdown grace; telemetry and head-server teardown also fall outside the managed-server grace.

Agent servers use their lifespan shutdown to close remaining sessions concurrently. One session_close_timeout_seconds budget bounds the whole drain, including incomplete setup and cleanup already in progress. This field still bounds individual close phases during normal operation. Its base default is 60 seconds; adapters may keep their own defaults, including 30 for Hermes and Stirrup. A timeout cancels pending cleanup and reports unconfirmed session IDs. Unconfirmed cleanup retains its handle and leaves no saved reply; process exit does not establish that a remote sandbox stopped, so its owner and provider TTL remain responsible after forced termination.

Allow the managed-server grace to exceed the longest agent drain plus Uvicorn request cancellation and process overhead. The default one-second grace does not guarantee a sixty-second agent drain. For example, if request cancellation and shutdown overhead fit within ten seconds:

server_shutdown_timeout_seconds: 70
my_agent:
  responses_api_agents:
    codex_agent:
      session_close_timeout_seconds: 60

An external supervisor’s shutdown deadline must exceed these budgets too, or it can kill Gym before graceful cleanup completes.

Model Endpoint Unreachable

2 model endpoint(s) never answered within 600s:
  - http://localhost:8000/v1 (from `openai_base_url`)
  - http://judge-host:9000/v1 (from `judge_base_url`)
Nothing accepted a connection there, or only a proxy or load balancer answered with 502, 503 or 504.
The server was never started, is still starting up, or the URL in your config does not match where
it is listening.

When: gym env start finished starting Gym’s own servers, but a model endpoint named in the config is not answering. Gym’s model server is a proxy that answers as soon as it starts, whether or not an inference server is behind it, so this check is separate from the server readiness check above it.

Fix: Check the config key named beside each URL, and confirm something answers there. The check probes /models under a base URL ending in /v1, and the URL itself otherwise, and connects directly rather than through a proxy, so mirror whichever applies:

curl --noproxy '*' -i https://example.com/v1/models
curl --noproxy '*' -i https://example.com

Any response other than 502, 503 or 504, including 401 or 404, means something is listening. A 502, 503 or 504 usually comes from a proxy or load balancer whose backend is still starting. Then correct that config key, or start the server it names. The Quickstart uses a hosted endpoint (https://api.openai.com/v1); a localhost URL means you serve the model yourself (see Configure Model).