> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Configuration Errors These errors appear when running `gym env start` or `gym eval run --no-serve`. NeMo Gym validates configuration at startup before launching servers, and checks the model endpoints the config names once its own servers are up. > **Note** > > See the [Configuration reference](/reference/configuration) for complete configuration syntax and options. --- ## Startup Errors Errors that prevent servers from starting. ### Unset Required Config Value(s) ``` nemo_gym.config_types.ConfigMissingValuesError: 1 required config value(s) are unset (still '???') after merging: - policy_api_key Provide each value via a CLI override, in env.yaml, or in a config you pass via config_paths. For example, on the command line: ++policy_api_key= ``` **When**: One or more required variables (usually in `env.yaml`) are left unset (`???`) after all config sources are merged. Every unset value is listed together in this single error. **Fix**: Add the missing value to `env.yaml`: ```yaml policy_api_key: sk-your-api-key ``` Or pass via command line: ```bash gym env start \ --config config.yaml \ --model-api-key sk-your-api-key ``` ### Server Reference Not Found ``` nemo_gym.config_types.ServerRefNotFoundError: In server instance 'my_agent', field 'resources_server' references resources_servers/'typo_weather', which is not defined in the merged config. Did you mean: 'weather'? ``` **When**: A server config references another server that doesn't exist or isn't loaded. The error names the referencing instance/field and then either suggests a similarly named server (`Did you mean: …`) or, when nothing is close, lists the available servers of that type (`Available resources_servers: …`). **Common causes**: * Typo in the server name * Referenced server's config file not included in the `--config` flags * Server defined in a different config file that wasn't loaded **Fix**: 1. Check server name spelling in your config 2. Ensure all required config files are passed with `--config`: ```bash gym env start \ --config model.yaml \ --config resource.yaml \ --config agent.yaml ``` ### Config Path Not Found ``` nemo_gym.config_types.ConfigPathNotFoundError: config_paths entry 'resources_servers/weather/config.yaml' was not found. Looked in: - /path/to/cwd/resources_servers/weather/config.yaml - /path/to/gym/resources_servers/weather/config.yaml Check the path is spelled correctly and is relative to your working directory or the Gym install root. ``` **When**: A `config_paths` entry (in a config file or a `--config` flag) points at a file that doesn't exist under any search root. The error lists every location that was searched. **Fix**: Correct the path. A relative entry is resolved against your working directory and then the Gym install root, so running from the repository root usually fixes it; an absolute path also works. ### Malformed `config_paths` ``` nemo_gym.config_types.MalformedConfigPathsError: 'config_paths' must be a list of paths. Got: 'resources_servers/weather/config.yaml'. Pass each config with --config (it builds the list for you), e.g.: gym env start --config resources_servers//configs/.yaml ``` **When**: `config_paths` is set to a scalar or mapping instead of a list of path strings. **Fix**: Make it a YAML list, or pass each config with `--config`: ```yaml config_paths: - resources_servers/weather/config.yaml ``` ### No Server Instances Configured ``` nemo_gym.config_types.NoServerInstancesError: No server instances are configured, so there is nothing to run. Pass one or more configs, e.g.: gym env start --config resources_servers//configs/.yaml --config responses_api_models//configs/.yaml ``` **When**: The merged config defines no server instances — usually a wrong/empty `--config` or a `config_paths` list that resolved to nothing. **Fix**: Pass the configs that define your servers, or check the `config_paths` entries actually load. > **Tip** > > Catch all of the above before launch with [`gym env validate`](/reference/cli-commands#gym-env-validate) — it > merges and validates the same config `gym env start` uses, then exits with a clean message (no traceback) > and a non-zero status on failure, so it fits in CI pre-flight checks. --- ## Validation Errors Errors where config structure is correct but values are invalid. ### Almost-Servers Detected ``` Configuration Warnings: Almost-Servers Detected Almost-Server Detected: 'example_simple_agent' This server configuration failed validation: - ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', 'long_context', 'safety', 'games', 'translation', 'e2e', 'rlhf' or 'other' ``` **When**: Config has the right structure (server type, entrypoint) but contains invalid field values. **Common causes**: * Invalid `domain` value for resources servers * Invalid `license` value in dataset configs * Missing required fields for the server type **Fix**: Check the validation error path (for example, `resources_servers -> example_server -> domain`) and update the field with a valid value. Refer to [configuration-reference](/reference/configuration) for valid field values. #### Bypass Strict Validation To continue with invalid configs (invalid servers will be skipped): ```yaml # In env.yaml error_on_almost_servers: false ``` ```bash # Or via command line gym env start \ --config config.yaml \ +error_on_almost_servers=false ``` > **Warning** > > Bypassing validation means invalid servers won't start. Use this only for debugging, not production. ### Agent Without an Environment Server ``` DEPRECATED: agents without an environment server: `my_simple_agent`. Rollout collection reaches an agent only through an environment server. This run generated a legacy_agent environment server for each agent above. Each one relays /run to its agent unchanged, so results are not affected. A future release will reject these configs with AgentWithoutEnvironmentServerError. ``` **When**: An agent instance has no environment server naming it in `agent_server`. Rollout collection reaches agents only through an environment server. Gym therefore generates a `legacy_agent` relay in front of the agent and logs this warning. Your config keeps running for now, but a future release will make this an error, so migrate it. **Fix**: Declare the environment servers in your config. From a NeMo Gym checkout, run the migration script on your config files or directories: ```bash # Report what would change python scripts/add_legacy_agent_environment_servers.py path/to/configs/ --check # Add the blocks python scripts/add_legacy_agent_environment_servers.py path/to/configs/ ``` The script appends a block to each config that needs one and leaves the existing content unchanged, comments included. It skips agents that an environment server already names. It also skips agents that only override fields of an agent defined elsewhere. Run without paths, it updates this repository's configs. It exits with 1 under `--check` when something needs adding, and with 2 when a file cannot be read. You can also add the block by hand. The warning prints the exact block for each agent; add it at the same level as the agents: ```yaml my_environment_server: environment_servers: legacy_agent: entrypoint: app.py agent_server: type: responses_api_agents name: my_simple_agent ``` For a top-level `_inherit_from` rename with one destination that still defines exactly one agent, Gym automatically points environment server references at the new name. The server keeps its name, so `environment_server_routes` still resolve. The migration script leaves these renames alone. If several destinations inherit the same source and remain agents, and an environment server still references the old name, Gym reports `AmbiguousAgentRenameError` with the destinations and the field to update. Give each new agent its own environment server by inheriting the original server and naming that agent, for example: ```yaml variant_a_environment_server: _inherit_from: my_environment_server environment_servers: legacy_agent: agent_server: name: variant_a ``` Apply this pattern to each variant so every agent has a declared environment server, including when `error_on_agent_without_environment_server` is enabled. Alternatively, point the existing server's field at one agent and ensure the other agents also have environment servers. Do not add a second server in front of an agent, because that makes agent-routed rows ambiguous. #### Require Declared Environment Servers To make a missing environment server an error instead (`AgentWithoutEnvironmentServerError`): ```yaml # In env.yaml error_on_agent_without_environment_server: true ``` ```bash # Or via command line gym env start \ --config config.yaml \ +error_on_agent_without_environment_server=true ``` --- ## Connection Errors Errors where the config is valid but an endpoint it names cannot be reached. ### Gym Server Startup Timeout Gym stops waiting when its managed servers have not become ready within `server_spinup_timeout_seconds` (default 600). The error names the server instances still waiting. This bound also catches a multi-worker Uvicorn manager that remains alive while repeatedly replacing workers that fail during application construction. The budget starts when Gym begins polling its servers for readiness, which happens after it has launched them and its head server is answering. Head-server startup does not count against it. The budget covers whatever each server still has to do before it opens its port. That includes installing the server's dependencies and, for a locally launched model such as `local_vllm_model`, downloading and loading the model weights: that server waits for its inference engine before the Gym port opens. `model_endpoint_readiness_timeout_seconds` is a separate, later check of the model endpoints named in the config, so raising it does not help at this stage. Raise the value for servers with legitimately long initialization with `++server_spinup_timeout_seconds=`, or set it to `0` to preserve an unbounded wait while debugging. The value must be a number of seconds; anything else is reported as a configuration error before any server starts. A deterministic startup exception should be fixed rather than hidden behind a larger timeout. ### Model Endpoint Unreachable ``` 2 model endpoint(s) never answered within 600s: - http://localhost:8000/v1 (from `openai_base_url`) - http://judge-host:9000/v1 (from `judge_base_url`) Nothing accepted a connection there. The server was never started, is still starting up, or the URL in your config does not match where it is listening. ``` **When**: `gym env start` finished starting Gym's own servers, but a model endpoint named in the config is not answering. Gym's model server is a proxy that answers as soon as it starts, whether or not an inference server is behind it, so this check is separate from the server readiness check above it. **Fix**: Check the config key named beside each URL, and confirm something answers there. The check probes `/models` under a base URL ending in `/v1`, and the URL itself otherwise, so mirror whichever applies: ```bash curl -i https://example.com/v1/models curl -i https://example.com ``` Any response, including `401` or `404`, means something is listening. Then correct that config key, or start the server it names. The [Quickstart](/get-started/quickstart) uses a hosted endpoint (`https://api.openai.com/v1`); a `localhost` URL means you serve the model yourself (see [Configure Model](/model-server)). > **Note** > > The check waits up to `model_endpoint_readiness_timeout_seconds` (default 600) so an inference server still loading weights is not treated as a failure. Raise it to wait longer, or set it to `0` to skip the check entirely. > > An endpoint whose hostname does not resolve is reported once and not waited on, since waiting cannot create a DNS record. This is what happens to a config that leaves a judge or user model at its placeholder default: you get one line about it rather than a stalled startup. > > Any HTTP answer counts as listening, including 401 and 404. A completed TLS handshake counts too, even against a certificate this machine does not trust, because it proves something is there.