Run a Custom NeMo Gym Environment

A custom Gym environment runs from a NeMo Helix FileSet rather than from the components bundled in the Gym task image. Evaluator stages the FileSet onto job storage, mounts it read-only in an OpenSandbox Gym host, and returns the rollouts through the normal agent-evaluation result bundle.

Prerequisites

• A running NeMo Helix with the Evals plugin.
• An OpenSandbox-capable Kubernetes deployment configured for sandboxed Gym. See Configure Sandboxed Gym.
• A taskset built from Gym rows as described in Evaluate a NeMo Gym Environment.
• A model endpoint reachable from the sandbox.

1. Package the environment

The FileSet must contain nemo-environment.yaml at its root. Evaluator accepts two formats, which differ only in where a server’s dependencies come from:

• native-v1: Gym installs each server’s requirements.txt from a package index at job start.
• wheels-v1: dependencies come from a flat, non-empty wheels/ directory in the package. No package index is contacted.

The runtime image already provides NeMo Gym and everything Gym itself needs. A wheels-v1 wheelhouse carries only what your servers add on top of that: the packages your requirements.txt files name, and their dependencies, as wheels built for the image’s Python and platform. Do not vendor nemo-gym, ray, or the rest of Gym’s own dependencies.

Every agent and resources-server directory that ships code needs exactly one of requirements.txt or pyproject.toml, in both formats. Gym recognises a server directory by that file; without it the directory is invisible to Gym, and with both Gym refuses to start it. A directory holding only configs/ is different: it references the image’s server of that name, such as responses_api_agents/simple_agent/configs/simple_agent.yaml to run the built-in simple_agent with your settings, and needs no marker. Evaluator checks these rules at submit time.

A minimal native-v1 resource-server package looks like this:

my-gym-environment/
├── nemo-environment.yaml
└── resources_servers/
    └── custom_greeting/
        ├── app.py
        ├── requirements.txt
        └── configs/
            └── custom_greeting.yaml

format: native-v1
config_paths:
  - resources_servers/custom_greeting/configs/custom_greeting.yaml
metadata:
  name: custom-greeting
  description: Return and score a requested greeting.

For wheels-v1, use the same manifest shape with format: wheels-v1, keep the server’s requirements.txt, and add the wheelhouse that satisfies it:

my-gym-environment/
├── nemo-environment.yaml
├── wheels/
│   ├── emoji-2.14.1-py3-none-any.whl
│   └── some_dependency-1.0.0-py3-none-any.whl
└── resources_servers/
    └── custom_greeting/
        ├── app.py
        ├── requirements.txt
        └── configs/
            └── custom_greeting.yaml

With requirements.txt listing emoji==2.14.1, this is enough:

pip download emoji==2.14.1 --only-binary :all: --python-version 3.13 --platform manylinux2014_x86_64 -d my-gym-environment/wheels

A wheels-v1 package may also carry no servers at all, only wheels. Use this when a built-in environment needs an extra package the image does not ship. Leave config_paths empty; the job runs the agent and resources server it selects, and the wheels are importable by both.

mcqa-extras/
├── nemo-environment.yaml
└── wheels/
    └── some_dependency-1.0.0-py3-none-any.whl

format: wheels-v1
config_paths: []
metadata:
  name: mcqa-extras

The package must follow these rules:

• Every config_paths entry is relative to the FileSet root and names an uploaded file.
• native-v1 config paths stay under resources_servers/ or responses_api_agents/.
• Every resources_servers/<name>/ and responses_api_agents/<name>/ directory that ships code has exactly one of requirements.txt or pyproject.toml, wherever its config file lives. A directory with only configs/ references the image’s server of that name.
• wheels/ is flat and contains only .whl files.
• Do not include responses_api_models/; model configuration is operator-owned.
• Do not include dataset .jsonl files. Store evaluation rows in a separate taskset.

2. Upload the environment

Create an environment FileSet and upload the directory contents:

nemo files filesets create my-gym-environment \
  --workspace default \
  --purpose environment \
  --description "Custom Gym environment"
nemo files upload ./my-gym-environment/ my-gym-environment \
  --workspace default
nemo files list my-gym-environment --workspace default

Keep the trailing slash on ./my-gym-environment/. It places nemo-environment.yaml at the FileSet root instead of creating an extra directory level.

3. Create the job specification

Create gym-custom-environment.yaml:

tasks: default/my-gym-taskset
target:
  kind: gym
  environment: default/my-gym-environment
  agent: simple_agent
  agent_config: responses_api_agents/simple_agent/configs/simple_agent.yaml
  resources_server: custom_greeting
  num_repeats: 1
  concurrency: 1
  hydra_params:
    policy_base_url: http://nemo-helix-api.default.svc.cluster.local:8080/apis/inference-gateway/v2/workspaces/default/model/my-model/-/v1
    policy_api_key: not-used
    policy_model_name: my-model
max_concurrent_tasks: 1
fail_fast: true
labels:
  benchmark: custom-greeting

The Evals plugin skill includes the same shape as a validated JSON starting point at skills/nemo-evals-plugin/assets/specs/gym_agent_eval.json.

Replace the service URL, workspace, model, taskset, and component names for your deployment. The model URL must be present in the deployment’s sandbox egress configuration.

If the package declares the selected agent instance, you can omit agent_config. Set agent_ref_name when the instance name registered by the package differs from agent.

For an environment variable containing a credential, use a platform secret reference:

target:
  kind: gym
  # ...
  env_secrets:
    EXTERNAL_MODEL_API_KEY: default/my-model-api-key

Do not place credentials in env_vars. Sandboxed submissions reject credential-shaped plaintext environment variables.

4. Submit the job

nemo evals agent-evaluate \
  --spec-file gym-custom-environment.yaml \
  --workspace default \
  --profile default

Evaluator compiles two ordered steps:

1. stage-environment downloads the FileSet onto job-scoped persistent storage.
2. agent-evaluate provisions nhx-gym-host, collects rollouts, scores them, and destroys the host.

Both platform steps use nhx-tasks; Gym and Ray run in the separate sandbox host.

5. Read the results

Wait for the job to finish, then download the result bundle:

nemo jobs results download agent-eval-results \
  --job <job-name> \
  --output-file agent-eval-results.tar.gz

The archive contains trials.jsonl, scores.jsonl, tasks.jsonl, and summary.json, plus Gym’s output under gym_run/: rollouts.jsonl, the materialized gym_input.jsonl, and the host’s model-call captures in gym_run/model_calls/. Evidence paths in trials.jsonl are relative to the bundle root, the directory that holds trials.jsonl. See Reading Results for the queryable result record and SDK retrieval path.

Troubleshooting

• FileSet purpose or manifest error: Confirm the FileSet uses purpose=environment and contains a valid root nemo-environment.yaml.
• Sandbox unavailable: Confirm the deployment enabled sandboxed Gym and configured OpenSandbox, the runtime image, a shared PVC, and model egress.
• PVC mismatch: The Jobs execution profile and sandbox_job_storage_pvc_claim must name the same claim.
• Agent returns no rollout: Set agent_ref_name to the agent instance registered by the package.
• Model connection failure: Add the model host and port to the deployment-owned egress allowlist.
• uv reports a package “not found in the cache”, or a Hugging Face load fails on a missing file: The host ran a wheels-v1 package offline because the deployment allows no egress beyond the model. Add the missing wheel to wheels/, or vendor the dataset in the package.

Related