> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.

# NeMo Gym Integration

> Set up NeMo Gym, run your reviewed Tier 3 cases in Docker, and open the SkillEvaluator report.

Run your Tier 3 tasks through **NeMo Gym**. SkillEvaluator prepares the cases,
grades the runs with and without your skill, and produces the comparison report.

> **Note**
>
> **Experimental · Docker · Linux (macOS untested).** Gym runs the agent and sandbox;
> SkillEvaluator supplies the shared verifier, scores, and reports.

#### [1. Set up](#install-the-pinned-gym)

Install Gym and connect your model.

#### [2. Run](#run-the-comparison)

Evaluate your reviewed dataset.

#### [3. See the result](#open-the-report)

Compare outcomes with and without the skill.

## Before you start

* [Install SkillEvaluator with all tiers](/skills/skillevaluator/installation#all-tiers), version **1.0.1** or later.
* Have **uv**, **Git**, and **Docker** available. Start Docker before the readiness check.
* Choose a skill folder containing `SKILL.md` and [prepare a reviewed dataset](/skills/skillevaluator/prepare-dataset) at `evals/evals.json`.

The example below uses **Codex with OpenAI**. Replace `./my-skill` with your
skill folder. Model calls can incur charges.

### Install the pinned Gym

SkillEvaluator finds Gym on `PATH` and checks its compatibility. It does not
install Gym automatically. For the Codex walkthrough, use uv 0.13.4 or later
and let uv install Gym in its own Python environment:

**`Install the pinned Gym once`**

```bash title="Install the pinned Gym once"
uv tool install --python ">=3.13.14,<3.14" --reinstall \
  "nemo-gym @ git+https://github.com/NVIDIA-NeMo/Gym@792fa142f58277aecff05d402061d647e9a92acf"
```

Use the pinned source above: public `nemo-gym==0.7.0` has an incompatible CLI.
The readiness check below verifies the required interface automatically.
For OpenCode, use the editable setup in the advanced reference below.

### Connect your model

Export these in the same terminal as the evaluation. The OpenAI key serves
both the evaluator and Gym's agent in this example.

**`Connect the evaluator and the live agent`**

```bash title="Connect the evaluator and the live agent"
export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'
export SKILL_EVAL_LLM_MODEL=gpt-5.4-mini
export SKILL_EVAL_NEMO_GYM_MODEL_URL=https://api.openai.com/v1
export SKILL_EVAL_NEMO_GYM_MODEL_API_KEY="$OPENAI_API_KEY"
```

Codex needs an endpoint that supports the **OpenAI Responses API**; NVIDIA
Build does not support this route. Use a model your account can access. The
commands below select `gpt-5.4-mini` for the agent too. For separate evaluator
credentials, see [Providers & Credentials](/skills/skillevaluator/configuration).

### Check your dataset and runtime

Open and review `./my-skill/evals/evals.json` **before continuing**. If it is
missing, [create your dataset first](/skills/skillevaluator/prepare-dataset).

**`Validate the cases and check Gym readiness`**

```bash title="Validate the cases and check Gym readiness"
skillevaluator tier3 validate ./my-skill --strict
skillevaluator tier3 doctor --backend nemo-gym --env-mode docker \
  --agents codex --agent-model codex=gpt-5.4-mini
```

Fix failed checks before continuing. Doctor checks the CLI, Docker, and endpoint
reachability; it does **not** prove that credentials or the model will work.

### Run the comparison

**`One attempt per selected case, in both arms`**

```bash title="One attempt per selected case, in both arms"
skillevaluator tier3 evaluate ./my-skill \
  --backend nemo-gym --env-mode docker \
  --agents codex --agent-model codex=gpt-5.4-mini \
  --n-attempts 1 --no-stop-on-pass
```

This uses your existing cases. It does not generate a missing dataset.
Saved run settings can limit which cases run; check the reported coverage.
Gym always runs every attempt. `--no-stop-on-pass` only matters when
`evals/config.yml` sets `harbor.stop_on_pass: true`, which Gym otherwise rejects;
without that setting you can leave it out.

## Open the report

```bash
skillevaluator view ./my-skill
```

| Check                     | What to look for                                                                                                             |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Status and coverage**   | Both arms completed with enough evidence to compare.                                                                         |
| **Skill Lift and trials** | How outcomes changed, and what happened in individual cases.                                                                 |
| **Backend**               | The report shows **Backend: NeMo Gym** (`nemo-gym` in `result.json`); compare runs with matching models, agents, and judges. |

The command prints the path to the latest `report.html` and opens it in your browser.
By default, the run directory is under `./my-skill/evals/results/`. It also contains
`result.json` and, when collected, Gym logs and rollouts under `_nemo-gym/`.
[Read the results guide](/skills/skillevaluator/tier3-results).

## Supported configurations

Use **Docker on Linux**. macOS is allowed but has not had a live run. Windows, local mode, and managed sandbox
backends are not supported by this adapter.

| Agent           | Current status                                                                                                                                                     |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Codex**       | Completed with-skill and baseline live runs on the pinned Gym with GPT-5.4-mini; the example above uses this path.                                                 |
| **Claude Code** | Completed with-skill and baseline live runs on the pinned Gym with Claude Sonnet 4.6; its mapping is experimental.                                                 |
| **OpenCode**    | Completed with-skill and baseline live runs on the pinned Gym with GPT-5.4-mini. Use the editable checkout below; its wheel installation fails during agent setup. |

Support for every agent, grading mode, and custom task combination has not
been qualified. Use the [Harbor walkthrough](/skills/skillevaluator/tier3-live-evaluation)
for other environments or unsupported run options.

## Troubleshooting

#### Gym is missing or its CLI is incompatible

Make sure `gym` on `PATH` comes from the pinned install. If needed, run
`uv tool update-shell` and restart your shell. An explicit
`SKILL_EVAL_NEMO_GYM_BIN` takes precedence; unset it to use `PATH`.
Installing public Gym 0.7.0 does not supply the required interface.

#### Doctor passes, but the model call fails

Check the policy URL, API key, and explicit `--agent-model` selection.
Codex and Claude Code need a Responses-compatible endpoint. Doctor's unauthenticated
reachability probe can pass even when the endpoint rejects authentication.
Check the failed trial and `_nemo-gym/` logs for the inference error.

#### An option or saved Harbor setting is rejected

Review `evals/config.yml` as well as CLI flags. Gym rejects unsupported
settings before starting; it never silently switches to Harbor. Common
examples are early stopping, baseline reuse, budget controls, and MCP
servers. Use the detailed options reference below or choose Harbor.

#### Docker fails, or a run is interrupted

Start Docker and check `docker info`. After a timeout or interruption,
check for leftover Gym servers and Docker sandboxes before retrying.
Inspect run status and execution errors before interpreting partial scores.

## Advanced reference

Expand only the details you need for another agent, custom tasks, or saved run settings.

#### Install Gym for OpenCode

OpenCode needs a persistent editable checkout so Gym can find its agent source
files. Keep the checkout after installing and export the executable path in the
terminal that runs the evaluation:

**`Install the pinned Gym in editable mode`**

```bash title="Install the pinned Gym in editable mode"
git clone --filter=blob:none https://github.com/NVIDIA-NeMo/Gym.git "$HOME/nemo-gym/source"
git -C "$HOME/nemo-gym/source" checkout --detach 792fa142f58277aecff05d402061d647e9a92acf
uv venv --python ">=3.13.14,<3.14" "$HOME/nemo-gym/.venv"
uv pip install --python "$HOME/nemo-gym/.venv/bin/python" \
  --editable "$HOME/nemo-gym/source"
export SKILL_EVAL_NEMO_GYM_BIN="$HOME/nemo-gym/.venv/bin/gym"
```

#### Gym interface and compatibility

**Why the pinned commit:** the public `nemo-gym==0.7.0` release reads JSONL inputs and
configured environments. It has no task-folder argument and no `--sandbox` option, so
SkillEvaluator can't use it. The pinned commit
[`792fa142`](https://github.com/NVIDIA-NeMo/Gym/blob/792fa142f58277aecff05d402061d647e9a92acf/nemo_gym/cli/main.py)
adds both. Install that commit until a public release includes them.

**What has been run live:** Docker runs completed both arms for Codex and OpenCode with
GPT-5.4-mini, and for Claude Code with Claude Sonnet 4.6, and kept Gym's execution and
verifier artifacts. These runs show the backend works end to end. They don't measure
skill quality or cover every model, grading mode, or custom Dockerfile.

#### Claude Code model routing and skill discovery

The Claude Code mapping uses the same pinned runtime version as Harbor, installed
by Gym inside the sandbox. It enables native discovery and projects staged skills
into the task workdir's `.claude/skills/` directory because Gym uses a private
Claude configuration directory. The baseline uses the same discovery settings
without the target skill. Gym's required Anthropic credential field receives only
a placeholder; policy credentials remain in the model-server environment. Claude Code
takes its policy endpoint from the same settings as Codex: `SKILL_EVAL_NEMO_GYM_MODEL_URL`,
`SKILL_EVAL_NEMO_GYM_MODEL_API_KEY` and a model from `--model` or
`--agent-model claude-code=MODEL`.
Like Codex, it needs an endpoint that supports the OpenAI Responses API; NVIDIA Build
returns 404 for `/v1/responses`.

#### Readiness probes and verifier contract

`tier3 doctor --backend nemo-gym` checks the Gym CLI interface and its version, that the Docker
daemon answers, the `SKILL_EVAL_NEMO_GYM_*` settings, each selected agent's policy model
endpoint, and the selected agents and `--env-mode`. The endpoint check uses
an isolated, unauthenticated HTTP probe with a three-second deadline. It honors
`HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (also lowercase), but sends no endpoint
or proxy credentials and does not follow redirects. Policy endpoint HTTP responses,
including authentication errors, count as reachable. Proxy authentication failures
(`407`), failed HTTPS tunnels, and plain-HTTP proxy gateway errors (`502`, `504`)
fail the check instead. HTTPS origin responses after a successful tunnel still
count as reachable, including `502` and `504`. It does not
validate credentials or model compatibility. The separate CLI interface check runs
`gym eval run --help` with a 15-second timeout and a combined stdout/stderr limit of
256 KiB, without policy credentials or server startup. Timeout, interruption, or output
overflow stops its isolated process group with bounded TERM/KILL cleanup. It rejects
missing task-folder arguments and required flags before dataset
generation or task staging. Passing this necessary check does not certify live runtime
compatibility. A Gym run reports
configuration, Docker image, per-agent and collection progress.

The adapter's required Gym contract is `gym eval run TASKS --sandbox docker`, with
`--agent`, `--model-type openai_model`, `--model`, `--model-url`, `--output`, and
`--concurrency`, plus Hydra overrides for repeats, credentials and cache disabling.
Its help must describe a standalone positional `TARGET`, `tasks`, `task_dir`,
`tasks_dir` or `task_folder` argument. Each rollout must identify its task and provide
`verifier_return_code` and `verifier_logs_dir`, where the SkillEvaluator verifier's rewards and trajectory can
be collected. The Gym implementation must execute inside Docker and resolve task
environment templates; a host-only agent harness does not meet this contract.

#### Options, saved configuration, and retained artifacts

**Agents and grading modes**

The adapter maps `codex`, `claude-code` and `opencode` and accepts `default`,
`default_plus_custom` and `custom_only` grading modes. These are adapter capabilities;
the live-run status in the table above does not qualify every combination.

**Works as on Harbor**

| Setting                                                                                                                                                                                                            | On Gym                                                                                                                                            |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evals/config.yml`: `harbor.n_attempts`, `pass_threshold`, `n_concurrent`, `max_cases`, `agents.<agent>.model`, `resources` (`cpus`, `memory_mb`), `runtime_env`, `grading.mode`, `skill_workspace.mode`/`include` | Unset flags fall back to these configuration keys, as on Harbor.                                                                                  |
| `--skip-baseline`, `--skip-with-skill`, `--max-cases`, `--agent-model`                                                                                                                                             | Apply the same way; `--agent-model` selects one model per agent.                                                                                  |
| `--timeout-multiplier`, `harbor.timeout_multiplier`                                                                                                                                                                | Scale each task's agent, verifier, and environment-build timeouts as on Harbor; Gym enforces them.                                                |
| `harbor.agent_workdir`, `harbor.pre_agent_setup` (alias `setup_commands`, with-skill-only runs)                                                                                                                    | Work as on Harbor: Gym starts the agent in that working directory and runs setup commands as the task's environment healthcheck before the agent. |
| `--copy-repo`                                                                                                                                                                                                      | Stages the whole repository under `/workspace/repo/` as on Harbor.                                                                                |
| `skill_workspace.decoy_manifest`                                                                                                                                                                                   | Stages the same verified decoy snapshot into every arm.                                                                                           |
| Native `evals/harbor/` tasks (`task_source: native_harbor`, or `auto` without `evals/evals.json`); `--override-cpus`, `--override-memory-mb`                                                                       | Use Harbor's own native staging and run on Gym; verifiers may write `reward.json` or a numeric `reward.txt`, and overrides apply as on Harbor.    |
| `harbor.base_image_mode` (`reuse`/`rebuild`)                                                                                                                                                                       | Builds Harbor's shared eval base image that the task images start from.                                                                           |
| `--custom-dockerfile-mode` (`rebase`, `preserve`)                                                                                                                                                                  | Behave as on Harbor.                                                                                                                              |
| `--environment-kwarg`                                                                                                                                                                                              | Rejected, as Harbor rejects it in Docker mode.                                                                                                    |

**Rejected before the run starts**

* **Environment modes:** Every non-Docker `--env-mode` (including `local`, `daytona`, `e2b` and `singularity`) is rejected on the NeMo Gym path; use `--backend harbor` for those modes.
* **Flags:** Gym rejects `--environment-kwarg`, `--stop-on-pass`, `--harbor-keep-jobs`, `--agent-runtime-preflight`, `--override-storage-mb`, `--reuse-baseline`, `--compare-to`, `--max-agent-tokens`, `--max-judge-calls`, `--max-attempts-total`, `--max-cost-usd`, `--price-file` and `--tolerate-baseline-timeouts`.
* **Configuration keys:** Gym rejects `resources.storage_mb`, `max_agent_tokens`, `max_judge_calls`, `max_attempts_total`, `max_cost_usd`, `price_file` and `tolerate_baseline_timeouts: true` under `harbor`.
* **Other conditions:** Gym rejects MCP servers in `evals/environment/mcp_servers.toml` or a native task's `[[environment.mcp_servers]]`, a model URL with embedded credentials, and Windows hosts, whether set by flag or config (a flag overrides its config key).
* These restrictions apply before anything starts. `tier3 PATH --autopilot` checks them before it writes a dataset. Gym never silently switches or falls back to Harbor.

**Credentials and environment**

`runtime_env` (alias `passthrough_env`) entries reach the agent's sandbox as
`${NAME}` references resolved from Gym's environment. Gym receives only the
variables Harbor's Docker backend allows (including `DOCKER_HOST`, `DOCKER_CONTEXT`
and TLS settings), Gym's `UV_*` package-index settings and credentials, `NETRC`, and
the policy and judge credentials; relative Docker and TLS paths are made absolute first.
Judge overrides (`LLM_JUDGE_MODEL`, `SKILL_EVAL_JUDGE_*`) reach Gym's verifier as on
Harbor, and Vertex AI ADC tokens are refreshed before every `gym eval run`.

**Failures, interruptions and artifacts**

Every agent and arm is staged from one skill snapshot before the first launch. A
missing or failed attempt counts as a failed trial, and the arm keeps its other
scored trials. Excess rollouts are reported as execution errors and scoring remains
bounded by the planned attempts. Any launch or rollout-count failure marks the run,
the agent's progress row and the report as failed. If Gym's process group cannot be
confirmed stopped after a timeout or interruption, no further arm starts beside it.
After a timeout, the run records that as an execution error, finishes as failed and
keeps the scores of the arms that finished. After an interruption, the progress
output reports it and the run ends. Either way, check for leftover Gym servers and
Docker sandboxes before you run again. A finished run updates `latest` and records
the same dataset snapshot and publication target as a Harbor run. As on Harbor
without `--harbor-keep-jobs`, the staged tasks and per-trial files are removed at
the end; Gym's own redacted `gym.log` and its `rollouts.jsonl` stay under
`_nemo-gym/` in the run folder, and the rollouts hold the raw agent responses.