Skip to navigation

NeMo Gym Integration

View as MarkdownOpen in Claude

Run your Tier 3 tasks through NeMo Gym. SkillEvaluator prepares the cases, grades the runs with and without your skill, and produces the comparison report.

Experimental · Docker · Linux (macOS untested). Gym runs the agent and sandbox; SkillEvaluator supplies the shared verifier, scores, and reports.

Before you start

The example below uses Codex with OpenAI. Replace ./my-skill with your skill folder. Model calls can incur charges.

1

Install the pinned Gym

SkillEvaluator finds Gym on PATH and checks its compatibility. It does not install Gym automatically. For the Codex walkthrough, use uv 0.13.4 or later and let uv install Gym in its own Python environment:

Install the pinned Gym once
uv tool install --python ">=3.13.14,<3.14" --reinstall \
"nemo-gym @ git+https://github.com/NVIDIA-NeMo/Gym@792fa142f58277aecff05d402061d647e9a92acf"

Use the pinned source above: public nemo-gym==0.7.0 has an incompatible CLI. The readiness check below verifies the required interface automatically. For OpenCode, use the editable setup in the advanced reference below.

2

Connect your model

Export these in the same terminal as the evaluation. The OpenAI key serves both the evaluator and Gym’s agent in this example.

Connect the evaluator and the live agent
export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'
export SKILL_EVAL_LLM_MODEL=gpt-5.4-mini
export SKILL_EVAL_NEMO_GYM_MODEL_URL=https://api.openai.com/v1
export SKILL_EVAL_NEMO_GYM_MODEL_API_KEY="$OPENAI_API_KEY"

Codex needs an endpoint that supports the OpenAI Responses API; NVIDIA Build does not support this route. Use a model your account can access. The commands below select gpt-5.4-mini for the agent too. For separate evaluator credentials, see Providers & Credentials.

3

Check your dataset and runtime

Open and review ./my-skill/evals/evals.json before continuing. If it is missing, create your dataset first.

Validate the cases and check Gym readiness
skillevaluator tier3 validate ./my-skill --strict
skillevaluator tier3 doctor --backend nemo-gym --env-mode docker \
--agents codex --agent-model codex=gpt-5.4-mini

Fix failed checks before continuing. Doctor checks the CLI, Docker, and endpoint reachability; it does not prove that credentials or the model will work.

4

Run the comparison

One attempt per selected case, in both arms
skillevaluator tier3 evaluate ./my-skill \
--backend nemo-gym --env-mode docker \
--agents codex --agent-model codex=gpt-5.4-mini \
--n-attempts 1 --no-stop-on-pass

This uses your existing cases. It does not generate a missing dataset. Saved run settings can limit which cases run; check the reported coverage. Gym always runs every attempt. --no-stop-on-pass only matters when evals/config.yml sets harbor.stop_on_pass: true, which Gym otherwise rejects; without that setting you can leave it out.

Open the report

skillevaluator view ./my-skill
CheckWhat to look for
Status and coverageBoth arms completed with enough evidence to compare.
Skill Lift and trialsHow outcomes changed, and what happened in individual cases.
BackendThe report shows Backend: NeMo Gym (nemo-gym in result.json); compare runs with matching models, agents, and judges.

The command prints the path to the latest report.html and opens it in your browser. By default, the run directory is under ./my-skill/evals/results/. It also contains result.json and, when collected, Gym logs and rollouts under _nemo-gym/. Read the results guide.

Supported configurations

Use Docker on Linux. macOS is allowed but has not had a live run. Windows, local mode, and managed sandbox backends are not supported by this adapter.

AgentCurrent status
CodexCompleted with-skill and baseline live runs on the pinned Gym with GPT-5.4-mini; the example above uses this path.
Claude CodeCompleted with-skill and baseline live runs on the pinned Gym with Claude Sonnet 4.6; its mapping is experimental.
OpenCodeCompleted with-skill and baseline live runs on the pinned Gym with GPT-5.4-mini. Use the editable checkout below; its wheel installation fails during agent setup.

Support for every agent, grading mode, and custom task combination has not been qualified. Use the Harbor walkthrough for other environments or unsupported run options.

Troubleshooting

Make sure gym on PATH comes from the pinned install. If needed, run uv tool update-shell and restart your shell. An explicit SKILL_EVAL_NEMO_GYM_BIN takes precedence; unset it to use PATH. Installing public Gym 0.7.0 does not supply the required interface.

Check the policy URL, API key, and explicit --agent-model selection. Codex and Claude Code need a Responses-compatible endpoint. Doctor’s unauthenticated reachability probe can pass even when the endpoint rejects authentication. Check the failed trial and _nemo-gym/ logs for the inference error.

Review evals/config.yml as well as CLI flags. Gym rejects unsupported settings before starting; it never silently switches to Harbor. Common examples are early stopping, baseline reuse, budget controls, and MCP servers. Use the detailed options reference below or choose Harbor.

Start Docker and check docker info. After a timeout or interruption, check for leftover Gym servers and Docker sandboxes before retrying. Inspect run status and execution errors before interpreting partial scores.

Advanced reference

Expand only the details you need for another agent, custom tasks, or saved run settings.

OpenCode needs a persistent editable checkout so Gym can find its agent source files. Keep the checkout after installing and export the executable path in the terminal that runs the evaluation:

Install the pinned Gym in editable mode
git clone --filter=blob:none https://github.com/NVIDIA-NeMo/Gym.git "$HOME/nemo-gym/source"
git -C "$HOME/nemo-gym/source" checkout --detach 792fa142f58277aecff05d402061d647e9a92acf
uv venv --python ">=3.13.14,<3.14" "$HOME/nemo-gym/.venv"
uv pip install --python "$HOME/nemo-gym/.venv/bin/python" \
--editable "$HOME/nemo-gym/source"
export SKILL_EVAL_NEMO_GYM_BIN="$HOME/nemo-gym/.venv/bin/gym"

Why the pinned commit: the public nemo-gym==0.7.0 release reads JSONL inputs and configured environments. It has no task-folder argument and no --sandbox option, so SkillEvaluator can’t use it. The pinned commit 792fa142 adds both. Install that commit until a public release includes them.

What has been run live: Docker runs completed both arms for Codex and OpenCode with GPT-5.4-mini, and for Claude Code with Claude Sonnet 4.6, and kept Gym’s execution and verifier artifacts. These runs show the backend works end to end. They don’t measure skill quality or cover every model, grading mode, or custom Dockerfile.

The Claude Code mapping uses the same pinned runtime version as Harbor, installed by Gym inside the sandbox. It enables native discovery and projects staged skills into the task workdir’s .claude/skills/ directory because Gym uses a private Claude configuration directory. The baseline uses the same discovery settings without the target skill. Gym’s required Anthropic credential field receives only a placeholder; policy credentials remain in the model-server environment. Claude Code takes its policy endpoint from the same settings as Codex: SKILL_EVAL_NEMO_GYM_MODEL_URL, SKILL_EVAL_NEMO_GYM_MODEL_API_KEY and a model from --model or --agent-model claude-code=MODEL. Like Codex, it needs an endpoint that supports the OpenAI Responses API; NVIDIA Build returns 404 for /v1/responses.

tier3 doctor --backend nemo-gym checks the Gym CLI interface and its version, that the Docker daemon answers, the SKILL_EVAL_NEMO_GYM_* settings, each selected agent’s policy model endpoint, and the selected agents and --env-mode. The endpoint check uses an isolated, unauthenticated HTTP probe with a three-second deadline. It honors HTTP_PROXY, HTTPS_PROXY and NO_PROXY (also lowercase), but sends no endpoint or proxy credentials and does not follow redirects. Policy endpoint HTTP responses, including authentication errors, count as reachable. Proxy authentication failures (407), failed HTTPS tunnels, and plain-HTTP proxy gateway errors (502, 504) fail the check instead. HTTPS origin responses after a successful tunnel still count as reachable, including 502 and 504. It does not validate credentials or model compatibility. The separate CLI interface check runs gym eval run --help with a 15-second timeout and a combined stdout/stderr limit of 256 KiB, without policy credentials or server startup. Timeout, interruption, or output overflow stops its isolated process group with bounded TERM/KILL cleanup. It rejects missing task-folder arguments and required flags before dataset generation or task staging. Passing this necessary check does not certify live runtime compatibility. A Gym run reports configuration, Docker image, per-agent and collection progress.

The adapter’s required Gym contract is gym eval run TASKS --sandbox docker, with --agent, --model-type openai_model, --model, --model-url, --output, and --concurrency, plus Hydra overrides for repeats, credentials and cache disabling. Its help must describe a standalone positional TARGET, tasks, task_dir, tasks_dir or task_folder argument. Each rollout must identify its task and provide verifier_return_code and verifier_logs_dir, where the SkillEvaluator verifier’s rewards and trajectory can be collected. The Gym implementation must execute inside Docker and resolve task environment templates; a host-only agent harness does not meet this contract.

Agents and grading modes

The adapter maps codex, claude-code and opencode and accepts default, default_plus_custom and custom_only grading modes. These are adapter capabilities; the live-run status in the table above does not qualify every combination.

Works as on Harbor

SettingOn Gym
evals/config.yml: harbor.n_attempts, pass_threshold, n_concurrent, max_cases, agents.<agent>.model, resources (cpus, memory_mb), runtime_env, grading.mode, skill_workspace.mode/includeUnset flags fall back to these configuration keys, as on Harbor.
--skip-baseline, --skip-with-skill, --max-cases, --agent-modelApply the same way; --agent-model selects one model per agent.
--timeout-multiplier, harbor.timeout_multiplierScale each task’s agent, verifier, and environment-build timeouts as on Harbor; Gym enforces them.
harbor.agent_workdir, harbor.pre_agent_setup (alias setup_commands, with-skill-only runs)Work as on Harbor: Gym starts the agent in that working directory and runs setup commands as the task’s environment healthcheck before the agent.
--copy-repoStages the whole repository under /workspace/repo/ as on Harbor.
skill_workspace.decoy_manifestStages the same verified decoy snapshot into every arm.
Native evals/harbor/ tasks (task_source: native_harbor, or auto without evals/evals.json); --override-cpus, --override-memory-mbUse Harbor’s own native staging and run on Gym; verifiers may write reward.json or a numeric reward.txt, and overrides apply as on Harbor.
harbor.base_image_mode (reuse/rebuild)Builds Harbor’s shared eval base image that the task images start from.
--custom-dockerfile-mode (rebase, preserve)Behave as on Harbor.
--environment-kwargRejected, as Harbor rejects it in Docker mode.

Rejected before the run starts

  • Environment modes: Every non-Docker --env-mode (including local, daytona, e2b and singularity) is rejected on the NeMo Gym path; use --backend harbor for those modes.
  • Flags: Gym rejects --environment-kwarg, --stop-on-pass, --harbor-keep-jobs, --agent-runtime-preflight, --override-storage-mb, --reuse-baseline, --compare-to, --max-agent-tokens, --max-judge-calls, --max-attempts-total, --max-cost-usd, --price-file and --tolerate-baseline-timeouts.
  • Configuration keys: Gym rejects resources.storage_mb, max_agent_tokens, max_judge_calls, max_attempts_total, max_cost_usd, price_file and tolerate_baseline_timeouts: true under harbor.
  • Other conditions: Gym rejects MCP servers in evals/environment/mcp_servers.toml or a native task’s [[environment.mcp_servers]], a model URL with embedded credentials, and Windows hosts, whether set by flag or config (a flag overrides its config key).
  • These restrictions apply before anything starts. tier3 PATH --autopilot checks them before it writes a dataset. Gym never silently switches or falls back to Harbor.

Credentials and environment

runtime_env (alias passthrough_env) entries reach the agent’s sandbox as ${NAME} references resolved from Gym’s environment. Gym receives only the variables Harbor’s Docker backend allows (including DOCKER_HOST, DOCKER_CONTEXT and TLS settings), Gym’s UV_* package-index settings and credentials, NETRC, and the policy and judge credentials; relative Docker and TLS paths are made absolute first. Judge overrides (LLM_JUDGE_MODEL, SKILL_EVAL_JUDGE_*) reach Gym’s verifier as on Harbor, and Vertex AI ADC tokens are refreshed before every gym eval run.

Failures, interruptions and artifacts

Every agent and arm is staged from one skill snapshot before the first launch. A missing or failed attempt counts as a failed trial, and the arm keeps its other scored trials. Excess rollouts are reported as execution errors and scoring remains bounded by the planned attempts. Any launch or rollout-count failure marks the run, the agent’s progress row and the report as failed. If Gym’s process group cannot be confirmed stopped after a timeout or interruption, no further arm starts beside it. After a timeout, the run records that as an execution error, finishes as failed and keeps the scores of the arms that finished. After an interruption, the progress output reports it and the run ends. Either way, check for leftover Gym servers and Docker sandboxes before you run again. A finished run updates latest and records the same dataset snapshot and publication target as a Harbor run. As on Harbor without --harbor-keep-jobs, the staged tasks and per-trial files are removed at the end; Gym’s own redacted gym.log and its rollouts.jsonl stay under _nemo-gym/ in the run folder, and the rollouts hold the raw agent responses.