NeMo Gym Integration
Run your Tier 3 tasks through NeMo Gym. SkillEvaluator prepares the cases, grades the runs with and without your skill, and produces the comparison report.
Experimental · Docker · Linux (macOS untested). Gym runs the agent and sandbox; SkillEvaluator supplies the shared verifier, scores, and reports.
Before you start
- Install SkillEvaluator with all tiers, version 1.0.1 or later.
- Have uv, Git, and Docker available. Start Docker before the readiness check.
- Choose a skill folder containing
SKILL.mdand prepare a reviewed dataset atevals/evals.json.
The example below uses Codex with OpenAI. Replace ./my-skill with your
skill folder. Model calls can incur charges.
Install the pinned Gym
SkillEvaluator finds Gym on PATH and checks its compatibility. It does not
install Gym automatically. For the Codex walkthrough, use uv 0.13.4 or later
and let uv install Gym in its own Python environment:
Use the pinned source above: public nemo-gym==0.7.0 has an incompatible CLI.
The readiness check below verifies the required interface automatically.
For OpenCode, use the editable setup in the advanced reference below.
Connect your model
Export these in the same terminal as the evaluation. The OpenAI key serves both the evaluator and Gym’s agent in this example.
Codex needs an endpoint that supports the OpenAI Responses API; NVIDIA
Build does not support this route. Use a model your account can access. The
commands below select gpt-5.4-mini for the agent too. For separate evaluator
credentials, see Providers & Credentials.
Check your dataset and runtime
Open and review ./my-skill/evals/evals.json before continuing. If it is
missing, create your dataset first.
Fix failed checks before continuing. Doctor checks the CLI, Docker, and endpoint reachability; it does not prove that credentials or the model will work.
Run the comparison
This uses your existing cases. It does not generate a missing dataset.
Saved run settings can limit which cases run; check the reported coverage.
Gym always runs every attempt. --no-stop-on-pass only matters when
evals/config.yml sets harbor.stop_on_pass: true, which Gym otherwise rejects;
without that setting you can leave it out.
Open the report
The command prints the path to the latest report.html and opens it in your browser.
By default, the run directory is under ./my-skill/evals/results/. It also contains
result.json and, when collected, Gym logs and rollouts under _nemo-gym/.
Read the results guide.
Supported configurations
Use Docker on Linux. macOS is allowed but has not had a live run. Windows, local mode, and managed sandbox backends are not supported by this adapter.
Support for every agent, grading mode, and custom task combination has not been qualified. Use the Harbor walkthrough for other environments or unsupported run options.
Troubleshooting
Gym is missing or its CLI is incompatible
Make sure gym on PATH comes from the pinned install. If needed, run
uv tool update-shell and restart your shell. An explicit
SKILL_EVAL_NEMO_GYM_BIN takes precedence; unset it to use PATH.
Installing public Gym 0.7.0 does not supply the required interface.
Doctor passes, but the model call fails
Check the policy URL, API key, and explicit --agent-model selection.
Codex and Claude Code need a Responses-compatible endpoint. Doctor’s unauthenticated
reachability probe can pass even when the endpoint rejects authentication.
Check the failed trial and _nemo-gym/ logs for the inference error.
An option or saved Harbor setting is rejected
Review evals/config.yml as well as CLI flags. Gym rejects unsupported
settings before starting; it never silently switches to Harbor. Common
examples are early stopping, baseline reuse, budget controls, and MCP
servers. Use the detailed options reference below or choose Harbor.
Docker fails, or a run is interrupted
Start Docker and check docker info. After a timeout or interruption,
check for leftover Gym servers and Docker sandboxes before retrying.
Inspect run status and execution errors before interpreting partial scores.
Advanced reference
Expand only the details you need for another agent, custom tasks, or saved run settings.
Install Gym for OpenCode
OpenCode needs a persistent editable checkout so Gym can find its agent source files. Keep the checkout after installing and export the executable path in the terminal that runs the evaluation:
Gym interface and compatibility
Why the pinned commit: the public nemo-gym==0.7.0 release reads JSONL inputs and
configured environments. It has no task-folder argument and no --sandbox option, so
SkillEvaluator can’t use it. The pinned commit
792fa142
adds both. Install that commit until a public release includes them.
What has been run live: Docker runs completed both arms for Codex and OpenCode with GPT-5.4-mini, and for Claude Code with Claude Sonnet 4.6, and kept Gym’s execution and verifier artifacts. These runs show the backend works end to end. They don’t measure skill quality or cover every model, grading mode, or custom Dockerfile.
Claude Code model routing and skill discovery
The Claude Code mapping uses the same pinned runtime version as Harbor, installed
by Gym inside the sandbox. It enables native discovery and projects staged skills
into the task workdir’s .claude/skills/ directory because Gym uses a private
Claude configuration directory. The baseline uses the same discovery settings
without the target skill. Gym’s required Anthropic credential field receives only
a placeholder; policy credentials remain in the model-server environment. Claude Code
takes its policy endpoint from the same settings as Codex: SKILL_EVAL_NEMO_GYM_MODEL_URL,
SKILL_EVAL_NEMO_GYM_MODEL_API_KEY and a model from --model or
--agent-model claude-code=MODEL.
Like Codex, it needs an endpoint that supports the OpenAI Responses API; NVIDIA Build
returns 404 for /v1/responses.
Readiness probes and verifier contract
tier3 doctor --backend nemo-gym checks the Gym CLI interface and its version, that the Docker
daemon answers, the SKILL_EVAL_NEMO_GYM_* settings, each selected agent’s policy model
endpoint, and the selected agents and --env-mode. The endpoint check uses
an isolated, unauthenticated HTTP probe with a three-second deadline. It honors
HTTP_PROXY, HTTPS_PROXY and NO_PROXY (also lowercase), but sends no endpoint
or proxy credentials and does not follow redirects. Policy endpoint HTTP responses,
including authentication errors, count as reachable. Proxy authentication failures
(407), failed HTTPS tunnels, and plain-HTTP proxy gateway errors (502, 504)
fail the check instead. HTTPS origin responses after a successful tunnel still
count as reachable, including 502 and 504. It does not
validate credentials or model compatibility. The separate CLI interface check runs
gym eval run --help with a 15-second timeout and a combined stdout/stderr limit of
256 KiB, without policy credentials or server startup. Timeout, interruption, or output
overflow stops its isolated process group with bounded TERM/KILL cleanup. It rejects
missing task-folder arguments and required flags before dataset
generation or task staging. Passing this necessary check does not certify live runtime
compatibility. A Gym run reports
configuration, Docker image, per-agent and collection progress.
The adapter’s required Gym contract is gym eval run TASKS --sandbox docker, with
--agent, --model-type openai_model, --model, --model-url, --output, and
--concurrency, plus Hydra overrides for repeats, credentials and cache disabling.
Its help must describe a standalone positional TARGET, tasks, task_dir,
tasks_dir or task_folder argument. Each rollout must identify its task and provide
verifier_return_code and verifier_logs_dir, where the SkillEvaluator verifier’s rewards and trajectory can
be collected. The Gym implementation must execute inside Docker and resolve task
environment templates; a host-only agent harness does not meet this contract.
Options, saved configuration, and retained artifacts
Agents and grading modes
The adapter maps codex, claude-code and opencode and accepts default,
default_plus_custom and custom_only grading modes. These are adapter capabilities;
the live-run status in the table above does not qualify every combination.
Works as on Harbor
Rejected before the run starts
- Environment modes: Every non-Docker
--env-mode(includinglocal,daytona,e2bandsingularity) is rejected on the NeMo Gym path; use--backend harborfor those modes. - Flags: Gym rejects
--environment-kwarg,--stop-on-pass,--harbor-keep-jobs,--agent-runtime-preflight,--override-storage-mb,--reuse-baseline,--compare-to,--max-agent-tokens,--max-judge-calls,--max-attempts-total,--max-cost-usd,--price-fileand--tolerate-baseline-timeouts. - Configuration keys: Gym rejects
resources.storage_mb,max_agent_tokens,max_judge_calls,max_attempts_total,max_cost_usd,price_fileandtolerate_baseline_timeouts: trueunderharbor. - Other conditions: Gym rejects MCP servers in
evals/environment/mcp_servers.tomlor a native task’s[[environment.mcp_servers]], a model URL with embedded credentials, and Windows hosts, whether set by flag or config (a flag overrides its config key). - These restrictions apply before anything starts.
tier3 PATH --autopilotchecks them before it writes a dataset. Gym never silently switches or falls back to Harbor.
Credentials and environment
runtime_env (alias passthrough_env) entries reach the agent’s sandbox as
${NAME} references resolved from Gym’s environment. Gym receives only the
variables Harbor’s Docker backend allows (including DOCKER_HOST, DOCKER_CONTEXT
and TLS settings), Gym’s UV_* package-index settings and credentials, NETRC, and
the policy and judge credentials; relative Docker and TLS paths are made absolute first.
Judge overrides (LLM_JUDGE_MODEL, SKILL_EVAL_JUDGE_*) reach Gym’s verifier as on
Harbor, and Vertex AI ADC tokens are refreshed before every gym eval run.
Failures, interruptions and artifacts
Every agent and arm is staged from one skill snapshot before the first launch. A
missing or failed attempt counts as a failed trial, and the arm keeps its other
scored trials. Excess rollouts are reported as execution errors and scoring remains
bounded by the planned attempts. Any launch or rollout-count failure marks the run,
the agent’s progress row and the report as failed. If Gym’s process group cannot be
confirmed stopped after a timeout or interruption, no further arm starts beside it.
After a timeout, the run records that as an execution error, finishes as failed and
keeps the scores of the arms that finished. After an interruption, the progress
output reports it and the run ends. Either way, check for leftover Gym servers and
Docker sandboxes before you run again. A finished run updates latest and records
the same dataset snapshot and publication target as a Harbor run. As on Harbor
without --harbor-keep-jobs, the staged tasks and per-trial files are removed at
the end; Gym’s own redacted gym.log and its rollouts.jsonl stay under
_nemo-gym/ in the run folder, and the rollouts hold the raw agent responses.