Integrate Existing Agents

You can use an existing agent in NeMo Gym, integrate an external one, or build your own from scratch.

To use an agent that Gym does not integrate yet, see Add an agent that is not listed.

Built-in agents

SimpleAgent is a native NeMo Gym agent that handles general-purpose multi-step tool calling with configurable max steps, and works with any Resources server out of the box. NeMo Gym also includes agents that integrate external tools: for example, MiniSWEAgent wraps an external coding harness running in Docker containers and converts its output back into the NeMo Gym format, and claude_code_agent runs the Claude Code CLI. Because every Gym model server speaks Anthropic Messages, you can point Claude Code at any backend — see Anthropic Messages.

Sandboxed harnesses

These adapters run a harness inside a sandbox. The single_agent_turn Environment Server drives them with a task sandbox when the Resources server supplies one; otherwise a configured provider creates an agent-owned sandbox. The harbor Environment Server drives them in the sandbox of a Harbor task folder. Each row summarizes the adapter’s README; the README is the reference for supported features, runtime requirements, and limits.

Declared means the benchmark’s Resources server lists the agent in allowed_agents. Gym rejects any other pairing unless you pass --allow-unsupported-pairing. Declared support is not a validation: it says nothing about accuracy or training readiness. Check that a pairing works with a real model and the benchmark’s verifier before you rely on it.

On Harbor task folders, you pick the harness with gym eval run <folder> --agent <harness>, and the allowed_agents lists above do not apply: a dataset limits harnesses with allowed_agents in the [gym] table of its dataset.toml. terminus_2_sandboxed_agent and miniswe_sandboxed_agent also run task folders. For tasks declaring a stdio MCP server or a skills_dir, use --agent mini_swe_agent/mini_swe_agent_sandboxed; the older miniswe_sandboxed_agent remains compatible but is deprecated.

Copilot pairings remain experimental pending full benchmark qualification and benchmark-owner review. Use --allow-unsupported-pairing for validation runs; the SWE-bench Pro pairing also requires task-network access to install its runtime.

The allowed_agents lists also name older adapters, such as opencode_sandboxed_agent and terminus_2_sandboxed_agent. They run the whole episode themselves behind a legacy_agent Environment Server, and shipped recipes such as terminal_bench_2_1/opencode use them. To run them or swap one for another, see Compare Harnesses.

Kilo MCP and sandbox ownership

Kilo’s session adapter accepts remote MCP grants from Resources. Set the Environment Server’s resources_tool_transports: [mcp] to expose Resources tools through MCP. Grant URLs and headers must be reachable and valid from the task sandbox. Required direct HTTP grants are still rejected; this does not make a recipe configured with [direct_http] compatible with Kilo.

The adapter configures Kilo’s remote MCP clients with each grant’s URL and headers and disables interactive OAuth. Before accepting a session with required MCP grants, it runs a supervised CLI check that connects to those servers and discovers tools without requesting a model completion. The activation’s CLI process connects again, and the bundled plugin confirms the required connections before its first model request. Kilo does not refresh connection status after that, so a server lost mid-run surfaces as failed tool calls. A failed connection or a missing readiness receipt fails the session or activation. Optional MCP connection failures do not fail this readiness check. Kilo cancels an MCP tool call after kilo_config.experimental.mcp_timeout milliseconds; kilocode_sandboxed.yaml sets 300000 (five minutes) instead of the MCP SDK’s 60-second default, so keep it above your slowest Resources tool and within timeout. Static servers in kilo_config.mcp remain available, but their names must not collide with grant names, including after Kilo replaces punctuation other than - and _ with _.

When Resources supplies sandbox_access, Kilo borrows that sandbox and leaves its destruction to Resources. When Resources supplies no sandbox, configure an agent-owned sandbox explicitly. For example, add these imports and overrides to an experiment that already defines policy_model, policy_model_name, and the Environment Server’s Resources/agent bindings:

config_paths:
  - responses_api_agents/kilocode_agent/configs/kilocode_sandboxed.yaml
  - nemo_gym/sandbox/providers/docker/configs/docker.yaml
kilocode_agent:
  responses_api_agents:
    kilocode_agent:
      sandbox_provider: sandbox
      sandbox_config:
        image: python:3.13-slim
        workdir: /app

The Docker provider needs a local Docker daemon. Configure the Gym host and network so the sandbox can reach both the Gym Model Server and the MCP endpoints; a host loopback URL is not reachable from Docker’s default bridge network. The Linux image must meet Kilo’s runtime requirements, including network access for the pinned CLI installation.

Agent-owned sandboxes inherit provider metadata defaults, then sandbox_config.metadata, with Gym’s agent/session/rollout labels taking precedence. The agent stops its sandbox on close or failed setup. If neither Resources sandbox access nor sandbox_provider is available, seed fails with a configuration error.

Pi and OpenCode prompt and tool controls

Both pi_agent and opencode_agent accept system_prompt_mode: append|replace, defaulting to append. Replacement uses the harness’s custom system prompt for the combined configured system prompt, request instructions and input system instructions. It keeps the user’s task separate. Append mode retains each adapter’s existing behavior. These controls apply to local calls and sandbox sessions; OpenCode continuations retain their established instructions. The harness may still add environment context, such as the date or working directory. OpenCode replacement selects its build agent, overriding a different default agent. Remove --agent selectors from extra_args in replacement mode; use append mode to retain custom agent selection. Replacement prompt text is passed literally, including {env:...} and {file:...} sequences. Other OpenCode configuration keeps its native substitution behavior.

Pi’s enable_tools defaults to true. Setting it to false disables Pi’s built-in tools through --no-builtin-tools; separately configured MCP tools and session grants remain available and retain their validation. For a run with no tools, also omit MCP configuration and grants. For OpenCode, withhold tools with opencode_config.permission rules, for example task: deny. To withhold every tool, also deny edit and bash through their own maps (edit: {"**": deny}, bash: {"*": deny}): opencode_agent’s permission profile allows them after its "*" rule, and a "*": deny override keeps that rule’s place before them. Do not use an opencode_config.tools map with opencode_agent: OpenCode places its keys before the profile’s "*": allow, which then overrides them (see the opencode_agent README).

Record prompt replacement and tool selection as evaluation treatments: they can change results relative to the harness’s default coding behavior.

Run an integrated agent on a benchmark

An experiment is a few composable YAML files: the model and provider, the benchmark, the agent, and the Environment Server that binds them. To change the agent or the benchmark, replace one file; the others stay as they are.

Harbor task folders need none of these files: gym eval run <folder> --agent <harness> writes the agent and Environment Server blocks to results/harbor/<taskset>/run_config_<harness>.yaml. See Run an agent.

Configure an experiment

The following evaluation example pairs SWE-bench Pro with Hermes. Before you start, install a current Gym checkout with the agent session adapters, Gym’s development environment, and the benchmark’s preparation dependencies. Save the four files below under experiments/onboarding/, and run all commands from the repository root. Entries in config_paths are relative to the root.

You also need a running vLLM-compatible model endpoint and an OpenSandbox deployment. Export the following variables in your shell:

• POLICY_BASE_URL, POLICY_API_KEY, and POLICY_MODEL_NAME for the model endpoint.
• OPENSANDBOX_DOMAIN and OPENSANDBOX_API_KEY for the sandbox provider.
• GYM_HOST, a bindable address of the Gym host that task sandboxes can reach. Do not use a loopback address or 0.0.0.0, because sandboxes cannot call back to them.

On Linux, you can find a candidate IPv4 address by running this command on the Gym node:

export GYM_HOST="$(getent ahostsv4 "$(hostname -f)" | awk 'NR==1 {print $1}')"

Use the result only if it is a local interface address that the task sandboxes can reach. If the command returns no address, or the host has several interfaces with different routes to the sandboxes, set GYM_HOST explicitly to the right address. Hostname resolution alone does not prove connectivity.

The sandbox must reach the Gym Model Server’s allocated port, and the Gym Model Server must reach the upstream endpoint. Keep credentials out of committed YAML files.

# experiments/onboarding/model-provider.yaml
config_paths:
  - responses_api_models/vllm_model/configs/vllm_model.yaml
  - nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml
default_host: ${oc.env:GYM_HOST}
policy_base_url: ${oc.env:POLICY_BASE_URL}
policy_api_key: ${oc.env:POLICY_API_KEY}
policy_model_name: ${oc.env:POLICY_MODEL_NAME}

Give the benchmark and the harness stable instance names. The outer keys, benchmark and agent, are instance names that references and the CLI use. The inner keys, such as swebench_pro and hermes_agent, select implementation directories. _inherit_from reuses the imported template and applies your overrides; it does not start an extra copy of the server.

# experiments/onboarding/benchmark.yaml
config_paths:
  - resources_servers/swebench_pro/configs/swebench_pro.yaml
benchmark:
  _inherit_from: swebench_pro_resources_server
  resources_servers:
    swebench_pro:
      datasets:
        - name: swebench_pro
          type: benchmark
          taskset: onboarding
          jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl
          prepare_script: benchmarks/swebench/pro/prepare.py
          num_repeats: 1

# experiments/onboarding/agent.yaml
config_paths:
  - responses_api_agents/hermes_agent/configs/hermes_agent.yaml
agent:
  _inherit_from: hermes_agent
  responses_api_agents:
    hermes_agent:
      resources_server: null
      num_workers: 1
      enabled_toolsets: [terminal]

The benchmark supplies the task sandbox. Its imported configuration sets sandbox_provider: sandbox, image selection, resources, and verification settings. The agent imports its runtime settings and the model_server: {type: responses_api_models, name: policy_model} binding. resources_server: null leaves benchmark selection to the Environment Server.

# experiments/onboarding/run.yaml
config_paths:
  - experiments/onboarding/model-provider.yaml
  - experiments/onboarding/benchmark.yaml
  - experiments/onboarding/agent.yaml
  - environment_servers/single_agent_turn/configs/single_agent_turn.yaml
environment_routing_mode: taskset
environment_server_routes:
  onboarding: experiment
experiment:
  _inherit_from: single_agent_turn
  environment_servers:
    single_agent_turn:
      resources_server: {type: resources_servers, name: benchmark}
      agent_server: {type: responses_api_agents, name: agent}
      resources_tool_transports: []
      max_concurrent_episodes: 1
      queue_timeout_seconds: 60
      # Above Hermes installation (900) + runner (21600) + SWE-bench Pro verification (2700), plus seed and agent close.
      default_episode_timeout_seconds: 28800
      # Each cleanup callback, such as agent close: Pi needs up to 5 x 60 + 60 = 360 seconds, Hermes 5 x 30 + 30.
      cleanup_timeout_seconds: 360

With taskset: onboarding, collation wraps each prepared row in task_id and task_input and keeps the benchmark’s task fields. environment_server_routes maps that taskset to the experiment instance, whose implementation is single_agent_turn. The collector assigns the episode identity and calls the instance’s /run endpoint. environment_routing_mode: taskset rejects flat rows submitted by mistake. For the row contract, see taskset data declarations.

Prepare the dataset and inspect the resolved configuration:

gym eval prepare --config experiments/onboarding/run.yaml
gym env resolve --config experiments/onboarding/run.yaml

Then run one task. This command collates the declared benchmark data into materialized task rows, starts the servers, collects rollouts, and shuts the servers down:

gym eval run \
  --config experiments/onboarding/run.yaml \
  --output results/onboarding/swe-pro-hermes/rollouts.jsonl \
  --split benchmark \
  --limit 1 --concurrency 1

The taskset route selects the Environment Server, and its agent_server reference selects the harness, so you do not need an --agent option. The source JSONL is not modified. The collated input for this run is written to results/onboarding/swe-pro-hermes/rollouts/preprocessed_datasets/benchmark.jsonl.

Before you increase concurrency, inspect the response, reward, failure sidecar, and cleanup evidence. When you raise --concurrency, raise max_concurrent_episodes to match and set queue_timeout_seconds above the expected episode duration; otherwise queued episodes fail admission after 60 seconds. A configuration that resolves has not yet been validated: resolution does not check task-image installation, model and tool behavior, or verification.

For callers that still submit flat rows, single_agent_turn_legacy wraps the same session lifecycle as a migration adapter. Name both resources_server and agent_server in its block. The separate legacy_agent implementation delegates to the agent’s old /run endpoint and names only agent_server. New experiments should use the taskset configuration above.

Swap the agent or the benchmark

Wait for the experiment to finish and its servers to shut down before you change the configuration. Then resolve and run it again. Use a new output directory for each variant, so that resume does not pick up another experiment’s results.

Replace Hermes with Pi. Replace the whole agent.yaml with the following file. Remove Hermes-only settings such as enabled_toolsets. The agent instance name does not change, so run.yaml, the taskset route, and the SWE-bench Pro preparation and input stay the same.

# experiments/onboarding/agent.yaml (Pi replacement)
config_paths:
  - responses_api_agents/pi_agent/configs/pi_agent.yaml
agent:
  _inherit_from: pi_agent
  responses_api_agents:
    pi_agent:
      resources_server: null
      num_workers: 1

Before you run, check Pi’s runtime and request requirements: the same model and input must satisfy the new adapter’s contract. Repeat the resolve and run commands with the output path results/onboarding/swe-pro-pi/rollouts.jsonl. Pi uses its own sandbox tools, so resources_tool_transports: [] still applies.

Replace SWE-bench Pro with Terminal Bench 2.1. Replace the whole benchmark.yaml with the following file, and keep whichever agent.yaml you selected. The dataset declaration selects the new preparation script and input, and taskset: onboarding keeps the route to experiment. The prepared rows supply each task’s image (docker_image), the new Resources Server configuration supplies its resource requirements and verifier, and run.yaml still references benchmark. Repeat the prepare, resolve, and run commands with a new output directory.

# experiments/onboarding/benchmark.yaml (Terminal Bench 2.1 replacement)
config_paths:
  - resources_servers/terminal_bench_2_1/configs/terminal_bench_2_1.yaml
benchmark:
  _inherit_from: terminal_bench_2_1_resources_server
  resources_servers:
    terminal_bench_2_1:
      datasets:
        - name: terminal_bench_2_1
          type: benchmark
          taskset: onboarding
          jsonl_fpath: benchmarks/terminal_bench_2_1/data/benchmark.jsonl
          prepare_script: benchmarks/terminal_bench_2_1/prepare.py
          num_repeats: 1

To swap a component, change the imported configuration and its matching implementation block together. Keep task data and preparation in the benchmark declaration, and keep harness selection in the Environment Server’s agent_server reference. The --agent-type shortcut for legacy agent-to-Resources bindings does not replace these edits. To compare agents on the same endpoint, keep the model and provider file unchanged. To compare models or providers, change that file in a separate experiment.

For where the harness runs and who cleans up, including borrowed and agent-owned sandboxes and host execution, see Select an execution mode.

Run Codex on Terminal-Bench-Science

Terminal-Bench-Science v0.1 ships a ready-made Codex profile, terminal_bench_science/codex, so you do not write the four files above. Unlike the Terminal Bench 2.1 adapters, its Resources Server is terminal_bench_4: it provisions each pinned task, lends the task sandbox to codex_agent, and grades in a separate offline verifier sandbox. The profile composes single_agent_turn_legacy, so it reads flat rows rather than a taskset. Like the other session paths, it is for evaluation only.

Before you run it:

• Build the task images. Upstream publishes none. Build each task’s agent and verifier image, push them to a registry that your sandbox deployment can pull from, and pass them as a tbsci_task_images map or as tbsci_agent_image and tbsci_verifier_image templates. See Images.
• Prepare the rows. Each row carries the task’s official instruction, read from the pinned package after its hashes are checked. The check needs pathspec, which Gym’s own environment leaves out, so gym eval prepare --benchmark terminal_bench_science/codex runs it in the Resources Server’s environment and sets that environment up first if needed, as the README shows.
• Bind Gym to a reachable address. Codex calls Gym’s model server from inside the task sandbox, so pass ++default_host with an address that the sandboxes can reach, as with GYM_HOST above.

codex_deepseek_flash.yaml is the run configuration for the leaderboard’s DeepSeek V4.1 Flash row, served by your own chat-completions server. Pass your images in a second --config file, here tbsci-images.yaml, that sets tbsci_task_images:

export POLICY_API_KEY=...  # any string if the server has no auth
gym eval run --config benchmarks/terminal_bench_science/codex_deepseek_flash.yaml \
  --config tbsci-images.yaml \
  --split benchmark --model-url http://SERVER_HOST:PORT/v1 \
  --output results/tbsci-deepseek/rollouts.jsonl --concurrency 8 \
  ++default_host=$GYM_HOST

The profile runs one trial per task. The leaderboard’s protocol is three trials per task, 210 in all: add ++num_repeats=3. For the Codex defaults and the model server settings, see Codex with DeepSeek V4.1 Flash.

Resuming harness sessions

OpenCode can resume its saved session across multiple activations in either an owned or borrowed sandbox. Opt in during /v1/agent_sessions seed with continuation: {mode: resume_harness_session}. Existing adapters reject continuation requirements they cannot satisfy before preparing the sandbox.

Use the existing /v1/responses request body and session cookie. Send x-ng-agent-activation: 1 for the first input delta, 2 for the next, and so on. Identical requests at the same index replay the original response. Conflicting retries, skipped indices, and overlapping new activations fail. The existing metadata.agent_timeout_sec sets the per-activation budget, including preparation, queueing and staging. Setup and cleanup retain their separate bounds.

Adapters that support continuation override _agent_continuation_capabilities() to return the supported mode and observations, implement _validate_agent_activation() before binding an index, _activate_agent_session_state() for execution, and _continuation_close_timeout() for their cleanup bound. The ordinary responses() handler delegates to activate_agent_session() only for opted-in sessions; its existing HTTP signature remains unchanged. The shared ledger protects active work from a disconnected waiter and fences it during close. These guarantees are process-local, separate from checkpoint recovery.

SandboxSession still launches once by default. An adapter that supports continuation may call begin_activation() with a fresh supervisor directory after the previous process has confirmed cleanup. Saved harness session storage remains outside these launch directories. Finalization stays private to the lifecycle; close removes the retained session directory tree before releasing the connection.

Emit one invocation per activation, recording harness_session_id as context, and tag each trajectory turn with its activation. OpenCode’s header plugin uses that activation-scoped invocation identity to attribute captured model calls. Responses carry only new output, usage and evidence; close supplies the cumulative record. cleanup_confirmed is omitted when an adapter does not report it. A final negative termination receipt fails the activation and skips verification; close releases the borrowed connection and returns cleanup_confirmed: false. Failures to release the connection remain retryable.

NOOA worker checkpoints

The NOOA sandbox worker publishes checkpoint.json as partial evidence at model-response and completed-tool boundaries. It projects and deeply copies the evidence on the rollout event loop, then uses a single background writer for JSON serialization, file writes, fsync, and atomic replacement. Projection and copying still run synchronously and grow with transcript length, including JSON conversion within trajectory projection. The writer never reads the live trace.

Only one snapshot is being written and one latest snapshot can be pending. New boundaries replace the pending snapshot; writes remain ordered. Normal completion flushes the pending evidence before publishing result.json. Cancellation waits for writes to finish before shutdown returns. An abrupt kill may lose the pending or in-progress update, but leaves the previous published checkpoint valid. Checkpoint failures are logged and do not fail an otherwise successful run.

Checkpoints carry partial: true and an incomplete response; they do not establish completion or authorize grading. A valid final result takes precedence. Missing or malformed final evidence with unsuccessful worker termination is an infrastructure failure. Callers must enforce a process deadline, including enough shutdown grace for pending writes; the worker has no independent deadline. Artifacts use mode 0600 and require the worker UID or privileged access for collection.

Tools in Agent vs. Resources Server

Existing agents may come with predefined tools, allowing you to leverage them directly and use the Resources server to supplement with any additional external tools. When building a new environment, prefer defining tools in the Resources server rather than the Agent server. This separation of concerns allows different agents to share the same Resources server without duplicating tool logic.

Add an agent that is not listed

• To score an agent you already run, without rewriting it, drive it through a Remote Agent.
• To get Gym’s models, tools, sandboxes, and observability, write an adapter with Add or Update a Harness. It also describes the session contract that an adapter implements.

See Choose a Path to compare the options.