> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Evaluate Terminal-Bench 2.1 on the Harbor Path

> Run the 89 Terminal-Bench 2.1 tasks as published Harbor task folders through the harbor environment server.

# Evaluate Terminal-Bench 2.1 on the Harbor Path

Terminal-Bench 2.1 is published as Harbor task folders. This page runs those folders exactly as published through
the generic [`harbor` environment server](https://github.com/NVIDIA-NeMo/Gym/blob/main/environment_servers/harbor/README.md),
with no Gym-side patching of `tests/test.sh` or `solution/solve.sh`. Results on this path are compared with Harbor's
own numbers for the dataset, not one to one with the
[`terminal_bench_2_1` Resources Server](https://github.com/NVIDIA-NeMo/Gym/blob/main/resources_servers/terminal_bench_2_1/README.md),
whose run-time fixes apply on its own path only (its benchmark folder is
[`benchmarks/terminal_bench_2_1`](https://github.com/NVIDIA-NeMo/Gym/blob/main/benchmarks/terminal_bench_2_1/README.md)).
Tasks whose published tests or solutions need those fixes score here as they do in Harbor.

Run the commands below from the Gym repository root with Gym installed and a sandbox provider configured
(the examples use `opensandbox`; see [OpenSandbox](/infrastructure/sandbox/opensandbox)).

## Get the tasks

Clone the dataset's repository and pin the commit the legacy benchmark runs from, so both paths grade the same
task files:

```bash
git clone https://github.com/harbor-framework/terminal-bench-2-1
git -C terminal-bench-2-1 checkout --detach 7131e4375048a0e408a8fb404b5f499d726b695b
```

`terminal-bench-2-1/tasks/` holds the 89 task folders (the `README.md` and `dataset.toml` beside them are not tasks
and are ignored). Every command below takes that folder as its target, and the loader generates the task mapping
and dataset rows for each run.

### Or fetch from the Harbor package store

The same dataset is published in Harbor's package store as `terminal-bench/terminal-bench-2-1`. A
`harbor:<org>/<name>` reference fetches it into Gym's shared datasets folder (`$NEMO_GYM_DATASETS_DIR`, default
`./datasets`) and takes the place of `terminal-bench-2-1/tasks` in every command below:

```bash
gym dataset validate harbor:terminal-bench/terminal-bench-2-1 --sandbox opensandbox --output tb21-oracle.jsonl
gym eval run harbor:terminal-bench/terminal-bench-2-1 --agent terminus_2_sandboxed_agent --sandbox opensandbox \
  --model-type openai_model --model <model> --model-url <url>
```

The plain reference resolves the store's `latest` tag, writes the task folders into `datasets/terminal-bench-2-1/`
and records the package and the version it resolved in the `manifest.toml` there. A later fetch keeps every folder
whose content still matches, and once `latest` has moved on it refuses the folder and names both versions, so a
stale copy is never mixed with a newer one. To pin the version, name its content hash; the folder then carries the
first twelve characters of the hash (`datasets/terminal-bench-2-1-7d7bdc1cbeda/`):

```bash
gym dataset validate harbor:terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \
  --sandbox opensandbox --output tb21-oracle.jsonl
```

Every task archive is checked against the content hash Harbor published for it before it takes its place in the
folder. The store version above holds the same 89 tasks as the pinned commit and differs from it in one file,
`sanitize-git-repo/tests/test_outputs.py`, so use the clone to compare with the `terminal_bench_2_1` Resources
Server and the store to compare with Harbor's own published numbers.

## Check the reference solutions

```bash
gym dataset validate terminal-bench-2-1/tasks --sandbox opensandbox --output tb21-oracle.jsonl
```

Each task's `solution/solve.sh` runs through its own verifier with an oracle harness. Every solution must score 1,
and the command exits non-zero otherwise. `--output` (`-o`) names the rollouts file, so the oracle run can be set
against a later model run. `--limit N` validates the first N tasks and `--concurrency N` bounds the parallel
sandboxes. On the package as published, the two tasks listed under [Known upstream noise](#known-upstream-noise)
fail their own reference solutions, so this full-set command exits non-zero and names them in its report. That
is the expected result, not a broken setup; every other task must score 1.

## Run a model

```bash
gym eval run terminal-bench-2-1/tasks --agent terminus_2_sandboxed_agent --sandbox opensandbox \
  --model-type openai_model --model <model> --model-url <url>
```

Terminus 2 drives the task sandbox the environment server started. The run's rows, generated configuration and
rollouts are the artifacts every other `gym eval run` produces, so `gym eval aggregate` and the health check apply
unchanged. Resource settings (CPUs, memory, storage) come from each task's `task.toml`. To match the legacy
server's sandbox sizing for a like-for-like comparison, add a `--config` overlay that sets
`sandbox_resources_override` on the generated `harbor_<taskset>_environment` block (an option of the `harbor`
environment server).

## Known upstream noise

Two tasks are broken as published and fail on Harbor and on Gym alike until their images are fixed upstream:

* `qemu-startup`
* `qemu-alpine-ssh`

Both images are Debian Bullseye, and their tests install packages from the Bullseye security repository, which the
live Debian mirror no longer serves. Gym runs the tasks as published and does not repair them, so count these two
as expected noise when comparing with Harbor's numbers. The `terminal_bench_2_1` Resources Server applies a
security-snapshot repair on its own path only; that repair is not part of this one.