Evaluate Terminal-Bench 2.1 on the Harbor Path

Terminal-Bench 2.1 is published as Harbor task folders. This page runs those folders exactly as published through the generic harbor environment server, with no Gym-side patching of tests/test.sh or solution/solve.sh. Results on this path are compared with Harbor’s own numbers for the dataset, not one to one with the terminal_bench_2_1 Resources Server, whose run-time fixes apply on its own path only (its benchmark folder is benchmarks/terminal_bench_2_1). Tasks whose published tests or solutions need those fixes score here as they do in Harbor.

Run the commands below from the Gym repository root with Gym installed and a sandbox provider configured (the examples use opensandbox; see OpenSandbox).

Get the tasks

Clone the dataset’s repository and pin the commit the legacy benchmark runs from, so both paths grade the same task files:

git clone https://github.com/harbor-framework/terminal-bench-2-1
git -C terminal-bench-2-1 checkout --detach 7131e4375048a0e408a8fb404b5f499d726b695b

terminal-bench-2-1/tasks/ holds the 89 task folders (the README.md and dataset.toml beside them are not tasks and are ignored). Every command below takes that folder as its target, and the loader generates the task mapping and dataset rows for each run.

Or fetch from the Harbor package store

The same dataset is published in Harbor’s package store as terminal-bench/terminal-bench-2-1. A harbor:<org>/<name> reference fetches it into Gym’s shared datasets folder ($NEMO_GYM_DATASETS_DIR, default ./datasets) and takes the place of terminal-bench-2-1/tasks in every command below:

gym dataset validate harbor:terminal-bench/terminal-bench-2-1 --sandbox opensandbox --output tb21-oracle.jsonl
gym eval run harbor:terminal-bench/terminal-bench-2-1 --agent terminus_2_sandboxed_agent --sandbox opensandbox \
  --model-type openai_model --model <model> --model-url <url>

The plain reference resolves the store’s latest tag, writes the task folders into datasets/terminal-bench-2-1/ and records the package and the version it resolved in the manifest.toml there. A later fetch keeps every folder whose content still matches, and once latest has moved on it refuses the folder and names both versions, so a stale copy is never mixed with a newer one. To pin the version, name its content hash; the folder then carries the first twelve characters of the hash (datasets/terminal-bench-2-1-7d7bdc1cbeda/):

gym dataset validate harbor:terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \
  --sandbox opensandbox --output tb21-oracle.jsonl

Every task archive is checked against the content hash Harbor published for it before it takes its place in the folder. The store version above holds the same 89 tasks as the pinned commit and differs from it in one file, sanitize-git-repo/tests/test_outputs.py, so use the clone to compare with the terminal_bench_2_1 Resources Server and the store to compare with Harbor’s own published numbers.

Check the reference solutions

gym dataset validate terminal-bench-2-1/tasks --sandbox opensandbox --output tb21-oracle.jsonl

Each task’s solution/solve.sh runs through its own verifier with an oracle harness. Every solution must score 1, and the command exits non-zero otherwise. --output (-o) names the rollouts file, so the oracle run can be set against a later model run. --limit N validates the first N tasks and --concurrency N bounds the parallel sandboxes. On the package as published, the two tasks listed under Known upstream noise fail their own reference solutions, so this full-set command exits non-zero and names them in its report. That is the expected result, not a broken setup; every other task must score 1.

Run a model

gym eval run terminal-bench-2-1/tasks --agent terminus_2_sandboxed_agent --sandbox opensandbox \
  --model-type openai_model --model <model> --model-url <url>

Terminus 2 drives the task sandbox the environment server started. The run’s rows, generated configuration and rollouts are the artifacts every other gym eval run produces, so gym eval aggregate and the health check apply unchanged. Resource settings (CPUs, memory, storage) come from each task’s task.toml. To match the legacy server’s sandbox sizing for a like-for-like comparison, add a --config overlay that sets sandbox_resources_override on the generated harbor_<taskset>_environment block (an option of the harbor environment server).

Known upstream noise

Two tasks are broken as published and fail on Harbor and on Gym alike until their images are fixed upstream:

• qemu-startup
• qemu-alpine-ssh

Both images are Debian Bullseye, and their tests install packages from the Bullseye security repository, which the live Debian mirror no longer serves. Gym runs the tasks as published and does not repair them, so count these two as expected noise when comparing with Harbor’s numbers. The terminal_bench_2_1 Resources Server applies a security-snapshot repair on its own path only; that repair is not part of this one.