Terminal-Bench 2.1 with NOOA

The NOOA recipe runs NOOA inside the task sandbox and reuses Terminal-Bench 2.1’s existing preparation and resources server. The run wrapper composes three parts:

• The reusable benchmark profile defines the benchmark resources-server instance. It owns the unchanged canonical dataset and preparer, using the terminal-bench-2.1 taskset.
• The reusable sandboxed NOOA profile defines the agent instance with resources_server: null. It uses the sandbox supplied for the episode and does not define an environment server of its own.
• The generic single_agent_turn environment-server instance, experiment, binds benchmark and agent.

To use another compatible harness, replace the harness profile and its settings while keeping the agent instance name. The environment server, benchmark profile, prepared data, and taskset can stay the same.

Configure a sandbox provider and your policy endpoint in a private private-policy-provider.yaml overlay; the provider must support the task images and allow the sandbox to reach Gym’s model server. Keep credentials in your local configuration. Set use_absolute_ip: true in that overlay and configure default_host when necessary so the advertised model URL is reachable from the sandbox, rather than loopback. Runtime installation also needs outbound package access unless the agent’s runtime_wheelhouse points to a prepared offline wheelhouse on the controller. The resources server owns sandbox creation, verification, and cleanup. Keep the private overlay last in the config list so its values take precedence. Benchmark overrides belong under benchmark.resources_servers.terminal_bench_2_1; NOOA overrides belong under agent.responses_api_agents.nooa_agent.

From the repository root:

gym eval prepare --config benchmarks/terminal_bench_2_1/nooa.yaml
gym env resolve \
    --config benchmarks/terminal_bench_2_1/nooa.yaml \
    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
    --config private-policy-provider.yaml
gym eval run \
    --config benchmarks/terminal_bench_2_1/nooa.yaml \
    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
    --config private-policy-provider.yaml \
    --split benchmark --limit 1 --concurrency 1 --num-repeats 1 \
    --output results/terminal-bench-2-1-nooa.jsonl

The recipe uses one worker with a 21,600-second (six-hour) sandbox runner ceiling. The 30,000-second episode deadline covers seed, agent, verification, and close together; it is not the agent’s time budget. The sandbox TTL is 30,600 seconds, leaving room for bounded cleanup after that deadline.

The canonical 1,800-second verification timeout applies per verifier command, including setup and the test command. It is not a total verification budget. The episode deadline bounds verification overall; if it expires during verification, the result is a non-terminal Episode timed out failure with no reward. The canonical preparer omits per-task agent and verifier timeout values from task.toml; this recipe does not restore those budgets.