Evaluate a Harbor Task Suite

Harbor is a container-based harness for agentic tasks: it runs each task in a Docker sandbox, lets an agent work in it, then runs a verifier that emits a reward (see Harbor’s Core Concepts for its task, trial, and job model). If you already have Harbor task datasets, the Harbor runner runs them and scores the verifier reward through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

The dataset

A Harbor dataset is a directory of task folders. For discovery, the runner needs two files per task — the rest of the task format (environment, verifier, and solution config) is Harbor’s own:

my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...

discover_harbor_tasks reads each folder into an AgentEvalTask: the [task] name becomes the task id and its human-readable intent, and instruction.md becomes inputs["instruction"] — the instruction the agent is prompted with. See Harbor’s task documentation for the full task format.

Run it

A Harbor run is a normal agent-eval run: AgentEvaluator().run(tasks=..., target=runner), exactly like the quickstart (a callable) and the deployed-agent guide (an HTTP target). Here the target is a HarborAgentTaskRunner.

1
import asyncio
2
from pathlib import Path
3
4
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
5
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
6
    HarborAgentTaskRunner,
7
    HarborRuntimeConfig,
8
    discover_harbor_tasks,
9
)
10
11
12
async def main() -> None:
13
    # INPUT: your Harbor task suite (a directory of task folders).
14
    tasks = discover_harbor_tasks("path/to/my-suite")
15
16
    runner = HarborAgentTaskRunner(
17
        config=HarborRuntimeConfig(
18
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
19
            jobs_dir=Path("./harbor-jobs"),
20
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
21
        )
22
    )
23
24
    # Same AgentEvaluator as every agent-eval run; only the target changes.
25
    result = await AgentEvaluator().run(tasks=tasks, target=runner)
26
27
    for aggregate in result.summary.scores.scores:
28
        print(f"{aggregate.name}: {aggregate.mean}")
29
30
31
asyncio.run(main())

The three pieces map onto the model: discover_harbor_tasks turns the suite into tasks, HarborAgentTaskRunner runs the Harbor job and returns one trial per Harbor trial, and AgentEvaluator scores each trial’s reward with HarborRewardMetric.

The oracle agent is Harbor’s reference agent — it produces a passing trial on a well-formed task — so for the one-task example above the verifier reward is 1.0:

harbor_reward.reward: 1.0

Swap agent_name (or agent_import_path) for your own agent to get a real score.

Choosing an agent

HarborRuntimeConfig decides what runs inside each sandbox (see Harbor’s agent documentation):

• agent_name — a built-in Harbor agent (for example "oracle", Harbor’s reference agent, handy as a smoke test that the harness and dataset are wired up).
• agent_import_path — your own Harbor agent, e.g. "my_agent_module:MyAgent". Set agent_dir as well when it’s a loose file rather than an installed package. Overrides agent_name.
• agent_model_name — the model slug handed to the agent.

How scoring works

1. Harbor runs each task in its sandbox and writes a <task>__<hash>/result.json per trial, including the reward from the task’s verifier.
2. The runner reads that reward onto each trial’s metadata, and HarborRewardMetric scores it — the reward value, or 0.0 for a trial whose verifier emitted none.
3. result.summary aggregates the reward across tasks (harbor_reward.reward), result.trials holds each trial’s status and evidence, and — if you pass a config with an output_dir to run() — the run bundle (including report.html) is written like any other agent-eval run.

Attempts, concurrency, and caching

• n_attempts — trials Harbor runs per task; n_concurrent_trials — how many run at once.
• The job directory doubles as a cache. Pin a stable job_name and a completed job whose results already cover every requested task is re-scored instead of re-run. The default timestamped job_name always runs fresh; set force_rerun=True to delete an existing job dir first.

Shortcut: run_harbor_eval

When a run is exactly “one Harbor suite, scored by its reward,” run_harbor_eval collapses the three steps above — discover, run, score — into a single call:

1
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
2
3
result = await run_harbor_eval(
4
    HarborRuntimeConfig(jobs_dir="./harbor-jobs", agent_name="oracle"),
5
    dataset_path="path/to/my-suite",
6
)

It uses the same AgentEvaluator and HarborRewardMetric under the hood. Prefer the explicit form above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across targets.

Next steps