Evaluate a Harbor Task Suite

Harbor is a container-based harness for agentic tasks: it runs each task in a Docker sandbox, lets an agent work in it, then runs a verifier that emits named rewards (see Harbor’s Core Concepts for its task, trial, and job model). If you already have Harbor task datasets, the Harbor runner runs them and scores the verifier rewards through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

uv sync --frozen --package nemo-evaluator-sdk --extra harbor

The dataset

A Harbor dataset is a directory of task folders. For discovery, the runner needs two files per task — the rest of the task format (environment, verifier, and solution config) is Harbor’s own:

my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...

discover_harbor_tasks reads each folder into an AgentEvalTask: the [task] name becomes the task id and its human-readable intent, and instruction.md becomes inputs["instruction"] — the instruction the agent is prompted with. See Harbor’s task documentation for the full task format.

Run it

A Harbor run is a normal agent-eval run: AgentEvaluator().run(tasks=..., target=runner), exactly like the quickstart (a callable) and the deployed-agent guide (an HTTP target). Here the target is a HarborAgentTaskRunner.

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
    HarborAgentTaskRunner,
    HarborRuntimeConfig,
    discover_harbor_tasks,
)
async def main() -> None:
    # INPUT: your Harbor task suite (a directory of task folders).
    tasks = discover_harbor_tasks("path/to/my-suite")
    runner = HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
            jobs_dir=Path("./harbor-jobs"),
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
            reward_key="reward",  # selects the required primary reward
        ),
    )
    # Same AgentEvaluator as every agent-eval run; only the target changes.
    result = await AgentEvaluator().run(tasks=tasks, target=runner)
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
asyncio.run(main())

The three pieces map onto the model: discover_harbor_tasks turns the suite into tasks, HarborAgentTaskRunner runs the Harbor job and returns one trial per Harbor trial, and AgentEvaluator scores each trial’s named rewards with HarborRewardMetric.

The oracle agent is Harbor’s reference agent — it produces a passing trial on a well-formed task — so for the one-task example above the verifier reward is 1.0:

harbor_reward.reward: 1.0

Swap agent_name (or agent_import_path) for your own agent to get a real score.

Choosing an agent

HarborRuntimeConfig decides what runs inside each sandbox (see Harbor’s agent documentation):

• agent_name — a built-in Harbor agent (for example "oracle", Harbor’s reference agent, handy as a smoke test that the harness and dataset are wired up).
• agent_import_path — your own Harbor agent, e.g. "my_agent_module:MyAgent". Set agent_dir as well when it’s a loose file rather than an installed package. Overrides agent_name.
• agent_model_name — the model slug handed to the agent.

How scoring works

1. Harbor runs each task in its sandbox and writes a <task>__<hash>/result.json per trial, including the reward mapping from the task’s verifier.
2. reward_key identifies the primary reward by name. If not specified, it defaults to reward. Mapping order and alphabetical order never select the primary.
3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged; a missing or unusable primary emits 0.0 and a diagnostic instead of skipping the trial.
4. Other keys from that task’s Harbor-valid results become optional secondaries:
   • Finite numbers are emitted.
   • Missing or Boolean values are omitted with a diagnostic; usable siblings are kept.
   • A null, nonnumeric string, or object in the reward mapping fails Harbor’s TrialResult check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes NaN and infinity as null, which hits this gate.
   • A secondary reward key discovered for one task does not apply to another task.
5. result.summary aggregates each named output, result.trials holds each trial’s status and evidence, and result.persist() writes the standard run bundle.

What counts as a numeric reward

Harbor’s task format defines each reward.json value as a float or integer (reward.txt is a single number, stored under the key reward). Harbor reads and validates that file before writing result.json:

• Finite numbers are retained. Numeric strings such as "1.25" are coerced to numbers.
• A nonnumeric string, null, or an object fails validation of the whole reward mapping.
• A JSON Boolean is coerced to 1.0 or 0.0.
• NaN and infinity pass Harbor’s numeric validation, but serialize as null in result.json.

The SDK first validates the whole result.json with Harbor’s TrialResult. A null, nonnumeric string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers, including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a diagnostic while retaining usable siblings.

Emit finite JSON numbers from the verifier. Use 1 and 0 when a reward represents pass/fail; do not rely on Harbor’s coercion of strings or Booleans.

Sparse secondary and errored reward examples

Missing secondary reward

For two attempts on task A:

a1: reward=1.0, format_ok=1.0
a2: reward=0.0, format_ok omitted

Resulting aggregates and coverage

Omission means unmeasured, not failure:

• It does not create a null, NaN, or zero per-trial metric output.
• Coverage records the missing measurement.
• Derived aggregates can therefore be unestimable (mean=None).

See Reading Results for denominator, failed-trial, semantic-view, and persistence behavior.

Errored attempt with a valid primary reward

For two attempts on task A:

a1: reward=1.0, no error
a2: reward=0.8, error=RuntimeError

Resulting statuses and aggregate

The error changes a2’s status and error rollup; it does not discard or replace its finite reward.

Attempts and concurrency

• n_attempts — desired attempts per task. On resume, Harbor runs only the missing attempts.
• n_concurrent_trials — maximum trials Harbor runs concurrently.

Retries

• max_retries — maximum extra attempts per trial during the current Harbor run; defaults to 0.
• Error policy — Harbor retries only allowed errors. Its default non-retryable errors include AgentTimeoutError.
• Repeated SDK calls — max_retries never reopens cached errored trials. Cache behavior is described below.

Caching

Caching activates only when HarborRuntimeConfig.job_name is pinned. Without it, every call creates a fresh timestamped job directory.

A cache hit requires both:

• A usable matching stamp for the requested inputs.
• At least n_attempts Harbor-valid results for every requested task.

On a repeated SDK call:

• Cache hit — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results count as completed, become PARTIAL, and are not rerun.
• Incomplete matching cache — preserves valid results, including errored results, then runs only attempts with missing or invalid results.
• Stale or unusable cache — deletes the job directory and reruns every requested attempt. This includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory.
• force_rerun=True — deletes the entire pinned job directory, then reruns every requested attempt.
• Selective cached-error rerun — unsupported. The SDK cannot rerun cached attempts by error type.
• Concurrent processes — must not share a pinned job_name; neither the SDK nor Harbor locks the job directory.

Cache identity

The SDK stores the cache stamp in jobs_dir/<job_name>/.nemo-eval-harbor-cache.json:

• version — stamp schema; must match exactly.
• options — SHA-256 of result-affecting HarborRuntimeConfig fields; must match exactly.
• agent — digest of agent_dir contents, or "<none>" when unset; must match exactly.
• tasks — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C can serve A, but cached A cannot serve A and B.
• Scoring and scheduling settings — reward_key, selected metrics, quiet, and n_concurrent_trials do not invalidate Harbor execution results.
• Installed agents — when agent_dir is unset, the stamp covers the agent selection or import path, not the installed package contents. Change the import path, use agent_dir, or force a rerun after changing installed agent code.

Force a complete rerun

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    job_name="my-suite",
    agent_name="oracle",
    force_rerun=True,
)
result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite")))

• Keep the previous job — use a new job_name, or omit it to create a fresh timestamped directory.

Shortcut: run_harbor_eval

When a run is exactly “one Harbor suite, scored by its reward,” run_harbor_eval collapses the three steps above — discover, run, score — into a single call:

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
    )
)

It uses the same AgentEvaluator and HarborRewardMetric under the hood. Prefer the explicit form above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across targets.

Next steps