Evaluate a Harbor Task Suite

Harbor is a container-based harness for agentic tasks: it runs each task in a Docker sandbox, lets an agent work in it, then runs a verifier that emits named rewards (see Harbor’s Core Concepts for its task, trial, and job model). If you already have Harbor task datasets, the Harbor runner runs them and scores the verifier rewards through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Install Harbor

nemo-evaluator-sdk is not published as a standalone package, so how you add Harbor depends on how you installed NeMo Platform.

uv pip install "nemo-platform[all]"

uv pip install "harbor>=0.20,<0.21"

uv sync has no pip equivalent — it resolves nemo-evaluator-sdk as a workspace member, which is why the source-checkout path needs the repository. The runner raises an error naming both of these install paths if harbor is missing.

The dataset

A Harbor dataset is a directory of task folders. For discovery, the runner needs two files per task — the rest of the task format (environment, verifier, and solution config) is Harbor’s own:

my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...

discover_harbor_tasks reads each folder into an AgentEvalTask: the [task] name becomes the task id and its human-readable intent, and instruction.md becomes inputs["instruction"] — the instruction the agent is prompted with. See Harbor’s task documentation for the full task format.

Those two files are the discovery contract, not a complete task — Harbor also needs an environment and a verifier to run the task and emit rewards. To author a new task, scaffold it with Harbor’s own CLI rather than writing the files by hand:

harbor init my-org/my-task --task

This writes a populated task.toml alongside instruction.md, environment, verifier, and solution stubs, then prints the files to fill in. Add --steps N for a multi-step task, or use -d to initialize a dataset. See Harbor’s documentation for the full task format and the rest of its CLI.

Run it

A Harbor run is a normal agent-eval run: AgentEvaluator().run(tasks=..., target=runner), exactly like the quickstart (a callable) and the deployed-agent guide (an HTTP target). Here the target is a HarborAgentTaskRunner.

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
    HarborAgentTaskRunner,
    HarborRuntimeConfig,
    discover_harbor_tasks,
)
async def main() -> None:
    # INPUT: your Harbor task suite (a directory of task folders).
    tasks = discover_harbor_tasks("path/to/my-suite")
    runner = HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
            jobs_dir=Path("./harbor-jobs"),
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
            reward_key="reward",  # selects the required primary reward
        ),
    )
    # Same AgentEvaluator as every agent-eval run; only the target changes.
    result = await AgentEvaluator().run(tasks=tasks, target=runner)
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
asyncio.run(main())

The three pieces map onto the model: discover_harbor_tasks turns the suite into tasks, HarborAgentTaskRunner runs the Harbor job and returns one trial per Harbor trial, and AgentEvaluator scores each trial’s named rewards with HarborRewardMetric.

The oracle agent is Harbor’s reference agent — it produces a passing trial on a well-formed task — so for the one-task example above the verifier reward is 1.0:

harbor_reward.reward: 1.0

Swap agent_name (or agent_import_path) for your own agent to get a real score.

Choosing an agent

HarborRuntimeConfig decides what runs inside each sandbox (see Harbor’s agent documentation):

• agent_name — a built-in Harbor agent (for example "oracle", Harbor’s reference agent, handy as a smoke test that the harness and dataset are wired up).

• agent_import_path — your own Harbor agent, e.g. "my_agent_module:MyAgent". Set agent_dir as well when it’s a loose file rather than an installed package. Overrides agent_name.

• agent_model_name — the model slug handed to the agent.

• agent_kwargs — keyword arguments forwarded to the agent’s constructor, the equivalent of Harbor’s --ak key=value flags. Values must be JSON (strings, numbers, booleans, lists, mappings). Do not put secrets here. Harbor copies its config into the job directory’s config.json and lock.json and into every trial’s config.json, lock.json, result.json and agent run spec, and needs the real value to construct the agent, so there is no later point at which the value can be taken back. Name credentials in agent_env_from_host instead; a ${NAME} template is fine here, since it carries no value.

  A credential-shaped value fails validation before the run starts — a credential-looking key (api_key, token, secret, authorization, …) holding plaintext, or any value carrying a recognised issued-token shape (nvapi-, sk-, hf_, a PEM block, …). Treat that as a backstop, not a guarantee: a secret under a key of your own naming, in a format the check does not know, will pass straight through. The rule is still that secrets do not belong in agent_kwargs.

• agent_env_from_host — host environment variables to forward to the agent. Each becomes a ${NAME} template on Harbor’s AgentConfig.env: Harbor resolves it from the host environment when it creates the agent and persists only the template, so the value never reaches config.json. Provenance records the names, never the values.

The agent fields map one-to-one onto Harbor’s AgentConfig, which Harbor’s agent factory unpacks into the agent’s constructor:

HarborRunnerTarget / HarborRuntimeConfig
        │
        ├─ agent_import_path   → Harbor AgentConfig.import_path
        ├─ agent_model_name    → Harbor AgentConfig.model_name
        ├─ agent_kwargs        → Harbor AgentConfig.kwargs
        └─ agent_env_from_host → Harbor AgentConfig.env   ({NAME: "${NAME}"}, resolved by Harbor)
                                       │
                                       ▼
                     Harbor AgentFactory constructs
                     FabricAgent(model_name=..., extra_env=..., **kwargs)

For example, NeMo Fabric’s FabricAgent is one of Harbor’s existing agents; its constructor requires fabric_adapter_id and accepts fabric_package and fabric_telemetry. For a complete, runnable version with a Nemotron model, see Run a NeMo Fabric Agent inside Harbor.

config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nemo_fabric.integrations.harbor:FabricAgent",
    agent_kwargs={
        "fabric_adapter_id": "nvidia.fabric.codex",
        "fabric_package": "nemo-fabric[codex]==0.3.0",
        "fabric_telemetry": "relay",
    },
)

The same fields appear on the platform job’s HarborRunnerTarget, the target of an AgentEvalInputSpec submitted with nemo evaluator agent-evaluate submit. Instead of agent_env_from_host, the platform target takes env_secrets, a mapping of environment variable name to NeMo Platform secret reference. The service resolves each reference into the job’s environment at compile time and the job hands Harbor a ${NAME} template, so the credential appears on neither the spec, the run bundle, nor config.json:

{
  "target": {
    "kind": "harbor",
    "agent_import_path": "nemo_fabric.integrations.harbor:FabricAgent",
    "agent_kwargs": {
      "fabric_adapter_id": "nvidia.fabric.codex",
      "fabric_package": "nemo-fabric[codex]==0.3.0"
    },
    "env_secrets": {
      "OPENAI_API_KEY": "my-workspace/openai-key"
    }
  }
}

How scoring works

1. Harbor runs each task in its sandbox and writes a <task>__<hash>/result.json per trial, including the reward mapping from the task’s verifier.
2. reward_key identifies the primary reward by name. If not specified, it defaults to reward. Mapping order and alphabetical order never select the primary.
3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged; a missing or unusable primary emits 0.0 and a diagnostic instead of skipping the trial.
4. Other keys from that task’s Harbor-valid results become optional secondaries:
   • Finite numbers are emitted.
   • Missing or Boolean values are omitted with a diagnostic; usable siblings are kept.
   • A null, nonnumeric string, or object in the reward mapping fails Harbor’s TrialResult check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes NaN and infinity as null, which hits this gate.
   • A secondary reward key discovered for one task does not apply to another task.
5. result.summary aggregates each named output, result.trials holds each trial’s status and evidence, and result.persist() writes the standard run bundle.

What counts as a numeric reward

Harbor’s task format defines each reward.json value as a float or integer (reward.txt is a single number, stored under the key reward). Harbor reads and validates that file before writing result.json:

• Finite numbers are retained. Numeric strings such as "1.25" are coerced to numbers.
• A nonnumeric string, null, or an object fails validation of the whole reward mapping.
• A JSON Boolean is coerced to 1.0 or 0.0.
• NaN and infinity pass Harbor’s numeric validation, but serialize as null in result.json.

The SDK first validates the whole result.json with Harbor’s TrialResult. A null, nonnumeric string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers, including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a diagnostic while retaining usable siblings.

Emit finite JSON numbers from the verifier. Use 1 and 0 when a reward represents pass/fail; do not rely on Harbor’s coercion of strings or Booleans.

Sparse secondary and errored reward examples

Missing secondary reward

For two attempts on task A:

a1: reward=1.0, format_ok=1.0
a2: reward=0.0, format_ok omitted

Resulting aggregates and coverage

Omission means unmeasured, not failure:

• It does not create a null, NaN, or zero per-trial metric output.
• Coverage records the missing measurement.
• Derived aggregates can therefore be unestimable (mean=None).

See Reading Results for denominator, failed-trial, semantic-view, and persistence behavior.

Errored attempt with a valid primary reward

For two attempts on task A:

a1: reward=1.0, no error
a2: reward=0.8, error=RuntimeError

Resulting statuses and aggregate

The error changes a2’s status and error rollup; it does not discard or replace its finite reward.

Attempts and concurrency

• n_attempts — desired attempts per task. On resume, Harbor runs only the missing attempts.
• n_concurrent_trials — maximum trials Harbor runs concurrently.

Retries

• max_retries — maximum extra attempts per trial during the current Harbor run; defaults to 0.
• Error policy — Harbor retries only allowed errors. Its default non-retryable errors include AgentTimeoutError.
• Repeated SDK calls — max_retries never reopens cached errored trials. Cache behavior is described below.

Caching

Caching activates only when HarborRuntimeConfig.job_name is pinned. Without it, every call creates a fresh timestamped job directory.

A cache hit requires both:

• A usable matching stamp for the requested inputs.
• At least n_attempts Harbor-valid results for every requested task.

On a repeated SDK call:

• Cache hit — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results count as completed, become PARTIAL, and are not rerun.
• Incomplete matching cache — preserves valid results, including errored results, then runs only attempts with missing or invalid results.
• Stale or unusable cache — deletes the job directory and reruns every requested attempt. This includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory.
• force_rerun=True — deletes the entire pinned job directory, then reruns every requested attempt.
• Selective cached-error rerun — unsupported. The SDK cannot rerun cached attempts by error type.
• Concurrent processes — must not share a pinned job_name; neither the SDK nor Harbor locks the job directory.

Cache identity

The SDK stores the cache stamp in jobs_dir/<job_name>/.nemo-eval-harbor-cache.json:

• version — stamp schema; must match exactly.
• options — SHA-256 of result-affecting HarborRuntimeConfig fields, including agent_kwargs; must match exactly.
• agent — digest of agent_dir contents, or "<none>" when unset; must match exactly.
• tasks — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C can serve A, but cached A cannot serve A and B.
• Scoring and scheduling settings — reward_key, selected metrics, quiet, and n_concurrent_trials do not invalidate Harbor execution results.
• agent_env_from_host — participates by name: adding or removing a forwarded variable invalidates, but the value it resolves to at run time is never fingerprinted, so rotating a credential keeps the cache valid.
• Installed agents — when agent_dir is unset, the stamp covers the agent selection or import path, not the installed package contents. Change the import path, use agent_dir, or force a rerun after changing installed agent code.

Force a complete rerun

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    job_name="my-suite",
    agent_name="oracle",
    force_rerun=True,
)
result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite")))

• Keep the previous job — use a new job_name, or omit it to create a fresh timestamped directory.

Shortcut: run_harbor_eval

When a run is exactly “one Harbor suite, scored by its reward,” run_harbor_eval collapses the three steps above — discover, run, score — into a single call:

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
    )
)

It uses the same AgentEvaluator and HarborRewardMetric under the hood. Prefer the explicit form above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across targets.

Next steps