Evaluate a Harbor Task Suite

Harbor is a container-based harness for agentic tasks: it runs each task in a Docker sandbox, lets an agent work in it, then runs a verifier that emits named rewards (see Harbor’s Core Concepts for its task, trial, and job model). If you already have Harbor task datasets, the Harbor runner runs them and scores the verifier rewards through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Install Harbor

nhx-evals-sdk is not published as a standalone package, so how you add Harbor depends on how you installed NeMo Helix.

uv pip install "nemo-helix[all]"

uv pip install "harbor>=0.20,<0.21"

uv sync has no pip equivalent — it resolves nhx-evals-sdk as a workspace member, which is why the source-checkout path needs the repository. The runner raises an error naming both of these install paths if harbor is missing.

The dataset

A Harbor dataset is a directory of task folders. For discovery, the runner needs two files per task — the rest of the task format (environment, verifier, and solution config) is Harbor’s own:

my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...

The standalone SDK’s discover_harbor_tasks() returns a HarborTaskCollection of HarborAgentEvalTask objects. The Harbor framework task name becomes id and intent; normalized instructions become inputs["instruction"]. Each HarborAgentEvalTask has an absolute source_dir pointing at its original location. The same objects work locally and with platform task resources. See Harbor’s task documentation for the full task format.

The collection supports repeatable iteration, indexing, and slicing; use list(tasks) if you need a mutable list. Discovery validates private copies and then releases them, so the original packages must remain available for execution or publication. To scaffold a complete package:

harbor init my-org/my-task --task

This writes a populated task.toml alongside instruction.md, environment, verifier, and solution stubs, then prints the files to fill in. Add --steps N for a multi-step task, or use -d to initialize a dataset. See Harbor’s documentation for the full task format and the rest of its CLI.

A single run takes either all discovered tasks or all plain AgentEvalTask objects, not a mix. To run plain tasks, including tasks reloaded from a results bundle, convert any discovered ones with as_base_task() and tell the runner where the packages live:

from nhx_evals_sdk.agent_eval.tasks import as_base_task
tasks = [as_base_task(task) for task in tasks]
runner = HarborAgentTaskRunner(config=config, dataset_path="./mydataset")

as_base_task() is also the way to get a plain copy of a discovered task: AgentEvalTask.model_validate(dict(task)) rejects the extra source_dir field.

Upload and Register Harbor Tasks

Discover local task packages, publish each task, then create a taskset from the returned IDs:

from pathlib import Path
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import discover_harbor_tasks
from nemo_evals.sdk.resources import Evaluator
from nemo_helix_plugin.evals.client import EvaluatorClient
evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
remote_task_ids = []
for task in discover_harbor_tasks("./mydataset"):
    # Native ids look like "my-org/my-task"; a registration name cannot contain "/".
    remote_task = evaluator.tasks.replace(task.id.replace("/", "-"), task=task)
    remote_task_ids.append(remote_task.id)
evaluator.tasksets.create("suite", tasks=remote_task_ids)

create requires a new name. replace creates or updates a task or taskset; identical content reuses its revision. Registration names are bare names of at most 255 characters, using letters, digits, underscores, hyphens or dots. Choose stable, distinct names for your tasks.

Task IDs identify stored records. A taskset pins their current revisions when it is published; concurrent task updates before that call may be included. Later task updates do not change the stored taskset. Task publication and taskset publication are separate operations.

To publish a single task, discover its directory and pass the task directly:

task = discover_harbor_tasks("./my_task")[0]
remote_task = evaluator.tasks.replace(task.id.replace("/", "-"), task=task)

The resource creates or reuses the default fileset, uploads and verifies the archive, then registers the task.

To publish additional metrics or views, see Custom metrics and views.

The default location is <workspace>/harbor-tasks, with an independent content-addressed path per task. Dataset directory names are not implicit storage prefixes. Archive references use workspace/fileset#relative/path. Every upload is downloaded and checksum-verified; registration and workers independently verify it. Existing archive bytes may be replaced by ordinary Files uploads. Cleanup is caller-managed because archives may be shared.

Discovery accepts one task directory or a dataset directory and excludes task_template with a warning. Regular files and hidden real directories at the dataset root are skipped. Special entries, malformed visible directories, empty datasets, and duplicate native IDs fail discovery. Each task is revalidated when published. The publication loop is not atomic: earlier task writes survive later failures. If an entity write fails after preparation, TaskPublicationError.prepared_task retains the prepared input for retry through replace, without reuploading. The original error is available as its cause. A lost task-write response remains an error; retrying create may conflict if it committed.

If you already have a typed platform client, use the supported factory:

from nemo_helix_plugin.client.client import NemoClient
client = NemoClient(base_url="http://localhost:8080")
evaluator = Evaluator.from_client(client)

Async resources provide the same workflow with awaited create and replace calls:

import asyncio
from pathlib import Path
from nemo_evals.sdk.resources import AsyncEvaluator
from nemo_helix_plugin.client.client import AsyncNemoClient
async def publish_task_async() -> None:
    async with AsyncNemoClient(base_url="http://localhost:8080") as async_client:
        async_evaluator = AsyncEvaluator.from_client(async_client)
        task = discover_harbor_tasks("./my_task")[0]
        stored_task = await async_evaluator.tasks.replace(
            task.id.replace("/", "-"), task=task, workspace="default",
        )
        await async_evaluator.tasksets.create(
            "async-suite", tasks=[stored_task.id], workspace="default",
        )
asyncio.run(publish_task_async())

A verified multistep task may have no root instruction; the captured archive remains authoritative.

Submission snapshots and worker preparation

Both task references and taskset references expand at submission. Before a job is created, the service validates stored definitions, unique task identities, metric/model dependencies, and views. The canonical job embeds each selected definition and its resolved scoring. Republishing or deleting task/taskset records afterward does not change that job’s selected tasks.

Workers download the referenced archives through the Files service, verify hashes, safely extract and validate native Harbor packages, and construct the local dataset. Archive availability, permissions, and native-package failures remain execution-time checks. Saved-trial rescoring skips these archive operations; metric evidence must still be available.

Custom metrics and views

Stored Harbor tasks support additional SDK metrics and semantic views. The plugin always scores HarborRewardMetric first, then the metrics you supply. An omitted or empty metrics list keeps reward-only scoring. Keep the discovered HarborRewardMetric as is: publication rejects a missing, duplicate, or customized one. To rename the primary reward, set reward_key on HarborRuntimeConfig.

Supply additional metrics and views directly on the publication source. Built-in metrics need no explicit packager; custom Python metrics require metric_bundle_packager=.

from pathlib import Path
from nemo_evals.api.fields import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborAgentTaskRunner, HarborRuntimeConfig, discover_harbor_tasks
from nhx_evals_sdk.agent_eval.tasks import SemanticReducer, SemanticView, ViewSignal
from nhx_evals_sdk.metrics.exact_match import ExactMatchMetric
from nemo_helix_plugin.evals.client import EvaluatorClient
evaluator_client = EvaluatorClient(base_url="http://localhost:8080", workspace="default")
evaluator = Evaluator(client=evaluator_client)
source = discover_harbor_tasks("./my_task")[0]
source.metrics.append(ExactMatchMetric(reference="Done", candidate="{{sample.output_text}}"))
source.views["quality"] = SemanticView(reducer=SemanticReducer.MEAN, signals=[
    ViewSignal(metric="harbor_reward", output="reward"),
    ViewSignal(metric="exact-match", output="exact-match"),
])
task = evaluator.tasks.replace(source.id.replace("/", "-"), task=source)
evaluator.tasksets.replace("scored-suite", tasks=[task.id])
job = evaluator.submit(
    tasks=TasksetRef("default/scored-suite"),
    target=HarborAgentTaskRunner(
        config=HarborRuntimeConfig(agent_name="oracle"),
    ),
)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)

• Scoring resolution: Evaluator loads metrics and views from each selected task revision. The runner supplies the primary Harbor reward, even when additional metrics are empty.
• Task publication: Resource methods package, upload and verify Harbor sources before registering them.
• Views: Reference the primary output (reward, or the target’s reward_key) and declared custom metric outputs.

Rescore saved Harbor trials

Pass saved trials to evaluator.submit(tasks=..., trials=...) instead of a live target.

Submit a stored taskset reference with saved SDK trials and omit target. Submission snapshots the selected definitions and resolves their scoring. The worker applies HarborRewardMetric, additional metrics, and views without looking up task/taskset records, downloading task archives, or running the agent or verifier. Metric evidence such as traces and artifacts must still be accessible from the worker.

With no target, rescoring uses the deployment’s normal evaluator execution backend rather than Harbor’s host subprocess override. Its configured worker image and dependencies must be available.

from pathlib import Path
from nemo_evals.api.schemas import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.trials import AgentEvalTrial
from nemo_helix_plugin.evals.client import EvaluatorClient
trials = [
    AgentEvalTrial.model_validate_json(line)
    for line in Path("./trials.jsonl").read_text().splitlines()
    if line.strip()
]
evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
job = evaluator.submit(tasks=TasksetRef("default/suite"), trials=trials)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)

• Change scoring: Publish task revisions with new metrics/views → update the taskset → submit with the same trials. Unknown trial tasks and selected tasks without trials fail.
• Resolution and size: New submissions resolve task/taskset selectors and metric/model dependencies again. Canonical jobs contain an ordered list of task definition snapshots, so job size grows with the selected definitions and inline metric bundles. Each snapshot retains its original entity name and revision digest as provenance.
• Reward key: Every trial must contain the same valid metadata["harbor_primary_reward_key"] (for example, "reward"). Rescoring preserves that primary selection and discovers secondary outputs from saved rewards; renaming an output does not reinterpret the primary value.
• Older trials: Missing reward-key metadata is rejected. Re-import original results with the current SDK’s build_trials_from_job_dir(job_dir, tasks, reward_key="reward") from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime, using SDK tasks with the original native IDs and the original reward key ("reward" is illustrative). This rebuilds metadata without rerunning Harbor. Keep or publish referenced evidence where the scoring worker can read it.

Harbor Tasks and Taskset Grouping in Fileset

Task resource publication stores new archives in <workspace>/harbor-tasks for both single-task and dataset directories. Existing archives in harbor-tasksets remain valid; their stored references are unchanged.

• workspace selects where Tasks and Tasksets are registered and the default <workspace>/harbor-tasks fileset.
• fileset_ref overrides the destination workspace/fileset without changing the registration workspace.
• path_prefix defaults to empty for every independent task publication. Set an explicit prefix to group archive paths under a dataset or release name.
• The first argument to tasks.create/replace is the registered task name.
• Publish a taskset separately with tasksets.create/replace(..., tasks=remote_task_ids).

For example, a single task with path_prefix="release-1" is stored at default/harbor-tasks#release-1/my_task/<H1>/task_archive.

default/harbor-tasks#my_task/
└── <H1>/
    └── task_archive
default/harbor-tasks#mydataset/
├── my_task/
│   └── <H1>/
│       └── task_archive
└── search/
    └── <H2>/
        └── task_archive

H1, H2 are SHA-256 of completed compressed task_archive bytes.

Each task_archive contains one named directory:

my_task/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile
├── tests/
│   └── test.sh
└── ...all other task files and empty directories...

Task Archiving Details

• The uploader captures a quiescent source and packages entries in sorted order.
• Tar timestamps are normalized, and the gzip header contains no timestamp. Compression is stable within the same tooling, but a compressor upgrade may produce a new hash.
• The uploader verifies the stored bytes and native structure after downloading the archive.
• The named task root, full permission bits (0o7777), dotfiles, ignored files, and empty directories are preserved.
• Original file ownership is not preserved. Archive UID and GID values are normalized to 0; extracted files remain owned by the worker process without using chown.
• Workers retain Harbor’s Docker transfer behavior.
• Symlinks inside a task are preserved as links when their target is relative and resolves inside the task directory, for example tests/run.sh -> test.sh. Absolute, escaping, dangling, and cyclic links are rejected, as are links at task.toml, instruction.md, .gitignore, and step directories. Harbor copies links as links, so a link that crosses directories Harbor transfers separately (such as tests/data -> ../environment/data) dangles inside the container.
• Special files and tar hard-link entries are rejected. A hard-linked regular file in the source task is archived as an ordinary file with its own copy of the content.

Constraints

Server registration and workers independently verify archive bytes, bounded extraction, native metadata and packaged step references. Initial limits are 100,000 entries, 4 GiB per file, 16 GiB payload, plus bounded compressed/expanded streams and tar metadata records.

Run it

Standalone SDK

A Harbor run is a normal agent-eval run: AgentEvaluator().run(tasks=..., target=runner), exactly like the quickstart (a callable) and the deployed-agent guide (an HTTP target). Here the target is a HarborAgentTaskRunner.

import asyncio
from pathlib import Path
from nhx_evals_sdk.agent_eval.evaluator import AgentEvaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import (
    HarborAgentTaskRunner,
    HarborRuntimeConfig,
    discover_harbor_tasks,
)
async def main() -> None:
    # INPUT: your Harbor task suite (a directory of task folders).
    tasks = discover_harbor_tasks("path/to/my-suite")
    runner = HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
            jobs_dir=Path("./harbor-jobs"),
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
            reward_key="reward",  # selects the required primary reward
        ),
    )
    # Same AgentEvaluator as every agent-eval run; only the target changes.
    result = await AgentEvaluator().run(tasks=tasks, target=runner)
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
asyncio.run(main())

The three pieces map onto the model: discover_harbor_tasks turns the suite into tasks, HarborAgentTaskRunner runs the Harbor job and returns one trial per Harbor trial, and AgentEvaluator scores each trial’s named rewards with HarborRewardMetric.

The oracle agent is Harbor’s reference agent — it produces a passing trial on a well-formed task — so for the one-task example above the verifier reward is 1.0:

harbor_reward.reward: 1.0
harbor_reward.reward.pass@1: 1.0

Swap agent_name (or agent_import_path) for your own agent to get a real score.

Platform Plugin

After uploading and registering the tasks, submit their taskset reference with a Harbor target. These examples use default/suite, matching the taskset created above. Use workspace/name#<full-content-digest> to select a specific taskset revision. The oracle agent runs the task’s reference solution without model credentials.

The job host needs Python 3.12+, Harbor, and a running Docker daemon. Local deployments with the Docker or none runtime predefine a subprocess execution profile named default, unless the subprocess executor is disabled. Harbor selects that provider explicitly. The examples use this profile; CLI submissions can select another configured subprocess profile by name. Running Harbor job in Kubernetes or Docker execution profile is not supported at this time.

from nemo_evals.api.fields import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborAgentTaskRunner, HarborRuntimeConfig, discover_harbor_tasks
from nemo_helix_plugin.evals.client import EvaluatorClient
evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
job = evaluator.submit(
    tasks=TasksetRef("default/suite"),
    target=HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            agent_name="oracle", reward_key="reward",
        ),
    ),
)
print(job.name)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)

Choosing an agent

HarborRuntimeConfig decides what runs inside each sandbox (see Harbor’s agent documentation):

• agent_name — a built-in Harbor agent (for example "oracle", Harbor’s reference agent, handy as a smoke test that the harness and dataset are wired up).

• agent_import_path — your own Harbor agent, e.g. "my_agent_module:MyAgent". Set agent_dir as well for a loose local source tree. Overrides agent_name. Platform-submitted jobs do not currently accept agent_dir; see Standalone and submitted custom agents.

• agent_model_name — the model slug handed to the agent.

  On the platform’s HarborRunnerTarget these three are one field, source, so a spec names exactly one agent: {"name": "oracle", "model_name": ...} for a built-in agent or {"import_path": "my_agent_module:MyAgent", "model_name": ...} for your own.

• agent_kwargs — keyword arguments forwarded to the agent’s constructor, the equivalent of Harbor’s --ak key=value flags. Values must be JSON (strings, numbers, booleans, lists, mappings). Do not put secrets here. Harbor copies its config into the job directory’s config.json and lock.json and into every trial’s config.json, lock.json, result.json and agent run spec, and needs the real value to construct the agent, so there is no later point at which the value can be taken back. Put credentials in env_secrets instead; a ${NAME} template is fine here, since it carries no value.

  A credential-shaped value fails validation before the run starts — a credential-looking key (api_key, token, secret, authorization, …) holding plaintext, or any value carrying a recognised issued-token shape (nvapi-, sk-, hf_, a PEM block, …). Treat that as a backstop, not a guarantee: a secret under a key of your own naming, in a format the check does not know, will pass straight through. The rule is still that secrets do not belong in agent_kwargs.

• env_secrets — credentials for the agent, as {ENV_NAME: secret-ref}. The same config runs standalone and on the platform:

  • Standalone submission:
    • a workspace ref such as my-workspace/openai-api-key is read from MY_WORKSPACE_OPENAI_API_KEY, falling back to OPENAI_API_KEY
    • a bare ref openai-api-key reads OPENAI_API_KEY. When both are set, the workspace-prefixed variable wins.
  • Platform submission:
    • the platform resolves the provided secret reference (a bare ref in the job’s workspace) into the job’s environment.
  • The agent receives each entry through Harbor’s agent env (self.extra_env, and the env of its environment.exec calls). A custom agent should read it from there, not from os.getenv.
  • Harbor scrubs a secret’s value from trial files only under keys matching KEY, SECRET, TOKEN, PASSWORD, CREDENTIAL or AUTH; other keys log a warning. A very short or common secret value (for example 1) is scrubbed everywhere it appears, which can corrupt JSON.

• env_vars — non-secret settings for the agent, as literal values that reach only the agent. Rejected: ${MY_ENV_VAR} templates, credential-shaped values, and keys Harbor treats as secrets (matching KEY, SECRET, TOKEN, PASSWORD, CREDENTIAL or AUTH, case-insensitively, such as MAX_TOKENS), since Harbor would mask them in config.json and scrub their values from every trial file.

The shared agent fields map onto Harbor’s AgentConfig, which Harbor’s agent factory unpacks into the agent’s constructor. agent_dir is an SDK-only local import aid, not an AgentConfig field:

HarborRuntimeConfig  (on the platform, HarborRunnerTarget.source fills the three agent lines)
        │
        ├─ agent_import_path   → Harbor AgentConfig.import_path
        ├─ agent_model_name    → Harbor AgentConfig.model_name
        ├─ agent_kwargs        → Harbor AgentConfig.kwargs
        └─ env_secrets/env_vars → Harbor AgentConfig.env ({NAME: "${<source var>}"} + literals)
                                       │
                                       ▼
                     Harbor AgentFactory constructs
                     FabricAgent(model_name=..., extra_env=..., **kwargs)

Standalone and submitted custom agents

Loose wrapper source through agent_dir is supported only by standalone nhx-evals-sdk runs. It is not supported by submitted nemo-evals plugin jobs.

For example, NeMo Fabric’s FabricAgent is one of Harbor’s existing agents; its constructor requires fabric_adapter_id and accepts fabric_package and fabric_telemetry. For a complete, runnable version with a Nemotron model, see Run a NeMo Fabric Agent inside Harbor.

config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nemo_fabric.integrations.harbor:FabricAgent",
    agent_kwargs={
        "fabric_adapter_id": "nvidia.fabric.codex",
        "fabric_package": "nemo-fabric[codex]==0.4.0",
        "fabric_telemetry": "relay",
    },
)

The fields shared with platform submission appear on HarborRunnerTarget, the target of an AgentEvalInputSpec submitted with nemo evals agent-evaluate, including env_secrets, a mapping of environment variable name to NeMo Helix secret reference, and env_vars. The service resolves each reference into the job’s environment at compile time and the job hands Harbor a ${NAME} template, so the credential appears on neither the spec, the run bundle, nor config.json. This installed NeMo Fabric adapter example needs no loose source directory:

{
  "target": {
    "kind": "harbor",
    "source": {"import_path": "nemo_fabric.integrations.harbor:FabricAgent"},
    "agent_kwargs": {
      "fabric_adapter_id": "nvidia.fabric.codex",
      "fabric_package": "nemo-fabric[codex]==0.4.0"
    },
    "env_secrets": {
      "OPENAI_API_KEY": "my-workspace/openai-key"
    }
  }
}

Built-in Codex agent

Harbor registers codex as a built-in installed agent. It installs the requested Codex CLI version inside each task container, so neither an agent_import_path nor wrapper-source upload is required. For a direct plugin job target:

from nemo_evals.jobs.agent_spec import HarborBuiltinAgentSource, HarborRunnerTarget
from nhx_evals_sdk import SecretRef
target = HarborRunnerTarget(
    source=HarborBuiltinAgentSource(name="codex", model_name="gpt-5.6-luna"),
    agent_kwargs={"version": "0.153.0"},
    env_secrets={
        "OPENAI_API_KEY": SecretRef(root="my-workspace/openai-key"),
    },
)

Replace the model and secret reference with values available to your account. The task image needs network access and the system tools Harbor’s Codex adapter uses to install the CLI. The CLI version above is an explicit reproducibility pin, not a platform-wide default.

For a complete runnable example, see the Harbor taskset evaluation notebook.

How scoring works

1. Harbor runs each task in its sandbox and writes a <task>__<hash>/result.json per trial, including the reward mapping from the task’s verifier.
2. reward_key identifies the primary reward by name. If not specified, it defaults to reward. Mapping order and alphabetical order never select the primary. The HarborRewardMetric that discover_harbor_tasks attaches to each task is a placeholder. The runner replaces it with one named after reward_key, so changing output_name, description or labels on the task has no effect. When rescoring saved trials, the key recorded in those trials is used; if you rescore with AgentEvaluator.run(tasks, trials) and no runner, the task’s metric is used as is.
3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged; a missing or unusable primary emits 0.0 and a diagnostic instead of skipping the trial.
4. Other keys from that task’s Harbor-valid results become optional secondaries:
   • Finite numbers are emitted.
   • Missing or Boolean values are omitted with a diagnostic; usable siblings are kept.
   • A null, nonnumeric string, or object in the reward mapping fails Harbor’s TrialResult check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes NaN and infinity as null, which hits this gate.
   • A secondary reward key discovered for one task does not apply to another task.
5. result.summary aggregates each named output, result.trials holds each trial’s status and evidence, and result.persist() writes the standard run bundle.

What counts as a numeric reward

Harbor’s task format defines each reward.json value as a float or integer (reward.txt is a single number, stored under the key reward). Harbor reads and validates that file before writing result.json:

• Finite numbers are retained. Numeric strings such as "1.25" are coerced to numbers.
• A nonnumeric string, null, or an object fails validation of the whole reward mapping.
• A JSON Boolean is coerced to 1.0 or 0.0.
• NaN and infinity pass Harbor’s numeric validation, but serialize as null in result.json.

The SDK first validates the whole result.json with Harbor’s TrialResult. A null, nonnumeric string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers, including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a diagnostic while retaining usable siblings.

Emit finite JSON numbers from the verifier. Use 1 and 0 when a reward represents pass/fail; do not rely on Harbor’s coercion of strings or Booleans.

Sparse secondary and errored reward examples

Missing secondary reward

For two attempts on task A:

a1: reward=1.0, format_ok=1.0
a2: reward=0.0, format_ok omitted

Resulting aggregates and coverage

Omission means unmeasured, not failure:

• It does not create a null, NaN, or zero per-trial metric output.
• Coverage records the missing measurement.
• Derived aggregates can therefore be unestimable (mean=None).

See Reading Results for denominator, failed-trial, semantic-view, and persistence behavior.

Errored attempt with a valid primary reward

For two attempts on task A:

a1: reward=1.0, no error
a2: reward=0.8, error=RuntimeError

Resulting statuses and aggregate

The error changes a2’s status and error rollup; it does not discard or replace its finite reward.

Attempts and concurrency

• n_attempts — desired attempts per task. On resume, Harbor runs only the missing attempts.
• n_concurrent_trials — maximum trials Harbor runs concurrently.

Retries

• max_retries — maximum extra attempts per trial during the current Harbor run; defaults to 0.
• Error policy — Harbor retries only allowed errors. Its default non-retryable errors include AgentTimeoutError.
• Repeated SDK calls — max_retries never reopens cached errored trials. Cache behavior is described below.

Caching

Caching activates only when HarborRuntimeConfig.job_name is pinned. Without it, every call creates a fresh timestamped job directory.

A cache hit requires both:

• A usable matching stamp for the requested inputs.
• At least n_attempts Harbor-valid results for every requested task.

On a repeated SDK call:

• Cache hit — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results count as completed, become PARTIAL, and are not rerun.
• Incomplete matching cache — preserves valid results, including errored results, then runs only attempts with missing or invalid results.
• Stale or unusable cache — deletes the job directory and reruns every requested attempt. This includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory.
• force_rerun=True — deletes the entire pinned job directory, then reruns every requested attempt.
• Selective cached-error rerun — unsupported. The SDK cannot rerun cached attempts by error type.
• Concurrent processes — must not share a pinned job_name; neither the SDK nor Harbor locks the job directory.

Cache identity

The SDK stores the cache stamp in jobs_dir/<job_name>/.nemo-eval-harbor-cache.json:

• version — stamp schema; must match exactly.
• options — SHA-256 of result-affecting HarborRuntimeConfig fields, including agent_kwargs; must match exactly.
• agent — digest of agent_dir contents, or "<none>" when unset; must match exactly.
• tasks — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C can serve A, but cached A cannot serve A and B.
• Scoring and scheduling settings — reward_key, selected metrics, quiet, and n_concurrent_trials do not invalidate Harbor execution results.
• env_secrets and env_vars — env_secrets participates by reference and env_vars by value. Secret values are never fingerprinted, so rotating a credential keeps the cache valid. Job dirs stamped by an SDK release before these fields existed re-run once.
• Installed agents — when agent_dir is unset, the stamp covers the agent selection or import path, not the installed package contents. Change the import path, use agent_dir, or force a rerun after changing installed agent code.

Force a complete rerun

import asyncio
from pathlib import Path
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    job_name="my-suite",
    agent_name="oracle",
    force_rerun=True,
)
result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite")))

• Keep the previous job — use a new job_name, or omit it to create a fresh timestamped directory.

Shortcut: run_harbor_eval

When a run is exactly “one Harbor suite, scored by its reward,” run_harbor_eval collapses the three steps above — discover, run, score — into a single call:

import asyncio
from pathlib import Path
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval
result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
    )
)

It uses the same AgentEvaluator and HarborRewardMetric under the hood. metrics= replaces the task metrics, so keep HarborRewardMetric() in the list or the reward is not scored:

from nhx_evals_sdk.metrics.exact_match import ExactMatchMetric
from nhx_evals_sdk.metrics.runner_rewards import HarborRewardMetric
result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
        metrics=[HarborRewardMetric(), ExactMatchMetric(reference="Done", candidate="{{sample.output_text}}")],
    )
)

Prefer the explicit form above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across targets.

Next steps