> This page is for version 0.8.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt
> - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt
> - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Evaluate a Harbor Task Suite

> Run an existing Harbor task dataset through NeMo Helix Evals's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits named rewards, which the SDK scores and reports like any other agent-eval run.

[Harbor](https://www.harborframework.com) is a container-based harness for agentic tasks: it runs each
task in a Docker sandbox, lets an agent work in it, then runs a **verifier** that emits named
**rewards**
(see Harbor's [Core Concepts](https://www.harborframework.com/docs/core-concepts) for its task, trial,
and job model). If you already have Harbor task datasets, the **Harbor runner** runs them and scores
the verifier rewards through agent-eval — the same
[`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the same result and bundle as the
[quickstart](/documentation/evaluate-models/agent-eval/quickstart). Only the runner changes.

> **Warning**
>
> Unlike the quickstart, this runner is **not** zero-dependency — it shells out to Harbor and Docker:
>
> * **Python `>=3.12,<3.14`** — the SDK's own floor is `>=3.12`, matching Harbor's. Harbor is still
>   imported lazily, so importing the SDK and building a Harbor config work without the extra
>   installed; execution and existing-result adaptation need it. The upper bound is
>   `nemo-helix`'s own; a source checkout resolves up to 3.14.
> * **A Docker-compatible container runtime** — Harbor invokes the `docker` executable on `PATH`
>   (`docker info` must succeed) and uses `docker compose`.
> * **Harbor** `>=0.20,<0.21` — see [Install Harbor](#install-harbor).

## Install Harbor

`nhx-evals-sdk` is not published as a standalone package, so how you add Harbor depends on how
you installed NeMo Helix.

#### Installed package

Harbor ships with the `nemo-helix` `all` extra — `services` and `plugins` include it as well:

```bash
uv pip install "nemo-helix[all]"
```

To add Harbor to an environment you already have, install it against the same constraint the SDK
declares:

```bash
uv pip install "harbor>=0.20,<0.21"
```

Use `pip install` in place of `uv pip install` if you prefer; both work here.

#### Source checkout

`uv sync --package` resolves `nhx-evals-sdk` as a member of the repository's uv workspace, so
it requires the workspace root `pyproject.toml` and does not work outside a checkout. Run it from
the repository root (see the
[setup guide](https://github.com/NVIDIA-NeMo/nemo-helix/blob/main/SETUP.md) for toolchain
prerequisites):

```bash
uv sync --frozen --package nhx-evals-sdk --extra harbor
```

`uv sync` has no `pip` equivalent — it resolves `nhx-evals-sdk` as a workspace member, which is
why the source-checkout path needs the repository. The runner raises an error naming both of these
install paths if `harbor` is missing.

## The dataset

A Harbor dataset is a directory of **task folders**. For discovery, the runner needs two files per
task — the rest of the task format (environment, verifier, and solution config) is Harbor's own:

```
my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...
```

The standalone SDK's `discover_harbor_tasks()` returns a `HarborTaskCollection` of
`HarborAgentEvalTask` objects. The Harbor framework task name becomes `id` and `intent`; normalized
instructions become `inputs["instruction"]`. Each `HarborAgentEvalTask` has an absolute `source_dir` pointing
at its original location. The same objects work locally and with platform task resources. See
[Harbor's task documentation](https://www.harborframework.com/docs/tasks) for the full task format.

The collection supports repeatable iteration, indexing, and slicing; use `list(tasks)` if you
need a mutable list. Discovery validates private copies and then releases them, so the original
packages must remain available for execution or publication. To scaffold a complete package:

```bash
harbor init my-org/my-task --task
```

This writes a populated `task.toml` alongside `instruction.md`, environment, verifier, and solution
stubs, then prints the files to fill in. Add `--steps N` for a multi-step task, or use `-d` to
initialize a dataset. See [Harbor's documentation](https://www.harborframework.com/docs/tasks) for
the full task format and the rest of its CLI.

A single run takes either all discovered tasks or all plain `AgentEvalTask` objects, not a mix.
To run plain tasks, including tasks reloaded from a results bundle, convert any discovered ones with
`as_base_task()` and tell the runner where the packages live:

```python
from nhx_evals_sdk.agent_eval.tasks import as_base_task

tasks = [as_base_task(task) for task in tasks]
runner = HarborAgentTaskRunner(config=config, dataset_path="./mydataset")
```

`as_base_task()` is also the way to get a plain copy of a discovered task:
`AgentEvalTask.model_validate(dict(task))` rejects the extra `source_dir` field.

> **Note**
>
> The repo ships a one-task
> [example dataset](https://github.com/NVIDIA-NeMo/nemo-helix/tree/main/packages/nhx_evals_sdk/examples/harbor/hello_world_dataset)
> you can clone and point at (it is not shipped in the installed wheel). Or point the dataset path at
> your own Harbor suite.

### Upload and Register Harbor Tasks

Discover local task packages, publish each task, then create a taskset from the returned IDs:

```python
from pathlib import Path

from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import discover_harbor_tasks
from nemo_evals.sdk.resources import Evaluator
from nemo_helix_plugin.evals.client import EvaluatorClient

evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
remote_task_ids = []
for task in discover_harbor_tasks("./mydataset"):
    # Native ids look like "my-org/my-task"; a registration name cannot contain "/".
    remote_task = evaluator.tasks.replace(task.id.replace("/", "-"), task=task)
    remote_task_ids.append(remote_task.id)
evaluator.tasksets.create("suite", tasks=remote_task_ids)
```

`create` requires a new name. `replace` creates or updates a task or taskset; identical content
reuses its revision. Registration names are bare names of at most 255 characters, using letters,
digits, underscores, hyphens or dots. Choose stable, distinct names for your tasks.

Task IDs identify stored records. A taskset pins their current revisions when it is published;
concurrent task updates before that call may be included. Later task updates do not change the
stored taskset. Task publication and taskset publication are separate operations.

To publish a single task, discover its directory and pass the task directly:

```python
task = discover_harbor_tasks("./my_task")[0]
remote_task = evaluator.tasks.replace(task.id.replace("/", "-"), task=task)
```

The resource creates or reuses the default fileset, uploads and verifies the archive, then
registers the task.

To publish additional metrics or views, see [Custom metrics and views](#custom-metrics-and-views).

The default location is `<workspace>/harbor-tasks`, with an independent content-addressed
path per task. Dataset directory names are not implicit storage prefixes. Archive references
use `workspace/fileset#relative/path`. Every upload is downloaded and checksum-verified;
registration and workers independently verify it. Existing archive bytes may be replaced by
ordinary Files uploads. Cleanup is caller-managed because archives may be shared.

Discovery accepts one task directory or a dataset directory and excludes `task_template` with
a warning. Regular files and hidden real directories at the dataset root are skipped. Special
entries, malformed visible directories, empty datasets, and duplicate native IDs fail discovery.
Each task is revalidated when published.
The publication loop is not atomic: earlier task writes survive later failures. If an entity
write fails after preparation, `TaskPublicationError.prepared_task` retains the prepared input
for retry through `replace`, without reuploading. The original error is available as its cause.
A lost task-write response remains an error; retrying `create` may conflict if it committed.

If you already have a typed platform client, use the supported factory:

```python
from nemo_helix_plugin.client.client import NemoClient

client = NemoClient(base_url="http://localhost:8080")
evaluator = Evaluator.from_client(client)
```

Async resources provide the same workflow with awaited `create` and `replace` calls:

```python
import asyncio

from pathlib import Path

from nemo_evals.sdk.resources import AsyncEvaluator
from nemo_helix_plugin.client.client import AsyncNemoClient

async def publish_task_async() -> None:
    async with AsyncNemoClient(base_url="http://localhost:8080") as async_client:
        async_evaluator = AsyncEvaluator.from_client(async_client)
        task = discover_harbor_tasks("./my_task")[0]
        stored_task = await async_evaluator.tasks.replace(
            task.id.replace("/", "-"), task=task, workspace="default",
        )
        await async_evaluator.tasksets.create(
            "async-suite", tasks=[stored_task.id], workspace="default",
        )

asyncio.run(publish_task_async())
```

A verified multistep task may have no root instruction; the captured archive remains authoritative.

#### Submission snapshots and worker preparation

Both task references and taskset references expand at submission. Before a job is created, the
service validates stored definitions, unique task identities, metric/model dependencies, and views.
The canonical job embeds each selected definition and its resolved scoring. Republishing or deleting
task/taskset records afterward does not change that job's selected tasks.

Workers download the referenced archives through the Files service, verify hashes, safely extract and
validate native Harbor packages, and construct the local dataset. Archive availability, permissions,
and native-package failures remain execution-time checks. Saved-trial rescoring skips these archive
operations; metric evidence must still be available.

#### Custom metrics and views

Stored Harbor tasks support additional SDK metrics and semantic views. The plugin always scores
`HarborRewardMetric` first, then the metrics you supply. An omitted or empty `metrics` list keeps
reward-only scoring. Keep the discovered `HarborRewardMetric` as is: publication rejects a missing,
duplicate, or customized one. To rename the primary reward, set `reward_key` on
`HarborRuntimeConfig`.

Supply additional metrics and views directly on the publication source. Built-in metrics need
no explicit packager; custom Python metrics require `metric_bundle_packager=`.

```python
from pathlib import Path

from nemo_evals.api.fields import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborAgentTaskRunner, HarborRuntimeConfig, discover_harbor_tasks
from nhx_evals_sdk.agent_eval.tasks import SemanticReducer, SemanticView, ViewSignal
from nhx_evals_sdk.metrics.exact_match import ExactMatchMetric
from nemo_helix_plugin.evals.client import EvaluatorClient

evaluator_client = EvaluatorClient(base_url="http://localhost:8080", workspace="default")
evaluator = Evaluator(client=evaluator_client)
source = discover_harbor_tasks("./my_task")[0]
source.metrics.append(ExactMatchMetric(reference="Done", candidate="{{sample.output_text}}"))
source.views["quality"] = SemanticView(reducer=SemanticReducer.MEAN, signals=[
    ViewSignal(metric="harbor_reward", output="reward"),
    ViewSignal(metric="exact-match", output="exact-match"),
])
task = evaluator.tasks.replace(source.id.replace("/", "-"), task=source)
evaluator.tasksets.replace("scored-suite", tasks=[task.id])
job = evaluator.submit(
    tasks=TasksetRef("default/scored-suite"),
    target=HarborAgentTaskRunner(
        config=HarborRuntimeConfig(agent_name="oracle"),
    ),
)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)
```

* **Scoring resolution:** Evaluator loads metrics and views from each selected task revision.
  The runner supplies the primary Harbor reward, even when additional metrics are empty.
* **Task publication:** Resource methods package, upload and verify Harbor sources before registering them.
* **Views:** Reference the primary output (`reward`, or the target's `reward_key`) and declared
  custom metric outputs.

#### Rescore saved Harbor trials

Pass saved trials to `evaluator.submit(tasks=..., trials=...)` instead of a live `target`.

Submit a stored taskset reference with saved SDK trials and omit `target`. Submission
snapshots the selected definitions and resolves their scoring. The worker applies `HarborRewardMetric`, additional metrics, and views without
looking up task/taskset records, downloading task archives, or running the agent or verifier. Metric evidence such as traces and
artifacts must still be accessible from the worker.

With no target, rescoring uses the deployment's normal evaluator execution backend rather than
Harbor's host subprocess override. Its configured worker image and dependencies must be available.

```python
from pathlib import Path

from nemo_evals.api.schemas import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.trials import AgentEvalTrial
from nemo_helix_plugin.evals.client import EvaluatorClient

trials = [
    AgentEvalTrial.model_validate_json(line)
    for line in Path("./trials.jsonl").read_text().splitlines()
    if line.strip()
]
evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
job = evaluator.submit(tasks=TasksetRef("default/suite"), trials=trials)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)
```

* **Change scoring:** Publish task revisions with new metrics/views → update the taskset → submit
  with the same trials. Unknown trial tasks and selected tasks without trials fail.
* **Resolution and size:** New submissions resolve task/taskset selectors and metric/model dependencies again.
  Canonical jobs contain an ordered list of task definition snapshots, so job size grows with the selected definitions
  and inline metric bundles. Each snapshot retains its original entity name and revision digest as provenance.
* **Reward key:** Every trial must contain the same valid `metadata["harbor_primary_reward_key"]`
  (for example, `"reward"`). Rescoring
  preserves that primary selection and discovers secondary outputs from saved rewards; renaming
  an output does not reinterpret the primary value.
* **Older trials:** Missing reward-key metadata is rejected. Re-import original results with the
  current SDK's `build_trials_from_job_dir(job_dir, tasks, reward_key="reward")` from
  `nhx_evals_sdk.agent_eval.runtimes.harbor.runtime`, using SDK tasks with the original native
  IDs and the original reward key (`"reward"` is illustrative). This rebuilds metadata without
  rerunning Harbor. Keep or publish referenced evidence where the scoring worker can read it.

#### Harbor Tasks and Taskset Grouping in Fileset

Task resource publication stores new archives in `<workspace>/harbor-tasks` for both
single-task and dataset directories. Existing archives in `harbor-tasksets` remain valid;
their stored references are unchanged.

* `workspace` selects where Tasks and Tasksets are registered and the default
  `<workspace>/harbor-tasks` fileset.
* `fileset_ref` overrides the destination workspace/fileset without changing the registration workspace.
* `path_prefix` defaults to empty for every independent task publication.
  Set an explicit prefix to group archive paths under a dataset or release name.
* The first argument to `tasks.create`/`replace` is the registered task name.
* Publish a taskset separately with `tasksets.create`/`replace(..., tasks=remote_task_ids)`.

For example, a single task with `path_prefix="release-1"` is stored at
`default/harbor-tasks#release-1/my_task/<H1>/task_archive`.

```text
default/harbor-tasks#my_task/
└── <H1>/
    └── task_archive
default/harbor-tasks#mydataset/
├── my_task/
│   └── <H1>/
│       └── task_archive
└── search/
    └── <H2>/
        └── task_archive
```

`H1`, `H2` are SHA-256 of completed compressed task\_archive bytes.

Each task\_archive contains one named directory:

```txt
my_task/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile
├── tests/
│   └── test.sh
└── ...all other task files and empty directories...
```

#### Task Archiving Details

* The uploader captures a quiescent source and packages entries in sorted order.
* Tar timestamps are normalized, and the gzip header contains no timestamp. Compression is stable
  within the same tooling, but a compressor upgrade may produce a new hash.
* The uploader verifies the stored bytes and native structure after downloading the archive.
* The named task root, full permission bits (`0o7777`), dotfiles, ignored files, and empty directories
  are preserved.
* Original file ownership is not preserved. Archive UID and GID values are normalized to `0`;
  extracted files remain owned by the worker process without using `chown`.
* Workers retain Harbor's Docker transfer behavior.
* Symlinks inside a task are preserved as links when their target is relative and resolves inside the
  task directory, for example `tests/run.sh -> test.sh`. Absolute, escaping, dangling, and cyclic
  links are rejected, as are links at `task.toml`, `instruction.md`, `.gitignore`, and step
  directories. Harbor copies links as links, so a link that crosses directories Harbor transfers
  separately (such as `tests/data -> ../environment/data`) dangles inside the container.
* Special files and tar hard-link entries are rejected. A hard-linked regular file in the source task is
  archived as an ordinary file with its own copy of the content.

#### Constraints

Server registration and workers independently verify archive bytes, bounded extraction, native metadata
and packaged step references. Initial limits are 100,000 entries, 4 GiB per file, 16 GiB payload, plus bounded
compressed/expanded streams and tar metadata records.

## Run it

### Standalone SDK

A Harbor run is a normal agent-eval run: `AgentEvaluator().run(tasks=..., target=runner)`, exactly like
the [quickstart](/documentation/evaluate-models/agent-eval/quickstart) (a callable) and the
[deployed-agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) guide (an HTTP
target). Here the target is a `HarborAgentTaskRunner`.

```python
import asyncio
from pathlib import Path

from nhx_evals_sdk.agent_eval.evaluator import AgentEvaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import (
    HarborAgentTaskRunner,
    HarborRuntimeConfig,
    discover_harbor_tasks,
)

async def main() -> None:
    # INPUT: your Harbor task suite (a directory of task folders).
    tasks = discover_harbor_tasks("path/to/my-suite")

    runner = HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
            jobs_dir=Path("./harbor-jobs"),
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
            reward_key="reward",  # selects the required primary reward
        ),
    )

    # Same AgentEvaluator as every agent-eval run; only the target changes.
    result = await AgentEvaluator().run(tasks=tasks, target=runner)

    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")

asyncio.run(main())
```

The three pieces map onto the model: `discover_harbor_tasks` turns the suite into tasks,
`HarborAgentTaskRunner` runs the Harbor job and returns one trial per Harbor trial, and `AgentEvaluator`
scores each trial's named rewards with `HarborRewardMetric`.

> **Note**
>
> **`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir`
> is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir/<job_name>/`,
> and that directory doubles as a re-run cache (see [below](#caching)).

The `oracle` agent is Harbor's reference agent — it produces a passing trial on a well-formed task —
so for the one-task example above the verifier reward is `1.0`:

```
harbor_reward.reward: 1.0
harbor_reward.reward.pass@1: 1.0
```

Swap `agent_name` (or `agent_import_path`) for your own agent to get a real score.

> **Note**
>
> Evals plugin submissions (`nemo evals agent-evaluate`) support the Harbor runner only
> through the host `subprocess` executor. Standalone SDK runs using `run_harbor_eval()` or
> `HarborAgentTaskRunner` invoke Harbor directly and do not use a Jobs execution profile.

### Platform Plugin

After [uploading and registering the tasks](#upload-and-register-harbor-tasks), submit their
taskset reference with a Harbor target. These examples use `default/suite`, matching the taskset
created above. Use `workspace/name#<full-content-digest>` to select a specific taskset revision. The `oracle` agent runs
the task's reference solution without model credentials.

The job host needs Python 3.12+, Harbor, and a running Docker daemon. Local deployments with the
Docker or `none` runtime predefine a `subprocess` execution profile named `default`, unless the
subprocess executor is disabled. Harbor selects that provider explicitly. The examples use this
profile; CLI submissions can select another configured subprocess profile by name.
Running Harbor job in Kubernetes or Docker execution profile is not supported at this time.

#### SDK

Use `evaluator.submit` with `HarborAgentTaskRunner`. It uses the deployment's **default**
execution profile, which must use the `subprocess` job executor.

```python
from nemo_evals.api.fields import TasksetRef
from nemo_evals.sdk.resources import Evaluator
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborAgentTaskRunner, HarborRuntimeConfig, discover_harbor_tasks
from nemo_helix_plugin.evals.client import EvaluatorClient

evaluator = Evaluator(client=EvaluatorClient(base_url="http://localhost:8080", workspace="default"))
job = evaluator.submit(
    tasks=TasksetRef("default/suite"),
    target=HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            agent_name="oracle", reward_key="reward",
        ),
    ),
)
print(job.name)
job.wait_until_done()
results = evaluator.agent_eval_results.list(job_id=job.name)
```

Submission rejects local dataset/task selection overrides, offline runners, and non-default
settings that the job target cannot preserve, including named local runs, `force_rerun`,
`quiet=False`, `agent_dir`, and timeout multipliers. For an agent that needs credentials, set
`env_secrets` on `HarborRuntimeConfig`; the platform resolves each reference at submission.
Non-secret settings go in `env_vars`.

The example prints the generated job name, waits for completion, and retrieves the stored results.

#### CLI

Save the job input as `harbor-eval.json`:

```bash
cat > harbor-eval.json <<'JSON'
{
  "tasks": "default/suite",
  "target": {
    "kind": "harbor",
    "source": {"name": "oracle"},
    "reward_key": "reward"
  }
}
JSON
```

Submit it from your NeMo Helix source checkout. Save the response so polling can reuse the
generated job name (`jq` is required):

```bash
uv run nemo evals agent-evaluate \
  --base-url http://localhost:8080 \
  --workspace default \
  --profile default \
  --spec-file harbor-eval.json > harbor-job.json
jq '{name, status}' harbor-job.json
```

Example output (your job name will differ):

```json
{
  "name": "nemo-evals.agent-27iuxxxc",
  "status": "created"
}
```

Poll every two seconds until the job completes or reaches an unsuccessful terminal status:

```bash
JOB_NAME=$(jq -er '.name' harbor-job.json)
while true; do
  STATUS=$(uv run nemo jobs get-status "$JOB_NAME" --workspace default -f json | jq -er '.status') || break
  printf '%s\n' "$STATUS"
  case "$STATUS" in
    completed|error|cancelled) break ;;
  esac
  sleep 2
done
```

Example output (intermediate states and repetitions depend on when polling starts):

```text
pending
active
completed
```

## Choosing an agent

`HarborRuntimeConfig` decides what runs inside each sandbox (see
[Harbor's agent documentation](https://www.harborframework.com/docs/agents)):

* **`agent_name`** — a built-in Harbor agent (for example `"oracle"`, Harbor's reference agent, handy
  as a smoke test that the harness and dataset are wired up).
* **`agent_import_path`** — your own Harbor agent, e.g. `"my_agent_module:MyAgent"`. Set `agent_dir`
  as well for a loose local source tree. Overrides `agent_name`. Platform-submitted jobs do not
  currently accept `agent_dir`; see [Standalone and submitted custom agents](#standalone-and-submitted-custom-agents).
* **`agent_model_name`** — the model slug handed to the agent.

  On the platform's `HarborRunnerTarget` these three are one field, `source`, so a spec names exactly
  one agent: `{"name": "oracle", "model_name": ...}` for a built-in agent or
  `{"import_path": "my_agent_module:MyAgent", "model_name": ...}` for your own.
* **`agent_kwargs`** — keyword arguments forwarded to the agent's constructor, the equivalent of
  Harbor's `--ak key=value` flags. Values must be JSON (strings, numbers, booleans, lists, mappings).
  Do not put secrets here. Harbor copies its config into the job directory's `config.json` and
  `lock.json` and into every trial's `config.json`, `lock.json`, `result.json` and agent run spec, and
  needs the real value to construct the agent, so there is no later point at which the value can be
  taken back. Put credentials in `env_secrets` instead;
  a `${NAME}` template is fine here, since it carries no value.

  A credential-shaped value fails validation before the run starts — a credential-looking key
  (`api_key`, `token`, `secret`, `authorization`, ...) holding plaintext, or any value carrying a
  recognised issued-token shape (`nvapi-`, `sk-`, `hf_`, a PEM block, ...). Treat that as a backstop,
  not a guarantee: a secret under a key of your own naming, in a format the check does not know, will
  pass straight through. The rule is still that secrets do not belong in `agent_kwargs`.
* **`env_secrets`** — credentials for the agent, as `{ENV_NAME: secret-ref}`. The same config runs
  standalone and on the platform:
  * Standalone submission:
    * a workspace ref such as `my-workspace/openai-api-key` is read from
      `MY_WORKSPACE_OPENAI_API_KEY`, falling back to `OPENAI_API_KEY`
    * a bare ref `openai-api-key` reads `OPENAI_API_KEY`. When both are set, the workspace-prefixed variable wins.
  * Platform submission:
    * the platform resolves the provided secret reference (a bare ref in the job's workspace) into the
      job's environment.
  * The agent receives each entry through Harbor's agent env (`self.extra_env`, and the env of its
    `environment.exec` calls). A custom agent should read it from there, not from `os.getenv`.
  * Harbor scrubs a secret's value from trial files only under keys matching `KEY`, `SECRET`,
    `TOKEN`, `PASSWORD`, `CREDENTIAL` or `AUTH`; other keys log a warning. A very short or common
    secret value (for example `1`) is scrubbed everywhere it appears, which can corrupt JSON.
* **`env_vars`** — non-secret settings for the agent, as literal values that reach only the agent.
  Rejected: `${MY_ENV_VAR}` templates, credential-shaped values, and keys Harbor treats as secrets
  (matching `KEY`, `SECRET`, `TOKEN`, `PASSWORD`, `CREDENTIAL` or `AUTH`, case-insensitively, such
  as `MAX_TOKENS`), since Harbor would mask them in `config.json` and scrub their values from every
  trial file.

> **Warning**
>
> Harbor's Docker task environments inherit the host environment. A task can read any variable you
> export, including `MY_WORKSPACE_*` secrets meant for the agent. Run untrusted task suites only with
> the variables they need exported.

The shared agent fields map onto Harbor's `AgentConfig`, which Harbor's agent factory unpacks into
the agent's constructor. `agent_dir` is an SDK-only local import aid, not an `AgentConfig` field:

```text
HarborRuntimeConfig  (on the platform, HarborRunnerTarget.source fills the three agent lines)
        │
        ├─ agent_import_path   → Harbor AgentConfig.import_path
        ├─ agent_model_name    → Harbor AgentConfig.model_name
        ├─ agent_kwargs        → Harbor AgentConfig.kwargs
        └─ env_secrets/env_vars → Harbor AgentConfig.env ({NAME: "${<source var>}"} + literals)
                                       │
                                       ▼
                     Harbor AgentFactory constructs
                     FabricAgent(model_name=..., extra_env=..., **kwargs)
```

### Standalone and submitted custom agents

Loose wrapper source through `agent_dir` is supported only by standalone `nhx-evals-sdk` runs.
It is not supported by submitted `nemo-evals` plugin jobs.

For example, NeMo Fabric's
[`FabricAgent`](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.4.0/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/fabric_agent.py)
is one of Harbor's [existing agents](https://www.harborframework.com/docs/agents#existing-agents);
its constructor requires `fabric_adapter_id` and accepts `fabric_package` and `fabric_telemetry`. For a
complete, runnable version with a Nemotron model, see
[Run a NeMo Fabric Agent inside Harbor](/documentation/evaluate-models/agent-eval/harbor-fabric-agent).

```python
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nemo_fabric.integrations.harbor:FabricAgent",
    agent_kwargs={
        "fabric_adapter_id": "nvidia.fabric.codex",
        "fabric_package": "nemo-fabric[codex]==0.4.0",
        "fabric_telemetry": "relay",
    },
)
```

The fields shared with platform submission appear on `HarborRunnerTarget`, the `target` of an
`AgentEvalInputSpec` submitted with `nemo evals agent-evaluate`, including `env_secrets`, a
mapping of environment variable name to NeMo Helix secret reference, and `env_vars`. The service
resolves each reference into the job's environment at
compile time and the job hands Harbor a `${NAME}` template, so the credential appears on neither the
spec, the run bundle, nor `config.json`. This installed NeMo Fabric adapter example needs no loose
source directory:

```json
{
  "target": {
    "kind": "harbor",
    "source": {"import_path": "nemo_fabric.integrations.harbor:FabricAgent"},
    "agent_kwargs": {
      "fabric_adapter_id": "nvidia.fabric.codex",
      "fabric_package": "nemo-fabric[codex]==0.4.0"
    },
    "env_secrets": {
      "OPENAI_API_KEY": "my-workspace/openai-key"
    }
  }
}
```

### Built-in Codex agent

Harbor registers `codex` as a built-in installed agent. It installs the requested Codex CLI version
inside each task container, so neither an `agent_import_path` nor wrapper-source upload is required.
For a direct plugin job target:

```python
from nemo_evals.jobs.agent_spec import HarborBuiltinAgentSource, HarborRunnerTarget
from nhx_evals_sdk import SecretRef

target = HarborRunnerTarget(
    source=HarborBuiltinAgentSource(name="codex", model_name="gpt-5.6-luna"),
    agent_kwargs={"version": "0.153.0"},
    env_secrets={
        "OPENAI_API_KEY": SecretRef(root="my-workspace/openai-key"),
    },
)
```

Replace the model and secret reference with values available to your account. The task image needs
network access and the system tools Harbor's Codex adapter uses to install the CLI. The CLI version
above is an explicit reproducibility pin, not a platform-wide default.

For a complete runnable example, see the
[Harbor taskset evaluation notebook](https://github.com/NVIDIA-NeMo/nemo-helix/blob/main/plugins/nemo-evals/examples/harbor_taskset/harbor_taskset_e2e.ipynb).

## How scoring works

1. Harbor runs each task in its sandbox and writes a `<task>__<hash>/result.json` per trial, including
   the reward mapping from the task's [verifier](https://www.harborframework.com/docs/tasks).
2. `reward_key` identifies the primary reward by name. If not specified, it defaults to `reward`.
   Mapping order and alphabetical order never select the primary.
   The `HarborRewardMetric` that `discover_harbor_tasks` attaches to each task is a placeholder. The
   runner replaces it with one named after `reward_key`, so changing `output_name`, `description` or
   `labels` on the task has no effect. When rescoring saved trials, the key recorded in those trials
   is used; if you rescore with `AgentEvaluator.run(tasks, trials)` and no runner, the task's metric
   is used as is.
3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged;
   a missing or unusable primary emits `0.0` and a diagnostic instead of skipping the trial.
4. Other keys from that task's Harbor-valid results become optional secondaries:
   * Finite numbers are emitted.
   * Missing or Boolean values are omitted with a diagnostic; usable siblings are kept.
   * A `null`, nonnumeric string, or object in the reward mapping fails Harbor's `TrialResult`
     check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes
     `NaN` and infinity as `null`, which hits this gate.
   * A secondary reward key discovered for one task does not apply to another task.
5. `result.summary` aggregates each named output, `result.trials` holds each trial's status and
   evidence, and `result.persist()` writes the standard run bundle.

### What counts as a numeric reward

Harbor's [task format](https://www.harborframework.com/docs/tasks) defines each `reward.json` value
as a float or integer (`reward.txt` is a single number, stored under the key `reward`). Harbor reads
and validates that file before writing `result.json`:

* Finite numbers are retained. Numeric strings such as `"1.25"` are coerced to numbers.
* A nonnumeric string, `null`, or an object fails validation of the whole reward mapping.
* A JSON Boolean is coerced to `1.0` or `0.0`.
* `NaN` and infinity pass Harbor's numeric validation, but serialize as `null` in `result.json`.

The SDK first validates the whole `result.json` with Harbor's `TrialResult`. A `null`, nonnumeric
string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored
or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers,
including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a
diagnostic while retaining usable siblings.

Emit finite JSON numbers from the verifier. Use `1` and `0` when a reward represents pass/fail; do
not rely on Harbor's coercion of strings or Booleans.

### Sparse secondary and errored reward examples

#### Missing secondary reward

For two attempts on task A:

```text
a1: reward=1.0, format_ok=1.0
a2: reward=0.0, format_ok omitted
```

**Resulting aggregates and coverage**

| Result field                     | Expected value                                                           |
| -------------------------------- | ------------------------------------------------------------------------ |
| `harbor_reward.reward`           | `mean=0.5`, `count=2`, `nan_count=0`                                     |
| `harbor_reward.format_ok`        | `mean=1.0`, `count=1`, `nan_count=1`                                     |
| `format_ok` coverage             | `total=2`, `scored=1`, `missing=1`, `failed=0`                           |
| `harbor_reward.format_ok.pass@1` | `mean=1.0`, `count=1`, `nan_count=0`; computed from one measured attempt |
| `harbor_reward.format_ok.pass@2` | unestimable: `count=0`, `nan_count=1`, `mean=None`                       |

Omission means unmeasured, not failure:

* It does not create a null, NaN, or zero per-trial metric output.
* Coverage records the missing measurement.
* Derived aggregates can therefore be unestimable (`mean=None`).

See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for denominator,
failed-trial, semantic-view, and persistence behavior.

#### Errored attempt with a valid primary reward

For two attempts on task A:

```text
a1: reward=1.0, no error
a2: reward=0.8, error=RuntimeError
```

**Resulting statuses and aggregate**

| Result field           | Expected value                       |
| ---------------------- | ------------------------------------ |
| SDK trial `a1`         | `COMPLETED`                          |
| SDK trial `a2`         | `PARTIAL`; remains scoreable         |
| `harbor_reward.reward` | `mean=0.9`, `count=2`, `nan_count=0` |
| `error_trial_ids`      | `{"RuntimeError": ["a2"]}`           |

The error changes `a2`'s status and error rollup; it does not discard or replace its finite reward.

## Attempts and concurrency

* **`n_attempts`** — desired attempts per task. On resume, Harbor runs only the missing attempts.
* **`n_concurrent_trials`** — maximum trials Harbor runs concurrently.

## Retries

* **`max_retries`** — maximum extra attempts per trial during the current Harbor run; defaults to `0`.
* **Error policy** — Harbor retries only allowed errors. Its
  [default non-retryable errors](https://github.com/harbor-framework/harbor/blob/v0.20.0/src/harbor/models/job/config.py#L288-L301)
  include `AgentTimeoutError`.
* **Repeated SDK calls** — `max_retries` never reopens cached errored trials. Cache behavior is
  described below.

## Caching

Caching activates only when `HarborRuntimeConfig.job_name` is pinned. Without it, every call creates a
fresh timestamped job directory.

A cache hit requires both:

* A usable matching stamp for the requested inputs.
* At least `n_attempts` Harbor-valid results for every requested task.

On a repeated SDK call:

* **Cache hit** — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results
  count as completed, become `PARTIAL`, and are not rerun.
* **Incomplete matching cache** — preserves valid results, including errored results, then runs only
  attempts with missing or invalid results.
* **Stale or unusable cache** — deletes the job directory and reruns every requested attempt. This
  includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory.
* **`force_rerun=True`** — deletes the entire pinned job directory, then reruns every requested attempt.
* **Selective cached-error rerun** — unsupported. The SDK cannot rerun cached attempts by error type.
* **Concurrent processes** — must not share a pinned `job_name`; neither the SDK nor Harbor locks the
  job directory.

### Cache identity

The SDK stores the cache stamp in `jobs_dir/<job_name>/.nemo-eval-harbor-cache.json`:

* **`version`** — stamp schema; must match exactly.
* **`options`** — SHA-256 of result-affecting `HarborRuntimeConfig` fields, including `agent_kwargs`;
  must match exactly.
* **`agent`** — digest of `agent_dir` contents, or `"<none>"` when unset; must match exactly.
* **`tasks`** — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C
  can serve A, but cached A cannot serve A and B.
* **Scoring and scheduling settings** — `reward_key`, selected metrics, `quiet`, and
  `n_concurrent_trials` do not invalidate Harbor execution results.
* **`env_secrets` and `env_vars`** — `env_secrets` participates by reference and `env_vars` by
  value. Secret values are never fingerprinted, so rotating a credential keeps the cache valid.
  Job dirs stamped by an SDK release before these fields existed re-run once.
* **Installed agents** — when `agent_dir` is unset, the stamp covers the agent selection or import path,
  not the installed package contents. Change the import path, use `agent_dir`, or force a rerun after
  changing installed agent code.

### Force a complete rerun

```python
import asyncio
from pathlib import Path

from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval

config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    job_name="my-suite",
    agent_name="oracle",
    force_rerun=True,
)
result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite")))
```

* **Keep the previous job** — use a new `job_name`, or omit it to create a fresh timestamped directory.

## Shortcut: `run_harbor_eval`

When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval` collapses the three
steps above — discover, run, score — into a single call:

```python
import asyncio
from pathlib import Path

from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval

result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
    )
)
```

It uses the same `AgentEvaluator` and `HarborRewardMetric` under the hood. `metrics=` replaces the
task metrics, so keep `HarborRewardMetric()` in the list or the reward is not scored:

```python
from nhx_evals_sdk.metrics.exact_match import ExactMatchMetric
from nhx_evals_sdk.metrics.runner_rewards import HarborRewardMetric

result = asyncio.run(
    run_harbor_eval(
        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
        dataset_path=Path("path/to/my-suite"),
        metrics=[HarborRewardMetric(), ExactMatchMetric(reference="Done", candidate="{{sample.output_text}}")],
    )
)
```

Prefer the explicit form
above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across
targets.

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Agent Evaluation Quickstart](/documentation/evaluate-models/agent-eval/quickstart)

#### [Reading Results](/documentation/evaluate-models/agent-eval/reading-results)

#### [Run a NeMo Fabric Agent inside Harbor](/documentation/evaluate-models/agent-eval/harbor-fabric-agent)