> This page is for version 0.8.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt
> - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt
> - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Run a NeMo Fabric Agent inside Harbor

> Evaluate a NeMo Fabric harness — here LangChain deepagents on a Nemotron model — inside Harbor's Docker sandbox, with the model API key delivered through env_secrets rather than the agent config.

The [Harbor runner](/documentation/evaluate-models/agent-eval/harbor-runner) runs any importable Harbor
agent, and [NeMo Fabric](https://github.com/NVIDIA/NeMo-Fabric) ships one. Combining them gives you a
Fabric harness inside Harbor's per-trial Docker sandbox, scored by the task's verifier, without leaving
agent-eval. This page runs the LangChain **deepagents** harness on a **Nemotron** model from
[build.nvidia.com](https://build.nvidia.com).

Compared with the [Fabric runner](/documentation/evaluate-models/agent-eval/fabric-runner), which runs
the harness on the host and scores with SDK metrics, this path trades the containerless loop for
Harbor's isolation, retries, and verifier rewards. Both stay supported. Choose Harbor when your tasks are a
Harbor task suite: each task ships its own container image and verifier, and the reward comes from that
verifier. Choose the Fabric runner when your tasks are prompts or files you score yourself with SDK
metrics, and a per-trial container would add nothing.

> **Warning**
>
> Prerequisites are the Harbor runner's (Python ≥ 3.12, Docker, the `harbor` extra) plus:
>
> * **`NVIDIA_API_KEY`** for build.nvidia.com, exported in the process that runs the evaluation.
> * A task image with **CPython and bash**. `NemoFabricAgent` installs Fabric and the harness into the
>   task container at setup time with `python3 -m venv` and pip. The repo ships
>   [`fabric_hello_world_dataset`](https://github.com/NVIDIA-NeMo/nemo-helix/tree/main/packages/nhx_evals_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset)
>   on `python:3.12-slim` for exactly this. To evaluate a task whose image you do not control, use
>   [`FabricInstalledAgent`](#run-on-task-images-without-python) instead, which brings its own
>   Python.
>
> Nothing Fabric-related needs to be installed on the host: the agent class below is part of the SDK,
> and the harness is installed inside the container.

## The agent

`nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent` extends
`nemo_fabric.integrations.harbor:FabricAgent`, Fabric's custom Harbor agent, and runs **one complete
Fabric agent config** inside the task container. Hand it the agent through `agent_kwargs["fabric_config"]`
— a Fabric `agent.yaml` as a JSON-shaped mapping, the same document the
[Fabric runner](/documentation/evaluate-models/agent-eval/fabric-runner) runs on the host:

```python
from pathlib import Path

from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig

fabric_config = {
    "metadata": {"name": "hello-world-deepagents"},
    "harness": {"adapter_id": "nvidia.fabric.langchain.deepagents"},
    "models": {
        "default": {
            "provider": "nvidia",
            "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
            "api_key_env": "NVIDIA_API_KEY",           # the harness reads the key from this variable
            "base_url": "https://integrate.api.nvidia.com/v1",
        }
    },
}
```

The config is used verbatim — harness, models, instructions, skills, MCP servers, tool policy, and
telemetry all come from it, and so does the agent's identity (`metadata.name`, the Relay `agent_name`).
Harbor decides only where a trial lives: the workspace and artifact paths inside the container, the
`text`/`message` runner schemas, and where Relay writes so the trajectory is collected with the trial.
Upstream `FabricAgent`'s flat keywords that *describe* an agent (`fabric_adapter_id`,
`fabric_system_instruction`, `fabric_max_turns`, …) are refused: build the whole config and pass it in.
The install/run keywords (`fabric_package`, `fabric_workspace`, `fabric_config_bundle`, …) are unchanged.

`agent_model_name`, when given, swaps `models.default.model` only; the endpoint and credential wiring
stay the config's. Files the config refers to by relative path (`skills.paths`) resolve against
`fabric_config_target` when a `fabric_config_bundle` directory is uploaded.

## Run on task images without Python

`NemoFabricAgent` installs Fabric with the task image's own `python3`, so it requires an image that
ships one. Many Harbor task images do not, and a task's Dockerfile usually comes with the dataset
rather than with your evaluation.

`nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_installed_agent:FabricInstalledAgent` removes
that constraint. It is a Harbor `BaseInstalledAgent` that wraps a `NemoFabricAgent` and replaces only
the install step, provisioning its own toolchain: `curl` and CA certificates through whatever package
manager the image has, then a pinned `uv`, then a uv-managed CPython, and only then the Fabric
virtualenv. The package-manager step is retried three times with linear backoff, and the uv installer
fetch is retried by `curl`, so a rate-limited or briefly unavailable package mirror does not end the
trial — Harbor's own retry restarts the trial rather than the failed step. Every
`agent_kwargs` key above still applies — they are passed through to the wrapped agent. The task's
Dockerfile needs no Python, pip, or curl.

Because the install runs through `BaseInstalledAgent`, its failures are classified by Harbor's
`ERROR_PATTERNS` — a DNS or TLS failure fetching `uv` raises `NetworkConnectionError` for
`--retry-include`. This does not extend to the harness itself: Fabric execs its runner inside the
container and reports model-side failures in its `RunResult` (for example
`deepagents_invocation_failed` on a 403), which Harbor records as a completed trial with reward `0`.

Swap the import path:

```python
from nhx_evals_sdk.values import SecretRef

config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_installed_agent:FabricInstalledAgent",
    agent_kwargs={
        "fabric_config": fabric_config,
        "fabric_package": "nemo-fabric[deepagents,relay]==0.4.0",
        "fabric_workspace": "/app",
    },
    # Standalone, an uppercase ref names the process env var directly: this reads NVIDIA_API_KEY.
    env_secrets={"NVIDIA_API_KEY": SecretRef("NVIDIA_API_KEY")},
    agent_setup_timeout_multiplier=12.0,
    agent_timeout_multiplier=5.0,
)
```

Four differences from `NemoFabricAgent`:

* `fabric_package` is required — the harness extra cannot be derived from the config. Its shape is
  `nemo-fabric[<harness extra>,relay]==<version>`: the extra is the harness the config's
  `adapter.id` names (`deepagents`, `hermes-agent`, `codex`, ...), `relay` is the piece that talks to
  the platform gateway, and the version should match the Fabric the platform ships so the config
  it translated is one the harness accepts. When you evaluate a registered agent through the
  platform instead, the resolver derives this value for you.
* `fabric_python_version` (default `"3.12"`) selects the interpreter uv provisions. Quote it in YAML
  so `3.10` does not parse as the float `3.1`.
* `fabric_uv_version` (default `"0.12.17"`) pins the uv installer the task container fetches, so two
  runs of the same eval provision the same toolchain. Quote it in YAML for the same reason.
* A turn budget of `50` is applied when the config sets no `runtime.max_turns`, instead of Fabric's
  unbounded `None`. When an unbounded harness cannot finish a task, Harbor eventually kills the agent
  phase, and a killed phase produces no `RunResult` — no trajectory and no error, only
  `AgentTimeoutError`. A budget lets the harness stop on its own terms; in our sample runs it accounted
  for most of the lost trials. Size it in the config for your dataset, or set `runtime.max_turns` to
  an explicit `null` there (or pass `fabric_default_max_turns=None`) for Fabric's unbounded behaviour.
  The budget and the timeout are complementary: `agent_timeout_multiplier` is the hard wall Harbor
  enforces, so raise it when trials end in `AgentTimeoutError`; the turn budget is what lets the
  harness stop cleanly inside that wall.

The packaged example runs it against a deliberately bare `ubuntu:24.04` task — no Python, no pip, no
curl — and `--dataset-dir` points it at any Harbor dataset, including one pulled from the Hub:

```bash
export NVIDIA_API_KEY=...
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_installed_example
harbor download terminal-bench-sample -o ./hub --export
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_installed_example \
    --dataset-dir ./hub/terminal-bench-sample
```

For a real dataset, build the `HarborRuntimeConfig` yourself rather than using the example: raise
`n_concurrent_trials`, and drop `agent_timeout_multiplier` to `1.0` where tasks already set a generous
`timeout_sec` of their own (terminal-bench sets 900s, so the example's 5× would allow 75 minutes a
task).

> **Note**
>
> Three requirements the agent cannot provision for you:
>
> * **bash.** Harbor's `BaseInstalledAgent._exec` prefixes `set -o pipefail` onto every command it runs,
>   so the environment backend has to execute through bash rather than `/bin/sh`. Docker and Daytona
>   do; Harbor's HF sandbox does not, and the install fails there before it starts. This applies to
>   every Harbor installed agent, not only this one.
> * **glibc.** `nemo-fabric-runtime` publishes no musllinux wheels, so an Alpine task fails at the final
>   `uv pip install` with an unsatisfiable resolution.
> * **A working package manager**, when the image has no `curl`. Archived distributions are the main
>   limitation: on `debian:bullseye-slim`, `apt-get install curl ca-certificates` exits 100 because the
>   baked-in index no longer resolves, and the image ships no curl, wget, python3, or busybox to fall
>   back on. Any installed agent hits this — terminal-bench's own `qemu-*` verifiers run the same
>   apt-then-`uv` sequence and fail the same way, so those tasks score `0` even under the oracle
>   agent.

## Run it from the SDK

```python
import asyncio
from pathlib import Path

from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval
from nhx_evals_sdk.values import SecretRef

async def main() -> None:
    config = HarborRuntimeConfig(
        jobs_dir=Path("./harbor-jobs"),
        agent_import_path="nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent",
        agent_kwargs={
            "fabric_config": fabric_config,  # the mapping from "The agent" above
            "fabric_package": "nemo-fabric[deepagents]==0.4.0",
            "fabric_workspace": "/app",  # the task image's working directory
        },
        # Standalone, an uppercase ref names the process env var directly: this reads NVIDIA_API_KEY.
        # Harbor gets a `${NVIDIA_API_KEY}` template and never writes the value into the job directory.
        env_secrets={"NVIDIA_API_KEY": SecretRef("NVIDIA_API_KEY")},
        # Installing Fabric and the harness in the container takes a few minutes the first time.
        agent_setup_timeout_multiplier=8.0,
        agent_timeout_multiplier=5.0,
    )
    result = await run_harbor_eval(
        config, "packages/nhx_evals_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset"
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")

asyncio.run(main())
```

The same thing is packaged as an example:

```bash
export NVIDIA_API_KEY=...
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_deepagents_example
```

What happens per trial: Harbor builds and starts the task container, `NemoFabricAgent` creates a venv
inside it and installs `nemo-fabric[deepagents]`, uploads a typed Fabric run spec, and runs the harness
against the task's `instruction.md`. The deepagents harness works in the Fabric workspace through its
filesystem tools, so it writes `/app/hello.txt` and Harbor's verifier awards `reward: 1.0`.

`agent_kwargs` is persisted verbatim by Harbor in the job directory's `config.json`, which is why the
key travels through `env_secrets` instead. Inspect the persisted agent entry after a run:

```json
{
  "import_path": "nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent",
  "kwargs": {
    "fabric_config": {"metadata": {"name": "hello-world-deepagents"}, "harness": {"...": "..."}, "models": {"...": "..."}},
    "fabric_package": "nemo-fabric[deepagents]==0.4.0",
    "fabric_workspace": "/app"
  },
  "env": {"NVIDIA_API_KEY": "${NVIDIA_API_KEY}"}
}
```

## Submit it as a platform job

On the platform the target is a `HarborRunnerTarget`, the `target` of an `AgentEvalInputSpec`. The
credential becomes an `env_secrets` entry: a NeMo Helix secret reference the service resolves into
the job's environment at compile time, which the job then forwards exactly as above.

Create the secret in your workspace first. Platform secret names are lowercase:

```bash
nemo secrets create nvidia-api-key --value "$NVIDIA_API_KEY" --workspace my-workspace
```

Then reference it as `my-workspace/nvidia-api-key`:

```json
{
  "target": {
    "kind": "harbor",
    "source": {"import_path": "nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent"},
    "agent_kwargs": {
      "fabric_config": {"metadata": {"name": "hello-world-deepagents"}, "harness": {"...": "..."}, "models": {"...": "..."}},
      "fabric_package": "nemo-fabric[deepagents]==0.4.0",
      "fabric_workspace": "/app"
    },
    "env_secrets": {
      "NVIDIA_API_KEY": "my-workspace/nvidia-api-key"
    }
  }
}
```

The value appears on neither the spec, the run bundle, nor `config.json`. See
[Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) for the rest of
the target's fields and the job dir's cache behaviour.

## When to use this instead of a built-in Harbor agent

Harbor ships its own agents, such as `codex`, which install the agent CLI in the task container and
drive it with Harbor's wrapper; if you only want that agent's reward on a Harbor suite, use them
directly (`agent_name`). Run the agent through Fabric when you want Fabric's surface: one typed agent
config that also runs in the [Fabric runner](/documentation/evaluate-models/agent-eval/fabric-runner)
outside Harbor, Fabric's tool policy, skills, and MCP wiring, and Relay ATIF telemetry for
trajectory-aware metrics.

## Choosing the harness and model

* **Harness**: `harness.adapter_id` in the config selects it; `fabric_package` must install the
  matching extra (`nemo-fabric[deepagents]`, `nemo-fabric[codex]`, ...). Harness-specific settings go
  in `harness.settings`.
* **Model**: any Nemotron id from build.nvidia.com works with the `nvidia` provider and
  `base_url: https://integrate.api.nvidia.com/v1`. The example uses
  `nvidia/nemotron-3.5-lightning-30b-a3b` for speed; larger models such as
  `nvidia/nemotron-3-super-120b-a12b` run the same way.
* **Telemetry**: leave `telemetry`/`relay` out of the config with deepagents; that adapter does not
  support Fabric's Relay observability configuration.

## Run a registered agent

An agent registered on the platform with `nemo agents create` can be the Harbor target's `source`,
in place of a built-in `name` or your own `import_path`:

```json
{
  "target": {
    "kind": "harbor",
    "source": {"agent": "calculator-agent"},
    "n_attempts": 2,
    "agent_setup_timeout_multiplier": 12.0,
    "agent_kwargs": {"fabric_workspace": "/app"}
  }
}
```

At submit time the service resolves the agent exactly as `nemo agents deploy` would — its models bound
to the workspace's Inference Gateway, the optional `environment` merged on top — and fills the target
in: the agent runs as the SDK's `FabricInstalledAgent` (so the task image does not need Python
preinstalled; the image requirements above still apply) and
`agent_kwargs.fabric_config` carries the whole translated config, with `fabric_package` derived from
the harness and pinned to the platform's Fabric version unless you set it. For the deepagents harness
the pin also covers `mcp` and `langchain-mcp-adapters` at the platform's versions, because a clean
install otherwise resolves an `mcp` major the adapter cannot import. `fabric_package` is split like a shell command
line, so your own pin can do the same; quote a requirement that contains spaces, such as one with an
environment marker. Installing the agent into
each task container takes minutes the first time, so raise `agent_setup_timeout_multiplier` as the
example does — Harbor's default is tuned for prebuilt agents. The persisted job spec
shows the filled-in target with `source.agent` qualified; the job never looks the agent up.
If the agent's config refers to skills or prompts by relative path, a snapshot of its Ethos FileSet taken
at submit (a job-owned `agent-files-<id>` FileSet, deleted when the run completes) is staged into the job
and uploaded into every task container as the Fabric config bundle.

`source.environment` takes the same spec a deployment does — redirect the agent's declared MCP servers to
mocks, add process env, bind secrets by reference — with one Harbor-specific caveat: fulfilment URLs
must be reachable from *inside* the task container, so `localhost` on the host is not. The same is
true of the Inference Gateway itself on a laptop, where the platform binds to `localhost`: point
`environment.model_provider_override.base_url` at the gateway through `host.docker.internal` for a
local run. A registered agent carries no `model_name`: its model is part of what it is, and a different model
is a different registered agent.

```json
{
  "target": {
    "kind": "harbor",
    "source": {
      "agent": "calculator-agent",
      "environment": {
        "mcp": {"calculator": {"url": "http://mock-calculator:8000/mcp"}},
        "env": {"CALC_MODE": "strict"},
        "secrets": {"CALC_TOKEN": "calc-token"},
        "model_provider_override": {
          "base_url": "http://host.docker.internal:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1"
        }
      }
    },
    "agent_setup_timeout_multiplier": 12.0,
    "agent_kwargs": {"fabric_workspace": "/app"}
  }
}
```

`mcp.calculator.url` replaces the URL of the `calculator` server the agent declares; a server the agent does
not declare is rejected. `env` is plain process environment, `secrets` binds an environment variable to a
Secrets entry by name (the value never enters the spec), and `model_provider_override.base_url` is the
laptop case described above.

One limitation inside the task container: a secret bound to a **stdio** MCP server through
`mcp.<name>.secrets` reaches the agent's environment but not the server itself. On the host Fabric
runner the resolver names it in the server's `env` as a `${NAME}` template and the runner fills it at
launch; the runner inside a task container expands no templates, so for a Harbor target the template
is dropped rather than handed to the server as a literal string. Use a network-transport server, or
pass the value another way, until that runner learns to.

## Next steps

#### [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner)

#### [Evaluate with a NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner)

#### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners)