> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Run a NeMo Fabric Agent inside Harbor

> Evaluate a NeMo Fabric harness — here LangChain deepagents on a Nemotron model — inside Harbor's Docker sandbox, with the model API key delivered through env_secrets rather than the agent config.

The [Harbor runner](/documentation/evaluate-models/agent-eval/harbor-runner) runs any importable Harbor
agent, and [NeMo Fabric](https://github.com/NVIDIA/NeMo-Fabric) ships one. Combining them gives you a
Fabric harness inside Harbor's per-trial Docker sandbox, scored by the task's verifier, without leaving
agent-eval. This page runs the LangChain **deepagents** harness on a **Nemotron** model from
[build.nvidia.com](https://build.nvidia.com).

Compared with the [Fabric runner](/documentation/evaluate-models/agent-eval/fabric-runner), which runs
the harness on the host and scores with SDK metrics, this path trades the containerless loop for
Harbor's isolation, retries, and verifier rewards. Both stay supported; pick by dataset shape.

Prerequisites are the Harbor runner's (Python ≥ 3.12, Docker, the `harbor` extra) plus:

* **`NVIDIA_API_KEY`** for build.nvidia.com, exported in the process that runs the evaluation.
* A task image with **CPython and bash**. `NemoFabricAgent` installs Fabric and the harness into the
  task container at setup time with `python3 -m venv` and pip. The repo ships
  [`fabric_hello_world_dataset`](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/packages/nemo_evaluator_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset)
  on `python:3.12-slim` for exactly this. To evaluate a task whose image you do not control, use
  [`FabricInstalledAgent`](#run-on-task-images-without-python) instead, which brings its own
  Python.

Nothing Fabric-related needs to be installed on the host: the agent class below is part of the SDK,
and the harness is installed inside the container.

## The agent

`nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent` is a thin subclass of
`nemo_fabric.integrations.harbor:FabricAgent`, Fabric's custom Harbor agent. It accepts every `FabricAgent` constructor
argument through `agent_kwargs`, and resolves what a non-OpenAI model provider needs from the
`provider/model` slug in `agent_model_name`:

| `agent_model_name`                  | provider | model sent to the endpoint          | credential variable                   | endpoint                              |
| ----------------------------------- | -------- | ----------------------------------- | ------------------------------------- | ------------------------------------- |
| `nvidia/nemotron-3-super-120b-a12b` | `nvidia` | `nvidia/nemotron-3-super-120b-a12b` | `NVIDIA_API_KEY`                      | `https://integrate.api.nvidia.com/v1` |
| `openai/gpt-5.4`                    | `openai` | `gpt-5.4`                           | `OPENAI_API_KEY`                      | OpenAI default                        |
| anything else                       | as given | as given                            | `fabric_model_api_key_env` (required) | `fabric_model_base_url`               |

Override the defaults with `fabric_model_api_key_env` and `fabric_model_base_url` in `agent_kwargs`,
for example to point at a self-hosted NIM.

## Run on task images without Python

`NemoFabricAgent` installs Fabric with the task image's own `python3`, so it requires an image that
ships one. Many Harbor task images do not, and a task's Dockerfile usually comes with the dataset
rather than with your evaluation.

`nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_installed_agent:FabricInstalledAgent` removes
that constraint. It is a Harbor `BaseInstalledAgent` that wraps a `NemoFabricAgent` and replaces only
the install step, provisioning its own toolchain: `curl` and CA certificates through whatever package
manager the image has, then a pinned `uv`, then a uv-managed CPython, and only then the Fabric
virtualenv. The package-manager step is retried three times with linear backoff, and the uv installer
fetch is retried by `curl`, so a rate-limited or briefly unavailable package mirror does not end the
trial — Harbor's own retry restarts the trial rather than the failed step. Every
`agent_kwargs` key above still applies — they are passed through to the wrapped agent. The task's
Dockerfile needs no Python, pip, or curl.

Because the install runs through `BaseInstalledAgent`, its failures are classified by Harbor's
`ERROR_PATTERNS` — a DNS or TLS failure fetching `uv` raises `NetworkConnectionError` for
`--retry-include`. This does not extend to the harness itself: Fabric execs its runner inside the
container and reports model-side failures in its `RunResult` (for example
`deepagents_invocation_failed` on a 403), which Harbor records as a completed trial with reward `0`.

Swap the import path:

```python
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_installed_agent:FabricInstalledAgent",
    agent_kwargs={
        "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
        "fabric_package": "nemo-fabric[deepagents,relay]==0.3.0b1",
        "fabric_workspace": "/app",
    },
    agent_model_name="nvidia/nemotron-3.5-lightning-30b-a3b",
    agent_env_from_host=["NVIDIA_API_KEY"],
    agent_setup_timeout_multiplier=12.0,
    agent_timeout_multiplier=5.0,
)
```

Four differences from `NemoFabricAgent`:

* `fabric_package` is required — the harness extra cannot be derived from the adapter id.
* `fabric_python_version` (default `"3.12"`) selects the interpreter uv provisions. Quote it in YAML
  so `3.10` does not parse as the float `3.1`.
* `fabric_uv_version` (default `"0.12.17"`) pins the uv installer the task container fetches, so two
  runs of the same eval provision the same toolchain. Quote it in YAML for the same reason.
* `fabric_max_turns` defaults to `50` instead of Fabric's unbounded `None`. When an unbounded harness
  cannot finish a task, Harbor eventually kills the agent phase, and a killed phase produces no
  `RunResult` — no trajectory and no error, only `AgentTimeoutError`. A turn budget lets the harness
  stop on its own terms instead; in our sample runs it accounted for most of the lost trials. Size it
  for your dataset, or pass `fabric_max_turns=None` for Fabric's unbounded behaviour.

The packaged example runs it against a deliberately bare `ubuntu:24.04` task — no Python, no pip, no
curl — and `--dataset-dir` points it at any Harbor dataset, including one pulled from the Hub:

```bash
export NVIDIA_API_KEY=...
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_installed_example
harbor download terminal-bench-sample -o ./hub --export
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_installed_example \
    --dataset-dir ./hub/terminal-bench-sample
```

For a real dataset, build the `HarborRuntimeConfig` yourself rather than using the example: raise
`n_concurrent_trials`, and drop `agent_timeout_multiplier` to `1.0` where tasks already set a generous
`timeout_sec` of their own (terminal-bench sets 900s, so the example's 5× would allow 75 minutes a
task).

Three requirements the agent cannot provision for you:

* **bash.** Harbor's `BaseInstalledAgent._exec` prefixes `set -o pipefail` onto every command it runs,
  so the environment backend has to execute through bash rather than `/bin/sh`. Docker and Daytona
  do; Harbor's HF sandbox does not, and the install fails there before it starts. This applies to
  every Harbor installed agent, not only this one.
* **glibc.** `nemo-fabric-runtime` publishes no musllinux wheels, so an Alpine task fails at the final
  `uv pip install` with an unsatisfiable resolution.
* **A working package manager**, when the image has no `curl`. Archived distributions are the main
  limitation: on `debian:bullseye-slim`, `apt-get install curl ca-certificates` exits 100 because the
  baked-in index no longer resolves, and the image ships no curl, wget, python3, or busybox to fall
  back on. Any installed agent hits this — terminal-bench's own `qemu-*` verifiers run the same
  apt-then-`uv` sequence and fail the same way, so those tasks score `0` even under the oracle
  agent.

## Run it from the SDK

```python
import asyncio
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval

async def main() -> None:
    config = HarborRuntimeConfig(
        jobs_dir=Path("./harbor-jobs"),
        agent_import_path="nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
        agent_kwargs={
            "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
            "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
            "fabric_workspace": "/app",  # the task image's working directory
        },
        agent_model_name="nvidia/nemotron-3.5-lightning-30b-a3b",
        # Forwarded to the agent as a `${NVIDIA_API_KEY}` template; Harbor resolves it from this
        # process's environment and never writes the value into the job directory.
        agent_env_from_host=["NVIDIA_API_KEY"],
        # Installing Fabric and the harness in the container takes a few minutes the first time.
        agent_setup_timeout_multiplier=8.0,
        agent_timeout_multiplier=5.0,
    )
    result = await run_harbor_eval(
        config, "packages/nemo_evaluator_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset"
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")

asyncio.run(main())
```

The same thing is packaged as an example:

```bash
export NVIDIA_API_KEY=...
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_deepagents_example
```

What happens per trial: Harbor builds and starts the task container, `NemoFabricAgent` creates a venv
inside it and installs `nemo-fabric[deepagents]`, uploads a typed Fabric run spec, and runs the harness
against the task's `instruction.md`. The deepagents harness works in the Fabric workspace through its
filesystem tools, so it writes `/app/hello.txt` and Harbor's verifier awards `reward: 1.0`.

`agent_kwargs` is persisted verbatim by Harbor in the job directory's `config.json`, which is why the
key travels through `agent_env_from_host` instead. Inspect the persisted agent entry after a run:

```json
{
  "import_path": "nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
  "model_name": "nvidia/nemotron-3.5-lightning-30b-a3b",
  "kwargs": {
    "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
    "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
    "fabric_workspace": "/app"
  },
  "env": {"NVIDIA_API_KEY": "${NVIDIA_API_KEY}"}
}
```

## Submit it as a platform job

On the platform the target is a `HarborRunnerTarget`, the `target` of an `AgentEvalInputSpec`. The
credential becomes an `env_secrets` entry: a NeMo Platform secret reference the service resolves into
the job's environment at compile time, which the job then forwards exactly as above.

```json
{
  "target": {
    "kind": "harbor",
    "agent_import_path": "nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
    "agent_model_name": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "agent_kwargs": {
      "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
      "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
      "fabric_workspace": "/app"
    },
    "env_secrets": {
      "NVIDIA_API_KEY": "my-workspace/nvidia-api-key"
    }
  }
}
```

The value appears on neither the spec, the run bundle, nor `config.json`. See
[Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) for the rest of
the target's fields and the job dir's cache behaviour.

## When to use this instead of a built-in Harbor agent

Harbor ships its own agents, such as `codex`, which install the agent CLI in the task container and
drive it with Harbor's wrapper; if you only want that agent's reward on a Harbor suite, use them
directly (`agent_name`). Run the agent through Fabric when you want Fabric's surface: one typed agent
config that also runs in the [Fabric runner](/documentation/evaluate-models/agent-eval/fabric-runner)
outside Harbor, Fabric's tool policy, skills, and MCP wiring, and Relay ATIF telemetry for
trajectory-aware metrics.

## Choosing the harness and model

* **Harness**: `fabric_adapter_id` selects it; `fabric_package` must install the matching extra
  (`nemo-fabric[deepagents]`, `nemo-fabric[codex]`, ...). Harness-specific settings go in
  `fabric_harness_settings`.
* **Model**: any Nemotron id from build.nvidia.com works with the `nvidia/` provider. The example
  uses `nvidia/nemotron-3.5-lightning-30b-a3b` for speed; larger models such as
  `nvidia/nemotron-3-super-120b-a12b` run the same way.
* **Telemetry**: leave `fabric_telemetry` at its default `none` with deepagents; that adapter does
  not support Fabric's Relay observability configuration.

## Next steps

#### [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner)

#### [Evaluate with a NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner)

#### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners)