Run a NeMo Fabric Agent inside Harbor

The Harbor runner runs any importable Harbor agent, and NeMo Fabric ships one. Combining them gives you a Fabric harness inside Harbor’s per-trial Docker sandbox, scored by the task’s verifier, without leaving agent-eval. This page runs the LangChain deepagents harness on a Nemotron model from build.nvidia.com.

Compared with the Fabric runner, which runs the harness on the host and scores with SDK metrics, this path trades the containerless loop for Harbor’s isolation, retries, and verifier rewards. Both stay supported. Choose Harbor when your tasks are a Harbor task suite: each task ships its own container image and verifier, and the reward comes from that verifier. Choose the Fabric runner when your tasks are prompts or files you score yourself with SDK metrics, and a per-trial container would add nothing.

The agent

nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent extends nemo_fabric.integrations.harbor:FabricAgent, Fabric’s custom Harbor agent, and runs one complete Fabric agent config inside the task container. Hand it the agent through agent_kwargs["fabric_config"] — a Fabric agent.yaml as a JSON-shaped mapping, the same document the Fabric runner runs on the host:

from pathlib import Path
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig
fabric_config = {
    "metadata": {"name": "hello-world-deepagents"},
    "harness": {"adapter_id": "nvidia.fabric.langchain.deepagents"},
    "models": {
        "default": {
            "provider": "nvidia",
            "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
            "api_key_env": "NVIDIA_API_KEY",           # the harness reads the key from this variable
            "base_url": "https://integrate.api.nvidia.com/v1",
        }
    },
}

The config is used verbatim — harness, models, instructions, skills, MCP servers, tool policy, and telemetry all come from it, and so does the agent’s identity (metadata.name, the Relay agent_name). Harbor decides only where a trial lives: the workspace and artifact paths inside the container, the text/message runner schemas, and where Relay writes so the trajectory is collected with the trial. Upstream FabricAgent’s flat keywords that describe an agent (fabric_adapter_id, fabric_system_instruction, fabric_max_turns, …) are refused: build the whole config and pass it in. The install/run keywords (fabric_package, fabric_workspace, fabric_config_bundle, …) are unchanged.

agent_model_name, when given, swaps models.default.model only; the endpoint and credential wiring stay the config’s. Files the config refers to by relative path (skills.paths) resolve against fabric_config_target when a fabric_config_bundle directory is uploaded.

Run on task images without Python

NemoFabricAgent installs Fabric with the task image’s own python3, so it requires an image that ships one. Many Harbor task images do not, and a task’s Dockerfile usually comes with the dataset rather than with your evaluation.

nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_installed_agent:FabricInstalledAgent removes that constraint. It is a Harbor BaseInstalledAgent that wraps a NemoFabricAgent and replaces only the install step, provisioning its own toolchain: curl and CA certificates through whatever package manager the image has, then a pinned uv, then a uv-managed CPython, and only then the Fabric virtualenv. The package-manager step is retried three times with linear backoff, and the uv installer fetch is retried by curl, so a rate-limited or briefly unavailable package mirror does not end the trial — Harbor’s own retry restarts the trial rather than the failed step. Every agent_kwargs key above still applies — they are passed through to the wrapped agent. The task’s Dockerfile needs no Python, pip, or curl.

Because the install runs through BaseInstalledAgent, its failures are classified by Harbor’s ERROR_PATTERNS — a DNS or TLS failure fetching uv raises NetworkConnectionError for --retry-include. This does not extend to the harness itself: Fabric execs its runner inside the container and reports model-side failures in its RunResult (for example deepagents_invocation_failed on a 403), which Harbor records as a completed trial with reward 0.

Swap the import path:

from nhx_evals_sdk.values import SecretRef
config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_installed_agent:FabricInstalledAgent",
    agent_kwargs={
        "fabric_config": fabric_config,
        "fabric_package": "nemo-fabric[deepagents,relay]==0.4.0",
        "fabric_workspace": "/app",
    },
    # Standalone, an uppercase ref names the process env var directly: this reads NVIDIA_API_KEY.
    env_secrets={"NVIDIA_API_KEY": SecretRef("NVIDIA_API_KEY")},
    agent_setup_timeout_multiplier=12.0,
    agent_timeout_multiplier=5.0,
)

Four differences from NemoFabricAgent:

• fabric_package is required — the harness extra cannot be derived from the config. Its shape is nemo-fabric[<harness extra>,relay]==<version>: the extra is the harness the config’s adapter.id names (deepagents, hermes-agent, codex, …), relay is the piece that talks to the platform gateway, and the version should match the Fabric the platform ships so the config it translated is one the harness accepts. When you evaluate a registered agent through the platform instead, the resolver derives this value for you.
• fabric_python_version (default "3.12") selects the interpreter uv provisions. Quote it in YAML so 3.10 does not parse as the float 3.1.
• fabric_uv_version (default "0.12.17") pins the uv installer the task container fetches, so two runs of the same eval provision the same toolchain. Quote it in YAML for the same reason.
• A turn budget of 50 is applied when the config sets no runtime.max_turns, instead of Fabric’s unbounded None. When an unbounded harness cannot finish a task, Harbor eventually kills the agent phase, and a killed phase produces no RunResult — no trajectory and no error, only AgentTimeoutError. A budget lets the harness stop on its own terms; in our sample runs it accounted for most of the lost trials. Size it in the config for your dataset, or set runtime.max_turns to an explicit null there (or pass fabric_default_max_turns=None) for Fabric’s unbounded behaviour. The budget and the timeout are complementary: agent_timeout_multiplier is the hard wall Harbor enforces, so raise it when trials end in AgentTimeoutError; the turn budget is what lets the harness stop cleanly inside that wall.

The packaged example runs it against a deliberately bare ubuntu:24.04 task — no Python, no pip, no curl — and --dataset-dir points it at any Harbor dataset, including one pulled from the Hub:

export NVIDIA_API_KEY=...
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_installed_example
harbor download terminal-bench-sample -o ./hub --export
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_installed_example \
    --dataset-dir ./hub/terminal-bench-sample

For a real dataset, build the HarborRuntimeConfig yourself rather than using the example: raise n_concurrent_trials, and drop agent_timeout_multiplier to 1.0 where tasks already set a generous timeout_sec of their own (terminal-bench sets 900s, so the example’s 5× would allow 75 minutes a task).

Run it from the SDK

import asyncio
from pathlib import Path
from nhx_evals_sdk.agent_eval.runtimes.harbor.runtime import HarborRuntimeConfig, run_harbor_eval
from nhx_evals_sdk.values import SecretRef
async def main() -> None:
    config = HarborRuntimeConfig(
        jobs_dir=Path("./harbor-jobs"),
        agent_import_path="nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent",
        agent_kwargs={
            "fabric_config": fabric_config,  # the mapping from "The agent" above
            "fabric_package": "nemo-fabric[deepagents]==0.4.0",
            "fabric_workspace": "/app",  # the task image's working directory
        },
        # Standalone, an uppercase ref names the process env var directly: this reads NVIDIA_API_KEY.
        # Harbor gets a `${NVIDIA_API_KEY}` template and never writes the value into the job directory.
        env_secrets={"NVIDIA_API_KEY": SecretRef("NVIDIA_API_KEY")},
        # Installing Fabric and the harness in the container takes a few minutes the first time.
        agent_setup_timeout_multiplier=8.0,
        agent_timeout_multiplier=5.0,
    )
    result = await run_harbor_eval(
        config, "packages/nhx_evals_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset"
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
asyncio.run(main())

The same thing is packaged as an example:

export NVIDIA_API_KEY=...
uv run python -m packages.nhx_evals_sdk.examples.harbor.fabric_agent.run_fabric_deepagents_example

What happens per trial: Harbor builds and starts the task container, NemoFabricAgent creates a venv inside it and installs nemo-fabric[deepagents], uploads a typed Fabric run spec, and runs the harness against the task’s instruction.md. The deepagents harness works in the Fabric workspace through its filesystem tools, so it writes /app/hello.txt and Harbor’s verifier awards reward: 1.0.

agent_kwargs is persisted verbatim by Harbor in the job directory’s config.json, which is why the key travels through env_secrets instead. Inspect the persisted agent entry after a run:

{
  "import_path": "nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent",
  "kwargs": {
    "fabric_config": {"metadata": {"name": "hello-world-deepagents"}, "harness": {"...": "..."}, "models": {"...": "..."}},
    "fabric_package": "nemo-fabric[deepagents]==0.4.0",
    "fabric_workspace": "/app"
  },
  "env": {"NVIDIA_API_KEY": "${NVIDIA_API_KEY}"}
}

Submit it as a platform job

On the platform the target is a HarborRunnerTarget, the target of an AgentEvalInputSpec. The credential becomes an env_secrets entry: a NeMo Helix secret reference the service resolves into the job’s environment at compile time, which the job then forwards exactly as above.

Create the secret in your workspace first. Platform secret names are lowercase:

nemo secrets create nvidia-api-key --value "$NVIDIA_API_KEY" --workspace my-workspace

Then reference it as my-workspace/nvidia-api-key:

{
  "target": {
    "kind": "harbor",
    "source": {"import_path": "nhx_evals_sdk.agent_eval.runtimes.harbor.fabric_agent:NemoFabricAgent"},
    "agent_kwargs": {
      "fabric_config": {"metadata": {"name": "hello-world-deepagents"}, "harness": {"...": "..."}, "models": {"...": "..."}},
      "fabric_package": "nemo-fabric[deepagents]==0.4.0",
      "fabric_workspace": "/app"
    },
    "env_secrets": {
      "NVIDIA_API_KEY": "my-workspace/nvidia-api-key"
    }
  }
}

The value appears on neither the spec, the run bundle, nor config.json. See Evaluate a Harbor Task Suite for the rest of the target’s fields and the job dir’s cache behaviour.

When to use this instead of a built-in Harbor agent

Harbor ships its own agents, such as codex, which install the agent CLI in the task container and drive it with Harbor’s wrapper; if you only want that agent’s reward on a Harbor suite, use them directly (agent_name). Run the agent through Fabric when you want Fabric’s surface: one typed agent config that also runs in the Fabric runner outside Harbor, Fabric’s tool policy, skills, and MCP wiring, and Relay ATIF telemetry for trajectory-aware metrics.

Choosing the harness and model

• Harness: harness.adapter_id in the config selects it; fabric_package must install the matching extra (nemo-fabric[deepagents], nemo-fabric[codex], …). Harness-specific settings go in harness.settings.
• Model: any Nemotron id from build.nvidia.com works with the nvidia provider and base_url: https://integrate.api.nvidia.com/v1. The example uses nvidia/nemotron-3.5-lightning-30b-a3b for speed; larger models such as nvidia/nemotron-3-super-120b-a12b run the same way.
• Telemetry: leave telemetry/relay out of the config with deepagents; that adapter does not support Fabric’s Relay observability configuration.

Run a registered agent

An agent registered on the platform with nemo agents create can be the Harbor target’s source, in place of a built-in name or your own import_path:

{
  "target": {
    "kind": "harbor",
    "source": {"agent": "calculator-agent"},
    "n_attempts": 2,
    "agent_setup_timeout_multiplier": 12.0,
    "agent_kwargs": {"fabric_workspace": "/app"}
  }
}

At submit time the service resolves the agent exactly as nemo agents deploy would — its models bound to the workspace’s Inference Gateway, the optional environment merged on top — and fills the target in: the agent runs as the SDK’s FabricInstalledAgent (so the task image does not need Python preinstalled; the image requirements above still apply) and agent_kwargs.fabric_config carries the whole translated config, with fabric_package derived from the harness and pinned to the platform’s Fabric version unless you set it. For the deepagents harness the pin also covers mcp and langchain-mcp-adapters at the platform’s versions, because a clean install otherwise resolves an mcp major the adapter cannot import. fabric_package is split like a shell command line, so your own pin can do the same; quote a requirement that contains spaces, such as one with an environment marker. Installing the agent into each task container takes minutes the first time, so raise agent_setup_timeout_multiplier as the example does — Harbor’s default is tuned for prebuilt agents. The persisted job spec shows the filled-in target with source.agent qualified; the job never looks the agent up. If the agent’s config refers to skills or prompts by relative path, a snapshot of its Ethos FileSet taken at submit (a job-owned agent-files-<id> FileSet, deleted when the run completes) is staged into the job and uploaded into every task container as the Fabric config bundle.

source.environment takes the same spec a deployment does — redirect the agent’s declared MCP servers to mocks, add process env, bind secrets by reference — with one Harbor-specific caveat: fulfilment URLs must be reachable from inside the task container, so localhost on the host is not. The same is true of the Inference Gateway itself on a laptop, where the platform binds to localhost: point environment.model_provider_override.base_url at the gateway through host.docker.internal for a local run. A registered agent carries no model_name: its model is part of what it is, and a different model is a different registered agent.

{
  "target": {
    "kind": "harbor",
    "source": {
      "agent": "calculator-agent",
      "environment": {
        "mcp": {"calculator": {"url": "http://mock-calculator:8000/mcp"}},
        "env": {"CALC_MODE": "strict"},
        "secrets": {"CALC_TOKEN": "calc-token"},
        "model_provider_override": {
          "base_url": "http://host.docker.internal:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1"
        }
      }
    },
    "agent_setup_timeout_multiplier": 12.0,
    "agent_kwargs": {"fabric_workspace": "/app"}
  }
}

mcp.calculator.url replaces the URL of the calculator server the agent declares; a server the agent does not declare is rejected. env is plain process environment, secrets binds an environment variable to a Secrets entry by name (the value never enters the spec), and model_provider_override.base_url is the laptop case described above.

One limitation inside the task container: a secret bound to a stdio MCP server through mcp.<name>.secrets reaches the agent’s environment but not the server itself. On the host Fabric runner the resolver names it in the server’s env as a ${NAME} template and the runner fills it at launch; the runner inside a task container expands no templates, so for a Harbor target the template is dropped rather than handed to the server as a literal string. Use a network-transport server, or pass the value another way, until that runner learns to.

Next steps