Run a NeMo Fabric Agent inside Harbor

The Harbor runner runs any importable Harbor agent, and NeMo Fabric ships one. Combining them gives you a Fabric harness inside Harbor’s per-trial Docker sandbox, scored by the task’s verifier, without leaving agent-eval. This page runs the LangChain deepagents harness on a Nemotron model from build.nvidia.com.

Compared with the Fabric runner, which runs the harness on the host and scores with SDK metrics, this path trades the containerless loop for Harbor’s isolation, retries, and verifier rewards. Both stay supported; pick by dataset shape.

The agent

nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent is a thin subclass of nemo_fabric.integrations.harbor:FabricAgent, Fabric’s custom Harbor agent. It accepts every FabricAgent constructor argument through agent_kwargs, and resolves what a non-OpenAI model provider needs from the provider/model slug in agent_model_name:

Override the defaults with fabric_model_api_key_env and fabric_model_base_url in agent_kwargs, for example to point at a self-hosted NIM.

Run on task images without Python

NemoFabricAgent installs Fabric with the task image’s own python3, so it requires an image that ships one. Many Harbor task images do not, and a task’s Dockerfile usually comes with the dataset rather than with your evaluation.

nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_installed_agent:FabricInstalledAgent removes that constraint. It is a Harbor BaseInstalledAgent that wraps a NemoFabricAgent and replaces only the install step, provisioning its own toolchain: curl and CA certificates through whatever package manager the image has, then a pinned uv, then a uv-managed CPython, and only then the Fabric virtualenv. The package-manager step is retried three times with linear backoff, and the uv installer fetch is retried by curl, so a rate-limited or briefly unavailable package mirror does not end the trial — Harbor’s own retry restarts the trial rather than the failed step. Every agent_kwargs key above still applies — they are passed through to the wrapped agent. The task’s Dockerfile needs no Python, pip, or curl.

Because the install runs through BaseInstalledAgent, its failures are classified by Harbor’s ERROR_PATTERNS — a DNS or TLS failure fetching uv raises NetworkConnectionError for --retry-include. This does not extend to the harness itself: Fabric execs its runner inside the container and reports model-side failures in its RunResult (for example deepagents_invocation_failed on a 403), which Harbor records as a completed trial with reward 0.

Swap the import path:

config = HarborRuntimeConfig(
    jobs_dir=Path("./harbor-jobs"),
    agent_import_path="nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_installed_agent:FabricInstalledAgent",
    agent_kwargs={
        "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
        "fabric_package": "nemo-fabric[deepagents,relay]==0.3.0b1",
        "fabric_workspace": "/app",
    },
    agent_model_name="nvidia/nemotron-3.5-lightning-30b-a3b",
    agent_env_from_host=["NVIDIA_API_KEY"],
    agent_setup_timeout_multiplier=12.0,
    agent_timeout_multiplier=5.0,
)

Four differences from NemoFabricAgent:

• fabric_package is required — the harness extra cannot be derived from the adapter id.
• fabric_python_version (default "3.12") selects the interpreter uv provisions. Quote it in YAML so 3.10 does not parse as the float 3.1.
• fabric_uv_version (default "0.12.17") pins the uv installer the task container fetches, so two runs of the same eval provision the same toolchain. Quote it in YAML for the same reason.
• fabric_max_turns defaults to 50 instead of Fabric’s unbounded None. When an unbounded harness cannot finish a task, Harbor eventually kills the agent phase, and a killed phase produces no RunResult — no trajectory and no error, only AgentTimeoutError. A turn budget lets the harness stop on its own terms instead; in our sample runs it accounted for most of the lost trials. Size it for your dataset, or pass fabric_max_turns=None for Fabric’s unbounded behaviour.

The packaged example runs it against a deliberately bare ubuntu:24.04 task — no Python, no pip, no curl — and --dataset-dir points it at any Harbor dataset, including one pulled from the Hub:

export NVIDIA_API_KEY=...
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_installed_example
harbor download terminal-bench-sample -o ./hub --export
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_installed_example \
    --dataset-dir ./hub/terminal-bench-sample

For a real dataset, build the HarborRuntimeConfig yourself rather than using the example: raise n_concurrent_trials, and drop agent_timeout_multiplier to 1.0 where tasks already set a generous timeout_sec of their own (terminal-bench sets 900s, so the example’s 5× would allow 75 minutes a task).

Run it from the SDK

import asyncio
from pathlib import Path
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
async def main() -> None:
    config = HarborRuntimeConfig(
        jobs_dir=Path("./harbor-jobs"),
        agent_import_path="nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
        agent_kwargs={
            "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
            "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
            "fabric_workspace": "/app",  # the task image's working directory
        },
        agent_model_name="nvidia/nemotron-3.5-lightning-30b-a3b",
        # Forwarded to the agent as a `${NVIDIA_API_KEY}` template; Harbor resolves it from this
        # process's environment and never writes the value into the job directory.
        agent_env_from_host=["NVIDIA_API_KEY"],
        # Installing Fabric and the harness in the container takes a few minutes the first time.
        agent_setup_timeout_multiplier=8.0,
        agent_timeout_multiplier=5.0,
    )
    result = await run_harbor_eval(
        config, "packages/nemo_evaluator_sdk/examples/harbor/fabric_agent/fabric_hello_world_dataset"
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
asyncio.run(main())

The same thing is packaged as an example:

export NVIDIA_API_KEY=...
uv run python -m packages.nemo_evaluator_sdk.examples.harbor.fabric_agent.run_fabric_deepagents_example

What happens per trial: Harbor builds and starts the task container, NemoFabricAgent creates a venv inside it and installs nemo-fabric[deepagents], uploads a typed Fabric run spec, and runs the harness against the task’s instruction.md. The deepagents harness works in the Fabric workspace through its filesystem tools, so it writes /app/hello.txt and Harbor’s verifier awards reward: 1.0.

agent_kwargs is persisted verbatim by Harbor in the job directory’s config.json, which is why the key travels through agent_env_from_host instead. Inspect the persisted agent entry after a run:

{
  "import_path": "nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
  "model_name": "nvidia/nemotron-3.5-lightning-30b-a3b",
  "kwargs": {
    "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
    "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
    "fabric_workspace": "/app"
  },
  "env": {"NVIDIA_API_KEY": "${NVIDIA_API_KEY}"}
}

Submit it as a platform job

On the platform the target is a HarborRunnerTarget, the target of an AgentEvalInputSpec. The credential becomes an env_secrets entry: a NeMo Platform secret reference the service resolves into the job’s environment at compile time, which the job then forwards exactly as above.

{
  "target": {
    "kind": "harbor",
    "agent_import_path": "nemo_evaluator_sdk.agent_eval.runtimes.harbor_fabric_agent:NemoFabricAgent",
    "agent_model_name": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "agent_kwargs": {
      "fabric_adapter_id": "nvidia.fabric.langchain.deepagents",
      "fabric_package": "nemo-fabric[deepagents]==0.3.0b1",
      "fabric_workspace": "/app"
    },
    "env_secrets": {
      "NVIDIA_API_KEY": "my-workspace/nvidia-api-key"
    }
  }
}

The value appears on neither the spec, the run bundle, nor config.json. See Evaluate a Harbor Task Suite for the rest of the target’s fields and the job dir’s cache behaviour.

When to use this instead of a built-in Harbor agent

Harbor ships its own agents, such as codex, which install the agent CLI in the task container and drive it with Harbor’s wrapper; if you only want that agent’s reward on a Harbor suite, use them directly (agent_name). Run the agent through Fabric when you want Fabric’s surface: one typed agent config that also runs in the Fabric runner outside Harbor, Fabric’s tool policy, skills, and MCP wiring, and Relay ATIF telemetry for trajectory-aware metrics.

Choosing the harness and model

• Harness: fabric_adapter_id selects it; fabric_package must install the matching extra (nemo-fabric[deepagents], nemo-fabric[codex], …). Harness-specific settings go in fabric_harness_settings.
• Model: any Nemotron id from build.nvidia.com works with the nvidia/ provider. The example uses nvidia/nemotron-3.5-lightning-30b-a3b for speed; larger models such as nvidia/nemotron-3-super-120b-a12b run the same way.
• Telemetry: leave fabric_telemetry at its default none with deepagents; that adapter does not support Fabric’s Relay observability configuration.

Next steps