> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Evaluate with a NeMo Fabric Harness

> Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on.

[NeMo Fabric](https://github.com/nvidia/nemo-fabric) runs an agent **harness** rather than a single
agent. Which harness runs is selected entirely by `config["harness"]["adapter_id"]`, so one runtime
covers several agent frontends without changing your evaluation. Fabric returns an **ATIF
trajectory** alongside the final answer, so a metric can score *how* the agent worked, not just what
it answered.

The result and bundle are the same as the
[quickstart](/documentation/evaluate-models/agent-eval/quickstart) — only the runner changes.

This runner executes the harness on the host. To run a Fabric harness inside Harbor's Docker sandbox and
score it with a Harbor verifier instead, see
[Run a NeMo Fabric Agent inside Harbor](/documentation/evaluate-models/agent-eval/harbor-fabric-agent).

## Harnesses

| `adapter_id`                         | Harness              | Transport              |
| ------------------------------------ | -------------------- | ---------------------- |
| `nvidia.fabric.codex`                | Codex CLI            | `cli` (subprocess)     |
| `nvidia.fabric.claude`               | Claude               | `cli`                  |
| `nvidia.fabric.hermes`               | Hermes SDK           | `library` (in-process) |
| `nvidia.fabric.langchain.deepagents` | LangChain deepagents | `library`              |

This runner is **not** zero-dependency. It needs:

* **The harness adapters**, from the `fabric` extra:

  ```bash
  uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact
  ```

* **The `nemo-relay` gateway binary**, which captures the trajectory for out-of-process harnesses.
  The pip package ships bindings only, so the daemon comes from a GitHub release asset:

  ```bash
  script/dev-install-fabric.sh
  ```

* **The harness's own CLI**, for `transport: cli` harnesses — `codex` on `PATH` and authenticated.

The `fabric` extra installs the Codex, Claude, and Hermes adapters. The deepagents adapter is
deliberately excluded from it, because that adapter does not support the Relay observability
configuration Fabric streaming generates; install its harness separately if you need it.

## The agent config

One mapping describes the whole agent — harness, runtime, environment, and model. This is a working
Codex configuration:

```python
config = {
    "schema_version": "fabric.agent/v1alpha1",
    "metadata": {"name": "eval-fabric"},
    "harness": {
        "adapter_id": "nvidia.fabric.codex",
        "resolution": "preinstalled",
        "settings": {"sandbox": "workspace-write"},
    },
    "runtime": {
        "mode": "oneshot",
        "transport": "cli",
        "input_schema": "text",
        "output_schema": "message",
        "timeout_seconds": 180,
    },
    "environment": {"provider": "local"},
    "models": {"default": {"provider": "openai", "model": "<provider-model>"}},
    "telemetry": {"enabled": False},
}
```

An `environment.workspace` set here is **overridden per task** — the runtime gives every task its own
fresh workspace under `work_root`, so setting one in the config has no effect.

Across harnesses the shape differs mainly in `adapter_id`, `runtime.transport`, and any
harness-specific `harness.settings`. Codex runs as a subprocess (`transport: cli`) while the Hermes
SDK harness runs in-library (`transport: library`). For complete Codex-CLI and Hermes-SDK
configurations, see `examples/fabric_harness_runtimes.py` in the SDK.

**The Codex adapter requires an explicit model provider.** It does not fall back to the Codex CLI's
own configured default, and starting without `models.default` fails the adapter lifecycle with
`codex_invalid_configuration`.

Fold the complete configuration into this mapping. Fabric profile overlays are not used here.

## Run it

```python
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk import StringCheckMetric
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask

tasks = [
    AgentEvalTask(
        id="reply-ok",
        intent="Reply with a fixed token.",
        inputs={"instruction": "Reply with the word OK."},
        metrics=[
            StringCheckMetric(
                operation="contains",
                left_template="{{sample.output_text}}",
                right_template="OK",
            )
        ],
    )
]

runtime = FabricAgentRuntime(
    config=config,
    work_root="fabric",
    capture_trajectory=True,
)

result = AgentEvaluator().run_sync(
    tasks=tasks,
    target=runtime,
    config=AgentEvalRunConfig(work_dir=Path("out"), parallelism=1),
)
```

### Configuration

| Field                | Required | Notes                                                                                             |
| -------------------- | -------- | ------------------------------------------------------------------------------------------------- |
| `config`             | yes      | the agent config mapping above                                                                    |
| `model`              | no       | overrides `models.default.model` without editing the config                                       |
| `base_dir`           | no       | base directory for relative paths in the config                                                   |
| `work_root`          | no       | where Fabric's per-run working directories are created                                            |
| `timeout_s`          | no       | overall run timeout, default `600`                                                                |
| `capture_trajectory` | no       | capture the ATIF trajectory as evidence, default `True`                                           |
| `trajectory_extra`   | no       | extra fields merged into the captured trajectory                                                  |
| `runtime_name`       | no       | the name recorded on `runner_info`, default `fabric`                                              |
| `skills`             | no       | skills injected into the agent's workspace                                                        |
| `sandbox`            | no       | a `SandboxProvider`; when set, each task runs inside a sandbox (see below)                        |
| `image`              | no       | the sandbox image; only valid with `sandbox`                                                      |
| `secrets`            | no       | secret references injected as environment variables inside the sandbox; only valid with `sandbox` |

## Seed files into the workspace

Every task runs in its own fresh workspace. Put files the agent should start from in the task's
`inputs["files"]`, and the runtime stages them before the harness runs:

```python
task = AgentEvalTask(
    id="read-seed",
    intent="Read a seeded file and echo its contents.",
    inputs={
        "instruction": "Read notes.txt in your workspace and reply with exactly its contents.",
        "files": {"notes.txt": "MARKER-7f3a"},
    },
    metrics=[
        StringCheckMetric(
            operation="contains",
            left_template="{{sample.output_text}}",
            right_template="MARKER-7f3a",
        )
    ],
)
```

With no `files` key this is a no-op, so tasks that need no starting state cost nothing.

## Read the results

The answer is on the trial, and scores come back under the metric's type:

```python
trial = result.trials[0]
print(trial.status)  # AgentEvalTrialStatus.COMPLETED
assert trial.output is not None
print(trial.output.output_text)  # MARKER-7f3a

for score in result.summary.scores.scores:
    print(score.name)
# string-check.string-check
# string-check.string-check.pass@1
```

### Evidence

This is where Fabric differs from the other runners. Each trial carries several evidence streams, so
a metric can score *how* the agent worked rather than only its final answer:

| Key                         | Kind                    | What it is                                                |
| --------------------------- | ----------------------- | --------------------------------------------------------- |
| `trace`                     | `trace` (format `atif`) | the ATIF trajectory — tool calls, reasoning, observations |
| `workspace`                 | `filesystem`            | the workspace's final file tree, after the agent finished |
| `result`                    | `json`                  | the harness's structured result                           |
| `stdout`                    | `log`                   | harness stdout                                            |
| `relay_atif` / `relay_atof` | `atif` / `atof`         | the raw Relay artifacts the trajectory is promoted from   |
| `relay_config`              | `telemetry_config`      | the Relay configuration used for the run                  |

```python
assert trial.evidence is not None
trace = trial.evidence.descriptors["trace"]
# trace.format == "atif"; trace.ref is a path to the trajectory JSON, which has a "steps" key

workspace = trial.evidence.descriptors["workspace"]
# workspace.ref is a directory — what the agent left behind, including any seeded files
```

The `workspace` tree is what lets a metric grade artifacts the agent produced on disk, and `trace`
is what lets it grade the process. See
[Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) for reading evidence,
and [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) for combining
several signals into one reported score.

## Submit as a platform job

`FabricRunnerTarget` carries the same config into a durable job:

```python
from nemo_evaluator.jobs.agent_spec import FabricRunnerTarget

target = FabricRunnerTarget(config=config, model="<provider>/<model>", capture_trajectory=True)
```

| Field                | Required | Notes                                                         |
| -------------------- | -------- | ------------------------------------------------------------- |
| `config`             | yes      | the same agent config, as a JSON-shaped mapping               |
| `model`              | no       | a `provider/model` slug applied as the config's default model |
| `timeout_s`          | no       | per-task timeout for the Fabric run                           |
| `capture_trajectory` | no       | capture the ATIF trajectory and attach it to trial evidence   |

Submit it with `nemo evaluator agent-evaluate submit`, or as the `target` of an
`AgentEvalInputSpec`. See
[Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners).

## Running in a sandbox

Pass `sandbox=` to run the same configuration inside a sandbox rather than on the host:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk.agent_eval.runtimes.sandbox.providers.docker import DockerSandboxProvider

runtime = FabricAgentRuntime(config=config, sandbox=DockerSandboxProvider())
```

| Field     | Required | Notes                                                                             |
| --------- | -------- | --------------------------------------------------------------------------------- |
| `sandbox` | yes      | a `SandboxProvider` — `DockerSandboxProvider` or `DockerComposeSandboxProvider`   |
| `image`   | no       | override the sandbox image; built on first run when omitted                       |
| `secrets` | no       | `{env_var: SecretRef}` resolved by the orchestrator and injected into the sandbox |

`model`, `timeout_s`, `capture_trajectory`, `trajectory_extra`, `skills`, `work_root`, and
`runtime_name` behave the same in both modes. `base_dir` only applies on the host and is rejected
together with `sandbox`; `image` and `secrets` are rejected without it.

Both modes lay evidence out the same way under the per-task directory and expose the same `result`,
`trace`, and `workspace` evidence, so the same metrics score either. Sandbox trials additionally carry
`logs` (the harness's stderr and the OTLP receiver log) and record `image` and `sandbox_provider` in
their metadata.

Use it when the agent should not run on the host — untrusted tasks, or a workspace that must be
discarded per task. Host mode is otherwise the simpler choice.

`FabricContainerRuntime(config, provider=...)` is a deprecated alias for
`FabricAgentRuntime(config, sandbox=...)` and warns on construction.

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners)

#### [Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics)