> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Evaluate with a NeMo Fabric Harness

> Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on.

[NeMo Fabric](https://github.com/nvidia/nemo-fabric) runs an agent **harness** rather than a single
agent. Which harness runs is selected entirely by `config["harness"]["adapter_id"]`, so one runtime
covers several agent frontends without changing your evaluation. Fabric returns an **ATIF
trajectory** alongside the final answer, so a metric can score *how* the agent worked, not just what
it answered.

The result and bundle are the same as the
[quickstart](/documentation/evaluate-models/agent-eval/quickstart) — only the runner changes.

## Harnesses

| `adapter_id`                         | Harness              | Transport              |
| ------------------------------------ | -------------------- | ---------------------- |
| `nvidia.fabric.codex`                | Codex CLI            | `cli` (subprocess)     |
| `nvidia.fabric.claude`               | Claude               | `cli`                  |
| `nvidia.fabric.hermes`               | Hermes SDK           | `library` (in-process) |
| `nvidia.fabric.langchain.deepagents` | LangChain deepagents | `library`              |

This runner is **not** zero-dependency. It needs:

* **The harness adapters**, from the `fabric` extra:

  ```bash
  uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact
  ```

* **The `nemo-relay` gateway binary**, which captures the trajectory for out-of-process harnesses.
  The pip package ships bindings only, so the daemon comes from a GitHub release asset:

  ```bash
  script/dev-install-fabric.sh
  ```

* **The harness's own CLI**, for `transport: cli` harnesses — `codex` on `PATH` and authenticated.

The `fabric` extra installs the Codex, Claude, and Hermes adapters. The deepagents adapter is
deliberately excluded from it, because that adapter does not support the Relay observability
configuration Fabric streaming generates; install its harness separately if you need it.

## The agent config

One mapping describes the whole agent — harness, runtime, environment, and model. This is a working
Codex configuration:

```python
config = {
    "schema_version": "fabric.agent/v1alpha1",
    "metadata": {"name": "eval-fabric"},
    "harness": {
        "adapter_id": "nvidia.fabric.codex",
        "resolution": "preinstalled",
        "settings": {"sandbox": "workspace-write"},
    },
    "runtime": {
        "mode": "oneshot",
        "transport": "cli",
        "input_schema": "text",
        "output_schema": "message",
        "timeout_seconds": 180,
    },
    "environment": {"provider": "local"},
    "models": {"default": {"provider": "openai", "model": "<provider-model>"}},
    "telemetry": {"enabled": False},
}
```

An `environment.workspace` set here is **overridden per task** — the runtime gives every task its own
fresh workspace under `work_root`, so setting one in the config has no effect.

Across harnesses the shape differs mainly in `adapter_id`, `runtime.transport`, and any
harness-specific `harness.settings`. Codex runs as a subprocess (`transport: cli`) while the Hermes
SDK harness runs in-library (`transport: library`). For complete Codex-CLI and Hermes-SDK
configurations, see `examples/fabric_harness_runtimes.py` in the SDK.

**The Codex adapter requires an explicit model provider.** It does not fall back to the Codex CLI's
own configured default, and starting without `models.default` fails the adapter lifecycle with
`codex_invalid_configuration`.

Fold the complete configuration into this mapping. Fabric profile overlays are not used here.

## Run it

```python
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk import StringCheckMetric
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask

tasks = [
    AgentEvalTask(
        id="reply-ok",
        intent="Reply with a fixed token.",
        inputs={"instruction": "Reply with the word OK."},
        metrics=[
            StringCheckMetric(
                operation="contains",
                left_template="{{sample.output_text}}",
                right_template="OK",
            )
        ],
    )
]

runtime = FabricAgentRuntime(
    config=config,
    work_root="fabric",
    capture_trajectory=True,
)

result = AgentEvaluator().run_sync(
    tasks=tasks,
    target=runtime,
    config=AgentEvalRunConfig(work_dir=Path("out"), parallelism=1),
)
```

### Configuration

| Field                | Required | Notes                                                       |
| -------------------- | -------- | ----------------------------------------------------------- |
| `config`             | yes      | the agent config mapping above                              |
| `model`              | no       | overrides `models.default.model` without editing the config |
| `base_dir`           | no       | base directory for relative paths in the config             |
| `work_root`          | no       | where Fabric's per-run working directories are created      |
| `timeout_s`          | no       | overall run timeout, default `600`                          |
| `capture_trajectory` | no       | capture the ATIF trajectory as evidence, default `True`     |
| `trajectory_extra`   | no       | extra fields merged into the captured trajectory            |
| `runtime_name`       | no       | the name recorded on `runner_info`, default `fabric`        |
| `skills`             | no       | skills injected into the agent's workspace                  |
| `task_hook`          | no       | a `FabricTaskRunHook` invoked around each task run          |

## Seed files into the workspace

Every task runs in its own fresh workspace. Put files the agent should start from in the task's
`inputs["files"]`, and the runtime stages them before the harness runs:

```python
task = AgentEvalTask(
    id="read-seed",
    intent="Read a seeded file and echo its contents.",
    inputs={
        "instruction": "Read notes.txt in your workspace and reply with exactly its contents.",
        "files": {"notes.txt": "MARKER-7f3a"},
    },
    metrics=[
        StringCheckMetric(
            operation="contains",
            left_template="{{sample.output_text}}",
            right_template="MARKER-7f3a",
        )
    ],
)
```

With no `files` key this is a no-op, so tasks that need no starting state cost nothing.

## Read the results

The answer is on the trial, and scores come back under the metric's type:

```python
trial = result.trials[0]
print(trial.status)  # AgentEvalTrialStatus.COMPLETED
assert trial.output is not None
print(trial.output.output_text)  # MARKER-7f3a

for score in result.summary.scores.scores:
    print(score.name)
# string-check.string-check
# string-check.string-check.pass@1
```

### Evidence

This is where Fabric differs from the other runners. Each trial carries several evidence streams, so
a metric can score *how* the agent worked rather than only its final answer:

| Key                         | Kind                    | What it is                                                |
| --------------------------- | ----------------------- | --------------------------------------------------------- |
| `trace`                     | `trace` (format `atif`) | the ATIF trajectory — tool calls, reasoning, observations |
| `workspace`                 | `filesystem`            | the workspace's final file tree, after the agent finished |
| `result`                    | `json`                  | the harness's structured result                           |
| `stdout`                    | `log`                   | harness stdout                                            |
| `relay_atif` / `relay_atof` | `atif` / `atof`         | the raw Relay artifacts the trajectory is promoted from   |
| `relay_config`              | `telemetry_config`      | the Relay configuration used for the run                  |

```python
assert trial.evidence is not None
trace = trial.evidence.descriptors["trace"]
# trace.format == "atif"; trace.ref is a path to the trajectory JSON, which has a "steps" key

workspace = trial.evidence.descriptors["workspace"]
# workspace.ref is a directory — what the agent left behind, including any seeded files
```

The `workspace` tree is what lets a metric grade artifacts the agent produced on disk, and `trace`
is what lets it grade the process. See
[Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) for reading evidence,
and [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) for combining
several signals into one reported score.

## Submit as a platform job

`FabricRunnerTarget` carries the same config into a durable job:

```python
from nemo_evaluator.jobs.agent_spec import FabricRunnerTarget

target = FabricRunnerTarget(config=config, model="<provider>/<model>", capture_trajectory=True)
```

| Field                | Required | Notes                                                         |
| -------------------- | -------- | ------------------------------------------------------------- |
| `config`             | yes      | the same agent config, as a JSON-shaped mapping               |
| `model`              | no       | a `provider/model` slug applied as the config's default model |
| `timeout_s`          | no       | per-task timeout for the Fabric run                           |
| `capture_trajectory` | no       | capture the ATIF trajectory and attach it to trial evidence   |

Submit it with `nemo evaluator agent-evaluate submit`, or as the `target` of an
`AgentEvalInputSpec`. See
[Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners).

## Running in a sandbox

`FabricContainerRuntime` takes the same config and runs it inside a sandbox rather than on the host:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.container_runtime import FabricContainerRuntime
from nemo_evaluator_sdk.agent_eval.runtimes.sandbox.providers.docker import DockerSandboxProvider

runtime = FabricContainerRuntime(config=config, provider=DockerSandboxProvider())
```

| Field      | Required | Notes                                                                           |
| ---------- | -------- | ------------------------------------------------------------------------------- |
| `config`   | yes      | the same agent config, or a `FabricConfig`                                      |
| `provider` | yes      | a `SandboxProvider` — `DockerSandboxProvider` or `DockerComposeSandboxProvider` |
| `secrets`  | no       | secret references made available inside the sandbox                             |
| `image`    | no       | override the sandbox image                                                      |
| `skills`   | no       | skills injected into the agent's workspace                                      |

Use it when the agent should not run on the host — untrusted tasks, or a workspace that must be
discarded per task. The host runtime is otherwise the simpler choice.

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners)

#### [Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics)