Evaluate with a NeMo Fabric Harness

NeMo Fabric runs an agent harness rather than a single agent. Which harness runs is selected entirely by config["harness"]["adapter_id"], so one runtime covers several agent frontends without changing your evaluation. Fabric returns an ATIF trajectory alongside the final answer, so a metric can score how the agent worked, not just what it answered.

The result and bundle are the same as the quickstart — only the runner changes.

This runner executes the harness on the host. To run a Fabric harness inside Harbor’s Docker sandbox and score it with a Harbor verifier instead, see Run a NeMo Fabric Agent inside Harbor.

Harnesses

The agent config

One mapping describes the whole agent — harness, runtime, environment, and model. This is a working Codex configuration:

config = {
    "schema_version": "fabric.agent/v1alpha1",
    "metadata": {"name": "eval-fabric"},
    "harness": {
        "adapter_id": "nvidia.fabric.codex",
        "resolution": "preinstalled",
        "settings": {"sandbox": "workspace-write"},
    },
    "runtime": {
        "mode": "oneshot",
        "transport": "cli",
        "input_schema": "text",
        "output_schema": "message",
        "timeout_seconds": 180,
    },
    "environment": {"provider": "local"},
    "models": {"default": {"provider": "openai", "model": "<provider-model>"}},
    "telemetry": {"enabled": False},
}

An environment.workspace set here is overridden per task — the runtime gives every task its own fresh workspace under work_root, so setting one in the config has no effect.

Across harnesses the shape differs mainly in adapter_id, runtime.transport, and any harness-specific harness.settings. Codex runs as a subprocess (transport: cli) while the Hermes SDK harness runs in-library (transport: library). For complete Codex-CLI and Hermes-SDK configurations, see examples/fabric_harness_runtimes.py in the SDK.

Fold the complete configuration into this mapping. Fabric profile overlays are not used here.

Run it

from pathlib import Path
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk import StringCheckMetric
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
tasks = [
    AgentEvalTask(
        id="reply-ok",
        intent="Reply with a fixed token.",
        inputs={"instruction": "Reply with the word OK."},
        metrics=[
            StringCheckMetric(
                operation="contains",
                left_template="{{sample.output_text}}",
                right_template="OK",
            )
        ],
    )
]
runtime = FabricAgentRuntime(
    config=config,
    work_root="fabric",
    capture_trajectory=True,
)
result = AgentEvaluator().run_sync(
    tasks=tasks,
    target=runtime,
    config=AgentEvalRunConfig(work_dir=Path("out"), parallelism=1),
)

Configuration

Seed files into the workspace

Every task runs in its own fresh workspace. Put files the agent should start from in the task’s inputs["files"], and the runtime stages them before the harness runs:

task = AgentEvalTask(
    id="read-seed",
    intent="Read a seeded file and echo its contents.",
    inputs={
        "instruction": "Read notes.txt in your workspace and reply with exactly its contents.",
        "files": {"notes.txt": "MARKER-7f3a"},
    },
    metrics=[
        StringCheckMetric(
            operation="contains",
            left_template="{{sample.output_text}}",
            right_template="MARKER-7f3a",
        )
    ],
)

With no files key this is a no-op, so tasks that need no starting state cost nothing.

Read the results

The answer is on the trial, and scores come back under the metric’s type:

trial = result.trials[0]
print(trial.status)  # AgentEvalTrialStatus.COMPLETED
assert trial.output is not None
print(trial.output.output_text)  # MARKER-7f3a
for score in result.summary.scores.scores:
    print(score.name)
# string-check.string-check
# string-check.string-check.pass@1

Evidence

This is where Fabric differs from the other runners. Each trial carries several evidence streams, so a metric can score how the agent worked rather than only its final answer:

assert trial.evidence is not None
trace = trial.evidence.descriptors["trace"]
# trace.format == "atif"; trace.ref is a path to the trajectory JSON, which has a "steps" key
workspace = trial.evidence.descriptors["workspace"]
# workspace.ref is a directory — what the agent left behind, including any seeded files

The workspace tree is what lets a metric grade artifacts the agent produced on disk, and trace is what lets it grade the process. See Writing Metrics for reading evidence, and Score by Component for combining several signals into one reported score.

Submit as a platform job

FabricRunnerTarget carries the same config into a durable job:

from nemo_evaluator.jobs.agent_spec import FabricRunnerTarget
target = FabricRunnerTarget(config=config, model="<provider>/<model>", capture_trajectory=True)

Submit it with nemo evaluator agent-evaluate submit, or as the target of an AgentEvalInputSpec. See Targets and Runners.

Running in a sandbox

Pass sandbox= to run the same configuration inside a sandbox rather than on the host:

from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk.agent_eval.runtimes.sandbox.providers.docker import DockerSandboxProvider
runtime = FabricAgentRuntime(config=config, sandbox=DockerSandboxProvider())

model, timeout_s, capture_trajectory, trajectory_extra, skills, work_root, and runtime_name behave the same in both modes. base_dir only applies on the host and is rejected together with sandbox; image and secrets are rejected without it.

Both modes lay evidence out the same way under the per-task directory and expose the same result, trace, and workspace evidence, so the same metrics score either. Sandbox trials additionally carry logs (the harness’s stderr and the OTLP receiver log) and record image and sandbox_provider in their metadata.

Use it when the agent should not run on the host — untrusted tasks, or a workspace that must be discarded per task. Host mode is otherwise the simpler choice.

FabricContainerRuntime(config, provider=...) is a deprecated alias for FabricAgentRuntime(config, sandbox=...) and warns on construction.

Next steps