> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Agent Evaluation

> The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics.

Agent evaluation is the **task-driven** shape of NeMo Evaluator (see
[Evaluation Approaches](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation)
for how it compares to metrics). You define **tasks**, an **agent** performs each one,
and you score the resulting **trial** — not just whether the final answer is right, but *how the agent
got there*. Each task carries its own metrics, so a single suite can grade heterogeneous work.

> **Note**
>
> **Local interfaces and durable platform interfaces are available.**
>
> * Use `await AgentEvaluator().run(tasks=..., target=...)` for local task-driven SDK runs
>   that do not require running nemo-helix.
> * Use the Evaluator plugin's `uv run nemo evaluator agent-evaluate` job for durable runs with inline
>   tasks or stored tasksets. For a `GymAgentTaskRunner` that you have already validated locally,
>   `evaluator.submit(tasks=TasksetRef(...), target=runner)` provides a high-level submission
>   path; build other runner targets in an agent-evaluate spec.

## The model

One run flows through five pieces:

**Task → (runner) → Trial → (metrics) → Scores → Result**

* **Task** (`AgentEvalTask`) — the unit of work. Its fields:
  * `intent` — a human-readable description of the goal; metadata for the suite's authors, **not** shown to the agent.
  * `inputs` — the instruction and anything the agent starts from (what the agent actually sees).
  * `reference` (optional) — grader-only, held-out ground truth the agent never sees.
  * `metrics` — the scorers for **this** task.
  * `views` (optional) — combine several of the task's metric outputs into one named, reported score (see *Score by component* under [Key properties](#key-properties)).
* **Runner** (`AgentTaskRunner`) — performs each task and returns trials through one small interface,
  `run_tasks(tasks) -> [AgentEvalTrial]`. The SDK ships runners for a plain async callable, a deployed
  agent over HTTP, and container/harness backends; you can supply your own. Scoring never depends on
  which runner produced a trial.
* **Trial** (`AgentEvalTrial`) — the durable record of one attempt: the agent's final `output`, the
  `evidence` it produced (an [ATIF](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trace of steps and tool calls or OTLP resource spans, final filesystem state, logs), a
  `status` (`completed` / `partial` / `failed`), and arbitrary metadata. The trial — not the runner — is what
  gets scored, and it can be re-scored offline later.
* **Metrics** (`Metric`) — each task's metrics score its trials through
  `compute_scores(input) -> MetricResult`. A metric can read the final output **and** the evidence, so
  it can grade the outcome (did it answer correctly?) or the trajectory (did it use the expected tool?).
* **Result** (`AgentEvaluator` → `AgentEvalResult`) — the evaluator orchestrates the run and returns
  the trials, per-trial scores, and an aggregated `summary`. Call `result.persist()` and it also
  writes a run **bundle**: `run.json`, `tasks.jsonl`, `trials.jsonl`, `scores.jsonl`, `summary.json`,
  `metadata.json`, and a browsable `report.html`.

## A minimal example

```python
import asyncio

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask
from nemo_evaluator_sdk.metrics.exact_match import ExactMatchMetric


async def my_agent(task: AgentEvalTask) -> str:
    return "Paris"


async def main() -> None:
    task = AgentEvalTask(
        id="capital-france",
        intent="Name the capital of France.",
        inputs={"instruction": "What is the capital of France?"},
        reference={"expected": "Paris"},  # grader-only; never shown to the agent
        metrics=[
            ExactMatchMetric(
                reference="{{item.reference.expected}}",
                candidate="{{sample.output_text}}",
            )
        ],
    )

    result = await AgentEvaluator().run(
        tasks=[task],
        target=CallableAgentTaskRunner(my_agent),
    )
    print(result.summary)


asyncio.run(main())
```

## Key properties

* **Runner-agnostic scoring.** The scorer only ever sees an `AgentEvalTrial`, so the same tasks and
  metrics score any runner's output — or previously stored trials re-scored offline.
* **Per-task metrics.** Metrics are attached to each task, not once for the whole run, so a suite can
  grade heterogeneous work (docs, Q\&A, tests, net-new code) each with the checks that fit it.
* **Score by component.** A single run can score at several levels:
  * the **task outcome** — the final answer;
  * the **trajectory** — how the agent worked (its tool use and steps);
  * **views** — named roll-ups you define on a task that combine two or more of its metric outputs into one reported score (for example, averaging an accuracy metric and a tool-use metric into a single `quality` score);
  * the **run-level aggregate** — results also roll up across the whole run.
* **Runs locally or durably.** A local run can produce the full bundle,
  including `report.html`, without Helix services. Use the plugin's
  `agent-evaluate` job when the task suite needs durable platform execution and
  persisted result metadata.
* **Measurement, not decisions.** The evaluator produces scores, aggregates, and provenance — it
  doesn't decide pass/fail, gate a release, or compare runs. Those decisions belong to whatever
  consumes the results.

## Targets

An evaluation target is what performs the tasks:

* a **`Model`** — a chat/completions endpoint;
* a deployed **`Agent`** reachable over HTTP — a `GenericAgent` (any JSON endpoint) or a
  `NemoAgentToolkitAgent`;
* a custom **runner** (`AgentTaskRunner`) — anything that can turn tasks into trials.

See [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
for how to describe an HTTP agent target.

## In this section

* [Quickstart](/documentation/evaluate-models/agent-eval/quickstart) — evaluate an agent end to end in
  a local process, no services required.
* [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
  — point a run at an agent behind an HTTP endpoint.
* [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) — run an
  existing Harbor dataset in Docker and score its verifier reward.
* [Run a Fabric Agent on Harbor](/documentation/evaluate-models/agent-eval/harbor-fabric-agent) —
  drive a Fabric harness as the agent inside a Harbor task container.
* [Evaluate a NeMo Gym Environment](/documentation/evaluate-models/agent-eval/gym-runner) — collect
  and score Gym rollouts locally or as a platform job.
* [Run a Custom Gym Environment](/documentation/evaluate-models/agent-eval/gym-custom-environment)
  — package a Gym environment as a FileSet and evaluate it in OpenSandbox.
* [Configure Sandboxed Gym](/documentation/evaluate-models/agent-eval/gym-sandbox-configuration) —
  deployment settings that govern sandboxed Gym execution.
* [Evaluate with a NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner) —
  drive a supported agent harness and capture its trajectory.
* [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) — score the
  trajectory (not just the answer) and roll metrics into a named view.
* [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners) — reference for
  what a run can point at.
* [Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) — reference for the
  `Metric` protocol.
* [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) — what a run returns and
  the on-disk bundle it writes.

## Related

#### [Evaluation Approaches](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation)

#### [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)

#### [Evaluation Metrics](/documentation/evaluate-models/metrics)

> The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics.

## Docs

- [Agent Evaluation Quickstart](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/quickstart.md): Evaluate a simple agent end to end with the NeMo Evaluator SDK — define tasks and a metric, run locally, and read the scores and HTML report. No Helix services or API keys required.
- [Evaluate a Deployed Agent over HTTP](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/evaluate-deployed-agent.md): Point NeMo Evaluator at an agent reachable over HTTP by describing it as a GenericAgent target — the request to send and where the answer is in the response — then score it exactly like the quickstart.
- [Evaluate a Harbor Task Suite](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/harbor-runner.md): Run an existing Harbor task dataset through NeMo Evaluator's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits named rewards, which the SDK scores and reports like any other agent-eval run.
- [Run a NeMo Fabric Agent inside Harbor](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/harbor-fabric-agent.md): Evaluate a NeMo Fabric harness — here LangChain deepagents on a Nemotron model — inside Harbor's Docker sandbox, with the model API key delivered through env_secrets rather than the agent config.
- [Evaluate a NeMo Gym Environment](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/gym-runner.md): Run an existing NeMo Gym environment through NeMo Evaluator's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward.
- [Run a Custom NeMo Gym Environment](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/gym-custom-environment.md): Package a custom NeMo Gym environment as a FileSet and run it as a sandboxed agent-evaluate job.
- [Configure Sandboxed Gym for Evaluator](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/gym-sandbox-configuration.md): Configure OpenSandbox, shared storage, images, and network policy for sandboxed Gym agent-evaluate jobs.
- [Evaluate with a NeMo Fabric Harness](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/fabric-runner.md): Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on.
- [Score by Component](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/score-by-component.md): Go beyond the final answer: add a metric that scores the agent's trajectory (its tool use), attach it alongside an outcome metric, and roll both into one named view — all runnable locally with no services.
- [Targets and Runners](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/targets-and-runners.md): Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, Gym, or your own) — with each target's key fields and when to use it.
- [Writing Metrics](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/writing-metrics.md): Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.
- [Reading Results](https://docs.nvidia.com/nemo-helix/v0.7.0/documentation/evaluate-models/agent-eval/reading-results.md): Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle you get by calling persist(), including the browsable HTML report.