> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Agent Evaluation Quickstart

> Evaluate a simple agent end to end with the NeMo Evaluator SDK — define tasks and a metric, run locally, and read the scores and HTML report. No platform services or API keys required.

In this quickstart you'll evaluate an agent on two tasks, score its answers, and produce a run
bundle with a browsable HTML report — all on your machine, with **no platform services and no API
keys**. It takes about five minutes.

For the concepts behind tasks, trials, runners, and metrics, see
[Agent Evaluation](/documentation/evaluate-models/agent-eval). This page is the hands-on version.

## Prerequisites

* Python 3.11–3.14
* The SDK:

```bash
pip install nemo-platform[nemo-evaluator-sdk]
```

Everything here runs in a single local Python process. You'll swap the stand-in agent for a real
model or deployed agent once you've seen the flow.

## 1. Define a metric

A metric scores one trial. It implements three things: a `type` (its name), an `output_spec()` (the
values it emits), and `compute_scores()` (the scoring logic). Here's a tiny one that scores `1.0`
when the agent's answer contains an expected keyword and `0.0` otherwise. It reads the agent's answer
from `input.candidate.output_text` and the task's grader-only `reference` from `input.row.data`.

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult


class KeywordMatchMetric:
    """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""

    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## 2. Define your tasks

Each `AgentEvalTask` is one unit of work: an `intent`, the `inputs` the agent acts on (here, an
`instruction`), a grader-only `reference` (held-out truth the agent never sees), and the `metrics`
that score **this** task. The `intent` is a human-readable note about the task's goal — metadata for
whoever reads or maintains the suite. It is **not** passed to the agent, which only ever sees
`inputs`. Because metrics live on the task, different tasks can use different scorers.

```python
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask


def make_tasks() -> list[AgentEvalTask]:
    return [
        AgentEvalTask(
            id="capital-france",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"},
            reference={"expected": "Paris"},
            metrics=[KeywordMatchMetric()],
        ),
        AgentEvalTask(
            id="capital-japan",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"},
            reference={"expected": "Tokyo"},
            metrics=[KeywordMatchMetric()],
        ),
    ]
```

## 3. Provide an agent

An agent is anything that turns a task into an answer. The simplest option is an async function; the
SDK provides a runner — `CallableAgentTaskRunner` — that wraps your function so it plugs into a run.
The runner hands your function the whole task and expects an answer back (here, a canned response
keyed by task `id`). We'll use a stand-in so the quickstart runs with no external services; later
you'll point the evaluation at a real model or a
[deployed agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) instead.

```python
async def my_agent(task: AgentEvalTask) -> str:
    # Stand-in agent — replace with a real model or deployed agent later.
    answers = {
        "capital-france": "The capital of France is Paris.",
        "capital-japan": "The capital of Japan is Tokyo.",
    }
    return answers[task.id]
```

## 4. Run the evaluation

`AgentEvaluator.run()` sends the tasks to the runner, collects the trials, scores them, and returns
an `AgentEvalResult`. Setting `output_dir` also writes a run bundle to disk. (`run()` is async, so it
lives inside an `async` function — see the full script below.)

```python
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig

result = await AgentEvaluator().run(
    tasks=make_tasks(),
    target=CallableAgentTaskRunner(my_agent),
    config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
)

for aggregate in result.summary.scores.scores:
    print(f"{aggregate.name}: {aggregate.mean}")
```

## 5. Read the results

`run()` returns an `AgentEvalResult` you can work with directly in Python:

* `result.summary` — aggregated scores per metric output (the mean you just printed), plus coverage
  counts (how many trials were scored, failed, or missing an output).
* `result.scores` — one entry per (task, trial, metric): the metric outputs and their status.
* `result.trials` — each trial: the agent's answer, the evidence it produced (trajectory, final
  state, logs), and its status (`completed` / `partial` / `failed`).
* `result.run_id` — a stable identifier for this run.

Because you set `output_dir`, the same data was also written to `./agent-eval-run/` as a run bundle:

| File             | Contents                                                                               |
| ---------------- | -------------------------------------------------------------------------------------- |
| `summary.json`   | aggregated scores per metric output (mean / min / max / std-dev / counts) and coverage |
| `scores.jsonl`   | one row per (task, trial, metric) — the metric outputs, status, and any diagnostics    |
| `trials.jsonl`   | one row per trial — the agent's output, its evidence, and status                       |
| `tasks.jsonl`    | the tasks that were evaluated                                                          |
| `run.json`       | the run manifest — the run id and a map of the artifact files                          |
| `benchmark.json` | benchmark-grouping metadata for the run                                                |
| `report.html`    | a browsable dashboard of the run — open it in a browser                                |

The in-memory result and the on-disk bundle hold the same information: use the result object for
programmatic follow-up, and the bundle (especially `report.html`) to inspect or share a run.

## Full script

```python
import asyncio

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult


class KeywordMatchMetric:
    """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""

    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])


def make_tasks() -> list[AgentEvalTask]:
    return [
        AgentEvalTask(
            id="capital-france",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"},
            reference={"expected": "Paris"},
            metrics=[KeywordMatchMetric()],
        ),
        AgentEvalTask(
            id="capital-japan",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"},
            reference={"expected": "Tokyo"},
            metrics=[KeywordMatchMetric()],
        ),
    ]


async def my_agent(task: AgentEvalTask) -> str:
    # Stand-in agent — replace with a real model or deployed agent later.
    answers = {
        "capital-france": "The capital of France is Paris.",
        "capital-japan": "The capital of Japan is Tokyo.",
    }
    return answers[task.id]


async def main() -> None:
    result = await AgentEvaluator().run(
        tasks=make_tasks(),
        target=CallableAgentTaskRunner(my_agent),
        config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")


asyncio.run(main())
```

Run it:

```bash
python quickstart.py
```

Expected output (both answers contain the expected keyword, so the mean is `1.0`):

```
keyword_match.score: 1.0
```

Open `./agent-eval-run/report.html` to see the run in a browser.

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)

* Swap the stand-in agent for a **real model or a deployed agent over HTTP**.
* Add a **trajectory** metric that scores *how* the agent worked (its tool use), not just the answer.
* Give a task **more than one metric**, or combine metric outputs into a named **view**.