Agent Evaluation

Agent evaluation is the task-driven shape of NeMo Evaluator (see Dataset-Driven vs Task-Driven Evaluation for how it compares to metrics). You define tasks, an agent performs each one, and you score the resulting trial — not just whether the final answer is right, but how the agent got there. Each task carries its own metrics, so a single suite can grade heterogeneous work.

The model

One run flows through five pieces:

Task → (runner) → Trial → (metrics) → Scores → Result

• Task (AgentEvalTask) — the unit of work. Its fields:
  • intent — a human-readable description of the goal; metadata for the suite’s authors, not shown to the agent.
  • inputs — the instruction and anything the agent starts from (what the agent actually sees).
  • reference (optional) — grader-only, held-out ground truth the agent never sees.
  • metrics — the scorers for this task.
  • views (optional) — combine several of the task’s metric outputs into one named, reported score (see Score by component under Key properties).
• Runner (AgentTaskRunner) — performs each task and returns trials through one small interface, run_tasks(tasks) -> [AgentEvalTrial]. The SDK ships runners for a plain async callable, a deployed agent over HTTP, and container/harness backends; you can supply your own. Scoring never depends on which runner produced a trial.
• Trial (AgentEvalTrial) — the durable record of one attempt: the agent’s final output, the evidence it produced (an ATIF trace of steps and tool calls or OTLP resource spans, final filesystem state, logs), a status (completed / partial / failed), and arbitrary metadata. The trial — not the runner — is what gets scored, and it can be re-scored offline later.
• Metrics (Metric) — each task’s metrics score its trials through compute_scores(input) -> MetricResult. A metric can read the final output and the evidence, so it can grade the outcome (did it answer correctly?) or the trajectory (did it use the expected tool?).
• Result (AgentEvaluator → AgentEvalResult) — the evaluator orchestrates the run and returns the trials, per-trial scores, and an aggregated summary. Point it at an output directory and it also writes a run bundle: run.json, trials.jsonl, scores.jsonl, summary.json, and a browsable report.html.

A minimal example

1
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
2
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask
3
4
task = AgentEvalTask(
5
    id="capital-france",
6
    intent="Name the capital of France.",
7
    inputs={"instruction": "What is the capital of France?"},
8
    reference={"expected": "Paris"},   # grader-only; never shown to the agent
9
    metrics=[OutcomeMetric()],         # this task's own scorer(s)
10
)
11
12
# target is a Model, a deployed Agent (HTTP), or any AgentTaskRunner.
13
result = await AgentEvaluator().run(tasks=[task], target=my_agent)
14
print(result.summary)

Key properties

• Runner-agnostic scoring. The scorer only ever sees an AgentEvalTrial, so the same tasks and metrics score any runner’s output — or previously stored trials re-scored offline.
• Per-task metrics. Metrics are attached to each task, not once for the whole run, so a suite can grade heterogeneous work (docs, Q&A, tests, net-new code) each with the checks that fit it.
• Score by component. A single run can score at several levels:
  • the task outcome — the final answer;
  • the trajectory — how the agent worked (its tool use and steps);
  • views — named roll-ups you define on a task that combine two or more of its metric outputs into one reported score (for example, averaging an accuracy metric and a tool-use metric into a single quality score);
  • the run-level aggregate — results also roll up across the whole run.
• Runs locally or durably. A local run can produce the full bundle, including report.html, without platform services. Use the plugin’s agent-evaluate job when the task suite needs durable platform execution and persisted result metadata.
• Measurement, not decisions. The evaluator produces scores, aggregates, and provenance — it doesn’t decide pass/fail, gate a release, or compare runs. Those decisions belong to whatever consumes the results.

Targets

An evaluation target is what performs the tasks:

• a Model — a chat/completions endpoint;
• a deployed Agent reachable over HTTP — a GenericAgent (any JSON endpoint) or a NemoAgentToolkitAgent;
• a custom runner (AgentTaskRunner) — anything that can turn tasks into trials.

See Evaluate a Deployed Agent over HTTP for how to describe an HTTP agent target.

In this section

• Quickstart — evaluate an agent end to end in a local process, no services required.
• Evaluate a Deployed Agent over HTTP — point a run at an agent behind an HTTP endpoint.
• Evaluate a Harbor Task Suite — run an existing Harbor dataset in Docker and score its verifier reward.
• Score by Component — score the trajectory (not just the answer) and roll metrics into a named view.
• Targets and Runners — reference for what a run can point at.
• Writing Metrics — reference for the Metric protocol.
• Reading Results — what a run returns and the on-disk bundle it writes.

Related