Evaluate Agents & Models

NeMo Evaluator scores how well your models, RAG pipelines, and agents actually do their job. You define what to score and how to score it once, then run it where you need to — in a local Python process for fast iteration, or as a durable platform job for production and regression tracking. Scoring covers deterministic and similarity checks, LLM-as-a-judge, and specialized metrics for RAG and agentic behavior.

Two shapes of evaluation

Evaluation comes in two shapes, told apart by what produces the thing you score:

• Dataset-driven evaluation — score a model or pipeline over a fixed dataset. Metrics compare each output against a reference, and the same metric set applies to every row. Reach for it for model quality checks, RAG pipelines, and regression testing against a labeled set. Entry point: Evaluation Metrics.
• Task-driven evaluation (agent evaluation) — score an agent that performs tasks. Each task produces a trial — the agent’s final output and how it got there (its trajectory, tool calls, and other evidence) — and carries its own metrics, so one suite can grade heterogeneous work. Reach for it when the system under test acts and the process matters as much as the outcome. Entry point: Agent Evaluation.

For how the two shapes differ in detail — the evidence each exposes, an at-a-glance comparison, and guidance on choosing — see Dataset-Driven vs Task-Driven Evaluation.

How evaluation runs

Evaluator separates evaluation definition from execution. You define the metric, the dataset, and the runtime configuration once, then choose where that definition runs. The definition is portable; only the caller changes.

1. Define the evaluation

Use the nemo_evaluator_sdk package to define your metric, dataset rows, and runtime configuration — these objects are context-agnostic and identical across every execution mode below:

1
from nemo_evaluator_sdk import RunConfig, ExactMatchMetric
2
3
# How to score: compare the output to the expected reference.
4
metric = ExactMatchMetric(
5
    reference="{{item.expected}}",
6
    candidate="{{item.output}}",
7
)
8
9
# What to score: the rows (inline here; a fileset or local path also works).
10
dataset = [
11
    {"expected": "Paris", "output": "Paris"},
12
    {"expected": "Berlin", "output": "Munich"},
13
]
14
15
# Runtime settings: sample limits and parallelism.
16
config = RunConfig(limit_samples=100, parallelism=8)

2. Run it — three dataset-driven modes

The same metric, dataset, and config run in three places. What changes is the caller — a bare SDK evaluator for local iteration, or the platform’s client.evaluator resource for plugin-backed and durable execution.

The platform caller is mounted on a NeMoPlatform client:

1
import os
2
3
from nemo_evaluator.sdk import Evaluator
4
from nemo_platform import NeMoPlatform
5
6
client = NeMoPlatform(
7
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
8
    workspace="default",
9
)
10
evaluator: Evaluator = client.evaluator
11
12
# Fast local iteration through the plugin runtime.
13
local_result = evaluator.run(metric=metric, dataset=dataset, config=config)
14
15
# Production evaluation as a durable platform job.
16
job = evaluator.submit(metric=metric, dataset=dataset, config=config)
17
job.wait_until_done()
18
result = job.get_result()

What the platform adds

When you move from the local SDK to the platform caller (client.evaluator), the definition stays the same and execution gains platform capabilities:

Live vs. jobs, online vs. offline

Two more distinctions cut across the modes above:

• Live (synchronous) vs. jobs (asynchronous). run() returns results immediately — best for fast iteration, metric development, and small payloads. submit() creates a durable job you monitor and fetch results from — best for production workloads, larger datasets, and recurring regression checks.
• Offline vs. online. Offline scores dataset rows that already contain outputs. Online generates outputs from a target model or agent as part of the evaluation, then scores them — pass target (and a prompt_template) to any of the modes above.

Tutorials

After setting up a local instance of the platform, use these step-by-step guides to run common evaluations.

A recommended path

Most teams get the best results by starting metric-first and scaling up:

1. Develop and validate your metrics. Start with Metrics to define how quality should be scored for your use case, and iterate quickly with local run() over small inline datasets.
2. Scale to durable jobs. When your metrics are validated, run them with submit() on larger datasets, using Filesets for production-scale inputs.
3. Add agent evaluation where behavior matters. For systems that act — tool use, multi-step reasoning — layer in Agent Evaluation to score the trajectory alongside the outcome.

Where to go next

• Define scoring: Evaluation Metrics — built-in and custom metrics, dataset sources, and how scores aggregate.
• Evaluate agents: Agent Evaluation — the task-driven model, targets, and runners.
• Run it in code: SDK Resources — the run() and submit() calling surface.
• Full API: Evaluator API Reference.