> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Evaluate Agents & Models

> Score agents and models in NeMo Helix — dataset-driven metrics over labeled data and task-driven agent evaluation — with the same scoring interface across local and platform execution.

NeMo Evaluator scores how well your models, RAG pipelines, and agents actually do their job. You
define **what** to score and **how** to score it once, then run it where you need to — in a local
Python process for fast iteration, or as a durable platform job for production and regression
tracking. Scoring covers deterministic and similarity checks, LLM-as-a-judge, and specialized
metrics for RAG and agentic behavior.

**Tutorials**

**Open Source SDK**

---

## Two shapes of evaluation

Evaluation comes in two shapes, told apart by what produces the thing you score:

* **Dataset-driven evaluation** — score a model or pipeline over a fixed dataset. Metrics compare each
  output against a reference, and the same metric set applies to every row. Reach for it for model
  quality checks, RAG pipelines, and regression testing against a labeled set. Entry point:
  [Evaluation Metrics](/documentation/evaluate-models/metrics).
* **Task-driven evaluation (agent evaluation)** — score an agent that performs *tasks*. Each task
  produces a *trial* — the agent's final output **and** how it got there (its trajectory, tool calls,
  and other evidence) — and carries its own metrics, so one suite can grade heterogeneous work. Reach
  for it when the system under test *acts* and the process matters as much as the outcome. Entry
  point: [Agent Evaluation](/documentation/evaluate-models/agent-eval).

> **Note**
>
> **Metrics are the shared scorer, not a third mode.** The same deterministic and LLM-as-a-judge
> scorers work in either shape — what changes is the input being scored, not the scorer.

For how the two shapes differ in detail — the evidence each exposes, an at-a-glance comparison, and
guidance on choosing — see
[Dataset-Driven vs Task-Driven Evaluation](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation).

---

## How evaluation runs

Evaluator separates **evaluation definition** from **execution**. You define the metric, the dataset,
and the runtime configuration once, then choose where that definition runs. The definition is
portable; only the caller changes.

> **Note**
>
> The snippets below are conceptual. For runnable examples see the
> [tutorials](/documentation/evaluate-models/tutorials) and
> [SDK Resources](/documentation/evaluate-models/sdk-resources).

### 1. Define the evaluation

Use the `nemo_evaluator_sdk` package to define your metric, dataset rows, and runtime configuration —
these objects are context-agnostic and identical across every execution mode below:

```python
from nemo_evaluator_sdk import RunConfig, ExactMatchMetric

# How to score: compare the output to the expected reference.
metric = ExactMatchMetric(
    reference="{{item.expected}}",
    candidate="{{item.output}}",
)

# What to score: the rows (inline here; a fileset or local path also works).
dataset = [
    {"expected": "Paris", "output": "Paris"},
    {"expected": "Berlin", "output": "Munich"},
]

# Runtime settings: sample limits and parallelism.
config = RunConfig(limit_samples=100, parallelism=8)
```

### 2. Run it — three dataset-driven modes

The same `metric`, `dataset`, and `config` run in two places. What changes is the **caller** — a
bare SDK evaluator for local iteration, or the platform's `client.evaluator` resource for durable
execution.

| Mode           | Caller                               | Call                                                                    | Best for                                                         |
| -------------- | ------------------------------------ | ----------------------------------------------------------------------- | ---------------------------------------------------------------- |
| **Local SDK**  | `Evaluator()` (`nemo_evaluator_sdk`) | `await evaluator.run(metrics=[metric], dataset=dataset, config=config)` | Fast in-process iteration with no platform services.             |
| **Remote job** | `client.evaluator`                   | `evaluator.submit(metric=metric, dataset=dataset, config=config)`       | Durable, monitored platform jobs for production and regressions. |

The platform caller is mounted on a `NeMoPlatform` client:

```python
import os

from nemo_evaluator.sdk import Evaluator
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
evaluator: Evaluator = client.evaluator

# Production evaluation as a durable platform job.
job = evaluator.submit(metric=metric, dataset=dataset, config=config)
job.wait_until_done()
result = job.get_result()
```

> **Note**
>
> **Agent evaluation has local interfaces and durable platform interfaces.**
>
> * Use `await AgentEvaluator().run(tasks=..., target=...)` for local task-driven SDK runs
>   that do not require running nemo-platform.
> * Use the Evaluator plugin's `uv run nemo evaluator agent-evaluate submit` job for durable runs with inline
>   tasks or stored tasksets. The high-level `client.evaluator.submit` interface
>   above remains dataset-driven only.

### What the platform adds

When you move from the local SDK to the platform caller (`client.evaluator`), the definition stays
the same and execution gains platform capabilities:

| Capability         | Local SDK                            | Platform (`client.evaluator`)                                                                                                          |
| ------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Execution**      | Local in-process run                 | Durable platform jobs                                                                                                                  |
| **Inference**      | Direct model or agent endpoint calls | The same, and can route through NeMo Helix [Inference Gateway](/documentation/models-and-inference) and platform-managed endpoints     |
| **Datasets**       | Inline rows and local files          | Inline rows, local paths resolved at submission time, and NeMo Helix [Filesets](/documentation/get-started/core-concepts/manage-files) |
| **Results**        | Returned in memory                   | Platform artifact storage with typed result download                                                                                   |
| **Authentication** | Local environment variables          | Local env vars for local runs; NeMo Helix [Secrets](/documentation/get-started/core-concepts/manage-secrets) for remote jobs           |

### Live vs. jobs, online vs. offline

Two more distinctions cut across the modes above:

* **Live (synchronous) vs. jobs (asynchronous).** `run()` returns results immediately — best for fast
  iteration, metric development, and small payloads. `submit()` creates a durable job you monitor and
  fetch results from — best for production workloads, larger datasets, and recurring regression
  checks.
* **Offline vs. online.** *Offline* scores dataset rows that already contain outputs. *Online*
  generates outputs from a `target` model or agent as part of the evaluation, then scores them — pass
  `target` (and a `prompt_template`) to any of the modes above.

---

## Tutorials

After [setting up a local instance of the platform](/documentation/get-started), use these
step-by-step guides to run common evaluations.

#### [Run an LLM Judge Eval](/documentation/evaluate-models/tutorials/run-llm-as-a-judge-evaluation)

Evaluate a fine-tuned model using the LLM-as-a-judge metric with a custom dataset.

custom-dataset

#### [Define and Run Custom Python Metrics](/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics)

Write a domain-specific Python metric, test it locally, and run it through the Evaluator service.

custom-metric

---

## A recommended path

Most teams get the best results by starting metric-first and scaling up:

1. **Develop and validate your metrics.** Start with [Metrics](/documentation/evaluate-models/metrics)
   to define how quality should be scored for your use case, and iterate quickly with local `run()`
   over small inline datasets.
2. **Scale to durable jobs.** When your metrics are validated, run them with `submit()` on larger
   datasets, using [Filesets](/documentation/get-started/core-concepts/manage-files) for
   production-scale inputs.
3. **Add agent evaluation where behavior matters.** For systems that act — tool use, multi-step
   reasoning — layer in [Agent Evaluation](/documentation/evaluate-models/agent-eval) to score the
   trajectory alongside the outcome.

---

## Where to go next

* **Define scoring:** [Evaluation Metrics](/documentation/evaluate-models/metrics) — built-in and
  custom metrics, dataset sources, and how scores aggregate.
* **Evaluate agents:** [Agent Evaluation](/documentation/evaluate-models/agent-eval) — the task-driven
  model, targets, and runners.
* **Compare runs:** [Experiments](/documentation/evaluate-models/experiments) — roll evaluation runs up
  into one leaderboard ranked by cost, latency, and evaluator scores.
* **Run it in code:** [SDK Resources](/documentation/evaluate-models/sdk-resources) — the `run()` and
  `submit()` calling surface.
* **Full API:** [Evaluator API Reference](/documentation/reference/api-reference).

> Score agents and models in NeMo Helix — dataset-driven metrics over labeled data and task-driven agent evaluation — with the same scoring interface across local and platform execution.

## Docs

- [Dataset-Driven vs Task-Driven Evaluation](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation.md): Two complementary shapes of evaluation in NeMo Evaluator — scoring model outputs over a dataset, versus scoring an agent that performs tasks — and how to choose between them.
- [Agent Evaluation](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval.md): The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics.
- [Agent Evaluation Quickstart](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/quickstart.md): Evaluate a simple agent end to end with the NeMo Evaluator SDK — define tasks and a metric, run locally, and read the scores and HTML report. No platform services or API keys required.
- [Evaluate a Deployed Agent over HTTP](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/evaluate-deployed-agent.md): Point NeMo Evaluator at an agent reachable over HTTP by describing it as a GenericAgent target — the request to send and where the answer is in the response — then score it exactly like the quickstart.
- [Evaluate a Harbor Task Suite](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/harbor-runner.md): Run an existing Harbor task dataset through NeMo Evaluator's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits a reward, which the SDK scores and reports like any other agent-eval run.
- [Score by Component](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/score-by-component.md): Go beyond the final answer: add a metric that scores the agent's trajectory (its tool use), attach it alongside an outcome metric, and roll both into one named view — all runnable locally with no services.
- [Targets and Runners](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/targets-and-runners.md): Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, or your own) — with each target's key fields and when to use it.
- [Writing Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/writing-metrics.md): Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.
- [Reading Results](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/agent-eval/reading-results.md): Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle you get by calling persist(), including the browsable HTML report.
- [Evaluation Tutorials](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/tutorials.md): 
- [Evaluate Response Quality with LLM-as-a-Judge](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/tutorials/run-llm-as-a-judge-evaluation.md): 
- [Define and Run Custom Python Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics.md): 
- [Experiments](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/experiments.md): Compare evaluation runs on a NeMo Experiments leaderboard with cost, latency, token, and evaluator rollups computed from NeMo Intake telemetry.
- [Evaluator NeMo Helix SDK Resources](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/sdk-resources.md): 
- [Manage Tasks & Tasksets](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/manage-tasks-tasksets.md): 
- [Evaluation Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics.md): 
- [Manage Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/manage-metrics.md): 
- [Evaluate with LLM-as-a-Judge](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/llm-as-a-judge.md): 
- [RAG Evaluation Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/rag-metrics.md): 
- [Similarity Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/similarity-metrics.md): 
- [Agentic Evaluation Metrics](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/agentic-metrics.md): 
- [Bring Your Own Metric](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/bring-your-own-metric.md): 
- [Agent Configuration](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/agent-configuration.md): 
- [Model Configuration](https://docs.nvidia.com/nemo-helix/v0.5.0/documentation/evaluate-models/metrics/model-configuration.md):