> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Evaluate Agents & Models

> Score agents and models in NeMo Helix — dataset-driven metrics over labeled data and task-driven agent evaluation — with the same scoring interface across local and platform execution.

Upgrading an existing deployment? Follow the [upgrade guidance](/documentation/evaluate-models/upgrading-from-evaluator)
to preserve configuration, Studio feature flags, and integrations.

NeMo Helix Evals scores how well your models, RAG pipelines, and agents actually do their job. You
define **what** to score and **how** to score it once, then run it where you need to — in a local
Python process for fast iteration, or as a durable platform job for production and regression
tracking. Scoring covers deterministic and similarity checks, LLM-as-a-judge, and specialized
metrics for RAG and agentic behavior.

**Tutorials**

**Open Source SDK**

---

## Three shapes of evaluation

Evaluation comes in three shapes, told apart by what produces the thing you score:

* **Dataset-driven evaluation** — score a model or pipeline over a fixed dataset. Metrics compare each
  output against a reference, and the same metric set applies to every row. Reach for it for model
  quality checks, RAG pipelines, and regression testing against a labeled set. Entry point:
  [Evaluation Metrics](/documentation/evaluate-models/metrics).
* **Task-driven evaluation (agent evaluation)** — score an agent that performs *tasks*. Each task
  produces a *trial* — the agent's final output **and** how it got there (its trajectory, tool calls,
  and other evidence) — and carries its own metrics, so one suite can grade heterogeneous work. Reach
  for it when the system under test *acts* and the process matters as much as the outcome. Entry
  point: [Agent Evaluation](/documentation/evaluate-models/agent-eval).
* **Retrieval-driven evaluation** — score how well a retriever ranks documents. A `Retrieval` target
  ranks a BEIR corpus against queries and relevance judgments, and retrieval metrics score the
  rankings. Reach for it when tuning embeddings or a reranker. Entry point: the `retrieve-eval` job,
  described in
  [Evaluation Approaches](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation).

> **Note**
>
> **Metrics are the shared scorer, not a shape of their own.** The same deterministic and
> LLM-as-a-judge scorers work in any of the three — what changes is the input being scored, not the
> scorer.

For how the two shapes differ in detail — the evidence each exposes, an at-a-glance comparison, and
guidance on choosing — see
[Evaluation Approaches](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation).

---

## How evaluation runs

Evaluator separates **evaluation definition** from **execution**. You define the metric, the dataset,
and the runtime configuration once, then choose where that definition runs. The definition is
portable; only the caller changes.

> **Note**
>
> The snippets below are conceptual. For runnable examples see the
> [tutorials](/documentation/evaluate-models/tutorials) and
> [SDK Resources](/documentation/evaluate-models/sdk-resources).

### 1. Define the evaluation

Use the `nhx_evals_sdk` package to define your metric, dataset rows, and runtime configuration —
these objects are context-agnostic and identical across every execution mode below:

```python
from nhx_evals_sdk import RunConfig, ExactMatchMetric

# How to score: compare the output to the expected reference.
metric = ExactMatchMetric(
    reference="{{item.expected}}",
    candidate="{{item.output}}",
)

# What to score: the rows (inline here; a fileset or local path also works).
dataset = [
    {"expected": "Paris", "output": "Paris"},
    {"expected": "Berlin", "output": "Munich"},
]

# Runtime settings: sample limits and parallelism.
config = RunConfig(limit_samples=100, parallelism=8)
```

### 2. Run it — two dataset-driven modes

The same `metric`, `dataset`, and `config` run in two places. What changes is the **caller** — a
bare SDK evaluator for local iteration, or the platform's `Evaluator` resource for durable
execution.

| Mode           | Caller                          | Call                                                                    | Best for                                                         |
| -------------- | ------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------- |
| **Local SDK**  | `Evaluator()` (`nhx_evals_sdk`) | `await evaluator.run(metrics=[metric], dataset=dataset, config=config)` | Fast in-process iteration with no Helix services.                |
| **Remote job** | `Evaluator.from_client(client)` | `evaluator.submit(metric=metric, dataset=dataset, config=config)`       | Durable, monitored platform jobs for production and regressions. |

The platform caller is built from a `NemoClient`:

```python
from nemo_evals.sdk import Evaluator
from nemo_helix_plugin.client.client import NemoClient

client = NemoClient.from_config()
evaluator = Evaluator.from_client(client)

# Production evaluation as a durable platform job.
job = evaluator.submit(metric=metric, dataset=dataset, config=config)
job.wait_until_done()
result = job.get_result()
```

> **Note**
>
> **Agent evaluation has local interfaces and durable platform interfaces.**
>
> * Use `await AgentEvaluator().run(tasks=..., target=...)` for local task-driven SDK runs
>   that do not require running nemo-helix.
> * Use the Evals plugin's `uv run nemo evals agent-evaluate` job for durable runs with inline
>   tasks or stored tasksets. A locally validated `GymAgentTaskRunner` can also be submitted with
>   `evaluator.submit(tasks=TasksetRef(...), target=runner)`; other runner targets use an
>   agent-evaluate spec.

### What the platform adds

When you move from the local SDK to the platform caller (`Evaluator.from_client(client)`), the definition stays
the same and execution gains platform capabilities:

| Capability         | Local SDK                            | Platform (`Evaluator.from_client(client)`)                                                                                             |
| ------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Execution**      | Local in-process run                 | Durable platform jobs                                                                                                                  |
| **Inference**      | Direct model or agent endpoint calls | The same, and can route through NeMo Helix [Inference Gateway](/documentation/models-and-inference) and platform-managed endpoints     |
| **Datasets**       | Inline rows and local files          | Inline rows, local paths resolved at submission time, and NeMo Helix [Filesets](/documentation/get-started/core-concepts/manage-files) |
| **Results**        | Returned in memory                   | Platform artifact storage with typed result download                                                                                   |
| **Authentication** | Local environment variables          | Local env vars for local runs; NeMo Helix [Secrets](/documentation/get-started/core-concepts/manage-secrets) for remote jobs           |

### Live vs. jobs, online vs. offline

Two more distinctions cut across the modes above:

* **Live (synchronous) vs. jobs (asynchronous).** `run()` returns results immediately — best for fast
  iteration, metric development, and small payloads. `submit()` creates a durable job you monitor and
  fetch results from — best for production workloads, larger datasets, and recurring regression
  checks.
* **Offline vs. online.** *Offline* scores dataset rows that already contain outputs. *Online*
  generates outputs from a `target` model or agent as part of the evaluation, then scores them — pass
  `target` (and a `prompt_template`) to any of the modes above.

---

## Tutorials

After [setting up a local instance of the platform](/documentation/get-started), use these
step-by-step guides to run common evaluations.

#### [Run an LLM Judge Eval](/documentation/evaluate-models/tutorials/run-llm-as-a-judge-evaluation)

Evaluate a fine-tuned model using the LLM-as-a-judge metric with a custom dataset.

custom-dataset

#### [Define and Run Custom Python Metrics](/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics)

Write a domain-specific Python metric, test it locally, and run it through the Evals service.

custom-metric

---

## A recommended path

Most teams get the best results by starting metric-first and scaling up:

1. **Develop and validate your metrics.** Start with [Metrics](/documentation/evaluate-models/metrics)
   to define how quality should be scored for your use case, and iterate quickly with local `run()`
   over small inline datasets.
2. **Scale to durable jobs.** When your metrics are validated, run them with `submit()` on larger
   datasets, using [Filesets](/documentation/get-started/core-concepts/manage-files) for
   production-scale inputs.
3. **Add agent evaluation where behavior matters.** For systems that act — tool use, multi-step
   reasoning — layer in [Agent Evaluation](/documentation/evaluate-models/agent-eval) to score the
   trajectory alongside the outcome.

---

## Where to go next

* **Define scoring:** [Evaluation Metrics](/documentation/evaluate-models/metrics) — built-in and
  custom metrics, dataset sources, and how scores aggregate.
* **Evaluate agents:** [Agent Evaluation](/documentation/evaluate-models/agent-eval) — the task-driven
  model, targets, and runners.
* **Compare runs:** [Experiments](/documentation/evaluate-models/experiments) — roll evaluation runs up
  into one leaderboard ranked by cost, latency, and evaluator scores.
* **Run it in code:** [SDK Resources](/documentation/evaluate-models/sdk-resources) — the `run()` and
  `submit()` calling surface.
* **Full API:** [Evals API Reference](/documentation/reference/api-reference).

> Score agents and models in NeMo Helix — dataset-driven metrics over labeled data and task-driven agent evaluation — with the same scoring interface across local and platform execution.

## Docs

- [Upgrading from NeMo Evaluator](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/upgrading-from-evaluator.md): Migrate deployment overrides and integrations to NeMo Helix Evals.
- [Evaluation Approaches](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation.md): Choose dataset-driven, task-driven, or retrieval-driven evaluation based on the unit being scored.
- [Agent Evaluation](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval.md): The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics.
- [Agent Evaluation Quickstart](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/quickstart.md): Evaluate a simple agent end to end with the NeMo Helix Evals SDK — define tasks and a metric, run locally, and read the scores and HTML report. No Helix services or API keys required.
- [Evaluate a Deployed Agent over HTTP](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/evaluate-deployed-agent.md): Point NeMo Helix Evals at an agent reachable over HTTP by describing it as a GenericAgent target — the request to send and where the answer is in the response — then score it exactly like the quickstart.
- [Evaluate a Harbor Task Suite](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/harbor-runner.md): Run an existing Harbor task dataset through NeMo Helix Evals's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits named rewards, which the SDK scores and reports like any other agent-eval run.
- [Run a NeMo Fabric Agent inside Harbor](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/harbor-fabric-agent.md): Evaluate a NeMo Fabric harness — here LangChain deepagents on a Nemotron model — inside Harbor's Docker sandbox, with the model API key delivered through env_secrets rather than the agent config.
- [Evaluate a NeMo Gym Environment](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/gym-runner.md): Run an existing NeMo Gym environment through NeMo Helix Evals's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward.
- [Run a Custom NeMo Gym Environment](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/gym-custom-environment.md): Package a custom NeMo Gym environment as a FileSet and run it as a sandboxed agent-evaluate job.
- [Configure Sandboxed Gym for Evaluator](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/gym-sandbox-configuration.md): Configure OpenSandbox, shared storage, images, and network policy for sandboxed Gym agent-evaluate jobs.
- [Evaluate with a NeMo Fabric Harness](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/fabric-runner.md): Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on.
- [Score by Component](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/score-by-component.md): Go beyond the final answer: add a metric that scores the agent's trajectory (its tool use), attach it alongside an outcome metric, and roll both into one named view — all runnable locally with no services.
- [Targets and Runners](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/targets-and-runners.md): Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, Gym, or your own) — with each target's key fields and when to use it.
- [Writing Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/writing-metrics.md): Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.
- [Reading Results](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/agent-eval/reading-results.md): Reference for what a run returns — the in-memory AgentEvalResult (summary, per-metric scores, trials, run id) — and the on-disk run bundle you get by calling persist(), including the browsable HTML report.
- [Evaluation Tutorials](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/tutorials.md): 
- [Evaluate Response Quality with LLM-as-a-Judge](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/tutorials/run-llm-as-a-judge-evaluation.md): 
- [Define and Run Custom Python Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics.md): 
- [Experiments](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/experiments.md): Compare evaluation runs on a NeMo Experiments leaderboard with cost, latency, token, and evaluator rollups computed from NeMo Intake telemetry.
- [Evaluator NeMo Helix SDK Resources](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/sdk-resources.md): 
- [Manage Tasks & Tasksets](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/manage-tasks-tasksets.md): 
- [Evaluation Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics.md): 
- [Manage Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/manage-metrics.md): 
- [Evaluate with LLM-as-a-Judge](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/llm-as-a-judge.md): 
- [RAG Evaluation Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/rag-metrics.md): 
- [Similarity Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/similarity-metrics.md): 
- [Agentic Evaluation Metrics](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/agentic-metrics.md): 
- [Bring Your Own Metric](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/bring-your-own-metric.md): 
- [Agent Configuration](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/agent-configuration.md): 
- [Model Configuration](https://docs.nvidia.com/nemo-helix/latest/documentation/evaluate-models/metrics/model-configuration.md):