> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Evaluation Approaches

> Choose dataset-driven, task-driven, or retrieval-driven evaluation based on the unit being scored.

NeMo Evaluator supports three complementary shapes of evaluation. They share the same scoring
foundation but differ in **what produces the thing you score** and **what you're allowed to look at**
when you score it.

* **Dataset-driven evaluation** — you have a dataset of homogeneously structured rows, a target
  generates an output for each row (or you supply outputs), and metrics score each output against a
  reference. This is the path
  behind [metrics](/documentation/evaluate-models/metrics).
* **Task-driven evaluation (agent evaluation)** — you have a set of *tasks*, an agent performs each
  task, and you score the resulting *trial* — the agent's final output **and** how it got there
  (its trajectory, tool calls, and other evidence).
* **Retrieval-driven evaluation** — you have a BEIR corpus, queries, and relevance judgments; a
  `Retrieval` target ranks corpus documents (embeddings, optionally a reranker) and ordinary
  retrieval metrics score those rankings.

Both shapes score with the **same `Metric` interface**. A deterministic/code scorer or an
LLM-as-a-judge scorer works in either one — the difference is the shape of the input being scored,
not the scorer.

## Dataset-driven evaluation

You start from a **dataset** — rows of inputs, usually with a reference/expected value. A **model**
(or pipeline) produces an output per row, either generated online at evaluation time or supplied
offline, and metrics score each output against its reference. The suite is uniform — the **same
metric set applies to every row**.

**Use it when** the thing under test is a model or pipeline you can run over a fixed dataset and you
care about output quality: model quality checks, RAG pipelines, and regression testing against a
labeled set.

Entry point: [Evaluation Metrics](/documentation/evaluate-models/metrics).

## Task-driven evaluation (agent evaluation)

You start from **tasks**. Each task gives an agent an instruction and any inputs it needs; the agent
performs the task through a **runner**, producing a **trial** — the agent's final output plus
evidence such as [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md)
steps and tool calls or OTLP resource spans, final filesystem state, and logs. Container-based runners
also provide an environment (and its final state) the agent works in. Metrics then score the trial.
Because trace evidence is part of the trial, you can score not
just *whether the answer is right* but *how the agent worked* — for example, whether it used the
expected tool.

Metrics are attached **per task**, not once for the whole run — each task carries its own scorers.
That lets a single evaluation grade **heterogeneous work**: a coding agent whose tasks span
documentation, Q\&A, unit tests, and net-new code can score each task type with the checks that fit
it — an LLM-as-a-judge for documentation clarity, a tests-pass check for code, an exact-match check
for a factual answer — all in one suite. Dataset-driven runs, by contrast, apply the same metric set
to every row.

**Use it when** the thing under test is an agent that *acts*: it calls tools, takes multiple steps,
or edits state — and you care about the process as well as the outcome, or you're comparing skills or
models on agentic tasks.

Entry point: [Agent Evaluation](/documentation/evaluate-models/agent-eval).

## Retrieval-driven evaluation

You start from a BEIR test fileset containing `corpus.jsonl`, `queries.jsonl`, and
`qrels/test.tsv`. A `Retrieval` target encodes queries and passages with an embedding NIM, optionally
reranks the dense hits, and `RetrievalNDCGMetric`, `RetrievalRecallMetric`,
`RetrievalPrecisionMetric`, and `RetrievalMAPMetric` score each query. Existing Range aggregation
averages those per-query scores.

Use `Evaluator.run(dataset=BeirDataset.from_path(...), target=Retrieval(...))` for local execution,
or submit the `evaluator.retrieve-eval` platform job. This path evaluates retrieval rankings
directly; it does not generate answers or use RAGAS/LLM-judge answer metrics.

```python
result = await Evaluator().run(
    metrics=[
        RetrievalNDCGMetric(k=[1, 5, 10, 100]),
        RetrievalRecallMetric(k=[1, 5, 10, 100]),
        RetrievalPrecisionMetric(k=[1, 5, 10, 100]),
        RetrievalMAPMetric(k=[1, 5, 10, 100]),
    ],
    dataset=BeirDataset.from_path(path),
    target=Retrieval(embeddings=embed_nim, reranker=rerank_nim, first_stage_k=100),
)
```

```bash
uv run nemo evaluator retrieve-eval submit --spec \
  '{"dataset":"default/eval-beir","target":"default/embed-nim","k":[1,5,10,100]}'
```

The job writes the BEIR-compatible `eval_results.json` artifact with `ndcg_cut_k`, `recall_k`,
`P_k`, and `map_cut_k` keys. Add `"baseline":"default/base-embed-nim"` to report relative
`ndcg_cut_10` and `recall_10` without turning those measurements into a release gate.

## At a glance

|                                 | Dataset-driven                                         | Task-driven (agent evaluation)                 | Retrieval-driven                                     |
| ------------------------------- | ------------------------------------------------------ | ---------------------------------------------- | ---------------------------------------------------- |
| **Unit of evaluation**          | a dataset row                                          | a task, and the trial it produces              | a ranked corpus per query                            |
| **What produces the candidate** | a model/pipeline generates an output per row           | an agent performs the task (via a runner)      | a `Retrieval` target (embeddings, optional reranker) |
| **What you score**              | the output, against a reference                        | the trial: final output **and** trace/evidence | ranked document IDs against qrels                    |
| **Metric assignment**           | one metric set applied to every row                    | each task defines its own metrics              | per-query retrieval metrics, Range-averaged          |
| **Entry point**                 | metrics (`RunConfig`, dataset rows)                    | `AgentEvaluator` + `AgentEvalTask`             | `dataset=` + `target=Retrieval` or `retrieve-eval`   |
| **Best for**                    | model quality, RAG answers, regression on labeled data | agents, tool use, multi-step behavior          | embedding retrievers                                 |

## Which should I use?

Start from *what produces the answer*. If you can generate outputs by running a model over a fixed
dataset, use **dataset-driven**. If an agent has to *do something* to produce the answer — and how it
does it matters — use **task-driven**.

* **Dataset-driven** — you have (or can produce) a labeled dataset and want to score outputs against
  references: LLM quality, RAG, or a regression suite.
* **Task-driven** — the system under test is an agent that uses tools or takes multiple steps, you
  want to score trace evidence as well as the outcome, your suite is **heterogeneous** so different
  tasks need different metrics (e.g. a coding agent producing docs, Q\&A, tests, and net-new code), or
  you're running a skills / model A/B on agentic tasks.
* **Retrieval-driven** — the output is a corpus ranking and relevance is expressed as BEIR qrels.
* **Combine them** — track base model quality, retrieval quality, and agentic behavior independently.

## What all three share

* **One scoring foundation.** Every scorer implements `Metric`; corpus-aware scorers additionally
  implement `CorpusMetric`. Choose metrics whose input matches the evaluation shape.
* **Measurement, not decisions.** The evaluator *measures* — it produces scores, aggregates, and
  provenance. It does not decide pass/fail, gate a release, or compare runs against each other —
  those decisions belong to whatever consumes the results. All evaluation shapes emit measurements;
  decisions live above them.

## Related

#### [Evaluation Metrics](/documentation/evaluate-models/metrics)

#### [Agent Evaluation](/documentation/evaluate-models/agent-eval)

#### [Agent Configuration (targets)](/documentation/evaluate-models/metrics/agent-configuration)