Dataset-Driven vs Task-Driven Evaluation

NeMo Evaluator supports two complementary shapes of evaluation. They share the same scoring foundation but differ in what produces the thing you score and what you’re allowed to look at when you score it.

• Dataset-driven evaluation — you have a dataset of homogeneously structured rows, a target generates an output for each row (or you supply outputs), and metrics score each output against a reference. This is the path behind metrics.
• Task-driven evaluation (agent evaluation) — you have a set of tasks, an agent performs each task, and you score the resulting trial — the agent’s final output and how it got there (its trajectory, tool calls, and other evidence).

Dataset-driven evaluation

You start from a dataset — rows of inputs, usually with a reference/expected value. A model (or pipeline) produces an output per row, either generated online at evaluation time or supplied offline, and metrics score each output against its reference. The suite is uniform — the same metric set applies to every row.

Use it when the thing under test is a model or pipeline you can run over a fixed dataset and you care about output quality: model quality checks, RAG pipelines, and regression testing against a labeled set.

Entry point: Evaluation Metrics.

Task-driven evaluation (agent evaluation)

You start from tasks. Each task gives an agent an instruction and any inputs it needs; the agent performs the task through a runner, producing a trial — the agent’s final output plus evidence such as its trajectory (an Agent Trajectory Interchange Format (ATIF) trace of steps and tool calls), final filesystem state, and logs. Container-based runners also provide an environment (and its final state) the agent works in. Metrics then score the trial. Because the trajectory is part of the trial, you can score not just whether the answer is right but how the agent worked — for example, whether it used the expected tool.

Metrics are attached per task, not once for the whole run — each task carries its own scorers. That lets a single evaluation grade heterogeneous work: a coding agent whose tasks span documentation, Q&A, unit tests, and net-new code can score each task type with the checks that fit it — an LLM-as-a-judge for documentation clarity, a tests-pass check for code, an exact-match check for a factual answer — all in one suite. Dataset-driven runs, by contrast, apply the same metric set to every row.

Use it when the thing under test is an agent that acts: it calls tools, takes multiple steps, or edits state — and you care about the process as well as the outcome, or you’re comparing skills or models on agentic tasks.

Entry point: Agent Evaluation.

At a glance

Which should I use?

• Dataset-driven — you have (or can produce) a labeled dataset and want to score outputs against references: LLM quality, RAG, or a regression suite.
• Task-driven — the system under test is an agent that uses tools or takes multiple steps, you want to score the trajectory as well as the outcome, your suite is heterogeneous so different tasks need different metrics (e.g. a coding agent producing docs, Q&A, tests, and net-new code), or you’re running a skills / model A/B on agentic tasks.
• Both — the two compose. It’s common to track base model quality with dataset-driven metrics and agentic behavior with task-driven evaluation.

What both share

• One scoring interface. Every scorer implements the same Metric protocol, so the same deterministic and LLM-as-a-judge scorers apply to either shape.
• Measurement, not decisions. The evaluator measures — it produces scores, aggregates, and provenance. It does not decide pass/fail, gate a release, or compare runs against each other — those decisions belong to whatever consumes the results. Both evaluation shapes emit measurements; decisions live above them.

Related