> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Score by Component

> Go beyond the final answer: add a metric that scores the agent's trajectory (its tool use), attach it alongside an outcome metric, and roll both into one named view — all runnable locally with no services.

The [quickstart](/documentation/evaluate-models/agent-eval/quickstart) scored one thing: whether the
final answer contained the right keyword. But an agent can reach the right answer the *wrong way* —
guessing instead of looking something up, or looping on a tool. Agent evaluation lets you score **how**
the agent worked, not just the outcome, and combine several signals into one reported **view**.

This guide extends the quickstart. You'll add a **trajectory metric** that reads the agent's tool
calls, keep the quickstart's **outcome metric**, and combine them into a `quality` **view**. It stays
zero-dependency — one local process, no API keys.

This builds on the [quickstart](/documentation/evaluate-models/agent-eval/quickstart)'s
`KeywordMatchMetric` and tasks. The full runnable script is at the end.

## 1. The outcome metric

Reuse the quickstart's `KeywordMatchMetric` — it scores the final answer, reading it from
`input.candidate.output_text` and the grader-only truth from `input.row.data`:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## 2. A trajectory metric

A metric can read more than the final output. `input.candidate.evidence` exposes the trial's
**evidence**. This guide produces an [ATIF](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md)
trace, whose handle returns modeled tool calls in order. Each `ToolCall` has a
`function_name` and `arguments`. This metric scores `1.0` when the agent used the tool you expected
and `0.0` otherwise:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE

class UsedExpectedToolMetric:
    """Score whether the agent's trajectory used a given tool."""

    def __init__(self, expected_tool: str) -> None:
        self._expected = expected_tool

    @property
    def type(self) -> str:
        return "used_expected_tool"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("tool_use")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        evidence = input.candidate.evidence
        if evidence is None:
            return MetricResult(outputs=[MetricOutput(name="tool_use", value=0.0)])
        try:
            handle = await evidence.trace(EVIDENCE_TRACE, format="atif")
        except KeyError:
            return MetricResult(outputs=[MetricOutput(name="tool_use", value=0.0)])
        calls = await handle.tool_calls()
        used = any(call.function_name == self._expected for call in calls)
        return MetricResult(outputs=[MetricOutput(name="tool_use", value=1.0 if used else 0.0)])
```

Asking for `format="atif"` raises `KeyError` when the trial recorded no ATIF trace, which this metric
scores the same as missing evidence: `0.0`. Harbor makes OTLP primary, so a trial that recorded only
an OTLP trace scores `0.0` here rather than failing — read that score as "this metric could not see a
trajectory," not as proof the tool went unused. An OTLP metric asks for `format="otlp"` and inspects
`resource_spans()` with exporter-specific logic.

## 3. Produce a trajectory

Runners that execute the agent capture this evidence for you — a
[deployed HTTP agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) returns a
trajectory when you set `trajectory_path`, and container/harness backends record one as the agent runs.
(A [Harbor](/documentation/evaluate-models/agent-eval/harbor-runner) run scores a verifier reward
instead.) Here the agent is a local callable, so it returns a `TrialDraft` — its final output plus,
when it used a tool, a small ATIF trace built from the SDK's trajectory models. One task's agent uses the `search` tool; the
other skips it and just guesses, returning no trace at all:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import TrialDraft
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask
from nemo_evaluator_sdk.agent_eval.trials import AgentOutput
from nemo_evaluator_sdk.values.atif import Step, ToolCall, Trajectory
from nemo_evaluator_sdk.values.evidence import (
    EVIDENCE_FORMAT_ATIF,
    EVIDENCE_TRACE,
    CandidateEvidence,
    EvidenceDescriptor,
)

def _trace(tool_name: str, query: str) -> CandidateEvidence:
    trajectory = Trajectory(
        schema_version="ATIF-v1.7",
        steps=[Step(source="agent", tool_calls=[ToolCall(function_name=tool_name, arguments={"query": query})])],
    )
    return CandidateEvidence(
        descriptors={
            EVIDENCE_TRACE: EvidenceDescriptor(
                kind="trace", format=EVIDENCE_FORMAT_ATIF, data=trajectory.model_dump(mode="json")
            )
        }
    )

async def my_agent(task: AgentEvalTask) -> TrialDraft:
    answers = {
        "capital-france": ("The capital of France is Paris.", "search"),  # used the expected tool
        "capital-japan": ("The capital of Japan is Tokyo.", None),        # right answer, but skipped the tool
    }
    text, tool = answers[task.id]
    evidence = _trace(tool, task.inputs["instruction"]) if tool else None
    return TrialDraft(output=AgentOutput(output_text=text), evidence=evidence)
```

## 4. Combine the signals into a view

Attach both metrics to each task, then define a **view** — a named roll-up of this task's metric
outputs. `SemanticView` reduces its `signals` (each a `metric.output`) into one score; `MEAN` averages
them. The view is reported per task and aggregated across the run as `view.<name>`.

```python
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask, SemanticReducer, SemanticView, ViewSignal

def _quality_view() -> SemanticView:
    return SemanticView(
        reducer=SemanticReducer.MEAN,
        signals=[
            ViewSignal(metric="keyword_match", output="score"),
            ViewSignal(metric="used_expected_tool", output="tool_use"),
        ],
    )

def make_tasks() -> list[AgentEvalTask]:
    metrics = [KeywordMatchMetric(), UsedExpectedToolMetric("search")]
    return [
        AgentEvalTask(
            id="capital-france", intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"}, reference={"expected": "Paris"},
            metrics=metrics, views={"quality": _quality_view()},
        ),
        AgentEvalTask(
            id="capital-japan", intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"}, reference={"expected": "Tokyo"},
            metrics=metrics, views={"quality": _quality_view()},
        ),
    ]
```

## 5. Run it and read the components

```python
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig

async def run_evaluation() -> None:
    result = await AgentEvaluator().run(
        tasks=make_tasks(),
        target=CallableAgentTaskRunner(my_agent),
        config=AgentEvalRunConfig(parallelism=2),
    )

    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")
```

Output:

```
keyword_match.score: 1.0
used_expected_tool.tool_use: 0.5
view.quality: 0.75
```

This is the whole point of scoring by component. The **outcome** looks perfect — every answer was
correct (`1.0`). But the **trajectory** metric shows only half the agents actually used the expected
tool (`0.5`); the other got the right answer by guessing. The **view** combines the two into a single
`quality` score (`0.75`) you can track over time. Scoring only the answer would have hidden the
shortcut entirely.

## Full script

```python
import asyncio

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner, TrialDraft
from nemo_evaluator_sdk.agent_eval.tasks import (
    AgentEvalRunConfig,
    AgentEvalTask,
    SemanticReducer,
    SemanticView,
    ViewSignal,
)
from nemo_evaluator_sdk.agent_eval.trials import AgentOutput
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
from nemo_evaluator_sdk.values.atif import Step, ToolCall, Trajectory
from nemo_evaluator_sdk.values.evidence import (
    EVIDENCE_FORMAT_ATIF,
    EVIDENCE_TRACE,
    CandidateEvidence,
    EvidenceDescriptor,
)

class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])

class UsedExpectedToolMetric:
    """Score whether the agent's trajectory used a given tool."""

    def __init__(self, expected_tool: str) -> None:
        self._expected = expected_tool

    @property
    def type(self) -> str:
        return "used_expected_tool"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("tool_use")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        evidence = input.candidate.evidence
        if evidence is None:
            return MetricResult(outputs=[MetricOutput(name="tool_use", value=0.0)])
        try:
            handle = await evidence.trace(EVIDENCE_TRACE, format="atif")
        except KeyError:
            return MetricResult(outputs=[MetricOutput(name="tool_use", value=0.0)])
        calls = await handle.tool_calls()
        used = any(call.function_name == self._expected for call in calls)
        return MetricResult(outputs=[MetricOutput(name="tool_use", value=1.0 if used else 0.0)])

def _trace(tool_name: str, query: str) -> CandidateEvidence:
    trajectory = Trajectory(
        schema_version="ATIF-v1.7",
        steps=[Step(source="agent", tool_calls=[ToolCall(function_name=tool_name, arguments={"query": query})])],
    )
    return CandidateEvidence(
        descriptors={
            EVIDENCE_TRACE: EvidenceDescriptor(
                kind="trace", format=EVIDENCE_FORMAT_ATIF, data=trajectory.model_dump(mode="json")
            )
        }
    )

def _quality_view() -> SemanticView:
    return SemanticView(
        reducer=SemanticReducer.MEAN,
        signals=[
            ViewSignal(metric="keyword_match", output="score"),
            ViewSignal(metric="used_expected_tool", output="tool_use"),
        ],
    )

def make_tasks() -> list[AgentEvalTask]:
    metrics = [KeywordMatchMetric(), UsedExpectedToolMetric("search")]
    return [
        AgentEvalTask(
            id="capital-france", intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"}, reference={"expected": "Paris"},
            metrics=metrics, views={"quality": _quality_view()},
        ),
        AgentEvalTask(
            id="capital-japan", intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"}, reference={"expected": "Tokyo"},
            metrics=metrics, views={"quality": _quality_view()},
        ),
    ]

async def my_agent(task: AgentEvalTask) -> TrialDraft:
    answers = {
        "capital-france": ("The capital of France is Paris.", "search"),  # used the expected tool
        "capital-japan": ("The capital of Japan is Tokyo.", None),        # right answer, but skipped the tool
    }
    text, tool = answers[task.id]
    evidence = _trace(tool, task.inputs["instruction"]) if tool else None
    return TrialDraft(output=AgentOutput(output_text=text), evidence=evidence)

async def main() -> None:
    result = await AgentEvaluator().run(
        tasks=make_tasks(),
        target=CallableAgentTaskRunner(my_agent),
        config=AgentEvalRunConfig(parallelism=2),
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")

asyncio.run(main())
```

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)