> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Writing Metrics

> Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.

A **metric** scores one trial. Metrics are attached to each task (not once per run), so a suite can
grade heterogeneous work — a Q\&A task and a coding task can carry different scorers. Every metric,
however simple or elaborate, implements the same small protocol.

## The protocol

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutputSpec, MetricResult


class MyMetric:
    @property
    def type(self) -> str:
        """A unique name for this metric within a task (also the summary key prefix)."""

    def output_spec(self) -> list[MetricOutputSpec]:
        """The named values this metric emits, and their types."""

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        """Score one trial and return its outputs."""
```

No base class — a metric is any object with these three members (a structural `Metric` protocol).

## What `compute_scores` receives

`input.candidate` is the trial under evaluation:

| Field                   | Type                        | What it holds                                                                        |
| ----------------------- | --------------------------- | ------------------------------------------------------------------------------------ |
| `candidate.output_text` | `str \| None`               | the agent's final answer                                                             |
| `candidate.evidence`    | `CandidateEvidence \| None` | trajectory, final filesystem state, logs — see [Reading evidence](#reading-evidence) |
| `candidate.metadata`    | `dict`                      | trial metadata (e.g. a reward a runner stamped on)                                   |

`input.row.data` is a dict describing the task and trial:

| Key                           | What it holds                                                               |
| ----------------------------- | --------------------------------------------------------------------------- |
| `input.row.data["reference"]` | grader-only ground truth (the task's `reference`), never shown to the agent |
| `input.row.data["inputs"]`    | the task `inputs` (`instruction`, …)                                        |
| `input.row.data["task"]`      | `{id, intent, metadata}`                                                    |
| `input.row.data["trial"]`     | `{id, task_id, status, metadata}`                                           |

So an **outcome** metric reads `candidate.output_text` and `row.data["reference"]`; a **trajectory**
metric reads `candidate.evidence`.

## Declaring outputs

A metric declares its outputs up front; the runtime validates that `compute_scores` returns **exactly**
those names, each coercible to the declared type (a missing or undeclared output raises). Build specs
with the `MetricOutputSpec` factories:

| Factory                                      | Value type       | Use for                            |
| -------------------------------------------- | ---------------- | ---------------------------------- |
| `MetricOutputSpec.continuous_score(name)`    | `float`          | a numeric score (0–1 or unbounded) |
| `MetricOutputSpec.discrete_score(name)`      | `int`            | counts or ordinal levels           |
| `MetricOutputSpec.boolean(name)`             | `bool`           | a pass/fail check                  |
| `MetricOutputSpec.label(name)`               | `str`            | a category label                   |
| `MetricOutputSpec.model(name, value_schema)` | your `BaseModel` | structured/custom values           |

A metric may emit **several** outputs — for example an efficiency metric returning both a boolean and a
count:

```python
def output_spec(self) -> list[MetricOutputSpec]:
    return [
        MetricOutputSpec.boolean("efficient_tool_use"),
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
    ]
```

Prefer `continuous_score` when you want a numeric **mean** in the run summary. A `boolean` output
reports per-trial pass/fail but does not aggregate to a numeric mean on its own (though it still
contributes as 0/1 to a [view](/documentation/evaluate-models/agent-eval/score-by-component)).

## Returning a result

Return a `MetricResult` whose `outputs` match `output_spec` by name:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult


class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## Reading evidence

`candidate.evidence` (a `CandidateEvidence`) holds named descriptors, each exposed through a typed
handle. Always guard first — a trial may not carry a given descriptor:

```python
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE

evidence = input.candidate.evidence
if evidence is not None and evidence.get(EVIDENCE_TRACE) is not None:
    calls = await (await evidence.trace(EVIDENCE_TRACE)).tool_calls()
```

| Evidence                                                          | Handle                            | Reads                                                                                                                                                                                            |
| ----------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **trace** (`EVIDENCE_TRACE`)                                      | `await evidence.trace(name)`      | `.tool_calls()`, `.steps()`, `.token_usage()` — the [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trajectory |
| **filesystem** (`EVIDENCE_FINAL_STATE`, `EVIDENCE_INITIAL_STATE`) | `await evidence.filesystem(name)` | `.run_verifier(command)`, `.diff(other)` — run a check or diff two snapshots                                                                                                                     |
| **logs**                                                          | `await evidence.logs(name)`       | `.read_text(file)`, `.tail(file)`                                                                                                                                                                |

The [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) guide has a
complete, runnable trajectory metric; the SDK's `example_metrics.py` (under
`examples/run_agent_eval/`) shows filesystem and trace metrics.

## How outputs are reported

Each output aggregates in `result.summary` under the key `<metric.type>.<output>` (mean / min / max /
std-dev). To roll several outputs into one reported score, define a **view** on the task — see
[Score by Component](/documentation/evaluate-models/agent-eval/score-by-component).

## Related

#### [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Quickstart](/documentation/evaluate-models/agent-eval/quickstart)