> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Writing Metrics

> Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.

A **metric** scores one trial. Metrics are attached to each task (not once per run), so a suite can
grade heterogeneous work — a Q\&A task and a coding task can carry different scorers. Every metric,
however simple or elaborate, implements the same small protocol.

## The protocol

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class MyMetric:
    @property
    def type(self) -> str:
        """A unique name for this metric within a task (also the summary key prefix)."""
        return "my_metric"

    def output_spec(self) -> list[MetricOutputSpec]:
        """The named values this metric emits, and their types."""
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        """Score one trial and return its outputs."""
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0)])
```

No base class — a metric is any object with these three members (a structural `Metric` protocol).

## What `compute_scores` receives

`input.candidate` is the trial under evaluation:

| Field                   | Type                        | What it holds                                                                        |
| ----------------------- | --------------------------- | ------------------------------------------------------------------------------------ |
| `candidate.output_text` | `str \| None`               | the agent's final answer                                                             |
| `candidate.evidence`    | `CandidateEvidence \| None` | trajectory, final filesystem state, logs — see [Reading evidence](#reading-evidence) |
| `candidate.metadata`    | `dict`                      | trial metadata (e.g. a reward a runner stamped on)                                   |

`input.row.data` is a dict describing the task and trial:

| Key                           | What it holds                                                               |
| ----------------------------- | --------------------------------------------------------------------------- |
| `input.row.data["reference"]` | grader-only ground truth (the task's `reference`), never shown to the agent |
| `input.row.data["inputs"]`    | the task `inputs` (`instruction`, …)                                        |
| `input.row.data["task"]`      | `{id, intent, metadata}`                                                    |
| `input.row.data["trial"]`     | `{id, task_id, status, metadata}`                                           |

So an **outcome** metric reads `candidate.output_text` and `row.data["reference"]`; a **trajectory**
metric reads `candidate.evidence`.

## Declaring outputs

A metric declares its outputs up front; the runtime validates that `compute_scores` returns **exactly**
those names, each coercible to the declared type (a missing or undeclared output raises). Build specs
with the `MetricOutputSpec` factories:

| Factory                                      | Value type       | Use for                            |
| -------------------------------------------- | ---------------- | ---------------------------------- |
| `MetricOutputSpec.continuous_score(name)`    | `float`          | a numeric score (0–1 or unbounded) |
| `MetricOutputSpec.discrete_score(name)`      | `int`            | counts or ordinal levels           |
| `MetricOutputSpec.boolean(name)`             | `bool`           | a pass/fail check                  |
| `MetricOutputSpec.label(name)`               | `str`            | a category label                   |
| `MetricOutputSpec.model(name, value_schema)` | your `BaseModel` | structured/custom values           |

A metric may emit **several** outputs — for example an efficiency metric returning both a boolean and a
count:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricOutputSpec

def output_spec(self) -> list[MetricOutputSpec]:
    return [
        MetricOutputSpec.boolean("efficient_tool_use"),
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
    ]
```

Prefer `continuous_score` when you want a numeric **mean** in the run summary. A `boolean` output
reports per-trial pass/fail but does not aggregate to a numeric mean on its own (though it still
contributes as 0/1 to a [view](/documentation/evaluate-models/agent-eval/score-by-component)).

## Returning a result

Return a `MetricResult` whose `outputs` match `output_spec` by name:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## Reading evidence

`candidate.evidence` (a `CandidateEvidence`) holds named descriptors, each exposed through a typed
handle. `trace()` returns the runner's primary view as an `ATIFTraceHandle | OTLPTraceHandle` union.
Always guard for missing evidence, then narrow on `handle.format` before using format-specific
methods:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE

async def read_trace(input: MetricInput) -> None:
    evidence = input.candidate.evidence
    if evidence is None or evidence.get(EVIDENCE_TRACE) is None:
        return

    handle = await evidence.trace(EVIDENCE_TRACE)
    if handle.format == "otlp":
        resource_spans = await handle.resource_spans()
        # extract tool calls based on the OTLP exporter specific logic
        # calls = await extract_tool_calls(resource_spans)
    else:
        calls = await handle.tool_calls()
```

| Evidence                                                          | Handle                                                    | Reads                                                                        |
| ----------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **trace** (`EVIDENCE_TRACE`)                                      | `ATIFTraceHandle` — `await evidence.trace(format="atif")` | `.trace()`, `.tool_calls()`, `.steps()`, `.token_usage()`                    |
| **trace** (`EVIDENCE_TRACE`)                                      | `OTLPTraceHandle` — `await evidence.trace(format="otlp")` | `.resource_spans()`                                                          |
| **filesystem** (`EVIDENCE_FINAL_STATE`, `EVIDENCE_INITIAL_STATE`) | `await evidence.filesystem(name)`                         | `.run_verifier(command)`, `.diff(other)` — run a check or diff two snapshots |
| **logs**                                                          | `await evidence.logs(name)`                               | `.read_text(file)`, `.tail(file)`                                            |

OTLP exposes the unflattened `resourceSpans` tree; it does not provide a generic `tool_calls()` method.
It's a responsibility of consumer to parse `resourceSpans` tree specific to OTLP exporter to extract the searched data.

### Asking for one trace format

A runner may record the same trial in more than one encoding. Harbor registers both under
`trace:atif` and `trace:otlp` when the agent under test emits both. Pass `format=` to choose one
and get a narrowed handle back, with no runtime branch:

```python
from nemo_evaluator_sdk.values.evidence import CandidateEvidence

async def read_both_views(evidence: CandidateEvidence) -> None:
    otlp = await evidence.trace(format="otlp")  # OTLPTraceHandle
    resource_spans = await otlp.resource_spans()

    atif = await evidence.trace(format="atif")  # ATIFTraceHandle
    calls = await atif.tool_calls()
```

`trace()` with no `format` still returns whichever encoding the runner made primary. For Harbor
that is the OTLP trace when the agent emits one and the ATIF trace otherwise — there is no setting
to choose between them. A `format=` request raises `KeyError` when the trial carries no trace in
that encoding, so a metric that depends on one specific view fails loudly instead of silently
scoring nothing.

The [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) guide has a
complete, runnable trajectory metric; the SDK's `example_metrics.py` (under
`examples/run_agent_eval/`) shows filesystem and trace metrics.

## How outputs are reported

Each output aggregates in `result.summary` under the key `<metric.type>.<output>` (mean / min / max /
std-dev). To roll several outputs into one reported score, define a **view** on the task — see
[Score by Component](/documentation/evaluate-models/agent-eval/score-by-component).

## Related

#### [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Quickstart](/documentation/evaluate-models/agent-eval/quickstart)