> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Writing Metrics

> Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported.

A **metric** scores one trial. Metrics are attached to each task (not once per run), so a suite can
grade heterogeneous work — a Q\&A task and a coding task can carry different scorers. Every metric,
however simple or elaborate, implements the same small protocol.

## The protocol

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class MyMetric:
    @property
    def type(self) -> str:
        """A unique name for this metric within a task (also the summary key prefix)."""
        return "my_metric"

    def output_spec(self) -> list[MetricOutputSpec]:
        """The named values this metric emits, and their types."""
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        """Score one trial and return its outputs."""
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0)])
```

No base class — a metric is any object with these three members (a structural `Metric` protocol).

## What `compute_scores` receives

`input.candidate` is the trial under evaluation:

| Field                   | Type                        | What it holds                                                                                                 |
| ----------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `candidate.output_text` | `str \| None`               | the agent's final answer                                                                                      |
| `candidate.response`    | `Any \| None`               | the raw model or agent response payload                                                                       |
| `candidate.trajectory`  | `Any \| None`               | the candidate trajectory when supplied directly                                                               |
| `candidate.evidence`    | `CandidateEvidence \| None` | trajectory, final filesystem state, logs — see [Reading evidence](#reading-evidence)                          |
| `candidate.metadata`    | `dict`                      | free-form trial and output metadata, excluding the dedicated fields above; empty when the trial has no output |

For a trial with an output, `candidate.metadata` starts with `AgentEvalTrial.metadata` and then adds
`AgentOutput.metadata`. If both contain the same key, the value from the output wins.

`input.row.data` is a dict describing the task and trial:

| Key                           | What it holds                                                               |
| ----------------------------- | --------------------------------------------------------------------------- |
| `input.row.data["reference"]` | grader-only ground truth (the task's `reference`), never shown to the agent |
| `input.row.data["inputs"]`    | the task `inputs` (`instruction`, …)                                        |
| `input.row.data["task"]`      | `{id, intent, metadata}`                                                    |
| `input.row.data["trial"]`     | `{id, task_id, status, error, measurements, metadata}`                      |

So an **outcome** metric reads `candidate.output_text` and `row.data["reference"]`; a **trajectory**
metric reads `candidate.evidence`.

## Declaring outputs

A metric declares its outputs up front. Outputs are required by default: `compute_scores` must return
every output whose spec has `required=True` (default is True if not specified). Set `required=False` only when the producer can
legitimately leave that value unmeasured. The runtime rejects missing required outputs, undeclared or
duplicate outputs, and present values that cannot be coerced to their declared type.

Build specs with the `MetricOutputSpec` factories. Pass `required=False` to mark an
output optional.

| Factory                                      | Value type       | Use for                            |
| -------------------------------------------- | ---------------- | ---------------------------------- |
| `MetricOutputSpec.continuous_score(name)`    | `float`          | a numeric score (0–1 or unbounded) |
| `MetricOutputSpec.discrete_score(name)`      | `int`            | counts or ordinal levels           |
| `MetricOutputSpec.boolean(name)`             | `bool`           | a pass/fail check                  |
| `MetricOutputSpec.label(name)`               | `str`            | a category label                   |
| `MetricOutputSpec.model(name, value_schema)` | your `BaseModel` | structured/custom values           |

A metric may emit **several** outputs — for example an efficiency metric returning both a boolean and a
count:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricOutputSpec

def output_spec(self) -> list[MetricOutputSpec]:
    return [
        MetricOutputSpec.boolean("efficient_tool_use"),
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
    ]
```

Prefer `continuous_score` when you want a numeric **mean** in the run summary. A `boolean` output
reports per-trial pass/fail but does not aggregate to a numeric mean on its own (though it still
contributes as 0/1 to a [view](/documentation/evaluate-models/agent-eval/score-by-component)).

### Optional outputs

Declare an output as optional when the metric cannot (or not expected) measure it on every otherwise scoreable trial.
Return the output when measured and omit it when unmeasured:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class AnswerFormatMetric:
    @property
    def type(self) -> str:
        return "answer_format"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [
            MetricOutputSpec.continuous_score("score"),
            # required=False makes the output optional
            MetricOutputSpec.continuous_score("format_ok", required=False),
        ]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        answer = input.candidate.output_text
        outputs = [MetricOutput(name="score", value=1.0 if answer else 0.0)]
        if answer is not None:
            outputs.append(MetricOutput(name="format_ok", value=float(answer.startswith("{"))))
        return MetricResult(outputs=outputs)
```

`required=False` flag is a producer contract, not an aggregation policy.
The metric author decides how to interpret and score missing optional metric scores.

#### Recommended

Omit the output when unmeasured, as in the example.

* The mean is calculated over finite measured values (`count`). An omitted output is not averaged in and does not count as zero.
* An omission stays in the applicable set as `nan_count` and coverage `missing`.
* `count + nan_count` equals that output's coverage `total`.

See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for the two-trial `format_ok` example.

#### Not recommended

A metric author is allowed to emit a filler. That puts the trial in the measured set and changes aggregates:

* `0.0` is a finite measurement: it enters `count` and `mean`, and coverage counts it as
  `scored`. The mean is pulled toward zero as if the metric measured failure.
* NaN is observed but non-finite: it is excluded from `count` and `mean`, increments
  `nan_count`, and coverage still counts it as `scored` (not `missing`). The trial looks
  measured when it was not.
* `None` is rejected.

Use a filler only when you intend that trial to count as measured.

## Returning a result

Return a `MetricResult` whose `outputs` match `output_spec` by name:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## Reading measurements

Metrics read measurements from `input.row.data["trial"]["measurements"]`. The complete
[`TrialMeasurements`](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py)
schema is always present: unavailable values are `None`, while a measured zero remains `0` (or
`0.0`). Persisted or imported measurement values belong exclusively in `measurements`; free-form
metadata values are never converted into measurements, even when their keys resemble measurement
names.

```python
class TotalTokensMetric:
    @property
    def type(self) -> str:
        return "usage"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.discrete_score("total_tokens", required=False)]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        total_tokens = input.row.data["trial"]["measurements"]["total_tokens"]
        outputs = []
        if total_tokens is not None:
            outputs.append(MetricOutput(name="total_tokens", value=total_tokens))
        return MetricResult(outputs=outputs)
```

## Reading evidence

`candidate.evidence` (a `CandidateEvidence`) holds named descriptors, each exposed through a typed
handle. `trace()` returns the runner's primary view as an `ATIFTraceHandle | OTLPTraceHandle` union.
Always guard for missing evidence, then narrow on `handle.format` before using format-specific
methods:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE

async def read_trace(input: MetricInput) -> None:
    evidence = input.candidate.evidence
    if evidence is None or evidence.get(EVIDENCE_TRACE) is None:
        return

    handle = await evidence.trace(EVIDENCE_TRACE)
    if handle.format == "otlp":
        resource_spans = await handle.resource_spans()
        # extract tool calls based on the OTLP exporter specific logic
        # calls = await extract_tool_calls(resource_spans)
    else:
        calls = await handle.tool_calls()
```

| Evidence                                                          | Handle                                                    | Reads                                                                        |
| ----------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **trace** (`EVIDENCE_TRACE`)                                      | `ATIFTraceHandle` — `await evidence.trace(format="atif")` | `.trace()`, `.tool_calls()`, `.steps()`, `.token_usage()`                    |
| **trace** (`EVIDENCE_TRACE`)                                      | `OTLPTraceHandle` — `await evidence.trace(format="otlp")` | `.resource_spans()`                                                          |
| **filesystem** (`EVIDENCE_FINAL_STATE`, `EVIDENCE_INITIAL_STATE`) | `await evidence.filesystem(name)`                         | `.run_verifier(command)`, `.diff(other)` — run a check or diff two snapshots |
| **logs**                                                          | `await evidence.logs(name)`                               | `.read_text(file)`, `.tail(file)`                                            |

OTLP exposes the unflattened span tree as parsed `ResourceSpans` protobuf messages — the same type
`opentelemetry-proto` defines and Intake ingests — so a reader walks `.scope_spans[].spans[]` and
typed attributes rather than decoded JSON. It does not provide a generic `tool_calls()` method: what
counts as a tool call depends on whatever produced the spans — the Gym projection, or a harness like
Fabric or Codex — each of which names and nests them differently, so extracting them is the
consumer's job.

### Asking for one trace format

A runner may record the same trial in more than one encoding. Harbor registers both under
`trace:atif` and `trace:otlp` when the agent under test emits both, and Fabric registers both
whenever it captures an OTLP trace from Relay. Pass `format=` to choose one and get a narrowed
handle back, with no runtime branch:

```python
from nemo_evaluator_sdk.values.evidence import CandidateEvidence

async def read_both_views(evidence: CandidateEvidence) -> None:
    otlp = await evidence.trace(format="otlp")  # OTLPTraceHandle
    resource_spans = await otlp.resource_spans()

    atif = await evidence.trace(format="atif")  # ATIFTraceHandle
    calls = await atif.tool_calls()
```

`trace()` with no `format` still returns whichever encoding the runner made primary. For Harbor
and Fabric alike that is the OTLP trace when one was recorded and the ATIF trace otherwise — there
is no setting to choose between them. A `format=` request raises `KeyError` when the trial carries no trace in
that encoding, so a metric that depends on one specific view fails loudly instead of silently
scoring nothing.

The [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) guide has a
complete, runnable trajectory metric; the SDK's `example_metrics.py` (under
`examples/run_agent_eval/`) shows filesystem and trace metrics.

## How outputs are reported

Each output aggregates in `result.summary` under `<metric.type>.<output>`. The aggregate reports
`mean`, extrema, population and sample statistics, `count`, and `nan_count`:

* `count` is the number of finite measured values.
* `nan_count` is the number of applicable opportunities that were unmeasured or failed.
* An optional output omitted everywhere keeps a row with `count=0` and `mean=None`.
* `summary.metric_coverage` separately reports `total`, `scored`, `missing`, and `failed`.

See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for exact examples. To
roll several outputs into one reported score, define a **view** on the task — see
[Score by Component](/documentation/evaluate-models/agent-eval/score-by-component).

## Related

#### [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)

#### [Reading Results](/documentation/evaluate-models/agent-eval/reading-results)

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Quickstart](/documentation/evaluate-models/agent-eval/quickstart)