Writing Metrics

A metric scores one trial. Metrics are attached to each task (not once per run), so a suite can grade heterogeneous work — a Q&A task and a coding task can carry different scorers. Every metric, however simple or elaborate, implements the same small protocol.

The protocol

1
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutputSpec, MetricResult
2
3
4
class MyMetric:
5
    @property
6
    def type(self) -> str:
7
        """A unique name for this metric within a task (also the summary key prefix)."""
8
9
    def output_spec(self) -> list[MetricOutputSpec]:
10
        """The named values this metric emits, and their types."""
11
12
    async def compute_scores(self, input: MetricInput) -> MetricResult:
13
        """Score one trial and return its outputs."""

No base class — a metric is any object with these three members (a structural Metric protocol).

What compute_scores receives

input.candidate is the trial under evaluation:

input.row.data is a dict describing the task and trial:

So an outcome metric reads candidate.output_text and row.data["reference"]; a trajectory metric reads candidate.evidence.

Declaring outputs

A metric declares its outputs up front; the runtime validates that compute_scores returns exactly those names, each coercible to the declared type (a missing or undeclared output raises). Build specs with the MetricOutputSpec factories:

A metric may emit several outputs — for example an efficiency metric returning both a boolean and a count:

1
def output_spec(self) -> list[MetricOutputSpec]:
2
    return [
3
        MetricOutputSpec.boolean("efficient_tool_use"),
4
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
5
    ]

Returning a result

Return a MetricResult whose outputs match output_spec by name:

1
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
2
3
4
class KeywordMatchMetric:
5
    @property
6
    def type(self) -> str:
7
        return "keyword_match"
8
9
    def output_spec(self) -> list[MetricOutputSpec]:
10
        return [MetricOutputSpec.continuous_score("score")]
11
12
    async def compute_scores(self, input: MetricInput) -> MetricResult:
13
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
14
        answer = (input.candidate.output_text or "").lower()
15
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])

Reading evidence

candidate.evidence (a CandidateEvidence) holds named descriptors, each exposed through a typed handle. Always guard first — a trial may not carry a given descriptor:

1
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE
2
3
evidence = input.candidate.evidence
4
if evidence is not None and evidence.get(EVIDENCE_TRACE) is not None:
5
    calls = await (await evidence.trace(EVIDENCE_TRACE)).tool_calls()

The Score by Component guide has a complete, runnable trajectory metric; the SDK’s example_metrics.py (under examples/run_agent_eval/) shows filesystem and trace metrics.

How outputs are reported

Each output aggregates in result.summary under the key <metric.type>.<output> (mean / min / max / std-dev). To roll several outputs into one reported score, define a view on the task — see Score by Component.

Related