Writing Metrics

A metric scores one trial. Metrics are attached to each task (not once per run), so a suite can grade heterogeneous work — a Q&A task and a coding task can carry different scorers. Every metric, however simple or elaborate, implements the same small protocol.

The protocol

from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
class MyMetric:
    @property
    def type(self) -> str:
        """A unique name for this metric within a task (also the summary key prefix)."""
        return "my_metric"
    def output_spec(self) -> list[MetricOutputSpec]:
        """The named values this metric emits, and their types."""
        return [MetricOutputSpec.continuous_score("score")]
    async def compute_scores(self, input: MetricInput) -> MetricResult:
        """Score one trial and return its outputs."""
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0)])

No base class — a metric is any object with these three members (a structural Metric protocol).

What compute_scores receives

input.candidate is the trial under evaluation:

For a trial with an output, candidate.metadata starts with AgentEvalTrial.metadata and then adds AgentOutput.metadata. If both contain the same key, the value from the output wins.

input.row.data is a dict describing the task and trial:

So an outcome metric reads candidate.output_text and row.data["reference"]; a trajectory metric reads candidate.evidence.

Declaring outputs

A metric declares its outputs up front. Outputs are required by default: compute_scores must return every output whose spec has required=True (default is True if not specified). Set required=False only when the producer can legitimately leave that value unmeasured. The runtime rejects missing required outputs, undeclared or duplicate outputs, and present values that cannot be coerced to their declared type.

Build specs with the MetricOutputSpec factories. Pass required=False to mark an output optional.

A metric may emit several outputs — for example an efficiency metric returning both a boolean and a count:

from nemo_evaluator_sdk.metrics.protocol import MetricOutputSpec
def output_spec(self) -> list[MetricOutputSpec]:
    return [
        MetricOutputSpec.boolean("efficient_tool_use"),
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
    ]

Optional outputs

Declare an output as optional when the metric cannot (or not expected) measure it on every otherwise scoreable trial. Return the output when measured and omit it when unmeasured:

from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
class AnswerFormatMetric:
    @property
    def type(self) -> str:
        return "answer_format"
    def output_spec(self) -> list[MetricOutputSpec]:
        return [
            MetricOutputSpec.continuous_score("score"),
            # required=False makes the output optional
            MetricOutputSpec.continuous_score("format_ok", required=False),
        ]
    async def compute_scores(self, input: MetricInput) -> MetricResult:
        answer = input.candidate.output_text
        outputs = [MetricOutput(name="score", value=1.0 if answer else 0.0)]
        if answer is not None:
            outputs.append(MetricOutput(name="format_ok", value=float(answer.startswith("{"))))
        return MetricResult(outputs=outputs)

required=False flag is a producer contract, not an aggregation policy. The metric author decides how to interpret and score missing optional metric scores.

Recommended

Omit the output when unmeasured, as in the example.

• The mean is calculated over finite measured values (count). An omitted output is not averaged in and does not count as zero.
• An omission stays in the applicable set as nan_count and coverage missing.
• count + nan_count equals that output’s coverage total.

See Reading Results for the two-trial format_ok example.

Not recommended

A metric author is allowed to emit a filler. That puts the trial in the measured set and changes aggregates:

• 0.0 is a finite measurement: it enters count and mean, and coverage counts it as scored. The mean is pulled toward zero as if the metric measured failure.
• NaN is observed but non-finite: it is excluded from count and mean, increments nan_count, and coverage still counts it as scored (not missing). The trial looks measured when it was not.
• None is rejected.

Use a filler only when you intend that trial to count as measured.

Returning a result

Return a MetricResult whose outputs match output_spec by name:

from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult
class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"
    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]
    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])

Reading measurements

Metrics read measurements from input.row.data["trial"]["measurements"]. The complete TrialMeasurements schema is always present: unavailable values are None, while a measured zero remains 0 (or 0.0). Persisted or imported measurement values belong exclusively in measurements; free-form metadata values are never converted into measurements, even when their keys resemble measurement names.

class TotalTokensMetric:
    @property
    def type(self) -> str:
        return "usage"
    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.discrete_score("total_tokens", required=False)]
    async def compute_scores(self, input: MetricInput) -> MetricResult:
        total_tokens = input.row.data["trial"]["measurements"]["total_tokens"]
        outputs = []
        if total_tokens is not None:
            outputs.append(MetricOutput(name="total_tokens", value=total_tokens))
        return MetricResult(outputs=outputs)

Reading evidence

candidate.evidence (a CandidateEvidence) holds named descriptors, each exposed through a typed handle. trace() returns the runner’s primary view as an ATIFTraceHandle | OTLPTraceHandle union. Always guard for missing evidence, then narrow on handle.format before using format-specific methods:

from nemo_evaluator_sdk.metrics.protocol import MetricInput
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE
async def read_trace(input: MetricInput) -> None:
    evidence = input.candidate.evidence
    if evidence is None or evidence.get(EVIDENCE_TRACE) is None:
        return
    handle = await evidence.trace(EVIDENCE_TRACE)
    if handle.format == "otlp":
        resource_spans = await handle.resource_spans()
        # extract tool calls based on the OTLP exporter specific logic
        # calls = await extract_tool_calls(resource_spans)
    else:
        calls = await handle.tool_calls()

OTLP exposes the unflattened span tree as parsed ResourceSpans protobuf messages — the same type opentelemetry-proto defines and Intake ingests — so a reader walks .scope_spans[].spans[] and typed attributes rather than decoded JSON. It does not provide a generic tool_calls() method: what counts as a tool call depends on whatever produced the spans — the Gym projection, or a harness like Fabric or Codex — each of which names and nests them differently, so extracting them is the consumer’s job.

Asking for one trace format

A runner may record the same trial in more than one encoding. Harbor registers both under trace:atif and trace:otlp when the agent under test emits both, and Fabric registers both whenever it captures an OTLP trace from Relay. Pass format= to choose one and get a narrowed handle back, with no runtime branch:

from nemo_evaluator_sdk.values.evidence import CandidateEvidence
async def read_both_views(evidence: CandidateEvidence) -> None:
    otlp = await evidence.trace(format="otlp")  # OTLPTraceHandle
    resource_spans = await otlp.resource_spans()
    atif = await evidence.trace(format="atif")  # ATIFTraceHandle
    calls = await atif.tool_calls()

trace() with no format still returns whichever encoding the runner made primary. For Harbor and Fabric alike that is the OTLP trace when one was recorded and the ATIF trace otherwise — there is no setting to choose between them. A format= request raises KeyError when the trial carries no trace in that encoding, so a metric that depends on one specific view fails loudly instead of silently scoring nothing.

The Score by Component guide has a complete, runnable trajectory metric; the SDK’s example_metrics.py (under examples/run_agent_eval/) shows filesystem and trace metrics.

How outputs are reported

Each output aggregates in result.summary under <metric.type>.<output>. The aggregate reports mean, extrema, population and sample statistics, count, and nan_count:

• count is the number of finite measured values.
• nan_count is the number of applicable opportunities that were unmeasured or failed.
• An optional output omitted everywhere keeps a row with count=0 and mean=None.
• summary.metric_coverage separately reports total, scored, missing, and failed.

See Reading Results for exact examples. To roll several outputs into one reported score, define a view on the task — see Score by Component.

Related