Reading Results

AgentEvaluator().run(...) returns an AgentEvalResult and writes nothing. Call result.persist() to store it as a run bundle on disk. The object and the bundle hold the same data — use the object for programmatic follow-up, and the bundle (especially report.html) to inspect or share a run.

The result object

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
result = await AgentEvaluator().run(tasks=..., target=...)

The summary

result.summary (an AgentEvalSummary):

• summary.scores.scores — the aggregates. Each is named <metric.type>.<output> (and view.<name> for a view), with mean, min, max, and std_dev, count, and nan_count. count is the number of finite measured values; nan_count is the number of applicable opportunities that were unmeasured or failed. This is what the guides print:

  for aggregate in result.summary.scores.scores:
      print(f"{aggregate.name}: {aggregate.mean}")

• summary.metric_coverage — per metric output, how many applicable trials were total, scored, missing, or failed, so you can tell a low mean from low coverage. For each output, scored + missing + failed == total.

• summary.task_metric_values — per task, the individual trial values behind those means, keyed <metric.type>.<output>. Each record carries the trial_id that produced it and its metric value, so you can answer “which tasks were flaky, and on which trial?” without regrouping result.scores yourself:

  for task_id, by_output in result.summary.task_metric_values.items():
      # .get: keys are per task, so a task scored by a different metric simply has none.
      print(task_id, [(a.trial_id, a.value) for a in by_output.get("reward.score", [])])

  A value of None is a trial that died before scoring — a trial that did not pass. A trial whose metric failed is absent entirely, because that leaves it unmeasured rather than unsuccessful. An omitted optional output is also absent; it is not stored as None. Look up by trial_id rather than by position: these rules mean lists for different outputs of one task need not be the same length.

  Values keep the type the metric produced them in — a count stays an int, a flag stays a bool, and a judge’s verdict stays a str. Each record’s value_type says which it is (number, label or missing), which is what tells a real NaN apart from a label that reads "NaN", since strict JSON has no NaN literal and both travel as strings. Before doing arithmetic, project with numeric_metric_values, which drops labels and keeps a dead trial’s None:

  from nemo_evaluator_sdk.agent_eval.results import numeric_metric_values
  records = result.summary.task_metric_values["task-47"]["reward.score"]
  scores = numeric_metric_values(records)  # [1.0, None, 0.0]

• summary.task_outcomes(metric_name=None) — the same data as models rather than nested dicts, sorted by task then metric, each naming its own task_id and metric_name. Pass a "<metric.type>.<output>" to narrow to one metric, which is what a report over a single metric wants:

  from nemo_evaluator_sdk.agent_eval.results import numeric_metric_values
  for per_task in result.summary.task_outcomes("reward.score"):
      for outcome in per_task.outcomes:
          values = numeric_metric_values(outcome.trials)
          print(per_task.task_id, outcome.metric_name, values)
  • When you narrow, a task the metric never measured is dropped — it was scored by a different metric, so listing it would invent missing coverage.
  • A task that declared the metric but produced no usable value keeps its entry with an empty trials list, because there the coverage really is missing.
  • Unfiltered, every task is returned.

• summary.task_count, summary.trial_count, summary.score_count.

Sparse output example

Suppose task A has two trials. Its Harbor verifier emits:

a1: reward=1.0, format_ok=1.0
a2: reward=0.0, format_ok omitted

reward is measured twice. format_ok is applicable twice but measured once:

These invariants expose the effective denominator:

• For an ordinary aggregate, count + nan_count equals the applicable opportunities.
• If all optional values are omitted, the aggregate, coverage, task-value, view, and pass@k rows remain present and unestimable instead of disappearing.
• Pass@k is calculated separately for every score-like output. Measured n includes failed trials as non-passes. A failed metric or an omitted optional output is left out of n (unmeasured, not unsuccessful).
• If task A declares format_ok and task B does not, format_ok’s count, nan_count, and coverage are only over A’s trials. B is omitted from that denominator, not counted as missing.

Per-metric scores

Each entry in result.scores carries: id, run_id, task_id, trial_id, metric_type, status (e.g. completed / failed), outputs (the metric’s named outputs), diagnostics, and metadata. Use these to drill from an aggregate down to the individual (task, metric) that produced it.

Trials

Each entry in result.trials carries: id, task_id, status (completed / partial / failed), output (the agent’s final answer), evidence (trajectory, final state, logs), and metadata. Trials are the durable, scorer-agnostic record — they can be re-scored offline later.

The run bundle

Call result.persist() and it writes these files (the same data, on disk):

persist() returns a BundleLocation telling you where the bundle landed:

from pathlib import Path
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
result = await AgentEvaluator().run(
    tasks=tasks,
    target=target,
    config=AgentEvalRunConfig(work_dir=Path("./agent-eval-run")),
)
location = result.persist()
# -> ./agent-eval-run/report.html, summary.json, scores.jsonl, trials.jsonl, ...
print(location.output_dir, location.dashboard_path)

report.html is the fastest way to eyeball a run or hand it to someone else; the .jsonl files are convenient for loading scores and trials into your own tooling.

An omitted optional output remains absent in scores.jsonl and after loading the bundle. Persistence does not replace it with JSON null, a NaN label, or 0.0.

Results from platform jobs

An agent-evaluate platform job persists the same JSON and JSONL data without report.html. It publishes two named job results:

• agent-eval-results — the complete bundle, including tasks, trials, scores, metadata, and summary.
• summary — summary.json by itself for lightweight retrieval.

Wait for the platform job to reach a terminal state before reading either result:

job.wait_until_done()
stored = client.evaluator.agent_eval_results.retrieve("<result-name>")
client.files.download(remote_path=stored.bundle_ref, local_path="agent-eval-run")

You can also download the bundle through the Jobs CLI:

nemo jobs results download agent-eval-results \
  --job <job-name> \
  --output-file agent-eval-results.tar.gz

The queryable agent_eval_results record stores aggregates, coverage, target identity, and the bundle reference. Individual trials remain in trials.jsonl inside the bundle.

Related