Reading Results

AgentEvaluator().run(...) returns an AgentEvalResult. Set an output_dir and it also writes a run bundle to disk. The object and the bundle hold the same data — use the object for programmatic follow-up, and the bundle (especially report.html) to inspect or share a run.

The result object

1
result = await AgentEvaluator().run(tasks=..., target=...)

The summary

result.summary (an AgentEvalSummary):

• summary.scores.scores — the aggregates. Each is named <metric.type>.<output> (and view.<name> for a view), with mean, min, max, and std_dev. This is what the guides print:

  1
  for aggregate in result.summary.scores.scores:
  2
      print(f"{aggregate.name}: {aggregate.mean}")

• summary.metric_coverage — per metric output, how many trials were total / scored / failed / missing, so you can tell a low mean from low coverage.

• summary.task_count, summary.trial_count, summary.score_count.

Per-metric scores

Each entry in result.scores carries: id, run_id, task_id, trial_id, metric_type, status (e.g. completed / failed), outputs (the metric’s named outputs), diagnostics, and metadata. Use these to drill from an aggregate down to the individual (task, metric) that produced it.

Trials

Each entry in result.trials carries: id, task_id, status (completed / partial / failed), output (the agent’s final answer), evidence (trajectory, final state, logs), and metadata. Trials are the durable, scorer-agnostic record — they can be re-scored offline later.

The run bundle

Set output_dir and run() writes these files (the same data, on disk):

1
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
2
3
result = await AgentEvaluator().run(
4
    tasks=..., target=..., config=AgentEvalRunConfig(output_dir="./agent-eval-run"),
5
)
6
# -> ./agent-eval-run/report.html, summary.json, scores.jsonl, trials.jsonl, ...

report.html is the fastest way to eyeball a run or hand it to someone else; the .jsonl files are convenient for loading scores and trials into your own tooling.

Related