Reports & Results

This page maps everything SkillEvaluator writes to disk and how to read it. Nothing here requires an API key — it is the output contract for every command, from a keyless quality-check to a full live evaluation.

Two kinds of artifacts exist. Tier 1 and Tier 2 commands render reports — one per format, selected with -r. Tier 3 (tier3 evaluate) additionally writes a results tree: a timestamped directory of raw scores, trials, and lift data that view and compare read back later.

Harbor is the open-source agent evaluation framework that runs those Tier 3 trials. SkillEvaluator collects its job artifacts into the results tree described on this page.

Report formats

Every Tier 1 and Tier 2 command accepts -r/--report and -o/--output-dir. The flag is repeatable and accepts comma- or space-separated values (-r cli,json,html or -r cli json html). By default — with no -r at all — the terminal shows a compact summary and the run writes html and json report files. An explicit -r is honored exactly, including -r cli, which prints the full terminal report and writes no files. Files land in reports/ unless you override it with -o.

For validate, that compact default is a pipeline view — a per-check ticker plus a summary. Pass --verbose to get the full per-check detail stream in the terminal instead.

skillevaluator validate ./my-skill -r json,html,markdown,sarif -o reports

For gating a pipeline on the JSON output, see Gate Your CI.

File naming

Report filenames follow two patterns, both under the output directory:

The extension matches the format (.json, .html, .md, .sarif.json); the cli format prints to the terminal and never writes a file. A full validate run’s timestamp is YYYYMMDDHHMMSS, so -r json produces something like skillevaluator-output-20260709141530.json and never clobbers the previous run.

BENCHMARK.md

When the target is a skill, validate always writes BENCHMARK.md to the output directory — regardless of which -r formats you picked, and even when validation fails. It is a decision-first publication card: the overall verdict and recommendation come first, followed by per-tier evidence, evaluator and dataset provenance, agent/model details, and a dimension-by-dimension Baseline → Skill Uplift table when Tier 3 ran. Public cards omit validation profile names and absolute host paths.

The card carries one overall verdict — PASS, NEUTRAL, FAIL, or INCOMPLETE. It fails closed: if a required scanner produced no trustworthy evidence, the verdict is INCOMPLETE and the card explicitly says not to use it to recommend publication. A Publication Recommendation section appears only on a clean PASS; NEUTRAL means the evidence is complete but at least one required dimension remains below the pass band. Tier 3 evidence is publication-required by default, so a missing, skipped, or incomplete live evaluation produces INCOMPLETE rather than a publication recommendation. A catalog may explicitly persist benchmark_policy.tier3_required = false; the card discloses that exception instead of implying that Tier 3 ran.

Evaluated source provenance

The Evaluation Metadata section distinguishes the source tree that was evaluated from the evaluator build that evaluated it:

- Evaluator version: `0.8.2`
- Evaluated source: `NVIDIA/NVFlare`
- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`
- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...`

Without that separation, two skills evaluated from different repositories by the same evaluator container publish cards whose only recorded revision is the shared container tag, and a reader cannot tell which version of a skill was evaluated or whether the card has gone stale.

The identity is supplied by the orchestration input and carried through the evaluation-result contract unchanged. It is never inferred from repository state while rendering, because the tree that renders a card is the evaluator checkout rather than the evaluated skill’s source. The supported seams are:

• --evaluated-source-repository, --evaluated-source-revision and --evaluator-container-revision, accepted by both validate and tier3 evaluate, which record the identity on the card and, for a Tier 3 run, persist it into that run’s run_config.json;
• the evaluated_source argument to build_agent_eval_payload or agent_eval_result_from_directory;
• an evaluated_source object in the run’s run_config.json. Note that SkillEvaluator writes that file itself, so a publication pipeline using this route has to add the key to the generated run artifact before the card is rendered;
• metadata["evaluated_source"] on any validation result, which is how a card that publishes without a completed Tier 3 run records its identity.

Where any two carriers supply a field and disagree, rendering fails rather than silently choosing one, so the order results happen to arrive in can never decide which source tree a card names. Evaluated source revision records a full Git object id of 40 or 64 hex characters, or a canonical sha256/sha384/sha512 digest of the evaluated skill content at that algorithm’s own width when a run has no upstream commit. A short revision prefix is refused, because it can grow a second match as a tree grows. Evaluator container revision records an image reference validated component by component: a repository path of at most 255 characters, an optional tag of at most 128 characters, and an optional digest at its own algorithm’s width. The 255 bound measures the path once the registry host has been split off it. Path components are lower case, as the OCI grammar requires, while a registry host may use any case and is read as a host only when it is localhost, carries a dot, or carries a port. A bare name is refused because it identifies a repository rather than the build that ran, and a published PASS additionally requires the digest, or a full implementation revision, because a tag can be repointed after the card is published. Fields the orchestration did not supply render as not recorded (not supplied by the orchestration input).

Keep the file with the skill and refresh it whenever the skill, eval dataset, agent/model, evaluator version, scoring policy, attempt policy, or execution environment materially changes. Follow the rollout and backfill runbook when regenerating existing published cards.

Safety guarantees

Report writes are hardened: every file is written atomically (a temporary file renamed into place), and SkillEvaluator refuses to write through symlinks, Windows junctions, or paths with parent traversal. A run that is interrupted mid-write never leaves a half-written report, and a report path can’t be used to redirect output somewhere unexpected.

Tier 3 results on disk

tier3 evaluate does not use -r/-o. It writes a results tree to the first of:

1. --results-dir <dir> on the command line
2. the SKILLEVALUATOR_RESULTS_DIR environment variable
3. <skill>/evals/results/ (the default)

With an external root (the first two options), each skill gets its own subdirectory — runs land under <dir>/<skill-name>/, not directly under <dir>.

Each run creates a collision-safe directory named by its run ID (YYYYMMDD_HHMMSS_<pid>_<12-hex-nonce>). Where symlinks are supported, a latest symlink points at the newest run; otherwise readers select the newest completed run directly:

• result.json — the run summary: run ID, per-agent scores and lift, pass@k, the resolved run configuration, and execution status.
• run_config.json / attempt_policy.json — exactly how the run was configured: environment, attempts, concurrency, grading mode, pass threshold.
• report.html — the human-readable report, generated automatically at the end of every run.
• comparison.json — the cross-agent comparison table; written only when you ran more than one agent.
• <agent>/lift.json and <agent>/pass_at_k_lift.json — per-metric deltas between the two arms; written only when both arms produced scores (a --skip-baseline run has neither).
• <agent>/{with-skill,without-skill}/ — each arm’s summary.json (scores, dimensions, pass@k, trial counts) and a trials/ directory of per-trial artifacts.

Reading a live-eval report

The terminal summary

A finished tier3 evaluate run prints two tables and closes with a compact Artifacts panel. Results by Evaluator lists each evaluation signal with its per-arm scores and Skill Lift; Results by Dimension rolls those signals up into the five dimensions below. A metric the run couldn’t score renders as NO SCORE — it is never coerced to 0.0 — and unscored metrics are omitted from the averages, so overall_score can be null in the JSON. The Artifacts panel shows the HTML report path, the output directory, and — only when a retained Harbor jobs directory exists — a ready-to-paste harbor-view command.

The five dimensions

With default or default_plus_custom grading, reports lead with five dimensions, each answering one question:

Dimension scores are 0.0–1.0 rollups of the underlying evaluation signals. The verdict bands are fixed: a dimension passes at 0.50 or above, is neutral from 0.40 to below 0.50, and fails below 0.40. The judge runs on the configured provider model; override it per run as described in Tier 3: Live Evaluation.

Skill Lift

Skill Lift is the with-skill score minus the without-skill baseline — the direct measurement of what your skill contributes. Because live agent runs are noisy (especially at low attempt counts), small deltas are deliberately kept neutral. The verdict bands:

These bands color-code Skill Lift as diagnostic evidence. They do not determine the combined validate --tier3 or BENCHMARK.md verdict. The overall verdict comes from the five dimensions: PASS when at least one successful agent has all five dimensions at 0.50 or above; NEUTRAL when no agent passes but at least one has no dimension below 0.40; and FAIL when every successful agent has a dimension below 0.40.

So a +0.03 lift isn’t a pass — it’s noise until more attempts say otherwise. Lift requires both arms; if you ran with --skip-baseline, there is no lift to report. In custom_only grading, your own grader defines the score and lift — see Custom Graders & Tasks.

pass@k

pass@k is the reliability signal, reported separately from the dimension scores. With --n-attempts k, each eval case runs k times per arm, and a case counts as passed when at least one attempt clears the --pass-threshold score. Comparing pass@k across arms (in pass_at_k_lift.json) tells you whether the skill makes success more repeatable, not just whether the average score moved. Each arm also records a case-level 95% Wilson score interval for its pass rate. When both arms contain the same identified cases, pass_at_k_lift.json records the paired outcome counts — skill-only passes, baseline-only passes, both-pass, and neither-pass — plus a two-sided exact McNemar diagnostic over the discordant pairs. Partial or unidentified pairing is labeled and does not receive that exact-test result.

The paired record also reports its pass-rate delta and the minimum p-value the exact test could attain with the observed number of discordant pairs. When that minimum is above 0.05, the report labels the test resolution-limited at that threshold. This is a mathematical resolution limit, not evidence of no effect; with five or fewer discordant pairs, a two-sided exact result cannot fall below 0.05 regardless of direction. The JSON always preserves compact decimal text for both probabilities. It also records a reduced rational in each *_exact field when the numerator and denominator fit both SkillEvaluator’s 4,300-digit safety cap and Python’s active integer-string conversion limit. Otherwise that field is null, *_omitted is true, and *_omitted_reason is "decimal_digit_limit". The numeric approximation is null, rather than a false zero, if binary floating-point conversion underflows.

These uncertainty fields describe the cases in one configured run. They do not make heterogeneous cases independent, correct dataset leakage, or promote a small p-value into proof that a skill generalizes. Keep the dataset, attempt policy, agent, model, and environment fixed when interpreting them.

Browse results

view

Opens the newest report.html for a skill in your browser, regenerating it from the stored run data if the file is missing.

skillevaluator view ./my-skill

compare

Prints a side-by-side terminal table of the most recent stored results per agent — score and lift per evaluator — across everything under the results root. Summaries whose execution status is anything other than succeeded are ignored, so a failed run can’t masquerade as a score.

skillevaluator compare ./my-skill

Both commands accept --results-dir to read from a non-default location.

harbor-view

Opens retained Harbor job artifacts in Harbor’s own trajectory browser — the deepest debugging view, down to individual agent transcripts. Staged-task and raw Harbor job directories are transient: they are deleted after collection by default, on success and failure alike. Pass --harbor-keep-jobs to retain them — the retention outcome is recorded in the results, and the Artifacts panel prints this command only when a retained jobs directory actually exists. Retained jobs live under _harbor-jobs/ inside the run directory.

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker --harbor-keep-jobs
skillevaluator harbor-view ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6/_harbor-jobs

(skillevaluator tier3 harbor-view is the same command under its expert alias.)

Machine-readable contract

For CI and tooling, two JSON entry points matter:

• Standalone tier3 evaluate runs — parse result.json in the run directory (or follow latest). It carries per-agent scores, dimensions, lift, pass@k, trial counts, the attempt policy, and execution status. Unscored metrics stay unscored rather than defaulting to 0.0, so overall_score can be null.
• validate --tier3 runs — the combined skillevaluator-output-<timestamp>.json report embeds the Tier 3 payload alongside the Tier 1 and Tier 2 results, so one file covers all three tiers. Each result carries finalized gating metadata. Tier 3 is advisory by default and becomes blocking with --block-on-agent-eval; Tier 2 is blocking by default and becomes advisory with --no-block-on-dedup.
• The evaluated source identity: every JSON report always carries a top-level evaluated_source object, resolved from every carrier the run recorded, so the JSON and BENCHMARK.md name the same source tree. It is null when the orchestration input supplied none. Carriers that name different sources abort the run before any report file is written, so a contradictory identity is never published in one format and reported in another.

Both paths speak the same dialect: standalone tier3 evaluate and validate --tier3 share one HTML renderer for report.html, and both embed the Tier 3 payload at schema version 2.0.

Exit codes and a ready-made JSON gating recipe live in Gate Your CI.

Next steps