Reports & Results

This page maps everything SkillEvaluator writes to disk and how to read it. Nothing here requires an API key — it is the output contract for every command, from a keyless quality-check to a full live evaluation.

Two kinds of artifacts exist. Tier 1 and Tier 2 commands render reports — one per format, selected with -r. Tier 3 (tier3 evaluate) additionally writes a results tree: a timestamped directory of raw scores, trials, and lift data that view and compare read back later.

Harbor is the open-source agent evaluation framework that runs those Tier 3 trials. SkillEvaluator collects its job artifacts into the results tree described on this page.

Report formats

Every Tier 1 and Tier 2 command accepts -r/--report and -o/--output-dir. The flag is repeatable and accepts comma- or space-separated values (-r cli,json,html or -r cli json html). By default — with no -r at all — the terminal shows a compact summary and the run writes html and json report files. An explicit -r is honored exactly, including -r cli, which prints the full terminal report and writes no files. Files land in reports/ unless you override it with -o. Report output must be outside the source tree whose publication identity it describes. When the default reports/ would be inside the target (for example, after cd my-skill), SkillEvaluator safely reserves the sibling directory my-skill-reports/ instead. A catalog uses the same rule and relocates to <catalog>-reports/. An explicit in-target -o is rejected; choose an external directory instead. Existing nonempty sibling directories are reused only when their SkillEvaluator ownership marker is valid, preventing the default from overwriting an unrelated directory. For Tier 3, an output directory inside the target repository or an included skill root must likewise be new, empty, or already carry a valid ownership marker. This applies to linked-context staging as well as --copy-repo. A nonempty unowned directory is rejected so selecting an authored subtree cannot silently remove that subtree from the agent’s runtime context.

For compatibility, commands launched from a parent directory keep the familiar layout when the default output is safe to claim: skillevaluator validate ./my-skill writes to ./reports/ when that directory is absent, empty, or already authenticated as SkillEvaluator output. If ./reports/ is nonempty and unowned, the default relocates to ./my-skill-reports/ instead of writing into unrelated content.

For validate, that compact default is a pipeline view — a per-check ticker plus a summary. Pass --verbose to get the full per-check detail stream in the terminal instead.

skillevaluator validate ./my-skill -r json,html,markdown,sarif -o reports

For gating a pipeline on the JSON output, see Gate Your CI.

File naming

Report filenames follow two patterns, both under the output directory:

The extension matches the format (.json, .html, .md, .sarif.json); the cli format prints to the terminal and never writes a file. A full validate run’s timestamp is YYYYMMDDHHMMSS, so -r json produces something like skillevaluator-output-20260709141530.json and never clobbers the previous run.

HTML report

The HTML report is one self-contained file. It opens on the answer and puts the evidence below it:

• Header. The skill name is the page title. The header shows one verdict, the date and version, and Download JSON.
• Summary. The verdict with a one-line reason, four numbers (checks passed, findings, blocking critical and high findings, quality score with its minimum), and Fix first: up to five places to fix, most severe first, each linking to its evidence. When a required scanner did not finish, it is listed first with the scanner’s own reason, such as a timeout. Two scanners reporting the same file and line take one entry.
• Findings and evidence. Every check with its findings and suggested fixes. Failed checks come first. Checks that report a raw ID, such as SCRIPT_LINT, show a plain name with the ID beside it. Scanner diagnostics and gate failures are always shown and printed; raw scanner lines that only repeat a finding are folded away.
• Run details. Scope, the target path, provenance and the gate policy.

A finding is one problem a check reports at one location, with a severity. A finding in a failed check blocks the verdict; a finding in a passed check is advisory. A quality score below the configured minimum fails the Quality score check, so it also blocks the verdict.

The HTML report, the CI index and the Pages history use the same verdict words: Passed, Changes required, Incomplete, Skipped and Neutral. When no check ran, the verdict is Skipped, not Passed. A tier whose gate does not block, such as deduplication with --no-block-on-dedup or live evaluation without --block-on-agent-eval, shows Failed (advisory) when it fails, because it does not require changes. Publication eligibility is shown next to the verdict, because a report can pass its checks and still not be eligible for publication. The report follows the system light or dark theme until you pick one, and always prints dark text on white paper.

BENCHMARK.md

When the target is a skill, validate always writes BENCHMARK.md to the output directory — regardless of which -r formats you picked, and even when validation fails. It is a decision-first publication card: the overall verdict and recommendation come first, followed by per-tier evidence, evaluator and dataset provenance, agent/model details, and a dimension-by-dimension Baseline → Skill Uplift table when Tier 3 ran. Public cards omit validation profile names and absolute host paths.

Table scores are rounded half-up to one decimal place. A score near a dimension verdict threshold keeps as many extra decimals as it needs to stay in its band, so 0.4995 is shown as 49.95%, not as a passing 50.0%. Uplift is the difference of the two displayed scores, so a cell such as 58.4% → 94.8% (+36.4 points) always adds up. A score outside 0–1 is shown as Not available.

The card carries one overall verdict — PASS, NEUTRAL, FAIL, or INCOMPLETE. It fails closed: if a required scanner produced no trustworthy evidence, the verdict is INCOMPLETE and the card explicitly says not to use it to recommend publication. A Publication Recommendation section appears only on a clean PASS; NEUTRAL means the evidence is complete but at least one required dimension remains below the pass band.

Tier 1 is always required for publication. Tier 2 and Tier 3 are also required by default, so a missing, skipped, or incomplete required tier produces INCOMPLETE rather than a publication recommendation. Catalog orchestration may explicitly persist benchmark_policy.tier2_required = false or benchmark_policy.tier3_required = false; the card discloses each exception instead of implying that the tier ran. An optional tier that is present but incomplete still makes the card INCOMPLETE because malformed evidence cannot certify publication. A default-passed result object is not sufficient. Tier 1 and Tier 2 evidence must carry a valid publication_evidence marker written by a built-in command wrapper, and every recognized result must record a success, finding, or positive check count that proves it executed. Tier 1 must also cover every default check that applies to the content type: schema, security, PII, license, code integrity, and Unicode, plus version, quality, and lint for skills. A run narrowed with --checks is reported but cannot certify Tier 1. Legacy and custom results remain visible, and their failures still block publication, but they cannot satisfy a required tier or create a waiver. Generic per-result optional metadata is not a publication waiver; only the resolved benchmark_policy from recognized evidence can make Tier 2 or Tier 3 optional. An unrecognized true claim can conservatively require a tier, but an unrecognized false claim cannot waive one.

Agent-evaluation payload metadata and its summary are duplicated claims at the same precedence level and must agree. A conflict between valid booleans at that level resolves to required evidence. Invalid entries do not count as policy values; when the level contains no valid boolean, resolution falls through to the next lower-precedence source, and no valid value anywhere means required. The consistent payload/summary claim takes precedence over recognized peer result metadata. Conflicting booleans among peer results likewise resolve to required evidence so aggregation order cannot create a publication waiver. Every Tier 1, Tier 2, and Tier 3 result used for publication carries a publication_target with the exact NFC filesystem-entry name, a SHA-256 source digest, and the versioned skill-evaluator-source-tree/2 recipe. The recipe binds author-owned relative paths, node kinds, executable bits, and file contents while excluding only known generated artifacts. In particular, authored evals/ inputs are included. The v2 exclusions are any-depth .git, .venv, __pycache__, and node_modules; root .evals, .results, .versions, results, and versions; evals/results; and the generated root files BENCHMARK.md, skill-card.md, and skill.oms.sig. Filesystem case aliases are treated according to the host filesystem, while a case-distinct authored name on a case-sensitive filesystem remains covered. A same-named file below another authored directory is still included. Tier 3 runtime projection and full --copy-repo staging omit the same generated inputs so excluded generated bytes inside the target cannot affect evaluation evidence. Tier 1 and Tier 2 skip the generated-artifact names (results, versions, evals, their dot variants, BENCHMARK.md, skill-card.md, and skill.oms.sig) at any depth. When one of those names appears below the target root, the digest would certify content that no scan read, so the Tier 1 run records the path and publication stays INCOMPLETE until it is renamed. The root evals/ directory holds Tier 3 inputs and is outside the Tier 1 scans by design. Split-job results must match this identity exactly. Tier 3 runs that stage any linked/full repository context or external reference/workspace skills, or task inputs and custom environments that carry .git, .venv, __pycache__, or node_modules entries, remain runnable but are marked INCOMPLETE, because that projection is not fully determined by this target-only publication identity. Legacy anonymous results remain readable, but their publication status is INCOMPLETE.

Built-in Tier 1 and Tier 2 results also carry a versioned publication_evidence object with exact schema_version, producer, tier, and canonical check_id fields. Reporters accept only the documented built-in producer/check combinations. This marker is provenance inside trusted local or CI artifacts; it is not a cryptographic signature for accepting result files from an untrusted party.

Publication model, evaluator, environment, and skill identities reject reserved placeholder text after Unicode confusable-skeleton comparison. This prevents a visually spoofed placeholder such as one containing a Cyrillic or Greek letter from becoming publication evidence while preserving ordinary multilingual identities.

Payload and summary policy claims are accepted only when their containing evidence matches that exact publication target; evidence from another source cannot waive a required tier. A skipped Tier 3 payload must satisfy the same target binding before its policy can waive evidence. A completed Tier 3 payload also duplicates its run-owned run_id in the payload and summary; missing or contradictory run identity is incomplete publication evidence. When multiple complete Tier 3 payloads disagree, reporters select the most conservative effective verdict (FAIL, then NEUTRAL, then PASS) before applying deterministic tie-breakers.

Execution controls do not rewrite that publication policy. --no-dedup and --tiers 1,3 select which validators run, while --no-block-on-dedup changes whether Tier 2 affects the command exit code. None of those flags persists a publication waiver. Without explicit policy metadata, a run that omits Tier 2 therefore exits according to its CLI gate but still produces an INCOMPLETE publication card.

Evaluated source provenance

The Evaluation Metadata section distinguishes the source tree that was evaluated from the evaluator build that evaluated it:

- Evaluator version: `0.8.2`
- Evaluated source: `NVIDIA/NVFlare`
- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`
- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...`

Without that separation, two skills evaluated from different repositories by the same evaluator container publish cards whose only recorded revision is the shared container tag, and a reader cannot tell which version of a skill was evaluated or whether the card has gone stale.

The identity is supplied by the orchestration input and carried through the evaluation-result contract unchanged. It is never inferred from repository state while rendering, because the tree that renders a card is the evaluator checkout rather than the evaluated skill’s source. The supported seams are:

• --evaluated-source-repository, --evaluated-source-revision and --evaluator-container-revision, accepted by both validate and tier3 evaluate, which record the identity on the card and, for a Tier 3 run, persist it into that run’s run_config.json;
• the evaluated_source argument to build_agent_eval_payload or agent_eval_result_from_directory;
• an evaluated_source object in the run’s run_config.json. Note that SkillEvaluator writes that file itself, so a publication pipeline using this route has to add the key to the generated run artifact before the card is rendered;
• metadata["evaluated_source"] on any validation result, which is how a card that publishes without a completed Tier 3 run records its identity.

Where any two carriers supply a field and disagree, rendering fails rather than silently choosing one, so the order results happen to arrive in can never decide which source tree a card names. Evaluated source revision records a full Git object id of 40 or 64 hex characters, or a canonical sha256/sha384/sha512 digest of the evaluated skill content at that algorithm’s own width when a run has no upstream commit. A short revision prefix is refused, because it can grow a second match as a tree grows. Evaluator container revision records an image reference validated component by component: a repository path of at most 255 characters, an optional tag of at most 128 characters, and an optional digest at its own algorithm’s width. The 255 bound measures the path once the registry host has been split off it. Path components are lower case, as the OCI grammar requires, while a registry host may use any case and is read as a host only when it is localhost, carries a dot, or carries a port. A bare name is refused because it identifies a repository rather than the build that ran, and a published PASS additionally requires the digest, or a full implementation revision, because a tag can be repointed after the card is published. Fields the orchestration did not supply render as not recorded (not supplied by the orchestration input).

Keep the file with the skill and refresh it whenever the skill, eval dataset, agent/model, evaluator version, scoring policy, attempt policy, or execution environment materially changes. Follow the rollout and backfill runbook when regenerating existing published cards.

Safety guarantees

Report writes are hardened: every file is written atomically (a temporary file renamed into place), and SkillEvaluator refuses to write through symlinks, Windows junctions, or paths with parent traversal. A run that is interrupted mid-write never leaves a half-written report, and a report path can’t be used to redirect output somewhere unexpected.

Tier 3 results on disk

tier3 evaluate does not use -r/-o. It writes a results tree to the first of:

1. --results-dir <dir> on the command line
2. the SKILLEVALUATOR_RESULTS_DIR environment variable
3. <skill>/evals/results/ (the default)

With an external root (the first two options), each skill gets its own subdirectory — runs land under <dir>/<skill-name>/, not directly under <dir>. Configured roots must be outside the skill. The canonical <skill>/evals/results/ path is the only supported in-skill result root; this keeps generated runs outside the versioned source-tree digest used to combine publication evidence.

Each run creates a collision-safe directory named by its run ID (YYYYMMDD_HHMMSS_<pid>_<12-hex-nonce>). Where symlinks are supported, a latest symlink points at the newest run; otherwise readers select the newest completed run directly:

• result.json — the bounded run summary: run ID, per-agent scores and lift, pass@k aggregates, the resolved run configuration, and execution status. When repeated case, security, or failure details would make this browser-facing file too large, result_projection records exactly which fields were omitted and points to the canonical per-agent JSON artifacts by relative path, byte size, and SHA-256 digest. The in-process return value remains complete; readers rebuilding a report from disk follow those artifact references.
• run_config.json / attempt_policy.json — exactly how the run was configured: environment, attempts, concurrency, grading mode, pass threshold.
• report.html — the human-readable report, generated automatically at the end of every run.
• comparison.json — the cross-agent comparison table; written only when you ran more than one agent.
• <agent>/lift.json and <agent>/pass_at_k_lift.json — per-metric deltas between the two arms; written only when both arms produced scores (a --skip-baseline or --skip-with-skill run has neither).
• <agent>/{with-skill,without-skill}/ — each arm’s summary.json (scores, dimensions, pass@k, trial counts) and a trials/ directory of per-trial artifacts: the trial’s reward.json, Harbor’s trajectory.json when available, and redacted, size-capped diagnostic copies such as trial.log, exception.txt, and agent logs (codex.txt, opencode.txt). With standard grading, reward.json records trajectory_source (the trajectory or agent log that was graded) and, under each LLM judge’s details, an omitted record showing whether its evidence was truncated to fit the evidence budget.

Reading a live-eval report

The terminal summary

A finished tier3 evaluate run prints two tables and closes with a compact Artifacts panel. Results by Evaluator lists each evaluation signal with its per-arm scores and Skill Lift; Results by Dimension rolls those signals up into the five dimensions below. A metric the run couldn’t score renders as NO SCORE — it is never coerced to 0.0 — and unscored metrics are omitted from the averages, so overall_score can be null in the JSON. In a baseline-only run (--skip-with-skill) the with-skill and Skill Lift cells read skipped instead, because that arm never ran. The Artifacts panel shows the HTML report path, the output directory, and — only when a retained Harbor jobs directory exists — a ready-to-paste harbor-view command.

The five dimensions

With default or default_plus_custom grading, reports lead with five dimensions, each answering one question:

Tool-efficiency checks classify explicit searches rooted at / as wasted work, whether or not the command names the expected skill or script. Scoped searches and / used as a search pattern retain their existing classification. This rule recognizes common search commands, including ones with redirections or wrapped in sh -c; it does not interpret expansions, comments, here-documents or other dynamically constructed shell programs.

Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated standard grader rejects non-finite values and clamps numeric scores with max(0.0, min(1.0, score)). A failed required judge produces no score instead of a numeric zero. The fixed dimension bands are pass at 0.50 or above, neutral from 0.40 to below 0.50, and fail below 0.40.

run_config.json records the configured judge provider, model, source, and catalog-verification status separately from the agent model. result.json embeds that run configuration alongside the scores. This provenance makes the LLM-judged values traceable, but it is not evidence that the judges are validated. The repository does not publish inter-rater agreement, calibration results, or a comparison against human-labeled ground truth.

The Quality Score displayed out of 100 in the HTML report is the Tier 1 static-analysis score. It is not an input to the Tier 3 dimensions, overall score, or Skill Lift.

The standard overall score gives each dimension one equal vote:

overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
Effectiveness = (goal_accuracy + behavior_check) / 2

This formula is shared by the report, per-attempt pass@k thresholding, Skill Lift, comparisons, and best-agent selection. Standard artifacts identify it as skill-evaluator-dimension-mean-v1 in score_policy and in the persisted attempt policy. Older metric sets and partial historical artifacts keep their legacy policy rather than being rescored as current data.

Overall comparisons require both the same scoring policy and the same metric set. A shared policy name alone does not make five-metric and six-metric means comparable. When two conditions use different metric sets under the same policy, their individual scores and shared per-metric deltas remain available, but overall Skill Lift and the pass@k comparison are omitted. Reports containing agents scored on different standard metric sets retain their individual scores and mark the overall comparison as incomplete, without selecting a best agent. A custom-only agent, one whose conditions use only its grader’s own overall (custom-overall-v1), is still ranked against standard agents by that score, and tier3 compare shows it beside them. tier3 compare refuses standard agents scored under different policies or metric sets.

Skill Lift

Skill Lift is with-skill overall score - without-skill overall score for comparable cases. It uses the same score units as both arms and ranges from -1.0 to +1.0. Because live agent runs are noisy, especially at low attempt counts, small deltas are kept neutral. The verdict bands:

These bands color-code Skill Lift as diagnostic evidence. They do not determine the combined validate --tier3 or BENCHMARK.md verdict. The overall verdict comes from the five dimensions: PASS when at least one successful agent has all five dimensions at 0.50 or above; NEUTRAL when no agent passes but at least one has no dimension below 0.40; and FAIL when every successful agent has a dimension below 0.40.

A +0.03 lift therefore remains neutral under these bands. Lift requires both arms; if you ran with --skip-baseline, there is no lift to report. In custom_only grading, your own grader defines the score and lift — see Custom Graders & Tasks.

When the HTML report has no lift for an agent or dimension, its label says why. no baseline means that row has no baseline score, for example after --skip-baseline. comparison unavailable means a baseline score was recorded but cannot be compared with the with-skill score, for example because the arms used different score policies or no case kept matching judge models.

For each LLM metric (accuracy, goal_accuracy, and behavior_check), cases that used different judge models across the arms are excluded from that metric’s lift. This includes model changes across repeated attempts or the scoring steps of a multi-step task. lift.json records the excluded case counts by metric in judge_mixed_cases. Arm scores still describe all scored cases, while evaluator, dimension, and overall lift use the eligible comparisons. A reported lift can therefore differ from subtracting the two displayed arm scores, even reversing their apparent direction. When no complete comparison with matching judge models remains, the affected metric, dimension, and overall lift are unavailable rather than zero.

If every positive with-skill trial has confirmed zero activation, the agent’s result is INCONCLUSIVE: skill never activated. Reports keep its arm scores and diagnostic trials but suppress its evaluator, dimension, overall, custom, and pass@k lift comparisons and skill-quality advice. Negative controls do not count as activation opportunities; missing or unsupported evidence remains unknown. Eligibility is determined separately for each agent, so another agent with observed activation can still contribute a comparison.

Measured agent resources

Each agent’s canonical JSON payload contains a resources object. The HTML table shows total prompt plus completion tokens and agent execution seconds for each arm, telemetry coverage, and the with-skill-minus-baseline difference. Routing/tool-productivity efficiency remains a separate score. Missing counters and incomplete arms produce null totals or deltas (N/A in HTML). Resource deltas require matching case counts and complete measurements in both arms; they do not measure evaluator tokens, setup time, price, or statistical savings.

pass@k

pass@k is the reliability signal, reported separately from the dimension scores. With --n-attempts k, each eval case runs k times per arm, and a case counts as passed when at least one attempt clears the --pass-threshold score using the same overall formula above. Comparing pass@k across arms (in pass_at_k_lift.json) tells you whether the skill makes success more repeatable, not just whether the average score moved. Each arm also records a case-level 95% Wilson score interval for its pass rate. When both arms contain the same identified cases, pass_at_k_lift.json records the paired outcome counts — skill-only passes, baseline-only passes, both-pass, and neither-pass — plus a two-sided exact McNemar diagnostic over the discordant pairs. Partial or unidentified pairing is labeled and does not receive that exact-test result.

Each logical attempt counts once: a multi-step native task’s step rewards combine into one attempt score for averages and pass@k. When one condition mixes standard-metric and custom-only rewards, its overall score and lift use that logical-attempt overall rather than the mean of the standard dimensions, which would cover only the standard rows.

Case identifiers used as JSON keys must be printable, free of credential material, and no larger than 512 UTF-8 bytes. A reward with an unsafe identity is retained as a failed diagnostic but is not scored. For large evaluations, summary.json keeps exact pass, attempt, failure, and pairing totals while publishing bounded case, attempt, and failure-detail samples. The matching *_total, *_shown, and *_truncated fields make every omission explicit; sampling does not change the aggregate score or paired comparison.

The paired record also reports its pass-rate delta and the minimum p-value the exact test could attain with the observed number of discordant pairs. When that minimum is above 0.05, the report labels the test resolution-limited at that threshold. This is a mathematical resolution limit, not evidence of no effect; with five or fewer discordant pairs, a two-sided exact result cannot fall below 0.05 regardless of direction. The JSON always preserves compact decimal text for both probabilities. It also records a reduced rational in each *_exact field when the numerator and denominator fit both SkillEvaluator’s 4,300-digit safety cap and Python’s active integer-string conversion limit. Otherwise that field is null, *_omitted is true, and *_omitted_reason is "decimal_digit_limit". The numeric approximation is null, rather than a false zero, if binary floating-point conversion underflows.

These uncertainty fields describe the cases in one configured run. They do not make heterogeneous cases independent, correct dataset leakage, or promote a small p-value into proof that a skill generalizes. Keep the dataset, attempt policy, agent, model, and environment fixed when interpreting them.

Browse results

view

Opens the newest report.html for a skill in your browser, regenerating it from the stored run data if the file is missing.

skillevaluator view ./my-skill

compare

Prints a side-by-side terminal table of the most recent stored results per agent — score and lift per evaluator — across everything under the results root. Summaries whose execution status is anything other than succeeded are ignored, so a failed run can’t masquerade as a score.

skillevaluator compare ./my-skill

Both commands accept --results-dir to read from a non-default location.

harbor-view

Opens retained Harbor job artifacts in Harbor’s own trajectory browser — the deepest debugging view, down to individual agent transcripts. Staged-task and raw Harbor job directories are transient: they are deleted after collection by default, on success and failure alike. Pass --harbor-keep-jobs to retain them — the retention outcome is recorded in the results, and the Artifacts panel prints this command only when a retained jobs directory actually exists. Retained jobs live under _harbor-jobs/ inside the run directory.

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker --harbor-keep-jobs
skillevaluator harbor-view ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6/_harbor-jobs

(skillevaluator tier3 harbor-view is the same command under its expert alias.)

Performance dashboard

Use the optional Performance Dashboard to filter saved Tier 3 reports and retained runs, compare paired scores and resource use, and download the visible tables as CSV. The guide covers installation, comparison eligibility, measurement limits, and reloading saved inputs.

Machine-readable contract

For CI and tooling, two JSON entry points matter:

• Standalone tier3 evaluate runs — parse result.json in the run directory (or follow latest). It carries per-agent scores, dimensions, lift, pass@k, trial counts, the attempt policy, and execution status. Unscored metrics stay unscored rather than defaulting to 0.0, so overall_score can be null.
• validate --tier3 runs — the combined skillevaluator-output-<timestamp>.json report embeds the Tier 3 payload alongside the Tier 1 and Tier 2 results, so one file covers all three tiers. Each built-in Tier 1 or Tier 2 result carries its validated publication_evidence producer marker alongside finalized gating metadata, and the top-level benchmark_policy records whether Tier 2 and Tier 3 evidence is required for publication. publication_status is the compact machine-readable verdict; publication carries eligibility, reasons, and the Tier 3 evidence status. This is intentionally separate from overall_status, which continues to describe the command/process gate. Tier 3 is advisory for the CLI exit code by default and becomes blocking with --block-on-agent-eval; Tier 2 is blocking for the exit code by default and becomes advisory with --no-block-on-dedup. Those exit-code controls are separate from benchmark_policy. A validator’s independent policy failures, such as a quality score below its required minimum, appear in gate_failures when present. Check those alongside findings; changing finding severity does not waive the minimum-score gate. The terminal, HTML report, and BENCHMARK.md also show these failures.
• The evaluated source identity: every JSON report always carries a top-level evaluated_source object, resolved from every carrier the run recorded, so the JSON and BENCHMARK.md name the same source tree. It is null when the orchestration input supplied none. Carriers that name different sources abort the run before any report file is written, so a contradictory identity is never published in one format and reported in another.

Both paths speak the same dialect: standalone tier3 evaluate and validate --tier3 share one HTML renderer for report.html, and both embed the Tier 3 payload at schema version 2.0.

Exit codes and a ready-made JSON gating recipe live in Gate Your CI.

Next steps