> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/skills/skillevaluator/_mcp/server.

# BENCHMARK.md Rollout and Backfill

> Regenerate published benchmark cards safely after scoring, renderer, or publication-policy changes.

Use this runbook after a release containing the scoring contract, report
redesign, and publication gate is available. Existing `BENCHMARK.md` files are
generated artifacts; merging the renderer does not rewrite cards already
published in skill repositories.

## Before and after

The legacy card led with dense tier output and mixed verdict evidence into the
details. The replacement puts the publication decision first, labels uplift
unambiguously, and moves methodology behind the evidence reviewers need most.

| Before: legacy card                                                                                  | After: decision-first publication card                                                                     |
| ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| ![Legacy BENCHMARK.md card before the rollout](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-skillevaluator.docs.buildwithfern.com/de540cc511ce79b844cbdf69407fa6b7a25e81c908d019f794bfafab483adbf0/_dot_dot_/docs/assets/benchmark-card-before.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T104210Z&X-Amz-Expires=604800&X-Amz-Signature=2659f37ef083cf0fc2a60924aae1e5da86e258b517c7a0ec7c5dd7581d6db3a0&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) | ![Decision-first BENCHMARK.md card after the rollout](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/nvidia-skillevaluator.docs.buildwithfern.com/3f2bff67ee02b30184162936a3e773fbf2a49fbe294d95ac8da7dd017f9112b1/_dot_dot_/docs/assets/benchmark-card-after.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260920T104210Z&X-Amz-Expires=604800&X-Amz-Signature=db6f2d256563191182a5f638cd7cbe61e5e65b298f90cbd18c59c0fbab98ddd4&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) |

## 1. Pin the evaluator

Install the merged SkillEvaluator release or immutable commit and record it for
the rollout:

```bash
skillevaluator --version
git -C /path/to/skill-evaluator rev-parse HEAD
```

Do not mix evaluator versions within one catalog backfill.

## 2. Inventory existing cards

From the catalog root:

```bash
find ./skills -name BENCHMARK.md -type f -print | sort
```

Record the skill, previous card path, intended Tier 2 catalog, Tier 3 agents,
attempt policy, and environment. Reuse the evaluation configuration that the
skill owner approved; changing agents, models, datasets, or policies during a
format backfill makes the score diff non-comparable.

## 3. Regenerate into a review directory

Generate reports outside the skill tree first:

```bash
skillevaluator validate ./skills/example-skill \
  --external \
  --agent-eval \
  --agents claude-code,codex \
  --env-mode docker \
  --output-dir ./benchmark-backfill/example-skill
```

Tier 3 is publication-required by default. Omitting `--agent-eval` therefore
produces an `INCOMPLETE` card with Tier 3 marked `NOT RUN`, and never a
publication recommendation. If a catalog intentionally makes Tier 3 optional,
its orchestration must persist `benchmark_policy.tier3_required = false`; the
card then discloses that policy. Do not reuse a prior live score while changing
only its label or date. If a baseline was not run, keep the explicit “uplift
unavailable” state.

The candidate card is:

```text
benchmark-backfill/example-skill/BENCHMARK.md
```

## 4. Review semantic and presentation diffs

Compare the existing and candidate cards:

```bash
diff -u \
  ./skills/example-skill/BENCHMARK.md \
  ./benchmark-backfill/example-skill/BENCHMARK.md
```

Expected presentation changes include:

* the verdict and recommendation moving to the top;
* `Baseline → Skill Uplift` cells with uplift labeled in percentage points;
* removal of the `Num` column;
* explicit `NOT RUN`, `SKIPPED`, and `INCOMPLETE` tier states;
* collapsible methodology and non-blocking observations;
* evaluator version, dataset digest, task composition, Tier 3 requirement,
  isolation wording, and freshness copy.

Investigate any numerical change. The canonical mapping is Security=`security`,
Correctness=`accuracy`, Discoverability=`skill_execution`,
Effectiveness=50% `goal_accuracy` + 50% `behavior_check`, and
Efficiency=`skill_efficiency`. A format-only rollout must not silently accept a
different dataset or agent/model.

A dimension passes at 50%. The overall verdict passes only when every configured
dimension passes for at least one supported agent; lift is diagnostic evidence
and does not override that gate.

## 5. Run the publication gate

Scan all candidates before copying them into skills:

```bash
python scripts/ci/check_public_benchmarks.py \
  --require-files \
  ./benchmark-backfill
```

Add `--require-source-provenance` for any tree whose cards were generated with
the evaluated-source identity supplied. SkillEvaluator's own CI runs the scan
with the flag. It requires every scanned card to carry `Evaluated source`,
`Evaluated source revision` and `Evaluator container revision`, and fails any
card publishing a `PASS` that does not record the repository and revision it
evaluated, or whose evaluator container is named by a mutable tag rather than
pinned by digest. That includes a card whose Tier 3 row is skipped under an
`optional by policy` contract. The flag stays opt-in because cards generated
before the identity contract existed cannot satisfy it; without it the scan
behaves exactly as it did before.

Supply the identity when regenerating a card:

```bash
skillevaluator validate ./my-skill \
  --evaluated-source-repository NVIDIA/NVFlare \
  --evaluated-source-revision 2263a2ebdab903e87f7e7c0a001d22c3a926a9cf \
  --evaluator-container-revision ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...
```

The linter rejects configured known leak patterns such as retired product
identities, any validation-profile metadata line, common absolute home paths,
ambiguous legacy uplift cells, the old `Num` column, missing metadata/decision
sections, and a publication `PASS` without completed required Tier 3 evidence.
A clean result means none of those configured patterns matched; it is a fixture
regression guard, not proof that a card is safe to publish. Keep human review
and the repository's broader boundary and security checks in the promotion
workflow.

## 6. Promote and verify

After review, copy each candidate beside its skill, then rerun the gate over the
catalog:

```bash
cp ./benchmark-backfill/example-skill/BENCHMARK.md \
  ./skills/example-skill/BENCHMARK.md

python scripts/ci/check_public_benchmarks.py --require-files ./skills
git diff --check
git diff -- ./skills/example-skill/BENCHMARK.md
```

Commit regenerated cards in reviewable batches. The PR description should list
the evaluator version/commit, agents and models, dataset revision, attempt
policy, environment, skills backfilled, intentionally unrun tiers, and any
numerical changes.

## 7. Freshness ownership

Regenerate a card when any of these inputs changes:

* skill content or dependencies;
* evaluation dataset or expected behavior;
* target agent or model;
* evaluator version or scoring policy;
* attempt count/pass threshold;
* execution environment or isolation mode.

The generated evaluation date must come from the live run artifact. Older
artifacts without an unambiguous timestamp remain “not recorded”; never replace
that state with the report-generation date.