Evaluate Agent Skills Before Publication

Scanning tells you whether a skill is safe to run. It does not tell you whether the skill is any good. A skill can be perfectly safe, well documented, correctly signed, and still leave an agent no better off than it was without it.

Evaluation answers that second question by measurement rather than assertion: an agent attempts a fixed set of tasks twice, once with the skill loaded and once without, and the difference is recorded.

What SkillEvaluator Runs

SkillEvaluator runs three tiers over a skill: deterministic validation, semantic deduplication against the catalog, and live agent evaluation in a sandbox. The security checks in Tier 1 are run by SkillSpector, which is part of SkillEvaluator rather than a separate tool you run alongside it.

What You Need to Ship

Tier 3 cannot run without a task set. Provide one at evals/evals.json, next to your SKILL.md. Other accepted locations are evals/*.json, eval/*.json, and benchmark/evals.json.

Without it, Tier 3 is skipped and the generated BENCHMARK.md reports no live results — which is treated the same as a missing report during review.

Write tasks that reflect what the skill is actually for. Include negative cases, where the correct behavior is for the agent not to use the skill.

Reading a BENCHMARK.md

Every evaluated skill publishes a BENCHMARK.md alongside it. It records the agents used, the evaluation date, per-dimension scores in both conditions, and an overall verdict.

Scores are shown as baseline → with skill (change) for each agent, across five dimensions: security, correctness, discoverability, effectiveness, and efficiency.

PASS requires every configured dimension to pass for at least one supported agent. Measured change is reported as diagnostic evidence and does not by itself override that gate.