SkillEvaluator

View as Markdown

SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.

Agent skills are folders of instructions and supporting files that extend AI agents, as defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline, published from the NVIDIA/skills catalog. Tier 1 integrates SkillSpector for specialized security scanning; see Installation for the security setup.

SkillEvaluator’s support level is Experimental: community-supported on a best-effort basis through GitHub Issues, with no SLA. See SUPPORT.md.

The three tiers

Each tier answers one question about your skill.

Tier 1: Validation

Is this skill safe and well-formed? Deterministic schema, quality, security, PII, license, and script checks that run offline — plus optional LLM-as-judge scoring.

Tier 2: Deduplication

Does it overlap with what exists? Embedding similarity finds repeated guidance inside one skill and overlapping skills across a collection or catalog.

Tier 3: Live Evaluation

Does it actually help the agent? A real agent runs generated tasks with and without the skill, inside a sandbox, and the difference is measured.

Every tier is an independent entry point — run any command directly; nothing requires running the earlier tiers first.

TierRepresentative commandsRequires
Tier 1tier1 PATH, quality-check, security-scan, pii-scan, lint-scripts, rubric-evalNo API key for deterministic checks; the security extra plus external Semgrep, SkillSpector, and Gitleaks executables for full scanner coverage; a provider key for LLM checks
Tier 2context-optimization-check, similarity-checkAn embeddings provider; intra-skill analysis also needs a chat LLM — local OpenAI-compatible endpoints work
Tier 3create-eval-dataset, tier3 evaluate, compareNo credential for keyless templates and report inspection; a provider key for LLM generation and grading; live evaluation also needs the agent CLI with its credential and a Docker, local OS, or cloud sandbox

Try it in two minutes

Install with uv and run the keyless quality check against your own skill — no API key, Docker, Gitleaks, or repository clone required:

No API key required
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator quality-check ./my-skill

You get a 0–100 quality score with an A–F grade. From there, the Quickstart walks through a complete offline Tier 1 run and wiring up a provider for the LLM-backed parts.

How a run flows

SkillEvaluator pipeline: skill directory through Tier 1 static validation, Tier 2 semantic deduplication, and Tier 3 live agent evaluation, producing reports and a verdict
The SkillEvaluator three-tier pipeline

skillevaluator validate ./my-skill runs all three tiers by default, with Tier 3 autopilot reusing an existing evaluation source or creating one starter case when none exists. Tier 1’s deterministic gates run offline. Tier 2 and the optional Tier 1 LLM checks need a provider key, and Tier 3 adds the agent credential and a sandbox. Every run ends in reports — CLI, JSON, HTML, or Markdown.

What Tier 3 measures

  • Five dimensions — Security, Correctness, Discoverability, Effectiveness, and Efficiency: human-readable scores that lead every standard grading report.
  • Skill Lift — the measured difference between the with-skill and without-skill runs; the number that says whether the skill earns its place.
  • pass@k — multi-attempt reliability, reported separately from the dimension scores.

How to read all three, and the on-disk results contract, live in Reports & Results.

Explore the docs

Quickstart

First evaluation result in about two minutes, no API key.

Gate Your CI

Make SkillEvaluator a merge gate — exit codes, a GitHub Actions recipe, and progressive adoption.

CLI Reference

Every command, every flag, exact defaults.

Contributing

Dev environment, tests, docs workflow, and how to submit a change.