> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/skills/skillevaluator/_mcp/server. # SkillEvaluator > An open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills. SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation. Agent skills are folders of instructions and supporting files that extend AI agents, as defined by the [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills). Tier 1 integrates [SkillSpector](https://github.com/NVIDIA/SkillSpector) for specialized security scanning; see [Installation](/skills/skillevaluator/installation#system-tools) for the security setup. SkillEvaluator's support level is **Experimental**: community-supported on a best-effort basis through [GitHub Issues](https://github.com/NVIDIA/SkillEvaluator/issues), with no SLA. See [SUPPORT.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/SUPPORT.md). ## The three tiers Each tier answers one question about your skill. #### [Tier 1: Validation](/skills/skillevaluator/tier1-validation) **Is this skill safe and well-formed?** Deterministic schema, quality, security, PII, license, and script checks that run offline — plus optional LLM-as-judge scoring. #### [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication) **Does it overlap with what exists?** Embedding similarity finds repeated guidance inside one skill and overlapping skills across a collection or catalog. #### [Tier 3: Live Evaluation](/skills/skillevaluator/tier3-live-evaluation) **Does it actually help the agent?** A real agent runs generated tasks with and without the skill, inside a sandbox, and the difference is measured. Every tier is an independent entry point — run any command directly; nothing requires running the earlier tiers first. | Tier | Representative commands | Requires | | ------------------------------------------------------ | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Tier 1](/skills/skillevaluator/tier1-validation) | `validate`, `quality-check`, `security-scan`, `pii-scan`, `lint-scripts`, `rubric-eval` | No API key for deterministic checks; the `security` extra plus external Semgrep, SkillSpector, and Gitleaks executables for full scanner coverage; a provider key for LLM checks | | [Tier 2](/skills/skillevaluator/tier2-deduplication) | `context-optimization-check`, `similarity-check` | An embeddings provider; intra-skill analysis also needs a chat LLM — local OpenAI-compatible endpoints work | | [Tier 3](/skills/skillevaluator/tier3-live-evaluation) | `create-eval-dataset`, `tier3 evaluate`, `compare` | No credential for keyless templates and report inspection; a provider key for LLM generation and grading; live evaluation also needs the agent CLI with its credential and a Docker, local OS, or cloud sandbox | ## Try it in two minutes Install with [uv](https://docs.astral.sh/uv/) and run the keyless quality check against your own skill — no API key, Docker, Gitleaks, or repository clone required: ```bash title="No API key required" uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git" skillevaluator quality-check ./my-skill ``` You get a 0–100 quality score with an A–F grade. From there, the [Quickstart](/skills/skillevaluator/quickstart) walks through a complete offline Tier 1 run and wiring up a provider for the LLM-backed parts. ## How a run flows SkillEvaluator pipeline: skill directory through Tier 1 static validation, Tier 2 semantic deduplication, and Tier 3 live agent evaluation, producing reports and a verdict Tier 1's deterministic gates run offline. Tier 2 and the optional Tier 1 LLM checks need a provider key, and Tier 3 adds the agent credential and a sandbox. Every run ends in [reports](/skills/skillevaluator/reports) — CLI, JSON, HTML, or Markdown. ## What Tier 3 measures * **Five dimensions** — Security, Correctness, Discoverability, Effectiveness, and Efficiency: human-readable scores that lead every standard grading report. * **Skill Lift** — the measured difference between the with-skill and without-skill runs; the number that says whether the skill earns its place. * **pass\@k** — multi-attempt reliability, reported separately from the dimension scores. How to read all three, and the on-disk results contract, live in [Reports & Results](/skills/skillevaluator/reports). ## Explore the docs #### [Quickstart](/skills/skillevaluator/quickstart) First evaluation result in about two minutes, no API key. #### [Gate Your CI](/skills/skillevaluator/ci-integration) Make SkillEvaluator a merge gate — exit codes, a GitHub Actions recipe, and progressive adoption. #### [CLI Reference](/skills/skillevaluator/cli-reference) Every command, every flag, exact defaults. #### [Contributing](/skills/skillevaluator/developer-guide) Dev environment, tests, docs workflow, and how to submit a change. * [SkillEvaluator on GitHub](https://github.com/NVIDIA/SkillEvaluator) * [Support](https://github.com/NVIDIA/SkillEvaluator/blob/main/SUPPORT.md) — best-effort, through GitHub Issues * [Security policy](https://github.com/NVIDIA/SkillEvaluator/blob/main/SECURITY.md) — report vulnerabilities privately, never in a public issue * [License](https://github.com/NVIDIA/SkillEvaluator/blob/main/LICENSE) — Apache-2.0