Skip to navigation

Evaluate Skills from Your Agent

View as MarkdownOpen in Claude

Add one skill to your coding agent, then ask: “Is my skill any good?” The agent picks the right check, runs the skillevaluator CLI, reads the report, and tells you what to fix. No commands or flags to remember.

Install

1

Add the skill with npx

You do this once, in your project. Run the command and pick your agents when it asks:

npx skills add NVIDIA/SkillEvaluator

To add it for one agent only, name the agent:

npx skills add NVIDIA/SkillEvaluator -a claude-code -y

The skill lands in .claude/skills/skillevaluator.

Either way, it adds the skill folder to your project. To confirm it is there:

npx skills list

npx comes with Node.js. Install Node.js, then run the command above.

Or skip npx: clone the repository and copy skills/skillevaluator yourself.

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cp -r SkillEvaluator/skills/skillevaluator .claude/skills/ # this repository
cp -r SkillEvaluator/skills/skillevaluator ~/.claude/skills/ # every repository
2

Install the CLI

The skill folder contains instructions for your agent; the checks run in the skillevaluator CLI. Install it before you ask the agent to run a check:

pip install --upgrade skillevaluator

uv tool install skillevaluator works too. See Installation for Python requirements and the extras that add Tier 2 and Tier 3.

3

Start a new agent session

Agents load skills when a session starts, so open a new one in the same project.

4

Ask

Ask your agent to check a skill. It uses the CLI you installed in the previous step.

Call it

Once the skill is installed, your agent can use it in two ways.

Just ask

Ask in plain language. The agent sees the skill in its list and loads it by itself.

Call it by name

Name the skill in your request. In Claude Code, typing /skillevaluator does the same.

Just ask
Is my skill any good? It's in ./my-skill
Call it by name
Use the skillevaluator skill to check ./my-skill

If you have many skills installed, or another one with a similar name, call it by name so the agent picks the right one.

See it work

One question in, one answer out. The agent runs the check for you and needs no API key.

You ask if your skill is any good. The agent uses the skillevaluator skill, runs skillevaluator tier1 ./my-skill --no-llm, which passes 11 of 11 checks with quality 76.5 out of 100, then lists three fixes
An example session. The command and its numbers come from a real run on a small sample skill; your agent's wording will differ.

How it works

You ask your agent in plain language, the SkillEvaluator skill picks the check, the CLI runs it on your machine, and you get a verdict, scores and what to fix

The scores come from the CLI, not from the agent’s opinion. They match what you get running SkillEvaluator by hand or in CI.

What you can ask

You askThe agent runsYou get
”Is my skill any good?”skillevaluator tier1 <skill-dir> --no-llmPass or fail, with security, PII, schema, and quality findings
”Score the quality of this SKILL.md”skillevaluator quality-check and rubric-evalQuality score and grade, rubric feedback
”Does it overlap with my other skills?”skillevaluator tier2Duplicate and similarity findings
”Write eval cases for my skill”Asks for real tasks and outcomes, writes <skill-dir>/evals/EVAL.md, then runs skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md, then tier3 validate <skill-dir> --strictA reviewed evals/evals.json with positive and negative cases
”Does my skill actually make the agent better?”skillevaluator tier3 evaluate, with and without the skillVerdict, five dimension scores, and Skill Lift
”Gate skills in our CI”skillevaluator ci initA setup PR with the right checks

The first row needs no key. The rubric, Tier 2, and Tier 3 need one provider key; see Providers & Credentials.

Keyless checks run without asking when their tools are already installed. Before installing the CLI, an extra or a missing scanner, the agent names the change and asks for permission unless you already authorized it. It also asks before using credentials, spending model tokens, or running an agent on your machine. For eval cases it asks which real tasks matter and what a good result looks like instead of inventing them. Live evaluation starts separate agent processes and a judge, so it needs a provider key on the host even when launched from inside your agent.

Good to know

Static Tier 1 checks need no key. The rubric, Tier 2, and Tier 3 need one provider, set in the shell where the agent runs:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY=...

OpenAI, Anthropic, and any OpenAI-compatible gateway also work. Keys stay in your environment; the skill tells the agent never to print or store them.

For “does my skill help”, the agent runs the same tasks twice in isolated sandboxes, once without your skill and once with it, and an independent judge scores both.

Tier 3 Evaluation: PASS
Agent: codex Attempts scored: 32/32
Baseline With skill Lift
Overall 82.5% 94.4% +11.9
Security 87.5% 100.0% +12.5
Correctness 91.3% 98.8% +7.5
Effectiveness 68.6% 95.3% +26.7

This is the SkillEvaluator skill evaluated by SkillEvaluator. Its own eval cases ship in skills/skillevaluator/evals/. The run above used an earlier version of case 001 that also asserted the agent read the skill’s SKILL.md, which the baseline cannot do; that assertion has since been removed.

Asking one agent to answer with and without a skill and grade itself is not an evaluation. Once it has read the skill, the baseline is contaminated, and the same model is judging its own work. The skill always routes “does it help” to real Tier 3 runs.

Tier 1 and Tier 2 run on your machine through the CLI, with no Docker. Tier 3 runs in a sandbox: Docker by default, or --env-mode local for trusted skills. See Agents & Sandboxes.

If an attempt errors or times out, the run is INCOMPLETE and no final score is reported. The skill tells the agent to say so, find the cause, and rerun, not to average partial results.

Next steps