Installation | SkillEvaluator Documentation

Start with the base package for deterministic quality checks, then add only the extras needed for your next command. No API key is needed to install or to run deterministic Tier 1 checks. LLM-backed features use a configured provider (Providers & Credentials); live Tier 3 evaluation adds an agent runtime and a sandbox.

For a complete first run against a bundled skill, use the keyless Quickstart, or add the llm extra for the NVIDIA Build route. Those routes use a source checkout and uv run; the standalone install below is convenient when you already have your own skill directory.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Requirements

uv for the recommended install paths (plain pip works too — see the source tab below).
Python 3.12, 3.13, or 3.14. uv tool install, uv run, and uv sync provision a supported interpreter automatically when passed --python 3.13 (or --python 3.14); only the plain-pip path needs one on PATH.
Git for installs from the repository and for the bundled-example checkout.
Docker, only for the container install and Tier 3 Docker-mode evaluation.
No API key is needed for deterministic Tier 1 checks. Full default scanner coverage requires the security extra; Tier 1 provisions SkillSpector and Gitleaks itself, and Semgrep is optional (see System tools). LLM-backed checks and Tiers 2–3 use a configured provider — see Providers & Credentials.

Install

uv tool (recommended)

From source

Docker

Install the base package as a standalone tool — uv provisions Python 3.13 for you:

Install the base package

uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator quality-check ./my-skill

Replace ./my-skill with a directory containing a SKILL.md. This quality check needs no API key, Docker, or security scanner. Full static scanner coverage has separate requirements under System tools.

Add the llm extra when you want to make your first rubric-evaluation call:

Add LLM support

uv tool install --python 3.13 --reinstall "skillevaluator[llm] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Then follow the NVIDIA Build setup, using skillevaluator rubric-eval ./my-skill for a standalone installation. Provider clients install with the extra; an API key is still required for the model call.

For other features, select their extras or use the optional evaluation bundle:

Install with selected extras

uv tool install --python 3.13 --reinstall "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[tier2,tier3] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

These are alternatives: include all the extras you need in the chosen command. all includes tier2, tier3, and security; it excludes the optional dashboard extra and external tools.

To pick up the latest commit later, rerun the install with --reinstall, keeping your chosen extras. Choose the command matching your install:

Upgrade to the latest commit

uv tool install --python 3.13 --reinstall "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[llm] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Then confirm the command resolves:

Verify

skillevaluator --version

Choosing extras

The base package completes the schema, PII, license, quality, Unicode, script-lint, and hygiene checks on its own. quality-check needs no scanner setup. When you run the full static suite, scanner-backed checks report INCOMPLETE until the security extra and the required external scanner executables are installed. Running without an LLM or API key does not remove those scanner requirements. Everything else lives behind an extra:

Harbor is the open-source agent evaluation framework that the tier3 extra installs for live task execution and environment backends.

Extra	What it unlocks
`llm`	Shared LLM and embedding clients used by LLM-backed checks, Tier 2, and Tier 3
`tier2`	Intra-skill deduplication and local-catalog inter-skill similarity; includes `llm`
`tier3`	Docker, local, and cloud live-agent evaluation through Harbor (pins `harbor==0.24.0` and requires `python-dotenv>=1.2.0`); includes `llm`
`security`	Bandit and pip-audit
`all`	`tier2`, `tier3`, and `security` in one bundle
`dashboard`	Local Streamlit tables for comparing saved reports and retained runs (`skillevaluator dashboard`); not included in `all`
`dev`	Build, test, coverage, and formatting tools for contributors

[all] installs the evaluator extras (tier2, tier3, security); add dashboard with [all,dashboard]. External scanner tools, Docker, agent CLIs, and the Harbor extras for managed sandbox backends (such as harbor[modal]==0.24.0) stay separate no matter which extra you pick — see Agents & Sandboxes.

System tools

Tier 1 provisions the external security scanners in their own environments, so their dependency stacks do not constrain the SkillEvaluator distribution:

SkillSpector: when no executable is found, Tier 1 downloads the 2.13.3 release wheel once into the SkillEvaluator runtime folder, verifies its SHA-256 checksum, and runs it with uv tool run. The first run needs uv and network access; later runs use the uv cache.
Semgrep: optional. When it is installed, Code Risk Analysis adds its rules to Bandit; when it is not, the check notes it was skipped. Set SKILLEVALUATOR_REQUIRE_SEMGREP=1 to report a missing Semgrep as INCOMPLETE (the Tier 1 CI image does).
Gitleaks: on first use, SkillEvaluator downloads the pinned 8.30.0 release into its runtime folder and verifies its SHA-256 checksum. There is no prompt, so CI runs install it too. Linux and macOS only.

Installed executables take precedence. To use your own, put them on PATH or set SKILLEVALUATOR_SEMGREP_PATH and SKILLEVALUATOR_SKILLSPECTOR_PATH:

Optional: preinstall the scanners

uv tool install semgrep
uv tool install git+https://github.com/NVIDIA/SkillSpector.git@v2.13.3
brew install gitleaks          # or a binary from https://github.com/gitleaks/gitleaks/releases

Docker is required only for Tier 3 Docker-mode evaluation. Agent CLIs and their credentials are user-supplied — see Agents & Sandboxes.

skillevaluator health-check --tier 1 reports the path and version of each installed scanner and exits non-zero when one is missing, misconfigured, or older than the supported release. Add --checks security,code-integrity,dependency to include pip-audit.

Verify your install

Two commands tell you whether the install landed:

Smoke test

skillevaluator --version
skillevaluator health-check

--version confirms the CLI is on PATH — that is the whole smoke test for a Tier 1-only install. health-check goes one step further: a quick readiness check of the CLI and the selected live-eval backend (the agent follows the configured provider; the environment defaults to docker), useful once you plan to run Tier 3. On an install without the tier3 extra or a configured provider it lists the missing pieces and exits non-zero — expected for a Tier 1-only setup, not a broken install. For a deeper diagnosis, doctor covers the same ground; add doctor --verify-models to make a live credential-bearing catalog request for each resolved agent model. A public or otherwise non-authoritative result is reported as a warning because catalog access alone does not prove inference access. See the CLI Reference for both commands.

Troubleshooting

The shell can't find skillevaluator after installing

uv installs tools to ~/.local/bin, which may not be on PATH yet. Run uv tool update-shell and open a new terminal.

Validation reports INCOMPLETE and exits non-zero

A required scanner could not run. Check that uv is on PATH and GitHub and PyPI are reachable (SkillSpector), or that GitHub is reachable for the Gitleaks download, or install the scanner named in the report. Missing required evidence leaves validation INCOMPLETE and non-zero. To run without that scanner, pass --skip with the check named in the fix list (e.g. --skip secrets); the check is then reported as skipped and the run cannot certify Tier 1.

Next steps

Quickstart

Check a bundled skill without an LLM, or use NVIDIA Build for a rubric review.

Providers & Credentials

Wire up an LLM provider for Tier 2, Tier 3, and LLM-backed checks.