Installation | SkillEvaluator Documentation

This page gets SkillEvaluator installed the way you want it — the full bundle, a smaller per-tier install, an editable source checkout, or a container. No API key is needed to install or to run the deterministic Tier 1 checks; keys only come into play later, for LLM-backed features (Providers & Credentials).

If you just want the fastest path to a first result, the Quickstart covers it in one command.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Requirements

uv for the recommended install paths (plain pip works too — see the source tab below).
Python 3.12 or 3.13. uv tool install and uv sync provision a supported interpreter automatically when passed --python 3.13; only the plain-pip path needs one on PATH.
Git for source installs and separately installed security scanners.
Docker, only for the container install and Tier 3 Docker-mode evaluation.
No API key is needed for deterministic Tier 1 checks. Full default scanner coverage requires the security extra plus separate Semgrep, SkillSpector, and Gitleaks executables. LLM-backed checks and Tiers 2–3 use a configured provider — see Providers & Credentials.

Install

uv tool (recommended)

From source

Docker

Install the full bundle as a standalone tool — uv provisions Python 3.13 for you:

Install everything

$ uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Prefer a smaller footprint? Swap the extras in the same one-liner:

Install with selected extras

$ uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"            # base checks; scanner checks report INCOMPLETE
$ uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"  # + bundled Python security scanners
$ uv tool install --python 3.13 "skillevaluator[tier2,tier3] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

To pick up the latest commit later, rerun the install with --reinstall:

Upgrade to the latest commit

$ uv tool install --python 3.13 --reinstall "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Then confirm the command resolves:

Verify

$ skillevaluator --version

Choosing extras

The base package completes the schema, PII, license, quality, Unicode, script-lint, and hygiene checks on its own; scanner-backed checks still run but report INCOMPLETE until the security extra and the required external scanner executables are installed. Everything else lives behind an extra:

Harbor is the open-source agent evaluation framework that the tier3 extra installs for live task execution and environment backends.

Extra	What it unlocks
`llm`	Shared LLM and embedding clients used by LLM-backed checks, Tier 2, and Tier 3
`tier2`	Intra-skill deduplication and local-catalog inter-skill similarity; includes `llm`
`tier3`	Docker and cloud live-agent evaluation through Harbor; includes `llm`
`security`	Bandit and pip-audit
`all`	`tier2`, `tier3`, and `security` in one bundle
`dev`	Build, test, coverage, and formatting tools for contributors

[all] installs every SkillEvaluator extra. External scanner tools, Docker, and agent CLIs stay separate no matter which extra you pick.

System tools

Install the required security scanners separately so their dependency stacks do not constrain the SkillEvaluator distribution:

External security scanners

$ brew install semgrep gitleaks                                  # macOS
$ uv tool install semgrep                                       # isolated Python tool environment
$ uv tool install git+https://github.com/NVIDIA/SkillSpector.git

On other systems, install Gitleaks with go install github.com/gitleaks/gitleaks/v8@latest or a binary from the Gitleaks releases. Semgrep and SkillSpector must be on PATH, or configured with SKILLEVALUATOR_SEMGREP_PATH and SKILLEVALUATOR_SKILLSPECTOR_PATH. RHEL 8 and compatible enterprise Linux systems can supply compatible executables independently of the SkillEvaluator Python environment.

Docker is required only for Tier 3 Docker-mode evaluation. Agent CLIs and their credentials are user-supplied — see Agents & Sandboxes.

Verify your install

Two commands tell you whether the install landed:

Smoke test

$ skillevaluator --version
$ skillevaluator health-check

--version confirms the CLI is on PATH — that is the whole smoke test for a Tier 1-only install. health-check goes one step further: a quick readiness check of the CLI and the selected live-eval backend (defaults: the codex agent, docker environment), useful once you plan to run Tier 3. On an install without the tier3 extra or a configured provider it lists the missing pieces and exits non-zero — expected for a Tier 1-only setup, not a broken install. For a deeper diagnosis, doctor covers the same ground plus model verification — see the CLI Reference for both.

Troubleshooting

The shell can't find skillevaluator after installing

uv installs tools to ~/.local/bin, which may not be on PATH yet. Run uv tool update-shell and open a new terminal.

Validation reports INCOMPLETE and exits non-zero

Install any missing required scanner named in the report. The security extra supplies Bandit and pip-audit; Semgrep, SkillSpector, and Gitleaks are separate executables. Missing required evidence leaves validation INCOMPLETE and non-zero.

Next steps

Quickstart

Run your first evaluation in about two minutes — no API key required.

Providers & Credentials

Wire up an LLM provider for Tier 2, Tier 3, and LLM-backed checks.