Installation Options
For the recommended setup, start with Install SkillEvaluator.
Upgrade
Use the command for your original installation method. Keep all the extras you need when changing an installation.
pip (PyPI)
uv tool (PyPI)
GitHub source
In the activated environment:
For an all-tier installation, keep the extras in the upgrade command:
python -m pip install --upgrade "skillevaluator[all]".
Run skillevaluator --version afterward. To move from a source install back to
PyPI, use python -m pip install --upgrade --force-reinstall skillevaluator
in its activated environment, or uv tool install --python 3.13 --reinstall skillevaluator
for a uv tool. Keep [all] or your other selected extras in that package name
when you want to retain their support.
Release and source differences
PyPI 0.9.0 supports the direct
Tier 1, Tier 2, and Tier 3 workflows. Harness Check, JUnit reports, and the
experimental NeMo Gym adapter are included in 1.0.1. Use its published
package when available, or the reviewed GitHub source before publication.
Check a command’s --help against your installed version.
Source checkouts can be newer than the published PyPI package; check the
installation source as well as the version. NVIDIA Build’s evaluator
default also differs: PyPI 0.9.0 uses nvidia/nemotron-3-super-120b-a12b, while
1.0.1 uses nvidia/nemotron-3.5-lightning-30b-a3b.
Other installation methods
Pip, uv, and direct repository installs are on Installation.
Run the CLI in a Docker container
This runs SkillEvaluator itself in a container. For the usual Tier 3 setup, install the CLI above and let it start agent containers; see Agents & Sandboxes.
To build the evaluator image, clone the repository and run these commands
from its root. The Dockerfile
includes [all] but not the optional dashboard.
The image runs as the unprivileged skillevaluator user (UID 10001).
/app/reports is writable and input mounts can be read-only. To retain
reports on a Linux host, use your user ID and a writable home:
SkillSpector and Gitleaks download on first use; each new container needs
network access to GitHub and PyPI unless their caches are retained. Missing
required scanner evidence leaves validation INCOMPLETE and exits nonzero.
Semgrep is optional and is not bundled. Provider credentials and sandbox
access must be configured for the container’s runtime user when needed.
For an editable checkout and contributor tools, use the development setup.
Choosing extras
Choose individual capabilities or add the dashboard
Append the extras you need to skillevaluator, such as skillevaluator[llm]
or skillevaluator[security]. Combine names with commas, such as
skillevaluator[tier2,tier3]. With no brackets, you get the lightweight base
package. Upgrade and replacement commands should name all the extras you
want to keep.
Harbor is the open-source agent
evaluation framework installed by tier3; that extra also requires
python-dotenv>=1.2.0. GKE/ACK dependencies are included; other managed
backends may need additional dependencies. See
backend setup.
System tools
A quality check needs no external scanner setup. For full static checks on
PyPI 0.9.0, add [security] to the lightweight install (or choose [all]).
1.0.1 also needs this extra for pip-audit when running Python
dependency audits:
pip (PyPI)
uv tool (PyPI)
GitHub source
If you already use other extras, include them in these commands too. A uv reinstall keeps only the extras named in the command. These dependencies support the scanner setup below:
For PyPI 0.9.0 installed with pip, install uv before full scanner validation:
Run this in the same activated environment. If you installed SkillEvaluator
with uv tool, uv is already available. SkillEvaluator 1.0.1 includes uv
as a base dependency. See Installation
for the supported installation paths.
The first scanner run needs network access; later runs use cached downloads.
Missing required scanner evidence is reported as INCOMPLETE.
Preinstall or troubleshoot security scanners
Tier 1 verifies the SHA-256 checksums of its downloaded scanners. PyPI 0.9.0 pins SkillSpector 2.13.3; 1.0.1 pins 2.13.4. Both pin Gitleaks 8.30.0. In 1.0.1, uv is installed with SkillEvaluator as a base dependency and runs SkillSpector in its own environment. PyPI 0.9.0 needs uv installed separately, as shown above. An existing uv on PATH is used first.
SkillEvaluator 1.0.1 accepts an automatically discovered SkillSpector only when it matches the pinned version; otherwise it provisions the pinned release. Explicit scanner path overrides are checked separately. To manage the tools yourself:
The last command is for macOS with Homebrew. Other platforms can use the
official Gitleaks releases.
To upgrade a manually installed SkillSpector, repeat its command with
--reinstall. Overrides use SKILLEVALUATOR_SEMGREP_PATH and
SKILLEVALUATOR_SKILLSPECTOR_PATH.
SKILLEVALUATOR_REQUIRE_SEMGREP=1 makes missing Semgrep incomplete; the Tier 1
CI image enables it. Otherwise, its absence is reported as skipped.
This checks scanner paths and versions; it does not run a skill evaluation.
Add --checks security,code-integrity,dependency to include pip-audit.
SkillSpector 2.13.4 can report JSON files containing a timeout key as
partial analysis. Inspect those diagnostics before treating a skill as validated.
Troubleshooting
The shell cannot find uv or skillevaluator
Open a new terminal after installing uv. If uv works but
skillevaluator does not, run uv tool update-shell and reopen the
terminal. With pip, activate the virtual environment where you installed
the package.
Installation says Git or a supported Python is missing
Git is required only for the GitHub source or checkout paths; check it with
git --version. The uv installation commands download Python 3.13
automatically; pip needs an existing Python 3.12–3.14 interpreter.
harness-check is not recognized
Harness Check is included in 1.0.1 and is absent from 0.9.0. Check
skillevaluator --version, then use the published 1.0.1 package when
available or the reviewed GitHub source option on
Installation.
health-check fails even though the CLI installed
skillevaluator --version verifies the CLI installation.
Bare health-check checks the live-evaluation runtime, so missing Docker,
a provider, or the tier3 extra can make it fail on a valid static-only
install. Check the prerequisites for your selected tier instead.
This project downloads and installs additional third-party open source software. Review its license terms before use.