Gate Your CI | SkillEvaluator Documentation

Turn SkillEvaluator into a merge gate for your skill repository. The recommended gate runs entirely offline — no API key, no network calls beyond installing the tools — so you can adopt it in any CI system today and layer LLM-backed checks on later.

Exit codes

skillevaluator validate communicates pass/fail through its exit code, so a plain run: step is already a gate — no output parsing required.

Exit code	Meaning	What to do
`0`	Success — every gating check passed	Merge
`1`	Validation failed — one or more gating checks failed	Fix the skill; read the report
`2`	Configuration error — bad flags or an unreadable target	Fix the pipeline, not the skill
`3`	Runtime error — an unexpected internal failure	Re-run; the skill was never judged. File an issue if it persists

Tier 1 checks gate the exit code — and so do Tier 2 deduplication findings when dedup runs (the recommended gate below disables it with --no-dedup). Tier 3 live-evaluation results are always advisory inside validate: they appear in the reports but never flip a passing run to 1. That means you can layer Tier 3 on later without destabilizing the gate.

The recommended gate

Keyless CI gate

$ skillevaluator validate ./skills --external --no-dedup -r json,markdown -o reports --min-score 70

Each flag earns its place:

--external pins the public-publication profile explicitly instead of relying on the environment, so the gate behaves identically on every runner. The active profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose), so a run always records which gate was applied.
--no-dedup skips Tier 2, which needs an embedding-capable provider key (--no-tier2 is an equivalent alias). Without it the run is fully keyless and hermetic. Drop this flag once you add a provider secret.
-r json,markdown writes a machine-readable report for your pipeline and a Markdown report you can post as a PR comment.
-o reports collects everything in one directory for artifact upload.
--min-score 70 is the default quality bar, written out explicitly so raising it later is a visible one-line diff.

GitHub Actions recipe

A complete workflow: install with uv, add the external Semgrep, SkillSpector, and Gitleaks scanners for a complete security result, validate, upload the reports, and post the Markdown report on the pull request.

ci.yml

1 name: Validate skills
2 
3 on:
4   pull_request:
5     paths:
6       - "skills/**"
7 
8 jobs:
9   validate:
10     runs-on: ubuntu-latest
11     permissions:
12       contents: read
13       pull-requests: write
14     steps:
15       - uses: actions/checkout@v7
16 
17       - name: Install uv
18         uses: astral-sh/setup-uv@v8
19         with:
20           python-version: "3.13"
21 
22       - name: Install SkillEvaluator
23         run: uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
24 
25       - name: Install external Python scanners
26         run: |
27           uv tool install semgrep
28           uv tool install git+https://github.com/NVIDIA/SkillSpector.git
29 
30       - name: Install Gitleaks
31         run: |
32           GITLEAKS_VERSION=8.30.0
33           curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
34             | tar -xz gitleaks
35           sudo mv gitleaks /usr/local/bin/
36 
37       - name: Validate skills
38         run: |
39           skillevaluator validate ./skills --external --no-dedup \
40             -r json,markdown -o reports --min-score 70
41 
42       - name: Upload reports
43         if: always()
44         uses: actions/upload-artifact@v7
45         with:
46           name: skillevaluator-reports
47           path: reports/
48 
49       - name: Post the Markdown report as a PR comment
50         if: always() && github.event_name == 'pull_request'
51         uses: actions/github-script@v9
52         with:
53           script: |
54             const fs = require("fs");
55             const [latest] = fs.readdirSync("reports")
56               .filter((f) => f.endsWith(".md"))
57               .sort()
58               .reverse();
59             if (!latest) return;
60             await github.rest.issues.createComment({
61               owner: context.repo.owner,
62               repo: context.repo.repo,
63               issue_number: context.issue.number,
64               body: fs.readFileSync(`reports/${latest}`, "utf8"),
65             });

The security extra installs Bandit and pip-audit. Semgrep and SkillSpector stay in separate uv tool environments, while Gitleaks is installed from its official releases. Report filenames embed a sortable timestamp (skillevaluator-output-<timestamp>.md), which is why a lexicographic sort finds the newest one. The if: always() conditions keep reports flowing even when validation fails — which is exactly when you want them.

The same shape ports to any CI system: install, run validate, gate on the exit code, archive reports/.

Validating collections

Point validate at a folder of skills and it validates each one:

Validate a whole collection

$ skillevaluator validate ./skills --external --no-dedup -c -r json,markdown -o reports

Two flags control how a multi-skill run behaves:

Flag	Behavior	Use it when
`-c`, `--continue-on-failure`	Runs the full pipeline without stopping early and records every issue; for folder targets, keeps scanning every skill past a CRITICAL finding	You want one complete report per CI run (recommended for gates)
`--fail-fast`	Stops on the first failing check	You want the fastest possible red signal

-c overrides --fail-fast if both are passed. For a merge gate, prefer -c: contributors get the complete picture in one run instead of a fix-push-repeat loop.

Adopt progressively

You do not have to turn on the strictest gate on day one. A path that works well:

Start advisory

Run the gate but never fail the build. Contributors see reports and get used to the findings before anything blocks a merge.

Report-only mode

$ skillevaluator validate ./skills --external --no-dedup -r markdown -o reports || true

Gate on Tier 1

Remove the || true. Exit code 1 now blocks merges on schema, security, PII, license, code-integrity, Unicode, and quality failures — all still keyless. (The lint check also runs, but its findings are advisory warnings and never fail validation.)

Raise the quality bar

Bump --min-score above the default 70 once your collection consistently clears it. The score and grade in each report tell you where the collection stands before you move the line.

Add LLM-backed checks

Add a provider key as a masked CI secret, set SKILL_EVAL_LLM_PROVIDER, then add --llm (and --llm-verify to suppress false positives) and drop --no-dedup so Tier 2 overlap checks run too. Note that once dedup runs, its findings gate the exit code alongside Tier 1. See Providers & Credentials for provider setup.

Add an advisory Tier 3 job

Run live agent evaluation as a separate, non-blocking job — either skillevaluator tier3 evaluate on its own or validate --tier3 (--agent-eval is a supported compatibility alias and is not currently deprecated; validate --full runs Tier 1+2+3 in one shot). Tier 3 results never gate validate’s exit code; the standalone tier3 evaluate exits non-zero only when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the env: block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the nv_build provider and --env-mode docker, a single NVIDIA_API_KEY secret covers the evaluator and all three agents. See Tier 3: Live Evaluation.

Keep provider and agent keys as masked CI secrets in the job environment — never in the repo or in evals/config.yml. SkillEvaluator enforces this: a harbor.runtime_env entry that names or ${VAR}-references an operator-owned credential (such as OPENAI_API_KEY or NVIDIA_API_KEY) fails the Tier 3 run with a hard error.

Custom policy in CI

If the default external profile is too loose or too strict for your repository, commit a policy overlay next to your skills and pin it in the gate. Severity decisions then get reviewed like code:

policy.yaml

1 profile: community-strict
2 identity:
3   author_email_regex: '<[^>]+@example\.org>'
4 severity_overrides:
5   SCHEMA.author_missing: critical
6   LICENSE.*: critical

Gate with a pinned policy

$ skillevaluator validate ./skills --external --policy ./policy.yaml --no-dedup -r json,markdown -o reports

The policy overlays the external profile, and the resulting profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose) — so a run always tells you which gate produced it. Full overlay syntax is on the Tier 1: Validation page.

Parse the JSON

When the exit code is not enough — dashboards, custom thresholds, badge generation — read reports/skillevaluator-output-<timestamp>.json. The fields CI cares about sit at the top level:

Field	Type	Contents
`overall_passed`	boolean	`true` when every check passed
`overall_status`	string	`passed`, `failed`, or `incomplete` (a required scanner produced no evidence)
`severity_counts`	object	Totals for `critical`, `high`, `medium`, `low`
`total_errors`, `total_warnings`	number	Aggregate counts across all validators
`skills`	array	Per-skill `{ name, passed, issue_count }`
`quality_summary`	array	Quality details with `overall_score` (0–100) and `grade` (A–F); a folder target reports the collection average plus a `skill_count`

A step that enforces a stricter score than the built-in gate:

Gate on quality score with jq

$ report=$(ls -t reports/skillevaluator-output-*.json | head -1)
$ jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"

jq -e exits non-zero when the expression is false, so the step fails the job on its own. For Tier 3 payloads and the full results-on-disk layout, see Reports & Results.

Troubleshooting

The run exits 2 instead of 1

Exit code 2 is a configuration error — a bad flag or an unreadable target path. Fix the pipeline; the skill was never judged. (Passing --llm without a configured provider does not exit 2: the LLM-backed scanner is marked incomplete and the run fails with 1.)

overall_status is 'incomplete'

A required scanner is missing or returned no evidence. Check the report for the named scanner: Semgrep, SkillSpector, and Gitleaks are all separate executables, and the security extra alone does not install them. The scanners fail closed: validate exits non-zero and the report stays non-green until every required scanner is installed and returns valid evidence. See the workflow above or Installation: System tools.

The report artifact is empty

Make sure the artifact path matches your -o directory and that the upload step carries if: always() — otherwise a failed gate skips the upload, and you lose the report exactly when you need it.

Next steps

Reports & Results

What every report format contains and how to read scores, severities, and Tier 3 payloads.

Tier 1: Validation

Every check behind the gate — what it inspects and how to fix findings.

Providers & Credentials

Add an LLM provider to unlock --llm, Tier 2 dedup, and Tier 3.

CLI Reference

The complete flag reference for validate and every other command.