Gate Your CI | SkillEvaluator Documentation

Turn SkillEvaluator into a merge gate for your skill repository. The recommended gate runs entirely offline — no API key, no network calls beyond installing the tools — so you can adopt it in any CI system today and layer LLM-backed checks on later.

Exit codes

skillevaluator validate communicates pass/fail through its exit code, so a plain run: step is already a gate — no output parsing required.

Exit code	Meaning	What to do
`0`	Success — every gating check passed	Merge
`1`	Validation failed — one or more gating checks failed	Fix the skill; read the report
`2`	Configuration error — bad flags or an unreadable target	Fix the pipeline, not the skill
`3`	Runtime error — an unexpected internal failure	Re-run; the skill was never judged. File an issue if it persists

Tier 1 checks always gate the exit code. Tier 2 deduplication findings gate when dedup runs unless you pass --no-block-on-dedup; the recommended keyless gate below instead disables Tier 2 with --no-dedup. Tier 3 live-evaluation results are advisory inside validate unless you pass --block-on-agent-eval. Reports record the effective choice for each tier.

The recommended gate

Keyless CI gate

skillevaluator validate ./skills --external --no-dedup -r json,markdown -o reports --min-score 70

Each flag earns its place:

--external pins the public-publication profile explicitly instead of relying on the environment, so the gate behaves identically on every runner. The active profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose), so a run always records which gate was applied.
--no-dedup skips Tier 2, which needs an embedding-capable provider key (--no-tier2 is an equivalent alias). Without it the run is fully keyless and hermetic. Drop this flag once you add a provider secret.
-r json,markdown,sarif writes machine-readable JSON, a Markdown PR comment, and a SARIF file for GitHub Code Scanning.
-o reports collects everything in one directory for artifact upload.
--min-score 70 is the default quality bar, written out explicitly so raising it later is a visible one-line diff.

GitHub Actions recipe

A complete workflow: install with uv, add the external Semgrep, SkillSpector, and Gitleaks scanners for a complete security result, validate, upload the reports, and post the Markdown report on the pull request.

ci.yml

name: Validate skills
on:
  pull_request:
    paths:
      - "skills/**"
jobs:
  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      security-events: write
    steps:
      - uses: actions/checkout@v7
      - name: Install uv
        uses: astral-sh/setup-uv@v8
        with:
          python-version: "3.13"
      - name: Install SkillEvaluator
        run: uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
      - name: Install external Python scanners
        run: |
          uv tool install semgrep
          uv tool install git+https://github.com/NVIDIA/SkillSpector.git
      - name: Install Gitleaks
        run: |
          GITLEAKS_VERSION=8.30.0
          curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
            | tar -xz gitleaks
          sudo mv gitleaks /usr/local/bin/
      - name: Validate skills
        run: |
          skillevaluator validate ./skills --external --no-dedup \
            -r json,markdown,sarif -o reports --min-score 70
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: skillevaluator-reports
          path: reports/
      - name: Find SARIF reports
        if: always()
        id: sarif
        run: |
          mapfile -t files < <(find reports -name '*.sarif.json' -type f 2>/dev/null | sort)
          if [ "${#files[@]}" -eq 0 ]; then
            echo "count=0" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          skillevaluator_python="$(dirname "$(readlink -f "$(command -v skillevaluator)")")/python"
          "$skillevaluator_python" - <<'PY'
          import json
          import pathlib
          from skillevaluator.reporting.sarif_reporter import merge_catalog_sarif_documents
          documents = []
          for raw in sorted(pathlib.Path("reports").rglob("*.sarif.json")):
              if raw.name == "catalog-sarif.json":
                  continue
              documents.append(json.loads(raw.read_text(encoding="utf-8")))
          if not documents:
              raise SystemExit(0)
          merged = merge_catalog_sarif_documents(documents)
          out = pathlib.Path("reports/catalog-sarif.json")
          out.write_text(json.dumps(merged, indent=2), encoding="utf-8")
          print(f"merged {len(documents)} child report(s) into one run at {out}")
          PY
          echo "count=${#files[@]}" >> "$GITHUB_OUTPUT"
          echo "path=reports/catalog-sarif.json" >> "$GITHUB_OUTPUT"
      - name: Upload SARIF to GitHub Code Scanning
        if: always() && steps.sarif.outputs.count != '0'
        uses: github/codeql-action/upload-sarif@v4
        with:
          sarif_file: ${{ steps.sarif.outputs.path }}
          category: skillevaluator
      - name: Post the Markdown report as a PR comment
        if: always() && github.event_name == 'pull_request'
        uses: actions/github-script@v9
        with:
          script: |
            const fs = require("fs");
            const [latest] = fs.readdirSync("reports")
              .filter((f) => f.endsWith(".md"))
              .sort()
              .reverse();
            if (!latest) return;
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: fs.readFileSync(`reports/${latest}`, "utf8"),
            });

The security extra installs Bandit and pip-audit. Semgrep and SkillSpector stay in separate uv tool environments, while Gitleaks is installed from its official releases. Report filenames embed a sortable timestamp (skillevaluator-output-<timestamp>.md), which is why a lexicographic sort finds the newest one. The if: always() conditions keep reports flowing even when validation fails — which is exactly when you want them.

The same shape ports to any CI system: install, run validate, gate on the exit code, archive reports/.

Validating collections

Point validate at a folder of skills and it validates each one:

Validate a whole collection

skillevaluator validate ./skills --external --no-dedup -c -r json,markdown -o reports

Two flags control how a multi-skill run behaves:

Flag	Behavior	Use it when
`-c`, `--continue-on-failure`	Runs the full pipeline without stopping early and records every issue; for folder targets, keeps scanning every skill past a CRITICAL finding	You want one complete report per CI run (recommended for gates)
`--fail-fast`	Stops on the first failing check	You want the fastest possible red signal

-c overrides --fail-fast if both are passed. For a merge gate, prefer -c: contributors get the complete picture in one run instead of a fix-push-repeat loop.

Adopt progressively

You do not have to turn on the strictest gate on day one. A path that works well:

Start advisory

Run the gate but never fail the build. Contributors see reports and get used to the findings before anything blocks a merge.

Report-only mode

skillevaluator validate ./skills --external --no-dedup -r markdown -o reports || true

Gate on Tier 1

Remove the || true. Exit code 1 now blocks merges on schema, security, PII, license, code-integrity, Unicode, and quality failures — all still keyless. (The lint check also runs, but its findings are advisory warnings and never fail validation.)

Raise the quality bar

Bump --min-score above the default 70 once your collection consistently clears it. The score and grade in each report tell you where the collection stands before you move the line.

Add LLM-backed checks

Add a provider key as a masked CI secret, set SKILL_EVAL_LLM_PROVIDER, then add --llm (and --llm-verify to suppress false positives) and drop --no-dedup so Tier 2 overlap checks run too. Once dedup runs, its findings gate alongside Tier 1 by default; add --no-block-on-dedup if you want evidence without enforcing it yet. See Providers & Credentials for provider setup.

Add an advisory Tier 3 job

Run live agent evaluation as a separate job — either skillevaluator tier3 evaluate on its own or validate --tier3 (--agent-eval is a supported compatibility alias and is not currently deprecated; validate --full runs Tier 1+2+3 in one shot). Attached Tier 3 is advisory by default; add --block-on-agent-eval when the team is ready for its findings or invalid source evidence to fail the merge gate. The standalone tier3 evaluate exits non-zero when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the env: block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the nv_build provider and --env-mode docker, a single NVIDIA_API_KEY secret covers the evaluator and all three agents. See Tier 3: Live Evaluation.

Keep provider and agent keys as masked CI secrets in the job environment — never in the repo or in evals/config.yml. SkillEvaluator enforces this: a harbor.runtime_env entry that names or ${VAR}-references an operator-owned credential (such as OPENAI_API_KEY or NVIDIA_API_KEY) fails the Tier 3 run with a hard error.

Custom policy in CI

If the default external profile is too loose or too strict for your repository, commit a policy overlay next to your skills and pin it in the gate. Severity decisions then get reviewed like code:

policy.yaml

profile: community-strict
identity:
  author_email_regex: '<[^>]+@example\.org>'
severity_overrides:
  SCHEMA.author_missing: critical
  LICENSE.*: critical

Gate with a pinned policy

skillevaluator validate ./skills --external --policy ./policy.yaml --no-dedup -r json,markdown -o reports

The policy overlays the external profile, and the resulting profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose) — so a run always tells you which gate produced it. Full overlay syntax is on the Tier 1: Validation page.

Parse the JSON

When the exit code is not enough — dashboards, custom thresholds, badge generation — read reports/skillevaluator-output-<timestamp>.json. The fields CI cares about sit at the top level:

Field	Type	Contents
`overall_passed`	boolean	`true` when every check passed
`overall_status`	string	`passed`, `failed`, or `incomplete` (a required scanner produced no evidence)
`severity_counts`	object	Totals for `critical`, `high`, `medium`, `low`
`total_errors`, `total_warnings`	number	Aggregate counts across all validators
`skills`	array	Per-skill `{ name, passed, issue_count }`
`quality_summary`	array	Quality details with `overall_score` (0–100) and `grade` (A–F); a folder target reports the collection average plus a `skill_count`

A step that enforces a stricter score than the built-in gate:

Gate on quality score with jq

report=$(ls -t reports/skillevaluator-output-*.json | head -1)
jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"

jq -e exits non-zero when the expression is false, so the step fails the job on its own. For Tier 3 payloads and the full results-on-disk layout, see Reports & Results.

Troubleshooting

The run exits 2 instead of 1

Exit code 2 is a configuration error — a bad flag or an unreadable target path. Fix the pipeline; the skill was never judged. (Passing --llm without a configured provider does not exit 2: the LLM-backed scanner is marked incomplete and the run fails with 1.)

overall_status is 'incomplete'

A required scanner is missing or returned no evidence. Check the report for the named scanner: Semgrep, SkillSpector, and Gitleaks are all separate executables, and the security extra alone does not install them. The scanners fail closed: validate exits non-zero and the report stays non-green until every required scanner is installed and returns valid evidence. See the workflow above or Installation: System tools.

The report artifact is empty

Make sure the artifact path matches your -o directory and that the upload step carries if: always() — otherwise a failed gate skips the upload, and you lose the report exactly when you need it.

Next steps

Reports & Results

What every report format contains and how to read scores, severities, and Tier 3 payloads.

Tier 1: Validation

Every check behind the gate — what it inspects and how to fix findings.

Providers & Credentials

Add an LLM provider to unlock --llm, Tier 2 dedup, and Tier 3.

CLI Reference

The complete flag reference for validate and every other command.