Gate Your CI

Turn SkillEvaluator into a merge gate for your skill repository. Tier 1 needs no model API key; full scanner-backed checks can still contact vulnerability services. Adopt the CLI in any CI system and add model-backed checks separately.

For the prebuilt runtime and one-file consumer setup, see GitHub Actions. The recipe below installs tools explicitly; its PR-comment step is separate from the reusable publisher’s hosted-feedback setup.

Exit codes

skillevaluator validate communicates pass/fail through its exit code, so a plain run: step is already a gate — no output parsing required.

Tier 1 checks always gate the exit code. Tier 2 deduplication findings gate when dedup runs unless you pass --no-block-on-dedup; the recommended keyless gate below selects --tiers 1 to exclude Tier 2 and Tier 3. Tier 3 live-evaluation results are advisory inside validate unless you pass --block-on-agent-eval. Reports record the effective choice for each tier. Without an explicit scope, validate runs all three tiers for skills, including Tier 3 autopilot.

The recommended gate

skillevaluator validate ./skills --external --tiers 1 -r json,markdown,sarif -o reports --min-score 70

Each flag earns its place:

• --external pins the public-publication profile explicitly instead of relying on the environment, so the gate behaves identically on every runner. The active profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose), so a run always records which gate was applied.
• --tiers 1 runs static checks without the provider-backed Tier 2 and Tier 3 stages. Use --tiers 1,2 once you add a provider secret, or omit the option to include Tier 3 after configuring its runtime. --no-dedup alone skips only Tier 2 and still runs Tier 3.
• -r json,markdown,sarif writes machine-readable JSON, a Markdown PR comment, and a SARIF file for GitHub Code Scanning.
• -o reports collects everything in one directory for artifact upload.
• --min-score 70 is the default quality bar, written out explicitly so raising it later is a visible one-line diff.

GitHub Actions recipe

A complete workflow: install with uv, add the external Semgrep, SkillSpector, and Gitleaks scanners for a complete security result, validate, upload the reports, and post the Markdown report on the pull request.

name: Validate skills
on:
  pull_request:
    paths:
      - "skills/**"
jobs:
  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      security-events: write
    steps:
      - uses: actions/checkout@v7
      - name: Install uv
        uses: astral-sh/setup-uv@v8
        with:
          python-version: "3.13"
      - name: Install SkillEvaluator
        run: uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
      - name: Install external Python scanners
        run: |
          uv tool install semgrep
          uv tool install git+https://github.com/NVIDIA/SkillSpector.git@v2.13.3
      - name: Install Gitleaks
        run: |
          GITLEAKS_VERSION=8.30.0
          curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
            | tar -xz gitleaks
          sudo mv gitleaks /usr/local/bin/
      - name: Validate skills
        run: |
          skillevaluator validate ./skills --external --tiers 1 \
            -r json,markdown,sarif -o reports --min-score 70
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: skillevaluator-reports
          path: reports/
      - name: Find SARIF reports
        if: always()
        id: sarif
        run: |
          mapfile -t files < <(find reports -name '*.sarif.json' -type f 2>/dev/null | sort)
          if [ "${#files[@]}" -eq 0 ]; then
            echo "count=0" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          skillevaluator_python="$(dirname "$(readlink -f "$(command -v skillevaluator)")")/python"
          "$skillevaluator_python" - <<'PY'
          import json
          import pathlib
          from skillevaluator.reporting.sarif_reporter import merge_catalog_sarif_documents
          documents = []
          for raw in sorted(pathlib.Path("reports").rglob("*.sarif.json")):
              if raw.name == "catalog-sarif.json":
                  continue
              documents.append(json.loads(raw.read_text(encoding="utf-8")))
          if not documents:
              raise SystemExit(0)
          merged = merge_catalog_sarif_documents(documents)
          out = pathlib.Path("reports/catalog-sarif.json")
          out.write_text(json.dumps(merged, indent=2), encoding="utf-8")
          print(f"merged {len(documents)} child report(s) into one run at {out}")
          PY
          echo "count=${#files[@]}" >> "$GITHUB_OUTPUT"
          echo "path=reports/catalog-sarif.json" >> "$GITHUB_OUTPUT"
      - name: Upload SARIF to GitHub Code Scanning
        if: always() && steps.sarif.outputs.count != '0'
        uses: github/codeql-action/upload-sarif@v4
        with:
          sarif_file: ${{ steps.sarif.outputs.path }}
          category: skillevaluator
      - name: Post the Markdown report as a PR comment
        if: always() && github.event_name == 'pull_request'
        uses: actions/github-script@v9
        with:
          script: |
            const fs = require("fs");
            const [latest] = fs.readdirSync("reports")
              .filter((f) => f.endsWith(".md"))
              .sort()
              .reverse();
            if (!latest) return;
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: fs.readFileSync(`reports/${latest}`, "utf8"),
            });

The security extra installs Bandit and pip-audit. Semgrep and SkillSpector stay in separate uv tool environments, while Gitleaks is installed from its official releases. Report filenames embed a sortable timestamp (skillevaluator-output-<timestamp>.md), which is why a lexicographic sort finds the newest one. The if: always() conditions keep reports flowing even when validation fails — which is exactly when you want them.

The same shape ports to any CI system: install, run validate, gate on the exit code, archive reports/.

Validating collections

Point validate at a folder of skills and it validates each one:

skillevaluator validate ./skills --external --tiers 1 -c -r json,markdown -o reports

Two flags control how a multi-skill run behaves:

-c overrides --fail-fast if both are passed. For a merge gate, prefer -c: contributors get the complete picture in one run instead of a fix-push-repeat loop.

Adopt progressively

You do not have to turn on the strictest gate on day one. A path that works well:

skillevaluator validate ./skills --external --tiers 1 -r markdown -o reports || true

Custom policy in CI

If the default external profile is too loose or too strict for your repository, commit a policy overlay next to your skills and pin it in the gate. Severity decisions then get reviewed like code:

profile: community-strict
identity:
  author_email_regex: '<[^>]+@example\.org>'
severity_overrides:
  SCHEMA.author_missing: critical
  LICENSE.*: critical

skillevaluator validate ./skills --external --policy ./policy.yaml --tiers 1 -r json,markdown -o reports

The policy overlays the external profile, and the resulting profile name is stamped into the HTML report and BENCHMARK.md (and printed in the run banner with --verbose) — so a run always tells you which gate produced it. Full overlay syntax is on the Tier 1: Validation page.

Parse the JSON

When the exit code is not enough — dashboards, custom thresholds, badge generation — read reports/skillevaluator-output-<timestamp>.json. The fields CI cares about sit at the top level:

overall_status and overall_passed describe the configured command gate. They can be green when an advisory tier did not run, so do not use them as a publication signal. Require publication_status == "pass" (or publication.eligible == true) before publishing a benchmark.

A step that enforces a stricter score than the built-in gate:

report=$(ls -t reports/skillevaluator-output-*.json | head -1)
jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"

jq -e exits non-zero when the expression is false, so the step fails the job on its own. For Tier 3 payloads and the full results-on-disk layout, see Reports & Results.

Troubleshooting

Next steps