> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.

# Tier 1 Reference

> Validation rules, profiles, scanner contracts, reports, and troubleshooting for Tier 1.

For your first check, follow [Tier 1: Validation](/skills/skillevaluator/tier1-validation).
This reference keeps the detailed rules and advanced workflows in one place.

Run Tier 1 directly to check a skill's structure, security, quality, and
dependencies. First, [install SkillEvaluator](/skills/skillevaluator/installation#install),
complete [scanner setup](/skills/skillevaluator/installation-options#system-tools) for full validation, then run:

**`Run Tier 1`**

```bash title="Run Tier 1"
skillevaluator tier1 ./my-skill
```

Static checks need no model key. With the [llm extra](/skills/skillevaluator/installation-options#choosing-extras)
and a configured provider, this command also
runs rubric judging, LLM security analysis, and finding verification. Without a
provider, those LLM stages are explicitly reported as skipped. Use `--no-llm`
for static checks only, or `--llm` to require a provider instead of allowing a
keyless run. See [scanner setup](/skills/skillevaluator/installation-options#system-tools)
for first-run downloads and optional tools; PyPI 0.9.0 installed with pip also
needs uv for automatic SkillSpector setup.

The two-variable [provider setup](/skills/skillevaluator/configuration) is enough to configure the
LLM stages for NVIDIA Build or OpenAI. Calling `skillevaluator tier1` without a
path shows help; expert subcommands remain available below.

## Tier 1 in automatic CI

The [all-tier CI setup](/skills/skillevaluator/advanced-ci) keeps Tier 1
as a required gate. GitHub calls the hardened legacy scanner workflow at the
same evaluator commit; GitLab runs the same digest-pinned scanner image.
Static CI checks receive no model key. Tier 2 and Tier 3 execute independently
and retain their own native evidence, so one tier's failure does not erase the
other reports. Changes to onboarding configuration trigger a full keyless
Tier 1 scan so the setup review exercises actual skills. With `scan-mode: auto`,
the provider-backed LLM Tier 1 pass, Tier 2 and Tier 3 run on a pull request only
when it changes a skill. Eligible pushes and manual runs on the default branch
run them for the whole collection.

Existing released Tier 1-only consumers keep their current workflow and
bundled image pins. See [GitHub Actions](/skills/skillevaluator/github-actions) or
[GitLab CI/CD](/skills/skillevaluator/gitlab-ci) to choose either path.

## What Tier 1 checks

The direct `tier1 PATH` workflow runs Tier 1 only, as does
`tier1 validate PATH`. The focused checks below are also available as
`skillevaluator tier1 <command>`. Top-level `validate` is the combined pipeline.

| Command              | Purpose                                                                               | Needs a key?                                                         |
| -------------------- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Top-level `validate` | Run the selected Tier 1 checks, Tier 2 deduplication, and Tier 3 autopilot for skills | Yes for the full pipeline; `--tiers 1` selects keyless static checks |
| `quality-check`      | Score skill quality across four weighted categories                                   | No                                                                   |
| `rubric-eval`        | Run the nine-criterion LLM-as-judge rubric                                            | Yes                                                                  |
| `security-scan`      | Run SkillSpector static scanning and optional LLM analysis                            | No (only with `--llm`/`--llm-verify`)                                |
| `pii-scan`           | Detect PII, credentials, secrets, and local identifiers                               | No (only with `--llm-verify`)                                        |
| `lint-scripts`       | Run advisory AST-based checks on Python scripts                                       | No                                                                   |

`validate` is the umbrella command: all three tiers run by default for skills, with autopilot creating one starter evaluation case if no source exists. Tier 1 checks always gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs and gates by default, and Tier 3 is advisory by default. Use `--tiers 1` for static checks only, `--no-block-on-dedup` to keep Tier 2 advisory, or `--block-on-agent-eval` to promote Tier 3 into the gate.

## Run the combined or focused validation commands

Unlike direct `tier1 PATH`, `validate` includes Tier 2 and Tier 3 by default
for skills. Its Tier 1 LLM checks still run only when requested.

### Run the default suite

**`All three tiers, provider and runtime required`**

```bash title="All three tiers, provider and runtime required"
skillevaluator validate ./my-skill
```

By default you get a compact pipeline view in the terminal — a per-check ticker per tier — and the run writes HTML and JSON report files unless you pass `-r` explicitly, which is then honored exactly (including `-r cli` for terminal-only output). Add `--verbose` for the full per-check detail stream instead of the compact view. The full pipeline needs the [provider and runtime setup](/skills/skillevaluator/tier3-live-evaluation#before-you-start); use `--tiers 1` for static checks only. The exit code is `0` when every gate passes and `1` when one fails.

### Run static checks only

Select only Tier 1 to run the static checks without model calls:

**`Static Tier 1`**

```bash title="Static Tier 1"
skillevaluator validate ./my-skill --tiers 1
```

Tier 1 can download required scanners on first use. For an offline environment,
prepare them using [Installation: System tools](/skills/skillevaluator/installation-options#system-tools).
Run `skillevaluator doctor --tier 1` to check the scanners' paths and supported versions.

### Narrow to specific checks

`--checks` (alias `--tier1-checks`) takes a comma-separated subset when you want fast, targeted feedback:

**`Selective checks`**

```bash title="Selective checks"
skillevaluator validate ./my-skill --tiers 1 --checks schema,security
```

Only the selected Tier 1 checks run; the others are skipped entirely.

### Add LLM analysis (optional)

With a provider configured, `--llm` (alias `--tier1-llm`; disable with `--no-llm` or `--no-tier1-llm`) adds LLM security analysis on top of the static findings, and `--llm-verify` runs a second pass that reviews each static finding in context:

**`LLM-backed validation`**

```bash title="LLM-backed validation"
skillevaluator validate ./my-skill --tiers 1 --llm --llm-verify
```

Confirmed findings keep their original severity. High-confidence false positives stay visible but are downgraded to INFO and carry verification metadata explaining the review. Each finding that gets a verdict names the provider and model that gave it as `llm_provider` and `llm_model_used` in its JSON metadata. For a Claude 5.5 or later model on the native Claude API, the model is the one the reply names; otherwise it is the configured model, unless a server-side `fallback` block in the reply names another model. Verdicts kept from a reply cut off at the output-token limit name the model that wrote that reply. Static scanning always runs first — if a requested LLM stage is unavailable or returns unusable evidence, the run does not silently turn green. `security-scan` and `pii-scan` accept `--llm-verify` too.

HTML, Markdown, and terminal reports show the recorded verification verdict, confidence, and reasoning beside each finding, together with its reported severity.

Direct `tier1 PATH` and `tier1 validate PATH` print the terminal report and
write JSON and HTML reports by default. Use `-r cli` for terminal-only output.

Reports for any of these runs are selected with `-r`/`--report` (`cli`, `json`, `html`, `markdown`, `sarif`) and written to `-o`/`--output-dir` (default `reports/`). The `junit` format is included in 1.0.1; see [Installation](/skills/skillevaluator/installation#install) for availability. It is not in 0.9.0. File naming, `BENCHMARK.md`, and the JSON contract are covered in [Reports & Results](/skills/skillevaluator/reports).

## Content types

`validate` auto-detects what you point it at. Use `--type` when the path is ambiguous:

| Type      | `--type` value | Auto-detection signal                                                                                        |
| --------- | -------------- | ------------------------------------------------------------------------------------------------------------ |
| Skill     | `skill`        | `SKILL.md` in `skills/` or `team-skills/`                                                                    |
| Rules     | `rules`        | `.mdc` files in `team-rules/`                                                                                |
| Workflows | `workflows`    | `workflow-rules.mdc` in a workflow directory                                                                 |
| Plugin    | `plugin`       | A bundle-reference `agent_plugin.yaml`/`.yml` manifest, or a contained `.claude-plugin/plugin.json` manifest |

Plugins are validated against their public contract; the `quality`, `lint`, and `version` checks are skill-only and are skipped for plugins, rules, and workflows.

When `--type auto` finds none of these signals, for example in an ordinary source repository, the target's type is `unknown`. There is no manifest to check, so Tier 1:

* Records `schema`, `version`, and `quality` as not applicable instead of failing them with `No skills found`. The terminal summary and the HTML report count them as N/A. The JSON and Markdown reports list them as skipped, with a skip reason that starts with `not applicable`. No report counts them as passes.
* Still runs the content scans (`security`, `pii`, `license`, `code-integrity`, `unicode`, `lint`, and `dependency` when selected), so a real secret or card number still fails the run. The security scan keeps its unrestricted scope.
* Passes when those scans pass. A misspelled `--checks` name still fails. Tier 2 deduplication still runs on the target unless you pass `--no-dedup` or `--tiers 1`, and can still fail the run.
* Adds a root `tier1_applicability` object to the JSON report, whichever checks `--checks` and `--skip` select: `applicability` (`not_applicable`), `reason_code` (`unsupported_content_type`), `detected_content_type` (`unknown`), and `supported_content_types`.
* Is never eligible for publication, because a not-applicable check is not evidence.

Auto-detection also reads the directory names in the target's absolute path. A repository whose path includes a directory named `skills`, `team-skills`, `team-rules`, `workflows`, or `team-workflows`, such as a CI checkout at `/home/runner/work/skills/skills`, is detected as that type, not as `unknown`, and Tier 1 validates it as that type, as before. `--type` forces a type but cannot select `unknown`.

A target folder that itself holds a manifest entry in any letter case (`SKILL.md`, `workflow-rules.mdc`, `agent_plugin.yaml`, `agent_plugin.yml`, or `.claude-plugin/plugin.json`), such as `Skill.md`, or a manifest symlink whose target is missing, is a broken artifact rather than a folder without one, so its manifest checks still run and fail. A `.claude-plugin` folder without a `plugin.json`, such as a marketplace's, does not count. This applies only to the target's own entries: a mis-cased manifest below the target is not discovered; validate that folder directly. Pass `--type skill` (or another type) to validate any target as that type; a missing `SKILL.md` then fails as before. A folder that holds skills below it, such as `.claude/skills/<name>/SKILL.md` or `.agents/skills/<name>/SKILL.md`, is still validated as a collection of skills.

For collections, pass a folder containing skills under `skills/` or `team-skills/`. Folder-level validation discovers each live `SKILL.md` below the target while excluding configured names such as `evals`, `results`, `versions`, `.git`, `.venv`, `node_modules`, and `__pycache__`.

**`Validate a skill or a collection`**

```bash title="Validate a skill or a collection"
skillevaluator validate ./skills/my-skill --type skill
skillevaluator validate ./skills --type skill --continue-on-failure
```

## Validation profiles and policies

Two policies are bundled. Both report malformed or unsafe author values and
blocked licenses (CRITICAL); they differ in which governance metadata a skill must carry.

|                                                                | `default`                       | `registry`                                        |
| -------------------------------------------------------------- | ------------------------------- | ------------------------------------------------- |
| Use it for                                                     | Public and private repositories | Registries with a documented publication contract |
| `metadata.author` missing                                      | LOW (advisory)                  | HIGH (blocks)                                     |
| Author is a name without an email, such as `NVIDIA cuOpt Team` | INFO                            | HIGH                                              |
| Author is malformed or unsafe                                  | HIGH                            | HIGH                                              |
| `metadata.id` missing                                          | Not reported                    | HIGH                                              |
| `metadata.id` invalid, or duplicated in a collection           | MEDIUM (advisory)               | HIGH                                              |

The Agent Skills baseline requires only `name` and `description`, so the
`default` policy treats author attribution and a stable ID as recommendations.
Neither policy restricts the author email domain. A policy controls validation
rules, not repository visibility or report hosting. The old `external` name and
`--external` flag remain compatibility aliases for `default`; there is no
separate bundled `internal` policy.

**`Profile selection`**

```bash title="Profile selection"
skillevaluator validate ./my-skill                         # default policy
skillevaluator validate ./my-skill --profile default       # explicit default policy
skillevaluator validate ./my-skill --profile registry      # registry publication contract
skillevaluator validate ./my-skill --external              # compatibility alias for default
skillevaluator validate ./my-skill --policy ./policy.yaml  # custom overlay on default
SKILLEVALUATOR_PROFILE=registry skillevaluator validate ./my-skill
```

Precedence, highest to lowest: `--policy` overlay, `--profile` flag, `SKILLEVALUATOR_PROFILE` environment variable, then the bundled `default` policy.

#### Write a custom policy overlay

A custom policy overlays the `--profile` rules (`default` unless you choose
another). It can narrow the accepted author domain, set the `metadata.id`
requirement, and change severities by `CATEGORY.check_name` or `CATEGORY.*`:

**`policy.yaml`**

```yaml title="policy.yaml"
profile: community-strict
identity:
  author_email_regex: '<[^>]+@example\.org>'
  metadata_id_requirement: recommend   # ignore | recommend | require
severity_overrides:
  SCHEMA.author_missing: critical
  LICENSE.*: critical
```

An `author_email_regex` needs an email address in that domain, so under it a
name without an email fails `SCHEMA.author_format`. An overlay that omits an
`identity` key keeps the base profile's value. An unrecognized
`metadata_id_requirement` value is an error, not a silent fallback.

The author checks are `SCHEMA.author_missing`, `SCHEMA.author_without_email`,
and `SCHEMA.author_format`. The ID checks are `SCHEMA.metadata_id_missing`,
`SCHEMA.metadata_id_format`, and `SCHEMA.metadata_id_duplicate`. Their default
severities follow `metadata_id_requirement`: a missing ID is LOW under
`recommend` and HIGH under `require`; an invalid or duplicate ID is MEDIUM
unless the requirement is `require`.

Severity overrides change how findings gate a run; they do not change the
quality score or waive its `--min-score` requirement. Even an override of
`QUALITY.*: info` leaves a below-minimum quality score failing. Nor do they
complete an `INCOMPLETE` scan, whose report keeps the error that says why.

#### Permit reviewed ownership contacts without allowing other email addresses

Contacts in the single-line `metadata.pic` and `metadata.email` fields of
the skill's own `SKILL.md` (the one at the root of the scanned skill) use the
separate `PII.ownership_emails` check. Both that check and ordinary
`PII.emails` findings remain **HIGH and blocking by default**.

If your repository policy permits those ownership contacts, pass an explicit
policy file. The finding stays visible in the report as informational.

**`ownership-policy.yaml`**

```yaml title="ownership-policy.yaml"
severity_overrides:
  PII.ownership_emails: info
```

```bash
skillevaluator tier1 ./skills/my-skill --no-llm --policy ./ownership-policy.yaml
```

Supported values are one address or `Name <address>` in valid YAML. The same
address in body text, a people directory, another metadata field, or another
file (including a `SKILL.md` nested in a subdirectory) still uses the ordinary
email policy. Ambiguous YAML, aliases, and block scalars also retain that
policy. Credential checks remain blocking.

Keep this policy under repository-owner review; the scanner does not discover
or trust a policy from the skill automatically. Existing `PII.emails` overrides
apply to ordinary emails; use `PII.ownership_emails` explicitly for these contacts.
This option is available on `tier1`, `tier1 validate`, and `validate`. The
released reusable CI templates do not yet expose a custom-policy input.

> **Note**
>
> The active profile name is stamped into the HTML report header and JSON report, and printed in the run banner under `--verbose`, so reviewers can see which gate was applied.

## The checks

For `validate`, the default set is `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, and `lint`. The `dependency` check requires explicit selection with `--checks` (alias `--tier1-checks`) for `validate`; direct `tier1 PATH` includes it by default. Accepted check-name aliases include `code`/`code-risk` for `code-integrity`, `scripts`/`script-lint` for `lint`, `dependencies`/`deps`/`dependency-audit` for `dependency`, and `licence`/`license-check` for `license`.

#### schema — frontmatter and repository governance

Validates `SKILL.md` frontmatter and repository placement:

* YAML frontmatter must parse and satisfy the skill schema.
* Frontmatter keys, including keys nested under `metadata`, must be strings. An unquoted key that YAML reads as a number, boolean, null, or date (`123`, `yes`, `null`, `2026-01-01`) is a HIGH finding that names the key in YAML form, such as `Field 'metadata.true': Keys should be strings`; quote the key (`"yes":`) to keep it. Rules, workflows, and plugin manifests report non-string keys as schema errors too.
* `name` and `description` are required. Names are 1–64 characters; descriptions are 1–1024 characters.
* Names use kebab-case, start with a letter, and cannot have trailing or consecutive hyphens. The directory name must match the frontmatter `name`. The reserved words `anthropic` and `claude` must not appear in a skill name.
* `alwaysApply` and `globs` are forbidden skill fields. Optional fields include `license`, `compatibility`, `metadata`, and `allowed-tools`.
* The standard hierarchy is `skills/<skill-name>/` or `team-skills/<team>/<skill-name>/`. A nonstandard standalone location is reported as an advisory finding.
* Recognized skill-root subdirectories are `agents/`, `references/`, `scripts/`, `assets/`, `evals/`, `tests/`, `tools/`, and `config/`; anything else gets a LOW advisory finding. Extend the set per repository with `SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS` (comma-separated names are added to the defaults, never replace them).
* `SKILL.md` should stay at or below 500 lines; larger files receive an advisory finding.
* Harness portability: patterns the spec allows but common agent harnesses (Claude Code, Codex, OpenCode, OpenClaw, Hermes, Copilot CLI) load differently are reported; override any of them with `SCHEMA.<check_name>` in the validation policy.
  * A duplicate frontmatter key is HIGH (`duplicate_key`; several harnesses drop the skill), and a UTF-8 BOM is MEDIUM (`manifest_bom`).
  * A description within 1,024 characters but over 1,024 UTF-16 units is MEDIUM (`description_utf16_length`), and no "Use when ..." trigger in the first 60 characters is LOW (`description_trigger`).
  * A symlinked `SKILL.md` (`manifest_symlink`) or a `SKILL.md` nested inside the skill (`nested_manifest`) is MEDIUM.
  * `manifest_size` is LOW above 32 KiB, MEDIUM above 50 KiB and HIGH above 256,000 bytes.
  * Fields outside the spec are LOW (`unknown_field`).
* `metadata.author` is optional under the `default` profile. Use `Name <email@host>` to include a contact address, or a person, team, or organization name such as `NVIDIA cuOpt Team` or `Jane Doe (@jdoe)`. A value that is malformed or unsafe is a HIGH `author_format` finding: a bare email or an email outside `Name <email@host>`, stray angle brackets, control, private-use, or bidirectional formatting characters (joiners such as ZWNJ and ZWJ, which Persian, Arabic and Devanagari names need, are accepted), no letters or digits, or more than 256 characters. A bare address counts even when its domain has no dot (`root@buildhost`), while a social handle such as `Jane Doe (@jdoe)` does not. The `registry` profile requires the `Name <email@host>` form.
* `metadata.id` is an optional stable skill ID. When present it must be an exact 1–64 character lowercase ASCII slug (`^[a-z0-9]+(-[a-z0-9]+)*$`, no surrounding whitespace); it may differ from `name`. A collection run reports an ID that more than one discovered skill declares; `validate` on a catalog also lists these in `catalog-summary.json` under `metadata_id_conflicts`. Under `registry` a shared ID blocks: the run exits nonzero, and each sharing skill's entry is `"passed": false` with the conflict repeated under `catalog_findings` and counted in its `severity_counts`. A catalog validates each skill on its own, so that skill's own report was written before the catalog-wide check ran and keeps its single-skill verdict, which the entry carries alongside as `report_verdict` beside `report_scope: "single_skill"`. Uniqueness is checked only across the skills validated together, never against a registry, and a single-skill run does not claim it. The ID does not change local catalog identity, which stays keyed by content type and relative path.
* `metadata.domain` accepts one category string or a list of them, such as `domain: [backend, ai-ml]`. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGH `Field 'metadata.domain'` finding. The 128-character limit also applies to a single string.

**To fix common findings:** rename the directory to match `name`, remove forbidden fields, and move the skill under `skills/`.

#### harness — harness compatibility

This check is included in 1.0.1; see [Installation](/skills/skillevaluator/installation#install) for availability. It is not included in 0.9.0.

The `harness` check reports, for each agent harness, whether the skill `loads`, is `degraded` (loads, but something is cut, hidden or ignored) or is `not_loaded`, with the rule IDs behind each verdict and the harness version and date the profile was verified against. Supported harnesses: `claude-code`, `codex`, `copilot-cli`, `hermes`, `openclaw`, `opencode`. All of this is static: no agent or model is run. See [Harness Check](/skills/skillevaluator/harness-check) for a walkthrough with example output.

`validate` runs it by default for all six and lists its findings in the HTML, JSON and SARIF reports; JSON reports also get a `harness_compatibility` block. Use `--checks harness` to run it alone and `--harness codex,claude-code` to narrow the harnesses.

The profiles ship with the package and come from repeated runs of each harness in which the request sent to the model was captured. The check doesn't change pass/fail: its findings are MEDIUM (`not_loaded`), LOW (`degraded`) or INFO (`advisory`), so it never fails a run unless a policy raises those severities. Install-location rules apply only when the skill sits under a harness root such as `.claude/skills` or `.agents/skills`. Use `--harness-profiles DIR` to try newer profile YAMLs without a release.

`validate` also shows a "Harness compatibility" panel after Tier 1, `! harness` in the checks line when a harness drops or degrades the skill, a `harness` line in the Tier 1 summary (kept out of the advisories count) and a note in the result box. `skillevaluator harness-check <skill>` runs only this check and shows the same panel, updated as each harness is checked. It accepts a folder of skills, `--harness`, `-v` and `--json PATH`, always exits 0, and points to `validate --tiers 1` when Tier 1 schema also fails. By default the panel is compact: one line per harness with its most serious finding (not loaded, then degraded, then note; ties go to the rule listed first in the profile) labelled with its outcome (`not found`, `rejected`, `listing shortened`, `delivery truncated` and so on), then `+N more` and the number of notes. `-v` (also on `validate`) lists every finding with its fix and each harness's verified version.

Frontmatter rules: duplicate keys (Codex, Copilot CLI, OpenClaw and OpenCode drop the skill), tab indentation (Codex, Copilot CLI and OpenClaw), a UTF-8 BOM (Codex), no front matter (Copilot CLI), and long custom frontmatter that never reaches the model.

Description rules: missing or empty descriptions, Claude Code and Codex listing cuts, Copilot CLI's 1,024 UTF-16 unit limit, quotes escaped in the Copilot CLI and OpenClaw lists, and Hermes's 60-character listing. `-v` also shows the description each harness puts in the model's skill list.

Location rules: `SKILL.md` casing, nested manifests, symlinked manifests and resources, OpenClaw symlink escapes, Codex scan depth, and files Hermes leaves out of its file list (outside `references/`, `templates/`, `assets/` and `scripts/`).

The check measures the skill together with the skills installed next to it. A single skill under a harness root such as `.claude/skills`, or in a workspace `skills/` folder, is counted with everything in that folder (symlinked skill folders included). Skills in different folders are separate installations, and a folder of skills is its own catalog. That drives the list-budget rules (Claude Code and Codex \~8,000 description chars, Copilot 32 full descriptions, OpenClaw 150 skills or 18,000 chars) and a duplicate-name rule per harness that says which copy wins. A `catalog` line in the panel names the folder. Copilot CLI and OpenCode also get an advisory note on when new or edited skills are picked up.

Delivery rules: Claude Code's body cap, OpenCode's 51,200-byte / 2,000-line reads and lines over 2,000 characters, OpenClaw's whole-or-nothing read against its result caps, Hermes's 100,000-character `skill_view` budget, and Codex's \~10K-token cut when the model reads `SKILL.md` through the shell.

Placeholder rules: `$ARGUMENTS` and `${CLAUDE_SKILL_DIR}` are substituted only by Claude Code, and price-like `$N` text is replaced with arguments by Claude Code and OpenCode.

Invocation rules: `disable-model-invocation` (ignored by Codex, Hermes and OpenCode), Codex's `agents/openai.yaml` implicit-invocation policy (Codex only), `allowed-tools`, Claude Code-only fields, and Hermes project trust.

Gate rules: Hermes `platforms` and OpenClaw `metadata.openclaw.requires` hide the skill on hosts that don't match.

`harness-check` on a plugin folder also shows a plugin panel: which plugin formats each harness accepts (`.claude-plugin`, `.codex-plugin`, Copilot, Agent Plugins v1, Hermes `plugin.yaml`), and an empty plugin.

Plugin component rules cover skill namespacing, Codex's plugin-skill cap and Agent Plugins hooks, Copilot CLI MCP server names, disabled plugins, and components a harness registers but doesn't deliver.

`harness-check` takes the model setup some limits depend on: `--context-tokens`, `--custom-provider`, `--explicit-invocation` and `--headless`.

Each rule records a confidence level and, where behaviour changed between releases, the versions it applies to (shown with `-v`). `--harness-version codex=0.170.0` adds a note when you run a release other than the one a profile was verified on.

#### security — vulnerability scanning

`security-scan`, and the `security` check inside `validate`, run SkillSpector. Static scanning covers common skill attack patterns such as prompt injection and data exfiltration; each finding carries the matched pattern, severity, and location.

**`Security scanning`**

```bash title="Security scanning"
skillevaluator security-scan ./my-skill
skillevaluator security-scan ./my-skill --llm
skillevaluator security-scan ./my-skill --llm --llm-verify
```

The LLM pass requires a configured provider. It enriches the static result but does not replace static findings or make missing scanner evidence pass.

SkillSpector's text pattern analyzers cannot read images, so it reports the analysis of a skill that references a PNG or JPEG file as `partial`. SkillEvaluator accepts that gap with a warning instead of `INCOMPLETE` only when the report shows no other gap apart from the missing references described next, and no fatal scope exclusion, and each referenced image is a regular, single-link (not hard-linked), non-executable file whose bytes have the chunk structure of a PNG or the marker structure of a JPEG and that is not reached through a symlink. The gap is accepted for at most 256 images and 128 MiB of image data in total. An image is never accepted inside a `hooks`, `scripts`, `tools`, `agents`, `commands`, `prompts`, `rules`, `team-rules` or `workflows` directory at any depth, or under a top-level `config`, `skills`, `.mcp`, `.agents`, `.claude`, `.codex`, `.continue`, `.cursor`, `.gemini` or `.windsurf` directory (names compared without regard to case). The structure check corroborates the file type; it does not decode the image or prove that it renders. PNG IDAT compressed data and ancillary chunks, and JPEG entropy-coded data and APPn or COM segments, are not inspected and can still carry other bytes. The image content itself is not assessed. Any other partial reason, except the missing references described next and the hooks described below, still leaves the scan `INCOMPLETE`. SkillSpector also reports a High `AE1` (incomplete referenced artifact analysis) finding for such an image; from v2.12.0 it omits that finding when the skill renders a PNG with Markdown image syntax, such as `![Logo](assets/logo.png)`. An `AE1` finding fails the check like any other High finding.

SkillSpector also reports the analysis as `partial` when the skill's instructions name a file that it did not scan, such as a file the skill writes at run time (`output/notes.md`) or a source file elsewhere in the repository. SkillEvaluator accepts that gap with a warning instead of `INCOMPLETE` when each such reference meets all of these conditions:

* It is a nonfatal `reference_resolution` row for one line of a file that the report lists once as a component and that is a regular, single-link file of at most 1,000,000 bytes inside the skill, not reached through a symlink.
* The report's complete `references` list records a `missing` reference on that line and quotes part of the line. Every `missing` entry must have an accepted ledger row at the same source path and line; an `ambiguous` entry anywhere, a truncation marker, or an unknown completeness field leaves the scan `INCOMPLETE`.
* The line, at most 10,000 characters long, names nothing in the skill that SkillSpector did not inspect: a symlink, a file the report does not list, a directory that holds no listed file, such as `versions/`, `node_modules/` or `evals/results/`, which SkillEvaluator leaves out of the scanned copy, anything inside such a directory, or a directory that holds any of these, even beside listed files. A `.git` directory counts by its own name, by its Git hooks other than `*.sample` templates, and by every other entry in it that is not part of Git's own layout, such as a script placed there, with everything inside that entry. Git's own files, such as `HEAD`, `config`, `index`, `packed-refs` or `ORIG_HEAD`, and what its `objects`, `refs`, `logs` and `info` directories hold, such as branch names, do not count. Names are compared without regard to case or Unicode normalization, in the line as written and as a Markdown renderer or a reader of file paths may read it: with backslash escapes of ASCII punctuation (such as `node\_modules`) removed, HTML character references (such as `&#118;`) resolved and percent-escapes decoded, up to three of these steps in any order. A skill with more than 50,000 files and directories, or nested more than 64 directories deep, is not checked, and its scan stays `INCOMPLETE`. The work of checking a report's lines is also bounded: when it would take longer, such as for many long lines beside thousands of files left out of the scan, the scan stays `INCOMPLETE`.
* A word in the line that a shell, cmd.exe, PowerShell or Windows would expand or unquote is read as a pattern, and none of those names may match it. In the pattern, quotes and escapes are removed, a group such as `(v)` stands for what it holds, also right after `]` as in `[v](e)rsions`, each expansion, set of alternatives, extended glob or numeric range may stand for any text, and each bracket expression for any one character. The word is read as follows:
  * The way POSIX shells read globs, brace and parameter expansions, `$(...)` and quotes (`v*/*.sh`, `vers?ons`, `v[e]rsions`, `v{e,x}rsions`, `$DIR/run.sh`, `ver""sions`).
  * The way zsh reads groups (`(v)ersions`, `(v|x)ersions`), numeric ranges (`run<1-9>.sh`), the `EXTENDED_GLOB` operators `^`, `#` and `~` (`ver^x`, `verr#sions`, and `v*~x`, read as `v*`), where `x#` as a whole part, such as `C#`, matches only names made of `x`, and globbing flags such as `(#a2)` or `(#c1)`, which make the whole word stand for any text (`(#a2)virsions/x`).
  * The way bash and ksh read extended globs (`@(v)ersions`, `!(x)`).
  * The way cmd.exe reads the `^` escape (`ver^sions`), variables (`%DIR%`, `!DIR!`) and batch parameters (`%1`, `%~dp0`).
  * The way PowerShell reads backtick escapes.
  * With `\` as a directory separator (`v*\i*.sh`).
  * With a Windows short (8.3) name or a Samba mangled name, a path part with text after a `~` such as `NODE_M~1/BOOTST~1.SH`, standing for any name that holds the part's first character, such as any name with an `n` for `NODE_M~1`. So `3~5` is read as written, as `3`, and as any name with a `3`.
  * A span between backticks is read as Markdown code, as written and with its backticks removed, not as a command, so a plural written as `` `topic`s `` is read as `topics`. A leading Markdown `* ` list marker is ignored.
  * A `$` before a space or the end of the line is read as a literal `$`, as a shell reads it, not as an expansion, so a shell prompt `$ python run.py` adds no pattern, and `sh *a$` matches only names that end in `a$`.
  * A bold or italic label outside code, such as `**Note:**`, `*Tip:*` or `**Step 1:**`, is read as its text, and each of its words, with its `*`, as a pattern that is not split at `:`, `=` or `,`. So as a pattern, `**Note:**` matches only names that hold `note:`, and `**Step 1:**` only names that end in `step` or start with `1:`. A standalone shell wildcard elsewhere, such as `bash *`, still counts, and so does a label inside code.
  * On a line with a space or shell delimiter between two of the same quote or after a backslash, as written or decoded as above, such as `setup' 'helper.sh`, `"hi there"` or `setup\ h*.sh`, a `$` before a space and a label are read like any other word, so they still stand for any text: a shell may join words there, which the check does not. A PowerShell backtick before a space does not count, as a backtick also ends Markdown code, so ``$ sh setup` h*.ps1`` does not name `setup helper.ps1`. A PowerShell backtick before a space does not count here: in ``**Note:** run setup` h*.ps1`` the label is read as its text, and ``setup` h*.ps1`` does not match a left-out `setup helper.ps1`, which PowerShell would run.
  * Prose with this syntax is read the same way: a price such as `$5`, a list such as `{a,b}`, `%APPDATA%` or a lone `^` stands for any text, so a line with one stays `INCOMPLETE` whenever anything is left out of the scan, and `approx~10` stays `INCOMPLETE` when a name left out holds an `a`.
* The check reads names that are written in the reported line, in the spellings above. It does not read other lines and does not run commands, so it does not detect a name on another line, such as `DIR=node_modules` set on one line and used as `sh $DIR/run.sh` on the next, or a name that a command prints or builds, such as `printf` output in backticks, a list piped into `xargs sh`, `find -exec`, or a PowerShell string joined with `+`. Nor does it join words at a quoted or escaped space as a shell does, so `setup' 'helper.sh` or `setup\ h*.sh` does not name `setup helper.sh`. It is meant to keep an instruction that names a left-out file `INCOMPLETE`, not to catch a skill written to hide one. Without this check, such a skill passes whenever SkillSpector reports no missing reference in it; with it, the skill also passes when every line SkillSpector reports is another missing reference, such as `output/notes.md`. SkillSpector's own analyzers, not this check, assess the instructions it scanned.
* When the target is a single file, such as `skillevaluator security-scan path/to/SKILL.md`, SkillSpector scans that file alone, so every other file and directory beside it counts as not inspected. A line in it that names a file beside it, such as `scripts/install.sh`, keeps the scan `INCOMPLETE`.
* Tier 3 output in its default location, `<skill>/evals/results`, is left out of the scanned copy too, so it counts as not inspected. The rest of `evals/`, such as `evals/evals.json` and input fixtures in `evals/files`, is scanned. After a Tier 3 run, a line that names `results`, a file the run leaves there, such as `report.html` or `summary.json`, or, when the run used `--harbor-keep-jobs`, any bundled file (the kept Harbor tasks hold a copy of the skill) keeps the scan `INCOMPLETE`, although the same line passed before the run. Writing Tier 3 results outside the skill, with `--results-dir` or `SKILLEVALUATOR_RESULTS_DIR`, keeps a run from changing the result.
* The scan covers the whole target. When the target is a folder or a plugin that holds skills, such as `skillevaluator security-scan path/to/skills/`, `skillevaluator validate path/to/skills/` (which checks each skill as its own job) or `validate` on a plugin, each skill in it is scanned on its own, and a line may name a file elsewhere in the target that the skill's scan did not inspect, so those skills' missing references keep their scans `INCOMPLETE`, also when a skill in a folder is a plugin of its own.

Every nonfatal scope exclusion must name an actual uninspected entry under the target. Accepted missing-reference warnings remain visible after the static and LLM stages deduplicate their findings, and after `--llm-verify` downgrades a finding. Any other gap must be a referenced image that SkillEvaluator accepts as described above, and the scan then carries both warnings. Without such images, the report must also show no partially or entirely uninspected file, no limitation and no fatal scope exclusion, and every analyzer must have completed or be not applicable or disabled. In either case the report must count as many findings after filtering as before it, and a `missing` reference without an accepted ledger row, an `ambiguous` reference or a truncation marker leaves the scan `INCOMPLETE` even when a referenced image is its only listed gap. The warning names the file and line of up to three of the references; it does not quote the referenced path. Files not in the scan, including any outside the scanned path, are not inspected.

SkillSpector v2.12.0 to v2.13.3 report these references as `reference_missing`, and v2.10 and v2.11 as `reference_unresolved`, which SkillEvaluator accepts under the same conditions. A v2.13.3 report whose only gaps are such references is also accepted from the report alone as a missing-reference caveat: an INFO finding that keeps SkillSpector's `partial` status and is recorded as `skillspector_reference_caveats` in the JSON report. It replaces the warning above, and whenever the scan has a known target, as through `validate`, `tier1` or `security-scan`, it is accepted only when the conditions above hold too. A reference that matches several bundled files (`reference_unresolved` from v2.12.0, `ambiguous` in the `references` list before it) still leaves the scan `INCOMPLETE`.

From v2.13.4, SkillSpector no longer reports such a scan as `partial`. When it inspected every bundled file and the only gaps are references to files the skill does not bundle, it reports the status `complete_with_caveats` and lists each reference as a caveat. SkillEvaluator accepts these caveats under the same conditions and shows the missing-reference warning above, and the findings of an accepted report are checked as in any other complete scan. A caveat on a path that is not a bundled file, or one that fails a condition above, such as a line that names a file left out of the scan, leaves the scan `INCOMPLETE`; as for a `partial` report, the report's findings are still checked and reported. Unlike a v2.13.3 caveat, these caveats are never accepted from the report alone, so they leave a scan without a known target `INCOMPLETE`, and an accepted report carries no `skillspector_reference_caveats` entry. SkillSpector gives this status only when every gap is a nonfatal `reference_missing` row, so a report with no caveat, or with a caveat that is not a nonfatal `reference_missing` row in phase `reference_resolution`, contradicts itself: SkillEvaluator does not trust it, and the scan is `INCOMPLETE` without its findings. The same holds for `complete_with_caveats` from a release before v2.13.4.

SkillSpector v2.12.0 and later cannot model what an ordinary command or HTTP hook, such as one that runs `echo hi` or a bundled script, does with its hook input. The exception is a hook that SkillSpector recognizes as sending hook content to a remote host, which it reports as **BH2**; a manifest whose command and HTTP hooks are all of that kind counts as modeled. For other hooks it discovers in manifests such as `hooks/hooks.json` and `.claude/settings.json`, SkillSpector reports a **BH1** finding whose evidence says the payload is `unmodeled`, records a nonfatal `partial` `opaque_content` row from `bundled_execution_surface` on the manifest, and reports the analysis as `partial` although the scan finished. This rule applies only to manifests SkillSpector actually reports with that evidence and ledger row; it does not establish coverage for inline plugin-manifest hooks or other hook locations the scanner does not report. The scan's findings cannot show that such hooks are safe: a hook script that sends its input over a socket, `nc` or `/dev/tcp` gets no blocking SkillSpector finding. SkillEvaluator therefore reports the manifest as the Critical `skillspector_hook_payload_unverified`, with a gate failure that neither a `--policy` severity override nor `--llm-verify` clears, instead of `INCOMPLETE` (see [Troubleshooting](#troubleshooting)). This applies when the row is a whole-file row that names only `bundled_execution_surface` and no error class, on a manifest whose BH1 says `unmodeled` (also when a baseline suppresses that BH1), and the report accounts for it exactly: one such row per manifest, `bundled_execution_surface` degraded by exactly those rows with nothing failed or skipped, its `Analyzer bundled_execution_surface status: degraded.` limitation, and one partially inspected file for each manifest that no other row names. The rest of the report is then judged on its own: it is complete when it shows no other gap, and missing references and referenced images can still be accepted with their warnings as described above. The v2.13.3 missing-reference caveat that is accepted from the report alone does not apply, because that check reads the whole report, hook row included. Any other gap still records the SkillSpector scan as incomplete (`incomplete_scans` in the JSON report, and publication eligibility **Incomplete**), with an error that names the other rows, or the other gaps when no other row remains. The check itself is then reported as `FAIL`, not `INCOMPLETE`, because the Critical and its gate failure are a confirmed failure, which takes precedence over missing evidence, also when a `--policy` override lowers the Critical (see [Reading the final status](/skills/skillevaluator/reports#reading-the-final-status)). A row that does not fit, such as one for a manifest in which SkillSpector parsed no hook (no `unmodeled` BH1), or a report that does not account for it, leaves the whole scan `INCOMPLETE` as before.

SkillSpector records the same row when it could not parse part of the manifest or read only its first 2,048 hook, permission and settings records. How much of that SkillEvaluator can see depends on the release:

* v2.13.4 reports `observed_records` and `limit_records` on the row. SkillEvaluator accepts the row only when `observed_records` equals the BH1 `declaration_count` and does not exceed `limit_records`. A manifest with more records than SkillSpector reads, or with a hook handler it counted but could not parse, therefore stays `INCOMPLETE`. So does a settings file with permission rules, `env` or other settings beside its hooks, because SkillSpector counts those records too. A part it does not count, such as a hook group that is not an object, is not visible and gets the Critical.
* v2.13.3 does not report them. SkillEvaluator then refuses only a BH1 that counts 2,048 or more declarations, because SkillSpector may have cut that manifest short. A manifest that SkillSpector parsed only in part, such as one with a malformed handler beside a valid one, gets the same Critical as a manifest it read in full, and still fails.

**Incomplete referenced artifacts.** SkillSpector reports a High **Incomplete referenced artifact analysis (AE1)** finding at each line that names a file it did not completely inspect. Releases before v2.12.0 leave the finding's name empty, so SkillEvaluator names it by its rule instead of `Unknown (AE1)`. A `severity_overrides` entry for these findings uses `SECURITY.Incomplete referenced artifact analysis (AE1)`; one keyed on `SECURITY.Unknown (AE1)` no longer matches. The finding's suggestion names the file, its disposition (`partial`, `failed`, `out_of_scope` or `uninspected`) and each reason SkillSpector gives: the reason code, such as `static_parse_limit` or `read_error`, the analyzers, the line in the file, the message and any limits, such as `limit_characters=8000`. The suggestion is part of the finding's `finding_id`, so it leaves out the run time and the time budget (`observed_seconds`, `limit_seconds`), which differ from run to run; the metadata keeps them. Reasons that differ only in the analyzer, the run time or the time budget are listed once. SkillSpector's own remedy is kept. Releases before v2.12.0 give no reasons in the finding and a generic remedy that says to make the file available: SkillEvaluator then reads the reasons from the report's `analysis_completeness` rows for that file and gives a remedy for a parse limit, a read error, or any other reason. Without such a row the finding keeps SkillSpector's text. The finding's metadata records the file, its disposition and up to 16 reasons under `skillspector_evidence`, with `reasons_truncated` when SkillSpector gave more, and up to 128 reference lines under `skillspector_occurrences`. In a folder of skills each reference line's `file` starts with the skill's `[name]` prefix, like the finding's own path, and `target_path` stays relative to that skill. Text with control characters is left out, credential-shaped text is masked, and a path outside the scanned target is dropped.

When the check fails, the terminal output, the `references` row of the terminal summary, and the Markdown and HTML reports add one line, such as `skillspector could not completely inspect 1 referenced artifact(s) across 2 reference occurrence(s); each AE1 finding names the artifact and any reasons SkillSpector gave`. The HTML and Markdown reports that the CI workflow renders again from the JSON report show the same line. The JSON report records the counts for every result, passed or failed, in `skillspector_reference_summary`: `affected_artifacts`, `reference_occurrences`, and `reference_occurrences_truncated`, which is `true` when SkillSpector listed more than 128 lines for one finding, so the count is a lower bound. For a folder of skills the counts are totals. None of this changes the finding's severity or whether the scan is complete. A gate changes only where a policy lowered `SECURITY.Unknown (AE1)` and that was what let the check pass.

**Agent control paths.** SkillSpector rules describe attacks on an agent, so a match is an agent threat only in a file the agent reads or runs. When `validate` or `tier1` detects a skill, rules, workflows, or plugin artifact, the `security` check keeps findings on agent control paths and suppresses the rest:

| File role                | Tier 1 finding | Examples                                                                                                                                                                                                                                                                                                                    |
| ------------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Agent instruction        | Kept           | `SKILL.md`, `AGENTS.md`, `*.mdc`, `commands/`, `prompts/`, `rules/`, `workflows/`, plugin `output-styles/`                                                                                                                                                                                                                  |
| Agent reference          | Kept           | `references/`, `templates/`, non-image `assets/`, other data and code files at the artifact root                                                                                                                                                                                                                            |
| Agent executable         | Kept           | `scripts/`, `tools/`, `hooks/`, plugin `.lsp.json` and `monitors/monitors.json`, and any file SkillSpector marks executable (for example `.py`, `.sh`, `.js`, `.tsx`)                                                                                                                                                       |
| Agent configuration      | Kept           | top-level `config/`, `.claude/`, `.codex/`, `.cursor/`, and other agent settings directories, plugin `.app.json`                                                                                                                                                                                                            |
| MCP and plugin manifests | Kept           | `.mcp.json`, `mcp.yaml`, `agent_plugin.yaml`, `plugin.json` in `.claude-plugin/`, `.codex-plugin/`, or `.cursor-plugin/`                                                                                                                                                                                                    |
| Referenced file          | Kept           | any file a kept file names by path, by file name alone (`helper.md` for `lib/helper.md`), by skill or plugin root variable, or by import, followed transitively                                                                                                                                                             |
| Ordinary file            | Suppressed     | source and UI files that SkillSpector does not mark executable (such as `.html`, CSS, `.vue`, and `.cpp`), test fixtures, `docs/` and `.rst` documents, notebooks, images, lockfiles, repository metadata (`.github/workflows/`, `.gitignore`, and top-level dotfiles that are not agent files, such as `.env`), other data |

SkillSpector marks source files executable wherever they are, including test suites under `tests/` and UI code under `web/`. Current releases (v2.12.0 and later) cover `.py`, `.js`, `.ts`, `.tsx`, `.jsx`, `.mjs`, `.cjs`, `.mts`, `.cts`, `.svg`, `.php`, `.class`, and `.wasm`. Those files are agent executables and seed the reference walk, so their findings stay blocking, and so do the findings of any fixture or other file they name.

The reference walk recognizes these forms, without regard to case:

* A path relative to the artifact or to the referring file, such as `docs/guide.md`, `./docs/guide.md`, `../lib/helper.py`, or `docs\guide.md`.
* A path behind a skill or plugin root: `${CLAUDE_SKILL_DIR}/`, `$SKILL_DIR/`, `$SKILL_ROOT/`, and `${CLAUDE_PLUGIN_ROOT}/`, in shell, PowerShell (`$env:NAME`), and cmd (`%NAME%`) forms, and a skill-directory placeholder whose name contains `skill` or `base`, such as `{baseDir}/`, `{{skill_dir}}/`, or `<skill-dir>/`.
* A path with a directory behind any other prefix, such as an install location (`~/.claude/skills/demo/docs/guide.md`), a root-relative path (`/docs/guide.md`), or another placeholder (`<root>/docs/guide.md`). A top-level file behind a prefix counts when the prefix ends with the artifact's directory name (`~/.claude/skills/demo/README.md` for a skill in `demo/`).
* A bare file name of a nested file, a directory name for every file under it, and Python and JavaScript imports, including `-m` module forms. In Markdown and other prose, a top-level directory name counts only with a slash, a `./` prefix, in quotes or backticks (`docs/`, `./docs`, `` `docs` ``), or as a word or glob inside a code span or fenced block, because the bare word also appears in ordinary prose such as "build a web page from the docs". In scripts and other code, including any file SkillSpector marks executable, a bare directory name also counts, because commands such as `find policies -type f` and `cd policies` take it as an argument.

SkillSpector can report identical matches from several files as one issue: `location` names one of the files and `occurrences` lists all of them. Such an issue is suppressed only when every file in it is outside agent control paths. If any occurrence is on a control path or in a referenced file, the issue is kept with that file's role.

A single-file target, such as one rules file, is always an agent control path. Supply-chain findings (`SC*`), YARA malware, webshell, and cryptominer signatures (`YR1`–`YR3`), and artifact-integrity findings for content that does not match its extension or a text file with embedded NUL bytes (`AE2`, `AE3`) are kept on every path because they describe what the artifact ships. A file named as a raster image (`.png`, `.jpg`, `.gif`, `.webp`, `.ico`, `.avif`) is treated as an image only when its content starts with an image signature; otherwise its findings are kept with file role `unverified_image`. Suppressed issues no longer count toward the aggregate SkillSpector risk-score gate. The terminal summary shows a `filtered` row, and the JSON report records a per-result `skillspector_applicability` block with the kept and suppressed counts, the suppressed counts by file role, and the raw and applicable risk scores. Kept findings carry `artifact_type`, `file_role`, and `applicability` metadata.

Classification fails closed when it cannot place a finding or finish the reference walk:

* An issue keeps its finding when a location or occurrence is missing, outside the artifact, or reached through an escaping symlink, or when its `occurrences` field is malformed.
* Every finding is kept when the report serializes fewer occurrence records than its `analysis_completeness.findings_after_filtering` count, because the unserialized findings have no location.
* Every finding is kept when the walk meets a symlinked directory, a file it cannot read as text, or a plugin manifest it cannot load, such as one with an invalid YAML date. Verified raster images and fonts are the exception, because they hold no path text. A PDF, Office document, or other binary file in a control location, such as `assets/guide.pdf`, therefore keeps the unrestricted scan for that skill.
* Every finding is kept when the walk exceeds a budget: a file it reads is over 1 MB, it reads more than 5 MB or 1,000 files, the artifact has more than 10,000 files, or it needs too many reference comparisons. Very large skills can therefore keep the unrestricted scan.

The walk recognizes references written as text. A file that a control file reaches only through a path it builds at run time, such as `"web/" + name`, is not followed. Neither is a top-level file named behind a prefix that is not a recognized root variable or placeholder and does not end with the artifact's directory name, such as `<root>/README.md` or an install directory with a different name. Top-level files are agent references whether or not they are named, except READMEs, `.rst` documents, dotfiles that are not recognized agent files (such as `.env`; `.cursorrules` and `.mcp.json` stay kept), test files, notebooks, images, lockfiles, and generated `skill-card.md` and `BENCHMARK.md` files.

`security-scan`, and `validate` on a target whose type is `unknown`, keep the unrestricted scan and report every SkillSpector finding.

To explain why two otherwise identical `--llm` scans differ, the reports record the LLM provenance that SkillSpector v2.12.0 and later return. The Security result in the JSON report carries:

* `skillspector_llm_provenance`: SkillSpector's provider adapter, the model and SkillSpector revision behind each LLM analyzer, the requested and forwarded temperature, seed, and reasoning-effort controls, and a determinism classification. Provider output is never guaranteed to be identical.
* `skillspector_provider_id`: the SkillEvaluator provider the scan was bridged from (`nv_build`, `openai`, `openai-compatible`, `anthropic`, or `bedrock`). `nv_build` and `openai-compatible` both run as SkillSpector's `openai` adapter. The field is omitted when `SKILLSPECTOR_PROVIDER` selects a SkillSpector adapter directly (`anthropic`, `bedrock`, or `openai`) or names an unsupported provider. `SKILLSPECTOR_PROVIDER=nv_build` is bridged like `SKILL_EVAL_LLM_PROVIDER=nv_build` and recorded as `nv_build`.

The HTML report shows the same data in the **Tier 1 LLM Provenance** table on the Tier 1 tab. The record contains no prompts, endpoints, or credentials. A record that does not match SkillSpector's provenance contract, or that contains a URL or credential-shaped value, is left out rather than partially shown. If the LLM stage fails or SkillSpector does not confirm LLM analysis, a valid record in SkillSpector's output is still kept to show what was attempted, and the Security result stays incomplete. Provenance is recorded when you scan a single skill. A scan of a folder with several skills does not record it yet.

SkillSpector 2.11.1 and later tags an exact read-only `/etc/passwd` bind mount, such as `docker run -v /etc/passwd:/etc/passwd:ro` or the Podman, `--volume`, and quoted forms, as likely benign but still reports it as a HIGH Credential Access (PE3) finding. SkillEvaluator ignores that finding only when SkillSpector tagged it benign and every `/etc/passwd` on the reported source line is the source or target of such a mount. Writable or implicit-mode mounts, other source or target paths, and reads such as `cat /etc/passwd` still block.

SkillSpector 2.13.3 can report separate findings for the mount's source and target; both must pass the same checks. Ignoring a mount never clears incomplete inspection, including refused or truncated LLM responses. A static report can say an LLM provider is available without having requested or applied LLM analysis; provider availability alone does not invalidate the static scan.

To accept reviewed findings, pass an audited baseline with `--skillspector-baseline`; see [Audited SkillSpector baselines](#audited-skillspector-baselines).

#### pii — PII, credentials, and local identifiers

`pii-scan`, and the `pii` check inside `validate`, use configurable regex patterns to detect sensitive values.
Instruction content in `.mdc` rules and workflow files is included, for both
direct file inputs and directory scans.

| Group                   | Examples                                                                                                     | Notes                                                                                                                                                                                                 |
| ----------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Personal information    | Personal paths, non-placeholder email addresses, phone numbers, SSNs, GPS coordinates, MAC addresses         | Personal macOS/Windows paths are flagged; SSNs are CRITICAL                                                                                                                                           |
| Credentials and secrets | Database connection strings, hardcoded API keys, cloud keys, GitHub tokens, private keys, JWTs, webhook URLs | Most credential findings are CRITICAL                                                                                                                                                                 |
| Network and financial   | Public IPs, credit-card numbers, cryptocurrency wallet addresses                                             | Private RFC1918, loopback (`127.x`), link-local (`169.254.x.x`, including cloud metadata endpoints such as `169.254.169.254`), and `0.x` (including the `0.0.0.0` placeholder) addresses are excluded |

Numeric values are judged by their context as well as their shape. A card-shaped number is not reported when a timestamp or digest label (`timestamp_microseconds:`, `event_timestamp:`, `epoch`, `epoch_us`, `unix_time`, `sha256:`, `checksum=`) directly precedes it or when it sits inside a longer ASCII token, such as a hex digest or an identifier like `id_1707938736269577` where an underscore joins it to a letter or digit (unless a payment word such as `card`, `cc`, or `pan` is part of that identifier, as in `card_4532015112830366`). A card in Markdown underscore emphasis (`_4532015112830366_`), after a string escape such as `\n` or `\t` or a percent-encoded byte such as `%3D`, or written directly next to Chinese, Japanese, Korean, or other non-ASCII letters (including the Turkish dotless `ı`) is still reported. Widely published test card numbers (`4111111111111111`, `4242424242424242`, `5555555555554444`, and similar) are excepted however their digits are grouped, without hiding a different card number on the same line. An international phone candidate that starts inside a scientific-notation value such as `1.000000e+00` is ignored. A GPS-shaped decimal pair is reported only with a geospatial cue on its line, on one of the two lines before it, or in the file name. Strong cues are GPS, latitude/longitude, lat/lng, `latlong`/`lonlat` keys, WGS 84, a `geo:` URI, a Google, Apple, OpenStreetMap, or Bing map link, and a GeoJSON geometry, `Feature`, or collection `"type"` or `"geometry"` member. Soft cues are location, office, address, degrees, and `°`. Cue words count in their plural forms, with a numeric suffix, and inside snake_case or camelCase keys and file names (`store_locations:`, `latlngs`, `lat1`, `homeLocation`, `gpsCoords`, `locations.json`). The GeoJSON `type` and `geometry` keys count double-quoted, single-quoted, or bare, so JSON, Python dicts, JS objects, and YAML all carry the cue; a bare type value must use GeoJSON's spelling (`Point`, `MultiPolygon`). Every position inside a GeoJSON geometry's `coordinates` array is cued, however deeply its rings are nested, when the geometry is written with braces (JSON, a Python dict, or a JS object); block-style YAML geometry is cued only through the two-line window. Technical labels outweigh soft cues but not strong ones: `waypoint`, `quaternion`, `pose`, `vector`, `calibration`, `roi`, and `scale` count only on the pair's own line and only before the pair (so `Location: 37.774929, -122.419416 (map scale 1:5000)` is still reported), while `<pixel|camera|world|...> coordinates` also counts on the two lines before it and in the file name. Pairs without a cue are not reported, for example `coordinates: 37.7749, -122.4194`, `Meet me at 37.7749, -122.4194`, or `home: 37.774929, -122.419416`.

Common placeholders such as `YOUR_API_KEY_HERE`, `example`, `test`, `dummy`, and `placeholder`, along with `@example.com` and `@test.com`, are excluded, as are configured allowed paths.

Proton (Steam Play) game prefixes are not personal paths: `pfx/drive_c/users/steamuser/` is the same fixed Windows profile for every player. Other user names, the same profile outside a `pfx/drive_c` prefix (for example `~/.wine/drive_c/users/steamuser/` or `/Users/steamuser/`), and a profile path followed by a `..` segment anywhere later on its line are still reported, as is any other personal path on the same line.

A card-shaped number is reported only when it passes the Luhn checksum; all-zero numbers do not. Digits inside a canonical GUID/UUID, such as the COM interface ID `{00000000-0000-0000-C000-000000000046}`, are treated as an identifier unless a payment label (`card_number`, `credit_card`, `cc_no`, `cc_number`, `pan`, and similar) directly precedes the GUID on its line or ends the line before it. Every card-shaped number on a line is checked, so an identifier or non-card number earlier on the line does not hide a card after it.

To accept a specific reviewed value, use a [PII waiver](#pii-waivers) rather than a `PII.*` severity override, which would relax every finding of that check.

The home-path check compares detected user paths against the submitter's identity. When no identity is available, that check is skipped and logged — set `SKILLEVALUATOR_SUBMITTER` (or add `author` to `SKILL.md`) to enable it. `SKILLEVALUATOR_SUBMITTER` is combined with `GITHUB_ACTOR`, `USER`, `LOGNAME`, `USERNAME`, the OS login name, and the skill's declared author — every identity that resolves is protected.

#### license — compliance

Looks for license evidence in frontmatter, common license files, and SPDX headers. Detected identifiers are normalized and matched against the packaged permissive allowlist and restrictive blocklist. Missing or unknown licenses are reported for review; blocked licenses fail according to the active policy (CRITICAL under the bundled `default` policy).

Frontmatter values and SPDX headers can be SPDX license expressions with `AND`, `OR`, `WITH`, and parentheses, such as `MIT OR Apache-2.0`. An expression passes only when every license in it is allowed, and any blocked license fails it. The `+`, `-only`, and `-or-later` forms of a blocked license, such as `GPL-3.0+` or `LGPL-2.1-only`, and deprecated ids such as `GPL-2.0-with-classpath-exception` are blocked too. A `WITH` exception keeps an allowed license allowed only when it is in the packaged `allowed_license_exceptions` list (`LLVM-exception`, `Swift-exception`); other exceptions are reported for review. An expression that uses an upper-case `AND`, `OR`, or `WITH` but has unbalanced parentheses or a missing or repeated operator fails with an `unsupported_license_expression` finding. A value without those operators, such as `MIT License` or `MIT License (MIT)`, is treated as a license name: it is matched by name as before or reported for review. A blocked license fails the value even when it does not parse, so `GPL-3.0 (or later)` and `(GPL-3.0 OR MIT` are blocked.

#### code-integrity — code risk, secrets, and hygiene

Combines three validation groups:

* **Code risk (Bandit and Semgrep)** — SQL or command injection, hardcoded passwords, insecure cryptography, unsafe deserialization, `shell=True`, `eval()`/`exec()`, CWE patterns, and language-specific anti-patterns.
* **Secrets (Gitleaks)** — API keys, tokens, passwords, private keys, database credentials, and other high-confidence secret patterns. Authored eval files, including `evals/evals.json`, graders such as `evals/grader.py`, and `evals/files` inputs, are scanned too. Generated Tier 3 output in `evals/results` is skipped. Files under a folder named `tests`, `examples`, `fixtures`, or `mocks`, or the singular form, are skipped at any depth, including the Harbor task graders in `evals/harbor/<case>/tests/`, so a secret there is not reported. To keep an intentional test token, add a `gitleaks:allow` comment on its line or list the finding in a `.gitleaksignore` file at the skill root.
* **Hygiene** — dead relative Markdown links, banned or unpinned dependency declarations, and static discovery of conventional Python test filenames.

Default Tier 1 does not import or execute target-controlled Python code. The `test_discovery` result is filename evidence only: it counts regular, in-tree, non-symlink `test_*.py` and `*_test.py` candidates without parsing or importing them. It reports `execution_performed=false` and `coverage_measured=false`; test success and code coverage must be established separately in a trusted project environment or an explicit sandbox.

A missing required scanner, or a timed-out, crashed, or malformed Bandit, Semgrep, Gitleaks, or SkillSpector run, is treated as `INCOMPLETE`: terminal and saved reports are non-green and `validate` exits non-zero. When Semgrep runs but cannot fully parse some files (for example a shell or JavaScript file with syntax Semgrep's parser rejects), the check stays `INCOMPLETE` because Semgrep skips the code it cannot parse, so findings there are missed. The report names up to 20 of those files, relative to the skill and with the line of the first error, keeps Semgrep's findings for the other files, and lists them in the `semgrep_unparsed_files` field of the JSON report, up to 1,000 files; `semgrep_unparsed_files_omitted` counts any beyond that. Fix the syntax error, or check whether it comes from Semgrep's parser or its rule configuration. See [Scanner setup](#scanner-setup) for installation.

#### unicode — smuggling detection

Detects invisible Unicode characters associated with ASCII smuggling, hidden-data encoding, and trojan-source attacks.

| Category                                    | Range                                                    | Default severity                           |
| ------------------------------------------- | -------------------------------------------------------- | ------------------------------------------ |
| Unicode Tags (ASCII smuggling)              | `U+E0000..U+E007F`                                       | CRITICAL; decodes the hidden ASCII payload |
| BiDi overrides (trojan source)              | `U+202A..U+202E`, `U+2066..U+2069`, `U+200E/F`, `U+061C` | HIGH                                       |
| Zero-width characters                       | `U+200B/C/D`, `U+034F`, `U+180E`, `U+2060`, `U+FEFF`     | MEDIUM or LOW                              |
| Invisible operators and deprecated controls | `U+2061..U+2064`, `U+206A..U+206F`                       | MEDIUM                                     |
| Variation selectors                         | `U+FE00..U+FE0F`, `U+E0100..U+E01EF`                     | LOW                                        |

Severity increases with suspicious run length. A BOM (`U+FEFF`) at byte zero is downgraded to INFO, and binary files are skipped using MIME and null-byte detection.

#### quality — weighted scoring

`quality-check`, and the `quality` check inside `validate`, produce a 0–100 composite score across four weighted categories.

| Category        | Weight | What it evaluates                                                                          |
| --------------- | ------ | ------------------------------------------------------------------------------------------ |
| Correctness     | 35%    | Frontmatter, naming, examples, paths, and skill-type-specific structure                    |
| Discoverability | 25%    | Description quality, trigger wording, purpose, scope, and naming                           |
| Reliability     | 25%    | Error handling, prerequisites, limitations, troubleshooting, and implementation safeguards |
| Efficiency      | 15%    | Estimated token budget, body length, repetition, instruction clarity, and reference use    |

The 5,000-token budget for `SKILL.md` is a recommendation, not a hard limit. Token counts are estimates (characters divided by 4), not tokenizer output: each model family counts differently, and dense files with tables, identifiers or code spans usually have fewer than 4 characters per token, so the estimate runs low. `metrics.estimated_tokens` holds the estimate and `metrics.estimate_method` names how it was made; `metrics.total_tokens` is kept as the same value for existing consumers. A larger `SKILL.md` gets a MEDIUM `quality_efficiency` finding and a 15-point Efficiency deduction. The finding does not fail the check by itself, but the deduction can still take the score below `--min-score`. The limit and the deduction are set in the quality config file described below.

Grades are A for scores of at least 90, B for at least 80, C for at least 70, D for at least 60, and F below 60. The default `--min-score` is 70. Passing requires the score to reach that minimum and no HIGH or CRITICAL quality findings under the active policy.

Each category starts at 100 and loses a fixed number of points per triggered finding. The [Metrics Reference](/skills/skillevaluator/metrics-reference#tier-1-quality-score) gives the composite formula, the deduction behind every finding, and how this 0–100 score differs from the similarly named Tier 3 dimensions.

The minimum score is an independent gate. Policy severity overrides cannot
waive it, and a collection average cannot hide an individual skill below
the minimum. A below-minimum result includes a diagnostic such as
`Quality score 67.8/100 is below the required 70/100`, retained in the JSON
result's `gate_failures` list. Improve the skill or explicitly choose the
required threshold with `--min-score`.

Correctness reports XML-style tags in any frontmatter value as a HIGH finding (potential prompt injection). A string `argument-hint`, or an unquoted hint such as `[<file>]` that YAML reads as a list of strings, may use command-line placeholders such as `<file> [--strict]`, `<plan|apply|verify>`, or `<codec e.g. av1|h264>`. A placeholder has at most four words, and `|` separates choices that are each one word or a quoted literal, after an optional label such as `stage:` or `codec e.g.`. Longer free text, shell command syntax (`;`, `&`, a backtick, `$(`, `${`, or a pipeline such as `<curl x | sh>`), closing tags, attributes, self-closing tags, and role or control tags such as `<system>`, `<tool_call>`, or `<script>` are still reported there. A role word (`system`, `developer`, `assistant`, `user`, `tool`, `context`, `instructions`, `prompt`, `override`, `rules`, or `policy`) followed by `prompt`, `message`, `call`, `use`, `result`, or `instruction` counts as a role tag (`<system_prompt>`, `<toolCall>`, `<user_message>`), while other names such as `<tool_name>`, `<user_id>`, `<prompt_file>`, or `<toolchain>` are ordinary placeholders. A hint with a redirect, that is a `>` that closes no group (`<cmd > file>`, `<cmd>file>`, `<cmd 2>file>`, `<curl x> /tmp/y>`), or a group with anything after a nested placeholder other than its own closing `>` (`<<curl x>/tmp/<y>>`, `<<x><y>>`) or anything other than a plain name before it (`<curl;sh<x>>`, `<$(curl x|sh)<x>>`), is not masked at all, so it is still reported. A placeholder nested directly after a plain name, such as `<R<NNN>>`, is masked. For a list value, each item and the space-joined hint are checked separately. An arrow or comparison between groups, such as `<old> -> <new>`, `<n> >= 1`, or `<count> (>0)`, is not a redirect. Every other field keeps the strict check.

The checker detects five skill shapes and applies relevant checks: **script-based** (Python, shell or Node `.mjs`, `.cjs` and `.js` files under `scripts/` or `tools/`), **lib-based** (a Python module containing `__init__.py`), **resource-based** (assets, templates, design-system content, or resources), **guide-only** (documentation without executable or resource content), and **hybrid** (scripts plus library or resource content).

A script-based or hybrid skill is expected to document its scripts in an `## Available Scripts` table and to show how to run them. The check looks for a `run_script` call. A skill whose scripts are all Node scripts is not asked for it, because that call names a runner such a skill does not use; a skill that also bundles Python or shell scripts keeps the check. Node scripts are not read by the error-handling reliability check, which looks at Python scripts only. A Node-script skill that was typed guide-only before can therefore score differently after upgrading.

Quality signals use boundary-aware lexical matching for unambiguous terms and CommonMark parsing for structural link evidence. Links inside code, images, or comments do not count, and external URLs do not satisfy local-reference checks. Deterministic Tier 1 checks do not infer author intent from ambiguous prose such as MCP negation, temporal language, README guidance, or exclusivity claims. Use `rubric-eval` for semantic judgments about the documentation and Tier 3 for observed agent behavior.

> **Tip**
>
> If a quality score changes unexpectedly, inspect the finding message rather than adding keywords. The checker rewards usable instructions and evidence in their relevant context, not vocabulary alone.

**Quality config file.** Pass `--quality-config PATH` to `quality-check`, `tier1`, or `validate` to adjust the rules below. The file has five optional sections: `expected_headers` and `error_handling` for Reliability, `efficiency` for the token budget, `discoverability` for the description length limits, and `correctness` for reserved words in the skill name. Anything you leave out keeps its packaged value from `quality-default.yaml`, and any list you set replaces the packaged list rather than merging with it.

**Expected headings.** Reliability deducts points when `SKILL.md` lacks an expected heading. The packaged `quality-default.yaml` expects `## Prerequisites` (or `## Requirements`), `## Limitations`, and `## Troubleshooting` (or `## Common Issues` or `## FAQ`), each worth 5 points. Omit `expected_headers` to keep that list. A list you set replaces it, so copy any default entries you want to keep. Set `expected_headers: []` to turn heading checks off.

**`quality.yaml`**

```yaml title="quality.yaml"
expected_headers:
  - heading: "## Prerequisites"
    alternatives:
      - "## Requirements"
    points: 5
    message: "No prerequisites/requirements documented"
    suggestion: "Document dependencies, API keys, or setup needed"
  - heading: "## Security"
    points: 8
    message: "No security notes documented"
    suggestion: "Add '## Security' describing permissions and data access"
  - heading: "## Examples"
```

**`Score with a custom heading list`**

```bash title="Score with a custom heading list"
skillevaluator quality-check ./my-skill --quality-config quality.yaml
skillevaluator tier1 ./my-skill --quality-config quality.yaml
skillevaluator validate ./my-skill --quality-config quality.yaml
```

| Field          | Required         | Meaning                                                                                                                                                                                                   |
| -------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `heading`      | Yes              | A single ATX Markdown heading, such as `## Security`                                                                                                                                                      |
| `alternatives` | No               | Other headings that satisfy the same entry                                                                                                                                                                |
| `points`       | No (default `5`) | Non-negative number deducted from Reliability when no accepted heading is found. With `points: 0` the missing heading is still reported as a zero-point (advisory) finding, but the score does not change |
| `message`      | No               | Finding message reported when the heading is missing. Defaults to `Missing expected heading: '## Examples'`, listing any alternatives                                                                     |
| `suggestion`   | No               | Fix suggestion attached to the finding. Defaults to `Add a '## Examples' section`, or `Add one of these sections: ...` when there are alternatives                                                        |

The packaged defaults set `message` and `suggestion` explicitly, so their findings keep the original wording.

An entry is satisfied when the heading or any alternative appears anywhere in `SKILL.md` as text, so `## Security` also matches `### Security Notes` or a heading inside a code block.

**Error-handling checks.** Reliability also runs three error-handling rules, set in the `error_handling` section of the same file. An omitted rule, or an omitted key within a rule, keeps its packaged value. Set `points: 0` to turn a rule off; it then adds no finding. This differs from `expected_headers`, where an entry with `points: 0` still reports a zero-point finding that does not change the score.

**`quality.yaml`**

```yaml title="quality.yaml"
error_handling:
  mentioned:
    terms: [error, failure, rollback, retry]
  scripts:
    points: 0
```

| Rule        | Deducts when                                                                                                                                                                    | Packaged points | List field                                                                                                                                                                                    |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mentioned` | `SKILL.md` contains none of the `terms`                                                                                                                                         | `10`            | `terms`: whole words, any case. `error` matches `Error` but not `errorless`. A term with spaces, such as `exit code`, also matches when its words are split by a line break or several spaces |
| `scripts`   | A `.py` file directly in `scripts/` or `tools/` has neither a `try:`/`except` pair nor any of the `markers`                                                                     | `5`             | `markers`: case-sensitive text found anywhere in the file, such as `raise` or `if not `                                                                                                       |
| `lib`       | A module in the skill's Python package, for skills with no `.py` or `.sh` files directly in `scripts/` or `tools/`, has neither a `try:`/`except` pair nor any of the `markers` | `5`             | `markers`: as for `scripts`                                                                                                                                                                   |

Each rule also takes `message` and `suggestion`. The `scripts` and `lib` findings append up to three file names to the message, as in `Scripts may lack error handling: run.py`. The `try:`/`except` pair always counts as error handling and is not configurable. See the packaged `quality-default.yaml` for every default term and marker. The `quality_config` digest in the report covers only the rules your file sets, not the packaged values it leaves out, so the digest changes only when your file's rules change.

**Token budget.** Efficiency deducts points when `SKILL.md` is larger than a token limit, set in the `efficiency` section under `token_budget`. As with `error_handling`, an omitted key keeps its packaged value and `points: 0` turns the rule off, so it adds no finding. This file raises the limit and changes nothing else:

**`quality.yaml`**

```yaml title="quality.yaml"
efficiency:
  token_budget:
    max_tokens: 8000
```

| Field        | Packaged value             | Meaning                                                                                                                                      |
| ------------ | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `max_tokens` | `5000`                     | A positive whole number. The rule deducts when the estimated size of `SKILL.md` is above it; a file estimated at exactly `max_tokens` passes |
| `points`     | `15`                       | Non-negative number deducted from Efficiency. `0` turns the rule off                                                                         |
| `message`    | `Large skill`              | Start of the finding message                                                                                                                 |
| `suggestion` | See `quality-default.yaml` | Fix suggestion attached to the finding, including the agentskills.io guidance on size                                                        |

The size is an estimate, not a tokenizer count: the number of characters in `SKILL.md`, frontmatter included, divided by 4 and rounded down. Files in `references/` and `scripts/` are not counted. The finding appends the estimate and the limit to `message`, as in `Large skill: ~6200 estimated tokens at 4 characters per token, recommended max <5000; a real tokenizer may count more`. Keep a custom `message` short: report summaries show only the first 160 characters of a finding message, so a long one pushes the estimate and the limit out of view. Reports show the limit in force as `recommended_max_tokens` in the quality metrics, whether or not the rule is turned off.

**Description length.** Discoverability deducts points when the frontmatter `description` is too short or too long, set in the `discoverability` section under `description_too_short` and `description_too_long`. An omitted key keeps its packaged value and `points: 0` turns a rule off, so it adds no finding. This file raises the minimum and the maximum and changes nothing else:

**`quality.yaml`**

```yaml title="quality.yaml"
discoverability:
  description_too_short:
    min_chars: 40
  description_too_long:
    max_chars: 300
```

| `description_too_short` field | Packaged value                                         | Meaning                                                                                                                                               |
| ----------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `min_chars`                   | `20`                                                   | A positive whole number. The rule deducts when the description has fewer characters than this; a description of exactly `min_chars` characters passes |
| `points`                      | `20`                                                   | Non-negative number deducted from Discoverability. `0` turns the rule off                                                                             |
| `message`                     | `Description too short (recommend 50-150 chars)`       | Start of the finding message                                                                                                                          |
| `suggestion`                  | `Add more context: what tasks does this skill handle?` | Fix suggestion attached to the finding                                                                                                                |

| `description_too_long` field | Packaged value                                         | Meaning                                                                                                                                              |
| ---------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `max_chars`                  | `200`                                                  | A positive whole number. The rule deducts when the description has more characters than this; a description of exactly `max_chars` characters passes |
| `points`                     | `5`                                                    | Non-negative number deducted from Discoverability. `0` turns the rule off                                                                            |
| `message`                    | `Description very long (recommend 50-150 chars)`       | Start of the finding message                                                                                                                         |
| `suggestion`                 | `Keep descriptions concise for progressive disclosure` | Fix suggestion attached to the finding                                                                                                               |

The length is counted in characters after leading and trailing whitespace is removed. A skill without a `description` key, or with an empty one (`description: ""`), is not reported by these rules. A `description:` key left without a value is not empty: YAML reads it as null, which is counted as the 4-character text `None` and reported as too short. Each finding appends the length and the limit to `message`, as in `Description too short (recommend 50-150 chars): 12 chars, minimum 20` and `Description very long (recommend 50-150 chars): 251 chars, maximum 200`. The two rules are checked independently, so a minimum set above the maximum makes both fire on a description between them. The other Discoverability rules are not configurable yet.

**Reserved words in the name.** Correctness deducts points when the frontmatter `name` contains a reserved word, set in the `correctness` section under `reserved_name`. An omitted key keeps its packaged value. A `terms` list you set replaces the packaged list, so copy `anthropic` and `claude` if you want to keep them:

**`quality.yaml`**

```yaml title="quality.yaml"
correctness:
  reserved_name:
    terms: [anthropic, claude, acme]
```

| Field        | Packaged value                          | Meaning                                                                                                                                                                                                                                                      |
| ------------ | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `terms`      | `[anthropic, claude]`                   | Whole words, any case. `claude` matches `claude-helper` but not `claudette-helper`. A term may contain only letters, digits and single hyphens between them, as in `claude-code`, because a skill name allows nothing else. Spaces around a term are ignored |
| `points`     | `15`                                    | Non-negative number deducted from Correctness. `0` turns the rule off                                                                                                                                                                                        |
| `message`    | `Name contains reserved word`           | Start of the finding message                                                                                                                                                                                                                                 |
| `suggestion` | `Remove reserved words from skill name` | Fix suggestion attached to the finding                                                                                                                                                                                                                       |

The finding appends the matched terms to `message`, as in `Name contains reserved word: claude`. Several matches still deduct `points` once. This is an error-level rule: its finding is HIGH and fails the quality check whatever the score, so `points: 0` removes that failure as well as the deduction. The schema check has its own fixed list (`anthropic`, `claude`) that this file does not change, so with the default checks a name containing either word is still rejected there. The other Correctness rules are not configurable yet.

The file is checked before any validation runs. A file that is missing, unreadable, not UTF-8, invalid YAML, has an unknown key (such as `point` instead of `points`), lists the same heading twice, has a negative or non-numeric `points`, a `max_tokens`, `min_chars` or `max_chars` that is not a whole number from 1 to 1,000,000,000, an empty `terms` or `markers` list, a repeated entry in one of those lists, or a reserved-name term that could never match a skill name (such as `Claude Code` or `claude_code`) is a usage error: the command exits with code 2 and names the file and the problem. `validate` checks the file even when `--checks` leaves out `quality`.

**`Example error`**

```text title="Example error"
Error: Invalid value for '--quality-config': quality.yaml: unknown key(s) in expected_headers[0]: point (allowed: alternatives, heading, message, points, suggestion)
```

#### rubric-eval — LLM-as-judge (requires a provider)

`rubric-eval` sends bounded skill documentation and selected supplementary content to your configured LLM provider. It scores nine criteria:

| # | Criterion                                                | Importance |
| - | -------------------------------------------------------- | ---------- |
| 1 | Description clarity and when to use the skill            | High       |
| 2 | Instruction clarity and actionable steps                 | High       |
| 3 | Example quality and query variation                      | High       |
| 4 | Documentation completeness                               | Medium     |
| 5 | Scope definition                                         | Medium     |
| 6 | Professional tone and formatting                         | Low        |
| 7 | Trigger simulation against positive and negative queries | High       |
| 8 | End-to-end workflow completeness                         | High       |
| 9 | Actionable error handling                                | Medium     |

Each criterion is scored from 0 to 10. The local evaluator, not the model's boolean, treats 7 or higher as passing. The overall score is an importance-weighted mean, and the rubric passes only when that score reaches `--min-score` and every criterion passes.

If the judge's reply is not valid JSON, misses, repeats or invents a criterion, or gives a criterion a score outside 0 to 10 or a pass flag that is not a boolean, `rubric-eval` asks the judge once more and names the problem, keeping the first 500 characters of it. A second such reply fails the rubric with a HIGH `llm_invalid_response` finding, or `llm_unavailable` when it is still not JSON. A rate limit, server error or other transient provider error is retried by the LLM client as set by `SKILL_EVAL_LLM_MAX_RETRIES` (see [LLM retries and judge time budget](/skills/skillevaluator/environment-variables#llm-retries-and-judge-time-budget)). A request that still fails after those retries, a refusal or a truncated reply is not asked again.

The JSON report records `judge_attempts`, the number of times `rubric-eval` asked the judge (1, or 2 after a rejected reply), in the skill's `rubric_eval` success detail or `llm_invalid_response` / `llm_unavailable` finding and, for a single skill, also in the `rubric_eval` object. The client's own retries are not counted. When the first request fails, for example because no provider is configured, no reply came back and `judge_attempts` is left out.

The JSON report names the provider and model that gave each score as `llm_provider` and `llm_model_used` in the skill's `rubric_eval` success detail and, for a single skill, also in the `rubric_eval` object. For a Claude 5.5 or later model on the native Claude API, the model is the one the reply names; otherwise it is the configured model, unless a server-side `fallback` block in the reply names another model. A Claude 5.5 or later model's refusal can be answered by the server-side fallback model, and the API can send a repeat of that request straight to it; the report then names the fallback model, not the configured model.

**`Rubric evaluation`**

```bash title="Rubric evaluation"
skillevaluator rubric-eval ./my-skill --min-score 70
```

#### lint — script linting (advisory)

`lint-scripts`, and the `lint` check inside `validate`, parse Python scripts under `scripts/` and report advisory findings. Node scripts count as scripts for skill-type detection and the script-based checks but are not linted; the result says how many Node scripts were not linted. These findings do not make the Tier 1 gate fail.

| Check                 | Severity | Detects                                                         |
| --------------------- | -------- | --------------------------------------------------------------- |
| `flat_script`         | MEDIUM   | No function definitions                                         |
| `deep_nesting`        | MEDIUM   | Control-flow nesting deeper than 6                              |
| `magic_numbers`       | LOW      | Raw numeric constants outside the safe constant set             |
| `missing_shebang`     | LOW      | No leading shebang (`#!`), or a shebang preceded by a UTF-8 BOM |
| `no_input_validation` | LOW      | No `argparse`, `click`, `typer`, or explicit raise pattern      |

#### version — semantic version check

The default version check validates an optional `metadata.version` as a three-part numeric value in `major.minor.patch` form:

**`SKILL.md frontmatter`**

```yaml title="SKILL.md frontmatter"
metadata:
  version: "1.2.3"
```

When `--previous-version` or `SKILLEVALUATOR_PREVIOUS_VERSION` is set, the current version must be present and numerically greater than the previous value. Without a previous-version bound, the version remains optional and the skill can rely on commit history. A previous-version bound is valid only when checking one skill; catalog validation must check each skill separately with its own bound.

**`Version check against a previous release`**

```bash title="Version check against a previous release"
skillevaluator validate ./my-skill --previous-version 1.2.2
```

#### dependency — CVE audit (opt-in)

Scans `requirements*.txt` for known Python package vulnerabilities. For
`pyproject.toml`, reads static PEP 621 dependencies and every optional group,
then audits their exact version pins without installing the target or
executing its build scripts. This covers direct declarations only;
transitive dependencies require a resolved requirements file. Markers are
evaluated against the scanner's Python/platform and optional-group context;
nonmatching declarations are reported as skipped. Version ranges, URL/path
dependencies, dynamic or Poetry metadata, malformed declarations, and
missing or failed scanner evidence leave the audit `INCOMPLETE`. Supply
pinned `requirements.txt` for unresolved declarations. A requirements line
that ends in a backslash continues on the next line unless it is a comment
line, as in `pip install -r`, so a pin after a comment line that ends in a
backslash is audited. `pip-audit` is the
primary scanner; Safety supplies secondary coverage for requirements files
when installed separately. The audit may use network-backed vulnerability databases.
JSON and HTML export data retain `dependency_coverage` per target, including
requested and actually audited pins. Unreadable declarations have unknown
declaration counts and an empty audited inventory, not a completed scan.

Each advisory is a `DEPENDENCY` finding whose check name is the advisory ID
(a numeric Safety ID reads `SAFETY-<id>`). It points at the requirements
file and the line that pins the package, or at `pyproject.toml`, so it
appears in JSON `findings`, the severity counts, and SARIF results.
`validate` on a folder of skills validates each skill as its own job, so
each skill's report lists its advisories. When one dependency check covers
several skills, as with `validate --type rules` or `--type workflows` on a
folder, or `DependencySecurityValidator().validate()` on a folder in
Python, a skill whose audit passes is summarized as one passed check, so
its medium or low advisories are not listed; validate that skill on its own
to see them. The finding metadata records:

* `package_name` and `resolved_version`: the package and version the
  scanner checked.
* `declared_constraint` and `declaration`: the exact pin as the target
  spells it, such as `PyYAML==5.3`, and where it is declared (`line 3`, or
  `optional group docs, declaration 2` in `pyproject.toml`). The raw line,
  with its markers and options, is never copied.
* `resolution_status` and `dependency_relationship`: `resolved` and
  `direct` when the advisory matches an audited exact pin. Only exact pins
  are audited, with `--no-deps`, so the advisory applies to the version the
  target declares. Safety reads the raw requirements file, so it can
  report a version that no audited pin declares, such as a pin skipped
  because its marker does not match the scanner platform. That advisory is
  `unbound`: `declared_constraint`, `declaration`, and
  `dependency_relationship` are `null`, and the finding has no line.
* `dependency_role`: `runtime`, `dev`, `test`, `docs`, or `build`, from a
  whole word of the requirements file name or the optional group name.
  Words are split at `-`, `_`, and `.`. `dev`, `devel`, `develop`,
  `development`, `devtools`, `lint`, and `linting` mean `dev`; `test`,
  `tests`, `testing`, `testutils`, `pytest`, and `unittest` mean `test`;
  `doc`, `docs`, and `documentation` mean `docs`; `build` means `build`.
  So `requirements-dev.txt` is `dev`, while `requirements-latest.txt` and
  `requirements-docker.txt` are `runtime`.
* `advisory_id`, `advisory_source` (`pip-audit` or `Safety`),
  `affected_range` (Safety only; `null` for pip-audit), `fixed_versions`,
  and `aliases`.

A `high` or `critical` advisory fails the run. A Python caller that builds
`DependencySecurityValidator(fail_on_medium=True)` also fails on a `medium`
pip-audit advisory; those findings record `fail_on_medium: true` in their
metadata, so a policy overlay keeps that rule. Safety's severity is its
level word or, when Safety runs with an API key, its CVSS object: the
CVSS v3 level, then the v3 score, then the v2 score and level. A score
that is not a number from 0 to 10 is skipped, and a Safety advisory with
none of these is `medium`. When pip-audit already reported
the same advisory ID or CVE for that package version, Safety's copy is
left out, so one vulnerability is one finding. A critical advisory does not
stop a folder run at that skill, as a critical finding of other checks
does without `--continue-on-failure`: every skill is audited.

A policy overlay can change one advisory with `DEPENDENCY.<advisory-id>`,
such as `DEPENDENCY.PYSEC-2020-176: low`, or every advisory with
`DEPENDENCY.*`. When one dependency check covers several skills, the
overlay sees only the advisories that check lists, so it cannot raise a
medium or low advisory of a skill whose audit passed. The warnings that say
why an audit is `INCOMPLETE`, such as a line without an exact pin, still
appear in the CLI and Markdown reports next to the advisories, and after a
policy overlay changes an advisory's severity.

**`Opt-in dependency audit`**

```bash title="Opt-in dependency audit"
skillevaluator validate ./my-skill --checks dependency
```

## Author attribution and contact emails

When `metadata.author` carries a contact, write it as `Name <email>` in
`SKILL.md`. The PII check accepts that recognized author line. The same address
in another field, body text, or a reference document is a separate occurrence
and can still fail the check. A team or organization name without an email is
also valid attribution under the `default` profile.

Use a name, team handle, or contact-directory link for additional contact
information. If a real address must remain, record a reviewed, time-limited
[PII waiver](#pii-waivers) for that occurrence. An organization-owned email
domain does not automatically exempt every address in the repository.

## PII waivers

A PII waiver accepts one reviewed value at one place, for example a public support address that a skill must document. Pass the waiver file with `--pii-waivers PATH` on `validate`, `tier1`, or `pii-scan`. Waivers are never discovered automatically inside the scanned skill, so a skill cannot waive its own findings. Keep it outside the skill: a waiver file inside the scanned tree ships with the skill and is PII-scanned like any other file, owners and reasons included.

**`pii-waivers.yaml`**

```yaml title="pii-waivers.yaml"
version: 1
waivers:
  - id: PII-001
    category: emails
    file: skills/support/SKILL.md
    line: 12
    evidence_sha256: 3f1d…  # 64 hex characters, copied from the PII finding
    reason: Public support address of the documented service
    owner: docs-team
    created: 2026-10-01
    expires: 2027-01-01
```

**`Apply reviewed waivers`**

```bash title="Apply reviewed waivers"
skillevaluator validate ./skills/support --pii-waivers ./pii-waivers.yaml
```

* `category` is the PII check name (`emails`, `personal_paths`, `phone_numbers`, and so on) and `line` is the line of the value. Credential checks (`database_credentials`, `hardcoded_secrets`, `jwt_tokens`, `github_tokens`, `aws_identifiers`, `webhook_urls`, `private_keys`) cannot be waived: remove or rotate the credential instead.
* `file` is relative to the directory that holds the waiver file (the directory of the link, when the waiver file is a symlink), so one file covers single-skill, folder, and catalog runs. Absolute paths, `..`, and control characters are rejected. A path through a symlinked directory, such as `.claude/skills/support/SKILL.md` when `.claude/skills/support` links to `skills/support`, names the same file as the path without the link.
* `evidence_sha256` comes from the `evidence_sha256` metadata of the PII finding in the JSON report. It is a SHA-256 digest of the check name and the case-folded value, so the waiver file never contains the value. A digest is an identifier, not encryption: a short value such as an SSN can be recovered from it by trying every candidate. Findings report a digest only for a value the report shows in full, so credential findings, and any other finding whose value reports mask because it contains a credential, carry none.
* `reason` needs at least 8 characters, `owner` names the reviewer or team, and the waiver applies through `expires`. Dates are UTC calendar dates; `created` may be at most one day after the UTC date, so a reviewer east of UTC can enter their local date, and `expires` may not be before `created` or more than 366 days after it, so every waiver comes up for review at least yearly.

A waiver applies only while the check, file, line, and value all match. When the value changes or moves, the finding is reported again, and other values on the same line are reported as usual. The outcome of each waiver the run judges is in the reports under the `PII_WAIVER` category:

| Check                     | Severity | Meaning                                                       |
| ------------------------- | -------- | ------------------------------------------------------------- |
| `applied-pii-waiver`      | INFO     | The waiver suppressed the finding.                            |
| `stale-pii-waiver`        | LOW      | No current finding matches the waiver; update or remove it.   |
| `expired-pii-waiver`      | MEDIUM   | The waiver is past `expires`; the finding is reported again.  |
| `invalid-pii-waiver-file` | HIGH     | The file could not be applied, so no waiver from it was used. |

A run judges only the waivers whose file lies in a skill or folder it scanned; a waiver for a deleted file there is stale. A run on one skill, and each skill of a catalog run, reports the waivers for that skill's files and leaves the other waivers in the file unjudged. Waiver findings name the file the way the PII findings of the same report do, for example `[support] SKILL.md` in a folder run; the waiver's own `file` value is in the `waiver_file` metadata. They do not repeat the waiver's `evidence_sha256`, because the run may now mask that value.

The command line rejects a malformed waiver file before the run starts, with exit code 2 and the problem named: unknown or missing fields, duplicate ids or keys, unknown or credential check names, and malformed paths, digests, or dates. Waivers apply only to the `pii` check; findings from other checks for the same value, such as Gitleaks secrets, are unaffected.

## Scanner setup

Bandit installs with SkillEvaluator. Tier 1 provisions SkillSpector and, on
Linux and macOS, Gitleaks when they are missing; the first run needs network
access. Semgrep is optional unless `SKILLEVALUATOR_REQUIRE_SEMGREP=1` is set.
See the [installation guide](/skills/skillevaluator/installation-options#system-tools)
for scanner setup, offline preparation, and the optional dependency-audit extra.

Semgrep uses a policy packaged with the installed distribution, with metrics and version checks disabled; Tier 1 never fetches a registry policy at runtime. The other public Tier 1 configuration also ships inside the package under `skillevaluator/config/`: `pii_patterns.yaml`, `unicode_smuggle_patterns.yaml`, `license_config.yaml`, and `profiles/default.yaml`.

Bundled Python scanners resolve next to the SkillEvaluator interpreter before `PATH`; external Semgrep and SkillSpector installations resolve from `PATH`. Intentional replacements require an auditable absolute executable path in `SKILLEVALUATOR_BANDIT_PATH`, `SKILLEVALUATOR_SEMGREP_PATH`, or `SKILLEVALUATOR_SKILLSPECTOR_PATH`.

> **Warning**
>
> Scanner path overrides fail closed: a relative, missing, or non-executable override path fails the run rather than silently falling back. Required scanners that are missing or produce malformed output leave the run `INCOMPLETE` and non-green.

## Audited SkillSpector baselines

Record reviewed SkillSpector exceptions in a SkillSpector YAML or JSON baseline and pass it with `--skillspector-baseline` to `validate`, `tier1`, or `security-scan`. SkillEvaluator never discovers a baseline on its own, including a `.skillspector-baseline.yaml` shipped inside the skill: applying exceptions is an operator or CI decision. A scan that reports suppressed findings without a requested baseline stays `INCOMPLETE`.

**`Apply a reviewed baseline`**

```bash title="Apply a reviewed baseline"
skillevaluator validate ./my-skill \
  --skillspector-baseline ./trusted/skillspector-baseline.yaml
```

Keep the baseline outside the skill under review, for example in a checkout of the base branch or a separate policy repository. A baseline kept inside the skill is only as trusted as the review of that file: whoever edits the skill can edit its exceptions, SkillSpector leaves the selected baseline file out of its own scan so extra text in it is never inspected, and a version 2 rule needs only a non-empty `reason`. In a pull-request pipeline, the checked-out skill is the PR's version, so pass a baseline from the target branch instead.

Prefer narrow rules. Every field a rule sets must match the finding. `id` (the rule ID), `path` (the file), and `message` (the finding text) are all case-insensitive globs, and `reason` is required.

**`config/skillspector-baseline.yaml`**

```yaml title="config/skillspector-baseline.yaml"
version: 2
rules:
  - id: "PE5"
    path: "SKILL.md"
    message: "*--privilege[d]*"
    reason: "Reviewed: the diagnostics container needs host device access."
```

The bracket class `[d]` matches the letter `d`, so the rule still matches `--privileged`, but the baseline file does not contain the literal text that triggers the finding. SkillEvaluator rejects a rule unless its `id` has at least two literal letters or digits in a row, or its `path` or `message` has at least four. Wildcards and multi-character classes break a run. This rejects `id: "*"` and also `message: "*e*"`, which matches almost every finding because SkillSpector matches the glob case-insensitively against the finding text. Before the scan it also rejects a version 2 rule or fingerprint without a `reason`, `rules` or `fingerprints` that are not lists or contain entries that are not mappings, a file that is not valid YAML or JSON (including an impossible date such as `2026-06-31`), and a file larger than 2 MiB or whose aliases and merge keys expand beyond SkillSpector's own limits. An empty or comment-only file is an empty baseline, as in SkillSpector. SkillSpector checks the rest of the schema. If it rejects the baseline, it exits with code 2, the security check is `INCOMPLETE`, and the error suggests checking the file with `skillspector scan <skill> --baseline <file>`.

To waive individual findings exactly, generate fingerprints with SkillSpector. The command records every current finding, so review each entry before you commit the file:

**`Generate exact fingerprints`**

```bash title="Generate exact fingerprints"
skillspector baseline ./my-skill --no-llm \
  --reason "Reviewed: <rationale>" \
  -o ./my-skill/config/skillspector-baseline.yaml
```

Fingerprints bind to the scanned file content and to the SkillSpector version that generated them. After a source edit, the affected findings are reported again. After a scanner upgrade, SkillSpector ignores every fingerprint, and SkillEvaluator reports `fingerprint_version_mismatch` with both versions: as an error when blocking findings remain, otherwise as a warning. Rules keep working across scanner versions.

Fingerprints only match static-only runs: `tier1 --no-llm`, or `validate` and `security-scan` without `--llm`. When the LLM security stage runs (`tier1` by default when a provider is configured, or `--llm`), SkillSpector's meta-analysis rewrites each finding's message, explanation, remediation, confidence, or tags before the baseline is applied. Those fields are part of the fingerprint, so fingerprints from `skillspector baseline --no-llm` no longer match, and the LLM stage reports the findings again. SkillEvaluator then reports `llm_stage_unmatched` with both stages' suppressed counts: as an error when blocking findings remain, otherwise as a warning. For runs with LLM analysis, use narrow `id`, `path`, and `message` rules instead; they still match because SkillSpector also checks `message` against the matched text.

A baseline stored inside the skill is excluded from the scan, including when SkillEvaluator scans a staged copy without `evals/` outputs. For a folder of skills, the same baseline applies to each skill, and rule paths are relative to each skill.

For a single-skill target, when SkillSpector is installed, JSON reports record the outcome under `results[].skillspector_baseline`:

* `status`: `applied`, `no_matches`, `fingerprint_version_mismatch`, `llm_stage_unmatched`, or `not_applied`.
* `file` and `digest`: the baseline's file name and a `sha256:` digest of its bytes. The local path is not recorded.
* The baseline and runtime scanner versions, and the rule and fingerprint counts.
* `suppressed_findings` from the static stage and, when the LLM stage completed, `llm_stage_suppressed_findings`; and the active and active blocking finding counts. All are taken from SkillSpector's reports before any `--llm-verify` downgrade or policy severity override.
* `suppressed_by_severity` and `suppressed_rule_ids`: the static stage's suppressed findings by severity and their rule IDs, so CI can refuse a baseline that hides anything blocking. The Markdown and HTML line shows the severity counts.
* `message`: the outcome or the reason the baseline was rejected.

Markdown and HTML reports show the same outcome as a `skillspector_baseline` line in the Security result, with the file name and digest, and a scan whose only findings were suppressed reads "No active security findings; N finding(s) suppressed by the requested baseline" rather than a clean result.

`security-scan` and `tier1 security-scan` on a folder of skills write one merged report without the JSON field. When a skill in the folder passes, its outcome is appended to that skill's "All checks passed" line in the Markdown and HTML reports. `validate` on a folder validates each skill as its own job, so each skill's JSON report records the field. The field keeps the `fingerprint_version_mismatch` and `llm_stage_unmatched` messages even when a later `--llm-verify` downgrade or policy severity override rebuilds the error and warning lists. See the [SkillSpector suppression reference](https://github.com/NVIDIA/SkillSpector/blob/v2.13.3/docs/SUPPRESSION.md) for the full baseline schema.

## Exit behavior and scoring

| Exit code | Meaning                                                                                                                                                                 |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`       | Success — every gate passed                                                                                                                                             |
| `1`       | Validation failed — a check failed or required scanner evidence is incomplete                                                                                           |
| `2`       | Usage error — an unknown flag, a value outside an option's type or choices, a target path that does not exist, or an invalid `--quality-config` or `--pii-waivers` file |

Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it runs unless `--no-block-on-dedup` is set. Tier 3 is advisory unless `--block-on-agent-eval` is set. The `quality` check requires the composite score to reach `--min-score` (default `70`) as well as no HIGH or CRITICAL quality findings under the active policy. Severity overrides cannot waive the minimum. Reports record the effective gate membership for every tier so saved evidence and process status agree.

`--fail-fast` stops after the first failed check. `-c`/`--continue-on-failure` overrides it, records the full pipeline, and keeps scanning a collection past a CRITICAL finding. Wiring these exit codes into a merge gate — including a complete GitHub Actions recipe — is covered in [Gate Your CI](/skills/skillevaluator/ci-integration).

## Troubleshooting

#### Why is SkillSpector analysis incomplete?

Open the security check's execution error in the CLI, HTML, or Markdown
report. It now shows up to three file/line locations with their scanner
phase and reason, for example:

```text
SKILL.md:14 [reference_resolution; reference_missing]
```

This means the scanner could not resolve or inspect a reference; it is not
itself proof of a vulnerability. Review the named source line and whether
the reference should be bundled. If the scanner mistook a version or API
name for a file, report that detection issue to SkillSpector.

For more detail, each JSON result's `skillspector_completeness.records`
retains the scan stage, up to 20 ledger exceptions, analyzer names, and up
to eight limitations. Omission counts make truncation explicit. Folder
reports retain the skill name with each record. These diagnostics never
turn an incomplete scan into a pass.

#### Gitleaks is not installed

On Linux and macOS, SkillEvaluator downloads Gitleaks on first use. If that
download cannot complete, follow [scanner setup](/skills/skillevaluator/installation-options#system-tools)
to prepare it for your environment.

#### A scanner is missing

Run `skillevaluator doctor --tier 1` to identify the scanner and its resolved
path or version, then follow [scanner setup](/skills/skillevaluator/installation-options#system-tools).
Required scanner failures remain `INCOMPLETE` until the scanner returns
valid evidence. Missing optional Semgrep is reported as skipped unless
`SKILLEVALUATOR_REQUIRE_SEMGREP=1` is set.

#### SkillSpector reports incomplete analysis for a valid helper script

SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a `static_parse_limit`, `obfuscated_instruction_text`, or `runtime_limit` ledger exception. The security check is then `INCOMPLETE`, often with a High **Incomplete referenced artifact analysis (AE1)** finding at each line that names the helper. The finding's suggestion names the helper and the reason, such as `static_parse_limit (static_patterns_tool_misuse)`. Check the resolved release with `skillevaluator doctor --tier 1` and follow [scanner setup](/skills/skillevaluator/installation-options#system-tools) to update an older installation. If the exception remains after the upgrade, investigate it like any other incomplete scan.

#### SkillSpector could not fully model the hooks

With SkillSpector v2.12.0 and later, a skill, folder or hooks-only plugin
with an ordinary command or HTTP hook fails the security check with the
Critical `skillspector_hook_payload_unverified`, for example
`SkillSpector could not fully model the hooks in hooks/hooks.json (analyzer bundled_execution_surface could not verify what they do with hook input), so this check cannot pass while these hooks are present.`
The check also records the gate failure
`hooks/hooks.json: SkillSpector could not model these hooks; this check cannot pass while they are present`.
Earlier SkillEvaluator releases reported the same scan as `INCOMPLETE`
("Security Scan didn't run") with
`hooks/hooks.json [static; opaque_content; analyzers=bundled_execution_surface]`.
The scan itself finished, and the other findings and the `--llm` stage are
reported as for any complete scan. In a folder, this Critical does not stop
the scan at that skill, so the other skills are still scanned without
`--continue-on-failure`.

SkillSpector cannot tell what these hooks do with their input, so review
the manifest and every file its hooks run. SkillEvaluator has no way to
record that review: removing the hooks is the only way to pass the check.
A `--policy` severity override can lower the finding's severity, but the
gate failure keeps the check failed, also when another gap records the
scan as incomplete. `--llm-verify` does not send the
finding to the verifier, so it keeps its severity and gets no verdict. The
finding's metadata records the manifest's SHA-256 (`manifest_sha256`) and
SkillSpector's row for it.

SkillSpector records the same row for a manifest it read only in part.
With v2.13.4, a manifest with more records than SkillSpector reads or with
a handler it could not parse, and a settings file with permission rules or
other settings beside its hooks, leave the scan `INCOMPLETE` with the row
instead of the Critical. With v2.13.3, only a manifest whose BH1 counts
2,048 or more declarations does. See the `security` check under
[The checks](#the-checks).

SkillSpector v2.13.4 also reads a hook's `"timeout"` key as a command
wrapper it cannot resolve and records
`hooks/hooks.json [static; static_parse_limit; analyzers=static_patterns_tool_misuse]`.
That row is an ordinary analysis gap, so with v2.13.4 the SkillSpector
scan of such a manifest is also recorded as incomplete (`incomplete_scans`
in the JSON report, and publication eligibility **Incomplete**), with an
error that names only that row. The check itself is still reported as
`FAIL` because of the Critical and its gate failure. v2.13.3 does not
record the row.

#### SkillSpector times out

Each SkillSpector run (the static stage, and the LLM stage with `--llm`)
is stopped after 1,200 seconds by default, and the security scan is reported
`INCOMPLETE` with `skillspector timed out after 1200 seconds`. SkillSpector
also ends a scan at its own deadline (1,080 seconds by default in v2.13.3) and reports partial
analysis, which is `INCOMPLETE` as well. Slow or local LLM providers can
need more time. Raise both limits, keeping SkillEvaluator's watchdog higher:

```bash
export SKILLSPECTOR_MAX_WORKFLOW_SECONDS=1800
export SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS=2400
skillevaluator validate ./my-skill --tiers 1 --llm
```

Each stage gets the full watchdog, so a scan that stalls in both stages can
take up to twice the value before it is reported. See
[Environment Variables](/skills/skillevaluator/environment-variables#validation-tier-1).

#### The SkillSpector LLM provider keeps failing

With `--llm`, SkillSpector v2.12.0 and later retries a failing provider call
a bounded number of times. When the provider still fails, for example
because of rate limits or an outage, SkillSpector records the
`llm_connection_retries_exhausted` reason and ends the LLM stage as failed.
If that is the only cause in a report from SkillSpector v2.12.0 or later,
the security error reads
`skillspector-llm did not complete: the LLM provider kept failing after SkillSpector's bounded retries`
and says that this is a retriable provider failure, not a security
finding. Retry the full scan when the provider is available.

The gate is unchanged: the security scan is `INCOMPLETE`, the command
exits `1`, and findings from the static stage are kept, so a High or
Critical static finding still makes the result `FAILED`.

SkillEvaluator shows this message only when every failure that
SkillSpector recorded is a provider-retry failure on an LLM analyzer.
Reports that mix it with another cause, are truncated or malformed, or come
from an older SkillSpector keep the generic error, such as
`skillspector-llm failed with unexpected exit code 2`. SkillSpector v2.9.1
to v2.11.2 used the same reason only for a connection error left after the
provider SDK's own retries, so SkillEvaluator does not read it from those
releases. A provider error that SkillSpector records under a different
reason also keeps the generic error.

#### An LLM-backed stage fails immediately

Configure a supported provider or select a keyless Tier 1 run with
`skillevaluator tier1 ./my-skill --no-llm` or
`skillevaluator validate ./my-skill --tiers 1 --no-llm` (without
`--llm-verify`). Direct Tier 1 automatically enables LLM stages when a
provider is configured; `--no-llm` explicitly disables them. Top-level
`validate --no-dedup` skips Tier 2 but still runs Tier 3 by default.

#### No scannable files are found

Confirm that the skill contains supported text or code extensions such as `.py`, `.sh`, `.yaml`, `.yml`, `.json`, `.md`, or `.txt`.

#### An executable override is rejected

Use an existing, executable absolute path for the relevant `SKILLEVALUATOR_*_PATH` variable.

#### SkillSpector stops at a runtime limit

When SkillSpector reports partial analysis, the security scan is
`INCOMPLETE`. The error quotes up to three of SkillSpector's completeness
exceptions as `path [phase; reason_code; analyzers=...]`. When there are more,
it adds how many files each reason affected, for example
`and 463 more (incomplete files by reason: runtime_limit 462, obfuscated_instruction_text 4)`.

For `runtime_limit`, the error also says what to do, with one piece of advice
for each kind of row the report contains:

* **Scan time budget.** Rows that name regular analyzers, or no analyzer,
  mean SkillSpector ran out of time for the bundle: at its own whole-scan
  deadline (1,080 seconds by default in v2.13.3) or at a shorter stage
  budget such as bundle discovery. A skill that bundles a large corpus, such
  as vendored documents, generated figures or renderer scripts, hits the
  same limit on every rerun. Keep large reference material outside the
  skill and point to it. A plain rerun helps only when a few files are
  affected on a busy machine. If SkillEvaluator's own SkillSpector timeout
  stops the run before SkillSpector's deadline, the error is
  `skillspector timed out after N seconds` instead, with no list of files.
* **Sub-analysis budget.** Analyzers named `<analyzer>_runtime_limit`, such as
  `static_patterns_supply_chain_bytecode_runtime_limit`, stopped at a time
  budget of their own, so bundle size does not matter. Releases before
  v2.13.0 charge the SC8 bytecode walk for time spent waiting behind other
  analyzers, so it can stop even on a skill with no bytecode. Rerun the scan,
  and update SkillSpector if the limit repeats.

## Next steps

#### [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication)

Check whether your skill repeats itself or overlaps the rest of your collection.

#### [Gate Your CI](/skills/skillevaluator/ci-integration)

Turn the Tier 1 exit code into a merge gate with a copy-paste workflow.

#### [CLI Reference](/skills/skillevaluator/cli-reference)

Every flag and default for validate and the standalone Tier 1 commands.

#### [Environment Variables](/skills/skillevaluator/environment-variables)

Profile, scanner override, and identity variables in one table.