Tier 1 Reference
For your first check, follow Tier 1: Validation. This reference keeps the detailed rules and advanced workflows in one place.
Run Tier 1 directly to check a skill’s structure, security, quality, and dependencies. First, install SkillEvaluator, complete scanner setup for full validation, then run:
Static checks need no model key. With the llm extra
and a configured provider, this command also
runs rubric judging, LLM security analysis, and finding verification. Without a
provider, those LLM stages are explicitly reported as skipped. Use --no-llm
for static checks only, or --llm to require a provider instead of allowing a
keyless run. See scanner setup
for first-run downloads and optional tools; PyPI 0.9.0 installed with pip also
needs uv for automatic SkillSpector setup.
The two-variable provider setup is enough to configure the
LLM stages for NVIDIA Build or OpenAI. Calling skillevaluator tier1 without a
path shows help; expert subcommands remain available below.
Tier 1 in automatic CI
The all-tier CI setup keeps Tier 1
as a required gate. GitHub calls the hardened legacy scanner workflow at the
same evaluator commit; GitLab runs the same digest-pinned scanner image.
Static CI checks receive no model key. Tier 2 and Tier 3 execute independently
and retain their own native evidence, so one tier’s failure does not erase the
other reports. Changes to onboarding configuration trigger a full keyless
Tier 1 scan so the setup review exercises actual skills. With scan-mode: auto,
the provider-backed LLM Tier 1 pass, Tier 2 and Tier 3 run on a pull request only
when it changes a skill. Eligible pushes and manual runs on the default branch
run them for the whole collection.
Existing released Tier 1-only consumers keep their current workflow and bundled image pins. See GitHub Actions or GitLab CI/CD to choose either path.
What Tier 1 checks
The direct tier1 PATH workflow runs Tier 1 only, as does
tier1 validate PATH. The focused checks below are also available as
skillevaluator tier1 <command>. Top-level validate is the combined pipeline.
validate is the umbrella command: all three tiers run by default for skills, with autopilot creating one starter evaluation case if no source exists. Tier 1 checks always gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs and gates by default, and Tier 3 is advisory by default. Use --tiers 1 for static checks only, --no-block-on-dedup to keep Tier 2 advisory, or --block-on-agent-eval to promote Tier 3 into the gate.
Run the combined or focused validation commands
Unlike direct tier1 PATH, validate includes Tier 2 and Tier 3 by default
for skills. Its Tier 1 LLM checks still run only when requested.
Run the default suite
By default you get a compact pipeline view in the terminal — a per-check ticker per tier — and the run writes HTML and JSON report files unless you pass -r explicitly, which is then honored exactly (including -r cli for terminal-only output). Add --verbose for the full per-check detail stream instead of the compact view. The full pipeline needs the provider and runtime setup; use --tiers 1 for static checks only. The exit code is 0 when every gate passes and 1 when one fails.
Run static checks only
Select only Tier 1 to run the static checks without model calls:
Tier 1 can download required scanners on first use. For an offline environment,
prepare them using Installation: System tools.
Run skillevaluator doctor --tier 1 to check the scanners’ paths and supported versions.
Narrow to specific checks
--checks (alias --tier1-checks) takes a comma-separated subset when you want fast, targeted feedback:
Only the selected Tier 1 checks run; the others are skipped entirely.
Add LLM analysis (optional)
With a provider configured, --llm (alias --tier1-llm; disable with --no-llm or --no-tier1-llm) adds LLM security analysis on top of the static findings, and --llm-verify runs a second pass that reviews each static finding in context:
Confirmed findings keep their original severity. High-confidence false positives stay visible but are downgraded to INFO and carry verification metadata explaining the review. Each finding that gets a verdict names the provider and model that gave it as llm_provider and llm_model_used in its JSON metadata. For a Claude 5.5 or later model on the native Claude API, the model is the one the reply names; otherwise it is the configured model, unless a server-side fallback block in the reply names another model. Verdicts kept from a reply cut off at the output-token limit name the model that wrote that reply. Static scanning always runs first — if a requested LLM stage is unavailable or returns unusable evidence, the run does not silently turn green. security-scan and pii-scan accept --llm-verify too.
HTML, Markdown, and terminal reports show the recorded verification verdict, confidence, and reasoning beside each finding, together with its reported severity.
Direct tier1 PATH and tier1 validate PATH print the terminal report and
write JSON and HTML reports by default. Use -r cli for terminal-only output.
Reports for any of these runs are selected with -r/--report (cli, json, html, markdown, sarif) and written to -o/--output-dir (default reports/). The junit format is included in 1.0.1; see Installation for availability. It is not in 0.9.0. File naming, BENCHMARK.md, and the JSON contract are covered in Reports & Results.
Content types
validate auto-detects what you point it at. Use --type when the path is ambiguous:
Plugins are validated against their public contract; the quality, lint, and version checks are skill-only and are skipped for plugins, rules, and workflows.
When --type auto finds none of these signals, for example in an ordinary source repository, the target’s type is unknown. There is no manifest to check, so Tier 1:
- Records
schema,version, andqualityas not applicable instead of failing them withNo skills found. The terminal summary and the HTML report count them as N/A. The JSON and Markdown reports list them as skipped, with a skip reason that starts withnot applicable. No report counts them as passes. - Still runs the content scans (
security,pii,license,code-integrity,unicode,lint, anddependencywhen selected), so a real secret or card number still fails the run. The security scan keeps its unrestricted scope. - Passes when those scans pass. A misspelled
--checksname still fails. Tier 2 deduplication still runs on the target unless you pass--no-dedupor--tiers 1, and can still fail the run. - Adds a root
tier1_applicabilityobject to the JSON report, whichever checks--checksand--skipselect:applicability(not_applicable),reason_code(unsupported_content_type),detected_content_type(unknown), andsupported_content_types. - Is never eligible for publication, because a not-applicable check is not evidence.
Auto-detection also reads the directory names in the target’s absolute path. A repository whose path includes a directory named skills, team-skills, team-rules, workflows, or team-workflows, such as a CI checkout at /home/runner/work/skills/skills, is detected as that type, not as unknown, and Tier 1 validates it as that type, as before. --type forces a type but cannot select unknown.
A target folder that itself holds a manifest entry in any letter case (SKILL.md, workflow-rules.mdc, agent_plugin.yaml, agent_plugin.yml, or .claude-plugin/plugin.json), such as Skill.md, or a manifest symlink whose target is missing, is a broken artifact rather than a folder without one, so its manifest checks still run and fail. A .claude-plugin folder without a plugin.json, such as a marketplace’s, does not count. This applies only to the target’s own entries: a mis-cased manifest below the target is not discovered; validate that folder directly. Pass --type skill (or another type) to validate any target as that type; a missing SKILL.md then fails as before. A folder that holds skills below it, such as .claude/skills/<name>/SKILL.md or .agents/skills/<name>/SKILL.md, is still validated as a collection of skills.
For collections, pass a folder containing skills under skills/ or team-skills/. Folder-level validation discovers each live SKILL.md below the target while excluding configured names such as evals, results, versions, .git, .venv, node_modules, and __pycache__.
Validation profiles and policies
Two policies are bundled. Both report malformed or unsafe author values and blocked licenses (CRITICAL); they differ in which governance metadata a skill must carry.
The Agent Skills baseline requires only name and description, so the
default policy treats author attribution and a stable ID as recommendations.
Neither policy restricts the author email domain. A policy controls validation
rules, not repository visibility or report hosting. The old external name and
--external flag remain compatibility aliases for default; there is no
separate bundled internal policy.
Precedence, highest to lowest: --policy overlay, --profile flag, SKILLEVALUATOR_PROFILE environment variable, then the bundled default policy.
Write a custom policy overlay
A custom policy overlays the --profile rules (default unless you choose
another). It can narrow the accepted author domain, set the metadata.id
requirement, and change severities by CATEGORY.check_name or CATEGORY.*:
An author_email_regex needs an email address in that domain, so under it a
name without an email fails SCHEMA.author_format. An overlay that omits an
identity key keeps the base profile’s value. An unrecognized
metadata_id_requirement value is an error, not a silent fallback.
The author checks are SCHEMA.author_missing, SCHEMA.author_without_email,
and SCHEMA.author_format. The ID checks are SCHEMA.metadata_id_missing,
SCHEMA.metadata_id_format, and SCHEMA.metadata_id_duplicate. Their default
severities follow metadata_id_requirement: a missing ID is LOW under
recommend and HIGH under require; an invalid or duplicate ID is MEDIUM
unless the requirement is require.
Severity overrides change how findings gate a run; they do not change the
quality score or waive its --min-score requirement. Even an override of
QUALITY.*: info leaves a below-minimum quality score failing. Nor do they
complete an INCOMPLETE scan, whose report keeps the error that says why.
Permit reviewed ownership contacts without allowing other email addresses
Contacts in the single-line metadata.pic and metadata.email fields of
the skill’s own SKILL.md (the one at the root of the scanned skill) use the
separate PII.ownership_emails check. Both that check and ordinary
PII.emails findings remain HIGH and blocking by default.
If your repository policy permits those ownership contacts, pass an explicit policy file. The finding stays visible in the report as informational.
Supported values are one address or Name <address> in valid YAML. The same
address in body text, a people directory, another metadata field, or another
file (including a SKILL.md nested in a subdirectory) still uses the ordinary
email policy. Ambiguous YAML, aliases, and block scalars also retain that
policy. Credential checks remain blocking.
Keep this policy under repository-owner review; the scanner does not discover
or trust a policy from the skill automatically. Existing PII.emails overrides
apply to ordinary emails; use PII.ownership_emails explicitly for these contacts.
This option is available on tier1, tier1 validate, and validate. The
released reusable CI templates do not yet expose a custom-policy input.
The active profile name is stamped into the HTML report header and JSON report, and printed in the run banner under --verbose, so reviewers can see which gate was applied.
The checks
For validate, the default set is schema, version, security, pii, license, code-integrity, unicode, quality, and lint. The dependency check requires explicit selection with --checks (alias --tier1-checks) for validate; direct tier1 PATH includes it by default. Accepted check-name aliases include code/code-risk for code-integrity, scripts/script-lint for lint, dependencies/deps/dependency-audit for dependency, and licence/license-check for license.
schema — frontmatter and repository governance
Validates SKILL.md frontmatter and repository placement:
- YAML frontmatter must parse and satisfy the skill schema.
- Frontmatter keys, including keys nested under
metadata, must be strings. An unquoted key that YAML reads as a number, boolean, null, or date (123,yes,null,2026-01-01) is a HIGH finding that names the key in YAML form, such asField 'metadata.true': Keys should be strings; quote the key ("yes":) to keep it. Rules, workflows, and plugin manifests report non-string keys as schema errors too. nameanddescriptionare required. Names are 1–64 characters; descriptions are 1–1024 characters.- Names use kebab-case, start with a letter, and cannot have trailing or consecutive hyphens. The directory name must match the frontmatter
name. The reserved wordsanthropicandclaudemust not appear in a skill name. alwaysApplyandglobsare forbidden skill fields. Optional fields includelicense,compatibility,metadata, andallowed-tools.- The standard hierarchy is
skills/<skill-name>/orteam-skills/<team>/<skill-name>/. A nonstandard standalone location is reported as an advisory finding. - Recognized skill-root subdirectories are
agents/,references/,scripts/,assets/,evals/,tests/,tools/, andconfig/; anything else gets a LOW advisory finding. Extend the set per repository withSKILLEVALUATOR_SCHEMA_ALLOWED_DIRS(comma-separated names are added to the defaults, never replace them). SKILL.mdshould stay at or below 500 lines; larger files receive an advisory finding.- Harness portability: patterns the spec allows but common agent harnesses (Claude Code, Codex, OpenCode, OpenClaw, Hermes, Copilot CLI) load differently are reported; override any of them with
SCHEMA.<check_name>in the validation policy.- A duplicate frontmatter key is HIGH (
duplicate_key; several harnesses drop the skill), and a UTF-8 BOM is MEDIUM (manifest_bom). - A description within 1,024 characters but over 1,024 UTF-16 units is MEDIUM (
description_utf16_length), and no “Use when …” trigger in the first 60 characters is LOW (description_trigger). - A symlinked
SKILL.md(manifest_symlink) or aSKILL.mdnested inside the skill (nested_manifest) is MEDIUM. manifest_sizeis LOW above 32 KiB, MEDIUM above 50 KiB and HIGH above 256,000 bytes.- Fields outside the spec are LOW (
unknown_field).
- A duplicate frontmatter key is HIGH (
metadata.authoris optional under thedefaultprofile. UseName <email@host>to include a contact address, or a person, team, or organization name such asNVIDIA cuOpt TeamorJane Doe (@jdoe). A value that is malformed or unsafe is a HIGHauthor_formatfinding: a bare email or an email outsideName <email@host>, stray angle brackets, control, private-use, or bidirectional formatting characters (joiners such as ZWNJ and ZWJ, which Persian, Arabic and Devanagari names need, are accepted), no letters or digits, or more than 256 characters. A bare address counts even when its domain has no dot (root@buildhost), while a social handle such asJane Doe (@jdoe)does not. Theregistryprofile requires theName <email@host>form.metadata.idis an optional stable skill ID. When present it must be an exact 1–64 character lowercase ASCII slug (^[a-z0-9]+(-[a-z0-9]+)*$, no surrounding whitespace); it may differ fromname. A collection run reports an ID that more than one discovered skill declares;validateon a catalog also lists these incatalog-summary.jsonundermetadata_id_conflicts. Underregistrya shared ID blocks: the run exits nonzero, and each sharing skill’s entry is"passed": falsewith the conflict repeated undercatalog_findingsand counted in itsseverity_counts. A catalog validates each skill on its own, so that skill’s own report was written before the catalog-wide check ran and keeps its single-skill verdict, which the entry carries alongside asreport_verdictbesidereport_scope: "single_skill". Uniqueness is checked only across the skills validated together, never against a registry, and a single-skill run does not claim it. The ID does not change local catalog identity, which stays keyed by content type and relative path.metadata.domainaccepts one category string or a list of them, such asdomain: [backend, ai-ml]. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGHField 'metadata.domain'finding. The 128-character limit also applies to a single string.
To fix common findings: rename the directory to match name, remove forbidden fields, and move the skill under skills/.
harness — harness compatibility
This check is included in 1.0.1; see Installation for availability. It is not included in 0.9.0.
The harness check reports, for each agent harness, whether the skill loads, is degraded (loads, but something is cut, hidden or ignored) or is not_loaded, with the rule IDs behind each verdict and the harness version and date the profile was verified against. Supported harnesses: claude-code, codex, copilot-cli, hermes, openclaw, opencode. All of this is static: no agent or model is run. See Harness Check for a walkthrough with example output.
validate runs it by default for all six and lists its findings in the HTML, JSON and SARIF reports; JSON reports also get a harness_compatibility block. Use --checks harness to run it alone and --harness codex,claude-code to narrow the harnesses.
The profiles ship with the package and come from repeated runs of each harness in which the request sent to the model was captured. The check doesn’t change pass/fail: its findings are MEDIUM (not_loaded), LOW (degraded) or INFO (advisory), so it never fails a run unless a policy raises those severities. Install-location rules apply only when the skill sits under a harness root such as .claude/skills or .agents/skills. Use --harness-profiles DIR to try newer profile YAMLs without a release.
validate also shows a “Harness compatibility” panel after Tier 1, ! harness in the checks line when a harness drops or degrades the skill, a harness line in the Tier 1 summary (kept out of the advisories count) and a note in the result box. skillevaluator harness-check <skill> runs only this check and shows the same panel, updated as each harness is checked. It accepts a folder of skills, --harness, -v and --json PATH, always exits 0, and points to validate --tiers 1 when Tier 1 schema also fails. By default the panel is compact: one line per harness with its most serious finding (not loaded, then degraded, then note; ties go to the rule listed first in the profile) labelled with its outcome (not found, rejected, listing shortened, delivery truncated and so on), then +N more and the number of notes. -v (also on validate) lists every finding with its fix and each harness’s verified version.
Frontmatter rules: duplicate keys (Codex, Copilot CLI, OpenClaw and OpenCode drop the skill), tab indentation (Codex, Copilot CLI and OpenClaw), a UTF-8 BOM (Codex), no front matter (Copilot CLI), and long custom frontmatter that never reaches the model.
Description rules: missing or empty descriptions, Claude Code and Codex listing cuts, Copilot CLI’s 1,024 UTF-16 unit limit, quotes escaped in the Copilot CLI and OpenClaw lists, and Hermes’s 60-character listing. -v also shows the description each harness puts in the model’s skill list.
Location rules: SKILL.md casing, nested manifests, symlinked manifests and resources, OpenClaw symlink escapes, Codex scan depth, and files Hermes leaves out of its file list (outside references/, templates/, assets/ and scripts/).
The check measures the skill together with the skills installed next to it. A single skill under a harness root such as .claude/skills, or in a workspace skills/ folder, is counted with everything in that folder (symlinked skill folders included). Skills in different folders are separate installations, and a folder of skills is its own catalog. That drives the list-budget rules (Claude Code and Codex ~8,000 description chars, Copilot 32 full descriptions, OpenClaw 150 skills or 18,000 chars) and a duplicate-name rule per harness that says which copy wins. A catalog line in the panel names the folder. Copilot CLI and OpenCode also get an advisory note on when new or edited skills are picked up.
Delivery rules: Claude Code’s body cap, OpenCode’s 51,200-byte / 2,000-line reads and lines over 2,000 characters, OpenClaw’s whole-or-nothing read against its result caps, Hermes’s 100,000-character skill_view budget, and Codex’s ~10K-token cut when the model reads SKILL.md through the shell.
Placeholder rules: $ARGUMENTS and ${CLAUDE_SKILL_DIR} are substituted only by Claude Code, and price-like $N text is replaced with arguments by Claude Code and OpenCode.
Invocation rules: disable-model-invocation (ignored by Codex, Hermes and OpenCode), Codex’s agents/openai.yaml implicit-invocation policy (Codex only), allowed-tools, Claude Code-only fields, and Hermes project trust.
Gate rules: Hermes platforms and OpenClaw metadata.openclaw.requires hide the skill on hosts that don’t match.
harness-check on a plugin folder also shows a plugin panel: which plugin formats each harness accepts (.claude-plugin, .codex-plugin, Copilot, Agent Plugins v1, Hermes plugin.yaml), and an empty plugin.
Plugin component rules cover skill namespacing, Codex’s plugin-skill cap and Agent Plugins hooks, Copilot CLI MCP server names, disabled plugins, and components a harness registers but doesn’t deliver.
harness-check takes the model setup some limits depend on: --context-tokens, --custom-provider, --explicit-invocation and --headless.
Each rule records a confidence level and, where behaviour changed between releases, the versions it applies to (shown with -v). --harness-version codex=0.170.0 adds a note when you run a release other than the one a profile was verified on.
security — vulnerability scanning
security-scan, and the security check inside validate, run SkillSpector. Static scanning covers common skill attack patterns such as prompt injection and data exfiltration; each finding carries the matched pattern, severity, and location.
The LLM pass requires a configured provider. It enriches the static result but does not replace static findings or make missing scanner evidence pass.
SkillSpector’s text pattern analyzers cannot read images, so it reports the analysis of a skill that references a PNG or JPEG file as partial. SkillEvaluator accepts that gap with a warning instead of INCOMPLETE only when the report shows no other gap apart from the missing references described next, and no fatal scope exclusion, and each referenced image is a regular, single-link (not hard-linked), non-executable file whose bytes have the chunk structure of a PNG or the marker structure of a JPEG and that is not reached through a symlink. The gap is accepted for at most 256 images and 128 MiB of image data in total. An image is never accepted inside a hooks, scripts, tools, agents, commands, prompts, rules, team-rules or workflows directory at any depth, or under a top-level config, skills, .mcp, .agents, .claude, .codex, .continue, .cursor, .gemini or .windsurf directory (names compared without regard to case). The structure check corroborates the file type; it does not decode the image or prove that it renders. PNG IDAT compressed data and ancillary chunks, and JPEG entropy-coded data and APPn or COM segments, are not inspected and can still carry other bytes. The image content itself is not assessed. Any other partial reason, except the missing references described next and the hooks described below, still leaves the scan INCOMPLETE. SkillSpector also reports a High AE1 (incomplete referenced artifact analysis) finding for such an image; from v2.12.0 it omits that finding when the skill renders a PNG with Markdown image syntax, such as . An AE1 finding fails the check like any other High finding.
SkillSpector also reports the analysis as partial when the skill’s instructions name a file that it did not scan, such as a file the skill writes at run time (output/notes.md) or a source file elsewhere in the repository. SkillEvaluator accepts that gap with a warning instead of INCOMPLETE when each such reference meets all of these conditions:
- It is a nonfatal
reference_resolutionrow for one line of a file that the report lists once as a component and that is a regular, single-link file of at most 1,000,000 bytes inside the skill, not reached through a symlink. - The report’s complete
referenceslist records amissingreference on that line and quotes part of the line. Everymissingentry must have an accepted ledger row at the same source path and line; anambiguousentry anywhere, a truncation marker, or an unknown completeness field leaves the scanINCOMPLETE. - The line, at most 10,000 characters long, names nothing in the skill that SkillSpector did not inspect: a symlink, a file the report does not list, a directory that holds no listed file, such as
versions/,node_modules/orevals/results/, which SkillEvaluator leaves out of the scanned copy, anything inside such a directory, or a directory that holds any of these, even beside listed files. A.gitdirectory counts by its own name, by its Git hooks other than*.sampletemplates, and by every other entry in it that is not part of Git’s own layout, such as a script placed there, with everything inside that entry. Git’s own files, such asHEAD,config,index,packed-refsorORIG_HEAD, and what itsobjects,refs,logsandinfodirectories hold, such as branch names, do not count. Names are compared without regard to case or Unicode normalization, in the line as written and as a Markdown renderer or a reader of file paths may read it: with backslash escapes of ASCII punctuation (such asnode\_modules) removed, HTML character references (such asv) resolved and percent-escapes decoded, up to three of these steps in any order. A skill with more than 50,000 files and directories, or nested more than 64 directories deep, is not checked, and its scan staysINCOMPLETE. The work of checking a report’s lines is also bounded: when it would take longer, such as for many long lines beside thousands of files left out of the scan, the scan staysINCOMPLETE. - A word in the line that a shell, cmd.exe, PowerShell or Windows would expand or unquote is read as a pattern, and none of those names may match it. In the pattern, quotes and escapes are removed, a group such as
(v)stands for what it holds, also right after]as in[v](e)rsions, each expansion, set of alternatives, extended glob or numeric range may stand for any text, and each bracket expression for any one character. The word is read as follows:- The way POSIX shells read globs, brace and parameter expansions,
$(...)and quotes (v*/*.sh,vers?ons,v[e]rsions,v{e,x}rsions,$DIR/run.sh,ver""sions). - The way zsh reads groups (
(v)ersions,(v|x)ersions), numeric ranges (run<1-9>.sh), theEXTENDED_GLOBoperators^,#and~(ver^x,verr#sions, andv*~x, read asv*), wherex#as a whole part, such asC#, matches only names made ofx, and globbing flags such as(#a2)or(#c1), which make the whole word stand for any text ((#a2)virsions/x). - The way bash and ksh read extended globs (
@(v)ersions,!(x)). - The way cmd.exe reads the
^escape (ver^sions), variables (%DIR%,!DIR!) and batch parameters (%1,%~dp0). - The way PowerShell reads backtick escapes.
- With
\as a directory separator (v*\i*.sh). - With a Windows short (8.3) name or a Samba mangled name, a path part with text after a
~such asNODE_M~1/BOOTST~1.SH, standing for any name that holds the part’s first character, such as any name with annforNODE_M~1. So3~5is read as written, as3, and as any name with a3. - A span between backticks is read as Markdown code, as written and with its backticks removed, not as a command, so a plural written as
`topic`sis read astopics. A leading Markdown*list marker is ignored. - A
$before a space or the end of the line is read as a literal$, as a shell reads it, not as an expansion, so a shell prompt$ python run.pyadds no pattern, andsh *a$matches only names that end ina$. - A bold or italic label outside code, such as
**Note:**,*Tip:*or**Step 1:**, is read as its text, and each of its words, with its*, as a pattern that is not split at:,=or,. So as a pattern,**Note:**matches only names that holdnote:, and**Step 1:**only names that end instepor start with1:. A standalone shell wildcard elsewhere, such asbash *, still counts, and so does a label inside code. - On a line with a space or shell delimiter between two of the same quote or after a backslash, as written or decoded as above, such as
setup' 'helper.sh,"hi there"orsetup\ h*.sh, a$before a space and a label are read like any other word, so they still stand for any text: a shell may join words there, which the check does not. A PowerShell backtick before a space does not count, as a backtick also ends Markdown code, so$ sh setup` h*.ps1does not namesetup helper.ps1. A PowerShell backtick before a space does not count here: in**Note:** run setup` h*.ps1the label is read as its text, andsetup` h*.ps1does not match a left-outsetup helper.ps1, which PowerShell would run. - Prose with this syntax is read the same way: a price such as
$5, a list such as{a,b},%APPDATA%or a lone^stands for any text, so a line with one staysINCOMPLETEwhenever anything is left out of the scan, andapprox~10staysINCOMPLETEwhen a name left out holds ana.
- The way POSIX shells read globs, brace and parameter expansions,
- The check reads names that are written in the reported line, in the spellings above. It does not read other lines and does not run commands, so it does not detect a name on another line, such as
DIR=node_modulesset on one line and used assh $DIR/run.shon the next, or a name that a command prints or builds, such asprintfoutput in backticks, a list piped intoxargs sh,find -exec, or a PowerShell string joined with+. Nor does it join words at a quoted or escaped space as a shell does, sosetup' 'helper.shorsetup\ h*.shdoes not namesetup helper.sh. It is meant to keep an instruction that names a left-out fileINCOMPLETE, not to catch a skill written to hide one. Without this check, such a skill passes whenever SkillSpector reports no missing reference in it; with it, the skill also passes when every line SkillSpector reports is another missing reference, such asoutput/notes.md. SkillSpector’s own analyzers, not this check, assess the instructions it scanned. - When the target is a single file, such as
skillevaluator security-scan path/to/SKILL.md, SkillSpector scans that file alone, so every other file and directory beside it counts as not inspected. A line in it that names a file beside it, such asscripts/install.sh, keeps the scanINCOMPLETE. - Tier 3 output in its default location,
<skill>/evals/results, is left out of the scanned copy too, so it counts as not inspected. The rest ofevals/, such asevals/evals.jsonand input fixtures inevals/files, is scanned. After a Tier 3 run, a line that namesresults, a file the run leaves there, such asreport.htmlorsummary.json, or, when the run used--harbor-keep-jobs, any bundled file (the kept Harbor tasks hold a copy of the skill) keeps the scanINCOMPLETE, although the same line passed before the run. Writing Tier 3 results outside the skill, with--results-dirorSKILLEVALUATOR_RESULTS_DIR, keeps a run from changing the result. - The scan covers the whole target. When the target is a folder or a plugin that holds skills, such as
skillevaluator security-scan path/to/skills/,skillevaluator validate path/to/skills/(which checks each skill as its own job) orvalidateon a plugin, each skill in it is scanned on its own, and a line may name a file elsewhere in the target that the skill’s scan did not inspect, so those skills’ missing references keep their scansINCOMPLETE, also when a skill in a folder is a plugin of its own.
Every nonfatal scope exclusion must name an actual uninspected entry under the target. Accepted missing-reference warnings remain visible after the static and LLM stages deduplicate their findings, and after --llm-verify downgrades a finding. Any other gap must be a referenced image that SkillEvaluator accepts as described above, and the scan then carries both warnings. Without such images, the report must also show no partially or entirely uninspected file, no limitation and no fatal scope exclusion, and every analyzer must have completed or be not applicable or disabled. In either case the report must count as many findings after filtering as before it, and a missing reference without an accepted ledger row, an ambiguous reference or a truncation marker leaves the scan INCOMPLETE even when a referenced image is its only listed gap. The warning names the file and line of up to three of the references; it does not quote the referenced path. Files not in the scan, including any outside the scanned path, are not inspected.
SkillSpector v2.12.0 to v2.13.3 report these references as reference_missing, and v2.10 and v2.11 as reference_unresolved, which SkillEvaluator accepts under the same conditions. A v2.13.3 report whose only gaps are such references is also accepted from the report alone as a missing-reference caveat: an INFO finding that keeps SkillSpector’s partial status and is recorded as skillspector_reference_caveats in the JSON report. It replaces the warning above, and whenever the scan has a known target, as through validate, tier1 or security-scan, it is accepted only when the conditions above hold too. A reference that matches several bundled files (reference_unresolved from v2.12.0, ambiguous in the references list before it) still leaves the scan INCOMPLETE.
From v2.13.4, SkillSpector no longer reports such a scan as partial. When it inspected every bundled file and the only gaps are references to files the skill does not bundle, it reports the status complete_with_caveats and lists each reference as a caveat. SkillEvaluator accepts these caveats under the same conditions and shows the missing-reference warning above, and the findings of an accepted report are checked as in any other complete scan. A caveat on a path that is not a bundled file, or one that fails a condition above, such as a line that names a file left out of the scan, leaves the scan INCOMPLETE; as for a partial report, the report’s findings are still checked and reported. Unlike a v2.13.3 caveat, these caveats are never accepted from the report alone, so they leave a scan without a known target INCOMPLETE, and an accepted report carries no skillspector_reference_caveats entry. SkillSpector gives this status only when every gap is a nonfatal reference_missing row, so a report with no caveat, or with a caveat that is not a nonfatal reference_missing row in phase reference_resolution, contradicts itself: SkillEvaluator does not trust it, and the scan is INCOMPLETE without its findings. The same holds for complete_with_caveats from a release before v2.13.4.
SkillSpector v2.12.0 and later cannot model what an ordinary command or HTTP hook, such as one that runs echo hi or a bundled script, does with its hook input. The exception is a hook that SkillSpector recognizes as sending hook content to a remote host, which it reports as BH2; a manifest whose command and HTTP hooks are all of that kind counts as modeled. For other hooks it discovers in manifests such as hooks/hooks.json and .claude/settings.json, SkillSpector reports a BH1 finding whose evidence says the payload is unmodeled, records a nonfatal partial opaque_content row from bundled_execution_surface on the manifest, and reports the analysis as partial although the scan finished. This rule applies only to manifests SkillSpector actually reports with that evidence and ledger row; it does not establish coverage for inline plugin-manifest hooks or other hook locations the scanner does not report. The scan’s findings cannot show that such hooks are safe: a hook script that sends its input over a socket, nc or /dev/tcp gets no blocking SkillSpector finding. SkillEvaluator therefore reports the manifest as the Critical skillspector_hook_payload_unverified, with a gate failure that neither a --policy severity override nor --llm-verify clears, instead of INCOMPLETE (see Troubleshooting). This applies when the row is a whole-file row that names only bundled_execution_surface and no error class, on a manifest whose BH1 says unmodeled (also when a baseline suppresses that BH1), and the report accounts for it exactly: one such row per manifest, bundled_execution_surface degraded by exactly those rows with nothing failed or skipped, its Analyzer bundled_execution_surface status: degraded. limitation, and one partially inspected file for each manifest that no other row names. The rest of the report is then judged on its own: it is complete when it shows no other gap, and missing references and referenced images can still be accepted with their warnings as described above. The v2.13.3 missing-reference caveat that is accepted from the report alone does not apply, because that check reads the whole report, hook row included. Any other gap still records the SkillSpector scan as incomplete (incomplete_scans in the JSON report, and publication eligibility Incomplete), with an error that names the other rows, or the other gaps when no other row remains. The check itself is then reported as FAIL, not INCOMPLETE, because the Critical and its gate failure are a confirmed failure, which takes precedence over missing evidence, also when a --policy override lowers the Critical (see Reading the final status). A row that does not fit, such as one for a manifest in which SkillSpector parsed no hook (no unmodeled BH1), or a report that does not account for it, leaves the whole scan INCOMPLETE as before.
SkillSpector records the same row when it could not parse part of the manifest or read only its first 2,048 hook, permission and settings records. How much of that SkillEvaluator can see depends on the release:
- v2.13.4 reports
observed_recordsandlimit_recordson the row. SkillEvaluator accepts the row only whenobserved_recordsequals the BH1declaration_countand does not exceedlimit_records. A manifest with more records than SkillSpector reads, or with a hook handler it counted but could not parse, therefore staysINCOMPLETE. So does a settings file with permission rules,envor other settings beside its hooks, because SkillSpector counts those records too. A part it does not count, such as a hook group that is not an object, is not visible and gets the Critical. - v2.13.3 does not report them. SkillEvaluator then refuses only a BH1 that counts 2,048 or more declarations, because SkillSpector may have cut that manifest short. A manifest that SkillSpector parsed only in part, such as one with a malformed handler beside a valid one, gets the same Critical as a manifest it read in full, and still fails.
Incomplete referenced artifacts. SkillSpector reports a High Incomplete referenced artifact analysis (AE1) finding at each line that names a file it did not completely inspect. Releases before v2.12.0 leave the finding’s name empty, so SkillEvaluator names it by its rule instead of Unknown (AE1). A severity_overrides entry for these findings uses SECURITY.Incomplete referenced artifact analysis (AE1); one keyed on SECURITY.Unknown (AE1) no longer matches. The finding’s suggestion names the file, its disposition (partial, failed, out_of_scope or uninspected) and each reason SkillSpector gives: the reason code, such as static_parse_limit or read_error, the analyzers, the line in the file, the message and any limits, such as limit_characters=8000. The suggestion is part of the finding’s finding_id, so it leaves out the run time and the time budget (observed_seconds, limit_seconds), which differ from run to run; the metadata keeps them. Reasons that differ only in the analyzer, the run time or the time budget are listed once. SkillSpector’s own remedy is kept. Releases before v2.12.0 give no reasons in the finding and a generic remedy that says to make the file available: SkillEvaluator then reads the reasons from the report’s analysis_completeness rows for that file and gives a remedy for a parse limit, a read error, or any other reason. Without such a row the finding keeps SkillSpector’s text. The finding’s metadata records the file, its disposition and up to 16 reasons under skillspector_evidence, with reasons_truncated when SkillSpector gave more, and up to 128 reference lines under skillspector_occurrences. In a folder of skills each reference line’s file starts with the skill’s [name] prefix, like the finding’s own path, and target_path stays relative to that skill. Text with control characters is left out, credential-shaped text is masked, and a path outside the scanned target is dropped.
When the check fails, the terminal output, the references row of the terminal summary, and the Markdown and HTML reports add one line, such as skillspector could not completely inspect 1 referenced artifact(s) across 2 reference occurrence(s); each AE1 finding names the artifact and any reasons SkillSpector gave. The HTML and Markdown reports that the CI workflow renders again from the JSON report show the same line. The JSON report records the counts for every result, passed or failed, in skillspector_reference_summary: affected_artifacts, reference_occurrences, and reference_occurrences_truncated, which is true when SkillSpector listed more than 128 lines for one finding, so the count is a lower bound. For a folder of skills the counts are totals. None of this changes the finding’s severity or whether the scan is complete. A gate changes only where a policy lowered SECURITY.Unknown (AE1) and that was what let the check pass.
Agent control paths. SkillSpector rules describe attacks on an agent, so a match is an agent threat only in a file the agent reads or runs. When validate or tier1 detects a skill, rules, workflows, or plugin artifact, the security check keeps findings on agent control paths and suppresses the rest:
SkillSpector marks source files executable wherever they are, including test suites under tests/ and UI code under web/. Current releases (v2.12.0 and later) cover .py, .js, .ts, .tsx, .jsx, .mjs, .cjs, .mts, .cts, .svg, .php, .class, and .wasm. Those files are agent executables and seed the reference walk, so their findings stay blocking, and so do the findings of any fixture or other file they name.
The reference walk recognizes these forms, without regard to case:
- A path relative to the artifact or to the referring file, such as
docs/guide.md,./docs/guide.md,../lib/helper.py, ordocs\guide.md. - A path behind a skill or plugin root:
${CLAUDE_SKILL_DIR}/,$SKILL_DIR/,$SKILL_ROOT/, and${CLAUDE_PLUGIN_ROOT}/, in shell, PowerShell ($env:NAME), and cmd (%NAME%) forms, and a skill-directory placeholder whose name containsskillorbase, such as{baseDir}/,{{skill_dir}}/, or<skill-dir>/. - A path with a directory behind any other prefix, such as an install location (
~/.claude/skills/demo/docs/guide.md), a root-relative path (/docs/guide.md), or another placeholder (<root>/docs/guide.md). A top-level file behind a prefix counts when the prefix ends with the artifact’s directory name (~/.claude/skills/demo/README.mdfor a skill indemo/). - A bare file name of a nested file, a directory name for every file under it, and Python and JavaScript imports, including
-mmodule forms. In Markdown and other prose, a top-level directory name counts only with a slash, a./prefix, in quotes or backticks (docs/,./docs,`docs`), or as a word or glob inside a code span or fenced block, because the bare word also appears in ordinary prose such as “build a web page from the docs”. In scripts and other code, including any file SkillSpector marks executable, a bare directory name also counts, because commands such asfind policies -type fandcd policiestake it as an argument.
SkillSpector can report identical matches from several files as one issue: location names one of the files and occurrences lists all of them. Such an issue is suppressed only when every file in it is outside agent control paths. If any occurrence is on a control path or in a referenced file, the issue is kept with that file’s role.
A single-file target, such as one rules file, is always an agent control path. Supply-chain findings (SC*), YARA malware, webshell, and cryptominer signatures (YR1–YR3), and artifact-integrity findings for content that does not match its extension or a text file with embedded NUL bytes (AE2, AE3) are kept on every path because they describe what the artifact ships. A file named as a raster image (.png, .jpg, .gif, .webp, .ico, .avif) is treated as an image only when its content starts with an image signature; otherwise its findings are kept with file role unverified_image. Suppressed issues no longer count toward the aggregate SkillSpector risk-score gate. The terminal summary shows a filtered row, and the JSON report records a per-result skillspector_applicability block with the kept and suppressed counts, the suppressed counts by file role, and the raw and applicable risk scores. Kept findings carry artifact_type, file_role, and applicability metadata.
Classification fails closed when it cannot place a finding or finish the reference walk:
- An issue keeps its finding when a location or occurrence is missing, outside the artifact, or reached through an escaping symlink, or when its
occurrencesfield is malformed. - Every finding is kept when the report serializes fewer occurrence records than its
analysis_completeness.findings_after_filteringcount, because the unserialized findings have no location. - Every finding is kept when the walk meets a symlinked directory, a file it cannot read as text, or a plugin manifest it cannot load, such as one with an invalid YAML date. Verified raster images and fonts are the exception, because they hold no path text. A PDF, Office document, or other binary file in a control location, such as
assets/guide.pdf, therefore keeps the unrestricted scan for that skill. - Every finding is kept when the walk exceeds a budget: a file it reads is over 1 MB, it reads more than 5 MB or 1,000 files, the artifact has more than 10,000 files, or it needs too many reference comparisons. Very large skills can therefore keep the unrestricted scan.
The walk recognizes references written as text. A file that a control file reaches only through a path it builds at run time, such as "web/" + name, is not followed. Neither is a top-level file named behind a prefix that is not a recognized root variable or placeholder and does not end with the artifact’s directory name, such as <root>/README.md or an install directory with a different name. Top-level files are agent references whether or not they are named, except READMEs, .rst documents, dotfiles that are not recognized agent files (such as .env; .cursorrules and .mcp.json stay kept), test files, notebooks, images, lockfiles, and generated skill-card.md and BENCHMARK.md files.
security-scan, and validate on a target whose type is unknown, keep the unrestricted scan and report every SkillSpector finding.
To explain why two otherwise identical --llm scans differ, the reports record the LLM provenance that SkillSpector v2.12.0 and later return. The Security result in the JSON report carries:
skillspector_llm_provenance: SkillSpector’s provider adapter, the model and SkillSpector revision behind each LLM analyzer, the requested and forwarded temperature, seed, and reasoning-effort controls, and a determinism classification. Provider output is never guaranteed to be identical.skillspector_provider_id: the SkillEvaluator provider the scan was bridged from (nv_build,openai,openai-compatible,anthropic, orbedrock).nv_buildandopenai-compatibleboth run as SkillSpector’sopenaiadapter. The field is omitted whenSKILLSPECTOR_PROVIDERselects a SkillSpector adapter directly (anthropic,bedrock, oropenai) or names an unsupported provider.SKILLSPECTOR_PROVIDER=nv_buildis bridged likeSKILL_EVAL_LLM_PROVIDER=nv_buildand recorded asnv_build.
The HTML report shows the same data in the Tier 1 LLM Provenance table on the Tier 1 tab. The record contains no prompts, endpoints, or credentials. A record that does not match SkillSpector’s provenance contract, or that contains a URL or credential-shaped value, is left out rather than partially shown. If the LLM stage fails or SkillSpector does not confirm LLM analysis, a valid record in SkillSpector’s output is still kept to show what was attempted, and the Security result stays incomplete. Provenance is recorded when you scan a single skill. A scan of a folder with several skills does not record it yet.
SkillSpector 2.11.1 and later tags an exact read-only /etc/passwd bind mount, such as docker run -v /etc/passwd:/etc/passwd:ro or the Podman, --volume, and quoted forms, as likely benign but still reports it as a HIGH Credential Access (PE3) finding. SkillEvaluator ignores that finding only when SkillSpector tagged it benign and every /etc/passwd on the reported source line is the source or target of such a mount. Writable or implicit-mode mounts, other source or target paths, and reads such as cat /etc/passwd still block.
SkillSpector 2.13.3 can report separate findings for the mount’s source and target; both must pass the same checks. Ignoring a mount never clears incomplete inspection, including refused or truncated LLM responses. A static report can say an LLM provider is available without having requested or applied LLM analysis; provider availability alone does not invalidate the static scan.
To accept reviewed findings, pass an audited baseline with --skillspector-baseline; see Audited SkillSpector baselines.
pii — PII, credentials, and local identifiers
pii-scan, and the pii check inside validate, use configurable regex patterns to detect sensitive values.
Instruction content in .mdc rules and workflow files is included, for both
direct file inputs and directory scans.
Numeric values are judged by their context as well as their shape. A card-shaped number is not reported when a timestamp or digest label (timestamp_microseconds:, event_timestamp:, epoch, epoch_us, unix_time, sha256:, checksum=) directly precedes it or when it sits inside a longer ASCII token, such as a hex digest or an identifier like id_1707938736269577 where an underscore joins it to a letter or digit (unless a payment word such as card, cc, or pan is part of that identifier, as in card_4532015112830366). A card in Markdown underscore emphasis (_4532015112830366_), after a string escape such as \n or \t or a percent-encoded byte such as %3D, or written directly next to Chinese, Japanese, Korean, or other non-ASCII letters (including the Turkish dotless ı) is still reported. Widely published test card numbers (4111111111111111, 4242424242424242, 5555555555554444, and similar) are excepted however their digits are grouped, without hiding a different card number on the same line. An international phone candidate that starts inside a scientific-notation value such as 1.000000e+00 is ignored. A GPS-shaped decimal pair is reported only with a geospatial cue on its line, on one of the two lines before it, or in the file name. Strong cues are GPS, latitude/longitude, lat/lng, latlong/lonlat keys, WGS 84, a geo: URI, a Google, Apple, OpenStreetMap, or Bing map link, and a GeoJSON geometry, Feature, or collection "type" or "geometry" member. Soft cues are location, office, address, degrees, and °. Cue words count in their plural forms, with a numeric suffix, and inside snake_case or camelCase keys and file names (store_locations:, latlngs, lat1, homeLocation, gpsCoords, locations.json). The GeoJSON type and geometry keys count double-quoted, single-quoted, or bare, so JSON, Python dicts, JS objects, and YAML all carry the cue; a bare type value must use GeoJSON’s spelling (Point, MultiPolygon). Every position inside a GeoJSON geometry’s coordinates array is cued, however deeply its rings are nested, when the geometry is written with braces (JSON, a Python dict, or a JS object); block-style YAML geometry is cued only through the two-line window. Technical labels outweigh soft cues but not strong ones: waypoint, quaternion, pose, vector, calibration, roi, and scale count only on the pair’s own line and only before the pair (so Location: 37.774929, -122.419416 (map scale 1:5000) is still reported), while <pixel|camera|world|...> coordinates also counts on the two lines before it and in the file name. Pairs without a cue are not reported, for example coordinates: 37.7749, -122.4194, Meet me at 37.7749, -122.4194, or home: 37.774929, -122.419416.
Common placeholders such as YOUR_API_KEY_HERE, example, test, dummy, and placeholder, along with @example.com and @test.com, are excluded, as are configured allowed paths.
Proton (Steam Play) game prefixes are not personal paths: pfx/drive_c/users/steamuser/ is the same fixed Windows profile for every player. Other user names, the same profile outside a pfx/drive_c prefix (for example ~/.wine/drive_c/users/steamuser/ or /Users/steamuser/), and a profile path followed by a .. segment anywhere later on its line are still reported, as is any other personal path on the same line.
A card-shaped number is reported only when it passes the Luhn checksum; all-zero numbers do not. Digits inside a canonical GUID/UUID, such as the COM interface ID {00000000-0000-0000-C000-000000000046}, are treated as an identifier unless a payment label (card_number, credit_card, cc_no, cc_number, pan, and similar) directly precedes the GUID on its line or ends the line before it. Every card-shaped number on a line is checked, so an identifier or non-card number earlier on the line does not hide a card after it.
To accept a specific reviewed value, use a PII waiver rather than a PII.* severity override, which would relax every finding of that check.
The home-path check compares detected user paths against the submitter’s identity. When no identity is available, that check is skipped and logged — set SKILLEVALUATOR_SUBMITTER (or add author to SKILL.md) to enable it. SKILLEVALUATOR_SUBMITTER is combined with GITHUB_ACTOR, USER, LOGNAME, USERNAME, the OS login name, and the skill’s declared author — every identity that resolves is protected.
license — compliance
Looks for license evidence in frontmatter, common license files, and SPDX headers. Detected identifiers are normalized and matched against the packaged permissive allowlist and restrictive blocklist. Missing or unknown licenses are reported for review; blocked licenses fail according to the active policy (CRITICAL under the bundled default policy).
Frontmatter values and SPDX headers can be SPDX license expressions with AND, OR, WITH, and parentheses, such as MIT OR Apache-2.0. An expression passes only when every license in it is allowed, and any blocked license fails it. The +, -only, and -or-later forms of a blocked license, such as GPL-3.0+ or LGPL-2.1-only, and deprecated ids such as GPL-2.0-with-classpath-exception are blocked too. A WITH exception keeps an allowed license allowed only when it is in the packaged allowed_license_exceptions list (LLVM-exception, Swift-exception); other exceptions are reported for review. An expression that uses an upper-case AND, OR, or WITH but has unbalanced parentheses or a missing or repeated operator fails with an unsupported_license_expression finding. A value without those operators, such as MIT License or MIT License (MIT), is treated as a license name: it is matched by name as before or reported for review. A blocked license fails the value even when it does not parse, so GPL-3.0 (or later) and (GPL-3.0 OR MIT are blocked.
code-integrity — code risk, secrets, and hygiene
Combines three validation groups:
- Code risk (Bandit and Semgrep) — SQL or command injection, hardcoded passwords, insecure cryptography, unsafe deserialization,
shell=True,eval()/exec(), CWE patterns, and language-specific anti-patterns. - Secrets (Gitleaks) — API keys, tokens, passwords, private keys, database credentials, and other high-confidence secret patterns. Authored eval files, including
evals/evals.json, graders such asevals/grader.py, andevals/filesinputs, are scanned too. Generated Tier 3 output inevals/resultsis skipped. Files under a folder namedtests,examples,fixtures, ormocks, or the singular form, are skipped at any depth, including the Harbor task graders inevals/harbor/<case>/tests/, so a secret there is not reported. To keep an intentional test token, add agitleaks:allowcomment on its line or list the finding in a.gitleaksignorefile at the skill root. - Hygiene — dead relative Markdown links, banned or unpinned dependency declarations, and static discovery of conventional Python test filenames.
Default Tier 1 does not import or execute target-controlled Python code. The test_discovery result is filename evidence only: it counts regular, in-tree, non-symlink test_*.py and *_test.py candidates without parsing or importing them. It reports execution_performed=false and coverage_measured=false; test success and code coverage must be established separately in a trusted project environment or an explicit sandbox.
A missing required scanner, or a timed-out, crashed, or malformed Bandit, Semgrep, Gitleaks, or SkillSpector run, is treated as INCOMPLETE: terminal and saved reports are non-green and validate exits non-zero. When Semgrep runs but cannot fully parse some files (for example a shell or JavaScript file with syntax Semgrep’s parser rejects), the check stays INCOMPLETE because Semgrep skips the code it cannot parse, so findings there are missed. The report names up to 20 of those files, relative to the skill and with the line of the first error, keeps Semgrep’s findings for the other files, and lists them in the semgrep_unparsed_files field of the JSON report, up to 1,000 files; semgrep_unparsed_files_omitted counts any beyond that. Fix the syntax error, or check whether it comes from Semgrep’s parser or its rule configuration. See Scanner setup for installation.
unicode — smuggling detection
Detects invisible Unicode characters associated with ASCII smuggling, hidden-data encoding, and trojan-source attacks.
Severity increases with suspicious run length. A BOM (U+FEFF) at byte zero is downgraded to INFO, and binary files are skipped using MIME and null-byte detection.
quality — weighted scoring
quality-check, and the quality check inside validate, produce a 0–100 composite score across four weighted categories.
The 5,000-token budget for SKILL.md is a recommendation, not a hard limit. Token counts are estimates (characters divided by 4), not tokenizer output: each model family counts differently, and dense files with tables, identifiers or code spans usually have fewer than 4 characters per token, so the estimate runs low. metrics.estimated_tokens holds the estimate and metrics.estimate_method names how it was made; metrics.total_tokens is kept as the same value for existing consumers. A larger SKILL.md gets a MEDIUM quality_efficiency finding and a 15-point Efficiency deduction. The finding does not fail the check by itself, but the deduction can still take the score below --min-score. The limit and the deduction are set in the quality config file described below.
Grades are A for scores of at least 90, B for at least 80, C for at least 70, D for at least 60, and F below 60. The default --min-score is 70. Passing requires the score to reach that minimum and no HIGH or CRITICAL quality findings under the active policy.
Each category starts at 100 and loses a fixed number of points per triggered finding. The Metrics Reference gives the composite formula, the deduction behind every finding, and how this 0–100 score differs from the similarly named Tier 3 dimensions.
The minimum score is an independent gate. Policy severity overrides cannot
waive it, and a collection average cannot hide an individual skill below
the minimum. A below-minimum result includes a diagnostic such as
Quality score 67.8/100 is below the required 70/100, retained in the JSON
result’s gate_failures list. Improve the skill or explicitly choose the
required threshold with --min-score.
Correctness reports XML-style tags in any frontmatter value as a HIGH finding (potential prompt injection). A string argument-hint, or an unquoted hint such as [<file>] that YAML reads as a list of strings, may use command-line placeholders such as <file> [--strict], <plan|apply|verify>, or <codec e.g. av1|h264>. A placeholder has at most four words, and | separates choices that are each one word or a quoted literal, after an optional label such as stage: or codec e.g.. Longer free text, shell command syntax (;, &, a backtick, $(, ${, or a pipeline such as <curl x | sh>), closing tags, attributes, self-closing tags, and role or control tags such as <system>, <tool_call>, or <script> are still reported there. A role word (system, developer, assistant, user, tool, context, instructions, prompt, override, rules, or policy) followed by prompt, message, call, use, result, or instruction counts as a role tag (<system_prompt>, <toolCall>, <user_message>), while other names such as <tool_name>, <user_id>, <prompt_file>, or <toolchain> are ordinary placeholders. A hint with a redirect, that is a > that closes no group (<cmd > file>, <cmd>file>, <cmd 2>file>, <curl x> /tmp/y>), or a group with anything after a nested placeholder other than its own closing > (<<curl x>/tmp/<y>>, <<x><y>>) or anything other than a plain name before it (<curl;sh<x>>, <$(curl x|sh)<x>>), is not masked at all, so it is still reported. A placeholder nested directly after a plain name, such as <R<NNN>>, is masked. For a list value, each item and the space-joined hint are checked separately. An arrow or comparison between groups, such as <old> -> <new>, <n> >= 1, or <count> (>0), is not a redirect. Every other field keeps the strict check.
The checker detects five skill shapes and applies relevant checks: script-based (Python, shell or Node .mjs, .cjs and .js files under scripts/ or tools/), lib-based (a Python module containing __init__.py), resource-based (assets, templates, design-system content, or resources), guide-only (documentation without executable or resource content), and hybrid (scripts plus library or resource content).
A script-based or hybrid skill is expected to document its scripts in an ## Available Scripts table and to show how to run them. The check looks for a run_script call. A skill whose scripts are all Node scripts is not asked for it, because that call names a runner such a skill does not use; a skill that also bundles Python or shell scripts keeps the check. Node scripts are not read by the error-handling reliability check, which looks at Python scripts only. A Node-script skill that was typed guide-only before can therefore score differently after upgrading.
Quality signals use boundary-aware lexical matching for unambiguous terms and CommonMark parsing for structural link evidence. Links inside code, images, or comments do not count, and external URLs do not satisfy local-reference checks. Deterministic Tier 1 checks do not infer author intent from ambiguous prose such as MCP negation, temporal language, README guidance, or exclusivity claims. Use rubric-eval for semantic judgments about the documentation and Tier 3 for observed agent behavior.
If a quality score changes unexpectedly, inspect the finding message rather than adding keywords. The checker rewards usable instructions and evidence in their relevant context, not vocabulary alone.
Quality config file. Pass --quality-config PATH to quality-check, tier1, or validate to adjust the rules below. The file has five optional sections: expected_headers and error_handling for Reliability, efficiency for the token budget, discoverability for the description length limits, and correctness for reserved words in the skill name. Anything you leave out keeps its packaged value from quality-default.yaml, and any list you set replaces the packaged list rather than merging with it.
Expected headings. Reliability deducts points when SKILL.md lacks an expected heading. The packaged quality-default.yaml expects ## Prerequisites (or ## Requirements), ## Limitations, and ## Troubleshooting (or ## Common Issues or ## FAQ), each worth 5 points. Omit expected_headers to keep that list. A list you set replaces it, so copy any default entries you want to keep. Set expected_headers: [] to turn heading checks off.
The packaged defaults set message and suggestion explicitly, so their findings keep the original wording.
An entry is satisfied when the heading or any alternative appears anywhere in SKILL.md as text, so ## Security also matches ### Security Notes or a heading inside a code block.
Error-handling checks. Reliability also runs three error-handling rules, set in the error_handling section of the same file. An omitted rule, or an omitted key within a rule, keeps its packaged value. Set points: 0 to turn a rule off; it then adds no finding. This differs from expected_headers, where an entry with points: 0 still reports a zero-point finding that does not change the score.
Each rule also takes message and suggestion. The scripts and lib findings append up to three file names to the message, as in Scripts may lack error handling: run.py. The try:/except pair always counts as error handling and is not configurable. See the packaged quality-default.yaml for every default term and marker. The quality_config digest in the report covers only the rules your file sets, not the packaged values it leaves out, so the digest changes only when your file’s rules change.
Token budget. Efficiency deducts points when SKILL.md is larger than a token limit, set in the efficiency section under token_budget. As with error_handling, an omitted key keeps its packaged value and points: 0 turns the rule off, so it adds no finding. This file raises the limit and changes nothing else:
The size is an estimate, not a tokenizer count: the number of characters in SKILL.md, frontmatter included, divided by 4 and rounded down. Files in references/ and scripts/ are not counted. The finding appends the estimate and the limit to message, as in Large skill: ~6200 estimated tokens at 4 characters per token, recommended max <5000; a real tokenizer may count more. Keep a custom message short: report summaries show only the first 160 characters of a finding message, so a long one pushes the estimate and the limit out of view. Reports show the limit in force as recommended_max_tokens in the quality metrics, whether or not the rule is turned off.
Description length. Discoverability deducts points when the frontmatter description is too short or too long, set in the discoverability section under description_too_short and description_too_long. An omitted key keeps its packaged value and points: 0 turns a rule off, so it adds no finding. This file raises the minimum and the maximum and changes nothing else:
The length is counted in characters after leading and trailing whitespace is removed. A skill without a description key, or with an empty one (description: ""), is not reported by these rules. A description: key left without a value is not empty: YAML reads it as null, which is counted as the 4-character text None and reported as too short. Each finding appends the length and the limit to message, as in Description too short (recommend 50-150 chars): 12 chars, minimum 20 and Description very long (recommend 50-150 chars): 251 chars, maximum 200. The two rules are checked independently, so a minimum set above the maximum makes both fire on a description between them. The other Discoverability rules are not configurable yet.
Reserved words in the name. Correctness deducts points when the frontmatter name contains a reserved word, set in the correctness section under reserved_name. An omitted key keeps its packaged value. A terms list you set replaces the packaged list, so copy anthropic and claude if you want to keep them:
The finding appends the matched terms to message, as in Name contains reserved word: claude. Several matches still deduct points once. This is an error-level rule: its finding is HIGH and fails the quality check whatever the score, so points: 0 removes that failure as well as the deduction. The schema check has its own fixed list (anthropic, claude) that this file does not change, so with the default checks a name containing either word is still rejected there. The other Correctness rules are not configurable yet.
The file is checked before any validation runs. A file that is missing, unreadable, not UTF-8, invalid YAML, has an unknown key (such as point instead of points), lists the same heading twice, has a negative or non-numeric points, a max_tokens, min_chars or max_chars that is not a whole number from 1 to 1,000,000,000, an empty terms or markers list, a repeated entry in one of those lists, or a reserved-name term that could never match a skill name (such as Claude Code or claude_code) is a usage error: the command exits with code 2 and names the file and the problem. validate checks the file even when --checks leaves out quality.
rubric-eval — LLM-as-judge (requires a provider)
rubric-eval sends bounded skill documentation and selected supplementary content to your configured LLM provider. It scores nine criteria:
Each criterion is scored from 0 to 10. The local evaluator, not the model’s boolean, treats 7 or higher as passing. The overall score is an importance-weighted mean, and the rubric passes only when that score reaches --min-score and every criterion passes.
If the judge’s reply is not valid JSON, misses, repeats or invents a criterion, or gives a criterion a score outside 0 to 10 or a pass flag that is not a boolean, rubric-eval asks the judge once more and names the problem, keeping the first 500 characters of it. A second such reply fails the rubric with a HIGH llm_invalid_response finding, or llm_unavailable when it is still not JSON. A rate limit, server error or other transient provider error is retried by the LLM client as set by SKILL_EVAL_LLM_MAX_RETRIES (see LLM retries and judge time budget). A request that still fails after those retries, a refusal or a truncated reply is not asked again.
The JSON report records judge_attempts, the number of times rubric-eval asked the judge (1, or 2 after a rejected reply), in the skill’s rubric_eval success detail or llm_invalid_response / llm_unavailable finding and, for a single skill, also in the rubric_eval object. The client’s own retries are not counted. When the first request fails, for example because no provider is configured, no reply came back and judge_attempts is left out.
The JSON report names the provider and model that gave each score as llm_provider and llm_model_used in the skill’s rubric_eval success detail and, for a single skill, also in the rubric_eval object. For a Claude 5.5 or later model on the native Claude API, the model is the one the reply names; otherwise it is the configured model, unless a server-side fallback block in the reply names another model. A Claude 5.5 or later model’s refusal can be answered by the server-side fallback model, and the API can send a repeat of that request straight to it; the report then names the fallback model, not the configured model.
lint — script linting (advisory)
lint-scripts, and the lint check inside validate, parse Python scripts under scripts/ and report advisory findings. Node scripts count as scripts for skill-type detection and the script-based checks but are not linted; the result says how many Node scripts were not linted. These findings do not make the Tier 1 gate fail.
version — semantic version check
The default version check validates an optional metadata.version as a three-part numeric value in major.minor.patch form:
When --previous-version or SKILLEVALUATOR_PREVIOUS_VERSION is set, the current version must be present and numerically greater than the previous value. Without a previous-version bound, the version remains optional and the skill can rely on commit history. A previous-version bound is valid only when checking one skill; catalog validation must check each skill separately with its own bound.
dependency — CVE audit (opt-in)
Scans requirements*.txt for known Python package vulnerabilities. For
pyproject.toml, reads static PEP 621 dependencies and every optional group,
then audits their exact version pins without installing the target or
executing its build scripts. This covers direct declarations only;
transitive dependencies require a resolved requirements file. Markers are
evaluated against the scanner’s Python/platform and optional-group context;
nonmatching declarations are reported as skipped. Version ranges, URL/path
dependencies, dynamic or Poetry metadata, malformed declarations, and
missing or failed scanner evidence leave the audit INCOMPLETE. Supply
pinned requirements.txt for unresolved declarations. A requirements line
that ends in a backslash continues on the next line unless it is a comment
line, as in pip install -r, so a pin after a comment line that ends in a
backslash is audited. pip-audit is the
primary scanner; Safety supplies secondary coverage for requirements files
when installed separately. The audit may use network-backed vulnerability databases.
JSON and HTML export data retain dependency_coverage per target, including
requested and actually audited pins. Unreadable declarations have unknown
declaration counts and an empty audited inventory, not a completed scan.
Each advisory is a DEPENDENCY finding whose check name is the advisory ID
(a numeric Safety ID reads SAFETY-<id>). It points at the requirements
file and the line that pins the package, or at pyproject.toml, so it
appears in JSON findings, the severity counts, and SARIF results.
validate on a folder of skills validates each skill as its own job, so
each skill’s report lists its advisories. When one dependency check covers
several skills, as with validate --type rules or --type workflows on a
folder, or DependencySecurityValidator().validate() on a folder in
Python, a skill whose audit passes is summarized as one passed check, so
its medium or low advisories are not listed; validate that skill on its own
to see them. The finding metadata records:
package_nameandresolved_version: the package and version the scanner checked.declared_constraintanddeclaration: the exact pin as the target spells it, such asPyYAML==5.3, and where it is declared (line 3, oroptional group docs, declaration 2inpyproject.toml). The raw line, with its markers and options, is never copied.resolution_statusanddependency_relationship:resolvedanddirectwhen the advisory matches an audited exact pin. Only exact pins are audited, with--no-deps, so the advisory applies to the version the target declares. Safety reads the raw requirements file, so it can report a version that no audited pin declares, such as a pin skipped because its marker does not match the scanner platform. That advisory isunbound:declared_constraint,declaration, anddependency_relationshiparenull, and the finding has no line.dependency_role:runtime,dev,test,docs, orbuild, from a whole word of the requirements file name or the optional group name. Words are split at-,_, and..dev,devel,develop,development,devtools,lint, andlintingmeandev;test,tests,testing,testutils,pytest, andunittestmeantest;doc,docs, anddocumentationmeandocs;buildmeansbuild. Sorequirements-dev.txtisdev, whilerequirements-latest.txtandrequirements-docker.txtareruntime.advisory_id,advisory_source(pip-auditorSafety),affected_range(Safety only;nullfor pip-audit),fixed_versions, andaliases.
A high or critical advisory fails the run. A Python caller that builds
DependencySecurityValidator(fail_on_medium=True) also fails on a medium
pip-audit advisory; those findings record fail_on_medium: true in their
metadata, so a policy overlay keeps that rule. Safety’s severity is its
level word or, when Safety runs with an API key, its CVSS object: the
CVSS v3 level, then the v3 score, then the v2 score and level. A score
that is not a number from 0 to 10 is skipped, and a Safety advisory with
none of these is medium. When pip-audit already reported
the same advisory ID or CVE for that package version, Safety’s copy is
left out, so one vulnerability is one finding. A critical advisory does not
stop a folder run at that skill, as a critical finding of other checks
does without --continue-on-failure: every skill is audited.
A policy overlay can change one advisory with DEPENDENCY.<advisory-id>,
such as DEPENDENCY.PYSEC-2020-176: low, or every advisory with
DEPENDENCY.*. When one dependency check covers several skills, the
overlay sees only the advisories that check lists, so it cannot raise a
medium or low advisory of a skill whose audit passed. The warnings that say
why an audit is INCOMPLETE, such as a line without an exact pin, still
appear in the CLI and Markdown reports next to the advisories, and after a
policy overlay changes an advisory’s severity.
Author attribution and contact emails
When metadata.author carries a contact, write it as Name <email> in
SKILL.md. The PII check accepts that recognized author line. The same address
in another field, body text, or a reference document is a separate occurrence
and can still fail the check. A team or organization name without an email is
also valid attribution under the default profile.
Use a name, team handle, or contact-directory link for additional contact information. If a real address must remain, record a reviewed, time-limited PII waiver for that occurrence. An organization-owned email domain does not automatically exempt every address in the repository.
PII waivers
A PII waiver accepts one reviewed value at one place, for example a public support address that a skill must document. Pass the waiver file with --pii-waivers PATH on validate, tier1, or pii-scan. Waivers are never discovered automatically inside the scanned skill, so a skill cannot waive its own findings. Keep it outside the skill: a waiver file inside the scanned tree ships with the skill and is PII-scanned like any other file, owners and reasons included.
categoryis the PII check name (emails,personal_paths,phone_numbers, and so on) andlineis the line of the value. Credential checks (database_credentials,hardcoded_secrets,jwt_tokens,github_tokens,aws_identifiers,webhook_urls,private_keys) cannot be waived: remove or rotate the credential instead.fileis relative to the directory that holds the waiver file (the directory of the link, when the waiver file is a symlink), so one file covers single-skill, folder, and catalog runs. Absolute paths,.., and control characters are rejected. A path through a symlinked directory, such as.claude/skills/support/SKILL.mdwhen.claude/skills/supportlinks toskills/support, names the same file as the path without the link.evidence_sha256comes from theevidence_sha256metadata of the PII finding in the JSON report. It is a SHA-256 digest of the check name and the case-folded value, so the waiver file never contains the value. A digest is an identifier, not encryption: a short value such as an SSN can be recovered from it by trying every candidate. Findings report a digest only for a value the report shows in full, so credential findings, and any other finding whose value reports mask because it contains a credential, carry none.reasonneeds at least 8 characters,ownernames the reviewer or team, and the waiver applies throughexpires. Dates are UTC calendar dates;createdmay be at most one day after the UTC date, so a reviewer east of UTC can enter their local date, andexpiresmay not be beforecreatedor more than 366 days after it, so every waiver comes up for review at least yearly.
A waiver applies only while the check, file, line, and value all match. When the value changes or moves, the finding is reported again, and other values on the same line are reported as usual. The outcome of each waiver the run judges is in the reports under the PII_WAIVER category:
A run judges only the waivers whose file lies in a skill or folder it scanned; a waiver for a deleted file there is stale. A run on one skill, and each skill of a catalog run, reports the waivers for that skill’s files and leaves the other waivers in the file unjudged. Waiver findings name the file the way the PII findings of the same report do, for example [support] SKILL.md in a folder run; the waiver’s own file value is in the waiver_file metadata. They do not repeat the waiver’s evidence_sha256, because the run may now mask that value.
The command line rejects a malformed waiver file before the run starts, with exit code 2 and the problem named: unknown or missing fields, duplicate ids or keys, unknown or credential check names, and malformed paths, digests, or dates. Waivers apply only to the pii check; findings from other checks for the same value, such as Gitleaks secrets, are unaffected.
Scanner setup
Bandit installs with SkillEvaluator. Tier 1 provisions SkillSpector and, on
Linux and macOS, Gitleaks when they are missing; the first run needs network
access. Semgrep is optional unless SKILLEVALUATOR_REQUIRE_SEMGREP=1 is set.
See the installation guide
for scanner setup, offline preparation, and the optional dependency-audit extra.
Semgrep uses a policy packaged with the installed distribution, with metrics and version checks disabled; Tier 1 never fetches a registry policy at runtime. The other public Tier 1 configuration also ships inside the package under skillevaluator/config/: pii_patterns.yaml, unicode_smuggle_patterns.yaml, license_config.yaml, and profiles/default.yaml.
Bundled Python scanners resolve next to the SkillEvaluator interpreter before PATH; external Semgrep and SkillSpector installations resolve from PATH. Intentional replacements require an auditable absolute executable path in SKILLEVALUATOR_BANDIT_PATH, SKILLEVALUATOR_SEMGREP_PATH, or SKILLEVALUATOR_SKILLSPECTOR_PATH.
Scanner path overrides fail closed: a relative, missing, or non-executable override path fails the run rather than silently falling back. Required scanners that are missing or produce malformed output leave the run INCOMPLETE and non-green.
Audited SkillSpector baselines
Record reviewed SkillSpector exceptions in a SkillSpector YAML or JSON baseline and pass it with --skillspector-baseline to validate, tier1, or security-scan. SkillEvaluator never discovers a baseline on its own, including a .skillspector-baseline.yaml shipped inside the skill: applying exceptions is an operator or CI decision. A scan that reports suppressed findings without a requested baseline stays INCOMPLETE.
Keep the baseline outside the skill under review, for example in a checkout of the base branch or a separate policy repository. A baseline kept inside the skill is only as trusted as the review of that file: whoever edits the skill can edit its exceptions, SkillSpector leaves the selected baseline file out of its own scan so extra text in it is never inspected, and a version 2 rule needs only a non-empty reason. In a pull-request pipeline, the checked-out skill is the PR’s version, so pass a baseline from the target branch instead.
Prefer narrow rules. Every field a rule sets must match the finding. id (the rule ID), path (the file), and message (the finding text) are all case-insensitive globs, and reason is required.
The bracket class [d] matches the letter d, so the rule still matches --privileged, but the baseline file does not contain the literal text that triggers the finding. SkillEvaluator rejects a rule unless its id has at least two literal letters or digits in a row, or its path or message has at least four. Wildcards and multi-character classes break a run. This rejects id: "*" and also message: "*e*", which matches almost every finding because SkillSpector matches the glob case-insensitively against the finding text. Before the scan it also rejects a version 2 rule or fingerprint without a reason, rules or fingerprints that are not lists or contain entries that are not mappings, a file that is not valid YAML or JSON (including an impossible date such as 2026-06-31), and a file larger than 2 MiB or whose aliases and merge keys expand beyond SkillSpector’s own limits. An empty or comment-only file is an empty baseline, as in SkillSpector. SkillSpector checks the rest of the schema. If it rejects the baseline, it exits with code 2, the security check is INCOMPLETE, and the error suggests checking the file with skillspector scan <skill> --baseline <file>.
To waive individual findings exactly, generate fingerprints with SkillSpector. The command records every current finding, so review each entry before you commit the file:
Fingerprints bind to the scanned file content and to the SkillSpector version that generated them. After a source edit, the affected findings are reported again. After a scanner upgrade, SkillSpector ignores every fingerprint, and SkillEvaluator reports fingerprint_version_mismatch with both versions: as an error when blocking findings remain, otherwise as a warning. Rules keep working across scanner versions.
Fingerprints only match static-only runs: tier1 --no-llm, or validate and security-scan without --llm. When the LLM security stage runs (tier1 by default when a provider is configured, or --llm), SkillSpector’s meta-analysis rewrites each finding’s message, explanation, remediation, confidence, or tags before the baseline is applied. Those fields are part of the fingerprint, so fingerprints from skillspector baseline --no-llm no longer match, and the LLM stage reports the findings again. SkillEvaluator then reports llm_stage_unmatched with both stages’ suppressed counts: as an error when blocking findings remain, otherwise as a warning. For runs with LLM analysis, use narrow id, path, and message rules instead; they still match because SkillSpector also checks message against the matched text.
A baseline stored inside the skill is excluded from the scan, including when SkillEvaluator scans a staged copy without evals/ outputs. For a folder of skills, the same baseline applies to each skill, and rule paths are relative to each skill.
For a single-skill target, when SkillSpector is installed, JSON reports record the outcome under results[].skillspector_baseline:
status:applied,no_matches,fingerprint_version_mismatch,llm_stage_unmatched, ornot_applied.fileanddigest: the baseline’s file name and asha256:digest of its bytes. The local path is not recorded.- The baseline and runtime scanner versions, and the rule and fingerprint counts.
suppressed_findingsfrom the static stage and, when the LLM stage completed,llm_stage_suppressed_findings; and the active and active blocking finding counts. All are taken from SkillSpector’s reports before any--llm-verifydowngrade or policy severity override.suppressed_by_severityandsuppressed_rule_ids: the static stage’s suppressed findings by severity and their rule IDs, so CI can refuse a baseline that hides anything blocking. The Markdown and HTML line shows the severity counts.message: the outcome or the reason the baseline was rejected.
Markdown and HTML reports show the same outcome as a skillspector_baseline line in the Security result, with the file name and digest, and a scan whose only findings were suppressed reads “No active security findings; N finding(s) suppressed by the requested baseline” rather than a clean result.
security-scan and tier1 security-scan on a folder of skills write one merged report without the JSON field. When a skill in the folder passes, its outcome is appended to that skill’s “All checks passed” line in the Markdown and HTML reports. validate on a folder validates each skill as its own job, so each skill’s JSON report records the field. The field keeps the fingerprint_version_mismatch and llm_stage_unmatched messages even when a later --llm-verify downgrade or policy severity override rebuilds the error and warning lists. See the SkillSpector suppression reference for the full baseline schema.
Exit behavior and scoring
Tier 1 checks always gate the exit code inside validate. Tier 2 gates when it runs unless --no-block-on-dedup is set. Tier 3 is advisory unless --block-on-agent-eval is set. The quality check requires the composite score to reach --min-score (default 70) as well as no HIGH or CRITICAL quality findings under the active policy. Severity overrides cannot waive the minimum. Reports record the effective gate membership for every tier so saved evidence and process status agree.
--fail-fast stops after the first failed check. -c/--continue-on-failure overrides it, records the full pipeline, and keeps scanning a collection past a CRITICAL finding. Wiring these exit codes into a merge gate — including a complete GitHub Actions recipe — is covered in Gate Your CI.
Troubleshooting
Why is SkillSpector analysis incomplete?
Open the security check’s execution error in the CLI, HTML, or Markdown report. It now shows up to three file/line locations with their scanner phase and reason, for example:
This means the scanner could not resolve or inspect a reference; it is not itself proof of a vulnerability. Review the named source line and whether the reference should be bundled. If the scanner mistook a version or API name for a file, report that detection issue to SkillSpector.
For more detail, each JSON result’s skillspector_completeness.records
retains the scan stage, up to 20 ledger exceptions, analyzer names, and up
to eight limitations. Omission counts make truncation explicit. Folder
reports retain the skill name with each record. These diagnostics never
turn an incomplete scan into a pass.
Gitleaks is not installed
On Linux and macOS, SkillEvaluator downloads Gitleaks on first use. If that download cannot complete, follow scanner setup to prepare it for your environment.
A scanner is missing
Run skillevaluator doctor --tier 1 to identify the scanner and its resolved
path or version, then follow scanner setup.
Required scanner failures remain INCOMPLETE until the scanner returns
valid evidence. Missing optional Semgrep is reported as skipped unless
SKILLEVALUATOR_REQUIRE_SEMGREP=1 is set.
SkillSpector reports incomplete analysis for a valid helper script
SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a static_parse_limit, obfuscated_instruction_text, or runtime_limit ledger exception. The security check is then INCOMPLETE, often with a High Incomplete referenced artifact analysis (AE1) finding at each line that names the helper. The finding’s suggestion names the helper and the reason, such as static_parse_limit (static_patterns_tool_misuse). Check the resolved release with skillevaluator doctor --tier 1 and follow scanner setup to update an older installation. If the exception remains after the upgrade, investigate it like any other incomplete scan.
SkillSpector could not fully model the hooks
With SkillSpector v2.12.0 and later, a skill, folder or hooks-only plugin
with an ordinary command or HTTP hook fails the security check with the
Critical skillspector_hook_payload_unverified, for example
SkillSpector could not fully model the hooks in hooks/hooks.json (analyzer bundled_execution_surface could not verify what they do with hook input), so this check cannot pass while these hooks are present.
The check also records the gate failure
hooks/hooks.json: SkillSpector could not model these hooks; this check cannot pass while they are present.
Earlier SkillEvaluator releases reported the same scan as INCOMPLETE
(“Security Scan didn’t run”) with
hooks/hooks.json [static; opaque_content; analyzers=bundled_execution_surface].
The scan itself finished, and the other findings and the --llm stage are
reported as for any complete scan. In a folder, this Critical does not stop
the scan at that skill, so the other skills are still scanned without
--continue-on-failure.
SkillSpector cannot tell what these hooks do with their input, so review
the manifest and every file its hooks run. SkillEvaluator has no way to
record that review: removing the hooks is the only way to pass the check.
A --policy severity override can lower the finding’s severity, but the
gate failure keeps the check failed, also when another gap records the
scan as incomplete. --llm-verify does not send the
finding to the verifier, so it keeps its severity and gets no verdict. The
finding’s metadata records the manifest’s SHA-256 (manifest_sha256) and
SkillSpector’s row for it.
SkillSpector records the same row for a manifest it read only in part.
With v2.13.4, a manifest with more records than SkillSpector reads or with
a handler it could not parse, and a settings file with permission rules or
other settings beside its hooks, leave the scan INCOMPLETE with the row
instead of the Critical. With v2.13.3, only a manifest whose BH1 counts
2,048 or more declarations does. See the security check under
The checks.
SkillSpector v2.13.4 also reads a hook’s "timeout" key as a command
wrapper it cannot resolve and records
hooks/hooks.json [static; static_parse_limit; analyzers=static_patterns_tool_misuse].
That row is an ordinary analysis gap, so with v2.13.4 the SkillSpector
scan of such a manifest is also recorded as incomplete (incomplete_scans
in the JSON report, and publication eligibility Incomplete), with an
error that names only that row. The check itself is still reported as
FAIL because of the Critical and its gate failure. v2.13.3 does not
record the row.
SkillSpector times out
Each SkillSpector run (the static stage, and the LLM stage with --llm)
is stopped after 1,200 seconds by default, and the security scan is reported
INCOMPLETE with skillspector timed out after 1200 seconds. SkillSpector
also ends a scan at its own deadline (1,080 seconds by default in v2.13.3) and reports partial
analysis, which is INCOMPLETE as well. Slow or local LLM providers can
need more time. Raise both limits, keeping SkillEvaluator’s watchdog higher:
Each stage gets the full watchdog, so a scan that stalls in both stages can take up to twice the value before it is reported. See Environment Variables.
The SkillSpector LLM provider keeps failing
With --llm, SkillSpector v2.12.0 and later retries a failing provider call
a bounded number of times. When the provider still fails, for example
because of rate limits or an outage, SkillSpector records the
llm_connection_retries_exhausted reason and ends the LLM stage as failed.
If that is the only cause in a report from SkillSpector v2.12.0 or later,
the security error reads
skillspector-llm did not complete: the LLM provider kept failing after SkillSpector's bounded retries
and says that this is a retriable provider failure, not a security
finding. Retry the full scan when the provider is available.
The gate is unchanged: the security scan is INCOMPLETE, the command
exits 1, and findings from the static stage are kept, so a High or
Critical static finding still makes the result FAILED.
SkillEvaluator shows this message only when every failure that
SkillSpector recorded is a provider-retry failure on an LLM analyzer.
Reports that mix it with another cause, are truncated or malformed, or come
from an older SkillSpector keep the generic error, such as
skillspector-llm failed with unexpected exit code 2. SkillSpector v2.9.1
to v2.11.2 used the same reason only for a connection error left after the
provider SDK’s own retries, so SkillEvaluator does not read it from those
releases. A provider error that SkillSpector records under a different
reason also keeps the generic error.
An LLM-backed stage fails immediately
Configure a supported provider or select a keyless Tier 1 run with
skillevaluator tier1 ./my-skill --no-llm or
skillevaluator validate ./my-skill --tiers 1 --no-llm (without
--llm-verify). Direct Tier 1 automatically enables LLM stages when a
provider is configured; --no-llm explicitly disables them. Top-level
validate --no-dedup skips Tier 2 but still runs Tier 3 by default.
No scannable files are found
Confirm that the skill contains supported text or code extensions such as .py, .sh, .yaml, .yml, .json, .md, or .txt.
An executable override is rejected
Use an existing, executable absolute path for the relevant SKILLEVALUATOR_*_PATH variable.
SkillSpector stops at a runtime limit
When SkillSpector reports partial analysis, the security scan is
INCOMPLETE. The error quotes up to three of SkillSpector’s completeness
exceptions as path [phase; reason_code; analyzers=...]. When there are more,
it adds how many files each reason affected, for example
and 463 more (incomplete files by reason: runtime_limit 462, obfuscated_instruction_text 4).
For runtime_limit, the error also says what to do, with one piece of advice
for each kind of row the report contains:
- Scan time budget. Rows that name regular analyzers, or no analyzer,
mean SkillSpector ran out of time for the bundle: at its own whole-scan
deadline (1,080 seconds by default in v2.13.3) or at a shorter stage
budget such as bundle discovery. A skill that bundles a large corpus, such
as vendored documents, generated figures or renderer scripts, hits the
same limit on every rerun. Keep large reference material outside the
skill and point to it. A plain rerun helps only when a few files are
affected on a busy machine. If SkillEvaluator’s own SkillSpector timeout
stops the run before SkillSpector’s deadline, the error is
skillspector timed out after N secondsinstead, with no list of files. - Sub-analysis budget. Analyzers named
<analyzer>_runtime_limit, such asstatic_patterns_supply_chain_bytecode_runtime_limit, stopped at a time budget of their own, so bundle size does not matter. Releases before v2.13.0 charge the SC8 bytecode walk for time spent waiting behind other analyzers, so it can stop even on a skill with no bytecode. Rerun the scan, and update SkillSpector if the limit repeats.