Tier 1: Static and Security Validation

A custom policy overlays the external profile. It can narrow the accepted author domain and change severities by CATEGORY.check_name or CATEGORY.*:

profile: community-strict
identity:
  author_email_regex: '<[^>]+@example\.org>'
severity_overrides:
  SCHEMA.author_missing: critical
  LICENSE.*: critical

Severity overrides change how findings gate a run; they do not change the quality score or waive its --min-score requirement. Even an override of QUALITY.*: info leaves a below-minimum quality score failing.

Validates SKILL.md frontmatter and repository placement:

• YAML frontmatter must parse and satisfy the skill schema.
• Frontmatter keys, including keys nested under metadata, must be strings. An unquoted key that YAML reads as a number, boolean, null, or date (123, yes, null, 2026-01-01) is a HIGH finding that names the key in YAML form, such as Field 'metadata.true': Keys should be strings; quote the key ("yes":) to keep it. Rules, workflows, and plugin manifests report non-string keys as schema errors too.
• name and description are required. Names are 1–64 characters; descriptions are 1–1024 characters.
• Names use kebab-case, start with a letter, and cannot have trailing or consecutive hyphens. The directory name must match the frontmatter name. The reserved words anthropic and claude must not appear in a skill name.
• alwaysApply and globs are forbidden skill fields. Optional fields include license, compatibility, metadata, and allowed-tools.
• The standard hierarchy is skills/<skill-name>/ or team-skills/<team>/<skill-name>/. A nonstandard standalone location is reported as an advisory finding.
• Recognized skill-root subdirectories are agents/, references/, scripts/, assets/, evals/, tests/, tools/, and config/; anything else gets a LOW advisory finding. Extend the set per repository with SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS (comma-separated names are added to the defaults, never replace them).
• SKILL.md should stay at or below 500 lines; larger files receive an advisory finding.
• The default profile expects metadata.author in Name <email@host> form but does not require a particular email domain.
• metadata.domain accepts one category string or a list of them, such as domain: [backend, ai-ml]. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGH Field 'metadata.domain' finding. The 128-character limit also applies to a single string.

To fix common findings: rename the directory to match name, remove forbidden fields, and move the skill under skills/.

security-scan, and the security check inside validate, run SkillSpector. Static scanning covers common skill attack patterns such as prompt injection and data exfiltration; each finding carries the matched pattern, severity, and location.

skillevaluator security-scan ./my-skill
skillevaluator security-scan ./my-skill --llm
skillevaluator security-scan ./my-skill --llm --llm-verify

The LLM pass requires a configured provider. It enriches the static result but does not replace static findings or make missing scanner evidence pass.

SkillSpector’s text pattern analyzers cannot read images, so it reports the analysis of a skill that references a PNG or JPEG file as partial. SkillEvaluator accepts that gap with a warning instead of INCOMPLETE only when the report shows no other gap and no fatal scope exclusion, and each referenced image is a regular, single-link (not hard-linked), non-executable file whose bytes have the chunk structure of a PNG or the marker structure of a JPEG and that is not reached through a symlink. The gap is accepted for at most 256 images and 128 MiB of image data in total. An image is never accepted inside a hooks, scripts, tools, agents, commands, prompts, rules, team-rules or workflows directory at any depth, or under a top-level config, skills, .mcp, .agents, .claude, .codex, .continue, .cursor, .gemini or .windsurf directory (names compared without regard to case). The structure check corroborates the file type; it does not decode the image or prove that it renders. PNG IDAT compressed data and ancillary chunks, and JPEG entropy-coded data and APPn or COM segments, are not inspected and can still carry other bytes. The image content itself is not assessed. Any other partial reason still leaves the scan INCOMPLETE. SkillSpector also reports a High AE1 (incomplete referenced artifact analysis) finding for such an image; from v2.12.0 it omits that finding when the skill renders a PNG with Markdown image syntax, such as ![Logo](assets/logo.png). An AE1 finding fails the check like any other High finding.

To explain why two otherwise identical --llm scans differ, the reports record the LLM provenance that SkillSpector v2.12.0 and later return. The Security result in the JSON report carries:

• skillspector_llm_provenance: SkillSpector’s provider adapter, the model and SkillSpector revision behind each LLM analyzer, the requested and forwarded temperature, seed, and reasoning-effort controls, and a determinism classification. Provider output is never guaranteed to be identical.
• skillspector_provider_id: the SkillEvaluator provider the scan was bridged from (nv_build, openai, openai-compatible, anthropic, or bedrock). nv_build and openai-compatible both run as SkillSpector’s openai adapter. The field is omitted when SKILLSPECTOR_PROVIDER selects a SkillSpector adapter directly (anthropic, bedrock, or openai) or names an unsupported provider. SKILLSPECTOR_PROVIDER=nv_build is bridged like SKILL_EVAL_LLM_PROVIDER=nv_build and recorded as nv_build.

The HTML report shows the same data in the Tier 1 LLM Provenance table on the Tier 1 tab. The record contains no prompts, endpoints, or credentials. A record that does not match SkillSpector’s provenance contract, or that contains a URL or credential-shaped value, is left out rather than partially shown. If the LLM stage fails or SkillSpector does not confirm LLM analysis, a valid record in SkillSpector’s output is still kept to show what was attempted, and the Security result stays incomplete. Provenance is recorded when you scan a single skill. A scan of a folder with several skills does not record it yet.

SkillSpector 2.11.1 and later tags an exact read-only /etc/passwd bind mount, such as docker run -v /etc/passwd:/etc/passwd:ro or the Podman, --volume, and quoted forms, as likely benign but still reports it as a HIGH Credential Access (PE3) finding. SkillEvaluator ignores that finding only when SkillSpector tagged it benign and every /etc/passwd on the reported source line is the source or target of such a mount. Writable or implicit-mode mounts, other source or target paths, and reads such as cat /etc/passwd still block.

SkillSpector 2.13.3 can report separate findings for the mount’s source and target; both must pass the same checks. Ignoring a mount never clears incomplete inspection, including refused or truncated LLM responses. A static report can say an LLM provider is available without having requested or applied LLM analysis; provider availability alone does not invalidate the static scan.

To accept reviewed findings, pass an audited baseline with --skillspector-baseline; see Audited SkillSpector baselines.

pii-scan, and the pii check inside validate, use configurable regex patterns to detect sensitive values. Instruction content in .mdc rules and workflow files is included, for both direct file inputs and directory scans.

Common placeholders such as YOUR_API_KEY_HERE, example, test, dummy, and placeholder, along with @example.com and @test.com, are excluded, as are configured allowed paths.

A card-shaped number is reported only when it passes the Luhn checksum; all-zero numbers do not. Digits inside a canonical GUID/UUID, such as the COM interface ID {00000000-0000-0000-C000-000000000046}, are treated as an identifier unless a payment label (card_number, credit_card, cc_no, cc_number, pan, and similar) directly precedes the GUID on its line or ends the line before it. Every card-shaped number on a line is checked, so an identifier or non-card number earlier on the line does not hide a card after it.

The home-path check compares detected user paths against the submitter’s identity. When no identity is available, that check is skipped and logged — set SKILLEVALUATOR_SUBMITTER (or add author to SKILL.md) to enable it. SKILLEVALUATOR_SUBMITTER is combined with GITHUB_ACTOR, USER, LOGNAME, USERNAME, the OS login name, and the skill’s declared author — every identity that resolves is protected.

Looks for license evidence in frontmatter, common license files, and SPDX headers. Detected identifiers are normalized and matched against the packaged permissive allowlist and restrictive blocklist. Missing or unknown licenses are reported for review; blocked licenses fail according to the active profile (CRITICAL under the default external profile).

Frontmatter values and SPDX headers can be SPDX license expressions with AND, OR, WITH, and parentheses, such as MIT OR Apache-2.0. An expression passes only when every license in it is allowed, and any blocked license fails it. The +, -only, and -or-later forms of a blocked license, such as GPL-3.0+ or LGPL-2.1-only, and deprecated ids such as GPL-2.0-with-classpath-exception are blocked too. A WITH exception keeps an allowed license allowed only when it is in the packaged allowed_license_exceptions list (LLVM-exception, Swift-exception); other exceptions are reported for review. An expression that uses an upper-case AND, OR, or WITH but has unbalanced parentheses or a missing or repeated operator fails with an unsupported_license_expression finding. A value without those operators, such as MIT License or MIT License (MIT), is treated as a license name: it is matched by name as before or reported for review. A blocked license fails the value even when it does not parse, so GPL-3.0 (or later) and (GPL-3.0 OR MIT are blocked.

Combines three validation groups:

• Code risk (Bandit and Semgrep) — SQL or command injection, hardcoded passwords, insecure cryptography, unsafe deserialization, shell=True, eval()/exec(), CWE patterns, and language-specific anti-patterns.
• Secrets (Gitleaks) — API keys, tokens, passwords, private keys, database credentials, and other high-confidence secret patterns.
• Hygiene — dead relative Markdown links, banned or unpinned dependency declarations, and static discovery of conventional Python test filenames.

Default Tier 1 does not import or execute target-controlled Python code. The test_discovery result is filename evidence only: it counts regular, in-tree, non-symlink test_*.py and *_test.py candidates without parsing or importing them. It reports execution_performed=false and coverage_measured=false; test success and code coverage must be established separately in a trusted project environment or an explicit sandbox.

A missing, timed-out, crashed, or malformed Bandit, Semgrep, Gitleaks, or SkillSpector run is treated as INCOMPLETE: terminal and saved reports are non-green and validate exits non-zero. See Scanner setup for installation.

Detects invisible Unicode characters associated with ASCII smuggling, hidden-data encoding, and trojan-source attacks.

Severity increases with suspicious run length. A BOM (U+FEFF) at byte zero is downgraded to INFO, and binary files are skipped using MIME and null-byte detection.

quality-check, and the quality check inside validate, produce a 0–100 composite score across four weighted categories.

The 5,000-token budget for SKILL.md is a recommendation, not a hard limit. A larger SKILL.md gets a MEDIUM quality_efficiency finding and a 15-point Efficiency deduction. The finding does not fail the check by itself, but the deduction can still take the score below --min-score.

Grades are A for scores of at least 90, B for at least 80, C for at least 70, D for at least 60, and F below 60. The default --min-score is 70. Passing requires the score to reach that minimum and no HIGH or CRITICAL quality findings under the active policy.

The minimum score is an independent gate. Policy severity overrides cannot waive it, and a collection average cannot hide an individual skill below the minimum. A below-minimum result includes a diagnostic such as Quality score 67.8/100 is below the required 70/100, retained in the JSON result’s gate_failures list. Improve the skill or explicitly choose the required threshold with --min-score.

Correctness reports XML-style tags in any frontmatter value as a HIGH finding (potential prompt injection). A string argument-hint, or an unquoted hint such as [<file>] that YAML reads as a list of strings, may use command-line placeholders such as <file> [--strict], <plan|apply|verify>, or <codec e.g. av1|h264>. A placeholder has at most four words, and | separates choices that are each one word or a quoted literal, after an optional label such as stage: or codec e.g.. Longer free text, shell command syntax (;, &, a backtick, $(, ${, or a pipeline such as <curl x | sh>), closing tags, attributes, self-closing tags, and role or control tags such as <system>, <tool_call>, or <script> are still reported there. A role word (system, developer, assistant, user, tool, context, instructions, prompt, override, rules, or policy) followed by prompt, message, call, use, result, or instruction counts as a role tag (<system_prompt>, <toolCall>, <user_message>), while other names such as <tool_name>, <user_id>, <prompt_file>, or <toolchain> are ordinary placeholders. A hint with a redirect, that is a > that closes no group (<cmd > file>, <cmd>file>, <cmd 2>file>, <curl x> /tmp/y>), or a group with anything after a nested placeholder other than its own closing > (<<curl x>/tmp/<y>>, <<x><y>>) or anything other than a plain name before it (<curl;sh<x>>, <$(curl x|sh)<x>>), is not masked at all, so it is still reported. A placeholder nested directly after a plain name, such as <R<NNN>>, is masked. For a list value, each item and the space-joined hint are checked separately. An arrow or comparison between groups, such as <old> -> <new>, <n> >= 1, or <count> (>0), is not a redirect. Every other field keeps the strict check.

The checker detects five skill shapes and applies relevant checks: script-based (Python or shell files under scripts/), lib-based (a Python module containing __init__.py), resource-based (assets, templates, design-system content, or resources), guide-only (documentation without executable or resource content), and hybrid (scripts plus library or resource content).

Quality signals use boundary-aware lexical matching for unambiguous terms and CommonMark parsing for structural link evidence. Links inside code, images, or comments do not count, and external URLs do not satisfy local-reference checks. Deterministic Tier 1 checks do not infer author intent from ambiguous prose such as MCP negation, temporal language, README guidance, or exclusivity claims. Use rubric-eval for semantic judgments about the documentation and Tier 3 for observed agent behavior.

Expected headings. Reliability deducts points when SKILL.md lacks an expected heading. The packaged quality-default.yaml expects ## Prerequisites (or ## Requirements), ## Limitations, and ## Troubleshooting (or ## Common Issues or ## FAQ), each worth 5 points. Pass --quality-config PATH to quality-check, tier1, or validate to use your own list instead. The file replaces the packaged defaults rather than merging with them, so copy any default entries you want to keep.

expected_headers:
  - heading: "## Prerequisites"
    alternatives:
      - "## Requirements"
    points: 5
    message: "No prerequisites/requirements documented"
    suggestion: "Document dependencies, API keys, or setup needed"
  - heading: "## Security"
    points: 8
    message: "No security notes documented"
    suggestion: "Add '## Security' describing permissions and data access"
  - heading: "## Examples"

skillevaluator quality-check ./my-skill --quality-config quality.yaml
skillevaluator tier1 ./my-skill --quality-config quality.yaml
skillevaluator validate ./my-skill --quality-config quality.yaml

The packaged defaults set message and suggestion explicitly, so their findings keep the original wording.

An entry is satisfied when the heading or any alternative appears anywhere in SKILL.md as text, so ## Security also matches ### Security Notes or a heading inside a code block.

The file is checked before any validation runs. A file that is missing, unreadable, not UTF-8, invalid YAML, has an unknown key (such as point instead of points), or lists the same heading twice is a usage error: the command exits with code 2 and names the file and the problem. validate checks the file even when --checks leaves out quality.

Error: Invalid value for '--quality-config': quality.yaml: unknown key(s) in expected_headers[0]: point (allowed: alternatives, heading, message, points, suggestion)

rubric-eval sends bounded skill documentation and selected supplementary content to your configured LLM provider. It scores nine criteria:

Each criterion is scored from 0 to 10. The local evaluator, not the model’s boolean, treats 7 or higher as passing. The overall score is an importance-weighted mean, and the rubric passes only when that score reaches --min-score and every criterion passes.

skillevaluator rubric-eval ./my-skill --min-score 70

lint-scripts, and the lint check inside validate, parse Python scripts under scripts/ and report advisory findings. These findings do not make the Tier 1 gate fail.

The default version check validates an optional metadata.version as a three-part numeric value in major.minor.patch form:

metadata:
  version: "1.2.3"

When --previous-version or SKILLEVALUATOR_PREVIOUS_VERSION is set, the current version must be present and numerically greater than the previous value. Without a previous-version bound, the version remains optional and the skill can rely on commit history. A previous-version bound is valid only when checking one skill; catalog validation must check each skill separately with its own bound.

skillevaluator validate ./my-skill --previous-version 1.2.2

Scans requirements*.txt for known Python package vulnerabilities. For pyproject.toml, reads static PEP 621 dependencies and every optional group, then audits their exact version pins without installing the target or executing its build scripts. This covers direct declarations only; transitive dependencies require a resolved requirements file. Markers are evaluated against the scanner’s Python/platform and optional-group context; nonmatching declarations are reported as skipped. Version ranges, URL/path dependencies, dynamic or Poetry metadata, malformed declarations, and missing or failed scanner evidence leave the audit INCOMPLETE. Supply pinned requirements.txt for unresolved declarations. pip-audit is the primary scanner; Safety supplies secondary coverage for requirements files when installed separately. The audit may use network-backed vulnerability databases. JSON and HTML export data retain dependency_coverage per target, including requested and actually audited pins. Unreadable declarations have unknown declaration counts and an empty audited inventory, not a completed scan.

skillevaluator validate ./my-skill --checks dependency

Install it with brew install gitleaks or download a binary from the official Gitleaks releases.

Install Bandit and pip-audit with skillevaluator[security]. Install Semgrep separately with brew install semgrep or uv tool install semgrep, and SkillSpector with uv tool install git+https://github.com/NVIDIA/SkillSpector.git@v2.13.3. Required scanner failures remain INCOMPLETE until the scanner is installed and returns valid evidence. skillevaluator doctor --tier 1 lists every scanner the selected checks need, with its resolved path and version.

SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a static_parse_limit, obfuscated_instruction_text, or runtime_limit ledger exception. The security check is then INCOMPLETE, often with a High Incomplete referenced artifact analysis (AE1) finding at each line that names the helper. Check the installed release with skillspector --version and upgrade with uv tool install git+https://github.com/NVIDIA/SkillSpector.git@v2.13.3. If the exception remains after the upgrade, investigate it like any other incomplete scan.

Each SkillSpector run (the static stage, and the LLM stage with --llm) is stopped after 1,200 seconds by default, and the security scan is reported INCOMPLETE with skillspector timed out after 1200 seconds. SkillSpector also ends a scan at its own deadline (1,080 seconds by default in v2.13.3) and reports partial analysis, which is INCOMPLETE as well. Slow or local LLM providers can need more time. Raise both limits, keeping SkillEvaluator’s watchdog higher:

export SKILLSPECTOR_MAX_WORKFLOW_SECONDS=1800
export SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS=2400
skillevaluator validate ./my-skill --tiers 1 --llm

Each stage gets the full watchdog, so a scan that stalls in both stages can take up to twice the value before it is reported. See Environment Variables.

Configure a supported provider or select a keyless Tier 1 run with skillevaluator tier1 ./my-skill --no-llm or skillevaluator validate ./my-skill --tiers 1 --no-llm (without --llm-verify). Direct Tier 1 automatically enables LLM stages when a provider is configured; --no-llm explicitly disables them. Top-level validate --no-dedup skips Tier 2 but still runs Tier 3 by default.

Confirm that the skill contains supported text or code extensions such as .py, .sh, .yaml, .yml, .json, .md, or .txt.

Use an existing, executable absolute path for the relevant SKILLEVALUATOR_*_PATH variable.

When SkillSpector reports partial analysis, the security scan is INCOMPLETE. The error quotes up to three of SkillSpector’s completeness exceptions as path [phase; reason_code; analyzers=...]. When there are more, it adds how many files each reason affected, for example and 463 more (incomplete files by reason: runtime_limit 462, obfuscated_instruction_text 4).

For runtime_limit, the error also says what to do, with one piece of advice for each kind of row the report contains:

• Scan time budget. Rows that name regular analyzers, or no analyzer, mean SkillSpector ran out of time for the bundle: at its own whole-scan deadline (1,080 seconds by default in v2.13.3) or at a shorter stage budget such as bundle discovery. A skill that bundles a large corpus, such as vendored documents, generated figures or renderer scripts, hits the same limit on every rerun. Keep large reference material outside the skill and point to it. A plain rerun helps only when a few files are affected on a busy machine. If SkillEvaluator’s own SkillSpector timeout stops the run before SkillSpector’s deadline, the error is skillspector timed out after N seconds instead, with no list of files.
• Sub-analysis budget. Analyzers named <analyzer>_runtime_limit, such as static_patterns_supply_chain_bytecode_runtime_limit, stopped at a time budget of their own, so bundle size does not matter. Releases before v2.13.0 charge the SC8 bytecode walk for time spent waiting behind other analyzers, so it can stop even on a skill with no bytecode. Rerun the scan, and update SkillSpector if the limit repeats.