> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/skills/skillevaluator/_mcp/server. # Tier 2: Semantic Deduplication > Detect redundant content within one skill and overlapping skills across a collection using embedding similarity. Tier 2 finds redundant content — inside a single skill and across a skill collection — using embedding similarity, with chat-LLM verification for intra-skill candidates. It needs a configured embeddings provider; intra-skill analysis also needs a chat LLM. See [Providers & Credentials](/skills/skillevaluator/configuration) to set both up, including the [fully local recipe](/skills/skillevaluator/configuration#fully-local-setup) when skill content must not leave your machine. Both checks read local files only. No external vector database or catalog service is required. ## Which check do you need #### [Is my skill repeating itself?](#detect-redundancy-inside-one-skill) **Intra-skill deduplication** — `context-optimization-check` walks one skill directory and flags sections that restate each other, so the skill stops burning context on duplicate guidance. #### [Does my skill overlap the collection?](#compare-skills-across-a-collection) **Inter-skill similarity** — `similarity-check` compares skills across a collection, or one candidate against a saved local catalog, and classifies each overlapping pair. `dedup-scan` is an alias for `context-optimization-check` with the same analysis options. The default report basenames identify which command name was invoked. The two checks differ in what they call out to: | | Intra-skill (`context-optimization-check`) | Inter-skill (`similarity-check`) | | --------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | **Scope** | Within one skill directory | Across a collection, or one candidate against a local catalog | | **Needs** | Embeddings provider **and** a chat LLM | Embeddings provider only — no chat LLM is ever called | | **Comparison** | Pairwise cosine similarity, then Union-Find clustering, then LLM verification | Pairwise cosine similarity, or one target compared with catalog entries | | **Embedding storage** | Ephemeral, in memory per run | Ephemeral for a direct scan, or a versioned local JSON catalog | ## Detect redundancy inside one skill The repository's exact SPDX-only HTML preamble is excluded from Markdown chunks, so required license metadata cannot create duplicate-content findings. Comments with any additional text or directive remain part of the comparison. As a skill grows, its directory accumulates repeated content: sections copied between `SKILL.md` and reference documents, near-identical guidance under different headings, script docstrings that restate the main instructions. That repetition consumes context without adding information. Not every similarity is redundant. A short overview in `SKILL.md` and a detailed explanation in `references/` are intentional progressive disclosure. Tier 2 therefore runs two stages — embeddings find likely overlap, then a chat LLM classifies its purpose: | Stage | What runs | What it produces | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | | **1. Embedding clustering** | Chunks are batch-embedded with the configured embeddings provider; pairs at or above the threshold are grouped with Union-Find | Candidate clusters of overlapping chunks | | **2. LLM verification** | The actual text of each candidate cluster goes to the configured provider model for classification | A verdict, a confidence score, reasoning, and a suggested author action | ```bash title="Scan one skill for repeated content" skillevaluator context-optimization-check ./my-skill ``` Useful options: | Flag | Default | Effect | | ------------- | -------------------------- | ----------------------------------------------------------------- | | `--threshold` | `0.8` | Minimum cosine score for a chunk pair to join a candidate cluster | | `--model` | configured embedding model | Embedding model override for this run | | `--llm-model` | configured provider model | Chat-LLM model override for the verification stage | The default intra-skill threshold (`0.80`) is higher than the inter-skill default because sections from the same skill naturally share more domain vocabulary. ### What gets chunked | Extension | Chunking strategy | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.md`, `.mdc` | Heading-based sections; sections over 3,000 characters are split at eligible paragraph boundaries when possible; chunks under 80 characters are dropped | | `.py` | Module docstrings plus class and function signatures with their docstrings; implementation bodies are not included | | `.sh` | Function bodies and comment blocks of at least three consecutive comment lines | | Other extensions | Skipped | Valid mapping frontmatter is removed from Markdown before chunking. Malformed, empty, or non-mapping frontmatter remains part of the analyzed text. Every retained chunk keeps its source file and line range, so findings point you to the exact overlapping sections. ### Verdicts and what to do about them | Verdict | Meaning | Your action | | ---------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------- | | `DUPLICATE` | Content repeats the same information without a meaningful addition | Consolidate into one location; the finding is actionable | | `INTENTIONAL_DETAIL` | One section summarizes content another section develops in detail | Keep it — progressive disclosure is not reported as a finding | | `RELATED_BUT_DISTINCT` | Sections cover the same topic for different purposes or from different angles | Keep it — not reported as a finding | Duplicate findings map to severities by confidence: | Condition | Severity | | --------------------------------------------------------------------------- | ------------------------ | | `DUPLICATE` with confidence >= 0.70 | HIGH — blocking | | `DUPLICATE` with confidence \< 0.70 | MEDIUM — advisory | | Short, same-file duplicate dominated by comment or configuration-style text | Capped at LOW — advisory | If chat-LLM analysis fails for a candidate cluster, Tier 2 reports the incomplete analysis as a blocking error rather than treating the content as clean. ## Compare skills across a collection `similarity-check` recursively discovers skill manifests, embeds each skill's name and description by default, and reports pairs at or above the selected threshold. ```bash title="Scan a collection for overlapping skills" skillevaluator similarity-check ./skills ``` Every match lands in one of five classifications: | Classification | Score | Severity and default behavior | | ----------------- | ------- | ------------------------------------------------------ | | `EXACT_DUPLICATE` | >= 0.95 | CRITICAL — blocking | | `HIGH_SIMILARITY` | >= 0.90 | HIGH — blocking | | `SIMILAR` | >= 0.75 | MEDIUM — advisory | | `LOOSELY_RELATED` | >= 0.50 | LOW — advisory | | `DISTINCT` | \< 0.50 | INFO — advisory; below the default reporting threshold | The threshold controls which score bands are returned; it does not change the classification boundaries. The default (`0.75`) reports `SIMILAR` and stronger matches. | Flag | Default | Effect | | -------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--threshold` | `0.75` | Minimum reported cosine score | | `--full-body` | off | Embed each discovered `SKILL.md` in full, including frontmatter, chunked at 512 tokens with 64-token overlap and average pooling — instead of only the name and description. Supporting files are not included | | `--model` | configured embedding model | Embedding model override for this run | | `--type skill` | `auto` | Explicit skill discovery when automatic detection is not suitable | ## Use a local catalog For repeated screening — say, every new skill submission against your team's existing set — save the collection once and query candidates against it. #### Build the catalog ```bash title="Save a reusable local catalog" skillevaluator similarity-check ./skills --save-catalog ./skill-catalog.json ``` `--save-catalog` builds a versioned JSON catalog from the supplied collection. #### Screen a candidate ```bash title="Compare one candidate against the catalog" skillevaluator similarity-check ./candidate-skill --catalog ./skill-catalog.json ``` `--catalog` requires an existing catalog and a candidate directory containing a root `SKILL.md`. It compares only that candidate against the catalog entries — it does not compare catalog entries with one another, and it never silently rebuilds a missing catalog. Catalogs contain finite embedding vectors, relative skill paths, display names, descriptions, content fingerprints, and provider, model, mode, and endpoint-fingerprint metadata for compatibility checks. They never contain API keys. The provider, model, endpoint, vector length, and description-versus-full-body mode used for a query must be compatible with the catalog. A missing, malformed, incompatible, non-finite, or oversized catalog is rejected with an actionable error — query with matching settings, or rebuild the catalog with the settings you intend to use. Catalogs stay local unless you explicitly share them. Because they contain embeddings and other data derived from skill content, review them before sharing like any other generated project artifact. ## Inside validate `validate` runs the intra-skill deduplication pass by default as part of a full evaluation. When the optional Tier 2 dependencies or an embeddings provider are unavailable, the pass skips gracefully with a warning — a keyless `validate` still completes. Use `--no-tier2` (or `--no-dedup`) to disable the pass explicitly: ```bash title="Skip the Tier 2 pass" skillevaluator validate ./my-skill --no-dedup ``` All Tier 2 commands accept the standard report options (`-r cli,json,html,markdown` and `-o`). HIGH and CRITICAL findings, along with blocking analysis errors, produce a non-zero exit code; MEDIUM and LOW findings are advisory. Exit behavior is identical in every output mode, so these commands work as CI gates — see [Gate Your CI](/skills/skillevaluator/ci-integration). ## Defaults and safety limits | Setting | Default | Purpose | | ---------------------------------- | ---------------------------- | ------------------------------------------------------------- | | Inter-skill reporting threshold | `0.75` | Reports `SIMILAR` and stronger matches | | Intra-skill clustering threshold | `0.80` | Selects candidate chunk pairs for clustering | | Full-body chunking | 512 tokens, 64-token overlap | Chunked average pooling for `--full-body` embeddings | | Minimum content chunk | 80 characters | Drops fragments too small to compare usefully | | Large Markdown section split point | 3,000 characters | Splits oversized sections at eligible paragraph boundaries | | High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM | Tier 2 treats skill content and catalog files as untrusted input. Before any provider call, it rejects linked roots, linked or escaping files, non-regular files, and paths outside the verified scan root. Explicit limits bound discovered paths, file counts, per-file and total input bytes, content chunks, candidate clusters, catalog size and entry count, vector length, returned matches, and pairwise scalar work. Catalog loading also rejects duplicate JSON keys, unexpected fields, invalid identities, non-finite vectors, and incompatible provider metadata. If a collection exceeds a limit, split it into intentional batches rather than scanning an unbounded directory tree. ### What leaves your machine * A default inter-skill scan sends skill names and descriptions to the configured embeddings provider. With `--full-body`, the scan sends each discovered `SKILL.md` in full, including frontmatter, in embedding chunks; supporting files are not sent. * Intra-skill analysis sends retained content chunks to the configured embeddings provider. Only candidate clusters found by the embedding stage are sent to the configured chat LLM for classification. * `similarity-check` never calls a chat LLM. * Catalog files are read and written locally; SkillEvaluator does not upload them to a catalog service. Use the [fully local provider recipe](/skills/skillevaluator/configuration#fully-local-setup) when skill content must not leave the machine, and review your provider's data handling terms before sending confidential skill content to a hosted endpoint. ### Edge cases #### Intra-skill analysis | Scenario | Behavior | | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Skill contains only `SKILL.md` | Checks repeated sections within that file | | Fewer than two chunks remain | Succeeds with "Not enough content to compare" | | Markdown has no headings | Treats the body as a preamble and attempts paragraph splitting when oversized; retains the original section if no eligible split is produced | | Unsupported extension | Skips the file | | Duplicate sections occur in one file | Reports the relevant headings and line ranges | | Skill carries large generated snapshots (`evals`, `.evals`, `results`, `.results`, `versions`, `.versions`) | Excluded directories are pruned during traversal, so their contents never count toward the discovered-path limit — a well-used skill with large evals or results snapshots no longer trips it, as it could in earlier releases | | A directory is a Windows junction or other reparse point | Rejected during the walk, like symbolic links | | Embedding or chat-LLM analysis is incomplete | Reports an error; does not claim a clean result | #### Inter-skill analysis | Scenario | Behavior | | --------------------------------------------------------- | --------------------------------------------------------------- | | Direct collection has fewer than two skills | Returns an actionable input error | | Catalog source collection is empty | Refuses to save an empty catalog | | Catalog path is missing | Returns an error and does not rebuild it silently | | Candidate lacks a root `SKILL.md` | Rejects the catalog query | | Candidate matches multiple catalog entries | Reports matches in descending similarity order | | Provider, endpoint, model, mode, or vector length differs | Rejects the catalog; query with matching settings or rebuild it | | Catalog is malformed, non-finite, or oversized | Rejects it before comparison | ## Next steps #### [Tier 3: Live Evaluation](/skills/skillevaluator/tier3-live-evaluation) Prove the skill changes agent behavior with a live A/B evaluation. #### [Providers & Credentials](/skills/skillevaluator/configuration) Configure an embeddings provider — including a fully local setup. #### [CLI Reference](/skills/skillevaluator/cli-reference) Every Tier 2 flag, default, and report option.