CLI Reference

This page lists every skillevaluator command with its full flag set and exact defaults. If you want task-oriented guidance instead, start from the tier guides: Tier 1: Validation, Tier 2: Deduplication, and Tier 3: Live Evaluation.

Harbor is the open-source agent evaluation framework behind the Tier 3 task, environment, trial, and trajectory commands referenced below.

Command map

Global conventions

Reports. Every Tier 1 and Tier 2 command accepts the same two report options. Direct tier1 PATH, tier1 validate PATH, and tier2 PATH default to cli,json,html; focused expert commands default to cli:

In practice: a default validate run (no -r) shows the compact pipeline view and writes json + html report files; passing -r takes over completely — -r cli really means terminal output only, no files. The focused expert commands take the cli default literally: they print the full terminal report and write files only for the formats you request with -r. A validate run writes skillevaluator-output-<timestamp>.json / .html (plus .md with -r markdown or .sarif.json with -r sarif); standalone commands write a fixed skillevaluator-<kind> basename (for example skillevaluator-quality.json), so a re-run overwrites the previous report. Direct workflows write skillevaluator-tier1 or skillevaluator-tier2 reports in the selected formats and also overwrite previous files with that basename. Filename conventions and report anatomy live in Reports & Results.

Exit codes. Commands communicate through a small exit-code contract — this is what your CI gates on (see Gate Your CI):

Global options. skillevaluator --version prints the installed version. -v, --verbose, given before the command (skillevaluator -v validate ./my-skill), enables debug logging; for validate it also replaces the compact pipeline view with the full detail stream. Every command accepts -h as a short form of --help.

Direct tier workflows

skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill --catalog ./skill-catalog.json
skillevaluator tier3 ./my-skill

There is no required run subcommand. A bare tier1, tier2, or tier3 shows help and does not evaluate the current directory. Prefix a path with ./ when its name matches an expert command: tier1 ./validate evaluates a directory named validate; tier1 validate ./my-skill invokes the Tier 1 workflow.

Direct Tier 1 can run keyless, but marks its LLM stages as skipped. Missing required scanner evidence remains incomplete. Direct Tier 2 needs both chat and embedding access and takes one skill, not a collection. Create its catalog with similarity-check ./skills --save-catalog ./skill-catalog.json. Direct Tier 3 preserves valid existing sources and rejects invalid ones; it does not overwrite a dataset to repair it.

validate PATH runs all three tiers for skills by default, including Tier 3 autopilot. It is separate from these tier-only workflows.

Direct tiers compared with validate --tiers

validate --tiers narrows the combined pipeline; it does not run a tier on its own. Tier 1 always runs and gates the exit code, so --tiers 2, --tiers 3, and --tiers 2,3 are rejected. Use tier2 PATH or tier3 PATH to run Tier 2 or Tier 3 without Tier 1.

For tier2 PATH, tier3 PATH, and tier1 PATH --llm, a missing provider is a configuration error that exits 1 before any report is written. tier1 PATH without --llm runs keyless. validate records Tier 2 and Tier 3 as skipped and still writes its reports.

Direct Tier 3 checks provider, option, and runtime configuration before it generates a starter case, so a configuration error leaves evals/ unchanged and no paid generation or agent run starts. The exception is --env-mode ack, cwsandbox, langsmith, openshift, and wandb: their backend checks contact a remote control plane, so they run after the starter case is generated. Usage errors, such as an option the workflow does not accept, exit 2.

Direct Tier 1 options

Also accepts -r/--report and -o/--output-dir described above.

Direct Tier 2 options

Both thresholds accept values from 0 to 1. Also accepts -r/--report and -o/--output-dir. Direct Tier 3 accepts the options in tier3 evaluate, with missing-source generation already enabled.

Expert tier subcommands

The commands below remain available for focused operations. skillevaluator tier1 validate PATH runs the same isolated Tier 1 workflow as tier1 PATH. Use top-level skillevaluator validate PATH for the combined pipeline.

health-check and models are top-level only. The expert commands tier3 evaluate, tier3 select-decoys, and tier3 validate are advertised only inside the tier3 group:

tier3 validate

Validate the Tier 3 evals/ directory and the optional Harbor bring-your-own-task contract before you spend money on a live run. The dataset and directory rules it enforces are documented in Eval Datasets.

skillevaluator tier3 validate ./my-skill --strict

validate

Validate a skill, rule, workflow, or plugin. For skills, all three tiers run by default. Tier 3 uses autopilot to reuse an existing evaluation source or generate one starter case when none exists; invalid sources are reported rather than replaced. Tier 1 static, security, and quality checks always gate the exit code. Tier 2 deduplication runs and gates by default, but --no-block-on-dedup keeps the scan while making its findings advisory. Tier 3 is advisory unless --block-on-agent-eval is set. Rules, workflows, and plugins retain their applicable validation stages without automatically enabling live skill evaluation.

skillevaluator validate ./my-skill                        # Tier 1 + Tier 2 + Tier 3 autopilot
skillevaluator validate ./my-skill --llm                  # add LLM security analysis
skillevaluator validate ./my-skill -r cli,json,html       # multiple report formats
skillevaluator validate ./my-skill --no-dedup             # skip Tier 2 dedup
skillevaluator validate ./my-skill --no-block-on-dedup    # run Tier 2 as advisory
skillevaluator validate ./my-skill --external             # strict publication profile
skillevaluator validate ./my-skill -a codex               # choose the Tier 3 agent
skillevaluator validate ./my-skill -a codex \
    --block-on-agent-eval                                  # gate on Tier 3 too
skillevaluator validate ./my-skill --no-autopilot         # require an existing eval source
skillevaluator validate ./my-skill --full -a codex        # compatibility: same default scope
skillevaluator validate ./my-skill --tiers 1,2            # skip live evaluation
skillevaluator validate ./my-skill --tiers 1              # static checks only
skillevaluator validate ./my-skill --tiers 1,3            # explicit tier selection
skillevaluator validate ./skills-folder                   # whole catalog, serially
skillevaluator validate ./my-skill -a codex,claude-code \
    --env-mode docker --harbor-keep-jobs                  # Tier 3, retain Harbor jobs

Pointing validate at a directory of skills (one with no root SKILL.md) runs the whole pipeline for each skill inside it, writes each skill’s reports under <output-dir>/<skill>/, and finishes with a per-skill summary — catalog validation in one command. Skills run serially by default; --workers N validates up to N skills at once in parallel processes. --previous-version is rejected for a directory of skills. By default the terminal shows a compact pipeline view; --verbose switches to the full per-check detail stream.

--help groups the flags into four panels, mirrored by the subsections below.

Run & Reports

Applies to the whole run: target typing, policy profile, reports, tier selection.

The evaluated source provenance flags also appear in this panel.

Auto-detection recognizes: a SKILL.md for skills, .mdc files for rules, workflow-rules.mdc for workflows, and — for plugins — a bundle-reference agent_plugin.yaml/.yml manifest or a contained .claude-plugin/plugin.json manifest. Plugins are validated against their public contract; quality, lint, and version checks are skill-only and skipped for plugins.

validate also takes the standard -r/--report and -o/--output-dir options described under Global conventions.

Tier 1 · Static & Security

Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.

Tier 2 · Deduplication

Embedding + LLM dedup; on by default, skips gracefully without a provider key.

Tier 3 · Live Agent Evaluation

Flags marked “unset” fall back to their matching keys in evals/config.yml where one exists, then to the built-in defaults (Eval Datasets documents the config keys and precedence). The exception is --results-dir, which resolves independently of evals/config.yml (see Reports & Results).

Evaluated source provenance

The identity is recorded on BENCHMARK.md, forwarded to every child of a parallel catalog run so each card names the same source tree, and persisted into a Tier 3 run’s run_config.json (see Reports & Results).

quality-check

Score skill quality across correctness, discoverability, reliability, and efficiency. Runs entirely offline — no API key.

skillevaluator quality-check ./my-skill

Plus the standard report options. A collection’s displayed average does not replace the minimum required of each skill. Scoring weights, grade bands, the independent minimum-score gate, and the --quality-config file format are documented in Tier 1: Validation.

rubric-eval

Run LLM-as-judge rubric evaluation for a skill against nine weighted criteria. Requires a configured LLM provider.

skillevaluator rubric-eval ./my-skill

Plus the standard report options.

security-scan

Scan for security vulnerabilities using the static scanner stack. Keyless by default; the LLM flags require a configured provider.

skillevaluator security-scan ./my-skill

Plus the standard report options.

pii-scan

Scan for PII and local identifiers (emails, credentials, home paths, and more).

skillevaluator pii-scan ./my-skill

Plus the standard report options.

lint-scripts

Run advisory lint checks on skill scripts. Takes only the standard report options.

skillevaluator lint-scripts ./my-skill

context-optimization-check

Detect redundant content within one skill (intra-skill deduplication). Needs an embeddings-capable provider plus a chat LLM for the verification stage — see Tier 2: Deduplication.

skillevaluator context-optimization-check ./my-skill

Plus the standard report options.

dedup-scan

Alias of context-optimization-check — same behavior, same flags (--threshold default 0.8, --model, --llm-model, plus the standard report options). Only the report basename differs: skillevaluator-dedup instead of skillevaluator-context.

skillevaluator dedup-scan ./my-skill

similarity-check

Detect duplicate content across a collection with embedding similarity (inter-skill deduplication). Takes a content path — a single item or a folder of them.

skillevaluator similarity-check ./skills

Plus the standard report options. Classification tiers and catalog rules are in Tier 2: Deduplication.

For older scripts, the hidden deprecated aliases --cache and --save-cache still map to --catalog and --save-catalog, respectively. Use the catalog spellings for new commands. A catalog query and a catalog build cannot be combined in one invocation, and each alias cannot be supplied alongside its replacement.

tier3 select-decoys

Choose neighboring skills from a local description-embedding catalog and save a frozen JSON manifest for group evaluation. This command makes an embedding request for the target; it does not launch an agent or modify the run policy.

skillevaluator tier3 select-decoys ./skills/my-skill \
  --catalog ./skill-catalog.json --catalog-root ./skills \
  --count 4 --output ./skills/my-skill/evals/decoys.json

The manifest records the catalog hash and embedding identity, selected paths and scores, target manifest hash, and selected runtime-content hashes. The target’s own collection uses a root relative to its parent directory, so an identical checkout can reuse the manifest. The hashes cover everything staging copies from a selected skill, including ignored or untracked files and empty folders, so generate a manifest you plan to commit from a fresh clone. Unrelated external collections retain an absolute root. Evaluator-only files are excluded from decoy hashes; changing a decoy’s evaluation config does not require a new selection. Output cannot overwrite evaluation inputs or an existing non-manifest file. Set skill_workspace.mode: group and skill_workspace.decoy_manifest in evals/config.yml to consume it. The manifest path is relative to the target skill’s parent directory. Evaluation validates and reuses the selection without contacting the embedding provider. See Select a reproducible decoy set for a complete example and the limits of similarity-based selection.

tier3 evaluate

Run Tier 3 live agent evaluation: real agents attempt your eval cases with and without the skill, and the difference becomes Skill Lift. Requires the tier3 extra, a provider key for grading, a credential the selected agents can use — with NVIDIA Build in --env-mode docker, kata, or local, the one NVIDIA_API_KEY covers both roles — and a running backend (Docker by default). The full walkthrough is Tier 3: Live Evaluation.

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker

Without a --model/--agent-model override, agent execution uses gpt-6.1-sol for OpenAI (OpenCode: openai/gpt-5.6-sol unless SKILL_EVAL_LLM_MODEL is set), claude-opus-5-5 for Anthropic, and nvidia/nemotron-3-super-120b-a12b for NVIDIA Build. OpenCode displays the last value as nvidia/nvidia/nemotron-3-super-120b-a12b because its adapter namespace precedes the NVIDIA catalog ID.

Flags marked “unset” fall back to their matching keys in evals/config.yml where one exists, then to the built-in defaults. The exception is --results-dir, which resolves independently of evals/config.yml.

create-eval-dataset

Create synthetic eval datasets for agent skill evaluation, written to evals/evals.json. Uses the configured LLM provider unless you pass --no-llm. Dataset anatomy and the four case buckets are documented in Eval Datasets.

skillevaluator create-eval-dataset ./my-skill --full

init-custom-grader

Create a bring-your-own-grader (BYOG) starter under evals/. The grader contract it scaffolds is documented in Custom Graders & Tasks.

skillevaluator init-custom-grader ./my-skill

init-harbor-task

Create a bring-your-own-task (BYOT) Harbor starter template under evals/harbor/. See Custom Graders & Tasks for how case IDs map to eval entries.

skillevaluator init-harbor-task ./my-skill

view

Open the latest HTML live-evaluation report for a skill. If the results exist but report.html is missing, view regenerates it before opening.

skillevaluator view ./my-skill

compare

Compare live-evaluation results across agents from previous runs.

skillevaluator compare ./my-skill

dashboard

Open the optional local tables dashboard. Install the dashboard extra first, then pass JSON reports, retained run directories, or directories containing reports. With no paths, upload JSON files in the browser.

skillevaluator dashboard reports/run-a.json reports/run-b.json
skillevaluator dashboard --no-browser --port 8502

The server listens on 127.0.0.1; Ctrl+C stops it. See the dashboard guide for measurements and comparison eligibility.

harbor-view

Open retained Harbor job artifacts with Harbor’s trajectory browser. Takes a jobs directory as its argument and has no options. Jobs are only retained when the original run passed --harbor-keep-jobs. skillevaluator tier3 harbor-view is an identical tier-prefixed spelling.

skillevaluator harbor-view ./path/to/jobs-dir

doctor

Check live-evaluation runtime readiness: the CLI install, the configured provider, the requested agents and their credentials, and the selected backend. Run it before your first tier3 evaluate — see Agents & Sandboxes for what each check means.

skillevaluator doctor --env-mode docker

--tier selects other readiness checks. Without it, doctor checks Tier 3 only, as above.

skillevaluator doctor --tier 1 --checks security,code-integrity,dependency
skillevaluator doctor --tier 2 --verify-models
skillevaluator doctor --tier all

• Tier 1 reports each external scanner the selected checks run: SkillSpector for security, Bandit, Semgrep and Gitleaks for code-integrity, and pip-audit for dependency. Each row shows the resolved executable, the override variable when one selected it, and the version from a bounded version probe. The probe never starts a shell. A scanner fails when it is missing, its override is malformed, or the probe exits nonzero or times out. It also fails when it is older than the release the validators parse: Bandit 1.9.4 or Gitleaks 8. An unreported version is a warning.
• Tier 2 checks that the embedding provider and the LLM provider used for duplicate analysis resolve, without a network request. With --verify-models, it also sends one embedding request and one chat request, each bounded by --timeout and sent without retries. Failures show only the HTTP status and a remedy, never the provider’s response body.
• all runs Tiers 1, 2 and 3 in order. A failed tier does not stop the next, and the exit code is nonzero when any tier fails.

An option for a tier that is not selected exits with a usage error before any check runs. For example, --checks without --tier 1 or all is an error, and so is --timeout with --tier 2 but no --verify-models, because offline Tier 2 sends no request.

-a, --env-mode, --environment-kwarg and --agent-model apply to Tier 3 only. skillevaluator tier3 doctor checks only Tier 3. It takes the Tier 3 flags, including --agent-model and --verify-models, and has no --tier, --checks or --timeout.

health-check

Quick readiness check for the CLI and the selected live-eval backend. Use it as an install smoke test or a fast CI preflight. It differs from doctor by two flags: health-check has no --agent-model and no --verify-models, so it never sends a live provider request. --tier and --checks work as in doctor, and --timeout bounds only the Tier 1 scanner probes.

skillevaluator health-check
skillevaluator health-check --tier 1    # CI preflight for the Tier 1 scanners

models

List models visible through the selected provider’s catalog. It uses the same provider and credential resolution as evaluation (see Providers & Credentials), so it checks catalog reachability and model membership. NVIDIA Build and some compatible gateways expose public catalogs, so a listed model does not prove that the inference key works or that the model is compatible with the evaluation harness. doctor --verify-models reports such successful but non-authoritative checks as warnings. Top-level only; there is no tier-prefixed twin.

skillevaluator models --limit 20

See also

• Environment Variables — every SKILL_EVAL_* and SKILLEVALUATOR_* variable
• Gate Your CI — exit codes and JSON reports as a merge gate
• Providers & Credentials — which commands need which key