Skip to navigation

Harness Check

View as MarkdownOpen in Claude

No LLM. No agent run. No API key. In about a second, the harness check predicts whether each agent harness loads your skill, degrades it, or drops it, and what to change.

skillevaluator harness-check ./my-skill

Results use versioned harness profiles. Add -v to see their verified versions and dates, or --harness-version HARNESS=VERSION to flag a different release. See the CLI reference for all options.

A skill can follow the Agent Skills specification and still never reach the model. Every harness has its own loader, with its own YAML parser, folders and size limits. The same SKILL.md can work in Claude Code and be dropped by Codex without an error. The harness check finds those differences before your users do.

Static

It reads the files on disk. No model, no agent, no sandbox, no key.

Six harnesses

Claude Code, Codex, Copilot CLI, Hermes, OpenClaw and OpenCode in one run.

Fixes included

Every finding names the rule, what the harness does, and the change that fixes it.

What you get

This example skill has two small problems. The front matter has license twice, and the body uses $ARGUMENTS.

skillevaluator harness-check ./my-skill
╭─ Harness compatibility · my-skill ───────────────────────────────────────────────╮
│ ✓ claude-code ✗ codex ✗ copilot-cli ! hermes ✗ openclaw ✗ opencode │
│ │
│ ✓ claude-code loads 1 note │
│ ✗ codex NOT LOADED Skill dropped: invalid YAML (tabs +1 more │
│ or duplicate keys). │
│ ✗ copilot-cli NOT LOADED Skill dropped: Copilot's YAML +1 more, 1 note │
│ parsing is strict. │
│ ! hermes degraded $ARGUMENTS is not substituted: the 1 note │
│ model sees the literal text. │
│ ✗ openclaw rejected Skill dropped: invalid YAML (tabs or +1 more │
│ duplicate keys). │
│ ✗ opencode rejected Skill dropped: duplicate frontmatter +1 more, 1 note │
│ keys. │
│ │
│ Doesn't change pass/fail. Details and fixes: -v │
╰──────────────────────────────────────────────────────────── ✗ 4 of 6 not loaded ─╯
Tier 1 schema also fails for my-skill: [SCHEMA-HIGH] Duplicate frontmatter key 'license': YAML parsers disagree on it (Codex, OpenCode, Copilot CLI and OpenClaw drop the skill; others keep the last value). Run `skillevaluator validate --tiers 1` for details.

Read it from the top:

  • The first row has one mark per harness, so you see the result at a glance.
  • Each line below is one harness: its verdict and its most serious finding. +1 more means it has more findings. 1 note counts notes that don’t change the verdict. A harness with nothing to report gets no line.
  • The bottom corner is the tally, here 4 of 6 not loaded.
  • The line under the panel appears when the same problem also fails a Tier 1 schema check, and points you to validate --tiers 1.

Each harness gets one of three verdicts:

MarkVerdictWhat it means
✓loadsNo loading or degradation rule matched. Read any notes: another installed copy may take precedence.
!degradedThe skill loads, but something is cut, hidden or ignored.
✗not_loadedThe harness drops the skill. The model never sees it.

The compact view labels a finding with its outcome when the rule names one, such as not found, rejected, listing shortened or delivery truncated.

See every finding and its fix

Add -v for the detailed view. --harness narrows it to the harnesses you care about.

skillevaluator harness-check ./my-skill --harness codex,hermes -v
╭─ Harness compatibility · my-skill ───────────────────────────────────────────────╮
│ ✗ codex ! hermes │
│ │
│ ✗ codex NOT LOADED verified 0.166.0, 2026-11-01 │
│ ✗ NOT LOADED Skill dropped: invalid YAML (tabs or duplicate keys). │
│ [CX-STRICT-YAML] │
│ → Use spaces, not tabs, and remove duplicate keys. │
│ confidence: high │
│ ! degraded $ARGUMENTS is not substituted: the model sees the │
│ literal text. [CX-ARGUMENTS] │
│ → Describe the expected input in words instead of │
│ $ARGUMENTS. │
│ confidence: medium │
│ ! hermes degraded verified main @ c419cf7c864a (binary reports │
│ 2026.9.24), 2026-11-01 │
│ · note Duplicate keys: last value wins. [HM-DUP-KEY] │
│ → Remove the duplicate frontmatter key. │
│ confidence: high │
│ ! degraded $ARGUMENTS is not substituted: the model sees the │
│ literal text. [HM-ARGUMENTS] │
│ → Describe the expected input in words instead of │
│ $ARGUMENTS. │
│ confidence: high │
│ │
│ Description in the model's skill list (from the profile): │
│ hermes cut to 60 chars “Write release notes from merged pull requests. Use │
│ when t…” │
│ │
│ Doesn't change pass/fail. │
╰──────────────────────────────────────────────────────────── ✗ 1 of 2 not loaded ─╯

The detailed view adds:

  • The rule ID in brackets, such as CX-STRICT-YAML, so you can search for it or refer to it.
  • The fix after the arrow.
  • A confidence level for each rule.
  • The harness release and date each profile was verified against.
  • Your description as the model sees it, for each harness that shortens it in its skill list.

Remove the duplicate key, replace $ARGUMENTS with words, and run it again:

skillevaluator harness-check ./my-skill
╭─ Harness compatibility · my-skill ───────────────────────────────────────────────╮
│ ✓ claude-code ✓ codex ✓ copilot-cli ✓ hermes ✓ openclaw ✓ opencode │
│ │
│ ✓ copilot-cli loads 1 note │
│ ✓ opencode loads 1 note │
│ │
│ Doesn't change pass/fail. Details and fixes: -v │
╰────────────────────────────────────────────────────────────────── ✓ 6 of 6 load ─╯

Fix what it finds

Every finding comes with its own fix, so recovering is a short loop.

1

Run it with -v

The detailed view prints the fix for each finding after the arrow. The JSON output has the same text in the fix field.

2

Fix ✗ not loaded first

Those harnesses never show your skill to the model, so nothing else matters until they are fixed. Then work through ! degraded. Notes are optional.

3

Run it again

Repeat until the tally reads 6 of 6 load. One change often clears several harnesses at once, because the same problem trips more than one loader.

These are the findings you are most likely to meet:

You seeWhyFix
Skill dropped: invalid YAMLCodex, Copilot CLI, OpenClaw and OpenCode parse front matter strictly.Indent with spaces, not tabs, and use each front matter key once.
No usable descriptionCodex, OpenClaw and OpenCode drop a skill without one. The others list it badly.Add a non-empty description as plain text.
Non-canonical nameskill.md or Skill.md is not SKILL.md. Codex and Hermes never load it.Rename the file to SKILL.md, with that exact casing.
Not foundThe skill sits in a folder this harness does not read.Install it in the folder the finding names, such as .claude/skills for Claude Code or .agents/skills for Codex.
Symlinked SKILL.md skippedCodex, Hermes and OpenClaw skip a symlinked manifest.Copy SKILL.md into the folder. If you symlink a folder, OpenClaw requires its target to stay within the skills root.
Description cut in the skill listCodex cuts at 1,024 characters, Claude Code at 1,536, and Copilot CLI drops the skill past 1,024 UTF-16 units.Put the trigger first and keep the description under 1,024 characters.
Skill list over budgetThe skills in this folder have more description text than the harness lists.Shorten descriptions, or install fewer skills in that folder.
Delivery truncatedSKILL.md is larger than the harness reads in one go.Move detail into references/ files. The tightest limit is OpenClaw, at about 15,000 characters.
$ARGUMENTS is not substitutedClaude Code substitutes $ARGUMENTS and ${CLAUDE_SKILL_DIR}. OpenCode also substitutes $ARGUMENTS through /command; its skill tool leaves it literal.Describe the expected input in words, and use paths relative to SKILL.md.
disable-model-invocation is ignoredCodex, Hermes and OpenCode don’t honour it.For Codex, add agents/openai.yaml with policy: {allow_implicit_invocation: false}.

Only ship to some harnesses? Narrow the check with --harness codex,claude-code, so findings for the others stay out of your way.

Two ways to run it

Use harness-check when this is the only answer you want. It takes a skill folder, a folder of skills, or a plugin folder. Its findings never change the exit code.

skillevaluator harness-check ./my-skill
Only the harnesses you ship to
skillevaluator harness-check ./my-skill --harness codex,claude-code
Every skill in a folder
skillevaluator harness-check ./skills
Save the result as JSON
skillevaluator harness-check ./my-skill --json harness.json
harness-checkvalidate
RunsOnly the harness checkSelected checks in the requested tiers (default: 1, 2, 3). Dependency auditing is opt-in; use --tiers 1 to limit the tiers
Exit code0 whatever it finds. A usage error or a folder with no skills is nonzeroSet by the selected checks and active policy. Harness findings are advisory by default; a policy override can make them blocking
Files--json FILEHTML and JSON reports by default. Choose formats with -r, for example -r json,html,sarif to include SARIF
Model setup flagsYes, see Match your setupNo

What it checks

The rules cover everything between your folder and the model’s context.

AreaWhat it looks for
Front matterDuplicate keys, tab indentation, a UTF-8 BOM, a missing front matter block, and long custom fields that never reach the model.
DescriptionMissing or empty descriptions and harness limits: Hermes shortens its skill-list description to 60 characters; Copilot CLI rejects descriptions over 1,024 UTF-16 units.
LocationSKILL.md casing, nested or symlinked manifests, how deep a harness scans, and whether the skill sits in a folder the harness reads.
CatalogYour skill counted with the skills installed next to it: skill-list budgets, duplicate names, and near-identical descriptions.
DeliverySize limits when the model opens the skill, such as OpenCode’s 51,200 bytes or 2,000 lines and Hermes’s 100,000 characters.
PlaceholdersWhether $ARGUMENTS and ${CLAUDE_SKILL_DIR} reach the model literally; substitution depends on the harness and invocation path.
Invocationdisable-model-invocation, allowed-tools and Claude Code-only fields that other harnesses ignore.
GatesHermes platforms and OpenClaw metadata.openclaw.requires, which hide the skill on hosts that don’t match.
PluginsWhich plugin formats each harness accepts, how plugin skills are named, and components a harness registers but doesn’t deliver.

The full rule list, harness by harness, is in Tier 1: Validation under harness.

Supported harnesses

HarnessName for --harness
Claude Codeclaude-code
OpenAI Codex CLIcodex
GitHub Copilot CLIcopilot-cli
Hermes Agenthermes
OpenClawopenclaw
OpenCodeopencode

Where the rules come from

Each harness has a profile that ships with SkillEvaluator. The profiles come from repeated runs of each harness in which the request sent to the model was captured, so a rule describes what the harness did, not what its documentation says.

Harnesses change between releases, so every profile records the release and date it was verified against, and -v shows them.

  • --harness-version codex=0.170.0 adds a note when you run a release other than the verified one.
  • --harness-profiles DIR uses your own profile YAML files, so you can try newer profiles without waiting for a release.

Match your setup

Some limits depend on how you run the harness. harness-check takes four flags for that:

FlagWhat you are telling it
--context-tokens NThe context window of your model, in tokens.
--custom-providerYour model is served through a custom or OpenAI-compatible provider.
--explicit-invocationSkills are activated by an explicit mention, such as Codex $skill.
--headlessThe harness runs non-interactively, such as claude -p in CI.

Without them, the check assumes an unknown context window, a native provider, a model that opens the skill by itself, and an interactive session. See the CLI reference for every flag.

JSON output

--json FILE writes a harness_compatibility block with one entry per skill and harness. validate puts the same block in its JSON report.

harness.json (one harness shown)
{
"harness_compatibility": {
"/path/to/my-skill": [
{
"harness": "codex",
"display_name": "OpenAI Codex CLI",
"status": "not_loaded",
"verified_version": "0.166.0",
"verified_date": "2026-11-01",
"findings": [
{
"id": "CX-STRICT-YAML",
"effect": "not_loaded",
"message": "Skill dropped: invalid YAML (tabs or duplicate keys).",
"fix": "Use spaces, not tabs, and remove duplicate keys.",
"confidence": "high"
},
{
"id": "CX-ARGUMENTS",
"effect": "degraded",
"message": "$ARGUMENTS is not substituted: the model sees the literal text.",
"fix": "Describe the expected input in words instead of $ARGUMENTS.",
"confidence": "medium"
}
],
"catalog": { "size": 1, "desc_chars": 504, "folder": "" }
}
]
}
}

status is loads, degraded or not_loaded. A finding’s effect is one of those last two, or advisory for a note. Each entry also has a listed field with the description as that harness lists it.

When it helps

  • Before you publish. One skill, six harnesses, about a second. Fix what gets dropped before anyone installs it.
  • When a skill never triggers. Check first that the harness loads it and how much of the description reaches the model’s skill list.
  • When you move a skill to another harness. A skill written for Claude Code often relies on placeholders and fields that other harnesses ignore.
  • When the skills folder grows. Harnesses give the skill list a budget. Past it, descriptions get shortened and skills get left out.
  • In CI. Include it in validate to report loading problems on each change. Findings are advisory by default; an active policy can make them block the run.

Good to know

No. Its findings are MEDIUM for not_loaded, LOW for degraded and INFO for a note. All three sit below the default gate, so the check never fails a run unless a policy raises those severities. harness-check exits 0 whatever it finds.

No. You get a verdict per harness and a tally such as 4 of 6 not loaded. For a quality score, run skillevaluator quality-check.

No. It predicts loader behavior from versioned profiles without running the agent. It doesn’t establish which skill reaches a live model or whether the skill helps once it is there. That is what Tier 3: Live Evaluation measures, by running the agent.

A harness lists your skill together with the skills installed next to it, and that list has a budget. A skill under a harness root such as .claude/skills is counted with everything in that folder. A catalog line in the panel names the folder.

Only when the skill sits under a harness root such as .claude/skills or .agents/skills. A skill in a working folder is not flagged for being in the wrong place.

Next steps