Eval Datasets

Everything a live evaluation runs on lives in one directory: evals/ inside your skill. This page is the contract for that directory — what goes in it, the dataset format tier3 evaluate reads, and how to generate and refine cases. You can generate a starter dataset with no API key; LLM-backed generation and refinement need a configured provider (see Providers & Credentials).

Tier 3 executes these cases through Harbor, the open-source agent evaluation framework. Native Harbor task bundles are the bring-your-own-task alternative to the evals.json dataset described below.

The evals/ directory

SkillEvaluator recognizes this tree inside your skill:

Only the dataset is required. Everything else is optional and additive:

Unrecognized files in evals/ are ignored, and the validator warns about them so typos don’t silently drop a fixture.

Dataset format

New datasets use the agentskills.io shape: a top-level object with skill_name and an evals array.

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "my-skill-001",
      "prompt": "Use my-skill to do the thing",
      "expected_output": "Expected outcome description",
      "assertions": [
        "Read my-skill/SKILL.md",
        "Returned correct result"
      ]
    }
  ]
}

The dataset file can be named evals.json, evals.jsonl, evals.yaml, or evals.yml — tier3 evaluate accepts all four.

Generate a dataset

create-eval-dataset writes evals/evals.json for you, grounded in your SKILL.md name, description, body, and script filenames:

skillevaluator create-eval-dataset ./my-skill --full

Without an LLM provider configured, generation automatically falls back to the same templates --no-llm uses, so the command always succeeds. Exact defaults for every flag are in the CLI Reference.

You don’t have to generate a dataset up front at all: validate PATH, tier3 PATH, or tier3 evaluate PATH --autopilot creates exactly one eval case when no evaluation source exists — LLM-generated with your configured provider, falling back to a deterministic template case when no key is available. Autopilot never overwrites an existing source; if evals/evals.json (or evals/harbor/) is already there, it runs with what you wrote. Dataset generation does not replace the provider and runtime requirements for the live evaluation. Use validate PATH --no-autopilot to require an existing source instead of generating one.

The four case buckets

--full with an LLM provider targets one case per bucket, including a negative case that should not trigger the skill. In --no-llm template mode, the negative bucket is omitted unless eval guidance includes an off-skill prompt under a supported negative heading (see below).

Guide generation with EVAL.md

Place an EVAL.md in evals/ and the generator picks it up automatically. Four headings are parsed as structured hints; everything else is general context:

## Questions
- Sample user prompts to use as eval questions
## Behaviors
- Expected agent behaviors to verify
## Negative Cases
- Off-skill prompts for the negative bucket in template `--full` mode
## Notes
- General context and constraints

Author guidance is the highest-priority input: behaviors you list here are preserved in the generated cases even when a trial agent skipped them — that is exactly the regression the eval should catch.

Refine from real runs

--refine grounds the dataset in real agent behavior. It looks for existing trajectories from a previous evaluation (checking --from-results or the latest resolved results first); if none exist, it runs the claude-code agent via Harbor to collect one, then rewrites expected_output and assertions from what the agent actually did:

skillevaluator create-eval-dataset ./my-skill --full --refine

Collecting a fresh trajectory needs Docker, the tier3 extra, and a configured provider. Refining from existing results does not need the tier3 extra: case IDs are read from each trial’s reward.json or Harbor result.json, and multimodal trajectory messages are read as text, with image and audio parts shown as [image] and [audio] markers. With --no-llm, refinement still works — it applies template-based updates from the trajectory instead of an LLM rewrite.

Run policy: evals/config.yml

The dataset says what to evaluate; evals/config.yml (or config.yaml) says how to run it. CLI flags always override config values.

schema_version: 1
harbor:
  task_source: auto              # auto | evals_json | native_harbor
  custom_dockerfile_mode: preserve   # preserve | rebase
  n_attempts: 3
  pass_threshold: 0.50
  n_concurrent: 2
  timeout_multiplier: 1.5
  resources:
    cpus: 2
    memory_mb: 4096
  runtime_env:
    - SERVICE_API_TOKEN          # expands to ${SERVICE_API_TOKEN} at run time
  pre_agent_setup:               # with-skill-only runs (--skip-baseline)
    - service-cli auth login --token "$SERVICE_API_TOKEN"
    - service-cli whoami >/dev/null
  agents:
    claude-code:
      model: your-model-id       # any model your agent CLI accepts
skill_workspace:
  mode: isolated                 # isolated | group
  include: []
grading:
  mode: default                  # default | default_plus_custom | custom_only

Environment constructor kwargs are deliberately not accepted in this skill-owned file. Operators can pass non-secret kwargs for Harbor-native non-Docker backends with repeatable --environment-kwarg KEY=VALUE / --ek KEY=VALUE CLI flags. Backend credentials stay in the host environment; runtime-policy, mount, network, lifecycle, and pod-security fields are reserved.

Fixtures and custom environments

Two optional directories shape the world the agent works in.

evals/files/ holds input fixtures — test data, configs, sample documents. An eval entry’s files string or list selects only those paths (relative to evals/) for its /workspace/input/. An explicit empty list or null stages no fixture. For backward compatibility, an entry that omits files receives the entire shared evals/files/ directory.

evals/environment/ customizes the container itself:

[[mcp_servers]]
name = "my-api"
url = "http://localhost:8080"
[[mcp_servers]]
name = "my-tool"
command = "python /workspace/tools/server.py"
transport = "stdio"

nodejs
npm
build-essential

Validate before you run

Check the whole contract — dataset fields, case IDs, config.yml, Dockerfile rules, MCP declarations — without launching anything:

skillevaluator tier3 validate ./my-skill

Run it in CI or before a long evaluation: it catches an empty dataset, missing required fields, duplicate case IDs, and invalid config in seconds.

validate PATH also checks the selected Tier 3 source (dataset entries and case IDs, config.yml, or the native Harbor task layout) before any live run. When that check fails, each error names the entry or file and what is wrong, for example entry case-3: Missing required fields: question:

• The terminal lists the errors under the Tier 3 finding when the full CLI report prints (-r cli or --verbose), otherwise in the compact Tier 3 panel. When the run also writes an HTML or JSON report, the terminal shows the first five and names that report for the rest. When it writes neither (for example -r cli alone, or --verbose without -r), the terminal lists every error.
• The HTML report shows the exact error on the Tier 3 dashboard when there is one, or the error count when there are several. Full Insights → Source Validation Errors lists every error in order.
• The JSON report keeps every error under tier3.provenance.source_validation.

Passing checks and warnings are not listed as source errors.

Next steps