Eval Datasets

Everything a live evaluation runs on lives in one directory: evals/ inside your skill. This page is the contract for that directory — what goes in it, the dataset format tier3 evaluate reads, and how to generate and refine cases. You can generate a starter dataset with no API key; LLM-backed generation and refinement need a configured provider (see Providers & Credentials).

Tier 3 executes these cases through Harbor, the open-source agent evaluation framework. Native Harbor task bundles are the bring-your-own-task alternative to the evals.json dataset described below.

The evals/ directory

SkillEvaluator recognizes this tree inside your skill:

Only the dataset is required. Everything else is optional and additive:

Unrecognized files in evals/ are ignored, and the validator warns about them so typos don’t silently drop a fixture.

Dataset format

New datasets use the agentskills.io shape: a top-level object with skill_name and an evals array.

1
{
2
  "skill_name": "my-skill",
3
  "evals": [
4
    {
5
      "id": "my-skill-001",
6
      "prompt": "Use my-skill to do the thing",
7
      "expected_output": "Expected outcome description",
8
      "assertions": [
9
        "Read my-skill/SKILL.md",
10
        "Returned correct result"
11
      ]
12
    }
13
  ]
14
}

The dataset file can be named evals.json, evals.jsonl, evals.yaml, or evals.yml — tier3 evaluate accepts all four.

Generate a dataset

create-eval-dataset writes evals/evals.json for you, grounded in your SKILL.md name, description, body, and script filenames:

$
skillevaluator create-eval-dataset ./my-skill --full

Without an LLM provider configured, generation automatically falls back to the same templates --no-llm uses, so the command always succeeds. Exact defaults for every flag are in the CLI Reference.

You don’t have to generate a dataset up front at all: tier3 evaluate --autopilot (and validate --full or validate --autopilot) creates exactly one eval case when no evaluation source exists — LLM-generated with your configured provider, falling back to a deterministic template case when no key is available. Autopilot never overwrites an existing source; if evals/evals.json (or evals/harbor/) is already there, it runs with what you wrote.

The four case buckets

--full generates one case per bucket. The mix matters: the negative case protects against a skill that triggers when it shouldn’t.

Guide generation with EVAL.md

Place an EVAL.md in evals/ and the generator picks it up automatically. Three headings are parsed as structured hints; everything else is general context:

1
## Questions
2
- Sample user prompts to use as eval questions
3
4
## Behaviors
5
- Expected agent behaviors to verify
6
7
## Notes
8
- General context and constraints

Author guidance is the highest-priority input: behaviors you list here are preserved in the generated cases even when a trial agent skipped them — that is exactly the regression the eval should catch.

Refine from real runs

--refine grounds the dataset in real agent behavior. It looks for existing trajectories from a previous evaluation (checking --from-results or the latest resolved results first); if none exist, it runs the claude-code agent via Harbor to collect one, then rewrites expected_output and assertions from what the agent actually did:

$
skillevaluator create-eval-dataset ./my-skill --full --refine

Collecting a fresh trajectory needs Docker, the tier3 extra, and a configured provider. With --no-llm, refinement still works — it applies template-based updates from the trajectory instead of an LLM rewrite.

Run policy: evals/config.yml

The dataset says what to evaluate; evals/config.yml (or config.yaml) says how to run it. CLI flags always override config values.

1
schema_version: 1
2
3
harbor:
4
  task_source: auto              # auto | evals_json | native_harbor
5
  custom_dockerfile_mode: preserve   # preserve | rebase
6
  n_attempts: 3
7
  pass_threshold: 0.50
8
  n_concurrent: 2
9
  timeout_multiplier: 1.5
10
  resources:
11
    cpus: 2
12
    memory_mb: 4096
13
  runtime_env:
14
    - SERVICE_API_TOKEN          # expands to ${SERVICE_API_TOKEN} at run time
15
  pre_agent_setup:
16
    - service-cli auth login --token "$SERVICE_API_TOKEN"
17
    - service-cli whoami >/dev/null
18
  agents:
19
    claude-code:
20
      model: your-model-id       # any model your agent CLI accepts
21
22
skill_workspace:
23
  mode: isolated                 # isolated | group
24
  include: []
25
26
grading:
27
  mode: default                  # default | default_plus_custom | custom_only

Fixtures and custom environments

Two optional directories shape the world the agent works in.

evals/files/ holds input fixtures — test data, configs, sample documents. The entire directory is copied into /workspace/input/ inside the container, so prompts can reference those paths directly.

evals/environment/ customizes the container itself:

1
[[mcp_servers]]
2
name = "my-api"
3
url = "http://localhost:8080"
4
5
[[mcp_servers]]
6
name = "my-tool"
7
command = "python /workspace/tools/server.py"
8
transport = "stdio"

Validate before you run

Check the whole contract — dataset fields, case IDs, config.yml, Dockerfile rules, MCP declarations — without launching anything:

$
skillevaluator tier3 validate ./my-skill

Run it in CI or before a long evaluation: it catches an empty dataset, missing required fields, duplicate case IDs, and invalid config in seconds.

Next steps