Skip to navigation

Prepare a Tier 3 Dataset

View as MarkdownOpen in Claude

Prepare the requests and expected outcomes before running an agent. Both the with-skill and without-skill arms receive the same cases. Already have evals/evals.json? Go straight to review and validate.

You need an installed CLI and a skill folder containing SKILL.md. Configure a provider and add the llm extra for model-backed generation, or use the keyless option with the lightweight install below. Docker is needed for the later agent run, not for dataset creation.

1

Write task guidance

Create eval-guidance.md in your working directory. Describe realistic requests, observable successful outcomes, and a request outside the skill’s scope. Adapt this text-analysis example to your skill:

eval-guidance.md
# Evaluation guidance for text-analyzer
## Questions
- Report the word count, character count, and most frequent word in "Red blue blue.".
- How many words and characters are in "Red blue blue.", and which word occurs most often?
- I am checking a short message: "Red blue blue.". Report its word count, character count, and most frequent word.
## Behaviors
- The response reports 3 words and 14 characters for "Red blue blue.".
- The response identifies "blue" as the most frequent word with 2 occurrences.
## Negative Cases
- What is 2 + 2?
## Notes
- All task input is inline; no external files are needed.
- Judge the reported results, regardless of how they were obtained.

Use outcomes that either arm can achieve. Requiring the agent to read the skill’s files or use its script makes the no-skill baseline unfair. More authoring examples and guidance sources.

2

Generate the cases

Replace ./my-skill with your skill folder:

Generate a dataset for review
skillevaluator create-eval-dataset ./my-skill --full --prompt eval-guidance.md

Open the generated ./my-skill/evals/evals.json. Model-backed generation uses your configured evaluator provider and can incur charges. It creates cases across explicit, implicit, contextual, and negative categories.

skillevaluator create-eval-dataset ./my-skill --full --no-llm --prompt eval-guidance.md

This produces local template cases, not model-written tasks. With the example guidance above, it creates three positive cases and one negative. Without an authored negative request, template mode does not invent one. Replace generic prompts and expected outcomes during review.

The generation command keeps an existing dataset by default. Review it directly, or use the regeneration and refinement options when you intend to change it. --force replaces the existing dataset; --refine refines it. Do not regenerate merely to run existing cases.

3

Review and validate

Open ./my-skill/evals/evals.json and review every case:

CheckReady when
Request (prompt)It is a realistic task with enough input to complete it.
Expected outcomeIt states observable success, not that the agent used the skill. For the example, use the exact counts for positive cases and 4 for the math case.
Inputs (files, when needed)The declared fixtures exist and are available to both arms. Inline-only cases need no fixture files.
CoverageCases cover the skill’s intended uses and an appropriate off-scope request.

Confirm that the file exists and contains your reviewed cases, then run:

Validate the reviewed dataset
skillevaluator tier3 validate ./my-skill --strict

If the entire evals/ directory is absent, the validator can report it as missing with exit code 0. Treat that as not ready.

Fix errors and warnings before starting a live run. Validation checks structure and flagged wording; it does not confirm that a human reviewed the cases or that the dataset is representative. A small starter is useful for checking the workflow, and produces limited evidence about skill performance.

Need the exact JSON fields, fixture layout, run policy, or sufficiency rules? Open the dataset format reference. For native Harbor task bundles, use Custom Graders & Tasks.