Prepare a Tier 3 Dataset
Prepare the requests and expected outcomes before running an agent.
Both the with-skill and without-skill arms receive the same cases.
Already have evals/evals.json? Go straight to review and validate.
You need an installed CLI and a
skill folder containing SKILL.md. Configure a provider
and add the llm extra
for model-backed generation, or use the keyless option with the lightweight install below. Docker is needed
for the later agent run, not for dataset creation.
Write task guidance
Create eval-guidance.md in your working directory. Describe realistic requests,
observable successful outcomes, and a request outside the skill’s scope.
Adapt this text-analysis example to your skill:
Use outcomes that either arm can achieve. Requiring the agent to read the skill’s files or use its script makes the no-skill baseline unfair. More authoring examples and guidance sources.
Generate the cases
Replace ./my-skill with your skill folder:
Open the generated ./my-skill/evals/evals.json. Model-backed generation uses
your configured evaluator provider and can incur charges. It creates cases
across explicit, implicit, contextual, and negative categories.
Generate a keyless draft instead
This produces local template cases, not model-written tasks. With the example guidance above, it creates three positive cases and one negative. Without an authored negative request, template mode does not invent one. Replace generic prompts and expected outcomes during review.
I already have a dataset
The generation command keeps an existing dataset by default. Review it
directly, or use the regeneration and refinement options
when you intend to change it. --force replaces the existing dataset;
--refine refines it. Do not regenerate merely to run existing cases.
Review and validate
Open ./my-skill/evals/evals.json and review every case:
Confirm that the file exists and contains your reviewed cases, then run:
If the entire evals/ directory is absent, the validator can report it as
missing with exit code 0. Treat that as not ready.
Fix errors and warnings before starting a live run. Validation checks structure and flagged wording; it does not confirm that a human reviewed the cases or that the dataset is representative. A small starter is useful for checking the workflow, and produces limited evidence about skill performance.
Need the exact JSON fields, fixture layout, run policy, or sufficiency rules? Open the dataset format reference. For native Harbor task bundles, use Custom Graders & Tasks.