Configure a Tier 3 Run
Start with a reviewed dataset, then choose how to run it. New to Tier 3? Follow the full walkthrough.
Before you run
- Install SkillEvaluator with the all-tier option.
- Configure a model provider and start Docker.
- Have reviewed cases in
./my-skill/evals/evals.json. Prepare your dataset if it is missing.
Treat a missing dataset as not ready, even if validation exits 0. Fix
errors and warnings before continuing. Native task bundles use
the task workflow instead of evals.json.
Choose a run
First comparison
Choose an agent
Repeat attempts
Runs one attempt per selected case in each arm: with the skill and without it.
This explicit command requires an existing evaluation source; it does not
create a missing dataset unless you add --autopilot.
Model and sandbox usage can incur charges. Planned trials are cases × agents × attempts × 2 arms; each trial can make multiple model calls. Attempt count does not limit the number of selected cases. Keep the baseline enabled when you need Skill Lift.
Use an existing run configuration
Settings in evals/config.yml still apply. Check harbor.max_cases and
budget settings before assuming the run covers the full dataset.
See run policy.
If that file enables harbor.stop_on_pass, turn it off when switching to
one attempt:
Read your result
Reports are written under ./my-skill/evals/results/ by default.
Interpret the report.