Skip to navigation

Configure a Tier 3 Run

View as MarkdownOpen in Claude

Start with a reviewed dataset, then choose how to run it. New to Tier 3? Follow the full walkthrough.

Before you run

Check the dataset and runtime
skillevaluator tier3 validate ./my-skill --strict
skillevaluator doctor --env-mode docker

Treat a missing dataset as not ready, even if validation exits 0. Fix errors and warnings before continuing. Native task bundles use the task workflow instead of evals.json.

Choose a run

Use your provider's default agent
skillevaluator tier3 evaluate ./my-skill --env-mode docker --n-attempts 1

Runs one attempt per selected case in each arm: with the skill and without it. This explicit command requires an existing evaluation source; it does not create a missing dataset unless you add --autopilot.

Model and sandbox usage can incur charges. Planned trials are cases × agents × attempts × 2 arms; each trial can make multiple model calls. Attempt count does not limit the number of selected cases. Keep the baseline enabled when you need Skill Lift.

Settings in evals/config.yml still apply. Check harbor.max_cases and budget settings before assuming the run covers the full dataset. See run policy.

If that file enables harbor.stop_on_pass, turn it off when switching to one attempt:

skillevaluator tier3 evaluate ./my-skill --env-mode docker --n-attempts 1 --no-stop-on-pass

Read your result

Open the latest report
skillevaluator view ./my-skill
CheckWhat to do
Run status and coverageResolve missing or failed trials before interpreting improvement.
Skill LiftCompare scored outcomes across the same cases in both arms.
Trial detailsInspect failures, skill activation, and outputs before changing the skill.

Reports are written under ./my-skill/evals/results/ by default. Interpret the report.

More control