> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.

# Configure a Tier 3 Run

> Start from a reviewed dataset, choose an agent and attempt count, then inspect the comparison.

Start with a **reviewed dataset**, then choose how to run it.
[New to Tier 3? Follow the full walkthrough](/skills/skillevaluator/tier3-live-evaluation).

#### [1. Prepare](#before-you-run)

Review your cases and check readiness.

#### [2. Choose a run](#choose-a-run)

Start with one agent and one attempt.

#### [3. Read the result](#read-your-result)

Check coverage before comparing scores.

## Before you run

* [Install SkillEvaluator](/skills/skillevaluator/installation#all-tiers) with the all-tier option.
* Configure a [model provider](/skills/skillevaluator/configuration) and start Docker.
* Have reviewed cases in `./my-skill/evals/evals.json`. [Prepare your dataset](/skills/skillevaluator/prepare-dataset) if it is missing.

**`Check the dataset and runtime`**

```bash title="Check the dataset and runtime"
skillevaluator tier3 validate ./my-skill --strict
skillevaluator doctor --env-mode docker
```

Treat a missing dataset as **not ready**, even if validation exits `0`. Fix
errors and warnings before continuing. Native task bundles use
the [task workflow](/skills/skillevaluator/custom-graders) instead of `evals.json`.

## Choose a run

#### First comparison

**`Use your provider's default agent`**

```bash title="Use your provider's default agent"
skillevaluator tier3 evaluate ./my-skill --env-mode docker --n-attempts 1
```

Runs one attempt per selected case in each arm: with the skill and without it.
This explicit command requires an existing evaluation source; it does not
create a missing dataset unless you add `--autopilot`.

#### Choose an agent

**`Example: use Codex with an OpenAI provider`**

```bash title="Example: use Codex with an OpenAI provider"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker --n-attempts 1
```

Configure OpenAI before using this example. For OpenCode, Claude Code, or a
model override, follow [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).

#### Repeat attempts

**`Try each case up to three times`**

```bash title="Try each case up to three times"
skillevaluator tier3 evaluate ./my-skill --env-mode docker --n-attempts 3 --stop-on-pass
```

A passing attempt stops the remaining attempts for that case. This can still
use more model calls than a single-attempt run. `--stop-on-pass` requires more
than one attempt.

> **Note**
>
> Model and sandbox usage can incur charges. Planned trials are
> **cases × agents × attempts × 2 arms**; each trial can make multiple model
> calls. Attempt count does not limit the number of selected cases. Keep the baseline enabled when
> you need Skill Lift.

#### Use an existing run configuration

Settings in `evals/config.yml` still apply. Check `harbor.max_cases` and
budget settings before assuming the run covers the full dataset.
See [run policy](/skills/skillevaluator/eval-datasets#run-policy-evalsconfigyml).

If that file enables `harbor.stop_on_pass`, turn it off when switching to
one attempt:

```bash
skillevaluator tier3 evaluate ./my-skill --env-mode docker --n-attempts 1 --no-stop-on-pass
```

## Read your result

**`Open the latest report`**

```bash title="Open the latest report"
skillevaluator view ./my-skill
```

| Check                   | What to do                                                                 |
| ----------------------- | -------------------------------------------------------------------------- |
| Run status and coverage | Resolve missing or failed trials before interpreting improvement.          |
| Skill Lift              | Compare scored outcomes across the same cases in both arms.                |
| Trial details           | Inspect failures, skill activation, and outputs before changing the skill. |

Reports are written under `./my-skill/evals/results/` by default.
[Interpret the report](/skills/skillevaluator/tier3-results).

## More control

#### Dataset staging, cost, and advanced run settings

* [Generate, review, and stage cases](/skills/skillevaluator/tier3-run-reference#prepare-and-review-a-fuller-evaluation)
* [Estimate trial volume and control cost](/skills/skillevaluator/tier3-run-reference#plan-for-cost)
* [All run settings and interactions](/skills/skillevaluator/tier3-run-reference#tune-the-run)
* [Reuse a baseline](/skills/skillevaluator/tier3-run-reference#reuse-a-baseline)
* [Compare skill versions](/skills/skillevaluator/tier3-run-reference#compare-skill-versions)
* [Baseline-only runs and timeouts](/skills/skillevaluator/tier3-run-reference#baseline-only-runs)
* [Budget limits and accounting](/skills/skillevaluator/tier3-run-reference#cost-limits)