> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/skills/skillevaluator/_mcp/server.

# Tier 3: Live Agent Evaluation

> Run a real agent with and without your skill in a sandboxed environment, then measure Skill Lift across five dimensions.

Tier 3 answers the question the other tiers can't: **does your skill actually
make an agent better at the task?** It is the one tier that needs real
infrastructure — an evaluator LLM key, the selected agent's own credential, and
Docker or another sandbox. This guide takes you from a skill directory to a
finished with-skill vs. without-skill comparison.

## What a live eval measures

SkillEvaluator runs a real agent (`codex`, `claude-code`, or `opencode` —
`claude` is accepted as an alias for `claude-code`) against the same eval
cases twice: once with the skill installed
(the with-skill arm) and, unless you pass `--skip-baseline`, once without it
(the baseline arm). Both arms execute inside a selected
[Harbor](https://github.com/harbor-framework/harbor) environment. Harbor is the
open-source agent evaluation framework that runs the trial, so the only
experimental variable is the skill itself. **Skill Lift** is the signed
difference between the two arms — the direct measurement of what your skill
contributes. With multiple attempts per case, **pass\@k** adds a reliability
signal alongside it.

With `default` or `default_plus_custom` grading, reports lead with five
human-readable dimensions:

| Dimension       | Question answered                         |
| --------------- | ----------------------------------------- |
| Security        | Is it safe to use?                        |
| Correctness     | Does it do what it's supposed to?         |
| Discoverability | Is it loaded when it should be?           |
| Effectiveness   | Is it better with the skill than without? |
| Efficiency      | Does it use fewer tool calls and tokens?  |

In `custom_only` mode, your own grader defines the score instead — see
[Custom Graders & Tasks](/skills/skillevaluator/custom-graders).

## Before you start

Every row below is checkable in seconds; the linked guide covers setup in
depth.

| Requirement                                       | How to check                                                                      | Covered in                                                        |
| ------------------------------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| The `tier3` extra installed (`[all]` includes it) | `skillevaluator health-check`                                                     | [Installation](/skills/skillevaluator/installation)               |
| An evaluator LLM provider key                     | `skillevaluator doctor --agents codex --env-mode docker` names any missing key    | [Providers & Credentials](/skills/skillevaluator/configuration)   |
| The agent and its credential                      | Same `doctor` run, with your agents; `tier3 evaluate` re-validates before any run | [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) |
| Docker running (the default environment)          | `docker info`                                                                     | [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) |

## Run your first evaluation

### Check readiness

```bash title="Verify the runtime before spending compute"
skillevaluator doctor --agents codex --env-mode docker
```

`doctor` takes seconds and names anything missing: the evaluator provider,
the selected backend's prerequisites, and unsupported agent names. Add
`--verify-models` to live-probe each agent's model against the provider's
catalog — one pass/fail row per agent — and `--agent-model AGENT=MODEL` to
check an override before committing to a run.
`skillevaluator health-check` is the quicker
CLI-and-backend variant; [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) covers
both in depth.

### Generate an eval dataset

```bash title="Write the four-bucket dataset to evals/evals.json"
skillevaluator create-eval-dataset ./my-skill --full
```

`--full` generates four case buckets — explicit, implicit, and contextual
positives plus a negative case that should *not* trigger the skill. Generation
uses your configured evaluator provider; `--no-llm` falls back to keyless local
templates. The dataset format, `EVAL.md` guidance, and refinement options live
in [Eval Datasets](/skills/skillevaluator/eval-datasets).

### Review the cases

Open `evals/evals.json`. Each entry has an `id`, a `prompt` the agent will
receive, an `expected_output`, and optional `assertions` the grader checks.
Edit freely — the generated cases are a starting point, and the run is only as
good as its dataset.

### Run the comparison

```bash title="With-skill vs. without-skill in Docker"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --agent-model codex=gpt-5.4-mini
```

Every supported agent harness works the same way:

```bash title="Choose an agent harness"
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker
```

Use `--agent-model AGENT=MODEL` whenever the evaluator provider's default model
is not valid for that agent — for example, `codex` needs an OpenAI-compatible
Responses model. When the run completes, results land in a timestamped
directory:

* evals/
  * results/
    * 20260709\_141530/
      * result.json
      * run\_config.json
      * report.html
      * codex/
        * lift.json
        * with-skill/
        * without-skill/

### Read the report

```bash title="Open the HTML report, then compare across agents"
skillevaluator view ./my-skill
skillevaluator compare ./my-skill
```

`view` opens the latest HTML report; `compare` summarizes stored results across
agents and arms. [Reports & Results](/skills/skillevaluator/reports) explains everything the run
wrote and how to interpret it.

## How a run works

`tier3 evaluate` reads the accepted dataset (or native task source) and builds
a Harbor task bundle. Before spending the full A/B matrix, a runtime preflight
runs one real, bounded smoke task per agent to confirm each agent runtime
actually works (on by default; disable with `--no-agent-runtime-preflight`). Harbor then
starts the selected environment and runs the agent in both arms; with standard
grading, each transcript is judged against the case's expected output and
assertions using the configured evaluator provider. Finally, SkillEvaluator
collects the artifacts and computes the dimension scores, pass rates, and
Skill Lift. Harbor task and job directories are transient: they are deleted
after collection whether the run succeeds or fails, unless you pass
`--harbor-keep-jobs`.

Two credential roles are involved: the evaluator provider generates datasets
and performs standard grading, while the live agent uses its own native
credential. Both come **only from the host environment** — a checked-out skill
can never supply, replace, or reroute them:

| Credential             | Used for                                | How to set                                                                                                            |
| ---------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Evaluator LLM provider | Dataset generation and standard grading | `SKILL_EVAL_LLM_PROVIDER` plus the provider key — see [Providers & Credentials](/skills/skillevaluator/configuration) |
| Live agent credential  | The agent actually performing the task  | Export it in the host environment where you run `skillevaluator`                                                      |

```bash title="Agent credentials are host-environment exports, never config entries"
export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY=sk-...          # evaluator, and codex's runtime
export ANTHROPIC_API_KEY=sk-ant-...   # claude-code's runtime
```

`evals/config.yml` may still add permitted task-scoped values through
`harbor.runtime_env`, but any entry that names — or `${VAR}`-references — an
operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
`NVIDIA_API_KEY`, the base-URL variables, `AWS_*`) fails the run with a hard
error. This prevents a skill from swapping a credential or routing a trusted
key to an attacker-controlled endpoint. [Providers &
Credentials](/skills/skillevaluator/configuration) lists every blocked name.

`runtime_env` also cannot override host launcher, language-runtime, Docker,
Compose, proxy, or tracing controls. Configure the selected backend in the
host environment instead.

One deliberate exception collapses the two roles: with the `nv_build` provider
in `--env-mode docker` or `local`, a single `NVIDIA_API_KEY` powers the
evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly;
Codex and experimental Claude Code use a Docker or local compatibility bridge.
SkillEvaluator never changes models silently: explicit overrides are preserved.
Without an override, bridged agents default to Nemotron Super
(`nvidia/nemotron-3-super-120b-a12b`); direct OpenCode defaults to
`nvidia/nemotron-3-nano-30b-a3b` (rendered
`nvidia/nvidia/nemotron-3-nano-30b-a3b`). Override models explicitly when you
want to compare harnesses or models:

```bash title="NVIDIA Build model overrides"
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
  --agent-model opencode=nvidia/nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --agent-model codex=nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker \
  --agent-model claude-code=nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
  --agent-model opencode=nvidia/meta/llama-3.1-8b-instruct
```

On cloud env-modes the bridges are
unavailable: `opencode` still works with `NVIDIA_API_KEY` directly, but
`codex` and `claude-code` fall back to their own native credentials and
explicit models — see [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).

## Where agents run

Docker is the default and needs only a running daemon. The same `--env-mode`
flag also selects **local mode**, which runs the agent CLI directly on the
host under an OS-sandbox policy, and 14 additional Harbor-native backends
(such as `daytona`, `e2b`, and `modal`), each requiring the matching Harbor
environment extra. The complete environment matrix, Docker resource tuning, local-mode
safety controls, and workspace modes live in
[Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).

Local mode executes the agent on your host rather than in a container. It is
experimental and intended for trusted skills and workspaces — use Docker or a
cloud backend for untrusted code.

## Two ways to run Tier 3

| Use case           | Command                                                   | When to use                                                      |
| ------------------ | --------------------------------------------------------- | ---------------------------------------------------------------- |
| Full validation    | `skillevaluator validate ./my-skill --tier3`              | Add advisory Tier 3 results after the standard validation stages |
| Focused evaluation | `skillevaluator tier3 evaluate ./my-skill --agents codex` | Iterate on datasets, agents, environments, and grading settings  |

Attached to `validate`, Tier 3 is advisory: its results appear in the combined
reports but never change the exit code. `--full` is the one-shot variant
— Tier 1+2+3 with `--autopilot` dataset generation — and `--autopilot` on its
own enables Tier 3 while generating an eval case when none exists.

`validate` forwards a dedicated Tier 3 flag group (`--agents`, `--env-mode`,
`--grading-mode`, and the rest) to the live-eval engine whenever Tier 3 is
enabled — via `--tier3`, `--autopilot`, `--full`, or the `--agent-eval` alias.
`--agent-eval` is a supported compatibility alias and is not currently
deprecated. See the [CLI Reference](/skills/skillevaluator/cli-reference) for the full group.

## Plan for cost

Tier 3 performs live model and agent calls, so it can incur provider charges
and, with a managed cloud environment, sandbox charges. A useful estimate of
the trial volume is:

```text
eval cases × agents × attempts × arms
```

A standard comparison has two arms (with-skill and without-skill). Each trial
can make multiple agent-model requests, standard grading adds evaluator-model
calls, dataset generation can add more, and the default runtime preflight adds
one small smoke task per agent. Actual charges therefore depend on the selected
models, token usage, agent behavior, and environment pricing; SkillEvaluator
does not estimate a fixed dollar amount. Check the current pricing for your
[NVIDIA API Catalog](https://build.nvidia.com/),
[OpenAI](https://openai.com/api/pricing/),
[Anthropic](https://docs.anthropic.com/en/docs/about-claude/pricing), or
[Amazon Bedrock](https://aws.amazon.com/bedrock/pricing/) provider and, when
applicable, your cloud sandbox vendor.

Start with one agent, a small reviewed dataset, and `--n-attempts 1`. During
dataset iteration, `--skip-baseline` removes the without-skill arm and usually
cuts the live trial count roughly in half, but that run cannot produce Skill
Lift. Re-enable the baseline for any result you intend to compare or publish.
For multi-attempt runs, `--stop-on-pass` can avoid remaining attempts after a
pass; it is not applicable with `--n-attempts 1`. By contrast,
`--n-concurrent` and `--max-agents` control parallelism and elapsed time; they
do not reduce the total planned trials or model work.

### Cost and local mode

`--env-mode local` avoids managed cloud-sandbox charges by running the agent
on your machine. It does **not** automatically eliminate model charges: a
hosted evaluator or agent provider is still billed normally, and local compute
and runtime become your responsibility. If both the evaluator and agent are
configured to use compatible locally hosted models, external API charges may
also be avoided.

```bash title="Small local trial"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode local \
  --n-attempts 1
```

Local mode is experimental and provides weaker isolation than Docker or a
managed sandbox. Use it only for trusted skills and workspaces; saving sandbox
cost is not worth running untrusted code directly on the host.

## Tune the run

The high-signal flags, with defaults:

| Flag                        | Default                                                   | Effect                                                                                                                                                                                   |
| --------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--n-attempts`              | `1`                                                       | Attempts per eval case; values above one add pass\@k context                                                                                                                             |
| `--pass-threshold`          | `0.5`                                                     | Score an attempt needs to count as passed                                                                                                                                                |
| `--n-concurrent`            | `4`                                                       | Concurrent eval cases per agent                                                                                                                                                          |
| `--max-agents`              | selected-agent count                                      | Maximum agents run in parallel                                                                                                                                                           |
| `--skip-baseline`           | off                                                       | Skip the without-skill arm; faster, but no Skill Lift                                                                                                                                    |
| `--stop-on-pass`            | off                                                       | Stop a case's remaining attempts once one passes                                                                                                                                         |
| `--agent-runtime-preflight` | on                                                        | One real, bounded smoke task per agent before the full A/B matrix; `--no-agent-runtime-preflight` skips it                                                                               |
| `--progress`                | `auto`                                                    | Progress rendering: `auto`, `rich` (live TTY view with secret redaction), `plain`, or `off`                                                                                              |
| `--autopilot`               | off                                                       | When no evaluation source exists, generate exactly one eval case — with the configured provider, falling back to a deterministic keyless template — never overwriting an existing source |
| `--grading-mode`            | `default`                                                 | `default`, `default_plus_custom`, or `custom_only` — see [Custom Graders & Tasks](/skills/skillevaluator/custom-graders)                                                                 |
| `--results-dir`             | `evals/results`, or `SKILLEVALUATOR_RESULTS_DIR` when set | Write results under an external root instead of the skill directory                                                                                                                      |
| `--harbor-keep-jobs`        | off                                                       | Retain Harbor job directories for inspection                                                                                                                                             |
| `--timeout-multiplier`      | `1.0`                                                     | Scale Harbor step timeouts                                                                                                                                                               |
| `--copy-repo`               | off                                                       | Copy the surrounding repository into the task environment                                                                                                                                |
| `--include-skills`          | none                                                      | Stage additional skill directories; requires `--skill-workspace-mode group`                                                                                                              |

Every `tier3 evaluate` flag, including resource overrides and Dockerfile
modes, is in the [CLI Reference](/skills/skillevaluator/cli-reference).

While iterating on a dataset, `--skip-baseline` roughly halves the runtime.
Re-enable the baseline arm for any run whose Skill Lift you intend to trust.

## Reading results

In the terminal, a finished run renders **Results by Evaluator** — each
evaluator's metrics plus Skill Lift — and **Results by Dimension**, the five
rollups, followed by a compact Artifacts panel pointing at the report and
output directory. A failed run renders the same structured display instead of
a bare exception.

Each dimension receives a score and a verdict: **PASS** at 0.7 or above,
**NEUTRAL** from 0.4, **FAIL** below. Skill Lift gets its own verdict with a
deliberate noise band, because live agent runs are noisy at low attempt
counts: **Pass** at +0.05 or above, **Fail** at −0.10 or below, **Neutral** in
between.

With multiple attempts, pass\@k shows whether at least one attempt per case met
`--pass-threshold` — reliability context alongside Skill Lift, not a
replacement for the dimensions. In `custom_only` mode, pass\@k uses the custom
overall reward; the dimension rows stay visible but read **NO SCORE**, and
custom metrics are never re-labeled as standard dimensions.

Missing metrics are never coerced to `0.0`: an unscored run reports an
`overall_score` of `None`, averages omit unscored metrics, and the affected
cells render **NO SCORE**. `compare` likewise ignores stored summaries whose
`execution_status` is not `succeeded`.

[Reports & Results](/skills/skillevaluator/reports) covers the full on-disk layout, the
machine-readable JSON contract, and how to judge whether a given lift is good.

## Troubleshooting

#### A public LLM provider is required for live evaluation

Set `SKILL_EVAL_LLM_PROVIDER` and the corresponding evaluator-provider key.
See [Providers & Credentials](/skills/skillevaluator/configuration).

#### Agent authentication fails

Export the agent's credential in the host environment where you run
`skillevaluator` — agent credentials are read only from the host, never
from `evals/config.yml`. It is a separate role from the evaluator key,
except with `nv_build` in Docker or local mode, where one
`NVIDIA_API_KEY` covers both.

#### evaluate cannot start the environment

Start Docker or configure the selected `--env-mode`, then re-run
`skillevaluator doctor` with the same agent and environment.

#### No eval dataset or native task source found

Run `skillevaluator create-eval-dataset ./my-skill`, or add an accepted
dataset file under `evals/`. See [Eval Datasets](/skills/skillevaluator/eval-datasets).

#### Agent model is rejected

Supply a model valid for that agent with `--agent-model AGENT=MODEL`. The
evaluator provider's default model is not automatically valid for every
agent.

#### Results look wrong or a case is hard to diagnose

Re-run with `--harbor-keep-jobs` — Harbor artifacts are transient and
deleted by default whether the run succeeds or fails. With retention on,
the Artifacts panel prints the exact
`skillevaluator tier3 harbor-view <JOBS_DIR>` command to browse the
retained jobs; `skillevaluator view` opens the HTML report.

## Next steps

#### [Eval Datasets](/skills/skillevaluator/eval-datasets)

The full evals/ contract — dataset format, generation, and config.yml run
policy.

#### [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes)

Every supported agent and execution environment, plus readiness and
debugging tools.

#### [Custom Graders & Tasks](/skills/skillevaluator/custom-graders)

Bring your own grader or a complete Harbor task when the defaults can't
express success.

#### [Reports & Results](/skills/skillevaluator/reports)

What every run writes to disk and how to read it — including the JSON
contract for CI.