Tier 3: Live Agent Evaluation

Tier 3 answers the question the other tiers can’t: does your skill actually make an agent better at the task? It is the one tier that needs real infrastructure — an evaluator LLM key, the selected agent’s own credential, and Docker or another sandbox. This guide takes you from a skill directory to a finished with-skill vs. without-skill comparison.

What a live eval measures

SkillEvaluator runs a real agent (codex, claude-code, or opencode — claude is accepted as an alias for claude-code) against the same eval cases twice: once with the skill installed (the with-skill arm) and, unless you pass --skip-baseline, once without it (the baseline arm). Both arms execute inside a selected Harbor environment. Harbor is the open-source agent evaluation framework that runs the trial, so the only experimental variable is the skill itself. Skill Lift is the signed difference between the two arms — the direct measurement of what your skill contributes. With multiple attempts per case, pass@k adds a reliability signal alongside it.

With default or default_plus_custom grading, reports lead with five human-readable dimensions:

In custom_only mode, your own grader defines the score instead — see Custom Graders & Tasks.

Before you start

Every row below is checkable in seconds; the linked guide covers setup in depth.

Run your first evaluation

$
skillevaluator doctor --agents codex --env-mode docker

$
skillevaluator create-eval-dataset ./my-skill --full

$
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
>
  --agent-model codex=gpt-5.4-mini

$
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker
$
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
$
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker

$
skillevaluator view ./my-skill
$
skillevaluator compare ./my-skill

How a run works

tier3 evaluate reads the accepted dataset (or native task source) and builds a Harbor task bundle. Before spending the full A/B matrix, a runtime preflight runs one real, bounded smoke task per agent to confirm each agent runtime actually works (on by default; disable with --no-agent-runtime-preflight). Harbor then starts the selected environment and runs the agent in both arms; with standard grading, each transcript is judged against the case’s expected output and assertions using the configured evaluator provider. Finally, SkillEvaluator collects the artifacts and computes the dimension scores, pass rates, and Skill Lift. Harbor task and job directories are transient: they are deleted after collection whether the run succeeds or fails, unless you pass --harbor-keep-jobs.

Two credential roles are involved: the evaluator provider generates datasets and performs standard grading, while the live agent uses its own native credential. Both come only from the host environment — a checked-out skill can never supply, replace, or reroute them:

$
export SKILL_EVAL_LLM_PROVIDER=openai
$
export OPENAI_API_KEY=sk-...          # evaluator, and codex's runtime
>
export ANTHROPIC_API_KEY=sk-ant-...   # claude-code's runtime

evals/config.yml may still add permitted task-scoped values through harbor.runtime_env, but any entry that names — or ${VAR}-references — an operator-owned credential (OPENAI_API_KEY, ANTHROPIC_API_KEY, NVIDIA_API_KEY, the base-URL variables, AWS_*) fails the run with a hard error. This prevents a skill from swapping a credential or routing a trusted key to an attacker-controlled endpoint. Providers & Credentials lists every blocked name.

runtime_env also cannot override host launcher, language-runtime, Docker, Compose, proxy, or tracing controls. Configure the selected backend in the host environment instead.

One deliberate exception collapses the two roles: with the nv_build provider in --env-mode docker or local, a single NVIDIA_API_KEY powers the evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly; Codex and experimental Claude Code use a Docker or local compatibility bridge. SkillEvaluator never changes models silently: explicit overrides are preserved. Without an override, bridged agents default to Nemotron Super (nvidia/nemotron-3-super-120b-a12b); direct OpenCode defaults to nvidia/nemotron-3-nano-30b-a3b (rendered nvidia/nvidia/nemotron-3-nano-30b-a3b). Override models explicitly when you want to compare harnesses or models:

$
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
>
  --agent-model opencode=nvidia/nvidia/nemotron-3-super-120b-a12b
$
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
>
  --agent-model codex=nvidia/nemotron-3-super-120b-a12b
$
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker \
>
  --agent-model claude-code=nvidia/nemotron-3-super-120b-a12b
$
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
>
  --agent-model opencode=nvidia/meta/llama-3.1-8b-instruct

On cloud env-modes the bridges are unavailable: opencode still works with NVIDIA_API_KEY directly, but codex and claude-code fall back to their own native credentials and explicit models — see Agents & Sandboxes.

Where agents run

Docker is the default and needs only a running daemon. The same --env-mode flag also selects local mode, which runs the agent CLI directly on the host under an OS-sandbox policy, and 14 additional Harbor-native backends (such as daytona, e2b, and modal), each requiring the matching Harbor environment extra. The complete environment matrix, Docker resource tuning, local-mode safety controls, and workspace modes live in Agents & Sandboxes.

Two ways to run Tier 3

Attached to validate, Tier 3 is advisory: its results appear in the combined reports but never change the exit code. --full is the one-shot variant — Tier 1+2+3 with --autopilot dataset generation — and --autopilot on its own enables Tier 3 while generating an eval case when none exists.

Plan for cost

Tier 3 performs live model and agent calls, so it can incur provider charges and, with a managed cloud environment, sandbox charges. A useful estimate of the trial volume is:

eval cases × agents × attempts × arms

A standard comparison has two arms (with-skill and without-skill). Each trial can make multiple agent-model requests, standard grading adds evaluator-model calls, dataset generation can add more, and the default runtime preflight adds one small smoke task per agent. Actual charges therefore depend on the selected models, token usage, agent behavior, and environment pricing; SkillEvaluator does not estimate a fixed dollar amount. Check the current pricing for your NVIDIA API Catalog, OpenAI, Anthropic, or Amazon Bedrock provider and, when applicable, your cloud sandbox vendor.

Start with one agent, a small reviewed dataset, and --n-attempts 1. During dataset iteration, --skip-baseline removes the without-skill arm and usually cuts the live trial count roughly in half, but that run cannot produce Skill Lift. Re-enable the baseline for any result you intend to compare or publish. For multi-attempt runs, --stop-on-pass can avoid remaining attempts after a pass; it is not applicable with --n-attempts 1. By contrast, --n-concurrent and --max-agents control parallelism and elapsed time; they do not reduce the total planned trials or model work.

Cost and local mode

--env-mode local avoids managed cloud-sandbox charges by running the agent on your machine. It does not automatically eliminate model charges: a hosted evaluator or agent provider is still billed normally, and local compute and runtime become your responsibility. If both the evaluator and agent are configured to use compatible locally hosted models, external API charges may also be avoided.

$
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode local \
>
  --n-attempts 1

Local mode is experimental and provides weaker isolation than Docker or a managed sandbox. Use it only for trusted skills and workspaces; saving sandbox cost is not worth running untrusted code directly on the host.

Tune the run

The high-signal flags, with defaults:

Every tier3 evaluate flag, including resource overrides and Dockerfile modes, is in the CLI Reference.

Reading results

In the terminal, a finished run renders Results by Evaluator — each evaluator’s metrics plus Skill Lift — and Results by Dimension, the five rollups, followed by a compact Artifacts panel pointing at the report and output directory. A failed run renders the same structured display instead of a bare exception.

Each dimension receives a score and a verdict: PASS at 0.7 or above, NEUTRAL from 0.4, FAIL below. Skill Lift gets its own verdict with a deliberate noise band, because live agent runs are noisy at low attempt counts: Pass at +0.05 or above, Fail at −0.10 or below, Neutral in between.

With multiple attempts, pass@k shows whether at least one attempt per case met --pass-threshold — reliability context alongside Skill Lift, not a replacement for the dimensions. In custom_only mode, pass@k uses the custom overall reward; the dimension rows stay visible but read NO SCORE, and custom metrics are never re-labeled as standard dimensions.

Missing metrics are never coerced to 0.0: an unscored run reports an overall_score of None, averages omit unscored metrics, and the affected cells render NO SCORE. compare likewise ignores stored summaries whose execution_status is not succeeded.

Reports & Results covers the full on-disk layout, the machine-readable JSON contract, and how to judge whether a given lift is good.

Troubleshooting

Next steps