Tier 3: Live Agent Evaluation
Tier 3: Live Agent Evaluation
Tier 3 answers the question the other tiers can’t: does your skill actually make an agent better at the task? It is the one tier that needs real infrastructure — an evaluator LLM key, the selected agent’s own credential, and Docker or another sandbox. This guide takes you from a skill directory to a finished with-skill vs. without-skill comparison.
What a live eval measures
SkillEvaluator runs a real agent (codex, claude-code, or opencode —
claude is accepted as an alias for claude-code) against the same eval
cases twice: once with the skill installed
(the with-skill arm) and, unless you pass --skip-baseline, once without it
(the baseline arm). Both arms execute inside a selected
Harbor environment. Harbor is the
open-source agent evaluation framework that runs the trial, so the only
experimental variable is the skill itself. Skill Lift is the signed
difference between the two arms — the direct measurement of what your skill
contributes. With multiple attempts per case, pass@k adds a reliability
signal alongside it.
With default or default_plus_custom grading, reports lead with five
human-readable dimensions:
In custom_only mode, your own grader defines the score instead — see
Custom Graders & Tasks.
Before you start
Every row below is checkable in seconds; the linked guide covers setup in depth.
Run your first evaluation
Check readiness
doctor takes seconds and names anything missing: the evaluator provider,
the selected backend’s prerequisites, and unsupported agent names. Add
--verify-models to live-probe each agent’s model against the provider’s
catalog — one pass/fail row per agent — and --agent-model AGENT=MODEL to
check an override before committing to a run.
skillevaluator health-check is the quicker
CLI-and-backend variant; Agents & Sandboxes covers
both in depth.
Generate an eval dataset
--full generates four case buckets — explicit, implicit, and contextual
positives plus a negative case that should not trigger the skill. Generation
uses your configured evaluator provider; --no-llm falls back to keyless local
templates. The dataset format, EVAL.md guidance, and refinement options live
in Eval Datasets.
Review the cases
Open evals/evals.json. Each entry has an id, a prompt the agent will
receive, an expected_output, and optional assertions the grader checks.
Edit freely — the generated cases are a starting point, and the run is only as
good as its dataset.
Each eval entry can declare task inputs with a files string or list. Paths are
relative to evals/ (for example, "files/case-001/input.txt") and only those
declared paths are staged into that task’s /workspace/input/. An explicit
empty list or null stages no task input. For backward compatibility, omitting
the files field stages the entire shared evals/files/ directory.
An evals/ directory directly owned by a skill package is evaluator-only. Tier
3 resolves the selected task source once and copies only the evaluator inputs it
can consume into one private temporary snapshot. That projection includes the
active dataset and configuration, selected fixtures, the first consumed grader,
the complete authored custom environment, and native tasks when selected.
Legacy entries that omit files retain the complete shared evals/files/
corpus. Unrelated evaluator subtrees and generated results are not copied. Every
agent and baseline arm reads the same snapshot, preventing a concurrent source
replacement from mixing dataset, fixture, grader, custom-environment, or
native-task identities. The snapshot is removed when the run returns and
requires temporary space only for the selected compatibility scope.
Tier 3 does not install evaluator-owned directories from the target or any
nested, reference, or workspace skill into agent-visible copies. --copy-repo
uses the same filtering and excludes generated result roots. Authenticated
historical result roots remain excluded after the configured output location is
changed; a present marker that is copied, stale, or cannot be authenticated
fails staging closed instead of copying that tree. Custom result roots created
outside evals/ by a pre-marker version cannot be identified safely after the
configured location changes. Move or delete that old content before
--copy-repo or another full-context evaluation, then rerun with the current
version if replacement evidence is needed. Runtime assets that the agent needs
as part of the installed skill should live outside skill-owned evals/. Task
fixtures should live under evals/files/ (preferred) or another non-control
subtree such as evals/data/, and should be declared as task inputs with
files. A files declaration cannot select evaluator control material:
dataset/configuration files, graders, developer eval guidance, benchmark
conversion reports, native Harbor tasks, custom environments, evaluator tests,
or generated results.
MCP declarations are read only from a bounded, stable, single-link regular
evals/environment/mcp_servers.toml through a descriptor-anchored evaluator
root. Linked configuration files or parents, hardlinks, reparse points, and
files replaced during the read fail task generation before output mutation. On
Windows, the selected-file handle also denies concurrent writes and deletion
for the read.
evals/environment/input/ is reserved and is not copied into generated tasks.
Directory-level custom Docker COPY input/ remains buildable for an empty entry
using an empty context directory. Missing, dynamic, linked, hardlinked, aliased,
or escaping input sources fail closed. Native Harbor tasks and Compose input
contexts use the same omitted-versus-explicit selection rules.
The final evaluator layer replaces /workspace/input, /workspace/repo, and
supported agent skill-discovery roots. Custom Docker preserve mode also
requires a statically known final USER so the projection can be installed and
the original user restored. Custom Dockerfiles and base images remain trusted
executable environment inputs: these controls isolate evaluation fixtures and
supported discovery roots; they are not a security boundary against an image
that intentionally changes system binaries or startup behavior.
Evaluator-owned in-skill output uses a path-bound authenticity marker backed by
a local key. A copied, forged, or stale marker cannot authorize replacement at
another path. A custom key path set through
SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE must remain outside every evaluated
skill source.
Run the comparison
Every supported agent harness works the same way:
Use --agent-model AGENT=MODEL whenever the evaluator provider’s default model
is not valid for that agent — for example, codex needs an OpenAI-compatible
Responses model. When the run completes, results land in a timestamped
directory:
Read the report
view opens the latest HTML report; compare summarizes stored results across
agents and arms. Reports & Results explains everything the run
wrote and how to interpret it.
How a run works
tier3 evaluate reads the accepted dataset (or native task source) and builds
a Harbor task bundle. Before spending the full A/B matrix, a runtime preflight
runs one real, bounded smoke task per agent to confirm each agent runtime
actually works (on by default; disable with --no-agent-runtime-preflight). Harbor then
starts the selected environment and runs the agent in both arms; with standard
grading, each transcript is judged against the case’s expected output and
assertions using the configured evaluator provider. Finally, SkillEvaluator
collects the artifacts and computes the dimension scores, pass rates, and
Skill Lift. Harbor task and job directories are transient: they are deleted
after collection whether the run succeeds or fails, unless you pass
--harbor-keep-jobs.
Two credential roles are involved: the evaluator provider generates datasets and performs standard grading, while the live agent uses its own native credential. Both come only from the host environment — a checked-out skill can never supply, replace, or reroute them:
evals/config.yml may still add permitted task-scoped values through
harbor.runtime_env, but any entry that names — or ${VAR}-references — an
operator-owned credential (OPENAI_API_KEY, ANTHROPIC_API_KEY,
NVIDIA_API_KEY, the base-URL variables, AWS_*) fails the run with a hard
error. This prevents a skill from swapping a credential or routing a trusted
key to an attacker-controlled endpoint. Providers &
Credentials lists every blocked name.
runtime_env also cannot override host launcher, language-runtime, Docker,
Compose, proxy, or tracing controls. Configure the selected backend in the
host environment instead.
One deliberate exception collapses the two roles: with the nv_build provider
in --env-mode docker or local, a single NVIDIA_API_KEY powers the
evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly;
Codex and experimental Claude Code use a Docker or local compatibility bridge.
SkillEvaluator never changes models silently: explicit overrides are preserved.
Without an override, bridged agents default to Nemotron Super
(nvidia/nemotron-3-super-120b-a12b); direct OpenCode defaults to
nvidia/nemotron-3-nano-30b-a3b (rendered
nvidia/nvidia/nemotron-3-nano-30b-a3b). Override models explicitly when you
want to compare harnesses or models:
On cloud env-modes the bridges are
unavailable: opencode still works with NVIDIA_API_KEY directly, but
codex and claude-code fall back to their own native credentials and
explicit models — see Agents & Sandboxes.
Where agents run
Docker is the default and needs only a running daemon. The same --env-mode
flag also selects local mode, which runs the agent CLI directly on the
host under an OS-sandbox policy, and 14 additional Harbor-native backends
(such as daytona, e2b, and modal), each requiring the matching Harbor
environment extra. The complete environment matrix, Docker resource tuning, local-mode
safety controls, and workspace modes live in
Agents & Sandboxes.
Local mode executes the agent on your host rather than in a container. It is experimental and intended for trusted skills and workspaces — use Docker or a cloud backend for untrusted code.
Two ways to run Tier 3
Attached to validate, Tier 3 is advisory: its results appear in the combined
reports but never change the exit code. --full is the one-shot variant
— Tier 1+2+3 with --autopilot dataset generation — and --autopilot on its
own enables Tier 3 while generating an eval case when none exists.
validate forwards a dedicated Tier 3 flag group (--agents, --env-mode,
--grading-mode, and the rest) to the live-eval engine whenever Tier 3 is
enabled — via --tier3, --autopilot, --full, or the --agent-eval alias.
--agent-eval is a supported compatibility alias and is not currently
deprecated. See the CLI Reference for the full group.
Plan for cost
Tier 3 performs live model and agent calls, so it can incur provider charges and, with a managed cloud environment, sandbox charges. A useful estimate of the trial volume is:
A standard comparison has two arms (with-skill and without-skill). Each trial can make multiple agent-model requests, standard grading adds evaluator-model calls, dataset generation can add more, and the default runtime preflight adds one small smoke task per agent. Actual charges therefore depend on the selected models, token usage, agent behavior, and environment pricing; SkillEvaluator does not estimate a fixed dollar amount. Check the current pricing for your NVIDIA API Catalog, OpenAI, Anthropic, or Amazon Bedrock provider and, when applicable, your cloud sandbox vendor.
Start with one agent, a small reviewed dataset, and --n-attempts 1. During
dataset iteration, --skip-baseline removes the without-skill arm and usually
cuts the live trial count roughly in half, but that run cannot produce Skill
Lift. Re-enable the baseline for any result you intend to compare or publish.
For multi-attempt runs, --stop-on-pass can avoid remaining attempts after a
pass; it is not applicable with --n-attempts 1. By contrast,
--n-concurrent and --max-agents control parallelism and elapsed time; they
do not reduce the total planned trials or model work.
Cost and local mode
--env-mode local avoids managed cloud-sandbox charges by running the agent
on your machine. It does not automatically eliminate model charges: a
hosted evaluator or agent provider is still billed normally, and local compute
and runtime become your responsibility. If both the evaluator and agent are
configured to use compatible locally hosted models, external API charges may
also be avoided.
Local mode is experimental and provides weaker isolation than Docker or a managed sandbox. Use it only for trusted skills and workspaces; saving sandbox cost is not worth running untrusted code directly on the host.
Tune the run
The high-signal flags, with defaults:
Every tier3 evaluate flag, including resource overrides and Dockerfile
modes, is in the CLI Reference.
While iterating on a dataset, --skip-baseline roughly halves the runtime.
Re-enable the baseline arm for any run whose Skill Lift you intend to trust.
Reading results
In the terminal, a finished run renders Results by Evaluator — each evaluator’s metrics plus Skill Lift — and Results by Dimension, the five rollups, followed by a compact Artifacts panel pointing at the report and output directory. A failed run renders the same structured display instead of a bare exception.
Each dimension receives a score and a verdict: PASS at 0.7 or above, NEUTRAL from 0.4, FAIL below. Skill Lift gets its own verdict with a deliberate noise band, because live agent runs are noisy at low attempt counts: Pass at +0.05 or above, Fail at −0.10 or below, Neutral in between.
With multiple attempts, pass@k shows whether at least one attempt per case met
--pass-threshold — reliability context alongside Skill Lift, not a
replacement for the dimensions. In custom_only mode, pass@k uses the custom
overall reward; the dimension rows stay visible but read NO SCORE, and
custom metrics are never re-labeled as standard dimensions.
Missing metrics are never coerced to 0.0: an unscored run reports an
overall_score of None, averages omit unscored metrics, and the affected
cells render NO SCORE. compare likewise ignores stored summaries whose
execution_status is not succeeded.
Reports & Results covers the full on-disk layout, the machine-readable JSON contract, and how to judge whether a given lift is good.
Troubleshooting
A public LLM provider is required for live evaluation
Set SKILL_EVAL_LLM_PROVIDER and the corresponding evaluator-provider key.
See Providers & Credentials.
Agent authentication fails
Export the agent’s credential in the host environment where you run
skillevaluator — agent credentials are read only from the host, never
from evals/config.yml. It is a separate role from the evaluator key,
except with nv_build in Docker or local mode, where one
NVIDIA_API_KEY covers both.
evaluate cannot start the environment
Start Docker or configure the selected --env-mode, then re-run
skillevaluator doctor with the same agent and environment.
No eval dataset or native task source found
Run skillevaluator create-eval-dataset ./my-skill, or add an accepted
dataset file under evals/. See Eval Datasets.
Agent model is rejected
Supply a model valid for that agent with --agent-model AGENT=MODEL. The
evaluator provider’s default model is not automatically valid for every
agent.
Results look wrong or a case is hard to diagnose
Re-run with --harbor-keep-jobs — Harbor artifacts are transient and
deleted by default whether the run succeeds or fails. With retention on,
the Artifacts panel prints the exact
skillevaluator tier3 harbor-view <JOBS_DIR> command to browse the
retained jobs; skillevaluator view opens the HTML report.
Next steps
The full evals/ contract — dataset format, generation, and config.yml run policy.
Every supported agent and execution environment, plus readiness and debugging tools.
Bring your own grader or a complete Harbor task when the defaults can’t express success.
What every run writes to disk and how to read it — including the JSON contract for CI.