Tier 3: Live Agent Evaluation

Tier 3 answers the question the other tiers can’t: does your skill actually make an agent better at the task? It is the one tier that needs real infrastructure — model access for the evaluator and selected agent, and Docker or another sandbox. NVIDIA Build in Docker, Kata, or local mode uses one key for both roles; OpenAI with Codex can also share its key. This guide takes you from a skill directory to a finished with-skill vs. without-skill comparison.

What a live eval measures

SkillEvaluator runs a real agent (codex, claude-code, or opencode — claude is accepted as an alias for claude-code) against the same eval cases twice: once with the skill installed (the with-skill arm) and, unless you pass --skip-baseline, once without it (the baseline arm). Both arms execute inside a selected Harbor environment. Harbor is the open-source agent evaluation framework that runs the trial, so the only experimental variable is the skill itself. Skill Lift is the signed difference between the two arms — the direct measurement of what your skill contributes. With multiple attempts per case, pass@k adds a reliability signal alongside it.

With default or default_plus_custom grading, reports lead with five human-readable dimensions:

In custom_only mode, your own grader defines the score instead — see Custom Graders & Tasks.

If every positive with-skill trial has a confirmed activation score of zero, the result is INCONCLUSIVE: skill never activated. Tier 3 keeps the arm scores for inspection but omits lift comparisons and skill-quality advice for that agent. A positive case with failed activation receives a skill_execution score of zero, even if unrelated commands succeeded. Activation counts exclude negative controls; missing or unsupported activation evidence remains unknown. When at least one positive trial activates the skill, lift remains eligible subject to the comparison requirements below. This also makes activation failures visible with --skip-baseline.

Before you start

Every row below is checkable in seconds; the linked guide covers setup in depth.

Harbor 0.24 compatibility

The tier3 extra installs Harbor 0.24.0, the supported Tier 3 backend for this release. The [all] extra includes tier3. Existing SkillEvaluator commands do not change: the public --agents and --env-mode options map built-in names and installed adapters to Harbor’s unified --agent and --env selectors.

The exact Harbor pin is deliberate: its CLI, Python models, result artifacts, and environment/agent subclass interfaces form one compatibility surface. This release targets stable Harbor 0.24.0 rather than unreleased main. Harbor stays behind the operator-selected tier3 extra, so static Tier 1 installs do not pull it in. The shared LLM client window (litellm>=1.92.0,<1.94.0.dev0) admits patched 1.92 and 1.93 releases validated with this integration while excluding prerelease and as-yet unvalidated 1.94 behavior.

The integration preserves SkillEvaluator’s runtime and artifact contracts while using Harbor 0.24:

• Every Harbor process starts in an empty, evaluator-owned working directory with .env loading and Harbor telemetry disabled. Harbor ignores .env.local and .env files around your project, so it sees only SkillEvaluator’s allowlisted environment. Relative host paths such as KUBECONFIG keep their meaning. Disabling .env loading needs python-dotenv 1.2.0 or newer, which the tier3 extra requires; with an older release, doctor and tier3 evaluate fail the prerequisite check until you reinstall the extra.
• Codex runs with reasoning_effort=high, the effort Harbor 0.22 used by default, so results stay comparable across Harbor upgrades.
• Harbor installs Codex 0.161.0 and Claude Code 2.1.292 rather than the latest release, so a new CLI release does not change results. Claude Code runs Claude 5.5 and later models with reasoning_effort=medium. Local mode uses your installed CLIs and checks that they are new enough for the model.
• Installed Codex adapters merge existing user and MCP configuration before the selected runtime provider and endpoint take precedence over conflicting route keys.
• Docker, Kata, and local execution stream stdout and stderr through secret redaction before Harbor callbacks receive them. Each command has a 16 MiB combined raw stdout/stderr limit; exceeding it terminates the command process tree and fails the trial instead of returning partial output.
• The parent Harbor orchestration process has its own 16 MiB combined stdout/stderr limit for the full job. Overflow or timeout terminates and reaps the Harbor POSIX process group or Windows task tree while retaining only a bounded, redacted diagnostic tail. A POSIX process that deliberately creates a new session can escape process-group cleanup, so the host-side Harbor CLI, configuration, and local-mode skills must be trusted. Run untrusted evaluated workloads in Docker or another isolated Harbor backend instead of local mode.
• Ctrl+C stops the whole run. Agents still waiting for a --max-agents slot, queued conditions, and remaining --stop-on-pass attempts never start. Each running Harbor job receives SIGTERM, which Harbor handles by stopping its trial environments, and has up to 30 seconds to exit before its POSIX process group is killed and reaped. The progress view shows the agents that are stopping; a second Ctrl+C skips the wait. On Windows the Harbor task tree is stopped immediately. A skipped or exceeded wait, or a Windows run, can leave trial containers or in-container agent processes running. Harbor’s Docker backends name each Compose project after its trial, so docker ps --filter label=com.docker.compose.project lists the candidates; check their project names before removing them with docker rm -f. The progress view marks unfinished stages as interrupted.
• Docker accepts Harbor stdin payloads and isolates main-container execution from sidecar operations without placing environment values on Compose argv. For NVIDIA Build, the credential reaches the trusted parent over stdin; each main-container exec then uses a short-lived, container-only stdin handoff that is removed before the requested command runs.
• Native Windows main-container artifact transfer requires tar in the image and accepts regular files and directories only. Links, junctions, hardlinks, sparse files, special files, compressed streams, and unsafe or colliding Windows path spellings fail closed before publication.
• Generated tasks retain schema 1.3, which Harbor 0.24 accepts. User-owned native tasks, including schema 1.4, are staged without a schema rewrite. Collection accepts Harbor 0.24 job, trial, reward, and ATIF v1.8 artifacts, including single-step step_results: null, authoritative multi-step results, and multimodal tool output, which judges and reports see as text with [image]/[audio] markers.
• Native tasks follow Harbor 0.24’s separate-verifier precedence. For a verify pass that runs in a separate verifier container, a docker_image in the step or task [verifier.environment], or a Dockerfile or docker-compose.yaml in the step or task tests/ directory, defines a dedicated verifier that must ship its own test script and never receives SkillEvaluator’s grader. Otherwise the separate verifier is built from environment/ and receives the uploaded tests. SkillEvaluator validates the verifier’s Compose model against its environment allowlist, and rejects docker-compose.yml verifier definitions, which Harbor ignores. Standard grading modes reject separate verifiers entirely; see Custom Graders & Tasks.
• Harbor 0.24 rejects a reward.json whose top-level values are not all finite numbers. SkillEvaluator’s verifier and wrapped custom graders already write numeric rewards; a native task’s own test script must do the same. When Harbor’s job statistics do not account for every trial, the failure names the first trial or step exception Harbor recorded, such as VerifierOutputParseError, instead of reporting only a trial count.

Start with one command

After the provider setup and runtime prerequisites above:

skillevaluator tier3 ./my-skill

The direct workflow uses Docker and selects an agent from the provider: OpenCode for NVIDIA Build, Codex for OpenAI, or Claude Code for Anthropic. If no accepted dataset or task source exists, it generates one starter case and proceeds to the live with-skill and without-skill comparison. It reuses valid existing sources unchanged; invalid sources produce an error rather than being replaced. A generated starter case lets you try the pipeline, but does not establish broad positive and negative coverage.

The same agent, model, environment, grading, and reporting options described for tier3 evaluate apply to this direct workflow. For example:

skillevaluator tier3 ./my-skill --agents opencode --env-mode docker

tier3 evaluate remains available for explicit execution; its missing-source generation requires --autopilot. To prepare and review a fuller dataset before running, follow the steps below.

Prepare and review a fuller evaluation

skillevaluator doctor --env-mode docker

skillevaluator create-eval-dataset ./my-skill --full

skillevaluator tier3 ./my-skill --env-mode docker

skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker

skillevaluator view ./my-skill
skillevaluator compare ./my-skill

How a run works

tier3 evaluate reads the accepted dataset (or native task source), resolves each agent’s actual provider route and the enabled standard-grading route, and makes bounded credential-bearing model-catalog requests before image preparation or task staging. Definitive native-provider authentication, authoritative model rejection, and deterministic Bedrock credential/configuration failures stop immediately. Non-authoritative OpenAI catalog permission or membership results, compatible-gateway catalog authentication, unsupported catalogs, and transient failures are reported as degraded rather than blocking the run. A successful NVIDIA Build or compatible-gateway listing is also inconclusive when that catalog does not authenticate inference. The optional real per-agent runtime preflight is the stronger check for agent routes. A degraded route used only by the standard grader remains unproven until grading begins. Native Harbor judge selection is likewise inconclusive until runtime when no host judge override is set. result.json retains the redacted target-by-target outcome under run_config.credential_validation, including when an enabled agent runtime preflight fails.

SkillEvaluator then builds the Harbor task bundle. Before spending the full A/B matrix, you can pass --agent-runtime-preflight to run one extra agent-only execution per agent. It uses the first staged evaluation task and its real prompt, so it can consume substantial tokens and exercise task actions; grading is the only part disabled. It is off by default. Harbor then starts the selected environment and runs the agent in both arms; with standard grading, each transcript is judged against the case’s expected output and assertions using the configured evaluator provider (see How standard grading judges a trial). Finally, SkillEvaluator collects the artifacts and computes the dimension scores, pass rates, and Skill Lift. Harbor task and job directories are transient: they are deleted after collection whether the run succeeds or fails, unless you pass --harbor-keep-jobs.

Two credential roles are involved: the evaluator provider generates datasets and performs standard grading, while the live agent uses its own native credential. Both come only from the host environment — a checked-out skill can never supply, replace, or reroute them:

export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY=sk-...          # evaluator, and codex's runtime
export ANTHROPIC_API_KEY=sk-ant-...   # claude-code's runtime

evals/config.yml may still add permitted task-scoped values through harbor.runtime_env, but any entry that names — or ${VAR}-references — an operator-owned credential (OPENAI_API_KEY, ANTHROPIC_API_KEY, NVIDIA_API_KEY, the base-URL variables, AWS_*) fails the run with a hard error. This prevents a skill from swapping a credential or routing a trusted key to an attacker-controlled endpoint. Providers & Credentials lists every blocked name.

runtime_env also cannot override host launcher, language-runtime, Docker, Compose, proxy, or tracing controls. Configure the selected backend in the host environment instead.

One deliberate exception collapses the two roles: with the nv_build provider in --env-mode docker, kata, or local, a single NVIDIA_API_KEY powers the evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly; Codex and experimental Claude Code use an in-container (Docker and Kata) or local compatibility bridge. Explicit model overrides are preserved exactly. Without an override, all three agents default to Nemotron Super (nvidia/nemotron-3-super-120b-a12b); direct OpenCode renders it as nvidia/nvidia/nemotron-3-super-120b-a12b because its adapter namespace precedes the NVIDIA catalog ID. Override models explicitly when you want to compare harnesses or models:

skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
  --agent-model opencode=nvidia/nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --agent-model codex=nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker \
  --agent-model claude-code=nvidia/nemotron-3-super-120b-a12b
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
  --agent-model opencode=nvidia/meta/llama-3.1-8b-instruct

On cloud env-modes the bridges are unavailable: opencode still works with NVIDIA_API_KEY directly, but codex and claude-code fall back to their own native credentials and explicit models — see Agents & Sandboxes.

Where agents run

--env-mode selects one of 27 environment modes: 26 Harbor-native backends, including the default Docker backend, plus SkillEvaluator’s local mode, which runs the agent CLI directly on the host under an OS-sandbox policy. Managed backends such as daytona, e2b, and modal require their mapped Harbor 0.24 extra; OpenShift, Apple Container, and Singularity instead use system CLIs and have no Harbor Python extra. These are selectable integration modes, not evidence that every managed backend was exercised end to end for this release. Harbor 0.24 transport verification covers Docker and local; Kata mode reuses the Docker transport but has not been exercised on a KVM host. Validate any managed backend’s extra, credentials, infrastructure, and constructor settings with doctor and its Harbor preflight before relying on it.

Harbor’s cua-cloud, opensandbox, hf-sandbox, podman, prime, and smol backends are not selectable because they cannot yet receive SkillEvaluator’s complete generated task bundle through a trusted image or backend-native provisioning path.

Environment kwargs are only for Harbor-native non-Docker backends. Pass non-secret constructor settings with the repeatable, operator-only --environment-kwarg KEY=VALUE / --ek KEY=VALUE option. Skill-owned evals/config.yml cannot set them. docker and local reject any environment kwargs, and kata accepts only kata_runtime and kata_dns. Fields owned by Harbor runtime policy are also reserved: resource overrides, mounts, network policy, extra Compose definitions, SkillEvaluator’s own lifecycle fields (delete, force_build, keep_containers), and Kubernetes pod security are not exposed through this surface. Other backend-specific timeout and retention knobs remain operator-owned. Use the same eligible-backend flags with doctor and health-check; never pass secrets through constructor kwargs. GKE requires cluster_name, region, namespace, registry_location, and registry_name; ACK requires namespace; EC2 requires region plus ami_id for its default ephemeral launch mode or instance_id with launch_mode=attach. The complete environment/extra matrix, examples, Docker resource tuning, local-mode safety controls, and workspace modes live in Agents & Sandboxes.

Two ways to run Tier 3

Attached to validate, Tier 3 is advisory by default: its results appear in the combined reports without changing the exit code. Add --block-on-agent-eval to make its findings gate, including evidence that the selected YAML/JSON dataset or native Harbor source is missing or invalid. A blocking Tier 3 passes only with a completed PASS verdict and complete attempt coverage; NEUTRAL and FAIL verdicts exit 1. Autopilot is enabled by default for skills: it generates one starter case when no source exists and reuses valid existing sources unchanged. --full remains supported but is unnecessary. Use --tiers 1,2 or --no-tier3 to skip live evaluation; --no-autopilot requires an existing evaluation source.

skillevaluator validate ./my-skill --agents codex \
  --block-on-agent-eval

Plan for cost

Tier 3 performs live model and agent calls, so it can incur provider charges and, with a managed cloud environment, sandbox charges. A useful estimate of the trial volume is:

eval cases × agents × attempts × arms

A standard comparison has two arms (with-skill and without-skill). Each trial can make multiple agent-model requests, standard grading adds evaluator-model calls, dataset generation can add more, and the optional runtime preflight adds one extra agent execution per agent. Actual charges therefore depend on the selected models, token usage, agent behavior, and environment pricing; SkillEvaluator does not estimate a fixed dollar amount. Check the current pricing for your NVIDIA API Catalog, OpenAI, Anthropic, or Amazon Bedrock provider and, when applicable, your cloud sandbox vendor.

Start with one agent, a small reviewed dataset, and --n-attempts 1. During dataset iteration, --skip-baseline removes the without-skill arm and usually cuts the live trial count roughly in half, but that run cannot produce Skill Lift. Re-enable the baseline for any result you intend to compare or publish. For multi-attempt runs, --stop-on-pass can avoid remaining attempts after a pass; it is rejected with --n-attempts 1. To try a large dataset before paying for every case, --max-cases N (or harbor.max_cases) runs only the first N cases in staged order: eval dataset order, or task-directory name order for native Harbor tasks. The run records the selection in run_config.json, its dataset snapshot covers only the cases that ran, and reports mark it publication-incomplete when cases were skipped. By contrast, --n-concurrent and --max-agents control parallelism and elapsed time; they do not reduce the total planned trials or model work.

Cost and local mode

--env-mode local avoids managed cloud-sandbox charges by running the agent on your machine. It does not automatically eliminate model charges: a hosted evaluator or agent provider is still billed normally, and local compute and runtime become your responsibility. If both the evaluator and agent are configured to use compatible locally hosted models, external API charges may also be avoided.

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode local \
  --n-attempts 1

Local mode is experimental and provides weaker isolation than Docker or a managed sandbox. Use it only for trusted skills and workspaces; saving sandbox cost is not worth running untrusted code directly on the host.

Tune the run

The high-signal flags, with defaults:

Every tier3 evaluate flag, including resource overrides and Dockerfile modes, is in the CLI Reference.

Baseline-only runs

--skip-with-skill is the counterpart of --skip-baseline: it stages and runs only the without-skill arm. Use it to measure how agents do on a task set before the skill exists, or to re-measure a baseline after a model or harness change without paying for with-skill trials.

skillevaluator tier3 evaluate ./my-skill --agents codex --skip-with-skill

The run is reported as a baseline calibration, not a skill evaluation. The overall score and best agent come from the baseline arm, the with-skill arm is recorded as skipped, with-skill scores and Skill Lift are unavailable (shown as skipped or N/A, never 0.0), and the verdict stays NEUTRAL because no skill was measured. The two flags are mutually exclusive, and baseline isolation rules still apply: a config that sets harbor.pre_agent_setup (or setup_commands) is rejected, because the baseline arm must stay isolated from skill-owned setup commands. validate does not accept --skip-with-skill, so a baseline-only run can never satisfy a Tier 3 gate.

How standard grading judges a trial

With default or default_plus_custom grading, SkillEvaluator’s verifier grades each trial after the agent finishes. security, skill_execution, and skill_efficiency are deterministic trajectory checks. accuracy, goal_accuracy, and behavior_check are LLM judges that call the configured evaluator provider. custom_only runs none of them.

Accuracy is calculated from the judge’s criterion verdicts, not its numeric total: passing applicable criteria divided by applicable criteria. Positive cases score all five criteria; a positive-case reply with no criteria falls back to the judge’s numeric score. When should_trigger is false or no expected_skill is declared, accuracy excludes SKILL_IDENTIFIED from both counts and scores the other four criterion verdicts. The reward details record the excluded criterion and applicable-accuracy-criteria/v1 scoring policy. Missing criterion evidence is an error for these cases.

The graded trajectory

The verifier grades Harbor’s ATIF trajectory.json. When that file is missing or has no steps, it rebuilds a trajectory from the agent’s own log in the trial’s agent log directory, trying claude-code.txt, cursor-cli.txt, cline.txt, opencode.txt, and codex.txt in that order:

• opencode.txt is OpenCode’s JSON event stream. Text events become agent messages, and completed tool events become tool calls whose output keeps the tool’s status, exit code, and error text.
• codex.txt is the Codex JSON event log. Shell commands, file changes, MCP tool calls, web_search actions (with the full search, open-page, or find-in-page payload), and collab_tool_call items become tool calls. Failed commands keep their status=failed and exit_code= evidence, so error-recovery checks still see them.

Each trial’s reward records the artifact it graded in trajectory_source. If neither trajectory.json nor a supported log yields any steps, the reward records No trajectory or reconstructible agent log. Custom graders read trajectory.json directly and do not get this rebuild.

Character-sensitive judging

Set SKILL_EVAL_JUDGE_CHARACTER_FACTS=1 in the evaluator’s host environment to append character facts to the accuracy, goal_accuracy, and behavior_check prompts. For example, a literal U+00A0 (NO-BREAK SPACE) in code can look like an ordinary U+0020 space even when the task treats them differently. The metadata identifies non-ASCII whitespace and Unicode format (Cf) characters by code point, Unicode name, position, and ASCII-escaped context. Their presence alone does not indicate an error; the task determines which characters are permitted.

The original prompt and evidence remain intact. Metadata is added separately, after evidence selection and compaction, with up to 20 occurrences per field and 20 code points of context on either side. Additional occurrences are counted. The metadata adds prompt tokens beyond the evidence budgets below. Line and column positions are one-based Unicode code points in that field; only LF advances the line number. Metadata cannot recover characters removed by earlier processing, and does not identify all invisible or look-alike characters.

The option is disabled by default. Enabling it selects the prompt-based goal judge instead of the optional RAGAS route, so keep this setting consistent when comparing scores. Facts can help a judge interpret evidence but do not guarantee correct judgments. Prefer executable checks for requirements they can verify.

Evidence budgets

Each LLM judge sees a bounded evidence bundle drawn from the trajectory rather than the full transcript. Sections are added in priority order until the judge’s character budget runs out, so lower-priority sections are dropped or clipped first:

When the case’s expected output or assertions contain a backtick-quoted snippet of four or more characters, or an output path under /workspace/output, /output, or /logs/agent, every bundle starts with a short deterministic fact list marking each one observed or not observed in the agent’s tool calls.

A judge can score only what its bundle contains. Evidence that was dropped or clipped cannot count, so on a long trajectory a behavior that appears only in truncated history can be judged as not observed. In the results tree, each judge’s details in a trial’s reward.json include an omitted record (count, truncated, and reason); when truncated is true on cases that matter, raise that judge’s budget. Larger budgets send longer judge prompts and use more evaluator tokens. The behavior budget never falls below the final-response limit plus 4,000 characters, so raising the limit leaves room for the tool history. Set these variables in the host environment; SkillEvaluator forwards them to the verifier. Defaults and accepted values are in Judge evidence budgets.

Judge requests and retries

• Structured output. Each judge requests a fixed JSON schema. OpenAI and OpenAI-compatible chat-completions endpoints, including NVIDIA Build, receive a strict JSON Schema response_format; Anthropic /v1/messages receives output_config. Bedrock judges use prompt-only JSON.
• Automatic downgrade. If an endpoint rejects the schema option itself with HTTP 400 or 422 (for example, response_format is unsupported, or a gateway turned the schema into a tool_choice the model rejects), the judge repeats that request without the schema and logs a warning. It then skips the schema for that provider, endpoint, and model for the remaining judge calls in that verifier run. Other 400 and 422 errors are not downgraded.
• Reply validation. Replies are checked against the expected fields either way. An empty, unparseable, or invalid reply, including an empty reply from a reasoning model that ran out of output tokens, gets one retry with a stricter format reminder. behavior_check can also salvage complete per-behavior results from a truncated reply and counts the missing behaviors as not passed.
• Reasoning models. GPT-5 and GPT-6 judges get no temperature and at least 32,768 output tokens, because reasoning tokens count against the limit; other judges get 16,384. GPT-6 judges and Claude 5.5 or later judges on the Anthropic API are sent reasoning effort medium; other models keep their default effort.
• Refusals. A safety-classifier refusal is never scored as an answer. On the native Claude API, Claude 5.5 or later judges use server-side fallback to claude-opus-4-8, and on Bedrock a filtered Claude 5.5 reply is retried once on Claude Opus 4.8 in the same geography. A refusal or an empty reply then moves on to the next LLM_JUDGE_FALLBACK_MODELS entry, if any. Each judge’s details in the trial’s skill_evaluator_reward.json record judge_model_used, judge_reasoning_effort, and judge_fallback_used. When a case uses different judge models across its two conditions, repeated attempts, or scoring steps of a multi-step task, it is left out of that judge metric’s Skill Lift and counted in judge_mixed_cases in lift.json. The arm scores keep all scored cases; the affected evaluator, dimension, and overall lift use the remaining eligible comparisons.
• RAGAS goal scorer. With the openai provider on the canonical https://api.openai.com/v1 endpoint and a judge model that is not a GPT-5 or GPT-6 reasoning model, goal_accuracy first tries the RAGAS AgentGoalAccuracyWithReference scorer through its own client, without the schema or retry handling above but within the same time budget. If RAGAS is unavailable or fails, the prompt judge runs instead.
• Transient failures. HTTP 408, 429, and every 5xx response except 501 and 505 are retried, including 520–524 proxy errors and 529 overload responses. Connection errors, timeouts, and Bedrock throttling or service-unavailable errors are also retried with jittered exponential backoff. TLS certificate verification failures are terminal, including when an SDK wraps them as a generic connection or service-unavailable error. A Retry-After header, in seconds or as an HTTP date, sets the wait; a value above the maximum retry delay fails immediately instead of stalling.
• Time budget. Each judge has its own wall-time budget (180 seconds by default) inside the generated task’s 600-second verifier timeout, which leaves time to write failure artifacts. Requests are cut off at the deadline, and a retry whose wait would reach it is not attempted. If you raise the per-judge budget, scale the verifier timeout with --timeout-multiplier so the three judges still fit.

Retry counts, delays, and the per-judge budget come from the host environment and are forwarded to the verifier; see LLM retries and judge time budget. Retries and downgrades do not change scoring formulas. A judge that still fails is an evaluation infrastructure failure, not a 0.0 score, and the affected condition publishes no scores; see Tier 3 judge failure semantics.

Reading results

In the terminal, a finished run renders Results by Evaluator — each evaluator’s metrics plus Skill Lift — and Results by Dimension, the five rollups, followed by a compact Artifacts panel pointing at the report and output directory. A failed run renders the same structured display instead of a bare exception.

Every evaluator score and dimension score is on a 0.0 to 1.0 scale. The generated standard grader rejects non-finite values and clamps numeric scores with max(0.0, min(1.0, score)). If a required judge fails, the result is unscored rather than converted to 0.0.

The Security check’s network rule warns when an agent’s curl, wget, or HTTPie command uploads data (for example -d, -F, -T, --post-data, or a POST, PUT, or PATCH method) or sends a secret-named variable. Every client named after another command, as in find . -exec curl -d @{} URL \;, is checked with its own rules, so a command that only names a client, such as apt-get install -y curl, which curl, grep -rn http src/, or prose in a here-document, is not a network command, and a plain request through a command that is not a runner, such as strace -f curl URL, is scored like a direct curl URL. A mention followed by an HTTP method, an upload flag, or a key=value word, such as http POST in prose or grep -n http server.log status=500, still warns.

A fixed set of runners that start a command in another host, container, root, user, or network namespace, such as ssh, docker, docker-compose, kubectl, apptainer, chroot, systemd-nspawn, bwrap, or ip netns exec, warns on any client named after it, even one named only as data, as in docker exec app apt-get install -y curl. That holds when the runner is the command itself, including after a compound keyword (do, then, {, (, !), a function header, a case pattern, or coproc, also when glued to it, as in while true; do ssh host curl URL; done, if(ssh host curl URL), f(){ ssh host curl URL; }, or case x in x)ssh host curl URL;; esac, when it follows a pass-through wrapper such as strace, flock, or watch, and when a function that wraps it is called. A runner reached through another command, as in find . -exec ssh host curl URL \; or git submodule foreach ssh host curl URL, and a runner outside the set get only the client’s upload rules.

Piped or redirected input stays with a text consumer such as grep, awk, sed, or jq, so env | grep -i http is not an HTTPie upload, while cat .env | strace -f http URL, sudo < .env http URL, while read -r l; do http URL; done < .env, case "$1" in *) < .env http URL;; esac, and f() { http URL; }; f < .env are. A function that wraps a client, as in up() { curl "$@"; } or up(){ curl "$@"; }, passes each call’s arguments to that client’s rules. A $VAR command word with no known value, as in for c in curl wget; do $c --post-file=.env -q URL; done, gets the curl and wget upload rules when the action names curl or wget as a word, so $GREP -F https://example.com src/ does not. A client or runner named by any command of a substitution, such as $(which curl), $(env which curl), $(cd /tmp && which curl), or $(command -v ssh), or printed with its arguments by $(echo curl -d @.env URL), is checked the same way. When a substitution names several, as in $(command -v wget || command -v curl), the command is checked once for each.

The run’s configured judge provider and model are stored under judge in run_config.json, separately from the agent’s provider and model. The same run configuration is embedded in result.json, so a judged score can be traced to the configuration that produced it. An allowed provider fallback may use a different model for an individual call.

This provenance does not establish that the LLM judgments are correct. The repository does not publish inter-rater agreement, calibration results, or a comparison against human-labeled ground truth for these judges.

The Quality Score shown as a value out of 100 in the HTML report comes from Tier 1 static analysis. It is separate from the Tier 3 evaluator scores, dimensions, overall score, and Skill Lift.

For standard grading, the overall score is the equal-weight mean of these five dimensions:

overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
Effectiveness = (goal_accuracy + behavior_check) / 2

Effectiveness therefore enters the overall score once; its two source evaluators do not receive an extra combined vote. The same aggregate drives per-attempt pass@k, Skill Lift, comparison output, and best-agent selection. Artifacts record this contract as score_policy: skill-evaluator-dimension-mean-v1. Legacy metric sets and historical partial reports retain their recorded scoring semantics instead of being silently reinterpreted.

The Security trace check excludes plain rm -rf commands whose operands are recognized regenerable caches, such as .pytest_cache, __pycache__, or ~/.cache/uv. Absolute cache paths are recognized only under /workspace/ and /app/. This narrow scoring exception does not authorize filesystem access. Mixed targets, dynamic shell expressions, compound commands and unsupported forms retain the existing destructive-command classification.

A destructive-command finding scores the Security dimension 0.0. Other recursive rm commands outside /tmp, and xargs rm with a recursive or force option, receive this finding. A force-only rm -f or rm --force without -r or --recursive receives it only when a target resolves under /bin, /boot, /etc, /home, /lib*, /root, /sbin, /usr, /var (except /var/tmp), ~, or $HOME. Brace lists such as /{etc,usr}/hosts and dot globs such as /tmp/.*/etc/hosts are expanded first. For find ... -exec rm -f {} +, and -execdir, -ok or -okdir, find’s start paths are the targets when the rm target can be a found file: {}, a variable such as "$1", or a relative path. An absolute target such as /tmp/lock is checked on its own. A relative target resolves against the directory the shell is in when the rm runs. That starts at the tool call’s workdir or cwd argument, or else where the previous call left the shell, at first /workspace; the grader cannot see a container’s own WORKDIR. Each cd, pushd or popd that runs before the rm then moves it, in order, so cd /etc && rm -f hosts receives the finding. A later cd replaces an earlier one, a ( ... ) subshell’s cd ends with the subshell, and cd in quoted text, another command’s arguments or a heredoc body leaves it alone, so cd /etc && cat hosts; cd /workspace && rm -f out.txt, (cd /etc && ls); rm -f out.txt and echo 'run: cd /etc'; rm -f out.txt do not receive it. An rm inside $( ... ), backticks or a heredoc body is checked against every directory the surrounding command line visits, and in text the grader cannot split into shell words, every cd before the rm counts, in order. An absolute target under a protected root receives the finding even when that is the task’s project directory, such as /usr/src/app/out.txt. Resolution work is capped per trace: after one tool call moves the shell through more than 32 distinct directories, relative targets stay unresolved for the rest of the trace, and once the work budget is spent, every force-only target counts as protected. Both receive the finding.

token_efficiency is a standalone report signal, not a dimension source. It never changes a dimension score or the overall verdict.

The report’s Measured Agent Resources table separately compares prompt plus completion tokens and Harbor’s measured agent execution seconds. Cache tokens are included in prompt totals, not subtracted or counted again. Time excludes environment setup, agent installation, and evaluator judging. Delta means with-skill total minus baseline total: negative values indicate less observed use. A delta requires complete telemetry, complete successful arms, and matching case counts including repeated attempts. Missing measurements, skipped baselines, partial runs, and unsupported multi-step timing remain N/A; they are not zero. These are observations for this run, not a graded efficiency or cost claim.

Each dimension receives a score and a diagnostic band: PASS at 0.50 or above, NEUTRAL from 0.40 to below 0.50, and FAIL below 0.40. A supported agent passes only when every dimension passes. The top-level Tier 3 verdict is:

Skill Lift is with-skill overall score - without-skill overall score for comparable cases. Judge-model exclusions can make it differ from subtracting the displayed arm means; when no complete comparison with matching judges remains, lift is unavailable. It uses the same score units as the two arms; because it is a difference, it ranges from -1.0 to +1.0. Its independent band is PASS at +0.05 or above, NEUTRAL between -0.10 and +0.05, and FAIL at -0.10 or below. Lift is diagnostic evidence and does not override the every-dimension gate.

With multiple attempts, pass@k shows whether at least one attempt per case met --pass-threshold — reliability context alongside Skill Lift, not a replacement for the dimensions. In custom_only mode, pass@k uses the custom overall reward; the dimension rows stay visible but read NO SCORE, and custom metrics are never re-labeled as standard dimensions.

Missing metrics are never coerced to 0.0: an unscored run reports an overall_score of None, averages omit unscored metrics, and the affected cells render NO SCORE. compare likewise ignores stored summaries whose execution_status is not succeeded.

The canonical Tier 3 payload embedded by both standalone evaluate and validate --agent-eval reports uses schema version 2.0. A completed standard-grading payload includes the summary, five dimensions, per-agent results, trials, pass@k, attempt policy, run-owned dataset summary and digest, evaluation timestamp, evaluator version, score policy, and verdict policy; advisory or skipped payloads may leave live-evidence sections empty.

Generated BENCHMARK.md cards put the verdict first, label each result column as Baseline → Skill Uplift, show all three tier states, and keep methodology collapsible. See the rollout and backfill runbook before regenerating already-published cards.

Reports & Results covers the full on-disk layout, the machine-readable JSON contract, and how to judge whether a given lift is good.

Troubleshooting

Next steps