Live Evaluation Grading Reference
For your first comparison, start with Run a live evaluation.
How standard grading judges a trial
With default or default_plus_custom grading, SkillEvaluator’s verifier
grades each trial after the agent finishes. security, skill_execution, and
skill_efficiency are deterministic trajectory checks. accuracy and
goal_accuracy are LLM judges that call the configured evaluator provider.
behavior_check combines the behavior judge with directly evaluated declared
checks; a checks-only case makes no behavior judge call. custom_only runs
none of them.
skill_efficiency averages skill routing with tool efficiency, the share of
scored tool calls that did productive work. For Claude Code, Codex and OpenCode
tool names, reading, searching, listing, writing, editing or patching files,
running commands and activating a skill are productive, and so are MCP tools
named mcp__server__tool (Claude Code). Codex and OpenCode MCP calls are not
recognized as MCP calls yet, so most of them count as wasted. Help, version and
install commands and searches rooted at / are wasted. Planning and
bookkeeping calls, such as TodoWrite, update_plan, task lists,
background-shell polling and sub-agent calls, are not counted, and a trial with
only such calls scores 0.0. Reports lists the tool classes.
The three LLM judges return verdicts only, and the verifier computes the
scores. A score field in a judge reply is ignored.
Error recovery distinguishes skill files, agent command errors, and missing
runtime dependencies. Missing tools/modules declared by or used by the skill
are environment problems: recovered failures have no deduction, while an
unrecovered environment failure without an unrecovered skill fault marks the
case environment_invalid. skill_execution then returns no score, the
verifier skips the judges, and the collector leaves the trial UNSCORED
(excluded from skill score averages). Any unrecovered skill fault in the same
trial is scored as a skill failure instead. Agent-only failures remain
unscored with the environment issue, and are included in its reason and
evaluation_errors. Reports include the missing name and dependency
declaration/installation guidance. Declare system tools in
scripts/apt-packages.txt, Python packages in scripts/requirements.txt, or
install them in evals/environment/Dockerfile. Agent-chosen tools the skill
does not use remain agent errors. Missing files inside the skill remain skill
defects (0.25 deduction if recovered); real unrecovered skill/agent failures
still score zero.
accuracyis passing criteria divided by five. The criteria grade the outcome:ACTION_CORRECT,FACTUALLY_ACCURATE,TASK_ADDRESSED,ACTIONABLE, andFORMAT_FOLLOWED(the response follows any output format the question explicitly requires; when none is required, it is true only if the response gives an answer). A response that gives no answer fails all five criteria and scores0.0, unless the expected answer is itself a refusal. Every case and both arms use the same five criteria. None of them asks whether the agent named or used the skill;skill_executionscores that. A reply without all five true/false verdicts is invalid.goal_accuracyis1.0when the judge decides the goal wasachievedand0.0otherwise, the same binary scale as the RAGAS scorer.behavior_checkincludes the share of expected behaviors observed and any directly evaluated declared checks. When expected behaviors are present, the reply must judge every behavior exactly once, identified by its 1-basedstepnumber; a reply that skips, repeats, or invents a step is invalid. A checks-only case is evaluated directly without a judge call.
Rewards record judge_contract: skill-evaluator-judges-v2. Results written
before this contract, whose accuracy included a SKILL_IDENTIFIED criterion,
are not directly comparable with newer ones: a correct no-skill baseline answer
used to be capped at 0.8 on positive cases.
The graded trajectory
The verifier grades Harbor’s ATIF trajectory.json. When that file is missing
or has no steps, it rebuilds a trajectory from the agent’s own log in the
trial’s agent log directory, trying claude-code.txt, cursor-cli.txt,
cline.txt, opencode.txt, and codex.txt in that order:
opencode.txtis OpenCode’s JSON event stream. Text events become agent messages, and completed tool events become tool calls whose output keeps the tool’s status, exit code, and error text.codex.txtis the Codex JSON event log. Shell commands, file changes, MCP tool calls,web_searchactions (with the full search, open-page, or find-in-page payload), andcollab_tool_callitems become tool calls. Failed commands keep theirstatus=failedandexit_code=evidence, so error-recovery checks still see them.
Each trial’s reward records the artifact it graded in trajectory_source. If
neither trajectory.json nor a supported log yields any steps, the reward
records No trajectory or reconstructible agent log. Custom graders read
trajectory.json directly and do not get this rebuild.
Character-sensitive judging
Set SKILL_EVAL_JUDGE_CHARACTER_FACTS=1 in the evaluator’s host environment to
append character facts to the accuracy, goal_accuracy, and behavior_check
prompts. For example, a literal U+00A0 (NO-BREAK SPACE) in code can look like an
ordinary U+0020 space even when the task treats them differently. The metadata
identifies non-ASCII whitespace and Unicode format (Cf) characters by code
point, Unicode name, position, and ASCII-escaped context. Their presence alone
does not indicate an error; the task determines which characters are permitted.
The original prompt and evidence remain intact. Metadata is added separately, after evidence selection and compaction, with up to 20 occurrences per field and 20 code points of context on either side. Additional occurrences are counted. The metadata adds prompt tokens beyond the evidence budgets below. Line and column positions are one-based Unicode code points in that field; only LF advances the line number. Metadata cannot recover characters removed by earlier processing, and does not identify all invisible or look-alike characters.
The option is disabled by default. Enabling it selects the prompt-based goal judge instead of the optional RAGAS route, so keep this setting consistent when comparing scores. Facts can help a judge interpret evidence but do not guarantee correct judgments. Prefer executable checks for requirements they can verify.
Evidence budgets
Each LLM judge sees a bounded evidence bundle drawn from the trajectory rather than the full transcript. Sections are added in priority order until the judge’s character budget runs out, so lower-priority sections are dropped or clipped first:
When the case’s expected output or assertions contain a backtick-quoted
snippet of four or more characters, or an output path under
/workspace/output, /output, or /logs/agent, every bundle starts with a
short deterministic fact list marking each one observed or not observed in the
agent’s tool calls.
A judge can score only what its bundle contains. Evidence that was dropped or
clipped cannot count, so on a long trajectory a behavior that appears only in
truncated history can be judged as not observed. In the results tree, each
judge’s details in a trial’s reward.json include an omitted record
(count, truncated, and reason); when truncated is true on cases that
matter, raise that judge’s budget. For behavior_check, the reason names each
setting that cut evidence: SKILL_EVAL_BEHAVIOR_CHECK_BUDGET,
SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT, or both. Raise each setting it
names; raising the budget does not clear a cut made by the final-response
limit. Larger budgets send longer judge prompts and use more evaluator
tokens. The default behavior budget never falls below the final-response
limit plus 4,000 characters, so raising the limit alone leaves room for the
tool history; a budget you set yourself is used exactly as set. Set these
variables in the host environment;
SkillEvaluator forwards them to the verifier. Defaults and accepted values are
in Judge evidence budgets.
behavior_check’s omitted record carries four further keys, because
truncated alone cannot say whether the judge still had what it needed:
incomplete_sections— the evidence sections whose complete source did not reach the judge, by name:final response,user request,file changes,compact tool history. Each is compared against its unclipped source, sofile changesandcompact tool historyalso appear when a single write body or tool result was longer than its own per-entry allowance, not only when the overall budget was exceeded.required_completeandrequired_missing— whether the required evidence survived. Onlyfinal responseis required: file changes and the tool history grow with the run, so any finite budget shortens them on a long trajectory, and their loss is reported but never withholds a score.expected_behaviors_verified— whether the deterministic fact list already settles every expected behavior, in which case the evidence that was cut cannot change the judge’s answer.
When required_complete is false, behavior_check returns an evaluation
error with no score instead of grading the evidence loss as a behavior
failure, and the case then has no overall score at all — an unscored
metric that is not “not applicable” leaves overall_score null, so the case
drops out of any gate or comparison keyed on it. Raise the two settings the
reason names, or shorten the case, and re-run. The one exception is
expected_behaviors_verified: when every expected behavior names an artifact
path or backtick-quoted snippet that the deterministic fact list above the
evidence already marks observed, the missing text cannot change the answer, so
the case is judged and scored normally.
Judge requests and retries
- Structured output. Each judge requests a fixed JSON schema. OpenAI and
OpenAI-compatible chat-completions endpoints, including NVIDIA Build, receive
a strict JSON Schema
response_format; Anthropic/v1/messagesreceivesoutput_config. Bedrock judges use prompt-only JSON. - Automatic downgrade. If an endpoint rejects the schema option itself
with HTTP 400 or 422 (for example,
response_formatis unsupported, or a gateway turned the schema into atool_choicethe model rejects), the judge repeats that request without the schema and logs a warning. It then skips the schema for that provider, endpoint, and model for the remaining judge calls in that verifier run. Other 400 and 422 errors are not downgraded. - Reply validation. Replies are checked against the expected fields either
way. An empty, unparseable, or invalid reply, including an empty reply from a
reasoning model that ran out of output tokens, gets one retry that names the
problem, for example
steps not judged: [2].behavior_checkcan also salvage complete per-behavior results from a truncated reply and counts the missing behaviors as not passed. - Saved judge evidence. The verifier writes every judge attempt to
judge_attempts.jsonnext to the trial’sskill_evaluator_reward.jsonas it happens, so the file also explains a judge that failed. Each entry records the metric, attempt number, method (llm,ragas, orsalvage), judge model, status (accepted,rejected,replied, orerror), and problem, along with the prompt and reply. Whenbehavior_checkis scored from a truncated reply, the LLM attempts keep their own status (rejected, orerrorif the retry failed in transport), and anacceptedsalvageentry, numbered like the truncated attempt, holds the rebuilt results. When the RAGAS goal scorer runs, each request it sends is saved as aragasentry with its messages as the prompt and the model named in the request. Arepliedentry holds the judge’s reply: the text or the tool-call arguments RAGAS parses, or the whole reply body when it has neither, such as a refusal or a content-filter stop. Anerrorentry holds the HTTP status and body, or the reason the reply was not received. A request that fails before any response arrives, such as a refused connection, saves no entry of its own. A lastragasentry isacceptedwith the score RAGAS computed, orerrorwith the reason the prompt judge ran instead. Only that last entry counts toward--max-judge-calls; the request entries do not. Configured credentials, common key shapes, URL userinfo,Authorization, API-key, token andCookieheader values, and credential flag values (--token,--password,curl -u user:password) are masked, also at the start of a line in JSON-escaped text such as RAGAS prompts, where a line break is the two characters\n, and inside JSON-escaped quotes (-u \"user:password\"), as in tool-call arguments. That includes a value whose flag or header name was masked together with an unquoted credential value that ends the line before (--password ...\n--token ...). Values whose quotes are escaped twice or three times, as when a RAGAS prompt repeats a tool call’s arguments, and single-quoted values whose quotes are escaped once or twice, as in Python source ([\'--password\', \'...\']), are masked too. Long prompts and replies are shortened. The prompts contain the task question, expected answer, and agent evidence. Pass--harbor-keep-jobsto keep the Harbor trial directories that hold this file. - Reasoning models. GPT-5 and GPT-6 judges get no
temperatureand at least 32,768 output tokens, because reasoning tokens count against the limit; other judges get 16,384. GPT-6 judges and Claude 5.5 or later judges on the Anthropic API are sent reasoning effortmedium; other models keep their default effort. - Refusals. A safety-classifier refusal is never scored as an answer. On
the native Claude API, Claude 5.5 or later judges use server-side fallback to
claude-opus-4-8, and on Bedrock a filtered Claude 5.5 reply is retried once on Claude Opus 4.8 in the same geography.SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODELchanges that target, or turns it off withnone; a target equal to the judge model is never sent. A refusal or an empty reply then moves on to the nextLLM_JUDGE_FALLBACK_MODELSentry, if any. Each judge’s details in the trial’sskill_evaluator_reward.jsonrecordjudge_model_used,judge_reasoning_effort, andjudge_fallback_used. When a case uses different judge models across its two conditions, repeated attempts, or scoring steps of a multi-step task, it is left out of that judge metric’s Skill Lift and counted injudge_mixed_casesinlift.json. The arm scores keep all scored cases; the affected evaluator, dimension, and overall lift use the remaining eligible comparisons. - RAGAS goal scorer. With the
openaiprovider on the canonicalhttps://api.openai.com/v1endpoint and a judge model that is not a GPT-5 or GPT-6 reasoning model,goal_accuracyfirst tries the RAGASAgentGoalAccuracyWithReferencescorer through its own client, without the schema or retry handling above but within the same time budget. A RAGAS score is recorded with"method": "ragas"and the judge-model fields above. If RAGAS is unavailable, fails, or returns a score other than0.0or1.0, the verifier log shows a warning with the redacted error, and the prompt judge runs instead ("method": "custom"). A case scored by RAGAS in one condition and by the prompt judge in the other counts as a judge mix even with the same model, so it is left out of thegoal_accuracylift the same way. - Transient failures. HTTP 408, 429, and every 5xx response except 501 and
505 are retried, including 520–524 proxy errors and 529 overload responses.
Connection errors, timeouts, and Bedrock throttling or service-unavailable
errors are also retried with jittered exponential backoff. TLS certificate
verification failures are terminal, including when an SDK wraps them as a
generic connection or service-unavailable error. A
Retry-Afterheader, in seconds or as an HTTP date, sets the wait; a value above the maximum retry delay fails immediately instead of stalling. - Time budget. Each judge has its own wall-time budget (180 seconds by
default) inside the generated task’s 600-second verifier timeout, which
leaves time to write failure artifacts. Requests are cut off at the
deadline, and a retry whose wait would reach it is not attempted. If you
raise the per-judge budget, scale the verifier timeout with
--timeout-multiplierso the three judges still fit.
Retry counts, delays, and the per-judge budget come from the host environment
and are forwarded to the verifier; see
LLM retries and judge time budget.
Retries and downgrades do not change scoring formulas. A judge that still fails
is an evaluation infrastructure failure, not a 0.0 score, and the affected
condition publishes no scores; see
Tier 3 judge failure semantics.
Judge fallback and retry policies
Two opt-in host variables limit which models may answer a standard judge and
how often it is asked. Both default to the behavior above. They are forwarded
like LLM_JUDGE_FALLBACK_MODELS: only to standard verifier jobs, never to the
agent environment or custom_only runs.
LLM_JUDGE_FALLBACK_POLICY=replace. Only the judge model and yourLLM_JUDGE_FALLBACK_MODELSentries answer. The built-in refusal fallback for Claude 5.5 or later judges (SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL, defaultclaude-opus-4-8) is not used, on the native Claude API or on Bedrock. The default,append, keeps it.LLM_JUDGE_RETRY_POLICY=strict. Each judge tries at most three models: the judge model, then the first twoLLM_JUDGE_FALLBACK_MODELSentries. It sends at most two requests to each, and all of them share the judge’s time budget.- A request that fails with a retryable HTTP status, a connection error, or
Bedrock throttling is retried at most once, then the next model is tried.
A model the provider reports as not found, invalid, or not accessible to
the key moves on without a retry. That covers the errors
LLM_JUDGE_FALLBACK_MODELSalready falls back on, OpenAI’smodel_not_found, a Claudenot_found_errorthat names the model, and Bedrock’sAccessDeniedException,ResourceNotFoundException, and invalid model identifier errors. ARetry-Afterwait above the maximum retry delay moves on without waiting. A wait or request that would pass the judge’s deadline ends the judge. - A refusal, a TLS certificate failure, an authentication error, and any
other non-retryable HTTP status end the judge. So does a provider error,
even on HTTP 429, that says a quota, budget, billing, or credit limit was
reached, or that a content filter, safety, or security policy rejected
the request. Only the provider’s error code, type, and message are read
(the first 500 characters of a body that is not JSON), and only whole
phrases such as
insufficient_quota,budget exceeded,credit balance, orcontent_filtercount, so a model id or proxy host name in the error does not. These are not retried or handed to another model, and the built-in refusal fallback is not used either. - The first completed reply is final. It must be exactly one JSON object,
with only whitespace around it: no prose, Markdown fences, second object,
duplicate key, or
NaN. An empty, invalid, or truncated reply fails the judge without the format retry orbehavior_checksalvage, andgoal_accuracyskips the RAGAS scorer, which sends its own requests. - The default,
legacy, keeps the retries, fallbacks, format retry, and salvage described above.
- A request that fails with a retryable HTTP status, a connection error, or
Bedrock throttling is retried at most once, then the next model is tried.
A model the provider reports as not found, invalid, or not accessible to
the key moves on without a retry. That covers the errors
Values are case-insensitive. Any other value stops a run with standard
grading at configuration, before any trial starts. If one still reaches the
verifier, every standard judge fails before a request is sent, so the affected
conditions publish no scores. When set, run_config.json records the values as
judge.fallback_policy and judge.retry_policy. A custom_only run does not
read them.