> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.

# Live Evaluation Grading Reference

> How standard judges use trajectories, evidence budgets, retries, and fallback models.

For your first comparison, start with [Run a live evaluation](/skills/skillevaluator/tier3-live-evaluation).

## How standard grading judges a trial

With `default` or `default_plus_custom` grading, SkillEvaluator's verifier
grades each trial after the agent finishes. `security`, `skill_execution`, and
`skill_efficiency` are deterministic trajectory checks. `accuracy` and
`goal_accuracy` are LLM judges that call the configured evaluator provider.
`behavior_check` combines the behavior judge with directly evaluated declared
checks; a checks-only case makes no behavior judge call. `custom_only` runs
none of them.

`skill_efficiency` averages skill routing with tool efficiency, the share of
scored tool calls that did productive work. For Claude Code, Codex and OpenCode
tool names, reading, searching, listing, writing, editing or patching files,
running commands and activating a skill are productive, and so are MCP tools
named `mcp__server__tool` (Claude Code). Codex and OpenCode MCP calls are not
recognized as MCP calls yet, so most of them count as wasted. Help, version and
install commands and searches rooted at `/` are wasted. Planning and
bookkeeping calls, such as `TodoWrite`, `update_plan`, task lists,
background-shell polling and sub-agent calls, are not counted, and a trial with
only such calls scores 0.0. [Reports](/skills/skillevaluator/reports) lists the tool classes.

The three LLM judges return verdicts only, and the verifier computes the
scores. A `score` field in a judge reply is ignored.

Error recovery distinguishes skill files, agent command errors, and missing
runtime dependencies. Missing tools/modules declared by or used by the skill
are environment problems: recovered failures have no deduction, while an
unrecovered environment failure without an unrecovered skill fault marks the
case `environment_invalid`. `skill_execution` then returns no score, the
verifier skips the judges, and the collector leaves the trial UNSCORED
(excluded from skill score averages). Any unrecovered skill fault in the same
trial is scored as a skill failure instead. Agent-only failures remain
unscored with the environment issue, and are included in its reason and
`evaluation_errors`. Reports include the missing name and dependency
declaration/installation guidance. Declare system tools in
`scripts/apt-packages.txt`, Python packages in `scripts/requirements.txt`, or
install them in `evals/environment/Dockerfile`. Agent-chosen tools the skill
does not use remain agent errors. Missing files inside the skill remain skill
defects (0.25 deduction if recovered); real unrecovered skill/agent failures
still score zero.

* `accuracy` is passing criteria divided by five. The criteria grade the
  outcome: `ACTION_CORRECT`, `FACTUALLY_ACCURATE`, `TASK_ADDRESSED`,
  `ACTIONABLE`, and `FORMAT_FOLLOWED` (the response follows any output format
  the question explicitly requires; when none is required, it is true only if
  the response gives an answer). A response that gives no answer fails all
  five criteria and scores `0.0`, unless the expected answer is itself a
  refusal. Every case and both arms use the same
  five criteria. None of them asks whether the agent named or used the skill;
  `skill_execution` scores that. A reply without all five true/false verdicts
  is invalid.
* `goal_accuracy` is `1.0` when the judge decides the goal was `achieved` and
  `0.0` otherwise, the same binary scale as the RAGAS scorer.
* `behavior_check` includes the share of expected behaviors observed and any
  directly evaluated declared checks. When expected behaviors are present, the
  reply must judge every behavior exactly once, identified by its 1-based
  `step` number; a reply that skips, repeats, or invents a step is invalid.
  A checks-only case is evaluated directly without a judge call.

Rewards record `judge_contract: skill-evaluator-judges-v2`. Results written
before this contract, whose `accuracy` included a `SKILL_IDENTIFIED` criterion,
are not directly comparable with newer ones: a correct no-skill baseline answer
used to be capped at `0.8` on positive cases.

### The graded trajectory

The verifier grades Harbor's ATIF `trajectory.json`. When that file is missing
or has no steps, it rebuilds a trajectory from the agent's own log in the
trial's agent log directory, trying `claude-code.txt`, `cursor-cli.txt`,
`cline.txt`, `opencode.txt`, and `codex.txt` in that order:

* `opencode.txt` is OpenCode's JSON event stream. Text events become agent
  messages, and completed tool events become tool calls whose output keeps the
  tool's status, exit code, and error text.
* `codex.txt` is the Codex JSON event log. Shell commands, file changes, MCP
  tool calls, `web_search` actions (with the full search, open-page, or
  find-in-page payload), and `collab_tool_call` items become tool calls. Failed
  commands keep their `status=failed` and `exit_code=` evidence, so
  error-recovery checks still see them.

Each trial's reward records the artifact it graded in `trajectory_source`. If
neither `trajectory.json` nor a supported log yields any steps, the reward
records `No trajectory or reconstructible agent log`. Custom graders read
`trajectory.json` directly and do not get this rebuild.

### Character-sensitive judging

Set `SKILL_EVAL_JUDGE_CHARACTER_FACTS=1` in the evaluator's host environment to
append character facts to the `accuracy`, `goal_accuracy`, and `behavior_check`
prompts. For example, a literal U+00A0 (NO-BREAK SPACE) in code can look like an
ordinary U+0020 space even when the task treats them differently. The metadata
identifies non-ASCII whitespace and Unicode format (`Cf`) characters by code
point, Unicode name, position, and ASCII-escaped context. Their presence alone
does not indicate an error; the task determines which characters are permitted.

The original prompt and evidence remain intact. Metadata is added separately,
after evidence selection and compaction, with up to 20 occurrences per field
and 20 code points of context on either side. Additional occurrences are counted.
The metadata adds prompt tokens beyond the evidence budgets below.
Line and column positions are one-based Unicode code points in that field; only
LF advances the line number. Metadata cannot recover characters removed by
earlier processing, and does not identify all invisible or look-alike characters.

The option is disabled by default. Enabling it selects the prompt-based goal
judge instead of the optional RAGAS route, so keep this setting consistent when
comparing scores. Facts can help a judge interpret evidence but do not guarantee
correct judgments. Prefer executable checks for requirements they can verify.

### Evidence budgets

Each LLM judge sees a bounded evidence bundle drawn from the trajectory rather
than the full transcript. Sections are added in priority order until the
judge's character budget runs out, so lower-priority sections are dropped or
clipped first:

| Judge            | Evidence, highest priority first                                                                                                  | Budget variable                    |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| `accuracy`       | Final response, produced files and writes, the six most recent tool results                                                       | `SKILL_EVAL_ACCURACY_BUDGET`       |
| `goal_accuracy`  | Final response, end-state file changes, the twelve most recent tool results                                                       | `SKILL_EVAL_GOAL_ACCURACY_BUDGET`  |
| `behavior_check` | File changes, final response (capped by `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`), the user request, and a compact tool history | `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` |

When the case's expected output or assertions contain a backtick-quoted
snippet of four or more characters, or an output path under
`/workspace/output`, `/output`, or `/logs/agent`, every bundle starts with a
short deterministic fact list marking each one observed or not observed in the
agent's tool calls.

A judge can score only what its bundle contains. Evidence that was dropped or
clipped cannot count, so on a long trajectory a behavior that appears only in
truncated history can be judged as not observed. In the results tree, each
judge's details in a trial's `reward.json` include an `omitted` record
(`count`, `truncated`, and `reason`); when `truncated` is `true` on cases that
matter, raise that judge's budget. For `behavior_check`, the reason names each
setting that cut evidence: `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET`,
`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`, or both. Raise each setting it
names; raising the budget does not clear a cut made by the final-response
limit. Larger budgets send longer judge prompts and use more evaluator
tokens. The *default* behavior budget never falls below the final-response
limit plus 4,000 characters, so raising the limit alone leaves room for the
tool history; a budget you set yourself is used exactly as set. Set these
variables in the host environment;
SkillEvaluator forwards them to the verifier. Defaults and accepted values are
in [Judge evidence budgets](/skills/skillevaluator/environment-variables#judge-evidence-budgets).

`behavior_check`'s `omitted` record carries four further keys, because
`truncated` alone cannot say whether the judge still had what it needed:

* `incomplete_sections` — the evidence sections whose complete source did not
  reach the judge, by name: `final response`, `user request`, `file changes`,
  `compact tool history`. Each is compared against its unclipped source, so
  `file changes` and `compact tool history` also appear when a single write
  body or tool result was longer than its own per-entry allowance, not only
  when the overall budget was exceeded.
* `required_complete` and `required_missing` — whether the *required*
  evidence survived. Only `final response` is required: file changes and the
  tool history grow with the run, so any finite budget shortens them on a long
  trajectory, and their loss is reported but never withholds a score.
* `expected_behaviors_verified` — whether the deterministic fact list already
  settles every expected behavior, in which case the evidence that was cut
  cannot change the judge's answer.

When `required_complete` is `false`, `behavior_check` returns an evaluation
error with no score instead of grading the evidence loss as a behavior
failure, and **the case then has no overall score at all** — an unscored
metric that is not "not applicable" leaves `overall_score` null, so the case
drops out of any gate or comparison keyed on it. Raise the two settings the
`reason` names, or shorten the case, and re-run. The one exception is
`expected_behaviors_verified`: when every expected behavior names an artifact
path or backtick-quoted snippet that the deterministic fact list above the
evidence already marks observed, the missing text cannot change the answer, so
the case is judged and scored normally.

### Judge requests and retries

* **Structured output.** Each judge requests a fixed JSON schema. OpenAI and
  OpenAI-compatible chat-completions endpoints, including NVIDIA Build, receive
  a strict JSON Schema `response_format`; Anthropic `/v1/messages` receives
  `output_config`. Bedrock judges use prompt-only JSON.
* **Automatic downgrade.** If an endpoint rejects the schema option itself
  with HTTP 400 or 422 (for example, `response_format` is unsupported, or a
  gateway turned the schema into a `tool_choice` the model rejects), the
  judge repeats that request without the schema and logs a warning. It then
  skips the schema for that provider, endpoint, and model for the remaining
  judge calls in that verifier run. Other 400 and 422 errors are not
  downgraded.
* **Reply validation.** Replies are checked against the expected fields either
  way. An empty, unparseable, or invalid reply, including an empty reply from a
  reasoning model that ran out of output tokens, gets one retry that names the
  problem, for example `steps not judged: [2]`. `behavior_check` can also
  salvage complete per-behavior results from a truncated reply and counts the
  missing behaviors as not passed.
* **Saved judge evidence.** The verifier writes every judge attempt to
  `judge_attempts.json` next to the trial's `skill_evaluator_reward.json` as it
  happens, so the file also explains a judge that failed. Each entry records
  the metric, attempt number, method (`llm`, `ragas`, or `salvage`), judge
  model, status (`accepted`, `rejected`, `replied`, or `error`), and problem,
  along with the prompt and reply. When `behavior_check` is scored from a
  truncated reply, the LLM attempts keep their own status (`rejected`, or
  `error` if the retry failed in transport), and an `accepted` `salvage`
  entry, numbered like the truncated attempt, holds the rebuilt results.
  When the RAGAS goal scorer runs, each request it sends is saved as a `ragas`
  entry with its messages as the prompt and the model named in the request. A
  `replied` entry holds the judge's reply: the text or the tool-call arguments
  RAGAS parses, or the whole reply body when it has neither, such as a refusal
  or a content-filter stop. An `error` entry holds the HTTP status and body,
  or the reason the reply was not received. A request that fails before any
  response arrives, such as a refused connection, saves no entry of its own. A
  last `ragas` entry is `accepted` with the score RAGAS computed, or `error`
  with the reason the prompt judge ran instead. Only that last entry counts
  toward `--max-judge-calls`; the request entries do not. Configured
  credentials, common key shapes, URL userinfo, `Authorization`, API-key,
  token and `Cookie` header values, and credential flag values (`--token`,
  `--password`, `curl -u user:password`) are masked, also at the start of a
  line in JSON-escaped text such as RAGAS prompts, where a line break is the
  two characters `\n`, and inside JSON-escaped quotes (`-u \"user:password\"`),
  as in tool-call arguments. That includes a value whose flag or header name
  was masked together with an unquoted credential value that ends the line
  before (`--password ...\n--token ...`). Values whose quotes are escaped twice
  or three times, as when a RAGAS prompt repeats a tool call's arguments, and
  single-quoted values whose quotes are escaped once or twice, as in Python
  source (`[\'--password\', \'...\']`), are masked too. Long prompts and
  replies are shortened. The prompts contain the task question, expected
  answer, and agent evidence. Pass `--harbor-keep-jobs` to keep the Harbor
  trial directories that hold this file.
* **Reasoning models.** GPT-5 and GPT-6 judges get no `temperature` and at
  least 32,768 output tokens, because reasoning tokens count against the
  limit; other judges get 16,384. GPT-6 judges and Claude 5.5 or later judges
  on the Anthropic API are sent reasoning effort `medium`; other models keep
  their default effort.
* **Refusals.** A safety-classifier refusal is never scored as an answer. On
  the native Claude API, Claude 5.5 or later judges use server-side fallback to
  `claude-opus-4-8`, and on Bedrock a filtered Claude 5.5 reply is retried
  once on Claude Opus 4.8 in the same geography.
  `SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL` changes that target, or turns it off
  with `none`; a target equal to the judge model is never sent. A refusal or an empty reply
  then moves on to the next `LLM_JUDGE_FALLBACK_MODELS` entry, if any.
  Each judge's details in the trial's `skill_evaluator_reward.json` record
  `judge_model_used`, `judge_reasoning_effort`, and `judge_fallback_used`.
  When a case uses different judge models across its two conditions,
  repeated attempts, or scoring steps of a multi-step task, it is left out of
  that judge metric's Skill Lift and counted in `judge_mixed_cases` in
  `lift.json`. The arm scores keep all scored cases; the affected evaluator,
  dimension, and overall lift use the remaining eligible comparisons.
* **RAGAS goal scorer.** With the `openai` provider on the canonical
  `https://api.openai.com/v1` endpoint and a judge model that is not a GPT-5 or
  GPT-6 reasoning model, `goal_accuracy` first tries the RAGAS
  `AgentGoalAccuracyWithReference` scorer through its own client, without the
  schema or retry handling above but within the same time budget. A RAGAS
  score is recorded with `"method": "ragas"` and the judge-model fields above.
  If RAGAS is unavailable, fails, or returns a score other than `0.0` or
  `1.0`, the verifier log shows a warning with the redacted error, and the
  prompt judge runs instead (`"method": "custom"`). A case scored by RAGAS in
  one condition and by the prompt judge in the other counts as a judge mix
  even with the same model, so it is left out of the `goal_accuracy` lift the
  same way.
* **Transient failures.** HTTP 408, 429, and every 5xx response except 501 and
  505 are retried, including 520–524 proxy errors and 529 overload responses.
  Connection errors, timeouts, and Bedrock throttling or service-unavailable
  errors are also retried with jittered exponential backoff. TLS certificate
  verification failures are terminal, including when an SDK wraps them as a
  generic connection or service-unavailable error. A `Retry-After` header,
  in seconds or as an HTTP date, sets the wait; a value
  above the maximum retry delay fails immediately instead of stalling.
* **Time budget.** Each judge has its own wall-time budget (180 seconds by
  default) inside the generated task's 600-second verifier timeout, which
  leaves time to write failure artifacts. Requests are cut off at the
  deadline, and a retry whose wait would reach it is not attempted. If you
  raise the per-judge budget, scale the verifier timeout with
  `--timeout-multiplier` so the three judges still fit.

Retry counts, delays, and the per-judge budget come from the host environment
and are forwarded to the verifier; see
[LLM retries and judge time budget](/skills/skillevaluator/environment-variables#llm-retries-and-judge-time-budget).
Retries and downgrades do not change scoring formulas. A judge that still fails
is an evaluation infrastructure failure, not a `0.0` score, and the affected
condition publishes no scores; see
[Tier 3 judge failure semantics](/skills/skillevaluator/environment-variables#tier-3-judge-failure-semantics).

### Judge fallback and retry policies

Two opt-in host variables limit which models may answer a standard judge and
how often it is asked. Both default to the behavior above. They are forwarded
like `LLM_JUDGE_FALLBACK_MODELS`: only to standard verifier jobs, never to the
agent environment or `custom_only` runs.

* **`LLM_JUDGE_FALLBACK_POLICY=replace`.** Only the judge model and your
  `LLM_JUDGE_FALLBACK_MODELS` entries answer. The built-in refusal fallback
  for Claude 5.5 or later judges (`SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL`,
  default `claude-opus-4-8`) is not used, on the native
  Claude API or on Bedrock. The default, `append`, keeps it.
* **`LLM_JUDGE_RETRY_POLICY=strict`.** Each judge tries at most three models:
  the judge model, then the first two `LLM_JUDGE_FALLBACK_MODELS` entries. It
  sends at most two requests to each, and all of them share the judge's time
  budget.
  * A request that fails with a retryable HTTP status, a connection error, or
    Bedrock throttling is retried at most once, then the next model is tried.
    A model the provider reports as not found, invalid, or not accessible to
    the key moves on without a retry. That covers the errors
    `LLM_JUDGE_FALLBACK_MODELS` already falls back on, OpenAI's
    `model_not_found`, a Claude `not_found_error` that names the model, and
    Bedrock's `AccessDeniedException`, `ResourceNotFoundException`, and
    invalid model identifier errors. A `Retry-After` wait above the maximum
    retry delay moves on without waiting. A wait or request that would pass
    the judge's deadline ends the judge.
  * A refusal, a TLS certificate failure, an authentication error, and any
    other non-retryable HTTP status end the judge. So does a provider error,
    even on HTTP 429, that says a quota, budget, billing, or credit limit was
    reached, or that a content filter, safety, or security policy rejected
    the request. Only the provider's error code, type, and message are read
    (the first 500 characters of a body that is not JSON), and only whole
    phrases such as `insufficient_quota`, `budget exceeded`,
    `credit balance`, or `content_filter` count, so a model id or proxy host
    name in the error does not. These are not retried or handed to another
    model, and the built-in refusal fallback is not used either.
  * The first completed reply is final. It must be exactly one JSON object,
    with only whitespace around it: no prose, Markdown fences, second object,
    duplicate key, or `NaN`. An empty, invalid, or truncated reply fails the
    judge without the format retry or `behavior_check` salvage, and
    `goal_accuracy` skips the RAGAS scorer, which sends its own requests.
  * The default, `legacy`, keeps the retries, fallbacks, format retry, and
    salvage described above.

Values are case-insensitive. Any other value stops a run with standard
grading at configuration, before any trial starts. If one still reaches the
verifier, every standard judge fails before a request is sent, so the affected
conditions publish no scores. When set, `run_config.json` records the values as
`judge.fallback_policy` and `judge.retry_policy`. A `custom_only` run does not
read them.