Grader Validation
Grader validation measures agreement with independent labels, not agent quality. It runs the real shipped verifier, including deterministic security, execution and efficiency checks and the configured accuracy, goal and behavior judges. It does not change judge prompts or scoring.
Quickstart
Check the bundled reference suite without credentials or model calls:
Configure a judge provider, then grade held-out assessment cases three times each:
The bundled suite contains 14 assessment and 2 tuning cases with hand-written calculator trajectories. It demonstrates every category, including recovery, missing evidence, correct non-use, a baseline without the skill and mixed per-step behavior. It is intentionally small and is not a reliability benchmark.
Suite contract
A directory contains suite.json and referenced ATIF trajectory JSON files.
The exact schema identifier is skillevaluator.grader-qualification/v1.
Unknown fields are rejected in every structured object, including entries and
checks. Case IDs are unique letters, digits, hyphens and underscores; paths must
be relative files inside the suite directory (including after symlink resolution).
Required case fields are shown above. Categories are positive, negative,
edge, failure_recovery, unavailable_evidence, non_use, and baseline.
Source kind, origin and trace ID are nonempty strings identifying the evidence.
Splits are assessment (held out) and tuning (used for development).
Entry fields accepted by this mode are id, question, ground_truth,
expected_behavior, expected_skill, expected_script, evaluated_skill,
should_trigger, has_skill, acceptable_skills, acceptable_alternates,
skill_workspace_mode, workspace_skill_names and checks. Question is required;
behavior, accepted skills and workspace skill names are lists of strings;
trigger and installed-skill flags are booleans. The verifier receives these
unchanged. Baseline skill metrics are N/A; a trusted evaluated skill identity
lets a non-use case receive credit for correctly avoiding that skill.
entry.checks accepts the dataset’s declarative checks,
including anchored output_matches patterns for exact final-response formats.
These run in the shipped verifier and contribute to Behavior alongside any
remaining expected_behavior judgments. file_exists is rejected: qualification
records only the trajectory, not the agent’s workspace files, so the check would
inspect the machine running qualification instead of the agent’s output.
When a case declares checks, label the
combined Behavior result with a scalar pass/fail. A boolean-list Behavior
label is rejected there, because it describes only the numbered model-judged
behaviors and cannot say whether the direct checks pass. label.checks below has a
different purpose: it verifies recorded evidence supporting the author’s label.
Expected overall is pass, fail or insufficient_evidence. Metric labels
also allow not_applicable; omit metrics without independent labels.
Behavior can instead be a boolean list exactly matching expected behavior length
(only for a case without checks).
A null trajectory is allowed only for unavailable-evidence cases labelled
insufficient evidence. Missing referenced files are errors, not abstentions.
Label metadata is required even for executable labels. Use reviewed with a
rationale for assertions that cannot be expressed mechanically.
Executable labels require at least one check. Checks assert recorded evidence:
final_response_contains, final_response_not_contains, tool_result_contains
each take a nonempty value; file_written_contains additionally takes path
and checks a recorded Write/write_file tool call, not a live filesystem file.
All checks run whenever the suite is loaded, even with --check-suite.
These checks anchor a reviewed label; a substring alone does not prove semantic
correctness. Authors must independently establish the expected outcome.
Reading the report
Each attempt runs the template in a separate subprocess with an isolated entry,
trajectory and reward directory. Attempts inherit provider credentials but
provenance does not record keys. --concurrency defaults to 1 and --timeout
to 300 seconds per attempt. Output directories must be empty to avoid mixing
old and new evidence.
Scores at or above the metric threshold pass. Defaults are accuracy 0.8, goal accuracy 1.0, security/execution/efficiency/behavior 1.0, and overall 0.5 (the Harbor pass threshold). Suite thresholds override defaults.
Accuracy and Behavior judge prompts instruct the judge to check explicit
final-answer formats against the entire final response. Correct values in tool
output or references do not establish format compliance. This clarification
does not add a critical-failure gate or establish improved semantic accuracy.
Accuracy still averages five criteria: if only FORMAT_FOLLOWED is false, its
score is 0.8 and passes the default accuracy threshold.
For exact output formats, declare an anchored output_matches check instead of
asking the judge to enforce a natural-language format rule. For example,
{"type": "output_matches", "pattern": "\\A[+-]?\\d+(?:\\.\\d+)?\\Z"}
requires the entire response to be a signed or unsigned integer/decimal.
Use \A and \Z; $ can match before a trailing newline. See the
dataset examples.
Keep semantic requirements in expected_behavior, and avoid counting the same
format rule in both lists. This configures an existing deterministic route;
it does not improve the language-model judge.
A live format diagnostic at commit 0d5e356583fee63c3d12e4930581c7dfb3352911
(0.9.0), using nv_build / nvidia/nemotron-3.5-bolt-e4b, measured Behavior
false passes of 9/12 and false fails of 3/18 over 10 distinct synthetic cases
with three dependent repeats (30 attempts). A separately frozen 20-case
author-labelled set, also repeated three times (60 attempts), measured 15/33
false passes and 3/27 false fails; it contains 18 format cases and two semantic
controls. Both had zero unjudgeable Behavior attempts. These labels are not
independent human labels. The wording and narrow synthetic coverage limit
these findings; they establish a known natural-language format limitation,
not representative judge reliability or an agreed acceptance threshold.
overall is the score policy’s dimension mean, compared with the default 0.5
threshold. The shipped policy averages security, accuracy, skill execution,
goal achievement (the mean of goal_accuracy and behavior_check), and skill
efficiency. N/A metrics are omitted from their dimension; dimensions with no
scored metrics are omitted from the overall mean. A known task failure with
clean deterministic dimensions can therefore still receive an overall pass.
The suite reports this as a false pass against the independent overall label
rather than hiding it or changing the verifier’s scoring.
Per-step behavior uses the verifier’s boolean results, not a new judge. N/A comes from detail status; missing trajectory is insufficient evidence. Judge failure is an unscored error, including overall when required judging failed. A process failure or timeout without a valid failure artifact is an error.
For every labelled metric, the behavior-step pool, and overall, JSON reports:
- Confusion matrix: expected pass/fail rows by observed pass, fail, N/A, error and insufficient evidence. Other expected labels have their own rows.
- False-pass rate: observed passes among known-fail attempts divided by known-fail attempts observed pass or fail. Errors, N/A and abstentions do not enter that denominator.
- False-fail rate: observed fails among known-pass attempts divided by known-pass attempts observed pass or fail.
- Unjudgeable rate: errors plus insufficient-evidence observations divided by all attempts with known pass/fail labels. Both counts are reported separately. N/A is separate, not counted as an error or as a scored attempt.
- Disagreement: count of any label/verdict mismatch, including N/A or errors.
- Expected-insufficient evidence: total attempts, correct abstentions, confident pass/fail verdicts and errors. An error is not a correct abstention.
- Variability: non-unanimous judgments divided by judgments repeated more than once. A judgment is one case/metric or case/behavior-step, not an attempt. Numeric score spreads are max minus min for each judgment, excluding missing scores. Verdict unanimity and score stability are different concepts.
- Counts: attempts, distinct judgments, cases, expected class counts, category and split counts, missing categories, cases with trajectories, and truncation count over details carrying omission metadata. The pooled behavior denominator counts steps, not just cases.
Rates include numerator, denominator and a Wilson 95% interval. Zero denominators produce null rates and intervals, not zero. Attempt-level intervals treat attempts as independent; repeated grades of one case are dependent, so these are descriptive bounds, not a substitute for independent case sampling. Metric kinds distinguish deterministic checks, model judgments and composite overall.
grader-validation.json contains the full report. Each case/attempt has
reward.json (Harbor numeric payload), skill_evaluator_reward.json (rich
details and contract metadata), judge_attempts.json and
expected-vs-observed.json. Contract provenance is read from the rich sidecar
because the shipped verifier intentionally keeps Harbor’s reward numeric-only.
Reproducible qualification procedure
- Establish outcomes independently using executable assertions and review. Record case/trace identity, reviewer, date, rationale and suite version.
- Freeze the assessment cases before changing judge configuration. Develop
on
--split tuning; never use assessment mistakes as undisclosed tuning data. - Validate the suite offline. Save its source and trajectories alongside results.
- Pin the package, provider/model, thresholds and non-secret judge settings. Run repeats with an explicit output directory.
- Read class counts and denominators before comparing rates. Inspect errors, abstentions, N/A, truncated evidence and repeated-grade variation separately.
- Save the report with its timestamp, selected case IDs, split, package version, verifier SHA-256, tracked-source Git commit and verifier-file dirty state when available, suite content SHA-256, judge contracts, actual models used (including fallback), goal method, thresholds, repeats and concurrency. The suite digest frames each relative filename, byte length and contents, covering suite.json plus all referenced trajectories, even unselected ones.
- Add independently reviewed cases when coverage gaps matter; version changes and rerun the frozen assessment suite. Do not claim a particular case count establishes statistical sufficiency.
Default output covers assessment only. --split all emits separate
by_split.assessment and by_split.tuning statistics with no mixed aggregate.
The top-level metrics object is empty ({}) in this mode; read the metric
summaries under those two by_split objects instead.
Explicit gates on an all-split run apply only to assessment.
Optional automation gates
Gates have no defaults. Exit 1 means at least one supplied rate limit was
strictly exceeded for any reported metric or the behavior-step pool; equality
passes. Undefined rates do not trip a gate: always inspect coverage. Exit 2
means invalid usage, suite, output path or provider configuration. Without gates,
disagreement and judge errors are reported rather than changing the exit code.
--check-suite never requires a provider.
Explicit qualification policy
--qualification-policy FILE adds operator-owned coverage requirements and risk
limits to an assessment-only run. It is off by default. --split tuning and
--split all with a policy are usage errors; development data cannot fill held-out
coverage. --case narrows the assessment scope, so the remaining cases must meet
the complete policy themselves. Existing standalone --max-*-rate gates remain
available and keep their existing behavior, including ignoring undefined rates.
A policy cannot be combined with any standalone --max-*-rate gate. Put all
qualification criteria in the retained policy file; combining them is usage exit 2
before provider access, including with --check-suite.
Review and freeze an immutable assessment suite and explicit policy before a run. The repository’s experimental example is an example for review, not a default or a release policy:
The example requires all eight metrics, at least one distinct case in each expected class per metric and at least one case per category. Its proposed limits are false-pass 0.10, false-fail 0.10 and unjudgeable 0.05. These availability floors do not establish statistical sufficiency. The bundled suite fails the example’s offline coverage check: security and skill_efficiency have no independently labelled failing cases. Its bytes and labels are unchanged. Do not weaken labels to obtain a pass; separately review and version representative evidence.
A policy is a regular UTF-8 JSON file no larger than 256 KiB with these fields:
Unknown or duplicate object fields, duplicate metrics, booleans used as numbers,
nonfinite numbers, malformed JSON and excessive parser complexity are rejected.
The report’s qualification stores the complete effective policy, policy_id
and policy_sha256. The SHA-256 covers UTF-8 canonical JSON with sorted keys,
compact separators, ASCII escapes and no nonfinite numbers. Whitespace and object
key ordering do not change the hash; policy content does.
qualification.scope names the selected assessment case IDs. Each required metric
has expected and scored distinct-case counts for each class, minimum checks, and
rate checks containing the existing measurement, limit and result. Category
checks separately report expected_cases, observed_cases, expected_met and
observed_met. A scored case has at least one observed pass/fail against that metric’s expected pass/fail label. Behavior-step
coverage counts a case once per class, even if it has many steps in that class;
one case with both pass and fail steps can count once in each. Aggregate
behavior_check expected classes use the suite’s metric threshold. Repeats never
create additional cases. A category’s observed count includes cases with a required
known-class observation scored pass/fail, or a required explicitly expected N/A or
insufficient-evidence observation correctly matched. Correct abstentions demonstrate
representative category evidence without creating binary scored-class cases or
rate denominators. Errors, unexpected abstentions and confident verdicts against
expected-abstention labels cannot fill observed category coverage. Missing category
criteria impose no category minimum.
Rates and Wilson intervals retain the attempt-level denominators described above;
no new success or scoring formula is introduced. Evaluated report attempt IDs
and any declared provenance case IDs/split must match the selected assessment
scope; the evaluator rejects broader/subset scope mismatches instead of borrowing
unselected cases’ rates. Observation case identities must match their attempt too.
In particular, zero denominators produce unavailable rates, never accepted zero-error results. Structured
qualification.reasons identify missing metrics/rates, class/category coverage
shortfalls and exceeded limits; human output names the same causes.
--check-suite with a policy prints suite validation and coverage readiness, exits
1 for insufficient expected labels, and writes no output. A live command with
insufficient expected labels first requires an empty output directory, then saves
grader-validation.json with execution_status: not_run, attempts: [], null
provider/model and the preflight decision. A covered live command runs the shipped
verifier and saves its final decision before writing the receipt. --json prints
the same complete report as the retained JSON. Policy failure or insufficient
evidence exits 1; malformed policy/options remain usage exit 2. Without a policy,
the command preserves its original gates and split behavior.
Provenance retains suite/verifier digests and contract IDs.
verifier_git_commit and verifier_git_dirty disclose the source checkout’s HEAD
and whether any actual verifier file has staged or unstaged changes, only when
all actual verifier files are tracked in that repository. Installed or untracked
copies and unavailable Git metadata produce null for both fields, even inside an
unrelated checkout. Unrelated dirty files do not make the verifier dirty. Git
metadata is read-only and does not change baseline reuse eligibility.
Run offline schema/executable-label checks on relevant changes. Manually trigger and retain live qualification when scorer prompts/code, judge/provider/model or the reviewed assessment suite changes, and before accepting a release candidate. Review the frozen suite, interpretation and approved cost bound before each paid run. This feature introduces no paid schedule or release-blocking workflow. This PR establishes neither live grader reliability nor release approval; those require separately reviewed evidence and an explicit promotion decision. Raw resource comparisons remain separate from scored Efficiency and Overall.
Separate diagnostic availability pilot
The separately versioned four-case pilot adds a forbidden shell-startup Write and a wasteful root search, each with a matched control. The security control quotes the same unsafe Write in read-only README output and issues no write. The Efficiency control searches only within the supplied calculator directory. These are public synthetic ATIF fixtures, not observed agent trajectories. The original reference suite is preserved.
Its explicit policy requires only security and skill_efficiency, one distinct
pass and fail case each, and false-pass/false-fail/unjudgeable limits of
0.10/0.10/0.05. No category floor is justified by this four-case design. Offline
ready establishes label availability only; current deterministic component
checks are supplemental conformance evidence. The
independent worksheet
derives labels from task, safety and search-scope requirements and keeps task
Overall failures separate from composite-score predictions. Other metric
labels are omitted where a full independent criterion oracle is absent.
Codex authored and self-reviewed the pilot; independent automated review is pending, and no human maintainer approval is claimed. Its bounded content manifest identifies the source revision, suite, policy, trajectories, worksheet and README. Retain actual source/model/run identities and any later review separately.
This diagnostic availability pilot establishes neither all-metrics
qualification, statistical representativeness, release approval nor Jira
acceptance. Its targeted policy does not satisfy experimental-all-metrics-v1.
Do not pool its rates with the preserved reference-suite receipt or relabel
task failures to agree with the composite. Three repeats still represent only
one distinct pass and fail case per target metric. No live qualification result
is claimed by these files.
Measured baseline
On 2026-11-07, SkillEvaluator commit
0205909b653f139602fc2e7681ba7d0112c0871a graded the bundled assessment split
with the openai-compatible provider: 14 cases × 3 repeats = 42 attempts per
model. Suite SHA-256 prefix: 5e85ceac…; the then-recorded eval.py SHA-256
prefix was d444e2c4….
The fractions below use the scored known-fail and known-pass denominators
defined above, not all 42 attempts.
For both models, the failure-recovery case received skill_execution 0.975, below its 1.0 threshold. Every known task failure exceeded the overall 0.5 dimension-mean threshold. There were no unjudgeable observations against pass/fail labels; all 3 unavailable-evidence attempts correctly abstained.
These results describe this suite, scorer and model only. They are a baseline,
not an acceptance threshold. No gate thresholds or CI cadence for live runs
are set yet; offline --check-suite runs in the test suite.
Limitations
Results describe this suite, scorer and model only. A fixed case count or score band is not evidence of statistical sufficiency or universal grader reliability. Labels can be wrong and synthetic evidence may not represent real agents. Agreement is not proof of causal validity. Repeated attempts can be correlated, providers can change, and a model may have seen public examples before.