nemo_gym.eval.recovery.grading
nemo_gym.eval.recovery.grading
Saved grading artifacts: whether a saved row’s grade is recorded, and why a saved row is re-verified.
Module Contents
Functions
Data
API
Whether the collector bound the saved answer and score together (_ng_collected_grading_binding).
COLLECT_SCRIPT’s own projection name, so an in-process reader projects a saved row as the collector does.
Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw
without non-whitespace text (has_text false). A message whose content was not saved counts as an answer.
The environment’s own reward record: result.reward_info.reward equal to the row’s reward.
Whether the row’s verifier invocations were observed but recorded no grading: no grading operation, no
verifier grading record, no saved grading artifact (grading_artifact) and no model-failure rule
(_model_failure). A row from a pin without verifier observations proves nothing and is not selected. Nor is
one whose verifier got an answer from a model (_verifier_model_answered) and wrote no judge-error marker
(_judge_error_marker): a judge the verifier does not observe (observe_llm_judge_grading) records no
grading however often it is asked again, and a grade it gave is never replaced. A row whose model calls all
failed, or that names a judge error, was not graded.
A guardrail-gated objective score the agent’s own rubric saved (response.metrics): the row’s reward equals
the saved aa_headline, and that headline follows from the saved counts: 0.0 when a guardrail was violated,
else the objectives passed over the objectives (0.0 without objectives). A row without any counted assertion (no
world or no assertions also scores 0.0) is no evidence and stays unobserved. The counts are the rubric’s own, not
a re-run of its assertions. The saved answer and score are bound.
Whether the row’s saved answer is a response left incomplete, without an error. An incomplete status alone says nothing about why: an agent also marks an answer incomplete that holds no message.
Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a top-level message) or an invalid judge response (top level or inside the rubric facts).
A non-empty judge_evaluations list without a judge error; when every evaluation’s verdict label parses
([[A=B]]/[[A!=B]] or Judgement: yes/no), the reward is 1.0 exactly when every verdict is equal/yes.
A saved judge verdict: a judge_response, a non-empty judgments list with a judge_score, or
non-empty judge_evaluations whose parsed verdicts agree with the reward (_judge_evaluations).
Whether a scored row is judge_unparseable: its judge gave no verdict (_unparsed_judge_verdict) or its
verifier’s grading did not resolve (_unknown_grading_outcome). Judges are stateless: it is re-judged.
Saved in-process metric scores: numeric sentence_* fields (sentence_chrf, sentence_spbleu).
The benchmark’s rule scored the model’s own failed answer without grading it: the verifier marked the answer
invalid (invalid_model_response), or the model left it incomplete, or it is empty and scored zero.
A native scorer’s saved score (native_metrics): the row’s reward equals the saved weighted_score in
[0, 1], the scorer saved its binary execution flag and its diagnostics. The score is the scorer’s own, not a
re-run of it. The saved answer and score are bound.
A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer’s final
Final Answer: [...] line, reproduces the saved predicted nodes and parse outcome, and scores them against the
saved expected node list as the verifier does; the row’s reward equals that re-graded F1 and the saved f1.
The saved answer and score are bound.
A saved repository test run of the answer’s patch: evaluation_completed true, the run’s resolved
verdict with the row’s reward equal to it (1.0 resolved, 0.0 not), saved test_results, no error, no
evaluation timeout, and a resolved verdict only for a patch that applied. The saved answer and score are bound.
A saved rubric judge’s verdict: verdict YES or NO, a finite rubric_score and scoring_threshold, no
invalid_judge_response, and the reward 1.0 exactly when the verdict is YES and the score reaches the threshold
(0.0 otherwise). The saved answer and score are bound.
Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector’s projected
facts, else the same checks on a full saved row (saved_model_patch, collected_patch). Only a verifier that
collects the patch itself can be given it back; one that reads the agent session at verify time cannot.
A scored row: unmasked, with a reward and no failure kind.
The verifier’s own reward file of the row’s session, read by the collector beside the artifacts
(resources/<session_id>/verifier/reward.json): the evaluation completed with no infrastructure error and the
row’s reward equals the file’s reward. Test counts and partial credit are metadata, never the grade. The saved
answer and score are bound.
Saved per-sub-step test results: a non-empty step_results list of booleans, one environment result per
step, num_steps_total and num_steps_passed equal to its length and its passes, problem_accuracy true
exactly when every step passed, and the row’s reward equal to it (1.0 or 0.0). The saved answer and score are
bound.
A saved symbolic answer check: a library_reward the row’s reward equals, or one beside a judge verdict
(the judge fallback).
A saved test run: evaluation_completed true with an integer verifier_exit_code and no
verifier_error.
Saved unit test results of the answer’s extracted code: at least one test, and the row’s reward 1.0 exactly
when every test passed (0.0 otherwise), or 0.0, the default penalty, for an answer whose reasoning format
violated the rule (reasoning_format_violation_rate 1.0). A configured non-zero penalty is not saved on the row,
so such a row stays unobserved. The saved answer and score are bound.
Whether a scored row’s verifier record reports a judge grading operation with outcome unknown: the grading
did not resolve, so its reward is no grade. Read from the verifier invocations the row already saves.
Whether a scored row’s grading is judge_unparseable for a verdict the judge did not give: a saved judge
evaluation recorded no verdict (_unparsed_verdict), or a pairwise game has none (games_unjudged). Its
reward is no grade.
Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.
Whether one model call of the row’s verifiers completed with a 2xx and a usable response, read from the call the row saved for each reference (by call id, else response id). A call the row does not hold has not answered.
Whether event and every explicit cause below it name only a lost verify call: an agent route failure or a
transport kind, with no reply status. A verifier’s own failure or an HTTP reply anywhere in the chain is not one.
Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure
or a transport kind, and a saved failure event of the verify operation has a transport-only cause chain.
The name of the first saved grading artifact (GRADING_ARTIFACTS) that records the row’s grading.
The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None.
A masked result masked_infrastructure_kind rejects only because it kept a saved answer (so every other rule of
that predicate applies: the kind must be registered, evaluation-side and not graded again), that has no recorded
grade (has_recorded_grade) and no re-verification cause (reverification_cause): its saved answer has no
gradeable artifact, a resume keeps it and re-verification cannot grade it. It runs again only with the explicit
--rerun-masked-without-artifact. row is a collector projection, or a saved row passed through
saved_row_evidence.
Why a saved result row is re-verified from its saved answer, or None when it is not.
REVERIFY_JUDGE: masked because its judge failed (judge_failed, judge_unparseable), or scored while
its judge gave no verdict or its grading did not resolve (_judge_unparseable).
REVERIFY_VERIFY_TRANSPORT: masked because its verify call was lost in transport (_verify_transport_failure).
REVERIFY_VERIFIER: masked because its verifier produced no verdict (verifier_error), with the workspace
patch it graded saved intact (saved_model_patch): the verifier runs again on that patch, never the agent.
REVERIFY_UNGRADED: scored, but no grading was recorded for it (_grading_unrecorded).
Only a result with a saved answer qualifies; a failures-sidecar row, a result a re-verification already settled
(its settlement is its grading) and every other masked result do not. row is a collector projection, or a
saved row passed through saved_row_evidence.
A full saved row with the evidence fields COLLECT_SCRIPT projects for the grading predicates: its scalar
grading facts, whether each assistant message of its answer has text, and its verifier invocations.