Skip to navigation

nemo_gym.eval.recovery.grading

View as MarkdownOpen in Claude

Saved grading artifacts: whether a saved row’s grade is recorded, and why a saved row is re-verified.

Module Contents

Functions

NameDescription
_bound_answerWhether the collector bound the saved answer and score together (_ng_collected_grading_binding).
_collector_functionCOLLECT_SCRIPT’s own projection name, so an in-process reader projects a saved row as the collector does.
_empty_answerWhether the saved answer holds no assistant message with text: none at all, or only ones the collector saw
_environment_rewardThe environment’s own reward record: result.reward_info.reward equal to the row’s reward.
_grading_facts-
_grading_unrecordedWhether the row’s verifier invocations were observed but recorded no grading: no grading operation, no
_guarded_objective_scoreA guardrail-gated objective score the agent’s own rubric saved (response.metrics): the row’s reward equals
_incomplete_answerWhether the row’s saved answer is a response left incomplete, without an error. An incomplete status alone
_judge_error_markerWhether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a
_judge_evaluationsA non-empty judge_evaluations list without a judge error; when every evaluation’s verdict label parses
_judge_recordA saved judge verdict: a judge_response, a non-empty judgments list with a judge_score, or
_judge_unparseableWhether a scored row is judge_unparseable: its judge gave no verdict (_unparsed_judge_verdict) or its
_metric_recordSaved in-process metric scores: numeric sentence_* fields (sentence_chrf, sentence_spbleu).
_model_failureThe benchmark’s rule scored the model’s own failed answer without grading it: the verifier marked the answer
_native_scorer_scoreA native scorer’s saved score (native_metrics): the row’s reward equals the saved weighted_score in
_node_set_f1A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer’s final
_repository_test_runA saved repository test run of the answer’s patch: evaluation_completed true, the run’s resolved
_reward-
_rubric_verdictA saved rubric judge’s verdict: verdict YES or NO, a finite rubric_score and scoring_threshold, no
_saved_model_patchWhether the row saved, intact, the workspace patch its verifier collected and graded: the collector’s projected
_scored_unmaskedA scored row: unmasked, with a reward and no failure kind.
_session_reward_fileThe verifier’s own reward file of the row’s session, read by the collector beside the artifacts
_substep_testsSaved per-sub-step test results: a non-empty step_results list of booleans, one environment result per
_symbolic_matchA saved symbolic answer check: a library_reward the row’s reward equals, or one beside a judge verdict
_test_executionA saved test run: evaluation_completed true with an integer verifier_exit_code and no
_transport_kind-
_unit_test_resultsSaved unit test results of the answer’s extracted code: at least one test, and the row’s reward 1.0 exactly
_unknown_grading_outcomeWhether a scored row’s verifier record reports a judge grading operation with outcome unknown: the grading
_unparsed_judge_verdictWhether a scored row’s grading is judge_unparseable for a verdict the judge did not give: a saved judge
_unparsed_verdictSaved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.
_verifier_model_answeredWhether one model call of the row’s verifiers completed with a 2xx and a usable response, read from the call
_verify_transport_chainWhether event and every explicit cause below it name only a lost verify call: an agent route failure or a
_verify_transport_failureWhether the row was masked because its verify call was lost in transport: its kind is an agent route failure
grading_artifactThe name of the first saved grading artifact (GRADING_ARTIFACTS) that records the row’s grading.
masked_without_artifact_kindThe kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None.
reverification_causeWhy a saved result row is re-verified from its saved answer, or None when it is not.
saved_row_evidenceA full saved row with the evidence fields COLLECT_SCRIPT projects for the grading predicates: its scalar

Data

GRADING_ARTIFACTS

REVERIFY_JUDGE

REVERIFY_UNGRADED

REVERIFY_VERIFIER

REVERIFY_VERIFY_TRANSPORT

_COLLECTOR_FUNCTIONS

_CUT_FINISH

_FAILED_RESPONSE_STATUSES

_REVERIFY_JUDGE_KINDS

API

nemo_gym.eval.recovery.grading._bound_answer(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the collector bound the saved answer and score together (_ng_collected_grading_binding).

nemo_gym.eval.recovery.grading._collector_function(
name: str
)

COLLECT_SCRIPT’s own projection name, so an in-process reader projects a saved row as the collector does.

nemo_gym.eval.recovery.grading._empty_answer(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw without non-whitespace text (has_text false). A message whose content was not saved counts as an answer.

nemo_gym.eval.recovery.grading._environment_reward(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

The environment’s own reward record: result.reward_info.reward equal to the row’s reward.

nemo_gym.eval.recovery.grading._grading_facts(
row: collections.abc.Mapping[str, typing.Any]
) -> collections.abc.Mapping[str, typing.Any]
nemo_gym.eval.recovery.grading._grading_unrecorded(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the row’s verifier invocations were observed but recorded no grading: no grading operation, no verifier grading record, no saved grading artifact (grading_artifact) and no model-failure rule (_model_failure). A row from a pin without verifier observations proves nothing and is not selected. Nor is one whose verifier got an answer from a model (_verifier_model_answered) and wrote no judge-error marker (_judge_error_marker): a judge the verifier does not observe (observe_llm_judge_grading) records no grading however often it is asked again, and a grade it gave is never replaced. A row whose model calls all failed, or that names a judge error, was not graded.

nemo_gym.eval.recovery.grading._guarded_objective_score(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A guardrail-gated objective score the agent’s own rubric saved (response.metrics): the row’s reward equals the saved aa_headline, and that headline follows from the saved counts: 0.0 when a guardrail was violated, else the objectives passed over the objectives (0.0 without objectives). A row without any counted assertion (no world or no assertions also scores 0.0) is no evidence and stays unobserved. The counts are the rubric’s own, not a re-run of its assertions. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._incomplete_answer(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the row’s saved answer is a response left incomplete, without an error. An incomplete status alone says nothing about why: an agent also marks an answer incomplete that holds no message.

nemo_gym.eval.recovery.grading._judge_error_marker(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a top-level message) or an invalid judge response (top level or inside the rubric facts).

nemo_gym.eval.recovery.grading._judge_evaluations(
facts: collections.abc.Mapping[str, typing.Any],
reward: float | None
) -> bool

A non-empty judge_evaluations list without a judge error; when every evaluation’s verdict label parses ([[A=B]]/[[A!=B]] or Judgement: yes/no), the reward is 1.0 exactly when every verdict is equal/yes.

nemo_gym.eval.recovery.grading._judge_record(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved judge verdict: a judge_response, a non-empty judgments list with a judge_score, or non-empty judge_evaluations whose parsed verdicts agree with the reward (_judge_evaluations).

nemo_gym.eval.recovery.grading._judge_unparseable(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether a scored row is judge_unparseable: its judge gave no verdict (_unparsed_judge_verdict) or its verifier’s grading did not resolve (_unknown_grading_outcome). Judges are stateless: it is re-judged.

nemo_gym.eval.recovery.grading._metric_record(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Saved in-process metric scores: numeric sentence_* fields (sentence_chrf, sentence_spbleu).

nemo_gym.eval.recovery.grading._model_failure(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

The benchmark’s rule scored the model’s own failed answer without grading it: the verifier marked the answer invalid (invalid_model_response), or the model left it incomplete, or it is empty and scored zero.

nemo_gym.eval.recovery.grading._native_scorer_score(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A native scorer’s saved score (native_metrics): the row’s reward equals the saved weighted_score in [0, 1], the scorer saved its binary execution flag and its diagnostics. The score is the scorer’s own, not a re-run of it. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._node_set_f1(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer’s final Final Answer: [...] line, reproduces the saved predicted nodes and parse outcome, and scores them against the saved expected node list as the verifier does; the row’s reward equals that re-graded F1 and the saved f1. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._repository_test_run(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved repository test run of the answer’s patch: evaluation_completed true, the run’s resolved verdict with the row’s reward equal to it (1.0 resolved, 0.0 not), saved test_results, no error, no evaluation timeout, and a resolved verdict only for a patch that applied. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._reward(
row: collections.abc.Mapping[str, typing.Any]
) -> float | None
nemo_gym.eval.recovery.grading._rubric_verdict(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved rubric judge’s verdict: verdict YES or NO, a finite rubric_score and scoring_threshold, no invalid_judge_response, and the reward 1.0 exactly when the verdict is YES and the score reaches the threshold (0.0 otherwise). The saved answer and score are bound.

nemo_gym.eval.recovery.grading._saved_model_patch(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector’s projected facts, else the same checks on a full saved row (saved_model_patch, collected_patch). Only a verifier that collects the patch itself can be given it back; one that reads the agent session at verify time cannot.

nemo_gym.eval.recovery.grading._scored_unmasked(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A scored row: unmasked, with a reward and no failure kind.

nemo_gym.eval.recovery.grading._session_reward_file(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

The verifier’s own reward file of the row’s session, read by the collector beside the artifacts (resources/<session_id>/verifier/reward.json): the evaluation completed with no infrastructure error and the row’s reward equals the file’s reward. Test counts and partial credit are metadata, never the grade. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._substep_tests(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Saved per-sub-step test results: a non-empty step_results list of booleans, one environment result per step, num_steps_total and num_steps_passed equal to its length and its passes, problem_accuracy true exactly when every step passed, and the row’s reward equal to it (1.0 or 0.0). The saved answer and score are bound.

nemo_gym.eval.recovery.grading._symbolic_match(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved symbolic answer check: a library_reward the row’s reward equals, or one beside a judge verdict (the judge fallback).

nemo_gym.eval.recovery.grading._test_execution(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

A saved test run: evaluation_completed true with an integer verifier_exit_code and no verifier_error.

nemo_gym.eval.recovery.grading._transport_kind(
kind: typing.Any
) -> bool
nemo_gym.eval.recovery.grading._unit_test_results(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Saved unit test results of the answer’s extracted code: at least one test, and the row’s reward 1.0 exactly when every test passed (0.0 otherwise), or 0.0, the default penalty, for an answer whose reasoning format violated the rule (reasoning_format_violation_rate 1.0). A configured non-zero penalty is not saved on the row, so such a row stays unobserved. The saved answer and score are bound.

nemo_gym.eval.recovery.grading._unknown_grading_outcome(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether a scored row’s verifier record reports a judge grading operation with outcome unknown: the grading did not resolve, so its reward is no grade. Read from the verifier invocations the row already saves.

nemo_gym.eval.recovery.grading._unparsed_judge_verdict(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether a scored row’s grading is judge_unparseable for a verdict the judge did not give: a saved judge evaluation recorded no verdict (_unparsed_verdict), or a pairwise game has none (games_unjudged). Its reward is no grade.

nemo_gym.eval.recovery.grading._unparsed_verdict(
facts: collections.abc.Mapping[str, typing.Any]
) -> bool

Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.

nemo_gym.eval.recovery.grading._verifier_model_answered(
row: collections.abc.Mapping[str, typing.Any],
verifiers: list[typing.Any]
) -> bool

Whether one model call of the row’s verifiers completed with a 2xx and a usable response, read from the call the row saved for each reference (by call id, else response id). A call the row does not hold has not answered.

nemo_gym.eval.recovery.grading._verify_transport_chain(
events: collections.abc.Mapping[str, typing.Any],
event: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether event and every explicit cause below it name only a lost verify call: an agent route failure or a transport kind, with no reply status. A verifier’s own failure or an HTTP reply anywhere in the chain is not one.

nemo_gym.eval.recovery.grading._verify_transport_failure(
row: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure or a transport kind, and a saved failure event of the verify operation has a transport-only cause chain.

nemo_gym.eval.recovery.grading.grading_artifact(
row: collections.abc.Mapping[str, typing.Any]
) -> str | None

The name of the first saved grading artifact (GRADING_ARTIFACTS) that records the row’s grading.

nemo_gym.eval.recovery.grading.masked_without_artifact_kind(
row: collections.abc.Mapping[str, typing.Any],
) -> str | None

The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None.

A masked result masked_infrastructure_kind rejects only because it kept a saved answer (so every other rule of that predicate applies: the kind must be registered, evaluation-side and not graded again), that has no recorded grade (has_recorded_grade) and no re-verification cause (reverification_cause): its saved answer has no gradeable artifact, a resume keeps it and re-verification cannot grade it. It runs again only with the explicit --rerun-masked-without-artifact. row is a collector projection, or a saved row passed through saved_row_evidence.

nemo_gym.eval.recovery.grading.reverification_cause(
row: collections.abc.Mapping[str, typing.Any]
) -> str | None

Why a saved result row is re-verified from its saved answer, or None when it is not.

REVERIFY_JUDGE: masked because its judge failed (judge_failed, judge_unparseable), or scored while its judge gave no verdict or its grading did not resolve (_judge_unparseable). REVERIFY_VERIFY_TRANSPORT: masked because its verify call was lost in transport (_verify_transport_failure). REVERIFY_VERIFIER: masked because its verifier produced no verdict (verifier_error), with the workspace patch it graded saved intact (saved_model_patch): the verifier runs again on that patch, never the agent. REVERIFY_UNGRADED: scored, but no grading was recorded for it (_grading_unrecorded). Only a result with a saved answer qualifies; a failures-sidecar row, a result a re-verification already settled (its settlement is its grading) and every other masked result do not. row is a collector projection, or a saved row passed through saved_row_evidence.

nemo_gym.eval.recovery.grading.saved_row_evidence(
row: collections.abc.Mapping[str, typing.Any]
) -> dict[str, typing.Any]

A full saved row with the evidence fields COLLECT_SCRIPT projects for the grading predicates: its scalar grading facts, whether each assistant message of its answer has text, and its verifier invocations.

nemo_gym.eval.recovery.grading.GRADING_ARTIFACTS = (('judge_record', _judge_record), ('rubric_verdict', _rubric_verdict), ('symboli...
nemo_gym.eval.recovery.grading.REVERIFY_JUDGE = 'judge'
nemo_gym.eval.recovery.grading.REVERIFY_UNGRADED = 'ungraded'
nemo_gym.eval.recovery.grading.REVERIFY_VERIFIER = 'verifier'
nemo_gym.eval.recovery.grading.REVERIFY_VERIFY_TRANSPORT = 'verify_transport'
nemo_gym.eval.recovery.grading._COLLECTOR_FUNCTIONS: dict[str, Any] = {}
nemo_gym.eval.recovery.grading._CUT_FINISH = frozenset({'length', 'content_filter', 'max_tokens', 'max_output_tokens'})
nemo_gym.eval.recovery.grading._FAILED_RESPONSE_STATUSES = frozenset({'failed', 'incomplete', 'cancelled'})
nemo_gym.eval.recovery.grading._REVERIFY_JUDGE_KINDS = frozenset({JUDGE_FAILED, JUDGE_UNPARSEABLE})