> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.grading

Saved grading artifacts: whether a saved row's grade is recorded, and why a saved row is re-verified.

## Module Contents

### Functions

| Name                                                                                           | Description                                                                                                            |
| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| [`_bound_answer`](#nemo_gym-eval-recovery-grading-_bound_answer)                               | Whether the collector bound the saved answer and score together (`_ng_collected_grading_binding`).                     |
| [`_collector_function`](#nemo_gym-eval-recovery-grading-_collector_function)                   | `COLLECT_SCRIPT`'s own projection `name`, so an in-process reader projects a saved row as the collector does.          |
| [`_empty_answer`](#nemo_gym-eval-recovery-grading-_empty_answer)                               | Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw             |
| [`_environment_reward`](#nemo_gym-eval-recovery-grading-_environment_reward)                   | The environment's own reward record: `result.reward_info.reward` equal to the row's reward.                            |
| [`_grading_facts`](#nemo_gym-eval-recovery-grading-_grading_facts)                             | -                                                                                                                      |
| [`_grading_unrecorded`](#nemo_gym-eval-recovery-grading-_grading_unrecorded)                   | Whether the row's verifier invocations were observed but recorded no grading: no grading operation, no                 |
| [`_guarded_objective_score`](#nemo_gym-eval-recovery-grading-_guarded_objective_score)         | A guardrail-gated objective score the agent's own rubric saved (`response.metrics`): the row's reward equals           |
| [`_incomplete_answer`](#nemo_gym-eval-recovery-grading-_incomplete_answer)                     | Whether the row's saved answer is a response left incomplete, without an error. An incomplete status alone             |
| [`_judge_error_marker`](#nemo_gym-eval-recovery-grading-_judge_error_marker)                   | Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a          |
| [`_judge_evaluations`](#nemo_gym-eval-recovery-grading-_judge_evaluations)                     | A non-empty `judge_evaluations` list without a judge error; when every evaluation's verdict label parses               |
| [`_judge_record`](#nemo_gym-eval-recovery-grading-_judge_record)                               | A saved judge verdict: a `judge_response`, a non-empty `judgments` list with a `judge_score`, or                       |
| [`_judge_unparseable`](#nemo_gym-eval-recovery-grading-_judge_unparseable)                     | Whether a scored row is `judge_unparseable`: its judge gave no verdict (`_unparsed_judge_verdict`) or its              |
| [`_metric_record`](#nemo_gym-eval-recovery-grading-_metric_record)                             | Saved in-process metric scores: numeric `sentence_*` fields (`sentence_chrf`, `sentence_spbleu`).                      |
| [`_model_failure`](#nemo_gym-eval-recovery-grading-_model_failure)                             | The benchmark's rule scored the model's own failed answer without grading it: the verifier marked the answer           |
| [`_native_scorer_score`](#nemo_gym-eval-recovery-grading-_native_scorer_score)                 | A native scorer's saved score (`native_metrics`): the row's reward equals the saved `weighted_score` in                |
| [`_node_set_f1`](#nemo_gym-eval-recovery-grading-_node_set_f1)                                 | A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer's final                        |
| [`_repository_test_run`](#nemo_gym-eval-recovery-grading-_repository_test_run)                 | A saved repository test run of the answer's patch: `evaluation_completed` true, the run's `resolved`                   |
| [`_reward`](#nemo_gym-eval-recovery-grading-_reward)                                           | -                                                                                                                      |
| [`_rubric_verdict`](#nemo_gym-eval-recovery-grading-_rubric_verdict)                           | A saved rubric judge's verdict: `verdict` YES or NO, a finite `rubric_score` and `scoring_threshold`, no               |
| [`_saved_model_patch`](#nemo_gym-eval-recovery-grading-_saved_model_patch)                     | Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector's projected        |
| [`_scored_unmasked`](#nemo_gym-eval-recovery-grading-_scored_unmasked)                         | A scored row: unmasked, with a reward and no failure kind.                                                             |
| [`_session_reward_file`](#nemo_gym-eval-recovery-grading-_session_reward_file)                 | The verifier's own reward file of the row's session, read by the collector beside the artifacts                        |
| [`_substep_tests`](#nemo_gym-eval-recovery-grading-_substep_tests)                             | Saved per-sub-step test results: a non-empty `step_results` list of booleans, one environment result per               |
| [`_symbolic_match`](#nemo_gym-eval-recovery-grading-_symbolic_match)                           | A saved symbolic answer check: a `library_reward` the row's reward equals, or one beside a judge verdict               |
| [`_test_execution`](#nemo_gym-eval-recovery-grading-_test_execution)                           | A saved test run: `evaluation_completed` true with an integer `verifier_exit_code` and no                              |
| [`_transport_kind`](#nemo_gym-eval-recovery-grading-_transport_kind)                           | -                                                                                                                      |
| [`_unit_test_results`](#nemo_gym-eval-recovery-grading-_unit_test_results)                     | Saved unit test results of the answer's extracted code: at least one test, and the row's reward 1.0 exactly            |
| [`_unknown_grading_outcome`](#nemo_gym-eval-recovery-grading-_unknown_grading_outcome)         | Whether a scored row's verifier record reports a judge grading operation with outcome `unknown`: the grading           |
| [`_unparsed_judge_verdict`](#nemo_gym-eval-recovery-grading-_unparsed_judge_verdict)           | Whether a scored row's grading is `judge_unparseable` for a verdict the judge did not give: a saved judge              |
| [`_unparsed_verdict`](#nemo_gym-eval-recovery-grading-_unparsed_verdict)                       | Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.               |
| [`_verifier_model_answered`](#nemo_gym-eval-recovery-grading-_verifier_model_answered)         | Whether one model call of the row's verifiers completed with a 2xx and a usable response, read from the call           |
| [`_verify_transport_chain`](#nemo_gym-eval-recovery-grading-_verify_transport_chain)           | Whether `event` and every explicit cause below it name only a lost verify call: an agent route failure or a            |
| [`_verify_transport_failure`](#nemo_gym-eval-recovery-grading-_verify_transport_failure)       | Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure           |
| [`grading_artifact`](#nemo_gym-eval-recovery-grading-grading_artifact)                         | The name of the first saved grading artifact (`GRADING_ARTIFACTS`) that records the row's grading.                     |
| [`masked_without_artifact_kind`](#nemo_gym-eval-recovery-grading-masked_without_artifact_kind) | The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None. |
| [`reverification_cause`](#nemo_gym-eval-recovery-grading-reverification_cause)                 | Why a saved result row is re-verified from its saved answer, or None when it is not.                                   |
| [`saved_row_evidence`](#nemo_gym-eval-recovery-grading-saved_row_evidence)                     | A full saved row with the evidence fields `COLLECT_SCRIPT` projects for the grading predicates: its scalar             |

### Data

[`GRADING_ARTIFACTS`](#nemo_gym-eval-recovery-grading-GRADING_ARTIFACTS)

[`REVERIFY_JUDGE`](#nemo_gym-eval-recovery-grading-REVERIFY_JUDGE)

[`REVERIFY_UNGRADED`](#nemo_gym-eval-recovery-grading-REVERIFY_UNGRADED)

[`REVERIFY_VERIFIER`](#nemo_gym-eval-recovery-grading-REVERIFY_VERIFIER)

[`REVERIFY_VERIFY_TRANSPORT`](#nemo_gym-eval-recovery-grading-REVERIFY_VERIFY_TRANSPORT)

[`_COLLECTOR_FUNCTIONS`](#nemo_gym-eval-recovery-grading-_COLLECTOR_FUNCTIONS)

[`_CUT_FINISH`](#nemo_gym-eval-recovery-grading-_CUT_FINISH)

[`_FAILED_RESPONSE_STATUSES`](#nemo_gym-eval-recovery-grading-_FAILED_RESPONSE_STATUSES)

[`_REVERIFY_JUDGE_KINDS`](#nemo_gym-eval-recovery-grading-_REVERIFY_JUDGE_KINDS)

### API

```python
nemo_gym.eval.recovery.grading._bound_answer(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the collector bound the saved answer and score together (`_ng_collected_grading_binding`).

```python
nemo_gym.eval.recovery.grading._collector_function(
    name: str
)
```

`COLLECT_SCRIPT`'s own projection `name`, so an in-process reader projects a saved row as the collector does.

```python
nemo_gym.eval.recovery.grading._empty_answer(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw
without non-whitespace text (`has_text` false). A message whose content was not saved counts as an answer.

```python
nemo_gym.eval.recovery.grading._environment_reward(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

The environment's own reward record: `result.reward_info.reward` equal to the row's reward.

```python
nemo_gym.eval.recovery.grading._grading_facts(
    row: collections.abc.Mapping[str, typing.Any]
) -> collections.abc.Mapping[str, typing.Any]
```

```python
nemo_gym.eval.recovery.grading._grading_unrecorded(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row's verifier invocations were observed but recorded no grading: no grading operation, no
verifier grading record, no saved grading artifact (`grading_artifact`) and no model-failure rule
(`_model_failure`). A row from a pin without verifier observations proves nothing and is not selected. Nor is
one whose verifier got an answer from a model (`_verifier_model_answered`) and wrote no judge-error marker
(`_judge_error_marker`): a judge the verifier does not observe (`observe_llm_judge_grading`) records no
grading however often it is asked again, and a grade it gave is never replaced. A row whose model calls all
failed, or that names a judge error, was not graded.

```python
nemo_gym.eval.recovery.grading._guarded_objective_score(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A guardrail-gated objective score the agent's own rubric saved (`response.metrics`): the row's reward equals
the saved `aa_headline`, and that headline follows from the saved counts: 0.0 when a guardrail was violated,
else the objectives passed over the objectives (0.0 without objectives). A row without any counted assertion (no
world or no assertions also scores 0.0) is no evidence and stays unobserved. The counts are the rubric's own, not
a re-run of its assertions. The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._incomplete_answer(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row's saved answer is a response left incomplete, without an error. An incomplete status alone
says nothing about why: an agent also marks an answer incomplete that holds no message.

```python
nemo_gym.eval.recovery.grading._judge_error_marker(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a
top-level message) or an invalid judge response (top level or inside the rubric facts).

```python
nemo_gym.eval.recovery.grading._judge_evaluations(
    facts: collections.abc.Mapping[str, typing.Any],
    reward: float | None
) -> bool
```

A non-empty `judge_evaluations` list without a judge error; when every evaluation's verdict label parses
(`[[A=B]]`/`[[A!=B]]` or `Judgement: yes/no`), the reward is 1.0 exactly when every verdict is equal/yes.

```python
nemo_gym.eval.recovery.grading._judge_record(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved judge verdict: a `judge_response`, a non-empty `judgments` list with a `judge_score`, or
non-empty `judge_evaluations` whose parsed verdicts agree with the reward (`_judge_evaluations`).

```python
nemo_gym.eval.recovery.grading._judge_unparseable(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether a scored row is `judge_unparseable`: its judge gave no verdict (`_unparsed_judge_verdict`) or its
verifier's grading did not resolve (`_unknown_grading_outcome`). Judges are stateless: it is re-judged.

```python
nemo_gym.eval.recovery.grading._metric_record(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Saved in-process metric scores: numeric `sentence_*` fields (`sentence_chrf`, `sentence_spbleu`).

```python
nemo_gym.eval.recovery.grading._model_failure(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

The benchmark's rule scored the model's own failed answer without grading it: the verifier marked the answer
invalid (`invalid_model_response`), or the model left it incomplete, or it is empty and scored zero.

```python
nemo_gym.eval.recovery.grading._native_scorer_score(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A native scorer's saved score (`native_metrics`): the row's reward equals the saved `weighted_score` in
\[0, 1], the scorer saved its binary `execution` flag and its diagnostics. The score is the scorer's own, not a
re-run of it. The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._node_set_f1(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer's final
`Final Answer: [...]` line, reproduces the saved predicted nodes and parse outcome, and scores them against the
saved expected node list as the verifier does; the row's reward equals that re-graded F1 and the saved `f1`.
The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._repository_test_run(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved repository test run of the answer's patch: `evaluation_completed` true, the run's `resolved`
verdict with the row's reward equal to it (1.0 resolved, 0.0 not), saved `test_results`, no `error`, no
evaluation timeout, and a resolved verdict only for a patch that applied. The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._reward(
    row: collections.abc.Mapping[str, typing.Any]
) -> float | None
```

```python
nemo_gym.eval.recovery.grading._rubric_verdict(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved rubric judge's verdict: `verdict` YES or NO, a finite `rubric_score` and `scoring_threshold`, no
`invalid_judge_response`, and the reward 1.0 exactly when the verdict is YES and the score reaches the threshold
(0.0 otherwise). The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._saved_model_patch(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector's projected
facts, else the same checks on a full saved row (`saved_model_patch`, `collected_patch`). Only a verifier that
collects the patch itself can be given it back; one that reads the agent session at verify time cannot.

```python
nemo_gym.eval.recovery.grading._scored_unmasked(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A scored row: unmasked, with a reward and no failure kind.

```python
nemo_gym.eval.recovery.grading._session_reward_file(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

The verifier's own reward file of the row's session, read by the collector beside the artifacts
(`resources/<session_id>/verifier/reward.json`): the evaluation completed with no infrastructure error and the
row's reward equals the file's `reward`. Test counts and partial credit are metadata, never the grade. The saved
answer and score are bound.

```python
nemo_gym.eval.recovery.grading._substep_tests(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Saved per-sub-step test results: a non-empty `step_results` list of booleans, one environment result per
step, `num_steps_total` and `num_steps_passed` equal to its length and its passes, `problem_accuracy` true
exactly when every step passed, and the row's reward equal to it (1.0 or 0.0). The saved answer and score are
bound.

```python
nemo_gym.eval.recovery.grading._symbolic_match(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved symbolic answer check: a `library_reward` the row's reward equals, or one beside a judge verdict
(the judge fallback).

```python
nemo_gym.eval.recovery.grading._test_execution(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A saved test run: `evaluation_completed` true with an integer `verifier_exit_code` and no
`verifier_error`.

```python
nemo_gym.eval.recovery.grading._transport_kind(
    kind: typing.Any
) -> bool
```

```python
nemo_gym.eval.recovery.grading._unit_test_results(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Saved unit test results of the answer's extracted code: at least one test, and the row's reward 1.0 exactly
when every test passed (0.0 otherwise), or 0.0, the default penalty, for an answer whose reasoning format
violated the rule (`reasoning_format_violation_rate` 1.0). A configured non-zero penalty is not saved on the row,
so such a row stays unobserved. The saved answer and score are bound.

```python
nemo_gym.eval.recovery.grading._unknown_grading_outcome(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether a scored row's verifier record reports a judge grading operation with outcome `unknown`: the grading
did not resolve, so its reward is no grade. Read from the verifier invocations the row already saves.

```python
nemo_gym.eval.recovery.grading._unparsed_judge_verdict(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether a scored row's grading is `judge_unparseable` for a verdict the judge did not give: a saved judge
evaluation recorded no verdict (`_unparsed_verdict`), or a pairwise game has none (`games_unjudged`). Its
reward is no grade.

```python
nemo_gym.eval.recovery.grading._unparsed_verdict(
    facts: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.

```python
nemo_gym.eval.recovery.grading._verifier_model_answered(
    row: collections.abc.Mapping[str, typing.Any],
    verifiers: list[typing.Any]
) -> bool
```

Whether one model call of the row's verifiers completed with a 2xx and a usable response, read from the call
the row saved for each reference (by call id, else response id). A call the row does not hold has not answered.

```python
nemo_gym.eval.recovery.grading._verify_transport_chain(
    events: collections.abc.Mapping[str, typing.Any],
    event: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether `event` and every explicit cause below it name only a lost verify call: an agent route failure or a
transport kind, with no reply status. A verifier's own failure or an HTTP reply anywhere in the chain is not one.

```python
nemo_gym.eval.recovery.grading._verify_transport_failure(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure
or a transport kind, and a saved failure event of the `verify` operation has a transport-only cause chain.

```python
nemo_gym.eval.recovery.grading.grading_artifact(
    row: collections.abc.Mapping[str, typing.Any]
) -> str | None
```

The name of the first saved grading artifact (`GRADING_ARTIFACTS`) that records the row's grading.

```python
nemo_gym.eval.recovery.grading.masked_without_artifact_kind(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None.

A masked result `masked_infrastructure_kind` rejects only because it kept a saved answer (so every other rule of
that predicate applies: the kind must be registered, evaluation-side and not graded again), that has no recorded
grade (`has_recorded_grade`) and no re-verification cause (`reverification_cause`): its saved answer has no
gradeable artifact, a resume keeps it and re-verification cannot grade it. It runs again only with the explicit
`--rerun-masked-without-artifact`. `row` is a collector projection, or a saved row passed through
`saved_row_evidence`.

```python
nemo_gym.eval.recovery.grading.reverification_cause(
    row: collections.abc.Mapping[str, typing.Any]
) -> str | None
```

Why a saved result row is re-verified from its saved answer, or None when it is not.

`REVERIFY_JUDGE`: masked because its judge failed (`judge_failed`, `judge_unparseable`), or scored while
its judge gave no verdict or its grading did not resolve (`_judge_unparseable`).
`REVERIFY_VERIFY_TRANSPORT`: masked because its verify call was lost in transport (`_verify_transport_failure`).
`REVERIFY_VERIFIER`: masked because its verifier produced no verdict (`verifier_error`), with the workspace
patch it graded saved intact (`saved_model_patch`): the verifier runs again on that patch, never the agent.
`REVERIFY_UNGRADED`: scored, but no grading was recorded for it (`_grading_unrecorded`).
Only a result with a saved answer qualifies; a failures-sidecar row, a result a re-verification already settled
(its settlement is its grading) and every other masked result do not. `row` is a collector projection, or a
saved row passed through `saved_row_evidence`.

```python
nemo_gym.eval.recovery.grading.saved_row_evidence(
    row: collections.abc.Mapping[str, typing.Any]
) -> dict[str, typing.Any]
```

A full saved row with the evidence fields `COLLECT_SCRIPT` projects for the grading predicates: its scalar
grading facts, whether each assistant message of its answer has text, and its verifier invocations.

```python
nemo_gym.eval.recovery.grading.GRADING_ARTIFACTS = (('judge_record', _judge_record), ('rubric_verdict', _rubric_verdict), ('symboli...
```

```python
nemo_gym.eval.recovery.grading.REVERIFY_JUDGE = 'judge'
```

```python
nemo_gym.eval.recovery.grading.REVERIFY_UNGRADED = 'ungraded'
```

```python
nemo_gym.eval.recovery.grading.REVERIFY_VERIFIER = 'verifier'
```

```python
nemo_gym.eval.recovery.grading.REVERIFY_VERIFY_TRANSPORT = 'verify_transport'
```

```python
nemo_gym.eval.recovery.grading._COLLECTOR_FUNCTIONS: dict[str, Any] = {}
```

```python
nemo_gym.eval.recovery.grading._CUT_FINISH = frozenset({'length', 'content_filter', 'max_tokens', 'max_output_tokens'})
```

```python
nemo_gym.eval.recovery.grading._FAILED_RESPONSE_STATUSES = frozenset({'failed', 'incomplete', 'cancelled'})
```

```python
nemo_gym.eval.recovery.grading._REVERIFY_JUDGE_KINDS = frozenset({JUDGE_FAILED, JUDGE_UNPARSEABLE})
```