> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.rows

Whether `gym eval resubmit` may queue a run again, decided from the run's persisted evidence only.

Every expected identity (`_ng_task_index`, `_ng_rollout_index`) of the materialized inventory gets one state:

* `scored`: a result row that is not a judge failure. Kept as saved; never re-run.
* `model_final`: a failures-sidecar row marked `_ng_failure_counts_as_zero`: the evaluated model's terminal failure,
  a scored zero that no infra retry may replace; or an `agent_timeout` row that kept the agent's answer
  (`is_answered_agent_deadline`), the model's result under its own deadline, which no resume runs again.
* `judge_recoverable`: a result row of class `judge_failed`. Its saved answer is re-judged by
  `gym eval reverify --judge-failed-only`; queueing the policy again is never needed for it.
* `retryable`: the last sidecar row is non-terminal and under its recorded attempt cap: a Gym resume retries it.
* `exhausted`: the last sidecar row is non-terminal and at its attempt cap: only a resubmit gives it a fresh budget.
* `terminal`: a terminal sidecar row that is not a scored zero; excluded, and the run cannot become complete.
* `missing`: no row at all (interrupted or never started); a resume restarts its input.
* `masked`: a result row the producer masked or did not evaluate. Collection keeps it as completed and never runs it
  again, so it is retained, but it is not measured and the run is not complete.
* `omitted`: a planned row that a staged collection's accepted partial stage omitted. It is retained, never
  planned again and not measured, and the run is complete without it.
* `unknown`: a failure whose kind is unregistered or absent, or a row whose identity is invalid; nobody owns it.

The run-level action follows: `stop` for a spent or unknown resubmit budget, or for an unknown identity that a resume
would run again; `noop` when nothing is left to do, `reverify` when only judge failures remain, else `resubmit`.
A run that ended before its inventory was written has nothing to recover: `fresh-restart` queues it again as a fresh
run with a longer walltime.
An unknown identity that a resume never runs again (the Store or the saved result holds it) does not block the
missing and retryable ones: the decision reports it apart from what is retried and what is retained. A decision never
reports a run as complete while an identity is terminal, missing or unknown.

## Module Contents

### Functions

| Name                                                                        | Description                                                                                                                     |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| [`_chain_blockers_note`](#nemo_gym-eval-recovery-rows-_chain_blockers_note) | For a stop on the identities' own states (exhausted, masked): what in the chain's budget would stop the resume                  |
| [`_live_reason`](#nemo_gym-eval-recovery-rows-_live_reason)                 | -                                                                                                                               |
| [`_populations`](#nemo_gym-eval-recovery-rows-_populations)                 | What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.                              |
| [`_unknown_label`](#nemo_gym-eval-recovery-rows-_unknown_label)             | The kind of an unknown identity with the rule that refused it, without naming the kind twice.                                   |
| [`decide_recovery`](#nemo_gym-eval-recovery-rows-decide_recovery)           | Classify every materialized identity and choose the run-level action.                                                           |
| [`omission_summary`](#nemo_gym-eval-recovery-rows-omission_summary)         | The omitted identities by stage and policy, e.g. "1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)". |

### Data

[`_ATTEMPTS_USED_TOKEN`](#nemo_gym-eval-recovery-rows-_ATTEMPTS_USED_TOKEN)

[`_CHAIN_BLOCKERS_LEAD`](#nemo_gym-eval-recovery-rows-_CHAIN_BLOCKERS_LEAD)

[`_EXTEND_RETRIES_SCOPE`](#nemo_gym-eval-recovery-rows-_EXTEND_RETRIES_SCOPE)

[`_INFRA_BUDGET_USED_PREFIX`](#nemo_gym-eval-recovery-rows-_INFRA_BUDGET_USED_PREFIX)

[`_MASKED_STOP_TOKEN`](#nemo_gym-eval-recovery-rows-_MASKED_STOP_TOKEN)

[`_NOT_RECORDED_TOKEN`](#nemo_gym-eval-recovery-rows-_NOT_RECORDED_TOKEN)

[`_RESERVATIONS_UNKNOWN`](#nemo_gym-eval-recovery-rows-_RESERVATIONS_UNKNOWN)

[`_WALLTIME_USED_TOKEN`](#nemo_gym-eval-recovery-rows-_WALLTIME_USED_TOKEN)

### API

```python
nemo_gym.eval.recovery.rows._chain_blockers_note(
    budget: nemo_gym.eval.recovery.budgets.RecoveryBudget
) -> str
```

For a stop on the identities' own states (exhausted, masked): what in the chain's budget would stop the resume
an extension or a rerun allows, as the stop texts name it, so one call can lift every blocker; else "".

```python
nemo_gym.eval.recovery.rows._live_reason(
    live_job_name: str | None
) -> str
```

```python
nemo_gym.eval.recovery.rows._populations(
    counts: collections.abc.Mapping[str, int],
    evidence: list[nemo_gym.eval.recovery.decision.IdentityEvidence]
) -> str
```

What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.

```python
nemo_gym.eval.recovery.rows._unknown_label(
    item: nemo_gym.eval.recovery.decision.IdentityEvidence
) -> str
```

The kind of an unknown identity with the rule that refused it, without naming the kind twice.

```python
nemo_gym.eval.recovery.rows.decide_recovery(
    materialized: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
    results: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
    failures: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
    budget: nemo_gym.eval.recovery.budgets.RecoveryBudget,
    default_max_attempts: int = 3,
    resubmit_offset: int = 0,
    outcome_policy: nemo_gym.eval.recovery.policy.RecoveryOutcomePolicyConfig | None = None,
    managed_selection: list[nemo_gym.eval.recovery.evidence.ManagedRecoveryFact] | None = None,
    planned_missing_continuation: bool = False,
    allocation_cut: bool = False,
    rerun_masked_without_artifact: bool = False
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

Classify every materialized identity and choose the run-level action.

`failures` are every failures-sidecar row in file order (`_line` is the row's line number when known). The
standing row of a rollout is chosen as rollout collection chooses it (`_latest_failure_rows`): rows that
`is_non_superseding_rollout_row` names are skipped, and an interrupted omitted row means the rollout is missing.
Only rows at or after `resubmit_offset` count towards the attempt cap. The rows are only read.

`allocation_cut` says that the allocation that last wrote the Store has ended, by any path (one of
`ALLOCATION_ENDED_STATES`), and no allocation of the run is live: a reservation with no saved row was in flight
when it ended and saved nothing, so it is not started for recovery purposes and the Store runs it again.

A masked result with a saved answer, no gradeable artifact and no recorded grade (`masked_without_artifact_kind`)
stays MASKED, named in `masked_without_artifact`, unless the Store reports an approval for it
(`ManagedRecoveryFact.rerun_approved`) or `rerun_masked_without_artifact` asks for it now: it is then planned
like an infrastructure-masked result, one failed attempt within its cap, and `rerun_masked` lists the retries
this call would have journaled.

```python
nemo_gym.eval.recovery.rows.omission_summary(
    evidence: collections.abc.Iterable[nemo_gym.eval.recovery.decision.IdentityEvidence]
) -> str
```

The omitted identities by stage and policy, e.g. "1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)".

```python
nemo_gym.eval.recovery.rows._ATTEMPTS_USED_TOKEN = 'used every attempt since the last resubmit'
```

```python
nemo_gym.eval.recovery.rows._CHAIN_BLOCKERS_LEAD = '; a resume of them would also stop, as '
```

```python
nemo_gym.eval.recovery.rows._EXTEND_RETRIES_SCOPE = '--extend-retries raises infra retries only; it does not raise the per-identity ...
```

```python
nemo_gym.eval.recovery.rows._INFRA_BUDGET_USED_PREFIX = 'the stored infra-retry budget is used'
```

```python
nemo_gym.eval.recovery.rows._MASKED_STOP_TOKEN = 'completed result(s) are masked and not measured'
```

```python
nemo_gym.eval.recovery.rows._NOT_RECORDED_TOKEN = 'asked in this call is not recorded, as nothing was queued: ask it again in the ...
```

```python
nemo_gym.eval.recovery.rows._RESERVATIONS_UNKNOWN = 'the allocation that made them is not known to have ended, so whether they ran i...
```

```python
nemo_gym.eval.recovery.rows._WALLTIME_USED_TOKEN = 'the stored walltime budget is used'
```