> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.planner

Whether `gym eval resubmit` may queue a run again: `decide_from_collected` turns the evidence
`COLLECT_SCRIPT` read (`CollectedEvidence`) into one `RecoveryDecision`.

It classifies every expected identity (`rows`), applies the run's budgets and an operator's explicit budget,
and names in the decision's reason what the run's chain, its drift and any raised limit add. It reads no files
and starts nothing.

## Module Contents

### Functions

| Name                                                                               | Description                                                                                                  |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [`_continuation_options`](#nemo_gym-eval-recovery-planner-_continuation_options)   | The options of the one `gym eval resubmit --recover` call that continues a run this STOP blocks, else \[].   |
| [`_decide_from_collected`](#nemo_gym-eval-recovery-planner-_decide_from_collected) | Decide from what `COLLECT_SCRIPT` read; a missing result file stops the decision.                            |
| [`_operator_continuation`](#nemo_gym-eval-recovery-planner-_operator_continuation) | Decide again under the operator's explicit budget, or keep the stop when continuing could be unsafe.         |
| [`_unrecorded_notes`](#nemo_gym-eval-recovery-planner-_unrecorded_notes)           | A stop that a budget, unknown accounting or a refused extension blocks records none of the options this call |
| [`decide_from_collected`](#nemo_gym-eval-recovery-planner-decide_from_collected)   | `_decide_from_collected`, with the reason naming any operational drift the run's latest resume crossed (see  |
| [`live_chain_job`](#nemo_gym-eval-recovery-planner-live_chain_job)                 | The chain job that still waits or runs, as 'id (STATE)' from its latest incarnation; None when none does.    |
| [`resume_drift_note`](#nemo_gym-eval-recovery-planner-resume_drift_note)           | The sentence a decision reason carries when the run's latest resume crossed operational drift.               |

### Data

[`MLFLOW_ROLLOUTS_NOT_UPLOADED`](#nemo_gym-eval-recovery-planner-MLFLOW_ROLLOUTS_NOT_UPLOADED)

[`SAVED_ROW_FILES`](#nemo_gym-eval-recovery-planner-SAVED_ROW_FILES)

[`_CHAIN_STOP_KIND`](#nemo_gym-eval-recovery-planner-_CHAIN_STOP_KIND)

[`_DRIFT_KEYS_SHOWN`](#nemo_gym-eval-recovery-planner-_DRIFT_KEYS_SHOWN)

[`_EXPLICIT_BUDGET`](#nemo_gym-eval-recovery-planner-_EXPLICIT_BUDGET)

[`_REPIN_IN_THE_SAME_RESUBMIT`](#nemo_gym-eval-recovery-planner-_REPIN_IN_THE_SAME_RESUBMIT)

[`__getattr__`](#nemo_gym-eval-recovery-planner-__getattr__)

### API

```python
nemo_gym.eval.recovery.planner._continuation_options(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision
) -> list[str]
```

The options of the one `gym eval resubmit --recover` call that continues a run this STOP blocks, else \[].

The stop's own cause decides: a spent budget (attempts, infra retries, walltime), unknown chain accounting (an
explicit budget fills it), masked results without an artifact (an approved rerun), or an extension the run's
pinned Gym cannot apply (a repin in the same call). What in the chain would stop next (`_chain_blockers_note`)
counts only once that cause has a continuation, and the note on masked results without an artifact, which any
stop carries, is no cause of its own. Every option a blocker needs is named, with each one the stopped call asked
for and did not record, so following the advice never drops one.

```python
nemo_gym.eval.recovery.planner._decide_from_collected(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    max_infra_retries: int | None,
    max_walltime_s: float | None,
    live_job: bool | None,
    outcome_policy: nemo_gym.eval.recovery.policy.RecoveryOutcomePolicyConfig | None = None,
    planned_missing_continuation: bool = False,
    allocation_cut: bool = False,
    extend_retries: int = 0,
    live_job_name: str | None = None,
    operator_budget: nemo_gym.eval.recovery.budgets.OperatorBudget | None = None,
    extend_attempts: int = 0,
    rerun_masked_without_artifact: bool = False
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

Decide from what `COLLECT_SCRIPT` read; a missing result file stops the decision.

`extend_attempts` is the attempts this recovery asks to add to every rollout's cap (`--extend-attempts`):
the Store's facts are read as they would be once it is recorded (see `_with_attempt_extension`).

`rerun_masked_without_artifact` asks to run again the masked results that kept a saved answer but have no gradeable
artifact and no grade (see `decide_recovery`); the run's Store reader must apply the approval, or the ask is refused.

When the chain accounting is unknown, `operator_budget` may continue the run instead (see
`_operator_continuation`); without it, the stop names the options that allow that.

A run with no inventory and no saved row is a fresh restart once no job of it is live. `allocation_cut` (see
`decide_recovery`) applies only with no live job. The infra-retry cap is the configured one plus the run's
recorded extension plus `extend_retries` that this recovery asks to record.

```python
nemo_gym.eval.recovery.planner._operator_continuation(
    stopped: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    inputs: dict[str, typing.Any],
    operator: nemo_gym.eval.recovery.budgets.OperatorBudget,
    configured_retries: int | None,
    max_walltime_s: float | None,
    extend_retries: int = 0
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

Decide again under the operator's explicit budget, or keep the stop when continuing could be unsafe.

The caller has checked that the scheduler answered and shows no live or pending job of the run. It fills only
unreadable counters: an unknown spent-retry count so that `operator.retries` retries remain under the cap, an
unknown used walltime with zero, so the walltime cap is fresh for this continuation, and an unknown extension with
zero. A readable counter keeps its value. When the spent-retry count is readable, an explicit `extend_retries`
(`--extend-retries`) is what it is on any recovery: that many more infra retries, recorded as the run's retry
extension in the continuation's writes and journal. Saved rows are classified as for any decision and never
changed.

```python
nemo_gym.eval.recovery.planner._unrecorded_notes(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    options: collections.abc.Mapping[str, typing.Any]
) -> list[str]
```

A stop that a budget, unknown accounting or a refused extension blocks records none of the options this call
asked for (nothing is queued): one note per option the reason does not already name, so the next advice keeps
it and one call can carry them all.

```python
nemo_gym.eval.recovery.planner.decide_from_collected(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    options: typing.Any = {}
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

`_decide_from_collected`, with the reason naming any operational drift the run's latest resume crossed (see
`resume_drift_note`) and a continuation that ended before it queued its follow-up, so an inspection, a recovery
and a chain continuation all say it.

```python
nemo_gym.eval.recovery.planner.live_chain_job(
    rows: list[tuple[str, str, int, int]] | None,
    job_chain: list[str]
) -> str | None
```

The chain job that still waits or runs, as 'id (STATE)' from its latest incarnation; None when none does.

```python
nemo_gym.eval.recovery.planner.resume_drift_note(
    keys: list[str] | None
) -> str | None
```

The sentence a decision reason carries when the run's latest resume crossed operational drift.

```python
nemo_gym.eval.recovery.planner.MLFLOW_ROLLOUTS_NOT_UPLOADED = 'records that the rollouts were not uploaded'
```

```python
nemo_gym.eval.recovery.planner.SAVED_ROW_FILES = ('rollouts_materialized_inputs.jsonl', 'rollouts.jsonl', 'rollouts_failures.json...
```

```python
nemo_gym.eval.recovery.planner._CHAIN_STOP_KIND = re.compile('^(?:Previous job \\S+ ended [^:]*: )?(?P<kind>[a-z_]+): ')
```

```python
nemo_gym.eval.recovery.planner._DRIFT_KEYS_SHOWN = 20
```

```python
nemo_gym.eval.recovery.planner._EXPLICIT_BUDGET = '--max-walltime HH:MM:SS --walltime-reason TEXT'
```

```python
nemo_gym.eval.recovery.planner._REPIN_IN_THE_SAME_RESUBMIT = '(--repin COMMIT --repin-reason TEXT) in the same resubmit'
```

```python
nemo_gym.eval.recovery.planner.__getattr__ = moved_names_getattr(__name__)
```