> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.decision

The recovery decision: identity states, run-level actions, and the decision and outcome records.

## Module Contents

### Classes

| Name                                                                          | Description                                                                                                |
| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`IdentityEvidence`](#nemo_gym-eval-recovery-decision-IdentityEvidence)       | -                                                                                                          |
| [`IdentityState`](#nemo_gym-eval-recovery-decision-IdentityState)             | -                                                                                                          |
| [`RecoveryAction`](#nemo_gym-eval-recovery-decision-RecoveryAction)           | -                                                                                                          |
| [`RecoveryDecision`](#nemo_gym-eval-recovery-decision-RecoveryDecision)       | -                                                                                                          |
| [`RecoveryOutcome`](#nemo_gym-eval-recovery-decision-RecoveryOutcome)         | What `recover` did: `dispatched` names the one scheduler or judge action taken, None when nothing ran.     |
| [`RerunMaskedApproval`](#nemo_gym-eval-recovery-decision-RerunMaskedApproval) | One masked result the operator approves to run again: its identity, the attempts counted for it before the |

### Functions

| Name                                                                                        | Description                                                                                                         |
| ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| [`_events`](#nemo_gym-eval-recovery-decision-_events)                                       | -                                                                                                                   |
| [`_judge_job_unsettled`](#nemo_gym-eval-recovery-decision-_judge_job_unsettled)             | -                                                                                                                   |
| [`_known_kind`](#nemo_gym-eval-recovery-decision-_known_kind)                               | -                                                                                                                   |
| [`failed_judge_jobs_blocking`](#nemo_gym-eval-recovery-decision-failed_judge_jobs_blocking) | The ids of the last `JUDGE_JOB_FAILURE_BOUND` judge-only jobs when all ended without settling a saved answer and no |
| [`failed_judge_jobs_stop`](#nemo_gym-eval-recovery-decision-failed_judge_jobs_stop)         | The stop text of `failed_judge_jobs_blocking`: the jobs, where to read them, and the explicit way on.               |

### Data

[`ACKNOWLEDGE_JUDGE_JOBS_FLAGS`](#nemo_gym-eval-recovery-decision-ACKNOWLEDGE_JUDGE_JOBS_FLAGS)

[`ALLOCATION_ENDED_STATES`](#nemo_gym-eval-recovery-decision-ALLOCATION_ENDED_STATES)

[`COLLECTED_CAUSE`](#nemo_gym-eval-recovery-decision-COLLECTED_CAUSE)

[`COUNTS_AS_ZERO`](#nemo_gym-eval-recovery-decision-COUNTS_AS_ZERO)

[`EXTEND_ATTEMPTS_HINT`](#nemo_gym-eval-recovery-decision-EXTEND_ATTEMPTS_HINT)

[`FAILURE_CLASS`](#nemo_gym-eval-recovery-decision-FAILURE_CLASS)

[`FAILURE_EVENTS`](#nemo_gym-eval-recovery-decision-FAILURE_EVENTS)

[`FAILURE_ID`](#nemo_gym-eval-recovery-decision-FAILURE_ID)

[`FRESH_RESTART_REASON`](#nemo_gym-eval-recovery-decision-FRESH_RESTART_REASON)

[`JUDGE_JOB_FAILURE_BOUND`](#nemo_gym-eval-recovery-decision-JUDGE_JOB_FAILURE_BOUND)

[`LIVE_JOB_STATES`](#nemo_gym-eval-recovery-decision-LIVE_JOB_STATES)

[`MAX_ATTEMPTS`](#nemo_gym-eval-recovery-decision-MAX_ATTEMPTS)

[`MAX_EXTEND_ATTEMPTS`](#nemo_gym-eval-recovery-decision-MAX_EXTEND_ATTEMPTS)

[`NO_INVENTORY`](#nemo_gym-eval-recovery-decision-NO_INVENTORY)

[`OPERATOR_CONTINUATION`](#nemo_gym-eval-recovery-decision-OPERATOR_CONTINUATION)

[`OPERATOR_CONTINUATION_NOTE`](#nemo_gym-eval-recovery-decision-OPERATOR_CONTINUATION_NOTE)

[`RERUN_MASKED_FLAGS`](#nemo_gym-eval-recovery-decision-RERUN_MASKED_FLAGS)

[`RERUN_MASKED_REASON`](#nemo_gym-eval-recovery-decision-RERUN_MASKED_REASON)

[`RESERVATION_WITHOUT_OUTCOME`](#nemo_gym-eval-recovery-decision-RESERVATION_WITHOUT_OUTCOME)

[`SCORED_UNPARSEABLE_REASON`](#nemo_gym-eval-recovery-decision-SCORED_UNPARSEABLE_REASON)

[`STORE_DOES_NOT_REPLAY`](#nemo_gym-eval-recovery-decision-STORE_DOES_NOT_REPLAY)

[`TERMINAL`](#nemo_gym-eval-recovery-decision-TERMINAL)

[`UNKNOWN_BUDGET_HINT`](#nemo_gym-eval-recovery-decision-UNKNOWN_BUDGET_HINT)

[`UNKNOWN_BUDGET_REASON`](#nemo_gym-eval-recovery-decision-UNKNOWN_BUDGET_REASON)

[`UNKNOWN_WALLTIME_REASON`](#nemo_gym-eval-recovery-decision-UNKNOWN_WALLTIME_REASON)

[`_FAILED_JUDGE_JOBS`](#nemo_gym-eval-recovery-decision-_FAILED_JUDGE_JOBS)

### API

```python
class nemo_gym.eval.recovery.decision.IdentityEvidence()
```

**Bases:** `BaseModel`

**`attempts`** `int = 0`

---

**`events`** `list[tuple[str | None, str | None, str | None, str | None, str | None]] = Field(default_factory=list)`

---

**`failure_id`** `str | None = None`

---

**`failure_kind`** `str | None = None`

---

**`http_status`** `int | None = None`

---

**`identity`** `Identity`

---

**`max_attempts`** `int | None = None`

---

**`reason`** `str | None = None`

---

**`stage`** `FailureStage | None = None`

---

**`state`** `IdentityState`

---

**`unscored`** `bool = False`

---

```python
class nemo_gym.eval.recovery.decision.IdentityState
```

**Bases:** `enum.Enum`

**`EXHAUSTED`** `= 'exhausted'`

---

**`JUDGE_RECOVERABLE`** `= 'judge_recoverable'`

---

**`MASKED`** `= 'masked'`

---

**`MISSING`** `= 'missing'`

---

**`MODEL_FINAL`** `= 'model_final'`

---

**`OMITTED`** `= 'omitted'`

---

**`RETRYABLE`** `= 'retryable'`

---

**`SCORED`** `= 'scored'`

---

**`SKIPPED`** `= 'skipped'`

---

**`TERMINAL`** `= 'terminal'`

---

**`UNKNOWN`** `= 'unknown'`

---

```python
class nemo_gym.eval.recovery.decision.RecoveryAction
```

**Bases:** `enum.Enum`

**`FRESH_RESTART`** `= 'fresh-restart'`

---

**`NOOP`** `= 'noop'`

---

**`RESUBMIT`** `= 'resubmit'`

---

**`REVERIFY`** `= 'reverify'`

---

**`STOP`** `= 'stop'`

---

**`UNKNOWN`** `= 'unknown'`

---

```python
class nemo_gym.eval.recovery.decision.RecoveryDecision()
```

**Bases:** `BaseModel`

**`accounting_unknown`** `Literal['budget', 'walltime'] | None = Field(default=None, exclude=True)`

---

**`action`** `RecoveryAction`

---

**`chain_stop_kind`** `str | None = Field(default=None, exclude=True)`

---

**`complete`** `bool`

---

**`continuation_refused`** `Literal['scheduler_unreachable', 'job_live', 'claim_unidentified', 'counters_changed', 'no_retry_cap'] | None = Field(default=None, exclude=True)`

---

**`counts`** `dict[str, int]`

---

**`duplicate_results`** `list[Identity] = Field(default_factory=list)`

---

**`evidence`** `list[IdentityEvidence]`

---

**`expected`** `int`

---

**`identities`** `dict[str, list[Identity]]`

---

**`judge_refusal`** `Literal['cpu_partition_unknown', 'cpu_placement', 'served_placement', 'unsupported_profile', 'refused', 'concurrent', 'other'] | None = Field(default=None, exclude=True)`

---

**`masked_without_artifact`** `list[Identity] = Field(default_factory=list)`

---

**`operator_continuation`** `dict[str, Any] | None = None`

---

**`outcome_policy`** `dict[str, Any] | None = None`

---

**`reason`** `str`

---

**`refused_option`** `str | None = Field(default=None, exclude=True)`

---

**`refused_over`** `RecoveryAction | None = Field(default=None, exclude=True)`

---

**`requested`** `list[str] = Field(default_factory=list, exclude=True)`

---

**`rerun_masked`** `list[RerunMaskedApproval] = Field(default_factory=list)`

---

**`reverify_first`** `list[Identity] = Field(default_factory=list)`

---

**`scheduler_live`** `bool = False`

---

**`target`** `str`

---

**`unverified`** `bool`

Whether results of the run were deliberately not verified (`judge_skipped`), so it is UNVERIFIED.

---

```python
class nemo_gym.eval.recovery.decision.RecoveryOutcome()
```

**Bases:** `BaseModel`

What `recover` did: `dispatched` names the one scheduler or judge action taken, None when nothing ran.

**`decision`** `RecoveryDecision`

---

**`dispatched`** `str | None`

---

**`gate_task`** `dict[str, Any] | None = None`

---

**`job_id`** `str | None = None`

---

**`judge_job`** `dict[str, Any] | None = None`

---

**`mlflow`** `dict[str, Any] | None = None`

---

**`ok`** `bool`

---

**`operator_verdict`** `dict[str, str] | None = None`

---

**`pin_plan`** `dict[str, Any] | None = None`

---

**`reader_token`** `str | None = None`

---

**`reverify`** `str | None = None`

---

**`scheduler_refusal`** `dict[str, Any] | None = None`

---

**`unsupported`** `str | None = None`

---

**`walltime_plan`** `dict[str, Any] | None = None`

---

```python
class nemo_gym.eval.recovery.decision.RerunMaskedApproval()
```

**Bases:** `BaseModel`

One masked result the operator approves to run again: its identity, the attempts counted for it before the
approval, and the attempt id of the masked row the approval covers (a later attempt is never covered).

**`attempt`** `int = Field(ge=0)`

---

**`attempts`** `int = Field(ge=0)`

---

**`identity`** `Identity`

---

```python
nemo_gym.eval.recovery.decision._events(
    row: collections.abc.Mapping[str, typing.Any]
) -> list[tuple[str | None, str | None, str | None, str | None, str | None]]
```

```python
nemo_gym.eval.recovery.decision._judge_job_unsettled(
    entry: collections.abc.Mapping[str, typing.Any]
) -> bool
```

```python
nemo_gym.eval.recovery.decision._known_kind(
    kind: typing.Any
) -> bool
```

```python
nemo_gym.eval.recovery.decision.failed_judge_jobs_blocking(
    judge_jobs: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]],
    acknowledgements: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]] = ()
) -> list[str | None] | None
```

The ids of the last `JUDGE_JOB_FAILURE_BOUND` judge-only jobs when all ended without settling a saved answer and no
acknowledgement names exactly them; else None. An acknowledgement covers the failed jobs it names, so a later pair of
failures (the one more job it let queue, and the one before it) needs an acknowledgement of its own.

```python
nemo_gym.eval.recovery.decision.failed_judge_jobs_stop(
    ids: collections.abc.Sequence[str | None],
    where: str
) -> str
```

The stop text of `failed_judge_jobs_blocking`: the jobs, where to read them, and the explicit way on.

```python
nemo_gym.eval.recovery.decision.ACKNOWLEDGE_JUDGE_JOBS_FLAGS = '--acknowledge-failed-judge-jobs --acknowledge-reason'
```

```python
nemo_gym.eval.recovery.decision.ALLOCATION_ENDED_STATES = frozenset({'TIMEOUT', 'PREEMPTED', 'NODE_FAIL', 'BOOT_FAIL', 'CANCELLED', 'FAILE...
```

```python
nemo_gym.eval.recovery.decision.COLLECTED_CAUSE = '_ng_collected_cause'
```

```python
nemo_gym.eval.recovery.decision.COUNTS_AS_ZERO = NG_FAILURE_COUNTS_AS_ZERO_KEY
```

```python
nemo_gym.eval.recovery.decision.EXTEND_ATTEMPTS_HINT = 'an exhausted identity whose attempts went to infrastructure failures or allocat...
```

```python
nemo_gym.eval.recovery.decision.FAILURE_CLASS = NG_FAILURE_CLASS_KEY
```

```python
nemo_gym.eval.recovery.decision.FAILURE_EVENTS = FailureReportRow.EVENTS
```

```python
nemo_gym.eval.recovery.decision.FAILURE_ID = FailureReportRow.FAILURE_ID
```

```python
nemo_gym.eval.recovery.decision.FRESH_RESTART_REASON = f'nothing to recover: {NO_INVENTORY}; resubmit it as a fresh run with a longer w...
```

```python
nemo_gym.eval.recovery.decision.JUDGE_JOB_FAILURE_BOUND = 2
```

```python
nemo_gym.eval.recovery.decision.LIVE_JOB_STATES = frozenset({'PENDING', 'RUNNING', 'CONFIGURING', 'COMPLETING', 'SUSPENDED', 'REQU...
```

```python
nemo_gym.eval.recovery.decision.MAX_ATTEMPTS = NG_FAILURE_MAX_ATTEMPTS_KEY
```

```python
nemo_gym.eval.recovery.decision.MAX_EXTEND_ATTEMPTS = 100
```

```python
nemo_gym.eval.recovery.decision.NO_INVENTORY = 'the run ended before the task inventory was written'
```

```python
nemo_gym.eval.recovery.decision.OPERATOR_CONTINUATION = 'OPERATOR-CONTINUATION'
```

```python
nemo_gym.eval.recovery.decision.OPERATOR_CONTINUATION_NOTE = f'Operator continuation: an explicit budget filled the unknown accounting; see {...
```

```python
nemo_gym.eval.recovery.decision.RERUN_MASKED_FLAGS = '--rerun-masked-without-artifact --rerun-reason'
```

```python
nemo_gym.eval.recovery.decision.RERUN_MASKED_REASON = f'retained, never run again, not measured: it reruns only with {RERUN_MASKED_FLA...
```

```python
nemo_gym.eval.recovery.decision.RESERVATION_WITHOUT_OUTCOME = 'reservation_without_outcome'
```

```python
nemo_gym.eval.recovery.decision.SCORED_UNPARSEABLE_REASON = 'judge gave no verdict (unparseable); re-judge planned'
```

```python
nemo_gym.eval.recovery.decision.STORE_DOES_NOT_REPLAY = "the policy retries it but the run's Store does not plan it again"
```

```python
nemo_gym.eval.recovery.decision.TERMINAL = NG_TERMINAL_KEY
```

```python
nemo_gym.eval.recovery.decision.UNKNOWN_BUDGET_HINT = 'An operator may continue it with an explicit budget: gym eval resubmit --recove...
```

```python
nemo_gym.eval.recovery.decision.UNKNOWN_BUDGET_REASON = 'the job chain or scheduler state is unavailable, so the budget is unknown'
```

```python
nemo_gym.eval.recovery.decision.UNKNOWN_WALLTIME_REASON = 'the run caps its walltime but the used walltime is unavailable'
```

```python
nemo_gym.eval.recovery.decision._FAILED_JUDGE_JOBS = re.compile(f'the last {JUDGE_JOB_FAILURE_BOUND} judge-only job\(s\) \((?P<ids>.*...
```