> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.budgets

The run's retry, attempt and walltime budgets as the recovery decision reads and extends them.

## Module Contents

### Classes

| Name                                                               | Description                                                                                                   |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| [`OperatorBudget`](#nemo_gym-eval-recovery-budgets-OperatorBudget) | An operator's explicit budget for a run whose chain accounting is unknown.                                    |
| [`RecoveryBudget`](#nemo_gym-eval-recovery-budgets-RecoveryBudget) | The run's stored infra-retry and walltime budget and what its job chain used, as the chain files record them. |

### Functions

| Name                                                                                   | Description                                                                                               |
| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [`_extend_attempts_refusal`](#nemo_gym-eval-recovery-budgets-_extend_attempts_refusal) | Why this run cannot take an attempt extension, or None when it can.                                       |
| [`_rerun_masked_refusal`](#nemo_gym-eval-recovery-budgets-_rerun_masked_refusal)       | Why this run cannot run masked results without an artifact again, or None when it can.                    |
| [`_saved_attempt_cap`](#nemo_gym-eval-recovery-budgets-_saved_attempt_cap)             | The run's saved per-rollout attempt cap, from the Store's facts; None when they do not say.               |
| [`_with_attempt_extension`](#nemo_gym-eval-recovery-budgets-_with_attempt_extension)   | The Store's facts as they read once `extend_attempts` more attempts are recorded for every rollout.       |
| [`retry_budget`](#nemo_gym-eval-recovery-budgets-retry_budget)                         | The run's two retry budgets as the gate reports them, each its configured cap and its recorded extension. |

### API

```python
class nemo_gym.eval.recovery.budgets.OperatorBudget()
```

**Bases:** `BaseModel`

An operator's explicit budget for a run whose chain accounting is unknown.

`retries` infra retries remain from now (`--extend-retries`, one by default) and `max_walltime` is the
walltime cap (`--max-walltime`, raised as a recovery raises it). It only fills counters that are unknown: a
readable counter is never lowered or reset.

**`max_walltime`** `str`

---

**`model_config`** `= ConfigDict(extra='forbid')`

---

**`reason`** `str = Field(min_length=1)`

---

**`retries`** `int = Field(default=1, ge=1, le=1000)`

---

**`user`** `str`

---

```python
class nemo_gym.eval.recovery.budgets.RecoveryBudget()
```

**Bases:** `BaseModel`

The run's stored infra-retry and walltime budget and what its job chain used, as the chain files record them.

`max_infra_retries` and `max_walltime_s` come from the benchmark's saved `resume_config`;
`infra_retries_used` from `.gym_infra_retries` and `walltime_used_s` from `.gym_accumulated_walltime`
(see `resume_script`). The chain records admitted retries and allows another only while
`infra_retries_used < max_infra_retries`. `None` means the evidence is unavailable and stops the decision.

**`infra_retries_used`** `int | None = Field(default=None, ge=0)`

---

**`job_chain`** `list[str]`

---

**`live_job`** `bool | None`

---

**`live_job_name`** `str | None`

---

**`max_infra_retries`** `int | None = Field(default=None, ge=0)`

---

**`max_walltime_s`** `float | None`

---

**`walltime_capped`** `bool`

---

**`walltime_used_s`** `float | None = Field(default=None, ge=0)`

---

```python
nemo_gym.eval.recovery.budgets._extend_attempts_refusal(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> str | None
```

Why this run cannot take an attempt extension, or None when it can.

```python
nemo_gym.eval.recovery.budgets._rerun_masked_refusal(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> str | None
```

Why this run cannot run masked results without an artifact again, or None when it can.

```python
nemo_gym.eval.recovery.budgets._saved_attempt_cap(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> int | None
```

The run's saved per-rollout attempt cap, from the Store's facts; None when they do not say.

```python
nemo_gym.eval.recovery.budgets._with_attempt_extension(
    facts: list[nemo_gym.eval.recovery.evidence.ManagedRecoveryFact] | None,
    extend_attempts: int
) -> list[nemo_gym.eval.recovery.evidence.ManagedRecoveryFact] | None
```

The Store's facts as they read once `extend_attempts` more attempts are recorded for every rollout.

Only a rollout the Store would run again under some cap (`replayable`) gets the higher cap, and it is pending
while it has attempts left. Any other rollout keeps its cap and state: a model failure, a judge failure that kept
its saved answer (its re-judgment budget is the saved cap), a saved result.

```python
nemo_gym.eval.recovery.budgets.retry_budget(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> dict[str, dict[str, int | None]]
```

The run's two retry budgets as the gate reports them, each its configured cap and its recorded extension.

`infra` is the chain's infra-retry cap (`--extend-retries`); `attempts` is every rollout's attempt cap
(`--extend-attempts`). A value that could not be read is None.