Skip to navigation

nemo_gym.eval.recovery.budgets

View as MarkdownOpen in Claude

The run’s retry, attempt and walltime budgets as the recovery decision reads and extends them.

Module Contents

Classes

NameDescription
OperatorBudgetAn operator’s explicit budget for a run whose chain accounting is unknown.
RecoveryBudgetThe run’s stored infra-retry and walltime budget and what its job chain used, as the chain files record them.

Functions

NameDescription
_extend_attempts_refusalWhy this run cannot take an attempt extension, or None when it can.
_rerun_masked_refusalWhy this run cannot run masked results without an artifact again, or None when it can.
_saved_attempt_capThe run’s saved per-rollout attempt cap, from the Store’s facts; None when they do not say.
_with_attempt_extensionThe Store’s facts as they read once extend_attempts more attempts are recorded for every rollout.
retry_budgetThe run’s two retry budgets as the gate reports them, each its configured cap and its recorded extension.

API

class nemo_gym.eval.recovery.budgets.OperatorBudget()

Bases: BaseModel

An operator’s explicit budget for a run whose chain accounting is unknown.

retries infra retries remain from now (--extend-retries, one by default) and max_walltime is the walltime cap (--max-walltime, raised as a recovery raises it). It only fills counters that are unknown: a readable counter is never lowered or reset.

max_walltime
str
model_config
= ConfigDict(extra='forbid')
reason
str = Field(min_length=1)
retries
int = Field(default=1, ge=1, le=1000)
user
str
class nemo_gym.eval.recovery.budgets.RecoveryBudget()

Bases: BaseModel

The run’s stored infra-retry and walltime budget and what its job chain used, as the chain files record them.

max_infra_retries and max_walltime_s come from the benchmark’s saved resume_config; infra_retries_used from .gym_infra_retries and walltime_used_s from .gym_accumulated_walltime (see resume_script). The chain records admitted retries and allows another only while infra_retries_used < max_infra_retries. None means the evidence is unavailable and stops the decision.

infra_retries_used
int | None = Field(default=None, ge=0)
job_chain
list[str]
live_job
bool | None
live_job_name
str | None
max_infra_retries
int | None = Field(default=None, ge=0)
max_walltime_s
float | None
walltime_capped
bool
walltime_used_s
float | None = Field(default=None, ge=0)
nemo_gym.eval.recovery.budgets._extend_attempts_refusal(
) -> str | None

Why this run cannot take an attempt extension, or None when it can.

nemo_gym.eval.recovery.budgets._rerun_masked_refusal(
) -> str | None

Why this run cannot run masked results without an artifact again, or None when it can.

nemo_gym.eval.recovery.budgets._saved_attempt_cap(
) -> int | None

The run’s saved per-rollout attempt cap, from the Store’s facts; None when they do not say.

nemo_gym.eval.recovery.budgets._with_attempt_extension(
extend_attempts: int

The Store’s facts as they read once extend_attempts more attempts are recorded for every rollout.

Only a rollout the Store would run again under some cap (replayable) gets the higher cap, and it is pending while it has attempts left. Any other rollout keeps its cap and state: a model failure, a judge failure that kept its saved answer (its re-judgment budget is the saved cap), a saved result.

nemo_gym.eval.recovery.budgets.retry_budget(
) -> dict[str, dict[str, int | None]]

The run’s two retry budgets as the gate reports them, each its configured cap and its recorded extension.

infra is the chain’s infra-retry cap (--extend-retries); attempts is every rollout’s attempt cap (--extend-attempts). A value that could not be read is None.