Skip to navigation

nemo_gym.eval.recovery.planner

View as MarkdownOpen in Claude

Whether gym eval resubmit may queue a run again: decide_from_collected turns the evidence COLLECT_SCRIPT read (CollectedEvidence) into one RecoveryDecision.

It classifies every expected identity (rows), applies the run’s budgets and an operator’s explicit budget, and names in the decision’s reason what the run’s chain, its drift and any raised limit add. It reads no files and starts nothing.

Module Contents

Functions

NameDescription
_continuation_optionsThe options of the one gym eval resubmit --recover call that continues a run this STOP blocks, else [].
_decide_from_collectedDecide from what COLLECT_SCRIPT read; a missing result file stops the decision.
_operator_continuationDecide again under the operator’s explicit budget, or keep the stop when continuing could be unsafe.
_unrecorded_notesA stop that a budget, unknown accounting or a refused extension blocks records none of the options this call
decide_from_collected_decide_from_collected, with the reason naming any operational drift the run’s latest resume crossed (see
live_chain_jobThe chain job that still waits or runs, as ‘id (STATE)’ from its latest incarnation; None when none does.
resume_drift_noteThe sentence a decision reason carries when the run’s latest resume crossed operational drift.

Data

MLFLOW_ROLLOUTS_NOT_UPLOADED

SAVED_ROW_FILES

_CHAIN_STOP_KIND

_DRIFT_KEYS_SHOWN

_EXPLICIT_BUDGET

_REPIN_IN_THE_SAME_RESUBMIT

__getattr__

API

nemo_gym.eval.recovery.planner._continuation_options(
) -> list[str]

The options of the one gym eval resubmit --recover call that continues a run this STOP blocks, else [].

The stop’s own cause decides: a spent budget (attempts, infra retries, walltime), unknown chain accounting (an explicit budget fills it), masked results without an artifact (an approved rerun), or an extension the run’s pinned Gym cannot apply (a repin in the same call). What in the chain would stop next (_chain_blockers_note) counts only once that cause has a continuation, and the note on masked results without an artifact, which any stop carries, is no cause of its own. Every option a blocker needs is named, with each one the stopped call asked for and did not record, so following the advice never drops one.

nemo_gym.eval.recovery.planner._decide_from_collected(
max_infra_retries: int | None,
max_walltime_s: float | None,
live_job: bool | None,
planned_missing_continuation: bool = False,
allocation_cut: bool = False,
extend_retries: int = 0,
live_job_name: str | None = None,
operator_budget: nemo_gym.eval.recovery.budgets.OperatorBudget | None = None,
extend_attempts: int = 0,
rerun_masked_without_artifact: bool = False

Decide from what COLLECT_SCRIPT read; a missing result file stops the decision.

extend_attempts is the attempts this recovery asks to add to every rollout’s cap (--extend-attempts): the Store’s facts are read as they would be once it is recorded (see _with_attempt_extension).

rerun_masked_without_artifact asks to run again the masked results that kept a saved answer but have no gradeable artifact and no grade (see decide_recovery); the run’s Store reader must apply the approval, or the ask is refused.

When the chain accounting is unknown, operator_budget may continue the run instead (see _operator_continuation); without it, the stop names the options that allow that.

A run with no inventory and no saved row is a fresh restart once no job of it is live. allocation_cut (see decide_recovery) applies only with no live job. The infra-retry cap is the configured one plus the run’s recorded extension plus extend_retries that this recovery asks to record.

nemo_gym.eval.recovery.planner._operator_continuation(
inputs: dict[str, typing.Any],
configured_retries: int | None,
max_walltime_s: float | None,
extend_retries: int = 0

Decide again under the operator’s explicit budget, or keep the stop when continuing could be unsafe.

The caller has checked that the scheduler answered and shows no live or pending job of the run. It fills only unreadable counters: an unknown spent-retry count so that operator.retries retries remain under the cap, an unknown used walltime with zero, so the walltime cap is fresh for this continuation, and an unknown extension with zero. A readable counter keeps its value. When the spent-retry count is readable, an explicit extend_retries (--extend-retries) is what it is on any recovery: that many more infra retries, recorded as the run’s retry extension in the continuation’s writes and journal. Saved rows are classified as for any decision and never changed.

nemo_gym.eval.recovery.planner._unrecorded_notes(
options: collections.abc.Mapping[str, typing.Any]
) -> list[str]

A stop that a budget, unknown accounting or a refused extension blocks records none of the options this call asked for (nothing is queued): one note per option the reason does not already name, so the next advice keeps it and one call can carry them all.

nemo_gym.eval.recovery.planner.decide_from_collected(
options: typing.Any = {}

_decide_from_collected, with the reason naming any operational drift the run’s latest resume crossed (see resume_drift_note) and a continuation that ended before it queued its follow-up, so an inspection, a recovery and a chain continuation all say it.

nemo_gym.eval.recovery.planner.live_chain_job(
rows: list[tuple[str, str, int, int]] | None,
job_chain: list[str]
) -> str | None

The chain job that still waits or runs, as ‘id (STATE)’ from its latest incarnation; None when none does.

nemo_gym.eval.recovery.planner.resume_drift_note(
keys: list[str] | None
) -> str | None

The sentence a decision reason carries when the run’s latest resume crossed operational drift.

nemo_gym.eval.recovery.planner.MLFLOW_ROLLOUTS_NOT_UPLOADED = 'records that the rollouts were not uploaded'
nemo_gym.eval.recovery.planner.SAVED_ROW_FILES = ('rollouts_materialized_inputs.jsonl', 'rollouts.jsonl', 'rollouts_failures.json...
nemo_gym.eval.recovery.planner._CHAIN_STOP_KIND = re.compile('^(?:Previous job \\S+ ended [^:]*: )?(?P<kind>[a-z_]+): ')
nemo_gym.eval.recovery.planner._DRIFT_KEYS_SHOWN = 20
nemo_gym.eval.recovery.planner._EXPLICIT_BUDGET = '--max-walltime HH:MM:SS --walltime-reason TEXT'
nemo_gym.eval.recovery.planner._REPIN_IN_THE_SAME_RESUBMIT = '(--repin COMMIT --repin-reason TEXT) in the same resubmit'
nemo_gym.eval.recovery.planner.__getattr__ = moved_names_getattr(__name__)