nemo_gym.eval.recovery.continuation
nemo_gym.eval.recovery.continuation
Whether a chain continues: the admission a terminal allocation runs in its installed runtime, and the accounting of a chain’s allocations, including requeued ones.
Module Contents
Functions
Data
API
In a sentinel: reconcile a requeued chain from the terminal records in CHAIN_ACCOUNTING_ENV and decide.
The same reconcile and decision as gym eval resubmit --recover: nothing else of the chain runs or waits (the
sentinel checks the queue), and every allocation ended. Prints the reconciled walltime when the run continues.
In the in-job guard: whether the allocation that last wrote the Store has ended.
The guard runs after its predecessor ended, and a chain runs one allocation at a time: every allocation the chain recorded before this one has ended. The writer is often an earlier allocation than the predecessor, for example when the predecessor timed out before it opened the Store. A writer the chain does not name stays unknown.
Elapsed seconds of all recorded allocations, only when each has one terminal accounting row.
stored_walltime_s None means no allocation stored a walltime: the prologue that accounts the previous
allocation stopped first (after an operator’s cancel, for one). The records’ sum is the used walltime then.
Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete.
Every chain job needs terminal records numbered 0..n by restarts, and some job a requeue (n > 0). The stored counter must equal the elapsed time of every incarnation of the jobs before the last requeued one: an earlier requeue was reconciled and stored before the chain went on, and the last requeued incarnation stops before it adds anything, as does every later allocation. Otherwise the budget epoch is unknown.
Whether the first incarnation of allocation_id has ended, by any path (ALLOCATION_ENDED_STATES).
Only the first incarnation (no restarts) runs work: the prologue stops a requeued one before any. A state that is absent, ambiguous or not terminal is not an ending.
Rows of sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT as (job id, state, elapsed s, restarts).
The state keeps its first word without a trailing + (“CANCELLED by 7” is CANCELLED). None when any row is
malformed.
Read saved state in the driver’s installed runtime before a terminal allocation continues.
The shell has already validated and persisted terminal elapsed accounting. This reader never changes
artifacts, starts servers, resets budgets, or treats an exit code as sufficient retry evidence.
outcome_policy is the run’s recovery_outcome_policy section as JSON, rendered from its saved config, so
an automatic continuation decides with the policy a recovery would apply.
Know a requeued chain’s budget again once every incarnation has a terminal scheduler record.
Only the requeue marker is reconciled: the counters themselves stayed readable and the missing fact is the elapsed time of the requeued incarnations, which complete terminal records give. Any other unknown reason stays unknown.
The last requeued job of a chain and what its allocations account for, as a decision states it.
Why the chain stopped when the follow-up job_id that its last allocation claimed is not in the chain.
accounting is sacct -D -X -n -P --format=FOLLOWUP_ACCOUNTING_FORMAT -j job_id. The prologue records an
allocation in the chain before anything else, so a follow-up that started and ended without that entry ran no
work. None when the records do not show that: the follow-up never started, is live, or its record is ambiguous.
Whether the allocation that last wrote the Store has ended (allocation_cut_from_accounting).
A coverage report from a worker older than allocation_id names no writer, but its writer is in the job chain:
the prologue records each allocation there before the Store opens. Then every allocation of the chain must have
ended. A report that records no allocation (a writer outside Slurm) stays unknown.
The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads
as an incomplete run and never as a finished one. Unsettled is wider than the coverage report’s open (the
count in the chain’s own stop line): it also holds final failures that were not counted as zero.