Skip to navigation

nemo_gym.eval.recovery.continuation

View as MarkdownOpen in Claude

Whether a chain continues: the admission a terminal allocation runs in its installed runtime, and the accounting of a chain’s allocations, including requeued ones.

Module Contents

Functions

NameDescription
_requeue_continuationIn a sentinel: reconcile a requeued chain from the terminal records in CHAIN_ACCOUNTING_ENV and decide.
_store_writer_endedIn the in-job guard: whether the allocation that last wrote the Store has ended.
accounted_chain_walltimeElapsed seconds of all recorded allocations, only when each has one terminal accounting row.
accounted_requeued_chain_walltimeElapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete.
allocation_cut_from_accountingWhether the first incarnation of allocation_id has ended, by any path (ALLOCATION_ENDED_STATES).
chain_accountingRows of sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT as (job id, state, elapsed s, restarts).
continuation_admissionRead saved state in the driver’s installed runtime before a terminal allocation continues.
reconcile_requeued_chainKnow a requeued chain’s budget again once every incarnation has a terminal scheduler record.
requeue_noteThe last requeued job of a chain and what its allocations account for, as a decision states it.
stopped_followupWhy the chain stopped when the follow-up job_id that its last allocation claimed is not in the chain.
store_writer_cut_from_accountingWhether the allocation that last wrote the Store has ended (allocation_cut_from_accounting).
unsettled_chain_noteThe chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads

Data

FOLLOWUP_ACCOUNTING_FORMAT

_FOLLOWUP_ENDED_STATES

API

nemo_gym.eval.recovery.continuation._requeue_continuation(
job: pathlib.Path,
max_retries: int,
max_walltime_s: float | None
) -> str

In a sentinel: reconcile a requeued chain from the terminal records in CHAIN_ACCOUNTING_ENV and decide.

The same reconcile and decision as gym eval resubmit --recover: nothing else of the chain runs or waits (the sentinel checks the queue), and every allocation ended. Prints the reconciled walltime when the run continues.

nemo_gym.eval.recovery.continuation._store_writer_ended(
previous_job_id: str
) -> bool

In the in-job guard: whether the allocation that last wrote the Store has ended.

The guard runs after its predecessor ended, and a chain runs one allocation at a time: every allocation the chain recorded before this one has ended. The writer is often an earlier allocation than the predecessor, for example when the predecessor timed out before it opened the Store. A writer the chain does not name stays unknown.

nemo_gym.eval.recovery.continuation.accounted_chain_walltime(
job_chain: list[str],
accounting: str,
stored_walltime_s: float | None = 0
) -> float | None

Elapsed seconds of all recorded allocations, only when each has one terminal accounting row.

stored_walltime_s None means no allocation stored a walltime: the prologue that accounts the previous allocation stopped first (after an operator’s cancel, for one). The records’ sum is the used walltime then.

nemo_gym.eval.recovery.continuation.accounted_requeued_chain_walltime(
job_chain: list[str],
rows: list[tuple[str, str, int, int]],
stored_walltime_s: float = 0
) -> float | None

Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete.

Every chain job needs terminal records numbered 0..n by restarts, and some job a requeue (n > 0). The stored counter must equal the elapsed time of every incarnation of the jobs before the last requeued one: an earlier requeue was reconciled and stored before the chain went on, and the last requeued incarnation stops before it adds anything, as does every later allocation. Otherwise the budget epoch is unknown.

nemo_gym.eval.recovery.continuation.allocation_cut_from_accounting(
rows: list[tuple[str, str, int, int]],
allocation_id: str | None
) -> bool

Whether the first incarnation of allocation_id has ended, by any path (ALLOCATION_ENDED_STATES).

Only the first incarnation (no restarts) runs work: the prologue stops a requeued one before any. A state that is absent, ambiguous or not terminal is not an ending.

nemo_gym.eval.recovery.continuation.chain_accounting(
accounting: str
) -> list[tuple[str, str, int, int]] | None

Rows of sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT as (job id, state, elapsed s, restarts).

The state keeps its first word without a trailing + (“CANCELLED by 7” is CANCELLED). None when any row is malformed.

nemo_gym.eval.recovery.continuation.continuation_admission(
job_dir: str,
max_retries: int,
max_walltime_s: float | None,
exit_code: str,
previous_job_id: str,
previous_state: str = 'FAILED',
outcome_policy: str | None = None
) -> str

Read saved state in the driver’s installed runtime before a terminal allocation continues.

The shell has already validated and persisted terminal elapsed accounting. This reader never changes artifacts, starts servers, resets budgets, or treats an exit code as sufficient retry evidence. outcome_policy is the run’s recovery_outcome_policy section as JSON, rendered from its saved config, so an automatic continuation decides with the policy a recovery would apply.

nemo_gym.eval.recovery.continuation.reconcile_requeued_chain(
rows: list[tuple[str, str, int, int]]

Know a requeued chain’s budget again once every incarnation has a terminal scheduler record.

Only the requeue marker is reconciled: the counters themselves stayed readable and the missing fact is the elapsed time of the requeued incarnations, which complete terminal records give. Any other unknown reason stays unknown.

nemo_gym.eval.recovery.continuation.requeue_note(
job_chain: list[str],
rows: list[tuple[str, str, int, int]]
) -> str | None

The last requeued job of a chain and what its allocations account for, as a decision states it.

nemo_gym.eval.recovery.continuation.stopped_followup(
job_id: str,
accounting: str
) -> str | None

Why the chain stopped when the follow-up job_id that its last allocation claimed is not in the chain.

accounting is sacct -D -X -n -P --format=FOLLOWUP_ACCOUNTING_FORMAT -j job_id. The prologue records an allocation in the chain before anything else, so a follow-up that started and ended without that entry ran no work. None when the records do not show that: the follow-up never started, is live, or its record is ambiguous.

nemo_gym.eval.recovery.continuation.store_writer_cut_from_accounting(
rows: list[tuple[str, str, int, int]],
) -> bool

Whether the allocation that last wrote the Store has ended (allocation_cut_from_accounting).

A coverage report from a worker older than allocation_id names no writer, but its writer is in the job chain: the prologue records each allocation there before the Store opens. Then every allocation of the chain must have ended. A report that records no allocation (a writer outside Slurm) stays unknown.

nemo_gym.eval.recovery.continuation.unsettled_chain_note(
) -> str

The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads as an incomplete run and never as a finished one. Unsettled is wider than the coverage report’s open (the count in the chain’s own stop line): it also holds final failures that were not counted as zero.

nemo_gym.eval.recovery.continuation.FOLLOWUP_ACCOUNTING_FORMAT = 'JobIDRaw,State,ElapsedRaw,ExitCode'
nemo_gym.eval.recovery.continuation._FOLLOWUP_ENDED_STATES = frozenset({'FAILED', 'CANCELLED', 'NODE_FAIL', 'BOOT_FAIL', 'PREEMPTED', 'OUT_OF...