> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.continuation

Whether a chain continues: the admission a terminal allocation runs in its installed runtime, and the
accounting of a chain's allocations, including requeued ones.

## Module Contents

### Functions

| Name                                                                                                          | Description                                                                                                |
| ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`_requeue_continuation`](#nemo_gym-eval-recovery-continuation-_requeue_continuation)                         | In a sentinel: reconcile a requeued chain from the terminal records in `CHAIN_ACCOUNTING_ENV` and decide.  |
| [`_store_writer_ended`](#nemo_gym-eval-recovery-continuation-_store_writer_ended)                             | In the in-job guard: whether the allocation that last wrote the Store has ended.                           |
| [`accounted_chain_walltime`](#nemo_gym-eval-recovery-continuation-accounted_chain_walltime)                   | Elapsed seconds of all recorded allocations, only when each has one terminal accounting row.               |
| [`accounted_requeued_chain_walltime`](#nemo_gym-eval-recovery-continuation-accounted_requeued_chain_walltime) | Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete. |
| [`allocation_cut_from_accounting`](#nemo_gym-eval-recovery-continuation-allocation_cut_from_accounting)       | Whether the first incarnation of `allocation_id` has ended, by any path (`ALLOCATION_ENDED_STATES`).       |
| [`chain_accounting`](#nemo_gym-eval-recovery-continuation-chain_accounting)                                   | Rows of `sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT` as (job id, state, elapsed s, restarts).      |
| [`continuation_admission`](#nemo_gym-eval-recovery-continuation-continuation_admission)                       | Read saved state in the driver's installed runtime before a terminal allocation continues.                 |
| [`reconcile_requeued_chain`](#nemo_gym-eval-recovery-continuation-reconcile_requeued_chain)                   | Know a requeued chain's budget again once every incarnation has a terminal scheduler record.               |
| [`requeue_note`](#nemo_gym-eval-recovery-continuation-requeue_note)                                           | The last requeued job of a chain and what its allocations account for, as a decision states it.            |
| [`stopped_followup`](#nemo_gym-eval-recovery-continuation-stopped_followup)                                   | Why the chain stopped when the follow-up `job_id` that its last allocation claimed is not in the chain.    |
| [`store_writer_cut_from_accounting`](#nemo_gym-eval-recovery-continuation-store_writer_cut_from_accounting)   | Whether the allocation that last wrote the Store has ended (`allocation_cut_from_accounting`).             |
| [`unsettled_chain_note`](#nemo_gym-eval-recovery-continuation-unsettled_chain_note)                           | The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads     |

### Data

[`FOLLOWUP_ACCOUNTING_FORMAT`](#nemo_gym-eval-recovery-continuation-FOLLOWUP_ACCOUNTING_FORMAT)

[`_FOLLOWUP_ENDED_STATES`](#nemo_gym-eval-recovery-continuation-_FOLLOWUP_ENDED_STATES)

### API

```python
nemo_gym.eval.recovery.continuation._requeue_continuation(
    job: pathlib.Path,
    max_retries: int,
    max_walltime_s: float | None
) -> str
```

In a sentinel: reconcile a requeued chain from the terminal records in `CHAIN_ACCOUNTING_ENV` and decide.

The same reconcile and decision as `gym eval resubmit --recover`: nothing else of the chain runs or waits (the
sentinel checks the queue), and every allocation ended. Prints the reconciled walltime when the run continues.

```python
nemo_gym.eval.recovery.continuation._store_writer_ended(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    previous_job_id: str
) -> bool
```

In the in-job guard: whether the allocation that last wrote the Store has ended.

The guard runs after its predecessor ended, and a chain runs one allocation at a time: every allocation the chain
recorded before this one has ended. The writer is often an earlier allocation than the predecessor, for example
when the predecessor timed out before it opened the Store. A writer the chain does not name stays unknown.

```python
nemo_gym.eval.recovery.continuation.accounted_chain_walltime(
    job_chain: list[str],
    accounting: str,
    stored_walltime_s: float | None = 0
) -> float | None
```

Elapsed seconds of all recorded allocations, only when each has one terminal accounting row.

`stored_walltime_s` None means no allocation stored a walltime: the prologue that accounts the previous
allocation stopped first (after an operator's cancel, for one). The records' sum is the used walltime then.

```python
nemo_gym.eval.recovery.continuation.accounted_requeued_chain_walltime(
    job_chain: list[str],
    rows: list[tuple[str, str, int, int]],
    stored_walltime_s: float = 0
) -> float | None
```

Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete.

Every chain job needs terminal records numbered 0..n by restarts, and some job a requeue (n > 0). The stored
counter must equal the elapsed time of every incarnation of the jobs before the last requeued one: an earlier
requeue was reconciled and stored before the chain went on, and the last requeued incarnation stops before it
adds anything, as does every later allocation. Otherwise the budget epoch is unknown.

```python
nemo_gym.eval.recovery.continuation.allocation_cut_from_accounting(
    rows: list[tuple[str, str, int, int]],
    allocation_id: str | None
) -> bool
```

Whether the first incarnation of `allocation_id` has ended, by any path (`ALLOCATION_ENDED_STATES`).

Only the first incarnation (no restarts) runs work: the prologue stops a requeued one before any. A state that is
absent, ambiguous or not terminal is not an ending.

```python
nemo_gym.eval.recovery.continuation.chain_accounting(
    accounting: str
) -> list[tuple[str, str, int, int]] | None
```

Rows of `sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT` as (job id, state, elapsed s, restarts).

The state keeps its first word without a trailing `+` ("CANCELLED by 7" is CANCELLED). None when any row is
malformed.

```python
nemo_gym.eval.recovery.continuation.continuation_admission(
    job_dir: str,
    max_retries: int,
    max_walltime_s: float | None,
    exit_code: str,
    previous_job_id: str,
    previous_state: str = 'FAILED',
    outcome_policy: str | None = None
) -> str
```

Read saved state in the driver's installed runtime before a terminal allocation continues.

The shell has already validated and persisted terminal elapsed accounting. This reader never changes
artifacts, starts servers, resets budgets, or treats an exit code as sufficient retry evidence.
`outcome_policy` is the run's `recovery_outcome_policy` section as JSON, rendered from its saved config, so
an automatic continuation decides with the policy a recovery would apply.

```python
nemo_gym.eval.recovery.continuation.reconcile_requeued_chain(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    rows: list[tuple[str, str, int, int]]
) -> nemo_gym.eval.recovery.evidence.CollectedEvidence
```

Know a requeued chain's budget again once every incarnation has a terminal scheduler record.

Only the requeue marker is reconciled: the counters themselves stayed readable and the missing fact is the elapsed
time of the requeued incarnations, which complete terminal records give. Any other unknown reason stays unknown.

```python
nemo_gym.eval.recovery.continuation.requeue_note(
    job_chain: list[str],
    rows: list[tuple[str, str, int, int]]
) -> str | None
```

The last requeued job of a chain and what its allocations account for, as a decision states it.

```python
nemo_gym.eval.recovery.continuation.stopped_followup(
    job_id: str,
    accounting: str
) -> str | None
```

Why the chain stopped when the follow-up `job_id` that its last allocation claimed is not in the chain.

`accounting` is `sacct -D -X -n -P --format=FOLLOWUP_ACCOUNTING_FORMAT -j job_id`. The prologue records an
allocation in the chain before anything else, so a follow-up that started and ended without that entry ran no
work. None when the records do not show that: the follow-up never started, is live, or its record is ambiguous.

```python
nemo_gym.eval.recovery.continuation.store_writer_cut_from_accounting(
    rows: list[tuple[str, str, int, int]],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> bool
```

Whether the allocation that last wrote the Store has ended (`allocation_cut_from_accounting`).

A coverage report from a worker older than `allocation_id` names no writer, but its writer is in the job chain:
the prologue records each allocation there before the Store opens. Then every allocation of the chain must have
ended. A report that records no allocation (a writer outside Slurm) stays unknown.

```python
nemo_gym.eval.recovery.continuation.unsettled_chain_note(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision
) -> str
```

The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads
as an incomplete run and never as a finished one. Unsettled is wider than the coverage report's `open` (the
count in the chain's own stop line): it also holds final failures that were not counted as zero.

```python
nemo_gym.eval.recovery.continuation.FOLLOWUP_ACCOUNTING_FORMAT = 'JobIDRaw,State,ElapsedRaw,ExitCode'
```

```python
nemo_gym.eval.recovery.continuation._FOLLOWUP_ENDED_STATES = frozenset({'FAILED', 'CANCELLED', 'NODE_FAIL', 'BOOT_FAIL', 'PREEMPTED', 'OUT_OF...
```