> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.chain

Recovery of a run's auto-resume chain: its evidence and decision, and the resubmit that continues it with
the journaled extensions, raises, repins and operator continuations it records.

## Module Contents

### Classes

| Name                                                                                                   | Description                                                                                           |
| ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| [`ChainRecovery`](#nemo_gym-orchestration-executors-slurm-chain-ChainRecovery)                         | The recovery half of `SlurmExecutor`: read a run's saved evidence, decide, and continue its chain.    |
| [`IncompatibleRerenderError`](#nemo_gym-orchestration-executors-slurm-chain-IncompatibleRerenderError) | A job script would be rendered again by a launcher of another Gym revision than the one the job runs. |

### Functions

| Name                                                                                                                 | Description                                                                                                  |
| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [`_epoch_walltime`](#nemo_gym-orchestration-executors-slurm-chain-_epoch_walltime)                                   | The used walltime of a chain after an operator continuation: its baseline plus every allocation since.       |
| [`_plan_walltime`](#nemo_gym-orchestration-executors-slurm-chain-_plan_walltime)                                     | How the raise `max_walltime` fits the partition `limit`: each segment requests the smaller of the raise      |
| [`_raised_cap`](#nemo_gym-orchestration-executors-slurm-chain-_raised_cap)                                           | The chain cap after a raise to `walltime`: at least the raise, and the raise itself when there was none.     |
| [`_recorded_spinup_timeout_seconds`](#nemo_gym-orchestration-executors-slurm-chain-_recorded_spinup_timeout_seconds) | The benchmark's server spin-up limit in seconds as the run starts its servers; 0 or less is no limit.        |
| [`_repinned_config`](#nemo_gym-orchestration-executors-slurm-chain-_repinned_config)                                 | `config` with the driver's Gym install at `ref`, and the benchmark's MLflow pin tags that named the old one. |
| [`_spinup_timeout_config`](#nemo_gym-orchestration-executors-slurm-chain-_spinup_timeout_config)                     | `config` with the benchmark's server spin-up limit `seconds`; every other field as saved.                    |
| [`_spinup_timeout_raise_refusal`](#nemo_gym-orchestration-executors-slurm-chain-_spinup_timeout_raise_refusal)       | Why `seconds` is not a raise of the `recorded` limit (0 is no limit), or None when it is.                    |
| [`_spinup_timeout_raise_text`](#nemo_gym-orchestration-executors-slurm-chain-_spinup_timeout_raise_text)             | The sentence a decision carries for a spin-up timeout raise.                                                 |
| [`_unsafe_resume_config`](#nemo_gym-orchestration-executors-slurm-chain-_unsafe_resume_config)                       | `config` with the benchmark's job resuming past a refused run identity check; every other field as saved.    |
| [`_walltime_config`](#nemo_gym-orchestration-executors-slurm-chain-_walltime_config)                                 | `config` with a chain cap of at least `walltime` and each Slurm segment requesting `segment` (by default     |

### Data

[`REPIN_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-chain-REPIN_JOURNAL_NAME)

[`RETRY_EXTENSION_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-chain-RETRY_EXTENSION_JOURNAL_NAME)

[`SPINUP_TIMEOUT_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-chain-SPINUP_TIMEOUT_JOURNAL_NAME)

[`UNSAFE_RESUME_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-chain-UNSAFE_RESUME_JOURNAL_NAME)

[`WALLTIME_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-chain-WALLTIME_JOURNAL_NAME)

[`_NAME_OPERATOR_JOB`](#nemo_gym-orchestration-executors-slurm-chain-_NAME_OPERATOR_JOB)

### API

```python
class nemo_gym.orchestration.executors.slurm.chain.ChainRecovery()
```

The recovery half of `SlurmExecutor`: read a run's saved evidence, decide, and continue its chain.

Mixed into `SlurmExecutor`, whose `_saved_config` and `installed_commit` its methods use.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._attempt_extension_commands(
    extend_attempts: int,
    keep_budget: bool
) -> list[str]
```

staticmethod

The locked steps that record `extend_attempts` more attempts for every rollout, restored unless queued.

The Store reads the record (artifacts/rollouts_attempt_extension) as an addition to the run's saved
`max_rollout_attempts`; a rollout that ended with a model failure is never run again.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._extension_commands(
    extend_retries: int,
    keep_budget: bool
) -> list[str]
```

staticmethod

The locked steps that record `extend_retries` more infra retries, restored unless the job is queued.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._fresh_restart_note(
    config: nemo_gym.orchestration.api.SubmitConfig,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    max_walltime: str
) -> str
```

staticmethod

The chain marker a fresh restart writes: why, the recorded walltime and the new one.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._judge_job_states(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> str | None
```

staticmethod

Ask the scheduler about the run's judge-only jobs that have no receipt (`CollectedEvidence.judge_jobs`,
state `queued`), set each one's `state` and `scheduler_state` from its record, and name the newest live
one as `judge-only job <id> (STATE)`; None when none is live, or when the query failed or knows none.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._operator_commands(
    continuation: collections.abc.Mapping[str, typing.Any],
    at: str
) -> list[str]
```

staticmethod

The locked steps of an operator continuation: fill the counters it names, replace the unknown marker and
journal the continuation with its epoch before sbatch, so a queued continuation always has its epoch.

Each file is kept aside first and put back unless the job is queued. A readable counter is never named, except
the retry extension an explicit `--extend-retries` raises when the spent count is readable.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._rerender_commands(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    config: nemo_gym.orchestration.api.SubmitConfig,
    keep_budget: bool,
    repin: str | None = None,
    repin_reason: str | None = None,
    max_walltime: str | None = None,
    walltime_reason: str | None = None,
    unsafe_resume_reason: str | None = None,
    server_spinup_timeout_seconds: int | None = None,
    spinup_timeout_reason: str | None = None,
    fresh_restart: bool = False,
    segment_walltime: str | None = None
) -> tuple[list[str], dict[str, str]]
```

The locked shell steps that render one benchmark's job.sh and saved config again, with a repin, a walltime
raise or both, and the journal entries to append once its job is queued. A fresh restart raises the walltime
above the recorded one without `keep_budget`.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery._rerun_masked_commands(
    approval: collections.abc.Mapping[str, typing.Any],
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    keep_budget: bool
) -> list[str]
```

staticmethod

The locked steps that journal the approval to run masked results again, taken back unless the job is queued.

The line goes to the rerun-masked journal beside the rollouts file, where the Store reads it when the job starts:
its time, user, reason and each approved identity with the attempt it covers and the attempts counted for it.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery.recovery_decision(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

The decision of `recovery_evidence` without a score key.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery.recovery_evidence(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    score_key: list[str | int] | None = None,
    score_agent: str | None = None,
    score_environment: str | None = None,
    outcome_policy: nemo_gym.eval.recovery.policy.RecoveryOutcomePolicyConfig | None = None,
    reader_ref: str | None = None,
    extend_retries: int = 0,
    extend_attempts: int = 0,
    max_walltime: str | None = None,
    operator_budget: nemo_gym.eval.recovery.budgets.OperatorBudget | None = None,
    server_spinup_timeout_seconds: int | None = None,
    spinup_timeout_reason: str | None = None,
    rerun_masked_without_artifact: bool = False
) -> tuple[nemo_gym.eval.recovery.decision.RecoveryDecision, nemo_gym.eval.recovery.evidence.CollectedEvidence | None]
```

Collect the job dir's identities, failures, quality summary, named score and chain budget over SSH.

`reader_ref` runs the Store reader at that full commit instead of the saved one: a repinned run resumes there.
`max_walltime` is a walltime raise the resubmit will render: the plan checks the chain cap it renders.
`server_spinup_timeout_seconds` (with `spinup_timeout_reason`) is a spin-up limit raise the resubmit
will render: the decision names it.
`rerun_masked_without_artifact` plans the masked results that kept a saved answer but have no gradeable
artifact and no grade as retries; the resubmit journals them (see `resubmit`).
`operator_budget` lets a run whose chain accounting is unknown continue (see `decide_from_collected`); with
it, a failed scheduler query is a stop that names it instead of an error.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery.resubmit(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    keep_budget: bool = False,
    accounted_walltime_s: float | None = None,
    accounting_snapshot: collections.abc.Mapping[str, str | None] | None = None,
    accounting_reconciled: bool = False,
    repin: str | None = None,
    repin_reason: str | None = None,
    extend_retries: int = 0,
    extend_attempts: int = 0,
    max_walltime: str | None = None,
    walltime_reason: str | None = None,
    operator_continuation: collections.abc.Mapping[str, typing.Any] | None = None,
    unsafe_resume_reason: str | None = None,
    server_spinup_timeout_seconds: int | None = None,
    spinup_timeout_reason: str | None = None,
    fresh_restart: bool = False,
    segment_walltime: str | None = None,
    rerun_masked: collections.abc.Mapping[str, typing.Any] | None = None
) -> nemo_gym.orchestration.jobs.BenchmarkJob
```

Queue `benchmark`'s job.sh again from its job dir and record the new job id.

With `keep_budget` (`--recover`) the job keeps the spent retry count, walltime and per-rollout attempts;
a job.sh rendered before that option would reset them, so it is refused. `accounting_snapshot` is the
collected evidence's `accounting_snapshot`; it is required with `keep_budget` and rechecked under the
resubmit lock before anything in the job dir changes, so a decision made on stale accounting is refused.
`accounting_reconciled` (with `keep_budget`) admits the requeue marker that the collected evidence
reconciled from terminal scheduler records; it is replaced by a note just before the job is queued.
`repin` (with `keep_budget` and `repin_reason`) continues the run on that full Gym commit, which must be
this launcher's: under the lock, after the live-job and snapshot checks, job.sh and resolved-config.yaml are
rendered again with the new ref (and the MLflow pin tags that named the old one), the old files are kept
beside them, and both are restored if the job is not queued. Rows, attempts and budget files stay as they are;
the queued job's record carries the new commit and the run dir's repin journal records the move.
`extend_retries` (with `keep_budget`) adds that many infra retries to the recorded extension that the
job's prologue adds to the configured cap; it is written just before sbatch, restored if the job is not
queued, and journaled once queued. The spent retry count is never reset.
`max_walltime` (with `keep_budget` and `walltime_reason`) raises the Slurm walltime the same way a repin
moves the commit, in the same render: `compute.<cluster>.walltime` becomes it, the chain cap becomes at least
it, and a value below the recorded walltime is refused. The run dir's walltime journal records the raise.
`segment_walltime` (from `walltime_plan`) is what each Slurm segment requests instead, when the partition's
MaxTime is below the raise: the raise stays the chain cap.
`rerun_masked` (with `keep_budget`) is the operator's approval to run again the masked results that kept a
saved answer but have no gradeable artifact and no grade: its `reason` and `identities` (identity, attempt
id and attempts counted) are appended as one line to the run's rerun-masked journal in artifacts/, which the
Store reads. The line is written just before sbatch and taken back if the job is not queued.
A job the scheduler refuses raises `SchedulerRefusedError` with sbatch's text and the restored job dir.
`unsafe_resume_reason` (with `keep_budget`) renders the job with `allow_unsafe_resume` the same way, so
its resume proceeds past a refused run identity check; the run dir's unsafe-resume journal records the reason
and the user once the job is queued.
`server_spinup_timeout_seconds` (with `spinup_timeout_reason`) renders the job with that server
spin-up limit the same way; it only raises the recorded limit (0 removes it), and the job dir's
spin-up timeout journal records both limits, the reason and the user once the job is queued.
`operator_continuation` (with `keep_budget`) is a decision's OPERATOR-CONTINUATION record for a run whose
accounting is unknown: under the lock it fills the counters the record names, replaces the unknown-accounting
marker with a note and keeps the readable counters; each step is undone unless the job is queued, and the job
dir's operator journal records the queued continuation.
`fresh_restart` (with `max_walltime`, `walltime_reason` and `accounting_snapshot`, without
`keep_budget`) queues a run that has no task inventory again as a fresh run: a fresh retry and walltime
budget, a walltime longer than the recorded one, the snapshot rechecked under the lock. Its walltime journal
entry, the receipt, carries `plan` FRESH-RESTART and `why`; `.gym_chain_incomplete` names the restart
and both walltimes until the queued job continues the chain.

Refuses while any of the caller's jobs runs in that job dir, so two jobs never write one run, and
refuses a run with rollouts whose `gym eval run` job.sh predates resubmit: it would start the run
fresh and delete them. A `command` benchmark's job.sh runs its own command, so it is not refused.

```python
nemo_gym.orchestration.executors.slurm.chain.ChainRecovery.walltime_plan(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    max_walltime: str
) -> dict[str, typing.Any]
```

Check a `--max-walltime` raise against the partition's MaxTime, read with sinfo on the cluster, before
any evidence is read (see `_plan_walltime`). A failed query keeps the whole raise per segment and says so.

```python
class nemo_gym.orchestration.executors.slurm.chain.IncompatibleRerenderError(
    pinned: str,
    launcher: str | None
)
```

**Bases:** `RuntimeError`

A job script would be rendered again by a launcher of another Gym revision than the one the job runs.

The launcher's job.sh templates may need modules the pinned revision does not have, and the capability checks
run later inspect only the newly rendered script, so the mismatch is refused before anything changes.

```python
nemo_gym.orchestration.executors.slurm.chain._epoch_walltime(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    rows: list[tuple[str, str, int, int]] | None
) -> float | None
```

The used walltime of a chain after an operator continuation: its baseline plus every allocation since.

The continuation filled or kept the counter as the baseline, so the allocations before it are not accounted again.
None when the records since then are incomplete or the counter does not match them.

```python
nemo_gym.orchestration.executors.slurm.chain._plan_walltime(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    benchmark: str,
    max_walltime: str,
    limit: dict[str, typing.Any]
) -> dict[str, typing.Any]
```

How the raise `max_walltime` fits the partition `limit`: each segment requests the smaller of the raise
and the partition's MaxTime, and the raise is the chain cap the segments continue under. Refused when no segment
fits: a run that does not chain segments needs the whole raise in one job, and a chained one at least its
recorded segment walltime.

```python
nemo_gym.orchestration.executors.slurm.chain._raised_cap(
    cap: str | None,
    walltime: str
) -> str
```

The chain cap after a raise to `walltime`: at least the raise, and the raise itself when there was none.

```python
nemo_gym.orchestration.executors.slurm.chain._recorded_spinup_timeout_seconds(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark: str
) -> float
```

The benchmark's server spin-up limit in seconds as the run starts its servers; 0 or less is no limit.

```python
nemo_gym.orchestration.executors.slurm.chain._repinned_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark: str,
    ref: str
) -> nemo_gym.orchestration.api.SubmitConfig
```

`config` with the driver's Gym install at `ref`, and the benchmark's MLflow pin tags that named the old one.

```python
nemo_gym.orchestration.executors.slurm.chain._spinup_timeout_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark: str,
    seconds: int
) -> nemo_gym.orchestration.api.SubmitConfig
```

`config` with the benchmark's server spin-up limit `seconds`; every other field as saved.

```python
nemo_gym.orchestration.executors.slurm.chain._spinup_timeout_raise_refusal(
    recorded: float,
    seconds: int
) -> str | None
```

Why `seconds` is not a raise of the `recorded` limit (0 is no limit), or None when it is.

```python
nemo_gym.orchestration.executors.slurm.chain._spinup_timeout_raise_text(
    recorded: float,
    seconds: int,
    reason: str | None
) -> str
```

The sentence a decision carries for a spin-up timeout raise.

```python
nemo_gym.orchestration.executors.slurm.chain._unsafe_resume_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark: str
) -> nemo_gym.orchestration.api.SubmitConfig
```

`config` with the benchmark's job resuming past a refused run identity check; every other field as saved.

```python
nemo_gym.orchestration.executors.slurm.chain._walltime_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    benchmark: str,
    walltime: str,
    segment: str | None = None
) -> nemo_gym.orchestration.api.SubmitConfig
```

`config` with a chain cap of at least `walltime` and each Slurm segment requesting `segment` (by default
`walltime` itself); every other field as saved. A chain without a cap takes `walltime` as its cap: a raise
always bounds the total.

```python
nemo_gym.orchestration.executors.slurm.chain.REPIN_JOURNAL_NAME = 'repin-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.chain.RETRY_EXTENSION_JOURNAL_NAME = 'retry-extension-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.chain.SPINUP_TIMEOUT_JOURNAL_NAME = 'spinup-timeout-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.chain.UNSAFE_RESUME_JOURNAL_NAME = 'unsafe-resume-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.chain.WALLTIME_JOURNAL_NAME = 'walltime-extension-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.chain._NAME_OPERATOR_JOB = "import json,os,sys; p=sys.argv[1]; lines=open(p).read().splitlines(); entry=jso...
```