> This page is for version 0.3.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.resume_script

The auto-resume prologue: a self-chaining Slurm job.

Rendered at the top of a benchmark's sbatch script, ahead of every
service/driver command, when `BenchmarkRunConfig.resumable` is set. Each job
in the chain inspects the PREVIOUS job (passed as `$1`) via `sacct`, decides
whether to continue, and unconditionally queues its own successor with an
`afternotok` dependency before doing any real work, and a sentinel that
continues the chain if that successor dies before its prologue. A job that
COMPLETEs leaves its queued successor with a dependency that can never be
satisfied, so Slurm would cancel it on its own, but the sentinel would then
start on a full allocation only to exit. So the epilogue, rendered after the
driver step, cancels both when the job finished the run.

## Module Contents

### Functions

| Name                                                                                                     | Description                                                                                          |
| -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`_walltime_to_seconds`](#nemo_gym-orchestration-executors-slurm-resume_script-_walltime_to_seconds)     | Convert a Slurm-style duration string to seconds.                                                    |
| [`fresh_restart_command`](#nemo_gym-orchestration-executors-slurm-resume_script-fresh_restart_command)   | The command that queues a run with no task inventory again, as a fresh run with a longer walltime.   |
| [`render_resume_epilogue`](#nemo_gym-orchestration-executors-slurm-resume_script-render_resume_epilogue) | Render the step after the driver of a resumable job: a job that finished the run ends its own chain. |
| [`render_resume_prologue`](#nemo_gym-orchestration-executors-slurm-resume_script-render_resume_prologue) | Render the auto-resume prologue for one benchmark's sbatch script.                                   |

### Data

[`ALLOCATION_ACCOUNTING_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-ALLOCATION_ACCOUNTING_MARKER)

[`BOUNDED_BUDGET_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-BOUNDED_BUDGET_MARKER)

[`CHAIN_ACCOUNTING_FORMAT`](#nemo_gym-orchestration-executors-slurm-resume_script-CHAIN_ACCOUNTING_FORMAT)

[`CHAIN_END_JOURNAL_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-CHAIN_END_JOURNAL_NAME)

[`CREDENTIALS_REJECTED_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-CREDENTIALS_REJECTED_REASON)

[`DRIVER_FAILURE_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-DRIVER_FAILURE_NAME)

[`EXPORT_RECEIPT_MISSING`](#nemo_gym-orchestration-executors-slurm-resume_script-EXPORT_RECEIPT_MISSING)

[`EXPORT_RECEIPT_MISSING_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-EXPORT_RECEIPT_MISSING_REASON)

[`FRESH_RESTART_MARKER_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-FRESH_RESTART_MARKER_NAME)

[`INCOMPLETE_MARKER_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-INCOMPLETE_MARKER_NAME)

[`INSTALL_ERROR_MAX_CHARS`](#nemo_gym-orchestration-executors-slurm-resume_script-INSTALL_ERROR_MAX_CHARS)

[`JOB_ALLOCATIONS_BUDGET_S`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_ALLOCATIONS_BUDGET_S)

[`JOB_ALLOCATIONS_FORMAT`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_ALLOCATIONS_FORMAT)

[`JOB_ALLOCATIONS_LOCK_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_ALLOCATIONS_LOCK_NAME)

[`JOB_ALLOCATIONS_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_ALLOCATIONS_NAME)

[`JOB_ALLOCATIONS_RETRY_S`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_ALLOCATIONS_RETRY_S)

[`JOB_CHAIN_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-JOB_CHAIN_NAME)

[`KEEP_BUDGET_ENV`](#nemo_gym-orchestration-executors-slurm-resume_script-KEEP_BUDGET_ENV)

[`NO_ELIGIBLE_RETRY`](#nemo_gym-orchestration-executors-slurm-resume_script-NO_ELIGIBLE_RETRY)

[`NO_ELIGIBLE_RETRY_EXIT`](#nemo_gym-orchestration-executors-slurm-resume_script-NO_ELIGIBLE_RETRY_EXIT)

[`NO_PROGRESS_SEGMENT_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-NO_PROGRESS_SEGMENT_REASON)

[`PREPARE_FAILED_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-PREPARE_FAILED_REASON)

[`PRE_INVENTORY_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-PRE_INVENTORY_REASON)

[`PROLOGUE_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-PROLOGUE_MARKER)

[`PROTOCOL_VIOLATION_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-PROTOCOL_VIOLATION_REASON)

[`QUOTA_EXHAUSTED_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-QUOTA_EXHAUSTED_REASON)

[`REQUEUE_RECONCILED_NOTE`](#nemo_gym-orchestration-executors-slurm-resume_script-REQUEUE_RECONCILED_NOTE)

[`REQUEUE_SETTLE_BOUND_S`](#nemo_gym-orchestration-executors-slurm-resume_script-REQUEUE_SETTLE_BOUND_S)

[`REQUEUE_SETTLE_POLL_S`](#nemo_gym-orchestration-executors-slurm-resume_script-REQUEUE_SETTLE_POLL_S)

[`RESUBMIT_ENV`](#nemo_gym-orchestration-executors-slurm-resume_script-RESUBMIT_ENV)

[`RESUBMIT_LOCK_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-RESUBMIT_LOCK_NAME)

[`RESUME_REFUSED_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-RESUME_REFUSED_REASON)

[`RETRY_EXTENSION_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-RETRY_EXTENSION_MARKER)

[`ROWS_AT_START_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-ROWS_AT_START_NAME)

[`RUNTIME_ADMISSION_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-RUNTIME_ADMISSION_MARKER)

[`SCHEDULER_CAUSE_MARKER`](#nemo_gym-orchestration-executors-slurm-resume_script-SCHEDULER_CAUSE_MARKER)

[`SPINUP_TIMEOUT_REASON`](#nemo_gym-orchestration-executors-slurm-resume_script-SPINUP_TIMEOUT_REASON)

[`_EPILOGUE_TEMPLATE`](#nemo_gym-orchestration-executors-slurm-resume_script-_EPILOGUE_TEMPLATE)

[`_JOB_SCRIPT_NAME`](#nemo_gym-orchestration-executors-slurm-resume_script-_JOB_SCRIPT_NAME)

[`_MAX_WALLTIME_CHECK_TEMPLATE`](#nemo_gym-orchestration-executors-slurm-resume_script-_MAX_WALLTIME_CHECK_TEMPLATE)

[`_PROLOGUE_TEMPLATE`](#nemo_gym-orchestration-executors-slurm-resume_script-_PROLOGUE_TEMPLATE)

[`_RECEIPT_ARTIFACTS`](#nemo_gym-orchestration-executors-slurm-resume_script-_RECEIPT_ARTIFACTS)

[`logger`](#nemo_gym-orchestration-executors-slurm-resume_script-logger)

### API

```python
nemo_gym.orchestration.executors.slurm.resume_script._walltime_to_seconds(
    duration: str
) -> int
```

Convert a Slurm-style duration string to seconds.

Covers the subset of Slurm's `--time` grammar that `max_walltime` accepts:
`MM`, `MM:SS`, `HH:MM:SS`, `D-HH`, `D-HH:MM`, `D-HH:MM:SS` -- the same
spellings Slurm itself takes, so a value can be copied straight from
`compute.walltime`.

```python
nemo_gym.orchestration.executors.slurm.resume_script.fresh_restart_command(
    gym_job_id: str,
    benchmark: str
) -> str
```

The command that queues a run with no task inventory again, as a fresh run with a longer walltime.

Submission ids and benchmark names are shell-safe by construction (see `new_gym_job_id` and the executor's
benchmark name check), so the command needs no quoting, and the prologue renders it with shell expansions of both.

```python
nemo_gym.orchestration.executors.slurm.resume_script.render_resume_epilogue() -> str
```

Render the step after the driver of a resumable job: a job that finished the run ends its own chain.

When the driver step succeeded and the run is complete, it writes the record the follow-up or its sentinel would
have written, cancels the sentinel and then the follow-up it queued, and journals both in
`CHAIN_END_JOURNAL_NAME`, so no allocation starts for a finished run. A driver that exits
`NO_ELIGIBLE_RETRY_EXIT` (nothing is eligible for another attempt) ends its chain the same way, with the
`NO_ELIGIBLE_RETRY_REASON` stop, unless it left in `DRIVER_FAILURE_NAME` a failure kind the follow-up's
prologue turns into its own stop (spin-up timeout, install failed, resume refused, credentials rejected). The job keeps the driver step's status.

```python
nemo_gym.orchestration.executors.slurm.resume_script.render_resume_prologue(
    resume: nemo_gym.orchestration.api.ResumeConfig,
    admission_command: str = 'false'
) -> str
```

Render the auto-resume prologue for one benchmark's sbatch script.

```python
nemo_gym.orchestration.executors.slurm.resume_script.ALLOCATION_ACCOUNTING_MARKER = '# --- Durable allocation accounting v1 ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.BOUNDED_BUDGET_MARKER = '# --- Bounded resume budgets v1 ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.CHAIN_ACCOUNTING_FORMAT = 'JobIDRaw,State,ElapsedRaw,Restarts'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.CHAIN_END_JOURNAL_NAME = 'chain-end-journal.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.CREDENTIALS_REJECTED_REASON = f'{PROVIDER_CREDENTIALS_REJECTED}: the provider refused the credentials; the cha...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.DRIVER_FAILURE_NAME = '.gym_driver_failure'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.EXPORT_RECEIPT_MISSING = 'run_complete_export_receipt_missing'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.EXPORT_RECEIPT_MISSING_REASON = f'{EXPORT_RECEIPT_MISSING}: the run is complete, but its MLflow export receipt (...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.FRESH_RESTART_MARKER_NAME = '.gym_fresh_restart'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.INCOMPLETE_MARKER_NAME = '.gym_chain_incomplete'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.INSTALL_ERROR_MAX_CHARS = 200
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_ALLOCATIONS_BUDGET_S = 30
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_ALLOCATIONS_FORMAT = 'JobIDRaw,State,Start,End,ElapsedRaw,NNodes,NodeList,AllocTRES,Restarts'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_ALLOCATIONS_LOCK_NAME = '.gym_job_allocations.lock'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_ALLOCATIONS_NAME = 'job-allocations.jsonl'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_ALLOCATIONS_RETRY_S = 5
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.JOB_CHAIN_NAME = '.gym_job_chain'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.KEEP_BUDGET_ENV = 'GYM_RESUBMIT_KEEP_BUDGET'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.NO_ELIGIBLE_RETRY = 'no_eligible_retry'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.NO_ELIGIBLE_RETRY_EXIT = 76
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.NO_PROGRESS_SEGMENT_REASON = "the segment saved no rollout: its walltime is shorter than this run's setup (Gy...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.PREPARE_FAILED_REASON = f'{ENVIRONMENT_PREPARE_FAILED}: EVAL STOPPED: data preparation failed; the chain...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.PRE_INVENTORY_REASON = 'the segment ended before the task inventory was written; nothing can be resumed...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.PROLOGUE_MARKER = '# --- Auto-resume chain ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.PROTOCOL_VIOLATION_REASON = f'{ENVIRONMENT_PROTOCOL_VIOLATION}: an environment server answered with a reply ...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.QUOTA_EXHAUSTED_REASON = f'{PROVIDER_QUOTA_EXHAUSTED}: the grading provider's quota is exhausted; the cha...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.REQUEUE_RECONCILED_NOTE = 'Requeued allocation accounting was reconciled from terminal scheduler records; ...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.REQUEUE_SETTLE_BOUND_S = 300
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.REQUEUE_SETTLE_POLL_S = 30
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.RESUBMIT_ENV = 'GYM_RESUBMIT'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.RESUBMIT_LOCK_NAME = '.gym_resubmit.lock'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.RESUME_REFUSED_REASON = f'{COLLECTION_RESUME_REFUSED}: the saved rollouts do not verify against this run...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.RETRY_EXTENSION_MARKER = '# --- Recorded retry extension v1 ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.ROWS_AT_START_NAME = '.gym_rows_at_start'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.RUNTIME_ADMISSION_MARKER = '# --- Installed-runtime continuation admission v1 ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.SCHEDULER_CAUSE_MARKER = '# --- Scheduler-cause retry admission v1 ---'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.SPINUP_TIMEOUT_REASON = f'{SERVER_SPINUP_TIMEOUT}: the Gym servers did not become ready within server_sp...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script._EPILOGUE_TEMPLATE = '# --- A job that finishes the run ends its own chain ---\n_gym_driver_rc=$?\n_g...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script._JOB_SCRIPT_NAME = 'job.sh'
```

```python
nemo_gym.orchestration.executors.slurm.resume_script._MAX_WALLTIME_CHECK_TEMPLATE = 'if (( _gym_accumulated >= {seconds} )); then\n    _gym_stop_incomplete "Accumul...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script._PROLOGUE_TEMPLATE = '# --- Auto-resume chain ---\nOUTPUT_DIR="$PWD"\n_this_script="$OUTPUT_DIR/{job_...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script._RECEIPT_ARTIFACTS = (('rollouts_jsonl_fpath', 'rollouts.jsonl'), ('resolved_config_yaml_fpath', 'rol...
```

```python
nemo_gym.orchestration.executors.slurm.resume_script.logger = logging.getLogger(__name__)
```