nemo_gym.orchestration.executors.slurm.resume_script
nemo_gym.orchestration.executors.slurm.resume_script
The auto-resume prologue: a self-chaining Slurm job.
Rendered at the top of a benchmark’s sbatch script, ahead of every
service/driver command, when BenchmarkRunConfig.resumable is set. Each job
in the chain inspects the PREVIOUS job (passed as $1) via sacct, decides
whether to continue, and unconditionally queues its own successor with an
afternotok dependency before doing any real work, and a sentinel that
continues the chain if that successor dies before its prologue. A job that
COMPLETEs leaves its queued successor with a dependency that can never be
satisfied, so Slurm would cancel it on its own, but the sentinel would then
start on a full allocation only to exit. So the epilogue, rendered after the
driver step, cancels both when the job finished the run.
Module Contents
Functions
Data
API
Convert a Slurm-style duration string to seconds.
Covers the subset of Slurm’s --time grammar that max_walltime accepts:
MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS — the same
spellings Slurm itself takes, so a value can be copied straight from
compute.walltime.
The command that queues a run with no task inventory again, as a fresh run with a longer walltime.
Submission ids and benchmark names are shell-safe by construction (see new_gym_job_id and the executor’s
benchmark name check), so the command needs no quoting, and the prologue renders it with shell expansions of both.
Render the step after the driver of a resumable job: a job that finished the run ends its own chain.
When the driver step succeeded and the run is complete, it writes the record the follow-up or its sentinel would
have written, cancels the sentinel and then the follow-up it queued, and journals both in
CHAIN_END_JOURNAL_NAME, so no allocation starts for a finished run. A driver that exits
NO_ELIGIBLE_RETRY_EXIT (nothing is eligible for another attempt) ends its chain the same way, with the
NO_ELIGIBLE_RETRY_REASON stop, unless it left in DRIVER_FAILURE_NAME a failure kind the follow-up’s
prologue turns into its own stop (spin-up timeout, install failed, resume refused, credentials rejected). The job keeps the driver step’s status.
Render the auto-resume prologue for one benchmark’s sbatch script.