nemo_gym.orchestration.executors.resume_script
nemo_gym.orchestration.executors.resume_script
The auto-resume prologue: a self-chaining Slurm job.
Rendered at the top of a benchmark’s sbatch script, ahead of every
service/driver command, when BenchmarkRunConfig.resumable is set. Each job
in the chain inspects the PREVIOUS job (passed as $1) via sacct, decides
whether to continue, and unconditionally queues its own successor with an
afternotok dependency before doing any real work, and a sentinel that
continues the chain if that successor dies before its prologue. A job that
COMPLETEs leaves its queued successor with a dependency that can never be
satisfied, so Slurm would cancel it on its own, but the sentinel would then
start on a full allocation only to exit. So the epilogue, rendered after the
driver step, cancels both when the job finished the run.
Module Contents
Functions
Data
API
Convert a Slurm-style duration string to seconds.
Covers the subset of Slurm’s --time grammar that max_walltime accepts:
MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS — the same
spellings Slurm itself takes, so a value can be copied straight from
compute.walltime.
The command that queues a run with no task inventory again, as a fresh run with a longer walltime.
Submission ids and benchmark names are shell-safe by construction (see new_gym_job_id and the executor’s
benchmark name check), so the command needs no quoting, and the prologue renders it with shell expansions of both.
Leave the registered kind error carries in the failure file the driver step names, for the next guard.
The driver step of a resumable job names the file in DRIVER_FAILURE_ENV; elsewhere this does nothing.
Render the step after the driver of a resumable job: a job that finished the run ends its own chain.
When the driver step succeeded and the run is complete, it writes the record the follow-up or its sentinel would
have written, cancels the sentinel and then the follow-up it queued, and journals both in
CHAIN_END_JOURNAL_NAME, so no allocation starts for a finished run. A driver that exits
NO_ELIGIBLE_RETRY_EXIT (nothing is eligible for another attempt) ends its chain the same way, with the
NO_ELIGIBLE_RETRY_REASON stop, unless it left in DRIVER_FAILURE_NAME a failure kind the follow-up’s
prologue turns into its own stop (spin-up timeout, install failed, resume refused, credentials rejected). The job keeps the driver step’s status.
Render the auto-resume prologue for one benchmark’s sbatch script.