nemo_gym.orchestration.executors.resume_script

The auto-resume prologue: a self-chaining Slurm job.

Rendered at the top of a benchmark’s sbatch script, ahead of every service/driver command, when BenchmarkRunConfig.resumable is set. Each job in the chain inspects the PREVIOUS job (passed as $1) via sacct, decides whether to continue, and unconditionally queues its own successor with an afternotok dependency before doing any real work. A job that COMPLETEs leaves its queued successor with a dependency that can never be satisfied, so Slurm cancels it on its own — the chain needs no explicit “we’re done” step on the success path.

Module Contents

Functions

Name	Description
`_walltime_to_seconds`	Convert a Slurm-style duration string to seconds.
`render_resume_prologue`	Render the auto-resume prologue for one benchmark’s sbatch script.

Data

_JOB_SCRIPT_NAME

_MAX_WALLTIME_CHECK_TEMPLATE

_PROLOGUE_TEMPLATE

API

nemo_gym.orchestration.executors.resume_script._walltime_to_seconds(
    duration: str
) -> int

Convert a Slurm-style duration string to seconds.

Covers the subset of Slurm’s --time grammar that max_walltime accepts: MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS — the same spellings Slurm itself takes, so a value can be copied straight from compute.walltime.

nemo_gym.orchestration.executors.resume_script.render_resume_prologue(
    resume: nemo_gym.orchestration.api.ResumeConfig
) -> str

Render the auto-resume prologue for one benchmark’s sbatch script.

nemo_gym.orchestration.executors.resume_script._JOB_SCRIPT_NAME = 'job.sh'

nemo_gym.orchestration.executors.resume_script._MAX_WALLTIME_CHECK_TEMPLATE = '        if (( _gym_accumulated >= {seconds} )); then\n            echo "Accumul...

nemo_gym.orchestration.executors.resume_script._PROLOGUE_TEMPLATE = '# --- Auto-resume chain ---\nOUTPUT_DIR="$PWD"\n_this_script="$OUTPUT_DIR/{job_...