nemo_gym.orchestration.executors.slurm_control
nemo_gym.orchestration.executors.slurm_control
How the Slurm executor stops the jobs it queued.
Stopping a benchmark is more than scancel on its latest job. Under auto-resume every job queues its own
follow-up, so a follow-up that is still pending would start after its parent is cancelled. The script finds
everything of the benchmark that is queued or running — by the directory it runs in, and by the job ids
the chain and the follow-up claim files name — and cancels the pending jobs before the running ones, so no
follow-up can start in between.
A job that is being cancelled can still queue its follow-up in the moments before it dies, after the queue was read. So the queue is read again until nothing of the benchmark is left in it, and whatever appeared in between is cancelled too.
Module Contents
Functions
Data
API
The job ids render_terminate_script cancelled, and what went wrong, if anything.
The commands that cancel every queued or running job of job: the pending ones first, then the rest.