Skip to navigation

nemo_gym.orchestration.executors.slurm_control

View as MarkdownOpen in Claude

How the Slurm executor stops the jobs it queued.

Stopping a benchmark is more than scancel on its latest job. Under auto-resume every job queues its own follow-up, so a follow-up that is still pending would start after its parent is cancelled. The script finds everything of the benchmark that is queued or running — by the directory it runs in, and by the job ids the chain and the follow-up claim files name — and cancels the pending jobs before the running ones, so no follow-up can start in between.

A job that is being cancelled can still queue its follow-up in the moments before it dies, after the queue was read. So the queue is read again until nothing of the benchmark is left in it, and whatever appeared in between is cancelled too.

Module Contents

Functions

NameDescription
_line-
parse_terminate_outputThe job ids render_terminate_script cancelled, and what went wrong, if anything.
render_terminate_scriptThe commands that cancel every queued or running job of job: the pending ones first, then the rest.

Data

FOLLOWUP_CLAIM_PREFIX

_DRAIN_PASSES

_DRAIN_WAIT_S

_FLAT_GROUP

_FLAT_OUT

_FLAT_QUEUE

_KIND_CANCELLED

_KIND_ERROR

_KIND_NONE

_MARK

_SEP

API

nemo_gym.orchestration.executors.slurm_control._line(
kind: str,
payload: str
) -> str
nemo_gym.orchestration.executors.slurm_control.parse_terminate_output(
output: str
) -> tuple[list[str], list[str]]

The job ids render_terminate_script cancelled, and what went wrong, if anything.

nemo_gym.orchestration.executors.slurm_control.render_terminate_script(
) -> list[str]

The commands that cancel every queued or running job of job: the pending ones first, then the rest.

nemo_gym.orchestration.executors.slurm_control.FOLLOWUP_CLAIM_PREFIX = '.gym_followup_queued.'
nemo_gym.orchestration.executors.slurm_control._DRAIN_PASSES = 6
nemo_gym.orchestration.executors.slurm_control._DRAIN_WAIT_S = 5
nemo_gym.orchestration.executors.slurm_control._FLAT_GROUP = '$(echo $_group | tr " " ",")'
nemo_gym.orchestration.executors.slurm_control._FLAT_OUT = '$(echo "$out" | tr "\\n" " ")'
nemo_gym.orchestration.executors.slurm_control._FLAT_QUEUE = '$(echo "$squeue_out" | tr "\\n" " ")'
nemo_gym.orchestration.executors.slurm_control._KIND_CANCELLED = 'cancelled'
nemo_gym.orchestration.executors.slurm_control._KIND_ERROR = 'error'
nemo_gym.orchestration.executors.slurm_control._KIND_NONE = 'none'
nemo_gym.orchestration.executors.slurm_control._MARK = '__GYM_CONTROL'
nemo_gym.orchestration.executors.slurm_control._SEP = ':'