Skip to navigation

nemo_gym.orchestration.executors.slurm_status

View as MarkdownOpen in Claude

How the Slurm executor looks at the jobs it queued.

One script per host asks Slurm about every job directory of every record at once: a single squeue for the whole batch, a single sacct over every job id any of them has had, and the state files each job directory holds. The cost grows with the number of hosts, not with the number of submissions. Jobs are found by where they run, not by the recorded id, so a job started by hand, a resumed one, or one resubmitted from another machine is still seen.

Module Contents

Functions

NameDescription
_exit_statusThe exit code of sacct’s code:signal pair.
_observe-
parse_read_files_outputThe contents render_read_files_script printed, None for a file it did not find.
parse_status_outputWhat render_status_script reported, as one ObservedStatus per job in refs.
render_read_files_scriptThe commands that print each readable file of paths, encoded so a file cannot break the output.
render_status_scriptThe commands that report the state of every job in refs, on one connection.
state_from_sacctSlurm’s own state name as a JobState; a name this does not know is UNKNOWN.

Data

STATE_FILES

_ACTIVE_STATES

_FAILED_STATES

_KIND_ACCT

_KIND_ACCT_ERROR

_KIND_ACTIVE

_KIND_CHAIN

_KIND_FILE

_KIND_SQUEUE_ERROR

_MARK

_PENDING_STATES

_SEP

_STATE_FILE_BYTES

API

nemo_gym.orchestration.executors.slurm_status._exit_status(
text: str
) -> int | None

The exit code of sacct’s code:signal pair.

nemo_gym.orchestration.executors.slurm_status._observe(
chain: list[str],
active: list[tuple[str, str]],
files: dict[str, str],
accounting: dict[str, tuple[str, int | None]],
squeue_error: str | None,
acct_error: str | None
nemo_gym.orchestration.executors.slurm_status.parse_read_files_output(
output: str,
paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]

The contents render_read_files_script printed, None for a file it did not find.

nemo_gym.orchestration.executors.slurm_status.parse_status_output(
output: str,

What render_status_script reported, as one ObservedStatus per job in refs.

nemo_gym.orchestration.executors.slurm_status.render_read_files_script(
paths: collections.abc.Iterable[pathlib.Path]
) -> list[str]

The commands that print each readable file of paths, encoded so a file cannot break the output.

nemo_gym.orchestration.executors.slurm_status.render_status_script(
refs: collections.abc.Iterable[nemo_gym.orchestration.jobs.JobRef]
) -> list[str]

The commands that report the state of every job in refs, on one connection.

nemo_gym.orchestration.executors.slurm_status.state_from_sacct(
native: str

Slurm’s own state name as a JobState; a name this does not know is UNKNOWN.

nemo_gym.orchestration.executors.slurm_status.STATE_FILES = (CHAIN_INCOMPLETE_FILE, EVAL_EXIT_FILE)
nemo_gym.orchestration.executors.slurm_status._ACTIVE_STATES = {'RUNNING', 'COMPLETING', 'CONFIGURING', 'SUSPENDED', 'RESIZING', 'SIGNALING', '...
nemo_gym.orchestration.executors.slurm_status._FAILED_STATES = {'FAILED', 'TIMEOUT', 'OUT_OF_MEMORY', 'NODE_FAIL', 'PREEMPTED', 'BOOT_FAIL', 'D...
nemo_gym.orchestration.executors.slurm_status._KIND_ACCT = 'acct'
nemo_gym.orchestration.executors.slurm_status._KIND_ACCT_ERROR = 'acct-error'
nemo_gym.orchestration.executors.slurm_status._KIND_ACTIVE = 'active'
nemo_gym.orchestration.executors.slurm_status._KIND_CHAIN = 'chain'
nemo_gym.orchestration.executors.slurm_status._KIND_FILE = 'file'
nemo_gym.orchestration.executors.slurm_status._KIND_SQUEUE_ERROR = 'squeue-error'
nemo_gym.orchestration.executors.slurm_status._MARK = '__GYM_STATE'
nemo_gym.orchestration.executors.slurm_status._PENDING_STATES = {'PENDING', 'REQUEUED', 'REQUEUE_HOLD', 'REQUEUE_FED', 'RESV_DEL_HOLD'}
nemo_gym.orchestration.executors.slurm_status._SEP = ':'
nemo_gym.orchestration.executors.slurm_status._STATE_FILE_BYTES = 4096