Skip to navigation

nemo_gym.orchestration.executors.slurm.scheduler

View as MarkdownOpen in Claude

The Slurm commands the executor’s parts share: sbatch and its reply, partition limits, live checks.

Module Contents

Functions

NameDescription
_chain_live_checkShell lines that fail unless the scheduler confirms that no job of chain waits or runs.
_parse_sbatch_resultsBenchmark name to (job_id, error), exactly one of which is set.
_partition_time_limitThe partition a job of compute queues on and its MaxTime, read with sinfo over conn.
_sbatch_commandOne sbatch that reports its own benchmark, exit status and output.
_slurm_durationseconds as a Slurm time limit, HH:MM:SS.
_slurm_time_limit_secondsA partition time limit as sinfo prints it (%l) in seconds; None for UNLIMITED or infinite.
_submission_env_exportsForward explicitly selected values over SSH stdin, never into artifacts.
_submission_exportsWhat run() exports ahead of sbatch: the forwarded values, then the telemetry token when it is on.

Data

_JOB_ID_RE

_MARKER

_RESUBMIT_LOCK

_VALID_BENCHMARK_NAME

API

nemo_gym.orchestration.executors.slurm.scheduler._chain_live_check(
chain: list[str],
live_error: str
) -> list[str]

Shell lines that fail unless the scheduler confirms that no job of chain waits or runs.

Slurm drops a finished job from the controller after MinJobAge, and squeue -j then exits 1 with “Invalid job id specified” once it knows none of the ids. That is no answer about the jobs, so sacct, which keeps finished jobs, decides: a job whose latest incarnation (highest Restarts; sacct -D lists every requeue) is in one of the LIVE_JOB_STATES is live, and one it does not know is not. Any other squeue failure, or a failing sacct, leaves the scheduler unknown.

nemo_gym.orchestration.executors.slurm.scheduler._parse_sbatch_results(
output: str
) -> dict[str, tuple[str | None, str | None]]

Benchmark name to (job_id, error), exactly one of which is set.

nemo_gym.orchestration.executors.slurm.scheduler._partition_time_limit(
) -> dict[str, typing.Any]

The partition a job of compute queues on and its MaxTime, read with sinfo over conn.

The partitions are the node pools’ ones, else extra_args.partition (a list a,b allowed), else the cluster’s default one. Every one must be listed, and the lowest limit applies: partition names it, max_time is sinfo’s text and max_time_s its seconds (None when unlimited); error says why they are unknown.

nemo_gym.orchestration.executors.slurm.scheduler._sbatch_command(
benchmark: str,
script: pathlib.Path
) -> str

One sbatch that reports its own benchmark, exit status and output.

rc is captured immediately: $? after the tr pipeline would be tr’s status, which is zero however badly sbatch failed. tr flattens sbatch’s multi-line error messages so the whole result stays on one marker line.

nemo_gym.orchestration.executors.slurm.scheduler._slurm_duration(
seconds: int
) -> str

seconds as a Slurm time limit, HH:MM:SS.

nemo_gym.orchestration.executors.slurm.scheduler._slurm_time_limit_seconds(
text: str
) -> int | None

A partition time limit as sinfo prints it (%l) in seconds; None for UNLIMITED or infinite.

Takes Slurm’s duration spellings (MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS) and raises ValueError for anything else, such as n/a.

nemo_gym.orchestration.executors.slurm.scheduler._submission_env_exports(
) -> list[str]

Forward explicitly selected values over SSH stdin, never into artifacts.

nemo_gym.orchestration.executors.slurm.scheduler._submission_exports(
) -> list[str]

What run() exports ahead of sbatch: the forwarded values, then the telemetry token when it is on.

nemo_gym.orchestration.executors.slurm.scheduler._JOB_ID_RE = re.compile('^\\d+(;\\S+)?$')
nemo_gym.orchestration.executors.slurm.scheduler._MARKER = '__GYM_JOB:'
nemo_gym.orchestration.executors.slurm.scheduler._RESUBMIT_LOCK = RESUBMIT_LOCK_NAME
nemo_gym.orchestration.executors.slurm.scheduler._VALID_BENCHMARK_NAME = re.compile('^[A-Za-z0-9._-]+$')