nemo_gym.orchestration.executors.slurm.scheduler
nemo_gym.orchestration.executors.slurm.scheduler
The Slurm commands the executor’s parts share: sbatch and its reply, partition limits, live checks.
Module Contents
Functions
Data
API
Shell lines that fail unless the scheduler confirms that no job of chain waits or runs.
Slurm drops a finished job from the controller after MinJobAge, and squeue -j then exits 1 with “Invalid job
id specified” once it knows none of the ids. That is no answer about the jobs, so sacct, which keeps finished
jobs, decides: a job whose latest incarnation (highest Restarts; sacct -D lists every requeue) is in one of the
LIVE_JOB_STATES is live, and one it does not know is not. Any other squeue failure, or a failing sacct, leaves
the scheduler unknown.
Benchmark name to (job_id, error), exactly one of which is set.
The partition a job of compute queues on and its MaxTime, read with sinfo over conn.
The partitions are the node pools’ ones, else extra_args.partition (a list a,b allowed), else the
cluster’s default one. Every one must be listed, and the lowest limit applies: partition names it,
max_time is sinfo’s text and max_time_s its seconds (None when unlimited); error says why they are
unknown.
One sbatch that reports its own benchmark, exit status and output.
rc is captured immediately: $? after the tr pipeline would be tr’s
status, which is zero however badly sbatch failed. tr flattens sbatch’s
multi-line error messages so the whole result stays on one marker line.
seconds as a Slurm time limit, HH:MM:SS.
A partition time limit as sinfo prints it (%l) in seconds; None for UNLIMITED or infinite.
Takes Slurm’s duration spellings (MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS) and
raises ValueError for anything else, such as n/a.
Forward explicitly selected values over SSH stdin, never into artifacts.
What run() exports ahead of sbatch: the forwarded values, then the telemetry token when it is on.