Workload Managers Quick Reference | NVIDIA Base Command Manager

F.1 Slurm

Slurm is a GPL-licensed workload management system and developed largely at Lawrence Livermore National Laboratory. The name was originally an acronym for Simple Linux Utility for Resource Management, but the acronym is deprecated because it no longer does justice to the advanced capabilities of Slurm.

The Slurm service and outputs are normally handled using the Base View or cmsh front end tools for CMDaemon (section 7.4).

From the command line, direct Slurm commands that may sometimes come in useful include the following:

sacct: used to report job or job step accounting information about active or completed jobs.

Example

# sacct -j 43 -o jobid,AllocCPUs,NCPUS,NNodes,NTasks,ReqCPUs
       JobID  AllocCPUS      NCPUS   NNodes   NTasks  ReqCPUS
------------ ---------- ---------- -------- -------- --------
43                    1          1        1                 1

salloc: used to allocate resources for a job in real time. Typically this is used to allocate resources and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks.
sattach used to attach standard input, output, and error plus signal capabilities to a currently running job or job step. One can attach to and detach from jobs multiple times.
sbatch: used to submit a job script for later execution. The script typically contains one or more srun commands to launch parallel tasks.
sbcast: used to transfer a file from local disk to local disk on the nodes allocated to a job. This can be used to effectively use diskless compute nodes or provide improved performance relative to a shared filesystem.
scancel: used to cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step.
scontrol: the administrative tool used to view and/or modify Slurm state. Note that many scontrol commands can only be executed as user root.

Example
```
[fred@basecm10 ~]$ scontrol show nodes
NodeName=basecm10 Arch=x86_64 CoresPerSocket=1
   CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=0.05 Features=(null)
...
```
If a node, for example node001, is stuck in a CG state (“completing”), and rebooting it is not feasible, then the following may clear it in some cases:

Example
```
[fred@basecm10 ~]$ scontrol update nodename=node001 state=down reason=hung
[fred@basecm10 ~]$ scontrol update nodename=node001 state=resume
```

sinfo: reports the state of partitions and nodes managed by Slurm. It has a wide variety of filtering, sorting, and formatting options.

Example

basecm10:~ # sinfo -o "%9P %.5a %.10l %.6D %.6t %C %N"
PARTITION AVAIL  TIMELIMIT  NODES  STATE CPUS(A/I/O/T) NODELIST
defq*        up   infinite      1  alloc 1/0/0/1 basecm10

smap: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology.

squeue: reports the state of jobs or job steps. It has a wide variety of filtering, sorting, and formatting options. By default, it reports the running jobs in priority order and then the pending jobs in priority order.

Example

basecm10:~ # squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %C %R"
JOBID PARTITION  NAME  USER ST   TIME  NODES CPUS NODELIST(REASON)
   43      defq  bash  fred  R  16:22      1 1 basecm10
...

srun: used to submit a job for execution or initiate job steps in real time. srun has a wide variety of options to specify resource requirements, including: minimum and maximum node count, processor count, specific nodes to use or not use, and specific node characteristics (so much memory, disk space, certain required features, etc.). A job can contain multiple job steps executing sequentially or in parallel on independent or shared nodes within the job’s node allocation.

Auto Scaler dynamic node issues with srun: A concern about srun (https://bugs.schedmd.com/show_bug.cgi?id=1333) at the time of writing (January 2023) is the following: After an srun job has been queued, and a new node is added, the slurm.conf file is not read again. This means that the new node resource is not seen by jobs using srun. Thus, with srun jobs, nodes launched dynamically by Auto Scaler remain unused, and can fail. Workarounds are to use salloc or sbatch instead of srun.
smap: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology.
strigger: used to set, get or view event triggers. Event triggers include things such as nodes going down or jobs approaching their time limit.
sview: a graphical user interface to get and update state information for jobs, partitions, and nodes managed by Slurm.

There are man pages for these commands. Full documentation on Slurm is available online at: http://slurm.schedmd.com/documentation.html.

F.2 Altair Grid Engine

Altair Grid Engine (GE) is a workload management system that was originally derived from Sun Grid Engine.

GE services should be handled using CMDaemon, as explained in section 7.4.

Managing queues and jobs can be done via the cmsh or Base View front ends, but can also be carried out via the GE tools.

The sge_qmaster daemon on the head node can be started, stopped, or viewed using service sgemaster start|stop|status, or alternatively handled at a lower, more fine-grained, level using the -k (kill), -s (show) and -a (activate) options of qconf. Further details on these options can be found in the qconf help text, or man qconf(1). For example, help on the -k options can be found by running:

qconf -help | grep '\-k'

or running:

man qconf(1) and scrolling down to the interesting part by searching for shutdown).
The sge_execd execution daemon running on each compute node accepts, manages, and returns the results of the jobs on the compute nodes. The daemon can be managed via service sgeexecd start|stop|status, or alternatively (de)registered from qmaster via qconf options.
To see why a queue is in an error state, qstat -explain E shows the reason. Further details on these options can be found in the qconf help text, or man qstat(1). Queues in an error state can be cleared with a qmod -cq <queue name>.

GE can be configured and managed generally with the command line utility qconf, which is what most administrators become familiar with. GE also provides a GUI alternative, qmon.

GE commands are listed below. The details of these are in the man page of the command and the GE documentation.

qalter: modify existing batch jobs
qacct: show usage information from accounting data
qconf: configure GE
qdel: delete batch jobs
qhold: place hold on batch jobs
qhost: display compute node queues, states, jobs
qlogin: start login-based interactive session with a node
qmake: distributed, parallel make utility
qmod: suspend/enable queues and jobs
qmon: configure GE with an X11 GUI interface
qping: check sge_qmaster and sge_execd status
qquota: list resource quotas
qresub: create new jobs by copying existing jobs
qrdel: cancel advance reservations
qrls: release batch jobs from a held state
qrsh: start rsh-based interactive session with node
qrstat: show status of advance reservations
qrsub: submit advanced reservation
qselect: select queues based on argument values
qsh: start sh interactive session with a node
qstat: show status of batch jobs and queues
qsub: submit new jobs (related: qalter, qresub)

F.3 PBS Professional

The following commands can be used in PBS Professional to view queues, jobs, and server status:

qstat                query queue status
qstat -a             show only queued or running jobs for a destination, or all states for a job ID
qstat -r             show only running or suspended jobs for a destination, or all states for a job ID
qstat -q             show queue status for destinations
qstat -rn            only running or suspended jobs, with list of allocated nodes (`exec_host` string)
qstat -i             information on queued, held, waiting jobs is given for specified destination. Inform-
                     ation about the job is given if a job ID is specified, regardless of the job status
qstat -B             display server status for the specified servers
qstat -u <username>   show jobs for a user for the specified destination. Status
                     information for the job is displayed for a specified job ID.

Other useful commands are:

tracejob <job id>                 show what happened today to <job id>
tracejob -n <number> <job id>     search last <number> days for <job id>
qmgr              administrator interface to batch system (man qmgr.8B for more details)
qterm             terminates PBS server (but BCM starts pbs_server again)
pbsnodes <node>    query status of compute node
pbsnodes -a       query status of all compute nodes

The commands of PBS Professional are documented in the man pages, and also in the extensive documentation available via https://community.altair.com/community?id=altair_product_documentation.