> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Workload Managers Quick Reference ## F.1 Slurm Slurm is a GPL-licensed workload management system and developed largely at Lawrence Livermore National Laboratory. The name was originally an acronym for Simple Linux Utility for Resource Management, but the acronym is deprecated because it no longer does justice to the advanced capabilities of Slurm. The Slurm service and outputs are normally handled using the Base View or `cmsh` front end tools for CMDaemon (section [7.4](/admin/workloadmanagement#74-enabling-disabling-and-monitoring-workload-managers)). From the command line, direct Slurm commands that may sometimes come in useful include the following: * `sacct`: used to report job or job step accounting information about active or completed jobs. **Example** ```bash # sacct -j 43 -o jobid,AllocCPUs,NCPUS,NNodes,NTasks,ReqCPUs JobID AllocCPUS NCPUS NNodes NTasks ReqCPUS ------------ ---------- ---------- -------- -------- -------- 43 1 1 1 1 ``` * `salloc`: used to allocate resources for a job in real time. Typically this is used to allocate resources and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks. * `sattach` used to attach standard input, output, and error plus signal capabilities to a currently running job or job step. One can attach to and detach from jobs multiple times. * `sbatch`: used to submit a job script for later execution. The script typically contains one or more srun commands to launch parallel tasks. * `sbcast`: used to transfer a file from local disk to local disk on the nodes allocated to a job. This can be used to effectively use diskless compute nodes or provide improved performance relative to a shared filesystem. * `scancel`: used to cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step. * `scontrol`: the administrative tool used to view and/or modify Slurm state. Note that many scontrol commands can only be executed as user root. **Example** ```bash [fred@basecm11 ~]$ scontrol show nodes NodeName=basecm11 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=0.05 Features=(null) ... ``` If a node, for example `node001`, is stuck in a `CG` state (“completing”), and rebooting it is not feasible, then the following may clear it in some cases: **Example** ```bash [fred@basecm11 ~]$ scontrol update nodename=node001 state=down reason=hung [fred@basecm11 ~]$ scontrol update nodename=node001 state=resume ``` * `sinfo`: reports the state of partitions and nodes managed by Slurm. It has a wide variety of filtering, sorting, and formatting options. **Example** ```bash basecm11:~ # sinfo -o "%9P %.5a %.10l %.6D %.6t %C %N" PARTITION AVAIL TIMELIMIT NODES STATE CPUS(A/I/O/T) NODELIST defq* up infinite 1 alloc 1/0/0/1 basecm11 ``` * `smap`: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology. * `squeue`: reports the state of jobs or job steps. It has a wide variety of filtering, sorting, and formatting options. By default, it reports the running jobs in priority order and then the pending jobs in priority order. **Example** ```bash basecm11:~ # squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %C %R" JOBID PARTITION NAME USER ST TIME NODES CPUS NODELIST(REASON) 43 defq bash fred R 16:22 1 1 basecm11 ... ``` * `srun`: used to submit a job for execution or initiate job steps in real time. srun has a wide variety of options to specify resource requirements, including: minimum and maximum node count, processor count, specific nodes to use or not use, and specific node characteristics (so much memory, disk space, certain required features, etc.). A job can contain multiple job steps executing sequentially or in parallel on independent or shared nodes within the job’s node allocation. **Auto Scaler dynamic node issues with `srun`:** [A concern about `srun` (`https://bugs.schedmd.com/show_bug.cgi?id=1333`)](https://bugs.schedmd.com/show_bug.cgi?id=1333) at the time of writing (January 2023) is the following: After an `srun` job has been queued, and a new node is added, the `slurm.conf` file is not read again. This means that the new node resource is not seen by jobs using `srun`. Thus, with `srun` jobs, nodes launched dynamically by Auto Scaler remain unused, and can fail. Workarounds are to use `salloc` or `sbatch` instead of `srun`. * `smap`: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology. * `strigger`: used to set, get or view event triggers. Event triggers include things such as nodes going down or jobs approaching their time limit. * `sview`: a graphical user interface to get and update state information for jobs, partitions, and nodes managed by Slurm. There are man pages for these commands. Full documentation on Slurm is available online at: [http://slurm.schedmd.com/documentation.html](http://slurm.schedmd.com/documentation.html).