> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Slurm Slurm is a workload management system developed originally at the Lawrence Livermore National Laboratory. Slurm used to stand for Simple Linux Utility for Resource Management. However Slurm has evolved since then, and its advanced state nowadays means that the acronym is obsolete. Slurm has both a graphical interface and command line tools for submitting, monitoring, modifying and deleting jobs. It is normally used with *job scripts* to submit and execute jobs. Various settings can be put in the job script, such as number of processors, resource usage, and application specific variables. The steps for running a job through Slurm are to: * Create the script or executable that will be handled as a job * Create a job script that sets the resources for the script/executable * Submit the job script to the workload management system The details of Slurm usage depends upon the MPI implementation used. The description in this chapter will cover using Slurm’s Open MPI implementation, which is quite standard. Slurm documentation can be consulted ([https://slurm.schedmd.com/mpi_guide.html](https://slurm.schedmd.com/mpi_guide.html)) if the implementation the user is using is very different. ## 5.1 Loading Slurm Modules And Compiling The Executable In section [3.5.1.5](/user/mpi#3515-running-the-application) an MPI “Hello, world!” executable that can run in parallel is created and run in parallel outside a workload manager. The executable can be run in parallel using the Slurm workload manager. For this, the Slurm module should first be loaded by the user on top of the chosen MPI implementation, in this case Open MPI: **Example** ```bash [fred@basecm10 ~]$ module list Currently Loaded Modulefiles: 1) gcc/11.2.0 2) openmpi/gcc/64/4.1.1 [fred@basecm10 ~]$ module add slurm; module list Currently Loaded Modulefiles: 1) gcc/11.2.0 2) openmpi/gcc/64/4.1.1 3) slurm/slurm/23.11.11 ``` The “`hello world`” executable from section [3.5.1.5](/user/mpi#3515-running-the-application) can then be compiled and run for one task outside the workload manager, on the local host, as: ```bash [fred@basecm10 ~]$ mpicc hello.c -o hello [fred@basecm10 ~]$ mpirun -np 1 hello ``` Adding a full path to `mpirun`, and adding a machine file, would allow it to run on the machine file hosts, instead of just locally. ## 5.2 Running The Executable With `salloc` Running it as a job managed by Slurm can be done interactively with the Slurm allocation command, `salloc`, as follows ```bash [fred@basecm10 ~]$ salloc mpirun hello ``` Slurm is more typically run as a batch job (section [5.3](#53-running-the-executable-as-a-slurm-job-script)). However execution via `salloc` uses the same options, and it is more convenient as an introduction because of its interactive behavior. In a default BCM configuration, Slurm auto-detects the compute node cores available, and by default spreads the tasks across the cores as part of the allocation request. Specifying a machine file is therefore not required. To change how Slurm spreads the executable across compute nodes is typically determined by the options in the following table:
**Short** **Long**
**Option** **Option** **Description**
`-N` `--nodes=` Request this many nodes on the cluster.
Use all cores on each node by default
`-n` `--ntasks=` Request this many tasks on the cluster.
Defaults to 1 task per node.
`-c` `--cpus-per-task=` request this many CPUs per task.
(not implemented by Open MPI yet)
(none) `--ntasks-per-node=` request this number of tasks per node.
The full options list and syntax for `salloc` can be viewed with “`man salloc`”. The requirement of specified options to `salloc` must be met before the executable is allowed to run. So, for example, if `--nodes=4` and the cluster only has 3 nodes, then the executable does not run. ### 5.2.1 Node Allocation Examples The following session illustrates and explains some node allocation options and issues for Slurm using a cluster with just 1 compute node and 4 CPU cores: **Default settings:** The `hello` MPI executable with default settings of Slurm runs successfully over the first (and in this case, the only) node that it finds: ```bash [fred@basecm10 ~]$ salloc mpirun hello salloc: Granted job allocation 572 Hello world from process 0 out of 4, host name node001 Hello world from process 1 out of 4, host name node001 Hello world from process 2 out of 4, host name node001 Hello world from process 3 out of 4, host name node001 salloc: Relinquishing job allocation 572 ``` The preceding output also displays if `-N1` (indicating 1 node) is specified, or if `-n4` (indicating 4 tasks) is specified. The node and task allocation is almost certainly not going to be done by relying on defaults. Instead, node specifications are supplied to Slurm along with the executable. To understand Slurm node specifications, the following cases consider and explain where the node specification is valid and invalid. **Number of nodes requested:** The value assigned to the `-N`|`--nodes=` option is the number of nodes from the cluster that is requested for allocation for the executable. In the current cluster example it can only be 1. For a cluster with, for example, 1000 compute nodes, it could be a number up to 1000. A resource allocation request for 2 nodes with the `--nodes` option halts for the current cluster which only has 1 compute node: ```bash [fred@basecm10 ~]$ salloc -N2 mpirun hello salloc: Requested partition configuration not available now salloc: Pending job allocation 573 salloc: job 573 queued and waiting for resources ``` The default behavior is to patiently wait for resources to become available. This makes sense if a cluster can increase its available resources within a reasonable time period. The user can interupt an `salloc` in this state with a `ctrl-c`. **Number of tasks requested per cluster:** The value assigned to the `-n`|`--ntasks` option is the number of tasks that are requested for allocation from the cluster for the executable. In the current cluster example, which has a single compute node with 4 cores, it can be 1 to 4 tasks. The default CPU resources available on a cluster are the number of available processor cores on the compute nodes, which is 4 on this cluster with a single compute node of 4 cores. A resource allocation request for 5 tasks for this cluster halts because it exceeds the default resources available on the 4-core cluster: ```bash [fred@basecm10 ~]$ salloc -n5 mpirun hello salloc: Requested partition configuration not available now salloc: Pending job allocation 574 salloc: job 574 queued and waiting for resources ``` Adding and configuring just one more compute node to the current cluster would allow the resource allocation to succeed, since that would provide at least one more core to the cluster. **Number of tasks requested per node:** The value assigned to the `--ntasks-per-node` option is the number of tasks that are requested for allocation from each compute node on the cluster. In the current cluster example, it can be 1 to 4 tasks. A resource allocation request for 5 tasks per compute node with `--ntasks-per-node` results in a similar output on this cluster running a single compute node with 4-cores. It gives an output like: ```bash [fred@basecm10 ~]$ salloc --ntasks-per-node=5 mpirun hello salloc: Requested partition configuration not available now salloc: Pending job allocation 575 salloc: job 575 queued and waiting for resources ``` Adding and configuring another 4-core node to the current cluster would still not allow resource allocation to succeed, because the request is for at least 5 cores per compute node, rather than per cluster. **Restricting the number of tasks that can run per compute node:** A resource allocation request for 2 tasks per compute node with the `--ntasks-per-node` option, and simultaneously an allocation request for 1 task to run on the cluster using the `--ntasks` option, runs successfully, although it uselessly leaves 2 cores unused on the compute node: ```bash [fred@basecm10 ~]$ salloc --ntasks-per-node=2 --ntasks=1 mpirun hello salloc: Granted job allocation 576 Hello world from process 000 out of 002, processor name node001 Hello world from process 001 out of 002, processor name node001 salloc: Relinquishing job allocation 576 ``` The other way round, that is, a resource allocation request for 1 task per node with the `--ntasks-per-node` option, and simultaneously an allocation request for 2 tasks to run on the cluster using the `--ntasks` option, fails on this cluster running 1 compute node with 4 cores. This is because although 1 task can be allocated resources from the single node, resources for 2 tasks are being asked for on the cluster, which requires 2 nodes: ```bash [fred@basecm10 ~]$ salloc --ntasks-per-node=1 --ntasks=2 mpirun hello salloc: error: Job submit/allocate failed: Requested node configuration is not available salloc: Job allocation 577 has been revoked. ``` ## 5.3 Running The Executable As A Slurm Job Script Instead of using options appended to the `salloc` command line as in section [5.2](#52-running-the-executable-with-salloc), it is usually more convenient to send jobs to Slurm with the `sbatch` command acting on a job script. A job script is also sometimes called a batch file. In a job script, the user can add and adjust the Slurm options, which are the same as the `salloc` options of section [5.2](#52-running-the-executable-with-salloc). The various settings and variables that go with the application can also be adjusted. ### 5.3.1 Slurm Job Script Structure A job script submission for the Slurm batch job script format is illustrated by the following: ```bash [fred@basecm10 ~]$ cat slurmhello.sh #!/bin/sh #SBATCH -o my.stdout #SBATCH --time=30 #time limit to batch job #SBATCH -N 4 #SBATCH --ntasks=16 #SBATCH --ntasks-per-node=4 module add gcc/9.2.0 openmpi/gcc/64/1.10.7 slurm mpirun hello ``` The structure is: **shebang line:** shell definition line. **SBATCH lines:** optional job script *directives* (section [5.3.2](#532-slurm-job-script-options)). **shell commands:** optional shell commands, such as loading necessary modules. **application execution line:** execution of the MPI application using `sbatch`, the Slurm submission wrapper. In SBATCH lines, “`#SBATCH`” is used to submit options. The various meanings of lines starting with “`#`” are:
**Line Starts With** **Treated As**
`#` Comment in shell and Slurm
`#SBATCH` Comment in shell, option in Slurm
`# SBATCH` Comment in shell and Slurm
After the Slurm job script is run with the `sbatch` command (Section [5.3.4](#534-submitting-the-slurm-job-script-with-sbatch)), the output goes into file `my.stdout`, as specified by the “`-o`” command. If the output file is not specified, then the file takes a name of the form ”`slurm-` \<*`jobnumber`*>`.out`”, where \<*jobnumber*> is a number starting from 1. The command “`sbatch --usage`” lists possible options that can be used on the command line or in the job script. Command line values override script-provided values. ### 5.3.2 Slurm Job Script Options Options, sometimes called “directives”, can be set in the job script file using this line format for each option: ```bash #SBATCH {option} {parameter} ``` Directives are used to specify the resource allocation for a job so that Slurm can manage the job optimally. Available options and their descriptions can be seen with the output of `sbatch --help`. The more overviewable usage output from `sbatch --usage` may also be helpful. Some of the more useful ones are listed in the following table: | **Directive Description** | **Specified As** | | :----------------------------------------------------------------------------------------------------------- | :-------------------------------------------- | | Name the job \<*jobname*> | `#SBATCH -J` \<*`jobname`*> | | Request at least \<*minnodes*> nodes | `#SBATCH -N` \<*`minnodes`*> | | Request \<*minnodes*> to \<*maxnodes*> nodes | `#SBATCH -N` \<*`minnodes`*>`-<`*`maxnodes`*> | | Request at least \<*MB*> amount of temporary disk space | `#SBATCH --tmp` \<*`MB`*> | | Run job for a time of \<*walltime*> minutes | `#SBATCH -t` \<*`walltime`*> | | Run job at \<*time*> (format: HH:MM MM/DD/YY) | `#SBATCH --begin` \<*`time`*> | | Set the working directory to \<*directorypath*> | `#SBATCH -D` \<*`directorypath`*> | | Set error log name to \<*jobname.err*>\* | `#SBATCH -e` \<*`jobname.err`*> | | Set output log name to \<*jobname.log*>\* | `#SBATCH -o` \<*`jobname.log`*> | | Mail \<*user\@address*> on job state change | `#SBATCH --mail-user` \<*`user@address`*> | | Mail on all state changes | `#SBATCH --mail-type=ALL` | | Mail on job end | `#SBATCH --mail-type=END` | | Run job in partition | `#SBATCH -p` \<*`destination`*> | | Run job using GPU with ID \<*number*>, as described in section [8.5.2](/user/gpu#852-the-nvidia-smi-utility) | `#SBATCH --gres=gpu:` \<*`number`*> | \*By default, both standard output and standard error go to a file:\ `slurm-` \<*`%j`*>`.out`\ where \<*%j*> is the job number. ### 5.3.3 Slurm Environment Variables Available environment variables include: ```bash SLURM_CLUSTER_NAME - name of the Slurm cluster SLURM_CPUS_ON_NODE - CPUs on allocated node SLURM_JOB_ID - job ID of executing job SLURM_JOB_NODELIST - list of nodes allocated to job SLURM_JOB_NUM_NODES - total number of nodes in job's resource allocation SLURM_JOB_PARTITION - partition of job SLURM_NODEID - ID of the nodes allocated SLURM_NTASKS - total number of processes in current job (same as -n|--ntasks=) SLURM_PROCID - MPI rank (or relative process ID) of the current process SLURM_SUBMIT_DIR - directory from which job was launched SLURM_TASK_PID - process ID of task started SLURM_TASKS_PER_NODE - number of tasks to be run on each node (man page gives specification) ``` Typically, end users use `SLURM_PROCID` in a program so that an input of a parallel calculation depends on it. The calculation is thus spread across processors according to the assigned `SLURM_PROCID`, so that each processor handles the parallel part of the calculation with different values. More information on environment variables is also to be found in the man page for `sbatch`. ### 5.3.4 Submitting The Slurm Job Script With `sbatch` Submitting a Slurm job script created as in the previous section is done by executing the job script with `sbatch`: ```bash [fred@basecm10 ~]$ sbatch slurmhello.sh Submitted batch job 604 [fred@basecm10 ~]$ cat my.stdout Hello world from process 001 out of 016, processor name node001 ... ``` Queues in Slurm terminology are called “partitions”. Slurm has a default queue called `defq`. The administrator may have removed this or created others. If a particular queue is to be used, this is typically set in the job script using the `-p` or `--partition` option: ```bash #SBATCH --partition=bitcoinsq ``` It can also be specified as an option to the `sbatch` command during submission to Slurm. ### 5.3.5 Checking And Changing Queued Job Status With `squeue`, `scancel`, `scontrol` And `sview` #### 5.3.5.1 Job Queue Listing With `squeue` After a job has gone into a queue, the queue status can be checked using the `squeue` command. The job number can be specified with the `-j` option to avoid seeing other jobs. **Example** ```bash [fred@basecm10 ~]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 673 defq slurmhel fred PD 0:00 5 (PartitionNodeLimit) 683 defq slurmhel fred PD 0:00 4 (Resources) 684 defq slurmhel fred PD 0:00 4 (Resources) 685 defq slurmhel fred PD 0:00 4 (Resources) 682 defq slurmhel fred R 0:02 4 node[001-004] [fred@basecm10 ~]$ squeue -j 673 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 673 defq slurmhel fred PD 0:00 5 (PartitionNodeLimit) ``` The man page for `squeue` covers other options, and explains the meaning of the output. A backstory for this `squeue` listing is that in JOBID 673 the user has tried to batch submit the `slurmhello.sh` job, while requesting 5 nodes, using the entry `#SBATCH -N 5` in `slurmhello.sh`. However, since only 4 nodes were available for the cluster at the time, the submission is pending. After modifying the entry to `#SBATCH -N 4`, and submitting the job several times, the jobs 674 to 681 complete successfully. The job with the ID 682 is seen in the R (running) state, and the rest are in a PD (pending) state, with the reasons being either * `Resources` (that there are no resources yet available), or * `PartitionNodeLimit` (the number of nodes required by this job cannot yet be met because it is outside of its partition’s current limits). #### 5.3.5.2 Job Canceling With `scancel` Jobs can be canceled queitly by job ID with “`scancel `*``*”. The `-v` option gives some feedback. Verbosity increases with `-vv`, `-vvv`, and `-vvvv`. **Example** ```bash [fred@basecm10 ~]$ scancel -v 673 scancel: Terminating job 673 ``` Other cancel options, such as per state (with `-t|--state=`), per partition (with `-p|--partition=`), are also possible. The man page for `scancel` has details. #### 5.3.5.3 Queued Job Changes With `scontrol` The `scontrol` command allows users to see and change the job directives while the job is still queued. For example, a user may have specified a job, using the `--begin` directive, to start at 10am the next day by mistake. To change the job to start at 10pm tonight, something like the following session may take place: ```bash [fred@basecm10 ~]$ scontrol show jobid=254 | grep Time RunTime=00:00:04 TimeLimit=UNLIMITED TimeMin=N/A SubmitTime=2011-10-18T17:41:34 EligibleTime=2011-10-19T10:00:00 StartTime=2011-10-18T17:44:15 EndTime=Unknown SuspendTime=None SecsPreSuspend=0 ``` The parameter that should be changed is “`EligibleTime`”, which can be done as follows: ```bash [fred@basecm10 ~]$ scontrol update jobid=254 EligibleTime=2011-10-18T22:00:00 ``` #### 5.3.5.4 GUI Slurm Control `sview` An approximate GUI Slurm equivalent to `scontrol` is the `sview` tool. This allows the job to be viewed under its jobs tab, and the job to be edited with a right click menu item. It can also carry out many other functions, including canceling a job. Webbrowser-accessible job viewing is possible from the workload tab of the `User Portal` (section [12.2](/user/userportal#122-workload-page)).