Slurm
Slurm is a workload management system developed originally at the Lawrence Livermore National Laboratory. Slurm used to stand for Simple Linux Utility for Resource Management. However Slurm has evolved since then, and its advanced state nowadays means that the acronym is obsolete.
Slurm has both a graphical interface and command line tools for submitting, monitoring, modifying and deleting jobs. It is normally used with job scripts to submit and execute jobs. Various settings can be put in the job script, such as number of processors, resource usage, and application specific variables.
The steps for running a job through Slurm are to:
-
Create the script or executable that will be handled as a job
-
Create a job script that sets the resources for the script/executable
-
Submit the job script to the workload management system
The details of Slurm usage depends upon the MPI implementation used. The description in this chapter will cover using Slurm’s Open MPI implementation, which is quite standard. Slurm documentation can be consulted (https://slurm.schedmd.com/mpi_guide.html) if the implementation the user is using is very different.
5.1 Loading Slurm Modules And Compiling The Executable
In section 3.5.1.5 an MPI “Hello, world!” executable that can run in parallel is created and run in parallel outside a workload manager.
The executable can be run in parallel using the Slurm workload manager. For this, the Slurm module should first be loaded by the user on top of the chosen MPI implementation, in this case Open MPI:
Example
The “hello world” executable from section 3.5.1.5 can then be compiled and run for one task outside the workload manager, on the local host, as:
Adding a full path to mpirun, and adding a machine file, would allow it to run on the machine file hosts, instead of just locally.
5.2 Running The Executable With salloc
Running it as a job managed by Slurm can be done interactively with the Slurm allocation command, salloc, as follows
Slurm is more typically run as a batch job (section 5.3). However execution via salloc uses the same options, and it is more convenient as an introduction because of its interactive behavior.
In a default BCM configuration, Slurm auto-detects the compute node cores available, and by default spreads the tasks across the cores as part of the allocation request. Specifying a machine file is therefore not required.
To change how Slurm spreads the executable across compute nodes is typically determined by the options in the following table:
The full options list and syntax for salloc can be viewed with “man salloc”.
The requirement of specified options to salloc must be met before the executable is allowed to run. So, for example, if --nodes=4 and the cluster only has 3 nodes, then the executable does not run.
5.2.1 Node Allocation Examples
The following session illustrates and explains some node allocation options and issues for Slurm using a cluster with just 1 compute node and 4 CPU cores:
Default settings: The hello MPI executable with default settings of Slurm runs successfully over the first (and in this case, the only) node that it finds:
The preceding output also displays if -N1 (indicating 1 node) is specified, or if -n4 (indicating 4 tasks) is specified.
The node and task allocation is almost certainly not going to be done by relying on defaults. Instead, node specifications are supplied to Slurm along with the executable.
To understand Slurm node specifications, the following cases consider and explain where the node specification is valid and invalid.
Number of nodes requested: The value assigned to the -N|--nodes= option is the number of nodes from the cluster that is requested for allocation for the executable. In the current cluster example it can only be 1. For a cluster with, for example, 1000 compute nodes, it could be a number up to 1000.
A resource allocation request for 2 nodes with the --nodes option halts for the current cluster which only has 1 compute node:
The default behavior is to patiently wait for resources to become available. This makes sense if a cluster can increase its available resources within a reasonable time period. The user can interupt an salloc in this state with a ctrl-c.
Number of tasks requested per cluster: The value assigned to the -n|--ntasks option is the number of tasks that are requested for allocation from the cluster for the executable. In the current cluster example, which has a single compute node with 4 cores, it can be 1 to 4 tasks. The default CPU resources available on a cluster are the number of available processor cores on the compute nodes, which is 4 on this cluster with a single compute node of 4 cores.
A resource allocation request for 5 tasks for this cluster halts because it exceeds the default resources available on the 4-core cluster:
Adding and configuring just one more compute node to the current cluster would allow the resource allocation to succeed, since that would provide at least one more core to the cluster.
Number of tasks requested per node: The value assigned to the --ntasks-per-node option is the number of tasks that are requested for allocation from each compute node on the cluster. In the current cluster example, it can be 1 to 4 tasks. A resource allocation request for 5 tasks per compute node with --ntasks-per-node results in a similar output on this cluster running a single compute node with 4-cores. It gives an output like:
Adding and configuring another 4-core node to the current cluster would still not allow resource allocation to succeed, because the request is for at least 5 cores per compute node, rather than per cluster.
Restricting the number of tasks that can run per compute node: A resource allocation request for 2 tasks per compute node with the --ntasks-per-node option, and simultaneously an allocation request for 1 task to run on the cluster using the --ntasks option, runs successfully, although it uselessly leaves 2 cores unused on the compute node:
The other way round, that is, a resource allocation request for 1 task per node with the --ntasks-per-node option, and simultaneously an allocation request for 2 tasks to run on the cluster using the --ntasks option, fails on this cluster running 1 compute node with 4 cores. This is because although 1 task can be allocated resources from the single node, resources for 2 tasks are being asked for on the cluster, which requires 2 nodes:
5.3 Running The Executable As A Slurm Job Script
Instead of using options appended to the salloc command line as in section 5.2, it is usually more convenient to send jobs to Slurm with the sbatch command acting on a job script.
A job script is also sometimes called a batch file. In a job script, the user can add and adjust the Slurm options, which are the same as the salloc options of section 5.2. The various settings and variables that go with the application can also be adjusted.
5.3.1 Slurm Job Script Structure
A job script submission for the Slurm batch job script format is illustrated by the following:
The structure is:
shebang line: shell definition line.
SBATCH lines: optional job script directives (section 5.3.2).
shell commands: optional shell commands, such as loading necessary modules.
application execution line: execution of the MPI application using sbatch, the Slurm submission wrapper.
In SBATCH lines, “#SBATCH” is used to submit options. The various meanings of lines starting with “#” are:
After the Slurm job script is run with the sbatch command (Section 5.3.4), the output goes into file my.stdout, as specified by the “-o” command.
If the output file is not specified, then the file takes a name of the form ”slurm- <jobnumber>.out”, where <jobnumber> is a number starting from 1.
The command “sbatch --usage” lists possible options that can be used on the command line or in the job script. Command line values override script-provided values.
5.3.2 Slurm Job Script Options
Options, sometimes called “directives”, can be set in the job script file using this line format for each option:
Directives are used to specify the resource allocation for a job so that Slurm can manage the job optimally. Available options and their descriptions can be seen with the output of sbatch --help. The more overviewable usage output from sbatch --usage may also be helpful.
Some of the more useful ones are listed in the following table:
*By default, both standard output and standard error go to a file:
slurm- <%j>.out
where <%j> is the job number.
5.3.3 Slurm Environment Variables
Available environment variables include:
Typically, end users use SLURM_PROCID in a program so that an input of a parallel calculation depends on it. The calculation is thus spread across processors according to the assigned SLURM_PROCID, so that each processor handles the parallel part of the calculation with different values.
More information on environment variables is also to be found in the man page for sbatch.
5.3.4 Submitting The Slurm Job Script With sbatch
Submitting a Slurm job script created as in the previous section is done by executing the job script with sbatch:
Queues in Slurm terminology are called “partitions”. Slurm has a default queue called defq. The administrator may have removed this or created others.
If a particular queue is to be used, this is typically set in the job script using the -p or --partition option:
It can also be specified as an option to the sbatch command during submission to Slurm.
5.3.5 Checking And Changing Queued Job Status With squeue, scancel, scontrol And sview
5.3.5.1 Job Queue Listing With squeue
After a job has gone into a queue, the queue status can be checked using the squeue command. The job number can be specified with the -j option to avoid seeing other jobs.
Example
The man page for squeue covers other options, and explains the meaning of the output.
A backstory for this squeue listing is that in JOBID 673 the user has tried to batch submit the slurmhello.sh job, while requesting 5 nodes, using the entry #SBATCH -N 5 in slurmhello.sh. However, since only 4 nodes were available for the cluster at the time, the submission is pending.
After modifying the entry to #SBATCH -N 4, and submitting the job several times, the jobs 674 to 681 complete successfully. The job with the ID 682 is seen in the R (running) state, and the rest are in a PD (pending) state, with the reasons being either
-
Resources(that there are no resources yet available), or -
PartitionNodeLimit(the number of nodes required by this job cannot yet be met because it is outside of its partition’s current limits).
5.3.5.2 Job Canceling With scancel
Jobs can be canceled quietly by job ID with “scancel <job number>”. The -v option gives some feedback. Verbosity increases with -vv, -vvv, and -vvvv.
Example
Other cancel options, such as per state (with -t|--state=), per partition (with -p|--partition=), are also possible. The man page for scancel has details.
5.3.5.3 Queued Job Changes With scontrol
The scontrol command allows users to see and change the job directives while the job is still queued. For example, a user may have specified a job, using the --begin directive, to start at 10am the next day by mistake. To change the job to start at 10pm tonight, something like the following session may take place:
The parameter that should be changed is “EligibleTime”, which can be done as follows:
5.3.5.4 GUI Slurm Control sview
An approximate GUI Slurm equivalent to scontrol is the sview tool. This allows the job to be viewed under its jobs tab, and the job to be edited with a right click menu item. It can also carry out many other functions, including canceling a job.
Webbrowser-accessible job viewing is possible from the workload tab of the User Portal (section 9.2).