Workload Management

For clusters that have many users and a significant load, a workload manager (WLM) system allows a more efficient use of resources to be enforced for all users than if there were no such system in place. This is because without resource management, there is a tendency for each individual user to over-exploit common resources.

When a WLM is used, the end user can submit a job to it. This can be done interactively, but it is typically done as a non-interactive batch job.

The WLM assigns resources to the job, and checks the current availability as well as checking its estimates of the future availability of the cluster resources that the job is asking for. The WLM then schedules and executes the job based on the assignment criteria that the administrator has set for the WLM system. After the job has finished executing, the job output is delivered back to the user.

Among the hardware resources that can be used for a job are GPUs. Installing CUDA software to enable the use of GPUs is described in section 9 of the Installation Manual. Configuring GPU settings for BCM is described in section 3.16.2 of the Administrator Manual. Configuring GPU settings for an individual WLM is described in the section on getting that particular WLM up and running.

The details of job submission from a user’s perspective are covered in the User Manual.

Sections 7.1–7.5 cover the installation procedure to get a WLM up and running.

Sections 7.6 –7.7 describe how Base View and cmsh are used to view and handle jobs, queues and node drainage.

Section 7.8 shows examples of WLM assignments handled by BCM.

Section 7.9 describes the power saving features of WLMs.

Section 7.10 describes cgroups, a resources limiter, mostly in the context of WLMs.

Section 7.11 describes WLM customizations for settings other than the common settings covered by BCM.

7.1 Workload Managers Choices

Installation of new PBS and GE workload managers is no longer offered by BCM. Legacy setups that are already installed continue to work, and documentation for PBS and GE is therefore still available in the manuals.

Some WLM packages are installed by default, others require registration from the distributor before installation.

During cluster installation, a WLM can be chosen (figure 3.9 of the Installation Manual) for setting up. The choices are:

• Slurm: Available as version 23.02 or 23.11. Slurm is a free (GPL) job scheduler, with commercial support.

• LSF v10.1: IBM Spectrum LSF (Load Sharing Facility) version 10.1, is a further development of what used to be IBM Platform LSF.

• None: For clusters that need no HPC job-scheduling.

The WLMs in the preceding list can also be chosen and set up later using the cm-wlm-setup tool (section 7.3).

After installation, if there are no major changes in the WLM for updated versions of the workload managers, then

• WLMs that are packaged with BCM (Slurm) can have their packages updated using standard package update commands (yum update and similar). The installation and configuration of the WLM from the updated packages is carried out as described later on in this chapter.

• WLMs that are installed by picking up software from the vendor (LSF) can be updated by following vendor guidelines.

7.2 Forcing Jobs To Run In A Workload Management System

Another preliminary step is to consider forcing users to run jobs only within the WLM system. Having jobs run via a WLM is normally a best practice.

For convenience, BCM defaults to allowing users to log in via ssh to a node, using the authorized keys files stored in each users directory in /home (section 2.3.2). This allows users to run their processes without restriction, that is, outside the WLM system. For clusters with a significant load this policy results in a sub-optimal use of resources, since such unplanned-for jobs disturb any already-running jobs.

Disallowing user logins to nodes, so that users have to run their jobs through the WLM system, means that jobs are then distributed to the nodes only according to the planning of the WLM. If planning is based on sensible assignment criteria, then resources use is optimized—which is the entire aim of a WLM in the first place.

7.2.1 Disallowing User Logins To Regular Nodes Via cmsh

The usernodelogin setting of cmsh restricts direct user logins from outside the WLM, and is thus one way of preventing the user from using node resources in an unaccountable manner. The usernodelogin setting is applicable to node categories only, rather than to individual nodes.

In cmsh the attribute of usernodelogin is set from within category mode:

Example

[root@basecm10 ~]# cmsh
[basecm10]% category use default
[basecm10->category[default]]% set usernodelogin onlywhenjob
[basecm10->category*[default*]]% commit

The attributes for usernodelogin are:

• always (the default): This allows all users to ssh directly into a node at any time.

• never: This allows no user other than root to directly ssh into the node.

• onlywhenjob: This allows the user to ssh directly into the node when a job is running on it. It typically also prevents other users from doing a direct ssh into the same node during the job run, since typically the WLM is set up so that only one job runs per node. However, an ssh session that is already running is not automatically terminated after the job is done.

  • Some cluster administrators may wish to allow some special user accounts to override the onlywhenjob setting, for example for diagnostic purposes. Before giving the details of how to override the setting, some background explanation is probably useful:

    The onlywhenjob setting works with the PAM system, and adds the following line to /etc/pam.d/sshd on the regular nodes:

        account          required       pam_bright.so

    Nodes with the onlywhenjob restriction can be configured to allow a particular set of users to access them, despite the restriction, by allowing them with the PAM system, as follows: Within the software image <node image> used by the node, that is under /cm/images/ <node image>, the administrator can add the set of user accounts to the file etc/security/pam_bright.d/pam_allow.conf. This file is installed in the software image with a chroot installation (section 11.4) of the cm-libpam package.

    Groups of users can be allowed using the file etc/security/pam_bright.d/pam_allow_group.conf.

    Other adjustments to PAM configuration, such as the number of attempted logins and the associated wait time per login, can be carried by editing the /etc/security/pam_bright.d/cm-check-alloc.conf file.

    The image can then be updated to allow the users, by running the imageupdate command in cmsh (section 5.6.2), or by clicking the Update node option in Base View (section 5.6.3).

7.2.2 Disallowing User Logins To Regular Nodes Via Base View

In Base View, user node login access is set via a category setting, for example for the default category via the navigation path in figure 7.1: Grouping > Categories[default] > Edit > Settings > User node login

Figure 7.1: Disallowing user logins to nodes via Base View

7.2.3 Disallowing Other User Processes Outside Of Workload Manager User Processes

Besides disabling user logins, administrators may choose to disable interactive jobs in the WLM as an additional measure to prevent users from starting jobs on other nodes.

Administrators may also choose to set up scripts that run after job execution. Such scripts can terminate user processes outside the WLM, as part of a policy, or for general administrative hygiene. These are Epilog scripts and are part of the WLM.

The WLM documentation has more on configuring these options.

7.2.4 High Availability By Workload Managers

NVIDIA Base Command Manager uses the existing built-in high availability (HA) functionalities of workload managers as much as possible. A double server HA configuration with the existing built-in HA functionality can be carried out using the cm-wlm-setup utility (section 7.3), or using the Base View HA wizard (section 7.4.1). The built-in functionality makes use of the primary WLM server and secondary WLM server, which are placed on separate nodes.

The HA configuration aspect (section 17.1.3) in this case means that a WLM server role is assigned to both the WLM primary and the WLM secondary nodes. These primary WLM and secondary WLM servers can then:

• both be on head nodes,

• both be on regular (compute) nodes

It is not possible to configure a mixed setup—that is with one head node and one regular node—for the WLM servers on an HA setup.

If the head nodes are configured with the WLM server roles, then in the cluster entity configuration the primary server value is empty. This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node.

If two regular nodes are configured with the WLM server roles instead, then in the cluster entity configuration the primary server parameter is set to one of the compute nodes.

The cluster primary head node and the WLM primary server should not be confused. In particular, the active cluster head node and active workload manager server are not necessarily the same for the case of the Slurm, PBS Professional, or LSF workload managers. For these, if the passive cluster head node becomes an active cluster head node without a crash, then this does not trigger a passive WLM server on the newly active head node to also become an active WLM server. Thus, the WLM server can be active on a passive head node, and vice versa, because the WLM primary server is independent of the cluster primary head node.

7.2.4.1 Slurm High Availability And scontrol takeover

Slurm can speed up takeover by running scontrol takeover from the Bash shell. This stops the primary slurmctld and triggers the Slurm failover algorithm.

However, by default BCM automatically restarts slurmctld, so the automatic restart of the slurmctld service must be disabled manually. There are also some other related housekeeping tasks that may need to be taken care of.

Instead of trying to run an scontrol takeover from Bash and then having to work around those kind of issues, BCM can be used to automate the takeover. That is to say, to run scontrol takeover cleanly from Bash with HA, the following configuration steps can be carried out:

1. Within CMDaemon, for Slurm, the takeover parameter is set to true. Disabling the automatic restart of the slurmctld service is then unnecessary:

   [root@basecm10 ~]# cmsh
   [basecm10]% wlm use slurm
   [basecm10->wlm[slurm]]% set --extra takeover true
   [basecm10->wlm*[slurm*]]% commit

   The takeover setting can be cleared with:

   [basecm10->wlm[slurm]]% clear -e takeover
   [basecm10->wlm*[slurm*]]% commit

2. If the installation of Slurm with cm-wlm-setup was carried out on what was a non-HA cluster at that time, then the prefailoverscript parameter must be set within the failover submode of partition mode:

   [basecm10->partition[base]->failover]% set prefailoverscript /cm/local/apps/cmd/scripts/slurm.takeover.sh
   [basecm10->partition*[base*]->failover*]% commit

   This ensures that the active head node is taken over by Slurm during a change of active head node

Table 7.2.4.1 shows what slurmctld does when the takeover parameter is set. The detailed behavior per head node depends on whether the head nodes are active or primary:

* any value of yes or no

7.3 Installation Of Workload Managers

Normally the administrator selects a WLM to be used during BCM installation (figure 3.9 of the Installation Manual). A WLM may however also be added and configured after BCM has been installed, using cm-wlm-setup, or the Base View WLM wizard.

With most other objects, BCM front ends—cmsh and Base View—can be used to create a new object from scratch, or can clone a new object from another existing object. However, the front ends cannot do this for a WLM object. An attempt to create or clone a new WLM object via cmsh or Base View is prohibited by the front ends, because there are many pitfalls possible in configuration.

A new WLM object, and WLM instance, can therefore only be installed via cm-wlm-setup, the Base View WLM wizard, or by selecting a WLM during the initial BCM installation.

7.3.1 Running cm-wlm-setup In CLI Mode

The recommended way to run the cm-wlm-setup utility is without options or arguments, in which case a TUI dialog starts up. A TUI session run with cm-wlm-setup is covered in section 7.3.2.

However, the cm-wlm-setup utility can alternatively be used in a non-GUI, command-line, mode, with options and arguments. The utility has the following usage:

[root@basecm10 ~]# cm-wlm-setup -h
usage: Workload manager setup cm-wlm-setup [-c <config_file>]
                                           [--setup | --disable]
                                           [--wlm <name>]
                                           [--server-nodes SERVER_NODES]
                                           [--server-primary SERVER_PRIMARY]
                                           [--server-overlay-name SERVER_OVERLAY_NAME]
                                           [--server-overlay-priority SERVER_OVERLAY_PRIORITY]
                                           [--client-categories CLIENT_CATEGORIES]
                                           [--client-nodes CLIENT_NODES]
                                           [--client-overlay-name CLIENT_OVERLAY_NAME]
                                           [--client-overlay-priority CLIENT_OVERLAY_PRIORITY]
                                           [--client-slots <slots>]
                                           [--submit-categories SUBMIT_CATEGORIES]
                                           [--submit-nodes SUBMIT_NODES]
                                           [--submit-overlay-name SUBMIT_OVERLAY_NAME]
                                           [--submit-overlay-priority SUBMIT_OVERLAY_PRIORITY]
                                           [--wlm-cluster-name WLM_CLUSTER_NAME]
                                           [--reboot] [--reset-cgroups]
                                           [--yes-i-really-mean-it]
                                           [--archives-location <path>]
                                           [--license <license>] [--purge]
                                           [--accounting-overlay-name ACCOUNTING_OVERLAY_NAME]
                                           [--accounting-overlay-priority ACCOUNTING_OVERLAY_PRIORITY]
                                           [--with-pyxis]
                                           [--add-pyxis]
                                           [--reinstall-pyxis]
                                           [--remove-pyxis]
                                           [--pyxis-data-directory <path>]
                                           [--nvidia-gpus NVIDIA_GPUS] [-v]
                                           [--no-distro-checks] [--json]
                                           [--output-remote-execution-runner]
                                           [--on-error-action {debug,remotedebug,undo,abort}]
                                           [--skip-packages]
                                           [--min-reboot-timeout <reboot_timeout_seconds>]
                                           [--allow-running-from-secondary]
                                           [--dev] [-h]

The help output from running cm-wlm-setup -h continues on beyond the preceding text output, and presents more options.

These options can be grouped as follows:

7.3.1.1 Optional Arguments

• --setup: Helps set up a server, enable roles, and create the default queues/partitions

• --disable: Disable WLM services

• -h, --help: Displays the help screen

7.3.1.2 Common Arguments

• -c <YAML configuration file>: Loads a runtime configuration for plugins, from a YAML configuration file.

7.3.1.3 Options For Installing Or Managing A WLM

• --wlm <WLM name>: Specifies which WLM is to be set up. Choices for <WLM name> are:

  • openpbs

  • pbspro

  • slurm

  • uge

  • lsf

• --wlm-cluster-name <WLM cluster name>: Specifies the name for the new WLM cluster that is to be set up.

• --reboot: Reboot after install

7.3.1.4 Server Role Settings

• --server-nodes <server nodes>: Sets the server roles of the WLM to the value set for <server nodes>, which is a comma-separated list of nodes. Default value: HEAD, which is a reserved name for the head node.

• --server-primary <primary server>: Sets the hostname used for the primary server to <primary server>. Default name: HEAD.

• --server-overlay-name <server overlay name>: Sets the server role configuration overlay name to <server overlay name>. Default name: <WLM name>-server, where <WLM name> is the name specified in the --wlm option.

• --server-overlay-priority <server overlay priority>: Sets the server role configuration overlay priority to <server overlay priority>. Default value: 500.

7.3.1.5 Client Role Settings

• --client-categories <client categories>: Sets the client roles of the WLM to the value set for <client categories nodes>, which is a comma-separated list of node categories. Default value: default.

• --client-nodes <client nodes>: Sets the client roles of the WLM to the value set for <client nodes>, which is a comma-separated list of nodes. No value set by default.

• --client-overlay-name <client overlay name>: Sets the client role configuration overlay name to <client overlay name>. Default name: <WLM name>-client, where <WLM name> is the name specified in the --wlm option.

• --client-overlay-priority <client overlay priority>: Sets the client role configuration overlay priority to <client overlay priority>. Default value: 500.

• --client-slots <slots>: Sets the number of slots on the client to <slots>.

7.3.1.6 Submit Role Settings

• --submit-categories <submit categories>: Sets the submit roles of the WLM to the value set for <submit categories nodes>, which is a comma-separated list of node categories that are submit nodes. Default value: default.

• --submit-nodes <submit nodes>: Sets the submit roles of the WLM to the value set for <submit nodes>, which is a comma-separated list of submit nodes. No value set by default.

• --submit-overlay-name <submit overlay name>: Sets the submit role configuration overlay name to <submit overlay name>. Default name: <WLM name>-submit, where <WLM name> is the name specified in the --wlm option.

• --submit-overlay-priority <submit overlay priority>: Sets the submit role configuration overlay priority to <submit overlay priority>. Default value: 500.

7.3.1.7 Disable Options

• --reset-cgroups: Disable joining cgroup controllers with systemd setting JoinControllers

• --yes-i-really-mean-it: Required for additional safety

7.3.1.8 Workload Manager Specific Options

• --archives-location <path>: Set the directory path for the archive files, only for LSF and GE. This parameter is mandatory for LSF installation.

• --license <path>: Set the path to the PBSPro, LSF, or GE license.

• --purge: Remove the directories on disable, for GE and LSF.

7.3.1.9 Slurm Accounting Role Settings

• --accounting-overlay-name <accounting overlay name>: Sets the accounting role configuration overlay

• --accounting-overlay-priority <accounting overlay priority>: Sets the accounting role configuration overlay priority to <accounting overlay priority>. Default value: 500.

7.3.1.10 Slurm Pyxis Settings

• --add-pyxis: Add Pyxis to an existing cluster

• --reinstall-pyxis: Reinstall or upgrade Pyxis on all the clusters using it

• --remove-pyxis: Remove Pyxis from an existing cluster

• --with-pyxis: Enable Pyxis

• --pyxis-data-directory <path>: Sets directory where images will be stored

7.3.1.11 Slurm GPU Settings

• --nvidia-gpus <NVIDIA GPUs specification>: Sets the NVIDIA GPUs that will be used in the configuration, specifying type and number of GPUs. For example: --nvidia-gpus=A100:4

• --gpu-client-nodes <NVIDIA GPU client nodes>: Sets a comma-separated list of nodes assigned to the GPU overlay.

• --gpu-client-categories <NVIDIA GPU client categories>: Sets a comma-separated list of node categories assigned to the GPU overlay.

7.3.1.12 Advanced Options

• -v, --verbose: This displays a more verbose output. It can be helpful in troubleshooting.

• --no-distro-checks: Disables distribution checks based on ds.json.

• --json: Use json formatting for logs printed to STDOUT.

• --output-remote-execution-runner: Format output for CMDaemon.

• --on-error-action {debug,remotedebug,undo,abort}: Upon encountering a critical error, instead of asking the user for choice, the setup will do the selected action.

• --skip-packages: Skip the stages which install packages. Requires packages to be already installed.

• --min-reboot-timeout <timeout>: How long to wait for nodes to finish reboot, in seconds. Minimum value: 300. Default value: 300.

• --allow-running-from-secondary: Allow the wizard to be run from the secondary when it is the active head node.

• --dev: Enables additional command line arguments for developers.

For example, if GE packages are downloaded to /root/ugedl, then a GE setup could be carried out with (much output elipsized):

Example

[basecm10 ~]# cm-wlm-setup -v --setup --wlm uge --archives-location /root/ugedl --client-categories default
I 19-12-05 15:29:46 |
D 19-12-05 15:29:46 | Please wait...
D 19-12-05 15:29:46 | ##########################################################################
D 19-12-05 15:29:46 | ######################## START ###########################################
D 19-12-05 15:29:46 | ##########################################################################
D 19-12-05 15:29:46 |
D 19-12-05 15:29:46 | ### Logging Current Runtime environment
...
...
I 19-12-05 15:30:20 | Took:     00:33 min.
I 19-12-05 15:30:20 | Progress: 100/100
I 19-12-05 15:30:20 | ########### Finished execution for 'Workload manager setup', status: completed
I 19-12-05 15:30:20 |
I 19-12-05 15:30:20 | Workload manager setup finished!
I 19-12-05 15:30:20 |
D 19-12-05 15:30:20 |
D 19-12-05 15:30:20 | ########################  END ###########################################
D 19-12-05 15:30:20 |
I 19-12-05 15:30:20 |

A check to see if the GE instance (WLM cluster) has been created could be:

Example

[basecm10 ~]# cmsh -c "wlm; list"
Type   Name (key)            Server nodes Submit nodes             Client nodes
------ --------------------- ------------ ------------------------ ----------------
UGE    uge                   basecm10     basecm10,node001,node002 node001,node002

The GE instance can then be removed with:

Example

[basecm10 ~]#
 cm-wlm-setup -v --disable --wlm uge --purge --yes-i-really-mean-it --wlm-cluster-name=uge

A log file of the setup can be seen at /var/log/cm-wlm-setup.log.

7.3.2 Running cm-wlm-setup As A TUI

Running cm-wlm-setup with no options and with no arguments brings up a TUI screen (figure 7.2).

Figure 7.2: cm-wlm-setup TUI initial screen

7.3.2.1 Express Installation

The Setup (Express) menu option allows the administrator to select the workload manager in the next screen (figure 7.3), and to install it with a minimal number of configuration steps. If it has already been installed, but disabled via cm-wlm-setup, then it can also be re-enabled, instead of installed from scratch.

Figure 7.3: cm-wlm-setup TUI WLM selection screen

7.3.2.2 Step-by-step Installation

If the Setup (Step By Step) menu option is chosen instead of the express option, then this also allows the administrator to select the workload manager in figure 7.3. But after selection, there are a number of extra configuration steps that can be carried out which are not available in the express configuration. Guidance is given for these extra steps, and sensible default values are already filled in for many options.

One part of the step-by-step session involves assigning the WLM client role to the compute nodes of the cluster. Only the non-GPU compute nodes should be assigned the (standard, non-GPU) WLM client role. The GPU compute nodes are assigned a GPU WLM client role in the section of the TUI wizard that deals with GPU configuration.

The WLM client role is assigned to the entire category—the default category— of non-head nodes by default (figure 7.4):

Figure 7.4: Slurm with cm-wlm-setup: WLM Client role category configuration screen

The WLM client role can be assigned to selected nodes only by setting the default category checkbox to blank, and then selecting the non-GPU nodes in the following screen. For example, as illustrated in figure 7.4, where the standard, non-GPU nodes are node002 and node003:

Figure 7.5: Slurm with cm-wlm-setup: WLM client role node configuration screen

7.3.2.3 GPU Configuration Screens

The GPU configuration screens are extra steps available during a Setup (Step By Step) session. The GPU configuration steps are discussed summarily in the quickstart section for GPUs, here.

The GPU configuration steps are covered in this section in more detail:

After configuring the WLM server, WLM submission and WLM client roles for the nodes of the cluster, a screen that asks if GPU resources should be configured is displayed (figure 1.1):

Figure 1.1: Slurm With cm-wlm-setup: GPU Configuration Entry Screen

Choosing yes means that some extra GPU configuration screens are presented. These are screens that allow:

• the configuration overlay (section 2.1.6) name to be set for the GPU WLM clients. By default the name is set to slurm-client-gpu.

• a GPU WLM client role to be assigned to a category, if, for example, all the GPU nodes have been given their own category.

• the GPU WLM client role to be assigned to individual nodes instead of to a category.

• a configuration overlay priority to be set for a GPU WLM client role. By default, this has a value of 450.

• automatic GPU detection, with the following options:

  • BCM autodetects GPUs

  • Slurm autodetects NVIDIA GPU details

  • Slurm autodetects AMD GPU details

  • Slurm autodetects Intel GPU details

  • Slurm autodetects AWS Trainium/Inferentia devices

  • Manual GPU configuration in cm-wlm-setup

  • Skip GPU configuration

7.3.2.4 Slurm Accounting Database Configuration

In the step-by-step configuration for the Slurm WLM, after the server and client parameters have been set, the Slurm accounting database configuration can be set.

This requires setting:

• one or two accounting nodes

• a primary accounting node

All nodes must be reachable from slurmctld.

The accounting nodes can be set with the Select accounting nodes screen (figure 7.7):

Figure 7.7: cm-wlm-setup selection of accounting nodes

Head nodes and compute nodes cannot be mixed for the accounting nodes. If two accounting nodes are set, then the slurmdbd high availability configuration is automatically set up.

The primary accounting node—that is, the node that is AccountingStorageHost in slurm.conf—can then be set in the Select the primary accounting server node screen (figure 7.8):

Figure 7.8: cm-wlm-setup selection of primary accounting node

The other accounting node is automatically set to be AccountingStorageBackupHost.

The type of node on which the slurmdbd database runs can then be set (figure 7.9):

Figure 7.9: cm-wlm-setup selection of the type of node on which the Slurm accounting database runs

• If Use accounting node is selected, then the database is stored on the primary accounting node

• If Select cluster node is selected, then the next screen allows the selection of the BCM node on which the database is to be installed (figure 7.10)

Figure 7.10: cm-wlm-setup Selection of host where the Slurm accounting database used by slurmdbd is stored

• If Select external node is selected, then a non-BCM hostname is asked for, and the cluster administrator is responsible for the installation and configuration of the DBMS

Modifying the settings after installation is discussed here.

7.3.2.5 Disabling An Installation

The Disable option in figure 7.2 allows the administrator to disable an existing instance.

7.3.2.6 Summary Screen

The screen that appears after the configuration steps are completed, is the Summary screen (figure 7.11). This screen allows the configuration to be viewed, saved, or saved and deployed.

Figure 7.11: cm-wlm-setup TUI summary screen

If deployment is carried out, then several screens of output are displayed. After the deployment is completed, the log file can be viewed at /var/log/cm-wlm-setup.log.

7.3.3 Installation And Configuration Of Enroot And Pyxis With Slurm To Run Containerized Jobs

7.3.3.1 What Is Enroot?

As the README file for Enroot says, Enroot is an open source tool to turn container images into unprivileged sandboxes. Enroot can be thought of as an enhanced unprivileged chroot. It uses user and mount namespaces, as well as other modern kernel features, in order to create such sandboxes. It uses the same underlying technologies as containers, but removes much of the isolation that they inherently provide, while preserving filesystem separation. Further details on Enroot can be found at https://github.com/NVIDIA/enroot.

Enroot can be used with different workload managers, but for now only Slurm has been tightly integrated.

7.3.3.2 What Is Pyxis?

Pyxis (https://github.com/NVIDIA/pyxis) is a SPANK plugin for Slurm. SPANK (Slurm Plug-in architecture for Node and job (K)control, man spank.8) is a generic interface for job launch code control in Slurm. The Pyxis plugin requires the Enroot utility, and allows the user’s jobs to be executed seamlessly over Enroot in unprivileged containers. The plugin enables the Slurm submission utilities to provide container-related command line options.

7.3.3.3 Enroot And Pyxis Packages

BCM provides two Enroot package flavors: standard and hardened. The standard binaries are compiled as follows:

• Open file descriptors are inherited

• Spectre variant 2 (IBPB/STIBP) mitigations are disabled

• Spectre variant 4 (SSBD) mitigations are disabled

As a rule of thumb: the hardened flavor is slightly more secure but suffers a larger overhead.

• The standard packages are already installed by default with BCM, and are:

  • enroot: provides the main utility and helper files.

  • enroot+caps: a nearly empty package which runs a post-installation script to grant extra capabilities to unprivileged users. This allows them to import and convert container images.

  • pyxis-sources: installs a tarball with the Pyxis plugin source files. The tarball is used by cm-wlm-setup to compile the Pyxis plugin on a cluster. The version of the package (section 11.1), found using yum info pyxis-sources or apt-cache show pyxis-sources, implies the Pyxis source version that is provided. For example: pyxis-sources-0.20.0-[...].rpm provides the Pyxis source for version 0.20.0.

• The hardened packages that can be installed from the BCM repositories to replace the standard enroot and enroot+caps are:

  • enroot-hardened: provides hardened main utility and helper files.

  • enroot-hardened+caps: Provides extra capabilities to unprivileged users so that they can import and convert containers themselves.

The package pyxis-sources installs the tarball at /cm/local/apps/slurm/var/pyxis/pyxis-sources.tar.gz. If the administrator needs a version of Pyxis other than the one provided by the package, then the archive can be replaced with a source tarball of the same name.

7.3.3.4 How Are Enroot And Pyxis Set Up In BCM?

Pyxis and Enroot can be set up or removed by the administrator by choosing the appropriate options when Slurm is set up. In order to choose the appropriate options, cm-wlm-setup can be run in step-by-step mode (here). Alternatively, Pyxis-related command line options can be specified for the cm-wlm-setup arguments --add-pyxis and --reinstall-pyxis.

The step-by-step mode eventually presents a screen where the Pyxis setup can be enabled via a plugin:

Figure 7.12: cm-wlm-setup Pyxis setup screen

The Pyxis screen is available for RHEL8-based systems and Ubuntu 20 and beyond. Older systems are not supported.

If the plugin is enabled in the Pyxis setup screen, then cm-wlm-setup installs the enroot and enroot+caps packages from https://github.com/NVIDIA/enroot/releases into the software images where the Slurm client role is to be assigned. The cm-wlm-setup uitility also installs them directly on the head node if the head node was selected to run jobs. Pyxis sources are downloaded from GitHub, compiled with the installed Slurm, and installed in the appropriate Slurm directory.

If the administrator enables the Pyxis plugin, then a new screen with Enroot settings is shown:

Figure 7.13: cm-wlm-setup Enroot settings screen

The Enroot settings have reasonable default values. The administrator can change these:

• Share raw images among users and nodes: If enabled, then the raw container image is shared among users and nodes, and the administrator must ensure that the cache directory is shared among all the compute nodes. Enabling the option disables the creation of a UID subdirectory by the Enroot prolog script.

  The following points about implementing Enroot cache sharing should be considered by the administrator:

  • Enabling the checkbox for the option changes the default value of the Enroot cache directory to /cm/shared/enroot/cache. The default value can also be changed in the TUI, in the Cache directory field, to any shared directory. If the specified directory does not exist, then cm-wlm-setup creates it with Unix directory permissions set to chmod 00777, which allows all users to share container images in the cache with each other on all the nodes.

  • Restrictions can be placed on the ability of regular users to share images and to create files in the cache, by setting more restrictive permissions for the directory manually, after the cm-wlm-setup run.

    For example, the permissions can be set to 00770, while the ownership group of the directory is changed to a group with all the Pyxis users in it. In this case, only the users within the Pyxis users group can create the image layers in the cache.

  • An alternative to cache sharing of images among users and nodes is to save Squashfs images in a shared directory, outside the cache. Then the users can specify a full path to a shared Squashfs image on the cluster in the srun/sbatch command line. In this case there is no need to share the cache among the nodes, as the Squashfs images are not copied over to cache before execution.

  Only the administrator should have access to updating the image.

• Share unpacked container images among nodes: Enables sharing of unpacked container images (image filesystems) among nodes. A user job can modify the filesystem, so it is not recommended to share this directory among users.

  Enabling the option disables removal of the Enroot data directory, if the container name is specified. This allows a user to keep the container filesystem between jobs run by the user. It is up to the user or the administrator to clean up the data directory if the option is enabled.

  If the data directory is shared among nodes, then it is up to a user to ensure that the container filesystem is unpacked at least once before real jobs start. Otherwise, a race condition is possible when the images are extracted simultaneously on several nodes.

• Cache directory: Path to the cache directory where raw container image layers are stored.

  If the directory is not shared, then the Slurm prolog script creates a subdirectory with a name that is the job user ID, while epilog cleans up that subdirectory. If the directory is shared, then neither prolog nor epilog touches the directory.

• Data directory: Directory where the container filesystems (unpackaged images) are stored. If the directory is shared, then the epilog script checks if the container name is specified for the job, and skips removal of the container subdirectory.

• Runtime directory: A working directory with temporary files created by Enroot.

When Pyxis is set up, cm-wlm-setup also prepares the following configuration files for the compute nodes:

• /etc/enroot/enroot.conf: This is a symlink to /cm/shared/apps/slurm/var/etc/enroot.conf. The configuration file provides reasonable default settings that allows Enroot to be used by many users. Important settings in the file are:

  • ENROOT_RUNTIME_PATH: working directory for enroot, created per user. Default value: /run/enroot/runtime/$(id -u)

  • ENROOT_CACHE_PATH: directory where container layers are stored. Default value: /run/enroot/cache/$(id -u)

  • ENROOT_DATA_PATH: directory where the filesystems of running containers are stored. Default value: /run/enroot/data/$(id -u)

  • ENROOT_SQUASH_OPTIONS: options passed to mksquashfs to produce container images. Default value: -noI -noD -noF -noX -no-duplicates

  • ENROOT_MOUNT_HOME: mount the current user’s home directory by default. Default value: yes.

  The administrator can change the symlink, or replace the file with a customized enroot.conf, if other values are preferred.

• /etc/sysctl.d/80-enroot.conf: symlink to /cm/shared/apps/slurm/var/etc/enroot-sysctl.conf. The file tunes sysctl parameters for Enroot.

• /cm/local/apps/slurm/var/prologs/50-prolog-enroot.sh: symlink to /cm/shared/apps/slurm/var/cm/prolog-enroot.sh. This is the slurmd prolog that creates appropriate directories, with appropriate user permissions, that are used by Enroot.

• /cm/local/apps/slurm/var/epilogs/50-epilog-enroot.sh: symlink to /cm/shared/apps/slurm/var/cm/epilog-enroot.sh. Cleans the user directories used by Enroot.

When cm-wlm-setup finishes its Pyxis configuration run, then there is no need for the nodes to be rebooted. The plugin works immediately.

The new Slurm submission command option names start with either --container or --no-container. The full list of options can be displayed with the --help option. For example:

Example

[user@basecm10 ~]$ srun --help | grep container
      --container             Path to OCI container bundle
      --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH
                              [pyxis] the image to use for the container
      --container-mounts=SRC:DST[:FLAGS][,SRC:DST...]
                              [pyxis] bind mount[s] inside the container. Mount
      --container-workdir=PATH
                              [pyxis] working directory inside the container
      --container-name=NAME   [pyxis] name to use for saving and loading the
                              container on the host. Unnamed containers are
                              containers are not. If a container with this name
                              already exists, the existing container is used and
      --container-save=PATH   [pyxis] Save the container state to a squashfs
      --container-mount-home  [pyxis] bind mount the user's home directory.
      --no-container-mount-home
      --container-remap-root  [pyxis] ask to be remapped to root inside the
                              container. Does not grant elevated system
      --no-container-remap-root
                              [pyxis] do not remap to root inside the container
      --container-entrypoint  [pyxis] execute the entrypoint from the container
      --no-container-entrypoint
                              container image
      --container-writable    [pyxis] make the container filesystem writable
      --container-readonly    [pyxis] make the container filesystem read-only

Simple installation validation: The simplest way to validate the Pyxis/Enroot setup after Slurm setup is to try out an srun command:

Example

[user@basecm10 ~]$ module load slurm
[user@basecm10 ~]$ srun --container-image=ubuntu grep PRETTY /etc/os-release
pyxis: importing docker image: ubuntu
PRETTY_NAME="Ubuntu 20.04.3 LTS"

A more thorough installation validation: In order to perform a more thorough test of Pyxis/Enroot, an NCCL-based test can be used. NCCL is the NVIDIA Collective Communications Library (https://docs.nvidia.com/deeplearning/nccl), which is a library of multi-GPU collective communication primitives. The test can be found at https://github.com/NVIDIA/nccl-tests. A prebuilt container image, with the NCCL test already installed, can be started as shown in the example that follows.

It should be noted that running a multi-tenant cluster with all the export flags enabled as in the example may compromise security. It is therefore not recommended as a standard configuration.

Example

[user@basecm10 ~]$ module load slurm
[user@basecm10 ~]$ srun --export="NCCL_DEBUG=INFO,NCCL_IB_DISABLE=1,PMIX_MCA_gds=hash" -N 2 \
--ntasks-per-node=1 --gpus-per-task=1 --mpi=pmix --container-image=deepops/mpi-nccl-test \
/nccl_tests/build/all_reduce_perf -b 1M -e 4G -f 2 -g 1
pyxis: imported docker image: deepops/mpi-nccl-test
pyxis: imported docker image: deepops/mpi-nccl-test
# nThread 1 nGpus 1 minBytes 1048576 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 validation: 1
#
# Using devices
#   Rank  0 Pid 175945 on    node001 device  0 [0x00] Tesla V100-SXM3-32GB
#   Rank  1 Pid 180379 on    node002 device  0 [0x00] Tesla V100-SXM3-32GB
node001:175945:175945 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.5<0>
node001:175945:175945 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
node001:175945:175945 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
node001:175945:175945 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.5<0>
node001:175945:175945 [0] NCCL INFO Using network Socket
NCCL version 2.11.4+cuda11.6
node002:180379:180379 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.6<0>
node002:180379:180379 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
node002:180379:180379 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
node002:180379:180379 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.6<0>
node002:180379:180379 [0] NCCL INFO Using network Socket
node002:180379:182573 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
node001:175945:177561 [0] NCCL INFO Channel 00/02 :    0   1
node001:175945:177561 [0] NCCL INFO Channel 01/02 :    0   1
node001:175945:177561 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
node001:175945:177561 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [receive] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [receive] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [receive] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [receive] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [send] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [send] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [send] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [send] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Connected all rings
node002:180379:182573 [0] NCCL INFO Connected all rings
node001:175945:177561 [0] NCCL INFO Connected all trees
node001:175945:177561 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
node001:175945:177561 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
node002:180379:182573 [0] NCCL INFO Connected all trees
node002:180379:182573 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
node002:180379:182573 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
node001:175945:177561 [0] NCCL INFO comm 0x1551f0001000 rank 0 nranks 2 cudaDev 0 busId 60 - Init COMPLETE
#
#                                                       out-of-place                       in-place
#       size         count      type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                       (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
node001:175945:175945 [0] NCCL INFO Launch mode Parallel
node002:180379:182573 [0] NCCL INFO comm 0x1551f8001000 rank 1 nranks 2 cudaDev 0 busId 60 - Init COMPLETE
     1048576        262144     float     sum   1491.5    0.70    0.70  0e+00   1421.3    0.74    0.74  0e+00
     2097152        524288     float     sum   3077.5    0.68    0.68  0e+00   2568.8    0.82    0.82  0e+00
     4194304       1048576     float     sum   4617.3    0.91    0.91  0e+00   4622.5    0.91    0.91  0e+00
     8388608       2097152     float     sum   9483.1    0.88    0.88  0e+00   8911.9    0.94    0.94  0e+00
    16777216       4194304     float     sum    17516    0.96    0.96  0e+00    18613    0.90    0.90  0e+00
    33554432       8388608     float     sum    34799    0.96    0.96  0e+00    41837    0.80    0.80  0e+00
    67108864      16777216     float     sum    99790    0.67    0.67  0e+00    83126    0.81    0.81  0e+00
   134217728      33554432     float     sum   204614    0.66    0.66  0e+00   199530    0.67    0.67  0e+00
   268435456      67108864     float     sum   319701    0.84    0.84  0e+00   341630    0.79    0.79  0e+00
   536870912     134217728     float     sum   608809    0.88    0.88  0e+00   683016    0.79    0.79  0e+00
  1073741824     268435456     float     sum  1337187    0.80    0.80  0e+00  1247369    0.86    0.86  0e+00
  2147483648     536870912     float     sum  2638741    0.81    0.81  0e+00  2743454    0.78    0.78  0e+00
  4294967296    1073741824     float     sum  5996381    0.72    0.72  0e+00  5430549    0.79    0.79  0e+00
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.810575
#

The test demonstrates the usage of PMIX, MPI, and GPU in Enroot containers on multiple nodes. In the preceding example, 2 nodes with 1 GPU on each is requested by the job. For better results the cluster administrator can tune the test parameters.

It should be noted that the image is quite large, and requires enough free space under /var. Also, the transfer timeout (ENROOT_TRANSFER_TIMEOUT) in enroot.conf must be large enough to download such a large container image to the compute nodes. A value of at least 600 seconds is recommended.

If there are issues when executing MPI jobs with PMIX, then to help debug the issues the Pyxis documentation (https://github.com/NVIDIA/pyxis/wiki/Setup) suggests setting up the following environment variables:

• PMIX_MCA_ptl=^usock

• PMIX_MCA_psec=none

• PMIX_SYSTEM_TMPDIR=/var/empty

• PMIX_MCA_gds=hash (as configured here)

This configuration change is typically carried out in the software image of the regular nodes. For example, for a node category called default:

[root@basecm10 ~]# category=default
[root@basecm10 ~]# cat <<EOF >> /cm/images/${category}/etc/default/slurmd
PMIX_MCA_ptl=^usock
PMIX_MCA_psec=none
PMIX_SYSTEM_TMPDIR=/var/empty
PMIX_MCA_gds=hash
EOF
[root@basecm10 ~]# systemctl restart slurmd
[root@basecm10 ~]# cmsh -c "device; imageupdate -c ${category} -w"

7.3.4 Prolog And Epilog Scripts

7.3.4.1 What Prolog And Epilog Scripts Do

The workload manager runs prolog scripts before job execution, and epilog scripts after job execution. The purpose of these scripts can include:

• checking if a node is ready before submitting a job execution that may use it

• preparing a node in some way to handle the job execution

• cleaning up resources after job execution has ended.

The administrator can run custom prolog or epilog scripts for the queues from CMDaemon for GE, or LSF, by setting such scripts in the Base View or cmsh front ends.

Example

[basecm10->wlm[uge]->jobqueue]% use all.q
[basecm10->wlm[uge]->jobqueue[all.q]]% show | grep . | grep -i epilog
Epilog                           NONE

For PBS and Slurm, there are global prolog and epilog scripts, but editing them is not recommended. Indeed, in order to discourage editing them, the scripts cannot be set via the cluster manager front ends. Instead the scripts must be placed by the administrator in the software image, and the relevant nodes updated from the image.

7.3.4.2 Detailed Workings Of Prolog And Epilog Scripts

Even though it is not recommended, some administrators may nonetheless wish to link and edit the scripts directly for their own needs, outside of the Base View or cmsh front ends. A more detailed explanation of how the prolog scripts work therefore follows:

When a workload manager is configured via cm-wlm-setup or via the Base View setup wizard, then the workload manager is configured to run the generic prolog located in /cm/local/apps/cmd/scripts/prolog, and the generic epilog located in /cm/local/apps/cmd/scripts/epilog. The generic prolog and epilog scripts call a sequence of scripts for a particular workload manager in special directories. The directories have paths in the format:

1. /cm/local/apps/ <workload manager>/var/prologs/

2. /cm/local/apps/ <workload manager>/var/epilogs/

In these directories, scripts are stored with names that have suffixes and prefixes associated with them that make them run in special ways, as follows:

• suffixes used in the prolog/epilog directory:

  • -prejob script runs prior to all jobs

  • -cmjob: script runs prior to job run in a cloud

• prefixes used in the prolog/epilog directory:

  • 00- to

  • 99-

  Number prefixes determine the order of script execution, with scripts with a lower number running earlier.

The script names can therefore look like:

Example

• 01-prolog-prejob

• 10-prolog-cmjob

Return values for the prolog/epilog scripts have these meanings:

• 0: the next script in the directory is run.

• A non-zero return value: no further scripts are executed from the prolog/epilog directory.

Often, the script in a prolog/epilog directory is not a real script but a symlink, with the symlink going to a real file located in a different directory. The general script is then able to take care of what is expected of the symlink. The name of the symlink, and destination file, usually hints at what the script is expected to do.

For example, if any health checks are marked to run as prejob checks during cm-wlm-setup configuration, then each of the PBS workload manager variants use the symlink 01-prolog-prejob within the prolog directory /cm/local/apps/ <workload manager>/var/prologs/. The symlink links to the script /cm/local/apps/cmd/scripts/prolog-prejob. In this case, the script is expected to run prior to the job.

Example

[root@basecm10 apps]# pwd
/cm/local/apps
[root@basecm10 apps]# ls -l *pbs*/var/prologs/
openpbs/var/prologs/:
total 0
lrwxrwxrwx 1 root root ... 01-prolog-prejob -> /cm/local/apps/cmd/scripts/prolog-prejob
pbspro/var/prologs/:
total 0
lrwxrwxrwx 1 root root ... 01-prolog-prejob -> /cm/local/apps/cmd/scripts/prolog-prejob

Epilog scripts (which run after a job run) have the location /cm/local/apps/ <workload manager>/var/epilogs/. Epilog script names follow the same execution sequence pattern as prolog script names.

It should be noted that the 01-prolog-prejob symlink is created and removed by BCM on each compute node where prejob is enabled in the workload manager entity. Each such entity provides a Enable Prejob parameter that affects the symlink existence:

Example

[head->wlm[openpbs]]% get enableprejob
yes
[head->wlm[openpbs]]%

This parameter is set to yes by cm-wlm-setup when at least one health check is selected as a prejob one. If any healthcheck was configured as a prejob check before cm-wlm-setup execution, and the administrator had a checkmark for that health check, then the prejob is considered enabled.

7.3.4.3 Workload Manager Configuration For Prolog And Epilog Scripts

BCM configures generic prologs and epilogs during workload manager setup with cm-wlm-setup. The administrator can configure prologs and epilogs using appropriate parameters in the configuration of the workload managers, by creating the symlinks in the local prologs and epilogs directories.

Generic prologs and epilogs are configured by default to run on job compute nodes (one run per each node per job) for Slurm, PBS variants and LSF. However, for GE the generic prolog runs only on main job node due to the limitations of GE configuration.

The following parameters for prologs and epilogs can be configured with cmsh or Base View:

• Slurm

  • Prolog Slurmctld: the fully qualified path of a program to execute before granting a new job allocation. The program is executed on the same node where the slurmserver role is assigned. The parth corresponds to the PrologSlurmctld parameter in slurm.conf.

  • Epilog Slurmctld: the fully qualified path of a program to execute upon termination of a job allocation. The program is executed on the same node where the slurmserver role is assigned. Corresponds with the EpilogSlurmctld parameter in slurm.conf.

  • Prolog: the fully qualified path of a program to execute on job compute nodes before granting a new job or step allocation. The program corresponds to the Prolog parameter, and by default points to the generic prolog. This prolog runs on every node of the job if the Prolog flags parameter contains the flag Alloc (the default value), otherwise it is executed only on the first node of the job.

  • Epilog: the fully qualified path of a program to execute on job compute nodes when the job allocation is released.

• LSF

  • Prolog: the fully qualified path of a program to execute on the LSF server node on job allocation. As an LSF queue parameter, it corresponds to the PRE_EXEC parameter in lsb.queues.

  • Epilog: the fully qualified path of a program to execute on the LSF server node on job allocation release. As an LSF queue parameter, it corresponds to the POST_EXEC parameter in lsb.queues

  • Host prolog: the fully qualified path of a program to execute on each node of a job before the job is started. Corresponds to the HOST_PRE_EXEC parameter in lsb.queues. By default it is configured to run the generic prolog

  • Host epilog: the fully qualified path of a program to execute on each node of a job after the job is finished. Corresponds to the HOST_POST_EXEC parameter in lsb.queues. By default it is configured to run the generic epilog.

• GE

  • Prolog: ugeserver role parameter that points to the fully qualified path of a program to execute. By default it is the generic prolog. It is executed only on the GE server node.

  • Epilog: ugeserver role parameter that is empty by default. It can be configured to run the generic epilog. It is executed only on the GE server node.

• PBS variants

  Pelogs: prolog and epilog hooks that emulate the classic PBS prologue and epilogue scripts located in the pbs_mom directory. The pelogs are configured in the appropriate workload manager instance, within pelogs mode when configuring the PBS cluster entity. For example, in cmsh:

  Example

  [basecm10]% wlm use openpbs
  [basecm10->wlm[openpbs]]% pelogs
  [basecm10->wlm[openpbs]->pelogs]% list
  Name (key) Enabled  Order
  ---------- -------- --------
  cm_epilog  yes      99
  cm_prolog  yes      1
  [basecm10->wlm[openpbs]->pelogs]%
  [basecm10->wlm[openpbs]->pelogs]% show cm_prolog
  Parameter                        Value
  -------------------------------- ------------------------------------------------
  Enabled                          yes
  Name                             cm_prolog
  Events                           execjob_begin
  Path                             /cm/shared/apps/pbspro/var/cm/cm-pelog-hook.py
  Default action                   RERUN
  Enable parallel                  yes
  Verbose user output              no
  Torque compatible                no
  Order                            1
  Alarm                            35
  Debug                            no
  [basecm10->wlm[openpbs]->pelogs]% show cm_epilog
  Parameter                        Value
  -------------------------------- ------------------------------------------------
  Enabled                          yes
  Name                             cm_epilog
  Events                           execjob_end
  Path                             /cm/shared/apps/pbspro/var/cm/cm-pelog-hook.py
  Default action                   RERUN
  Enable parallel                  yes
  Verbose user output              no
  Torque compatible                no
  Order                            99
  Alarm                            35
  Debug                            no
  [basecm10->wlm[openpbs]->pelogs]%

  The parameters of each pelog correspond to appropriate parameters of PBS hooks:

  • Name: hook name that will be used in PBS

  • Enabled: flag to enable the hook in PBS

  • Events: list of PBS events that the hook will run on

  • Path: path to the hook script that will be imported to PBS in case the hook is not found. The script will not be imported if a hook with the same name already exists in PBS

  • Order: the hook execution order

  • Alarm: the hook alarm time (timeout), in seconds

  Additional parameters related to the prolog and epilog hooks only are:

  • Default action: The PBS default action when a prolog or epilog fails

  • Enable parallel: enable parallel prologues and epilogues, that run on sister moms

  • Verbose user output: provide verbose hook output to the user’s .o/.e file

  • Torque compatible: make torque compatible from prolog/epilog command line arguments point of view

  By default, two pelogs are added. These are to run the generic prolog and the generic epilog. If needed, the administrator can add more pelog hooks that will run on different events.

7.4 Enabling, Disabling, And Monitoring Workload Managers

7.4.0.1 Enabling And Disabling A WLM

A WLM can be disabled for all nodes with cm-wlm-setup. Disabling the WLM means the workload management services are stopped by removing roles, and removing the WLM cluster object.

Alternatively, a WLM can be enabled or disabled by the administrator via role addition and role removal with Base View or cmsh. This is described further on in this section.

Multiple WLM instances of the same type: Versions of NVIDIA Base Command Manager prior to 9.0 already had the ability to have different workload managers run at the same time. However, NVIDIA Base Command Manager version 9.0 introduced the additional ability to run many workload managers of the same kind at the same time.

Example

Two WLM instances, Slurm and OpenPBS, are already running at the same time in the cluster, with each WLM assigned to one category. Then, BCM can start up a third WLM instance, such as another Slurm WLM instance. These WLM instances are alternatively called WLM clusters, because they effectively allow one cluster to function as many separate clusters as far as running WLMs is concerned.

From the Base View or cmsh point of view a WLM consists of

• a WLM server, usually on the head node

• WLM clients, usually on the compute nodes

For the administrator, enabling or disabling the servers or clients is then simply a matter of assigning or unassigning a particular WLM server or client role on the head or compute nodes, as deemed appropriate.

The administrator typically also sets up an appropriate WLM environment module (slurm, uge, openpbs, pbspro, lsf), so that it is loaded up for the end user (section 2.2.3).

7.4.1 Enabling And Disabling A WLM With Base View

A particular WLM package may be installed, but the WLM may not be enabled. This can happen, for example, if disabling a WLM that was previously enabled.

If a WLM instance exists, then the WLM client, submission, and server roles can be enabled or disabled from Base View by assigning or removing the appropriate roles to nodes, categories, or configuration overlays. Within the role, the properties of the WLM may be further configured by setting options.

7.4.1.1 Workload Manager Role Assignment To An Individual Node With Base View

Workload Manager Server The following roles are WLM roles that can be assigned to a node:

• server

• submit

• accounting (for the Slurm WLM only, to configure and run the slurmdbd service)

• client

For example, a Slurm server role can be assigned to a head node, basecm10, via the navigation path:

Devices > Head Nodes[basecm10] > Edit > Settings > Role > Role list[ADD] > SlurmServerRole

Figure 7.14: Workload management role assignment on a head node

The role window for the server then opens up, and allows role options to be set for the workload manager server. For example, for Slurm, a builtin or backfill option can be set for the Scheduler parameter. The workload manager server role is then saved with the selected options (figure 7.15).

To have the server start up on non-head nodes (but not for a head node), the imageupdate command (section 5.6.2) can be run. The workload manager server process and any associated schedulers then automatically start up.

Figure 7.15: Workload management role assignment options on a head node

Workload Manager Client Similarly, the workload manager client process can be enabled on a node or head node by having the workload manager client role assigned to it. Some basic options can be set for the client role right away.

Saving the role, and then running imageupdate (section 5.6.2), automatically starts up the client process with the options chosen, and managed by CMDaemon.

7.4.1.2 Workload Manager Role Assignment To A Category With Base View

It is true that workload manager role assignment can be done as described in the preceding text for individual non-head nodes. However it is usually more efficient to assign roles using categories or configuration overlays, due to the large number of compute nodes in typical clusters.

For example, the case can be considered of all physical on-premises non-head nodes. By default these are in the default category. This means that, by default, roles in the category are automatically assigned to all those non-head nodes, unless, as an exception, an individual node configuration overrides the category setting and uses a role setting instead at node level.

Viewing the possible workload manager roles for the category default is done by using the navigation path: Grouping > Categories[default] > Edit > Settings > Roles > Add Once the role is selected, its options can be edited and saved.

For compute nodes, the role assigned is usually a workload manager client. If the assigned role is that of a workload manager client, then the node with that role can have queues, GPUs, and other parameters specified for it.

For example, queues can then be assigned via the navigation path:
HPC > WLM Management Clusters[cluster instance] > Job Queues while GPUs, if using Slurm as the workload manager with default settings, can then be specified via the navigation path: Configuration Overlays > slurm-client-gpu > roles > slurmclient > edit > Generic Resources > gpu

The workload manager server role can also be assigned to a non-head node. For example, a Slurm server role can be taken on by a non-head node. This is the equivalent to the --server-nodes option of cm-wlm-setup.

Saving the roles with their options and then running imageupdate (section 5.6.2) automatically starts up the newly-configured workload manager.

7.4.1.3 Workload Manager Role Options With Base View

Each compute node role (workload manager client role) has options that can be set for GPUs, Queues, and Slots. Generally, the value that is set for Slots is the number of jobs expected to run on a node simultaneously. This number can, for example, be set to the number of threads. Threads (virtual cores) in the x86_64 architecture are provided by Intel’s hyper-threading (HT), or by AMD’s simultaneous multithreading (SMT).

The physical CPU, the cores on the CPU, and the threads of a core (HT, SMT) should not be confused with each other, they are distinct concepts, and can all have different values.

• Slots, in a workload manager, corresponds in BCM to:

  • the CPUs setting (a NodeName parameter) in Slurm’s slurm.conf

  • the nproc setting in PBS,

  In LSF setting the number of slots for the client role to 0 means that the client node does not run jobs on itself, but becomes a submit host, which means it is able to forward jobs to other client nodes.

  The default value for Slots is AUTO, which means that the value for Slots is auto-detected. The parameter may be alternatively be set to a non-negative number. Each WLM has a different implementation on how this is done. For instance, for Slurm, BCM uses Slurm’s own auto-detection implementation. On the other hand, for UGE BCM sets its value in the client role configuration overlay to the lowest number of hyperthreads among the nodes.

• Queues with a specified name are available in their associated role after they are created. The creation of queues is described in sections 7.6.2 (using Base View) and 7.7.2 (using cmsh).

All server roles also provide the option to enable or disable the External Server setting. Enabling that means that the server is no longer managed by BCM, but provided by an external device.

The cmsh equivalent of enabling an external server is described here.

7.4.2 Enabling And Disabling A Workload Manager With cmsh

A particular workload manager package may be set up, but not enabled. This can happen, for example, if no WLM server or WLM client role has been assigned.

If a WLM instance exists, then the WLM client, server, or submit roles can be enabled from cmsh by assigning it from within the roles submode. Within the assigned role, the properties of the WLM may be further configured by setting options.

7.4.2.1 Workload Manager Role Assignment To A Configuration Overlay With cmsh

In cmsh, workload manager role assignment to a configuration overlay (section 2.1.5) can be done using configurationoverlay mode. By default cm-wlm-setup run as a TUI session creates some configuration overlays with suggestive names, and assigns roles to the configuration overlays according to what the names suggest. Thus, for example, with the cm-wlm-setup TUI session used to carry out an express setup for Slurm, the configuration overlays that get created are the following:

Example

[basecm10->configurationoverlay]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting     500        yes                                              slurmaccounting
slurm-client         500        no                              default          slurmclient
slurm-server         500        yes                                              slurmserver
slurm-submit         500        no                              default          slurmsubmit
wlm-headnode-submit  600        yes                                              slurmsubmit

Nodes in the default category can take on the slurmclient or slurmsubmit role by setting the nodes for the role using the associated configuration overlays slurm-client or slurm-submit.

The wlm-headnode-submit configuration overlay is a special overlay. It is applied only to the head node, and is shared among all installed workload managers. Setting this overlay means that the head node, by default, has a submit role for a given workload manager.

Example

[basecm10->configurationoverlay]% use slurm-client
[basecm10->configurationoverlay[slurm-client]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             slurm-client
Revision
All head nodes                   no
Priority                         500
Nodes
Categories                       default
Roles                            slurmclient
Customizations                   <0 in submode>
[basecm10->configurationoverlay[slurm-client]]% set nodes
node001  node002  node003  basecm10
[basecm10->configurationoverlay[slurm-client]]% set nodes node001..node002
[basecm10->configurationoverlay*[slurm-client*]]% commit
[basecm10->configurationoverlay[slurm-client]]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting     500        yes                                              slurmaccounting
slurm-client         500        no             node001,node002  default          slurmclient
slurm-server         500        yes                                              slurmserver
slurm-submit         500        no                              default          slurmsubmit
wlm-headnode-submit  600        yes                                              slurmsubmit
[basecm10->configurationoverlay[slurm-client]]%

All the head nodes can also be made to take on the configuration overlay role by setting its All head nodes value to yes. The union set of All head nodes with Nodes is the set of nodes to which the role is applied for that configuration overlay.

Values for the parameters in a role, such as the slurmclient role, can be set within the configuration overlay:

Example

[basecm10->configurationoverlay[slurm-client]]% roles
[basecm10->configurationoverlay[slurm-client]->roles]% use slurmclient
[basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                               slurmclient
Revision
Type                               SlurmClientRole
Add services                       yes
WLM cluster                        slurm
Slots                              0
All Queues                         no
Queues                             defq
Features
Sockets                            0
Cores Per Socket                   0
ThreadsPerCore                     0
Boards                             0
SocketsPerBoard                    0
RealMemory                         0B
NodeAddr
Weight                             0
Port                               0
TmpDisk                            0
Reason
CPU Spec List
Core Spec Count                    0
Mem Spec Limit                     0B
GPU auto detect                    BCM
Node Customizations                <0 in submode>
Generic Resources                  <0 in submode>
Cpu Bindings                       None
Slurm hardware probe auto detect   yes
Memory autodetection slack         10.0%
IMEX                               no
[basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]%

After the workload manager roles are assigned or unassigned, and after running imageupdate (section 5.6.2) for non-head nodes, the associated workload manager services automatically start up or stop as appropriate.

The configuration overlay role values are inherited by categories and nodes, unless the categories and nodes have their own values set. Thus, for role properties, a value set at node level overrides values set at category level, and a value set at configuration overlay level overrides a value set at category level. This is typical of how properties of objects are inherited in BCM levels.

7.4.2.2 Workload Manager Role Assignment To A Category With cmsh

In cmsh, workload manager role assignment to a node category can be done using category mode, using the category name, assigning a role from the roles submode, setting the WLM instance for that role, and committing the modified role:

Example

[root@basecm10 ~]# cmsh
[basecm10]% category
[basecm10->category]% use default
[basecm10->category[default]]% roles
[basecm10->category[default]->roles]% assign slurmclient
[basecm10->category[default]->roles*[slurmclient*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm10     basecm10,node001 node001,node002
[basecm10->category[default]->roles*[slurmclient*]]% set wlmcluster slurm1
[basecm10->category[default]->roles*[slurmclient*]]% commit

Settings that are assigned in the slurmclient role of the category overrule the slurmclient role configuration overlay settings.

The role assignment at category level requires the value for a WLM instance to be specified for wlmcluster before the commit command is successful.

7.4.2.3 Workload Manager Role Assignment To An Individual Node With cmsh

In cmsh, assigning a workload manager role to a head node can be done in device mode. This can be done by using the head node name as the device, assigning the workload manager role to the device, setting the WLM instance value to the role within the role submode, and committing the modified role.

Example

[root@basecm10 ~]# cmsh
[basecm10]% device
[basecm10->device]% use basecm10
[basecm10->device[basecm10]]% roles
[basecm10->device[basecm10]->roles]% assign slurmserver
[basecm10->category[default]->roles*[slurmserver*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm10     basecm10,node001 node001,node002
[basecm10->category[default]->roles*[slurmserver*]]% set wlmcluster slurm1
[basecm10->device*[basecm10*]->roles*[slurmserver*]]% commit
[basecm10->device[basecm10]->roles[slurmserver]]%

For regular nodes, role assignment is done via device mode, using the node name. Th node name is assigned the workload manager role, the WLM instance value is set for that role in the role submode, and the modified role is committed.

Example

[root@basecm10 ~]# cmsh
[basecm10]% device
[basecm10->device]% use node001
[basecm10->device[node001]]% roles
[basecm10->device[node001]->roles]% assign slurmclient
[basecm10->device[node001]->roles*[slurmclient*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm10     basecm10,node001 node001,node002
[basecm10->device[node001]->roles*[slurmclient*]]% set wlmcluster slurm1
[basecm10->device[node001]->roles*[slurmclient*]]% commit
[basecm10->device[node001]->roles[slurmclient]]%

The role assignment at node level requires the value for a WLM instance to be specified for wlmcluster before the commit command is successful.

Role assignment values set in device mode have precedence over any role assignment values set in category mode for that node. This means, for example, that if a node is originally in a node category with a slurmclient role and queues set, then when the node is assigned a slurmclient role from device mode, its queue properties are empty by default.

7.4.2.4 Setting Options For Workload Manager Settings With cmsh

In the preceding text, it is explained how the workload manager client or server is assigned a role (such as slurmclient or slurmserver) within the roles submode. It is done from within a main mode of cmsh. The main modes from which role assignment can be done are: configurationoverlay, category or device.

Options for workload managers in general: Whatever main mode is used, the workload manager options for a role can then be set with the usual object commands introduced in section 2.5.3.

• WLM client options: For example, the configuration options of a WLM client, such as the PBS Professional client, can be seen by using the show command on the role. Here it can be seen at a category level, for the default category default:

  Example

  [basecm10->category[default]->roles[pbsproclient]]% show
  Parameter                        Value
  -------------------------------- ------------------------------------------------
  Add services                     yes
  All Queues                       no
  GPUs                             0
  Name                             pbsproclient
  Properties
  Queues
  Revision
  Slots                            1
  Type                             PbsProClientRole
  WLM cluster
  Mom Settings                     <submode>
  Comm Settings                    <submode>
  Node Customizations              <0 in submode>

  The Slots option can be set in the role

  Example

  [basecm10->category[default]->roles[pbsproclient]]% set slots 2
  [basecm10->category*[default*]->roles*[pbsproclient*]]% commit
  [basecm10->category[default]->roles[pbsproclient]]%

• WLM server options: Similarly, WLM server options can be managed from an assigned server role. For PBS, the pbsproserver role for a device shows:

  Example

  [basecm10->device[basecm10]->roles]% use pbsproserver
  [basecm10->device[basecm10]->roles[pbsproserver]]% show
  Parameter                        Value
  -------------------------------- ------------------------------------------------
  Name                             pbsproserver
  Revision
  Type                             PbsProServerRole
  Add services                     yes
  WLM cluster
  External Server                  no
  Comm Settings                    <submode>

Option to set an external workload manager: A workload manager can be set to run as an external server from within a device mode role:

Example

[basecm10->device[basecm10]->roles[pbsproserver]]% set externalserver on
[basecm10->device[basecm10]->roles[pbsproserver*]]% commit

For convenience, setting it on the head node is recommended.

The Base View equivalent of configuring externalserver is described here.

7.4.3 Monitoring The Workload Manager Services

By default, the workload manager services are monitored. BCM attempts to restart the services using the service tools (section 3.14), unless the role for that workload manager service is disabled, or the service has been stopped.

Workload manager roles and corresponding services can be disabled using cm-wlm-setup (section 7.3), Base View role configuration (section 7.4.1), or cmsh role configuration (section 7.4.2).

The daemon service states can be viewed for each node via the shell, cmsh, or Base View (section 3.14).

Queue submission and scheduling daemons normally run on the head node. From Base View their states are viewable via the navigation path to the services running on the node. For example, on a head node (figure 7.16), via:

Devices > Head Nodes > [basecm10] > Settings > JUMP TO > Services

Figure 7.16: Services seen on head node in Base View

For a regular node, a similar navigation path for node001, for example, is: Devices > Nodes > node001 > Settings > JUMP TO > Services

and leads to a view of services on the regular nodes (figure 7.17):

Figure 7.17: Services seen on regular node in Base View

Considering only the WLMs: in figure 7.16 the pbsserver is seen running on the head node, while in figure 7.17 the pbsmom server is seen running on the compute node.

The navigation path:
Devices > Head Nodes > basecm10 > Settings > JUMP TO > Roles
shows the roles that result in the servers running on the head node (figure 7.18):

Figure 7.18: Roles seen on head node in Base View

Similarly, the navigation path:
Devices > Nodes > node001 > Settings > JUMP TO > Roles
shows the roles on a regular node such as node001 (figure 7.19):

Figure 7.19: Roles seen on regular node in Base View

The roles seen in these figures are from the defaults that cm-wlm-setup provides in an express setup.

For regular nodes, the inheritance of roles from category level or configuration overlay level is indicated by the values in the INHERITED column. Thus, in figure 7.19, the pbsprosubmit and pbsproclient roles are decided by the default setting from the category level.

The assignment of roles can be varied to taste for WLMs. This allows WLM services to run on the head node or on the regular nodes.

From cmsh the services states are viewable from within device mode, using the services command. One-liners from the shell to illustrate this are (output elided):

Example

[root@basecm10 ~]# cmsh -c "device services node001; status"
Service      Status
------------ -----------
nslcd        [   UP   ]
pbsmom       [   UP   ]
[root@basecm10 ~]# cmsh -c "device services basecm10; status"
Service      Status
------------ -----------
                   ...
pbsserver    [   UP   ]

Roles can be viewed from within the main modes of configurationoverlay, category, or device. One-liners to view these are:

Example

[root@basecm10 ~]# cmsh -c "configurationoverlay; list"
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
openpbs-client       500        no                              default          pbsproclient
openpbs-server       500        yes                                              pbsproserver
openpbs-submit       500        no                              default          pbsprosubmit
wlm-headnode-submit  600        yes                                              pbsprosubmit
[root@basecm10 ~]# cmsh -c "category; use default; roles; list -p"
Name (key)
----------------------------
[overlay:openpbs-client:500] pbsproclient
[overlay:openpbs-submit:500] pbsprosubmit
[root@basecm10 ~]# cmsh -c "device use node001; roles; list -p"
Name (key)
----------------------------------------
[overlay:openpbs-client:500] pbsproclient
[overlay:openpbs-submit:500] pbsprosubmit
[root@basecm10 ~]# cmsh -c "device use basecm10; roles; list -p"
Name (key)
----------------------------------------
[750] backup
[750] boot
[750] firewall
[750] headnode
[750] monitoring
[750] provisioning
[750] storage
[overlay:openpbs-server:500] pbsproserver
[overlay:openpbs-submit:500] pbsprosubmit

The -p|--priority option displays of the list command the priority setting for the roles.

7.5 Configuring And Running Individual Workload Managers

BCM deals with the various choices of workload managers in as generic a way as possible. This means that not all features of a particular workload manager can be controlled, so that fine-tuning must be done through the workload manager configuration files. Workload manager configuration files that are controlled by BCM should normally not be changed directly because BCM overwrites them. However, overwriting by CMDaemon is prevented on setting the directive:

FreezeChangesTo <workload manager>Config = <true|false>

in cmd.conf (Appendix C), where <workload manager> takes the value of Slurm, UGE, LSF, or PBSPro, as appropriate. The value of the directive defaults to false.

A list of configuration files that are changed by CMDaemon, the items changed, and the events causing such a change are listed in Appendix H.

A very short guide to some specific workload manager commands that can be used outside of the NVIDIA Base Command Manager 10 system is given in Appendix F.

7.5.1 Configuring And Running Slurm

7.5.1.1 Slurm Packages

At the time of writing of this section (March 2026), BCM can be integrated with Slurm packages for Slurm versions 23.02, 23.11, 24.05, 24.11, and 25.05.

For Slurm version 23.02, the following packages are available from the BCM repositories for RHEL8, RHEL9, and derivatives:

• slurm23.02: Simple Linux Utility for Resource Management, Slurm Workload Management

• slurm23.02-client: Spool dir and slurm conf installed on nodes

• slurm23.02-contribs: Perl tool to print Slurm job state information

• slurm23.02-libpmi: Slurm’s implementation of the pmi libraries

• slurm23.02-openlava: openlava/LSF wrappers for transition from OpenLava/LSF to Slurm

• slurm23.02-pam: PAM module for restricting access to compute nodes via Slurm.

• slurm23.02-perlapi: Perl API to Slurm.

• slurm23.02-slurmdbd: Slurm database daemon

• slurm23.02-slurmrestd: Slurm REST API translator

• slurm23.02-torque: Torque/PBS wrappers for transition from Torque/PBS to Slurm.

For Slurm version 23.11, the value of 23.02 is simply replaced by 23.11 in the preceding list of packages. Slurm version 23.11 also has an extra package:

• slurm23.11-sackd: Slurm authentication daemon. Used on login nodes that are not running slurmd daemons to allow authentication to the cluster.

For Slurm versions 24.05, 24.11, or 25.05, the value of 23.11 in the 23.11 package names is simply replaced by 24.05, 24.11, or 25.05 respectively. In addition, Sharp packages (here) are available for 24.05, and 24.11.

The distribution version of Slurm (package: slurm) is not integrated with BCM and conflicts with the preceding packages. It should not be used.

Important updates from upstream are patched into the BCM repositories. If updating Slurm packages, all the Slurm packages should be updated to the same version, on the compute nodes as well as on the scheduling node.

7.5.1.2 Updating From Earlier Slurm Versions To slurm23.11

Upgrading between major versions of Slurm is generally possible. It is a good idea to upgrade one version at a time, rather than jumping 2 or more versions ahead, which requires a full wipe of the Slurm configuration.

If Slurm is using Pyxis (section 7.3.3), then upgrading the Slurm version means that Pyxis needs to be reinstalled using cm-wlm-setup. The reinstallation run for Pyxis compiles Pyxis and recreates a plugin directory for the new Slurm version under /cm/shared/apps/slurm/.

An upgrade from one major version of Slurm to another can be carried out according to the following example, which is for an update from major version 22.05 to version 23.11, and avoids total reconfiguration of the Slurm configuration:

• It is recommended that no jobs are running. Draining nodes (section 7.7.3) is one way to arrange this over time. No new jobs run on a drained node, but old ones are allowed to finish.

• When all running jobs are finished, then Slurm server services—slurmctld and slurmdbd—should be stopped using cmsh or Base View (section 3.14.2):

  Example

  [basecm10->device[basecm10]->services]% stop slurmctld
  [basecm10->device[basecm10]->services]% stop slurmdbd

• The old Slurm packages should then be removed. There can be only one version of Slurm at a time, so there will be a package installation conflict if a new version is installed while an old one is still there.

  Removal can be carried out on RHEL-based systems with, for example:

  [root@basecm10 ~]# yum remove slurm22.05*

  The old packages must also be removed from each software image that uses it:

  [root@basecm10 ~]# cm-chroot-sw-img /cm/images/<software image>
  ...
  [root@<software image> /]# yum remove slurm22.05*
  ...removal takes place...
  [root@<software image> /]# exit

  The cm-chroot-sw-img wrapper utility is discussed in section 11.4.1.

• The new packages can then be installed. For installation onto the RHEL head node, the installation might be carried out as follows:

      [root@basecm10 ~]# yum install slurm23.11 slurm23.11-client slurm23.11-contribs \
      slurm23.11-perlapi slurm23.11-devel slurm23.11-pam slurm23.11-slurmdbd \
      slurm23.11-slurmrestd

  The client package can be installed in each software image with, for example:

  [root@basecm10 ~]# cm-chroot-sw-img /cm/images/<software image>
  ...
  [root@<software image> /]# yum install slurm23.11-client
  ...installation takes place
  [root@<software image> /]# exit

  Other Slurm packages from the repository may also be installed on the head node and within the software images, as needed.

• The new Slurm version is then set in cmsh or Base View, in the Slurm WLM cluster configuration:

  Example

  [root@basecm10 ~]# cmsh
  [basecm10]% wlm use slurm
  [basecm10->wlm[slurm]]% set version 23.11; commit

• Slurm server services slurmctld and slurmdbd should then be started again using cmsh or Base View:

  Example

  [basecm10->device[basecm10]->services]% start slurmdbd
  [basecm10->device[basecm10]->services]% start slurmctld

• The nodes can then have their new image placed on them, and the new Slurm configuration can then be taken up. This can be done in the following two ways:

  1. The regular nodes can then be restarted to supply the live nodes with the new image and get the new Slurm configuration running.

  2. Alternatively, the imageupdate command (section 5.6.2) can be run on the live nodes to supply them with the image.

     Running the imageupdate command in dry mode (the default) first is recommended. The synclog command can then be run to check there are no unexpected changes that will take place due to the update. If all is well, then imageupdate’s wet mode flag -w can be used in order to really carry out the task.

     For example, the change can be checked, and then actually carried out, for the image on node001 with:

     [root@basecm10 ~]# cmsh
     [basecm10]% device use node001
     [basecm10->device[node001]]% imageupdate
     Performing dry run (use synclog command to review result, then pass -w to perform real update)...
     ...some messages...
     imageupdate [ COMPLETED ]
     [basecm10->device[node001]]% synclog
     ...rsync dry run output...
     [basecm10->device[node001]]% imageupdate -w
     ...same messages as before, but this time it really happens...
     [basecm10->device[node001]]% commit

     The last commit command triggers the reconfiguration of the file /etc/systemd/system/slurmd.service.d/99-cmd.conf on the node. After a short time—around 30 seconds—the file is regenerated. The slurmd service on the node can then be restarted with:

     [root@basecm10 ~]# ssh node001 "systemctl daemon-reload"
     cmsh -c "device services node001; use slurmd; restart"

7.5.1.3 Slurm NVIDIA Sharp Plugin

NVIDIA’s Sharp packages are packages that offload some operations from CPUs and GPUs to the network. Sharp is the abbreviation used for the Scalable Hierarchical Aggregation and Reduction Protocol. The protocol refers to the reduction in the amount of data traversing the network and the reduction in the time for collective operations. Using Sharp frees up more CPUs and GPU resources for computation.

The Sharp binaries are packaged and available for Slurm versions 24.05 and 24.11.Running

apt-cache search slurm| grep sharp

is a way of listing the packages.

To use the NVIDIA Sharp plugin, the existing Slurm packages must be substituted by packages with the -sharp suffix. This allows the plugin and associated extra options to be used.

Example

The following session on an Ubuntu system illustrates the existing Slurm version 24.05 packages, and then carrying out a replacement of these with the corresponding Sharp versions:

    basecm10:~# dpkg --get-selections | grep slurm | cut -f1 | tr "\n" " " ; echo
slurm24.05 slurm24.05-client slurm24.05-contribs slurm24.05-devel slurm24.05-perlapi slurm24.05-slurmdbd
    basecm10:~# apt install slurm24.05-sharp slurm24.05-sharp-client slurm24.05-sharp-contribs\
    slurm24.05-sharp-devel slurm24.05-sharp-perlapi slurm24.05-sharp-slurmdbd

The change should be treated like the upgrade procedure here, and should likewise end with the cluster administrator selecting the -sharp Slurm version parameter in cmsh and committing it.

Example

root@basecm10:~# cmsh -c "wlm use slurm; get version"
24.05-sharp

This plugin is only useful for a network that has the necessary hardware and services configured. Further information about Sharp can be found at https://docs.nvidia.com/networking/display/sharpv300.

7.5.1.4 Configuring Slurm

After Slurm setup is configured and installed with cm-wlm-setup (section 7.3), the Slurm software components are installed in a symlinked directory /cm/shared/apps/slurm/current.

Slurm clients and servers can be configured to some extent via role assignment (sections 7.4.1 and 7.4.2).

Using cmsh, advanced option parameters can be set under the slurmclient and slurmserver roles. The settings for the roles can be done at configuration overlay, category, or node level (sections 2.1.5, 2.1.6).

By default, the cm-wlm-setup utility configures Slurm using configuration overlays.

Example

[basecm10->configurationoverlay]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting     500        yes                                              slurmaccounting
slurm-client         500        no                              default          slurmclient
slurm-server         500        yes                                              slurmserver
slurm-submit         500        no                              default          slurmsubmit
wlm-headnode-submit  600        yes                                              slurmsubmit

The settings within the roles can be viewed and modified. For example, the slurmclient role of the slurm-client configuration overlay can be viewed:

Example

[basecm10->configurationoverlay]% roles slurm-client
[basecm10->configurationoverlay[slurm-client]->roles]% show slurmclient
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             slurmclient
Revision
Type                             SlurmClientRole
Add services                     yes
WLM cluster                      slurm
Slots                            0
All Queues                       no
Queues                           defq
Features
Sockets                          0
Cores Per Socket                 0
ThreadsPerCore                   0
Boards                           0
SocketsPerBoard                  0
RealMemory                       0B
NodeAddr
Weight                           0
Port                             0
TmpDisk                          0
Reason
CPU Spec List
Core Spec Count                  0
Mem Spec Limit                   0B
GPU auto detect                  BCM
Node Customizations              <0 in submode>
Generic Resources                <0 in submode>
Cpu Bindings                     None
Slurm hardware probe autodetect  yes
Memory autodetection slack       0.0%
IMEX                             no

Slurm Hardware Autodetection (non-GPU): The setting Slurm hardware probe autodetect in the Slurm role enables automated hardware detection for Slurm for non-GPU hardware. (Specifically for GPUs is the related Slurm GPU auto detect setting (here)).

Slurm hardware probe autodetect enables detection for the following Slurm parameters:

• corespersocket

• threadspercore

• boards

• socketsperboard

• sockets

• realmemory

The autodetected parameters are placed in the NodeName line in slurm.conf.

The Slurm client role parameters can be modified. For example Core Spec Count:

Example

[basecm10->configurationoverlay[slurm-client]->roles]% set slurmclient corespeccount 2
[basecm10->configurationoverlay*[slurm-client*]->roles*]% commit

As usual, values set at node level override the values set at categories and configuration overlays level.

For example, to set corespeccount to 4, only for node001 but not for other nodes, the session might run further as:

Example

[basecm10->configurationoverlay[slurm-client]->roles]% device use node001
[basecm10->device[node001]]% roles
[basecm10->device[node001]->roles]% assign slurmclient
[basecm10->device*[node001*]->roles*[slurmclient*]]% set corespeccount 4
[basecm10->device*[node001*]->roles*[slurmclient*]]% commit
Field                    Message
------------------------ ----------------------------------------------------------------
wlmCluster               Error: The WLM cluster should be set
[basecm10->device*[node001*]->roles*[slurmclient*]]% wlm list
Type   Name (key)               Server nodes Submit nodes     Client nodes
------ ------------------------ ------------ ---------------- ----------------
Slurm  slurm                    basecm10     basecm10,node001 node001,node001
[basecm10->device*[node001*]->roles*[slurmclient*]]% set wlmcluster slurm
[basecm10->device*[node001*]->roles*[slurmclient*]]% commit

In the preceding session, the role needs to be assigned at node level with assign slurmclient because it does not initially exist at node level. If it already existed, then use slurmclient could have been used to descend into that role.

Also in the preceding session, one of the values that the Slurm client needs to know is wlmcluster, which decides which WLM it is to work with on the cluster. The value is selected from the list of WLM instance names in wlm mode.

The level of the active role can be seen with the list command. For example, the Slurm client role assignment at node level is seen here:

Example

[basecm10->device[node001]->roles[slurmclient]]% list
Name (key)
-----------------------------------
[overlay:slurm-submit] slurmsubmit
slurmclient

Removing the assignment has the list command display the configuration overlay Slurm client role assignment:

Example

[basecm10->device[node001]->roles[slurmclient]]% unassign slurmclient; commit
[basecm10->device[node001]->roles[slurmclient]]% list
Name (key)
-----------------------------------
[overlay:slurm-submit] slurmsubmit
[overlay:slurm-client] slurmclient

Slurm Hardware Autodetection For GPUs: The Slurm GPU auto detect setting is a setting seen in the session output here. It manages Slurm GRES configuration (https://slurm.schedmd.com/gres.conf.html) automatically, and can be set:

• globally, for all Slurm compute nodes in a Slurm instance, from within the wlm mode of cmsh

• for a particular role, such as in a device role, a category role, and a configuration overlay role for a Slurm client

  Example

  [basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]% set gpuautodetect [TAB][TAB]
  bcm     none    nrt    nvml    off     oneapi  rsmi

If GPU auto detect is assigned a value, then a corresponding value is assigned to the autodetect parameter in Slurm’s gres.conf file. One out of 7 values can be assigned to GPU auto detect:

1. bcm: to use BCM values, which are the values that CMDaemon automatically puts in for the vendor (NVIDIA, Intel, AWS, AMD) and for the number of GPUs.

2. none: to not have the autodetect parameter exist in gres.conf. In this case, there is actually no corresponding value that can be assigned to the autodetect of gres.conf.

3. nrt: to detect AWS Trainium/Inferentia devices. <— not yet, in October 2024, trunk, or bcm10.24.09

4. nvml: to detect NVIDIA GPUs.

5. off: to turn Slurm GPU autodetection off.

6. oneapi: to detect Intel GPUs.

7. rsmi: to detect AMD GPUs

Slurm GPU autodetection is described further here.

Slurm accounting database configuration in cmsh: After package setup is carried out with cm-wlm-setup (section 7.3), cmsh can be used to modify the settings for the Slurm accounting database. Changes can be carried out in the slurm-accounting configuration overlay:

[basecm10->configurationoverlay[slurm-accounting]]% show
Parameter                        Value
-------------------------------- ---------------------------
Name                             slurm-accounting
Revision
All head nodes                   no
Priority                         500
Nodes                            node001,node002
Categories
Roles                            slurmaccounting
Customizations                   <0 in submode>

In the preceding overlay, slurmdbd runs on the accounting node(s). It is possible to run slurmdbd on both head nodes or on one or two compute nodes, as set in the configuration overlay. It is not possible to mix up head nodes and compute nodes for this configuration, and it is not possible to run slurmdbd on more than two nodes.

The other Slurm accounting changes can be found within the slurmaccounting role. For example, if node001 has been set during setup with cm-wlm-setup to be the primary accounting storage host, and HA has been enabled so that accounting runs on both nodes, then the parameter values may be displayed as follows:

Example

[basecm10->configurationoverlay[slurm-accounting]->roles[slurmaccounting]]% show
Parameter                        Value
-------------------------------- --------------------------
ha                               true
primary                          node001
Name                             slurmaccounting
Revision
Type                             SlurmAccountingRole
Add services                     yes
DbdPort                          6819
StorageHost                      node001
StoragePort                      3306
StorageLoc                       slurm_acct_db
StorageUser                      slurm
Slurm Clusters                   slurm

StorageHost, StoragePort, StorageLoc and StorageUser are settings for connecting to the MySQL database, If these are for an external host, then they must be configured manually so that they are reachable by the nodes running slurmdbd.

Generic resources (gres) configuration in Slurm: In order to configure generic resources, the genericresources mode can be used to set a list of objects. Each object then represents one generic resource available on nodes.

Each value of name in genericresources must already be defined in the list of GresTypes. The list of GresTypes is defined in the wlm role for the instance.

Example

[basecm10->wlm[slurm]]% get grestypes
gpu

Several generic resources entries can have the same value for name (for example gpu), but must have a unique alias. The alias is a string that is used to manage the resource entry in cmsh or in Base View. The string is enclosed in square brackets in cmsh, and is used instead of the name for the object. The alias does not affect Slurm configuration.

For example, to add two GPUs for all the nodes in the default category which are of type k20xm, and to assign them to different CPU cores, the following cmsh commands can be run:

Example

[basecm10]% configurationoverlay use slurm-client
[basecm10->configurationoverlay[slurm-client]]% roles
[basecm10->configurationoverlay[slurm-client]->roles*]% use slurmclient
[...[slurmclient]]% genericresources
[...[slurmclient]->genericresources]% add gpu0
[...[slurmclient*]->genericresources*[gpu0*]]% set name gpu
[...[slurmclient*]->genericresources*[gpu0*]]% set file /dev/nvidia0
[...[slurmclient*]->genericresources*[gpu0*]]% set cores 0-7
[...[slurmclient*]->genericresources*[gpu0*]]% set type k20xm
[...[slurmclient*]->genericresources*[gpu0*]]% add gpu1
[...[slurmclient*]->genericresources*[gpu1*]]% set name gpu
[...[slurmclient*]->genericresources*[gpu1*]]% set file /dev/nvidia1
[...[slurmclient*]->genericresources*[gpu1*]]% set cores 8-15
[...[slurmclient*]->genericresources*[gpu1*]]% set type k20xm
[...[slurmclient*]->genericresources*[gpu1*]]% commit
[...[slurmclient]->genericresources[gpu1]]% list
Alias (key) Name     Type     Count    File
----------- -------- -------- -------- ----------------
gpu0        gpu      k20xm             /dev/nvidia0
gpu1        gpu      k20xm             /dev/nvidia1
[...[slurmclient]->genericresources[gpu1]]%

Typically this configuration is done automatically during the GPU configuration process as outlined in the GPU Configuration Screens section here, where by default the configuration overlay is given the name slurm-client-gpu.

In Base View, the navigation path:

Configuration Overlays > slurm-client-gpu > Edit > Roles > slurmclient > Edit > Generic Resources > ADD

provides the equivalent (figure 7.20):

Figure 7.20: Base View access to NVIDIA GPU configuration options

After the generic resources are committed, BCM updates the gres.conf file.

Since NVIDIA Base Command Manager version 8.2 and higher, a single gres.conf configuration file, located at /cm/shared/apps/slurm/var/etc/slurm/gres.conf is used.

If the category consists of node001 and node002, then the entries to the gres.conf file in this case would look like:

Example

# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia0 Cores=0-7
NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia1 Cores=8-15
# END AUTOGENERATED SECTION   -- DO NOT REMOVE
[root@basecm10 ~]#

Slurm topology configuration: Slurm supports topology-aware resource allocation to optimize job performance. For example, on a hierarchical network, the topology/tree plugin works so that Slurm allocates resources to jobs to minimize network contention. The network topology configuration for a Slurm instance <slurmcluster> is described by the file:

/cm/shared/apps/slurm/var/etc/ <slurmcluster>/topology.conf

The file is generated and modified by the topograph service (here) when the cluster administrator modifies the topology.

The following parameters can be set to modify the topology:

• topology switches: A Slurm cluster configuration parameter that defines the list of switches used to write the topology file.

  Example

  [root@basecm10 ~]# cmsh
  [basecm10]% wlm use slurm
  [basecm10->wlm[slurm]]% set topologyswitches switch01 switch02 switch03
  [basecm10->wlm[slurm*]]% commit

• switch ports: The switch port definitions of a node or a switch. This information is used by BCM to construct the topology tree for topology.conf.

  Example

  [root@basecm10 ~]# cmsh
  [basecm10]% device use node001
  [basecm10->device[node001]]% set switchports switch01:3
  [basecm10->device[node001*]]% commit
  [basecm10->device[node001]]% use switch01
  [basecm10->device[switch01]]% set switchports switch02:15
  [basecm10->device[switch01*]]% commit

• SlurmTopology: an extra node value that allows the Slurm network topology to be configured without an actual need to configure switches and switch ports in BCM.

  Example

  [root@basecm10 ~]# cmsh
  [basecm10]% device
  [basecm10->device]% set node001 -e SlurmTopology switch02 switch01
  [basecm10->device*]% set node002 -e SlurmTopology switch02 switch01
  [basecm10->device*]% set node003 -e SlurmTopology switch03
  [basecm10->device*]% commit