Workload Management

For clusters that have many users and a significant load, a workload manager (WLM) system allows a more efficient use of resources to be enforced for all users than if there were no such system in place. This is because without resource management, there is a tendency for each individual user to over-exploit common resources.

When a WLM is used, the end user can submit a job to it. This can be done interactively, but it is typically done as a non-interactive batch job.

The WLM assigns resources to the job, and checks the current availability as well as checking its estimates of the future availability of the cluster resources that the job is asking for. The WLM then schedules and executes the job based on the assignment criteria that the administrator has set for the WLM system. After the job has finished executing, the job output is delivered back to the user.

Among the hardware resources that can be used for a job are GPUs. Installing CUDA software to enable the use of GPUs is described in section 9 of the Installation Manual. Configuring GPU settings for BCM is described in section 3.17.2 of the Administrator Manual. Configuring GPU settings for an individual WLM is described in the section on getting that particular WLM up and running.

The details of job submission from a user’s perspective are covered in the User Manual.

Sections 7.1–7.5 cover the installation procedure to get a WLM up and running.

Sections 7.6 –7.7 describe how Base View and cmsh are used to view and handle jobs, queues and node drainage.

Section 7.8 shows examples of WLM assignments handled by BCM.

Section 7.9 describes the power saving features of WLMs.

Section 7.10 describes cgroups, a resources limiter, mostly in the context of WLMs.

Section 7.11 describes WLM customizations for settings other than the common settings covered by BCM.

7.1 Workload Managers Choices

Some WLM packages are installed by default, others require registration from the distributor before installation.

During cluster installation, a WLM can be chosen (figure 3.9 of the Installation Manual) for setting up. The choices are:

• Slurm: Available as version 25.05 or 25.11. Slurm is a free (GPL) job scheduler, with commercial support.

• LSF v10.1: IBM Spectrum LSF (Load Sharing Facility) version 10.1, is a further development of what used to be IBM Platform LSF.

• None: For clusters that need no HPC job-scheduling.

The WLMs in the preceding list can also be chosen and set up later using the cm-wlm-setup tool (section 7.3).

After installation, if there are no major changes in the WLM for updated versions of the workload managers, then

• WLMs that are packaged with BCM (Slurm) can have their packages updated using standard package update commands (yum update and similar). The installation and configuration of the WLM from the updated packages is carried out as described later on in this chapter.

• WLMs such as LSF that are installed by picking up software from the vendor can be updated by following vendor guidelines.

7.2 Forcing Jobs To Run In A Workload Management System

Another preliminary step is to consider forcing users to run jobs only within the WLM system. Having jobs run via a WLM is normally a best practice.

For convenience, BCM defaults to allowing users to log in via ssh to a node, using the authorized keys files stored in each users directory in /home (section 2.3.2). This allows users to run their processes without restriction, that is, outside the WLM system. For clusters with a significant load this policy results in a sub-optimal use of resources, since such unplanned-for jobs disturb any already-running jobs.

Disallowing user logins to nodes, so that users have to run their jobs through the WLM system, means that jobs are then distributed to the nodes only according to the planning of the WLM. If planning is based on sensible assignment criteria, then resources use is optimized—which is the entire aim of a WLM in the first place.

7.2.1 Disallowing User Logins To Regular Nodes Via cmsh

The usernodelogin setting of cmsh restricts direct user logins from outside the WLM, and is thus one way of preventing the user from using node resources in an unaccountable manner. The usernodelogin setting is applicable to node categories only, rather than to individual nodes.

In cmsh the attribute of usernodelogin is set from within category mode:

Example

[root@basecm11 ~]# cmsh
[basecm11]% category use default
[basecm11->category[default]]% set usernodelogin onlywhenjob
[basecm11->category*[default*]]% commit

The attributes for usernodelogin are:

• always (the default): This allows all users to ssh directly into a node at any time.

• never: This allows no user other than root to directly ssh into the node.

• onlywhenjob: This allows the user to ssh directly into the node when a job is running on it. It typically also prevents other users from doing a direct ssh into the same node during the job run, since typically the WLM is set up so that only one job runs per node. However, an ssh session that is already running is not automatically terminated after the job is done.

  • Some cluster administrators may wish to allow some special user accounts to override the onlywhenjob setting, for example for diagnostic purposes. Before giving the details of how to override the setting, some background explanation is probably useful:

    The onlywhenjob setting works with the PAM system, and adds the following line to /etc/pam.d/sshd on the regular nodes:

        account          required       pam_bright.so

    Nodes with the onlywhenjob restriction can be configured to allow a particular set of users to access them, despite the restriction, by allowing them with the PAM system, as follows: Within the software image <node image> used by the node, that is under /cm/images/ <node image>, the administrator can add the set of user accounts to the file etc/security/pam_bright.d/pam_allow.conf. This file is installed in the software image with a chroot installation (section 9.4) of the cm-libpam package.

    Groups of users can be allowed using the file etc/security/pam_bright.d/pam_allow_group.conf.

    Other adjustments to PAM configuration, such as the number of attempted logins and the associated wait time per login, can be carried by editing the /etc/security/pam_bright.d/cm-check-alloc.conf file.

    The image can then be updated to allow the users, by running the imageupdate command in cmsh (section 5.6.2), or by clicking the Update node option in Base View (section 5.6.3).

7.2.2 Disallowing User Logins To Regular Nodes Via Base View

In Base View, user node login access is set via a category setting, for example for the default category via the navigation path in figure 7.1: Grouping > Categories[default] > > Settings > User node login

Figure 7.1: Disallowing user logins to nodes via Base View

7.2.3 Disallowing Other User Processes Outside Of Workload Manager User Processes

Besides disabling user logins, administrators may choose to disable interactive jobs in the WLM as an additional measure to prevent users from starting jobs on other nodes.

Administrators may also choose to set up scripts that run after job execution. Such scripts can terminate user processes outside the WLM, as part of a policy, or for general administrative hygiene. These are Epilog scripts and are part of the WLM.

The WLM documentation has more on configuring these options.

7.2.4 High Availability By Workload Managers

NVIDIA Base Command Manager uses the existing built-in high availability (HA) functionalities of workload managers as much as possible. A double server HA configuration with the existing built-in HA functionality can be carried out using the cm-wlm-setup utility (section 7.3), or using the Base View HA wizard (section 7.4.1). The built-in functionality makes use of the primary WLM server and secondary WLM server, which are placed on separate nodes.

The HA configuration aspect (section 15.1.3) in this case means that a WLM server role is assigned to both the WLM primary and the WLM secondary nodes. These primary WLM and secondary WLM servers can then:

• both be on head nodes,

• both be on regular (compute) nodes

It is not possible to configure a mixed setup—that is with one head node and one regular node—for the WLM servers on an HA setup.

In a non-HA cluster, the WLM primaryserver value is set to the head node by default.

Example

[head->wlm[slurm]]% get primaryserver
head

This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node.

If both the head nodes are configured with the WLM server roles, then the WLM primaryserver value is set to the primary cluster head node. This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node, and the backup WLM server on the secondary head node.

Example

[head1->wlm[slurm]]% get primaryserver
head1
[head1->wlm[slurm]]% exit
root@head1:~# ssh head2 "cmha makeactive"
...
...[logging into the newly active head node, and checking where the primary server is:]...
[head2->wlm[slurm]]% get primaryserver
head1

If the parameter primaryserver is unset in wlm mode, and the Slurm server role is assigned to the head nodes, then the slurmctld service is always started on both head nodes. During a failover, the Slurm configuration is regenerated and both slurmctld services are restarted. The primary slurmctld is always on the active head node.

If two regular nodes are configured with the WLM server roles instead, then in the cluster entity configuration the primaryserver parameter is set to one of the compute nodes.

The cluster primary head node and the WLM primary server should not be confused. In particular, the active cluster head node and active workload manager server are not necessarily the same for the case of the Slurm or LSF workload managers. For these, if the passive cluster head node becomes an active cluster head node without a crash, then this does not trigger a passive WLM server on the newly active head node to also become an active WLM server. Thus, the WLM server can be active on a passive head node, and vice versa, because the WLM primary server is independent of the cluster primary head node.

7.2.4.1 Slurm High Availability, scontrol takeover And slurmstartpolicy

This section considers some Slurm high availability issues on running scontrol takeover with BCM integration.

Logical steps during the Slurm takeover between head nodes: Slurm high availability normally waits a short while with moving its control activity over to a passive head node that has just been made active. After a short delay, the Slurm failover algorithm is followed so that the Slurm controller (slurmctld) takes over operations on the newly active head node.

Slurm can speed up takeover by running scontrol takeover from the Bash shell on the node with the secondary slurmctld. Running scontrol takeover stops the primary slurmctld and triggers the Slurm failover algorithm.

This can be useful if the administrator wants to carry out maintenance on the usually active head node. Carrying out the takeover lets the Slurm service remain available, and stops the Slurm primary controller following the newly active head node.

The logic on how this works in BCM is:

1. The production state starts with

   • an active primary

   • a passive secondary

   • the primary head node set up as the primary Slurm controller

2. the administrator runs cmha makeactive on the secondary passive. This causes the following to happen:

• scontrol takeover is run by CMDaemon on the secondary passive

• CMDaemon goes into the passive state on the primary node, and slurmctld on the primary node stops

• CMDaemon goes into the active state on the secondary node, and slurmctld on the secondary node starts

3. there is no interruption in Slurm services, because with scontrol takeover CMDaemon has forced the backup slurmctld to take over

4. the administrator can now carry out maintenance on the primary (now passive) head node

5. once maintenance has been completed, the administrator can optionally reboot the primary (passive) head node, and then make it active again with cmha makeactive to get back to the original production state

The problem with the preceding logic is that by default BCM automatically restarts a stopped slurmctld that it notices. This means that the results of scontrol takeover in steps 2 and 3 are unknown if carried out manually. That is, it is not practical to run the scontrol takeover command directly from Bash because it is possible to unintentionally configure the primary and backup slurmctld with an incorrect status.

To deal with this problem cleanly, CMDaemon provides the slurmctldstartpolicy parameter, described next.

The slurmctldstartpolicy parameter and Slurm takeover behavior: The slurmctldstartpolicy parameter can take the following values to decide the slurmctld behavior during scontrol takeover:

• TAKEOVER: The default value on an HA cluster.

• ALWAYS: The default value on a non-HA cluster.

• ACTIVEONLY: Useful only if /cm/shared is not mounted on both head nodes at the same time. For example, as in DAS (Direct Attached Storage), which should never be used anyway for shared storage in a production system.

1. If within NVIDIA Base Command Manager, in the SlurmServerRole, the slurmctldstartpolicy parameter is set to TAKEOVER, then disabling the automatic restart of the slurmctld service is unnecessary:

   [root@basecm11 ~]# cmsh
   [basecm11-]% configurationoverlay
   [basecm11-]>configurationoverlay]% use slurm-server
   [basecm11->configurationoverlay[slurm-server]]% roles
   [basecm11->configurationoverlay[slurm-server]->roles]% use slurmserver
   [basecm11->configurationoverlay[slurm-server]->roles[slurmserver]]% set slurmctldstartpolicy TAKEOVER
   [basecm11->configurationoverlay*[slurm-server*]->roles*[slurmserver*]]% commit

   The setting value can be cleared with:

   [basecm11->configurationoverlay[slurm-server]->roles[slurmserver]]% set slurmctldstartpolicy ALWAYS
   [basecm11->configurationoverlay*[slurm-server*]->roles*[slurmserver*]]% commit

2. If cm-wlm-setup installs Slurm on an HA cluster, then the slurmctldstartpolicy parameter can be set in one of the TUI screens during the session. Before HA is configured, the parameter has a default value of ALWAYS. After HA is configured it has a default value of TAKEOVER. The default value of TAKEOVER causes prefailoverscript in the failover submode (section 15.4.6) to take the value /cm/local/apps/cmd/scripts/slurm.takeover.sh

3. If cm-wlm-setup installs Slurm on a non-HA cluster, then the prefailoverscript parameter must be set within the failover submode of partition mode:

   [basecm11->partition[base]->failover]% set prefailoverscript /cm/local/apps/cmd/scripts/slurm.takeover.sh
   [basecm11->partition*[base*]->failover*]% commit

   This ensures that the active head node is taken over by Slurm during a change of the active head node.

The takeover is temporary. If the head node which has the Slurm primary slurmctld on it is restarted, or if the service itself is restarted manually, then the primary slurmctld takes back its role, as per Slurm documentation.

For a Slurm cluster with a single slurmctld, where the Slurm server role is assigned to a single compute node or the BCM cluster has only one head node, the parameter slurmctldstartpolicy should be set to ALWAYS, which is the default.

Table 7.2.4.1 shows what slurmctld does when the slurmctldstartpolicy parameter is set to TAKEOVER and when in SlurmWlmCluster the parameter primaryserver is set. The detailed behavior per head node depends on whether the head nodes are active or primary:

* any value of yes or no

Table 7.2.4.1 slurmctldstartpolicy parameter and slurmctld

Slurm controller high availability on compute nodes: If the Slurm server role is assigned to two compute nodes, the TAKEOVER policy is configured automatically only when both nodes belong to a failover group. In this configuration, slurmctld does not start on the primary node while the secondary node is active.

7.3 Installation Of Workload Managers

Normally the administrator selects a WLM to be used during BCM installation (figure 3.9 of the Installation Manual). A WLM may however also be added and configured after BCM has been installed, by using cm-wlm-setup, which is part of the cm-setup package, or by using the Base View WLM wizard.

With most other objects, BCM front ends—cmsh and Base View—can be used to create a new object from scratch, or can clone a new object from another existing object. However, the front ends cannot do this for a WLM object. An attempt to create or clone a new WLM object using cmsh or Base View is prohibited by the front ends, because there are many pitfalls possible in configuration.

A new WLM object, and WLM instance, can therefore only be installed via cm-wlm-setup, the Base View WLM wizard, or by selecting a WLM during the initial BCM installation.

7.3.1 Running cm-wlm-setup In CLI Mode

The recommended way to run the cm-wlm-setup utility is without options or arguments, in which case a TUI dialog starts up. A TUI session run with cm-wlm-setup is covered in section 7.3.2.

However, the cm-wlm-setup utility can alternatively be used in a non-GUI, command-line, mode, with options and arguments. The utility has the following usage:

[root@basecm11 ~]# cm-wlm-setup -h
usage: Workload manager setup cm-wlm-setup [-c <config_file>]
                                           [--setup | --disable]
                                           [--wlm <name>]
                                           [--server-nodes SERVER_NODES]
                                           [--server-primary SERVER_PRIMARY]
                                           [--server-overlay-name SERVER_OVERLAY_NAME]
                                           [--server-overlay-priority SERVER_OVERLAY_PRIORITY]
                                           [--client-categories CLIENT_CATEGORIES]
                                           [--client-nodes CLIENT_NODES]
                                           [--client-overlay-name CLIENT_OVERLAY_NAME]
                                           [--client-overlay-priority CLIENT_OVERLAY_PRIORITY]
                                           [--client-slots <slots>]
                                           [--submit-categories SUBMIT_CATEGORIES]
                                           [--submit-nodes SUBMIT_NODES]
                                           [--submit-overlay-name SUBMIT_OVERLAY_NAME]
                                           [--submit-overlay-priority SUBMIT_OVERLAY_PRIORITY]
                                           [--wlm-cluster-name WLM_CLUSTER_NAME]
                                           [--reboot] [--reset-cgroups]
                                           [--yes-i-really-mean-it]
                                           [--archives-location <path>]
                                           [--license <license>] [--purge]
                                           [--accounting-overlay-name ACCOUNTING_OVERLAY_NAME]
                                           [--accounting-overlay-priority ACCOUNTING_OVERLAY_PRIORITY]
                                           [--with-pyxis]
                                           [--add-pyxis]
                                           [--reinstall-pyxis]
                                           [--remove-pyxis]
                                           [--pyxis-data-directory <path>]
                                           [--nvidia-gpus NVIDIA_GPUS] [-v]
                                           [--no-distro-checks] [--json]
                                           [--output-remote-execution-runner]
                                           [--on-error-action {debug,remotedebug,undo,abort}]
                                           [--skip-packages]
                                           [--min-reboot-timeout <reboot_timeout_seconds>]
                                           [--allow-running-from-secondary]
                                           [--dev] [-h]

The help output from running cm-wlm-setup -h continues on beyond the preceding text output, and presents more options.

These options can be grouped as follows:

7.3.1.1 Optional Arguments

• --setup: Helps set up a server, enable roles, and create the default queues/partitions

• --disable: Disable WLM services

• -h, --help: Displays the help screen

7.3.1.2 Common Arguments

• -c <YAML configuration file>: Loads a runtime configuration for plugins, from a YAML configuration file.

7.3.1.3 Options For Installing Or Managing A WLM

• --wlm <WLM name>: Specifies which WLM is to be set up. Choices for <WLM name> are:

  • slurm

  • lsf

• --wlm-cluster-name <WLM cluster name>: Specifies the name for the new WLM cluster that is to be set up.

• --reboot: Reboot after install

7.3.1.4 Server Role Settings

• --server-nodes <server nodes>: Sets the server roles of the WLM to the value set for <server nodes>, which is a comma-separated list of nodes. Default value: HEAD, which is a reserved name for the head node.

• --server-primary <primary server>: Sets the hostname used for the primary server to <primary server>. Default name: HEAD.

• --server-overlay-name <server overlay name>: Sets the server role configuration overlay name to <server overlay name>. Default name: <WLM name>-server, where <WLM name> is the name specified in the --wlm option.

• --server-overlay-priority <server overlay priority>: Sets the server role configuration overlay priority to <server overlay priority>. Default value: 500.

7.3.1.5 Client Role Settings

• --client-categories <client categories>: Sets the client roles of the WLM to the value set for <client categories nodes>, which is a comma-separated list of node categories. Default value: default.

• --client-nodes <client nodes>: Sets the client roles of the WLM to the value set for <client nodes>, which is a comma-separated list of nodes. No value set by default.

• --client-overlay-name <client overlay name>: Sets the client role configuration overlay name to <client overlay name>. Default name: <WLM name>-client, where <WLM name> is the name specified in the --wlm option.

• --client-overlay-priority <client overlay priority>: Sets the client role configuration overlay priority to <client overlay priority>. Default value: 500.

• --client-slots <slots>: Sets the number of slots on the client to <slots>.

7.3.1.6 Submit Role Settings

• --submit-categories <submit categories>: Sets the submit roles of the WLM to the value set for <submit categories nodes>, which is a comma-separated list of node categories that are submit nodes. Default value: default.

• --submit-nodes <submit nodes>: Sets the submit roles of the WLM to the value set for <submit nodes>, which is a comma-separated list of submit nodes. No value set by default.

• --submit-overlay-name <submit overlay name>: Sets the submit role configuration overlay name to <submit overlay name>. Default name: <WLM name>-submit, where <WLM name> is the name specified in the --wlm option.

• --submit-overlay-priority <submit overlay priority>: Sets the submit role configuration overlay priority to <submit overlay priority>. Default value: 500.

7.3.1.7 Disable Options

• --reset-cgroups: Disable joining cgroup controllers with systemd setting JoinControllers

• --yes-i-really-mean-it: Required for additional safety

7.3.1.8 Workload Manager Specific Options

• --archives-location <path>: Set the directory path for the archive files, only for LSF. This parameter is mandatory for LSF installation.

• --license <path>: Set the path to the LSF license.

• --purge: Remove the directories on disable, for LSF.

7.3.1.9 Slurm Accounting Role Settings

• --accounting-overlay-name <accounting overlay name>: Sets the accounting role configuration overlay

• --accounting-overlay-priority <accounting overlay priority>: Sets the accounting role configuration overlay priority to <accounting overlay priority>. Default value: 500.

7.3.1.10 Slurm Pyxis Settings

• --add-pyxis: Add Pyxis to an existing cluster

• --reinstall-pyxis: Reinstall or upgrade Pyxis on all the clusters using it

• --remove-pyxis: Remove Pyxis from an existing cluster

• --with-pyxis: Enable Pyxis

• --pyxis-data-directory <path>: Sets directory where images will be stored

7.3.1.11 Slurm GPU Settings

• --nvidia-gpus <NVIDIA GPUs specification>: Sets the NVIDIA GPUs that will be used in the configuration, specifying type and number of GPUs. For example: --nvidia-gpus=A100:4

• --gpu-client-nodes <NVIDIA GPU client nodes>: Sets a comma-separated list of nodes assigned to the GPU overlay.

• --gpu-client-categories <NVIDIA GPU client categories>: Sets a comma-separated list of node categories assigned to the GPU overlay.

7.3.1.12 Advanced Options

• -v, --verbose: This displays a more verbose output. It can be helpful in troubleshooting.

• --no-distro-checks: Disables distribution checks based on ds.json.

• --json: Use json formatting for logs printed to STDOUT.

• --output-remote-execution-runner: Format output for CMDaemon.

• --on-error-action {debug,remotedebug,undo,abort}: Upon encountering a critical error, instead of asking the user for choice, the setup will do the selected action.

• --skip-packages: Skip the stages which install packages. Requires packages to be already installed.

• --min-reboot-timeout <timeout>: How long to wait for nodes to finish reboot, in seconds. Minimum value: 300. Default value: 300.

• --allow-running-from-secondary: Allow the wizard to be run from the secondary when it is the active head node.

• --dev: Enables additional command line arguments for developers.

7.3.2 Running cm-wlm-setup As A TUI

Running cm-wlm-setup with no options and with no arguments brings up a TUI screen (figure 7.2).

Figure 7.2: cm-wlm-setup TUI initial screen

7.3.2.1 Express Installation

The Setup (Express) menu option allows the administrator to select the workload manager in the next screen (figure 7.3), and to install it with a minimal number of configuration steps. If it has already been installed, but disabled using cm-wlm-setup, then it can also be re-enabled, instead of installed from scratch.

Figure 7.3: cm-wlm-setup TUI WLM selection screen

7.3.2.2 Step-by-step Installation

If the Setup (Step By Step) menu option is chosen instead of the express option, then this also allows the administrator to select the workload manager in figure 7.3. But after selection, there are a number of extra configuration steps that can be carried out which are not available in the express configuration. Guidance is given for these extra steps, and sensible default values are already filled in for many options.

One part of the step-by-step session involves assigning the WLM client role to the compute nodes of the cluster. Only the non-GPU compute nodes should be assigned the (standard, non-GPU) WLM client role. The GPU compute nodes are assigned a GPU WLM client role in the section of the TUI wizard that deals with GPU configuration.

The WLM client role is assigned to the entire category—the default category— of non-head nodes by default (figure 7.4):

Figure 7.4: Slurm with cm-wlm-setup: WLM Client role category configuration screen

The WLM client role can be assigned to selected nodes only by setting the default category checkbox to blank, and then selecting the non-GPU nodes in the following screen. For example, as illustrated in figure 7.4, where the standard, non-GPU nodes are node002 and node003:

Figure 7.5: Slurm with cm-wlm-setup: WLM client role node configuration screen

7.3.2.3 GPU Configuration Screens

The GPU configuration screens are extra steps available during a Setup (Step By Step) session. The GPU configuration steps are discussed summarily in the quickstart section for GPUs, here.

The GPU configuration steps are covered in this section in more detail:

After configuring the WLM server, WLM submission and WLM client roles for the nodes of the cluster, a screen that asks if GPU resources should be configured is displayed (figure 1.1):

Figure 1.1: Slurm With cm-wlm-setup: GPU Configuration Entry Screen

Following through brings up a GPU device settings configuration screen (figure 1.2):

Figure 1.2: Slurm With cm-wlm-setup: GPU Device Settings Configuration Screen

The help text option in the screen gives hints based on the descriptions at https://slurm.schedmd.com/gres.conf.html, and also as seen in Slurm’s man gres.conf.5.

Figure 1.2 shows 2 physical GPUs on the node being configured. The type is an arbitrary string for the GPU, and each CPU core is allocated an alias GPU device.

The next screen (figure 1.3) allows the NVIDIA CUDA MPS (Multi-Process Service) to be configured:

Figure 1.3: Configuring An NVIDIA GPU For Slurm With cm-wlm-setup: MPS Settings Configuration Screen

The help text for this screen gives hints on how the fields can be filled in. The number of GPU cores (figure 1.3) for a GPU device can be set.

Further screens allow:

• the configuration overlay (section 2.1.6) name to be set for the GPU WLM clients. By default the name is set to slurm-client-gpu.

• a GPU WLM client role to be assigned to a category, if, for example, all the GPU nodes have been given their own category.

• the GPU WLM client role to be assigned to individual nodes instead of to a category.

• a configuration overlay priority to be set for a GPU WLM client role. By default, this has a value of 450.

• automatic GPU detection, with the following options:

  • BCM autodetects GPUs

  • Slurm autodetects NVIDIA GPU details

  • Slurm autodetects AMD GPU details

  • Slurm autodetects Intel GPU details

  • Slurm autodetects AWS Trainium/Inferentia devices

  • Manual GPU configuration in cm-wlm-setup

  • Skip GPU configuration

7.3.2.4 Slurm Accounting Database Configuration

In the step-by-step configuration for the Slurm WLM, after the server and client parameters have been set, the Slurm accounting database configuration can be set.

This requires setting:

• one or two accounting nodes

• a primary accounting node

All nodes must be reachable from slurmctld.

The accounting nodes can be set with the Select accounting nodes screen (figure 7.9):

Figure 7.9: cm-wlm-setup selection of accounting nodes

Head nodes and compute nodes cannot be mixed for the accounting nodes. If two accounting nodes are set, then the slurmdbd high availability configuration is automatically set up.

The primary accounting node—that is, the node that is AccountingStorageHost in slurm.conf—can then be set in the Select the primary accounting server node screen (figure 7.10):

Figure 7.10: cm-wlm-setup selection of primary accounting node

The other accounting node is automatically set to be AccountingStorageBackupHost.

The type of node on which the slurmdbd database runs can then be set (figure 7.11):

Figure 7.11: cm-wlm-setup selection of the type of node on which the Slurm accounting database runs

Only two of the three options are offered at a time. Which options are offered is determined by the accounting node selection made on the preceding screen. The accounting nodes are either all head nodes or all compute nodes. A mixed selection is rejected.

• If Use accounting node is selected, then the database is stored on the primary accounting node. This option is offered when the accounting nodes are head nodes

• If Select cluster node is selected, then the next screen allows the selection of the BCM MySQL overlay which is used to manage the database (figure 7.12). This option is offered when the accounting nodes are compute nodes, that is, when slurmdbd is offloaded from the head nodes. Whether the head nodes are configured for high availability makes no difference. If no MySQL configuration overlay is found, then the selection is rejected, and MySQL must first be installed with cm-mysql-setup. If the MySQL installation is running on more than one node (section 14.12), then mysql-router is automatically installed on all accounting nodes.

Figure 7.12: cm-wlm-setup Selection of host(s) for where the Slurm accounting database used by slurmdbd is stored

• If Select external node is selected, then a non-BCM hostname is asked for, and the cluster administrator is responsible for the installation and configuration of the DBMS. This option is always offered

Modifying the settings after installation is discussed here.

7.3.2.5 Disabling An Installation

The Disable option in figure 7.2 allows the administrator to disable an existing instance.

7.3.2.6 Summary Screen

The screen that appears after the configuration steps are completed, is the Summary screen (figure 7.13). This screen allows the configuration to be viewed, saved, or saved and deployed.

Figure 7.13: cm-wlm-setup TUI summary screen

In the final summary screen, as with other scripts that come with the cm-setup package, the configuration setting can be one of the following kinds:

• minimal config: For this minimal kind of configuration setting, the configuration consists only of the settings that differ from the default application configuration. The remaining default application configuration settings are not included in the configuration.

• config: For this non-minimal kind of configuration setting, the configuration consists of the minimal configuration settings (the ones changed by BCM), as well as the remaining default configuration settings (the ones provided with the application). That is to say: the remaining default configuration application settings are included in the configuration. The default application configuration settings may change as the upstream packages change.

If deployment is carried out, then several screens of output are displayed. After the deployment is completed, the log file can be viewed at /var/log/cm-wlm-setup.log.

7.3.3 Installation And Configuration Of Enroot And Pyxis With Slurm To Run Containerized Jobs

7.3.3.1 What Is Enroot?

As the README file for Enroot says, Enroot is an open source tool to turn container images into unprivileged sandboxes. Enroot can be thought of as an enhanced unprivileged chroot. It uses user and mount namespaces, as well as other modern kernel features, in order to create such sandboxes. It uses the same underlying technologies as containers, but removes much of the isolation that they inherently provide, while preserving filesystem separation. Further details on Enroot can be found at https://github.com/NVIDIA/enroot.

Enroot can be used with different workload managers, but for now only Slurm has been tightly integrated.

7.3.3.2 What Is Pyxis?

Pyxis (https://github.com/NVIDIA/pyxis) is a SPANK plugin for Slurm. SPANK (Slurm Plug-in architecture for Node and job (K)control, man spank.8) is a generic interface for job launch code control in Slurm. The Pyxis plugin requires the Enroot utility, and allows the user’s jobs to be executed seamlessly over Enroot in unprivileged containers. The plugin enables the Slurm submission utilities to provide container-related command line options.

7.3.3.3 Enroot And Pyxis Packages

BCM provides two Enroot package flavors: standard and hardened. The standard binaries are compiled as follows:

• Open file descriptors are inherited

• Spectre variant 2 (IBPB/STIBP) mitigations are disabled

• Spectre variant 4 (SSBD) mitigations are disabled

As a rule of thumb: the hardened flavor is slightly more secure but suffers a larger overhead.

• The standard packages are already installed by default with BCM, and are:

  • enroot: provides the main utility and helper files.

  • enroot+caps: a nearly empty package which runs a post-installation script to grant extra capabilities to unprivileged users. This allows them to import and convert container images.

  • pyxis-sources: installs a tarball with the Pyxis plugin source files. The tarball is used by cm-wlm-setup to compile the Pyxis plugin on a cluster. The version of the package (section 9.1), found using yum info pyxis-sources or apt-cache show pyxis-sources, implies the Pyxis source version that is provided. For example: pyxis-sources-0.20.0-[...].rpm provides the Pyxis source for version 0.20.0.

• The hardened packages that can be installed from the BCM repositories to replace the standard enroot and enroot+caps are:

  • enroot-hardened: provides hardened main utility and helper files.

  • enroot-hardened+caps: Provides extra capabilities to unprivileged users so that they can import and convert containers themselves.

The package pyxis-sources installs the tarball at /cm/local/apps/slurm/var/pyxis/pyxis-sources.tar.gz. If the administrator needs a version of Pyxis other than the one provided by the package, then the archive can be replaced with a source tarball of the same name.

7.3.3.4 How Are Enroot And Pyxis Set Up In BCM?

Pyxis and Enroot can be set up or removed by the administrator by choosing the appropriate options when Slurm is set up. In order to choose the appropriate options, cm-wlm-setup can be run in step-by-step mode (here). Alternatively, Pyxis-related command line options can be specified for the cm-wlm-setup arguments --add-pyxis and --reinstall-pyxis.

The step-by-step mode eventually presents a screen where the Pyxis setup can be enabled using a plugin:

Figure 7.14: cm-wlm-setup Pyxis setup screen

The Pyxis screen is available for RHEL8-based systems and Ubuntu 20 and beyond. Older systems are not supported.

If the plugin is enabled in the Pyxis setup screen, then cm-wlm-setup installs the enroot and enroot+caps packages from https://github.com/NVIDIA/enroot/releases into the software images where the Slurm client role is to be assigned. The cm-wlm-setup uitility also installs them directly on the head node if the head node was selected to run jobs. Pyxis sources are downloaded from GitHub, compiled with the installed Slurm, and installed in the appropriate Slurm directory.

If the administrator enables the Pyxis plugin, then a new screen with Enroot settings is shown:

Figure 7.15: cm-wlm-setup Enroot settings screen

The Enroot settings have reasonable default values. The administrator can change these:

• Share raw images among users and nodes: If enabled, then the raw container image is shared among users and nodes, and the administrator must ensure that the cache directory is shared among all the compute nodes. Enabling the option disables the creation of a UID subdirectory by the Enroot prolog script.

  The following points about implementing Enroot cache sharing should be considered by the administrator:

  • Enabling the checkbox for the option normally changes the default value of the Enroot cache directory, as defined in the TUI in the Cache directory field, to ${XDG_CACHE_HOME}/enroot. This field is normally evaluated by the Enroot installation to the home directory of the user:

    ~/.cache/enroot

    when a job is executed.

    The default value can also be changed in the TUI to any shared directory. If the specified directory does not exist, then cm-wlm-setup creates it with Unix directory permissions set to chmod 00777, which allows all users to share container images in the cache with each other on all the nodes. The extra 0 prefix is a non-POSIX-compliant GNU/Linux extension that clears the SUID and SGID bits so that users do not get elevated privileges simply through sharing the images.

  • Restrictions can be placed on the ability of regular users to share images and to create files in the cache, by setting more restrictive permissions for the directory manually, after the cm-wlm-setup run.

    For example, the permissions can be set to 00770, while the ownership group of the directory is changed to a group with all the Pyxis users in it. In this case, only the users within the Pyxis users group can create the image layers in the cache.

  • An alternative to cache sharing of images among users and nodes is to save Squashfs images in a shared directory, outside the cache. Then the users can specify a full path to a shared Squashfs image on the cluster in the srun/sbatch command line. In this case there is no need to share the cache among the nodes, as the Squashfs images are not copied over to cache before execution.

  Only the administrator should have access to updating the image.

• Share unpacked container images among nodes: Enables sharing of unpacked container images (image filesystems) among nodes. A user job can modify the filesystem, so it is not recommended to share this directory among users.

  Enabling the option disables removal of the Enroot data directory, if the container name is specified. This allows a user to keep the container filesystem between jobs run by the user. It is up to the user or the administrator to clean up the data directory if the option is enabled.

  If the data directory is shared among nodes, then it is up to a user to ensure that the container filesystem is unpacked at least once before real jobs start. Otherwise, a race condition is possible when the images are extracted simultaneously on several nodes.

• Cache directory: Path to the cache directory where raw container image layers are stored.

  If the directory is not shared, then the Slurm prolog script creates a subdirectory with a name that is the job user ID, while epilog cleans up that subdirectory. If the directory is shared, then neither prolog nor epilog touches the directory.

• Data directory: Directory where the container filesystems (unpackaged images) are stored. If the directory is shared, then the epilog script checks if the container name is specified for the job, and skips removal of the container subdirectory.

• Runtime directory: A working directory with temporary files created by Enroot.

When Pyxis is set up, cm-wlm-setup also prepares the following configuration files for the compute nodes:

• /etc/enroot/enroot.conf: This is a symlink to /cm/shared/apps/slurm/etc/enroot.conf. The configuration file provides reasonable default settings that allows Enroot to be used by many users. Important settings in the file are:

  • ENROOT_RUNTIME_PATH: working directory for enroot, created per user. Default value: /run/enroot/runtime/$(id -u)

  • ENROOT_CACHE_PATH: directory where container layers are stored. Default value: /run/enroot/cache/$(id -u)

  • ENROOT_DATA_PATH: directory where the filesystems of running containers are stored. Default value: /run/enroot/data/$(id -u)

  • ENROOT_SQUASH_OPTIONS: options passed to mksquashfs to produce container images. Default value: -noI -noD -noF -noX -no-duplicates

  • ENROOT_MOUNT_HOME: mount the current user’s home directory by default. Default value: yes.

  The administrator can change the symlink, or replace the file with a customized enroot.conf, if other values are preferred.

• /etc/sysctl.d/80-enroot.conf: symlink to /cm/shared/apps/slurm/etc/enroot-sysctl.conf. The file tunes sysctl parameters for Enroot.

• /cm/local/apps/slurm/var/prologs/50-prolog-enroot.sh: symlink to /cm/shared/apps/slurm/prologs/prolog-enroot.sh. This is the slurmd prolog that creates appropriate directories, with appropriate user permissions, that are used by Enroot.

• /cm/local/apps/slurm/var/epilogs/50-epilog-enroot.sh: symlink to /cm/shared/apps/slurm/epilogs/epilog-enroot.sh. Cleans the user directories used by Enroot.

When cm-wlm-setup finishes its Pyxis configuration run, then there is no need for the nodes to be rebooted. The plugin works immediately.

The new Slurm submission command option names start with either --container or --no-container. The full list of options can be displayed with the --help option. For example:

Example

[user@basecm11 ~]$ srun --help | grep container
      --container             Path to OCI container bundle
      --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH
                              [pyxis] the image to use for the container
      --container-mounts=SRC:DST[:FLAGS][,SRC:DST...]
                              [pyxis] bind mount[s] inside the container. Mount
      --container-workdir=PATH
                              [pyxis] working directory inside the container
      --container-name=NAME   [pyxis] name to use for saving and loading the
                              container on the host. Unnamed containers are
                              containers are not. If a container with this name
                              already exists, the existing container is used and
      --container-save=PATH   [pyxis] Save the container state to a squashfs
      --container-mount-home  [pyxis] bind mount the user's home directory.
      --no-container-mount-home
      --container-remap-root  [pyxis] ask to be remapped to root inside the
                              container. Does not grant elevated system
      --no-container-remap-root
                              [pyxis] do not remap to root inside the container
      --container-entrypoint  [pyxis] execute the entrypoint from the container
      --no-container-entrypoint
                              container image
      --container-writable    [pyxis] make the container filesystem writable
      --container-readonly    [pyxis] make the container filesystem read-only

Simple installation validation:

The simplest way to validate the Pyxis/Enroot setup after Slurm setup is to try out an srun command:

Example

[user@basecm11 ~]$ module load slurm
[user@basecm11 ~]$ srun --container-image=ubuntu grep PRETTY /etc/os-release
pyxis: importing docker image: ubuntu
PRETTY_NAME="Ubuntu 24.04.2 LTS"

A more thorough installation validation: In order to perform a more thorough test of Pyxis/Enroot, an NCCL-based test can be used. NCCL is the NVIDIA Collective Communications Library (https://docs.nvidia.com/deeplearning/nccl), which is a library of multi-GPU collective communication primitives. The test can be found at https://github.com/NVIDIA/nccl-tests. A prebuilt container image, with the NCCL test already installed, can be started as shown in the example that follows.

It should be noted that running a multi-tenant cluster with all the export flags enabled as in the example may compromise security. It is therefore not recommended as a standard configuration.

Example

[user@basecm11 ~]$ module load slurm
[user@basecm11 ~]$ srun --export="NCCL_DEBUG=INFO,NCCL_IB_DISABLE=1,PMIX_MCA_gds=hash" -N 2 \
--ntasks-per-node=1 --gpus-per-task=1 --mpi=pmix --container-image=deepops/mpi-nccl-test \
/nccl_tests/build/all_reduce_perf -b 1M -e 4G -f 2 -g 1
pyxis: imported docker image: deepops/mpi-nccl-test
pyxis: imported docker image: deepops/mpi-nccl-test
# nThread 1 nGpus 1 minBytes 1048576 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 validation: 1
#
# Using devices
#   Rank  0 Pid 175945 on    node001 device  0 [0x00] Tesla V100-SXM3-32GB
#   Rank  1 Pid 180379 on    node002 device  0 [0x00] Tesla V100-SXM3-32GB
node001:175945:175945 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.5<0>
node001:175945:175945 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
node001:175945:175945 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
node001:175945:175945 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.5<0>
node001:175945:175945 [0] NCCL INFO Using network Socket
NCCL version 2.11.4+cuda11.6
node002:180379:180379 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.6<0>
node002:180379:180379 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
node002:180379:180379 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
node002:180379:180379 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.6<0>
node002:180379:180379 [0] NCCL INFO Using network Socket
node002:180379:182573 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
node001:175945:177561 [0] NCCL INFO Channel 00/02 :    0   1
node001:175945:177561 [0] NCCL INFO Channel 01/02 :    0   1
node001:175945:177561 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
node001:175945:177561 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [receive] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [receive] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [receive] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [receive] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [send] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [send] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [send] via NET/Socket/0
node002:180379:182573 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [send] via NET/Socket/0
node001:175945:177561 [0] NCCL INFO Connected all rings
node002:180379:182573 [0] NCCL INFO Connected all rings
node001:175945:177561 [0] NCCL INFO Connected all trees
node001:175945:177561 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
node001:175945:177561 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
node002:180379:182573 [0] NCCL INFO Connected all trees
node002:180379:182573 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
node002:180379:182573 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
node001:175945:177561 [0] NCCL INFO comm 0x1551f0001000 rank 0 nranks 2 cudaDev 0 busId 60 - Init COMPLETE
#
#                                                       out-of-place                       in-place
#       size         count      type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                       (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
node001:175945:175945 [0] NCCL INFO Launch mode Parallel
node002:180379:182573 [0] NCCL INFO comm 0x1551f8001000 rank 1 nranks 2 cudaDev 0 busId 60 - Init COMPLETE
     1048576        262144     float     sum   1491.5    0.70    0.70  0e+00   1421.3    0.74    0.74  0e+00
     2097152        524288     float     sum   3077.5    0.68    0.68  0e+00   2568.8    0.82    0.82  0e+00
     4194304       1048576     float     sum   4617.3    0.91    0.91  0e+00   4622.5    0.91    0.91  0e+00
     8388608       2097152     float     sum   9483.1    0.88    0.88  0e+00   8911.9    0.94    0.94  0e+00
    16777216       4194304     float     sum    17516    0.96    0.96  0e+00    18613    0.90    0.90  0e+00
    33554432       8388608     float     sum    34799    0.96    0.96  0e+00    41837    0.80    0.80  0e+00
    67108864      16777216     float     sum    99790    0.67    0.67  0e+00    83126    0.81    0.81  0e+00
   134217728      33554432     float     sum   204614    0.66    0.66  0e+00   199530    0.67    0.67  0e+00
   268435456      67108864     float     sum   319701    0.84    0.84  0e+00   341630    0.79    0.79  0e+00
   536870912     134217728     float     sum   608809    0.88    0.88  0e+00   683016    0.79    0.79  0e+00
  1073741824     268435456     float     sum  1337187    0.80    0.80  0e+00  1247369    0.86    0.86  0e+00
  2147483648     536870912     float     sum  2638741    0.81    0.81  0e+00  2743454    0.78    0.78  0e+00
  4294967296    1073741824     float     sum  5996381    0.72    0.72  0e+00  5430549    0.79    0.79  0e+00
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.810575
#

The test demonstrates the usage of PMIX, MPI, and GPU in Enroot containers on multiple nodes. In the preceding example, 2 nodes with 1 GPU on each is requested by the job. For better results the cluster administrator can tune the test parameters.

It should be noted that the image is quite large, and requires enough free space under /var. Also, the transfer timeout (ENROOT_TRANSFER_TIMEOUT) in enroot.conf must be large enough to download such a large container image to the compute nodes. A value of at least 600 seconds is recommended.

If there are issues when executing MPI jobs with PMIX, then to help debug the issues the Pyxis documentation (https://github.com/NVIDIA/pyxis/wiki/Setup) suggests setting up the following environment variables:

• PMIX_MCA_ptl=^usock

• PMIX_MCA_psec=none

• PMIX_SYSTEM_TMPDIR=/var/empty

• PMIX_MCA_gds=hash (as configured here)

This configuration change is typically carried out in the software image of the regular nodes. For example, for a node category called default:

[root@basecm11 ~]# category=default
[root@basecm11 ~]# cat <<EOF >> /cm/images/${category}/etc/default/slurmd
PMIX_MCA_ptl=^usock
PMIX_MCA_psec=none
PMIX_SYSTEM_TMPDIR=/var/empty
PMIX_MCA_gds=hash
EOF
[root@basecm11 ~]# systemctl restart slurmd
[root@basecm11 ~]# cmsh -c "device; imageupdate -c ${category} -w"

7.3.4 Prolog And Epilog Scripts

7.3.4.1 What Prolog And Epilog Scripts Do

The workload manager runs prolog scripts before job execution, and epilog scripts after job execution. The purpose of these scripts can include:

• checking if a node is ready before submitting a job execution that may use it

• preparing a node in some way to handle the job execution

• cleaning up resources after job execution has ended.

The administrator can run custom prolog or epilog scripts for the queues from CMDaemon for LSF, by setting such scripts in the Base View or cmsh front ends.

Example

[basecm11->wlm[lsf]->jobqueue]% use normal
[basecm11->wlm[lsf]->jobqueue[normal]]% show | grep -i epilog
Prolog/Epilog user               root
Epilog
Host epilog                      /cm/local/apps/cmd/scripts/epilog

For Slurm, there are global prolog and epilog scripts, but editing them is not recommended. Indeed, in order to discourage editing them, the scripts cannot be set using the cluster manager front ends. Instead the scripts must be placed by the administrator in the software image, and the relevant nodes updated from the image.

7.3.4.2 Detailed Workings Of Prolog And Epilog Scripts

Even though it is not recommended, some administrators may nonetheless wish to link and edit the scripts directly for their own needs, outside of the Base View or cmsh front ends. A more detailed explanation of how the prolog scripts work therefore follows:

When a workload manager is configured using cm-wlm-setup or using the Base View setup wizard, then the workload manager is configured to run the generic prolog located in /cm/local/apps/cmd/scripts/prolog, and the generic epilog located in /cm/local/apps/cmd/scripts/epilog. The generic prolog and epilog scripts call a sequence of scripts for a particular workload manager in special directories. The directories have paths in the format:

1. /cm/local/apps/ <workload manager>/var/prologs/

2. /cm/local/apps/ <workload manager>/var/epilogs/

In these directories, scripts are stored with names that have suffixes and prefixes associated with them that make them run in special ways, as follows:

• suffixes used in the prolog/epilog directory:

  • -prejob script runs prior to all jobs

• prefixes used in the prolog/epilog directory:

  • 00- to

  • 99-

  Number prefixes determine the order of script execution, with scripts with a lower number running earlier.

The script names can therefore look like:

Example

• 01-prolog-prejob

• 10-prolog-prejob

Return values for the prolog/epilog scripts have these meanings:

• 0: the next script in the directory is run.

• A non-zero return value: no further scripts are executed from the prolog/epilog directory.

Often, the script in a prolog/epilog directory is not a real script but a symlink, with the symlink going to a real file located in a different directory. The general script is then able to take care of what is expected of the symlink. The name of the symlink, and destination file, usually hints at what the script is expected to do.

For example, if any health checks are marked to run as prejob checks during cm-wlm-setup configuration, then Slurm uses the symlink 01-prolog-prejob within the prolog directory /cm/local/apps/ <workload manager>/var/prologs/. The symlink links to the script /cm/local/apps/cmd/scripts/prolog-prejob. In this case, the script is expected to run prior to the job.

Example

[root@basecm11 apps]# pwd
/cm/local/apps
[root@basecm11 apps]# ls -l *slurm*/var/prologs/
slurm/var/prologs/:
total 0
lrwxrwxrwx 1 root root ... 01-prolog-prejob -> /cm/local/apps/cmd/scripts/prolog-prejob

Epilog scripts (which run after a job run) have the location /cm/local/apps/ <workload manager>/var/epilogs/. Epilog script names follow the same execution sequence pattern as prolog script names.

It should be noted that the 01-prolog-prejob symlink is created and removed by BCM on each compute node where prejob is enabled in the workload manager entity. Each such entity provides a Enable Prejob parameter that affects the symlink existence:

Example

[head->wlm[slurm]]% get enableprejob
yes
[head->wlm[slurm]]%

This parameter is set to yes by cm-wlm-setup when at least one health check is selected as a prejob one. If any healthcheck was configured as a prejob check before cm-wlm-setup execution, and the administrator had a checkmark for that health check, then the prejob is considered enabled.

7.3.4.3 Workload Manager Configuration For Prolog And Epilog Scripts

BCM configures generic prologs and epilogs during workload manager setup with cm-wlm-setup. The administrator can configure prologs and epilogs using appropriate parameters in the configuration of the workload managers, by creating the symlinks in the local prologs and epilogs directories.

Generic prologs and epilogs are configured by default to run on job compute nodes (one run per each node per job) for Slurm and LSF.

The following parameters for prologs and epilogs can be configured with cmsh or Base View:

• Slurm

  • Prolog Slurmctld: the fully qualified path of a program to execute before granting a new job allocation. The program is executed on the same node where the slurmserver role is assigned. The parth corresponds to the PrologSlurmctld parameter in slurm.conf.

  • Epilog Slurmctld: the fully qualified path of a program to execute upon termination of a job allocation. The program is executed on the same node where the slurmserver role is assigned. Corresponds with the EpilogSlurmctld parameter in slurm.conf.

  • Prolog: the fully qualified path of a program to execute on job compute nodes before granting a new job or step allocation. The program corresponds to the Prolog parameter, and by default points to the generic prolog. This prolog runs on every node of the job if the Prolog flags parameter contains the flag Alloc (the default value), otherwise it is executed only on the first node of the job.

  • Epilog: the fully qualified path of a program to execute on job compute nodes when the job allocation is released.

• LSF

  • Prolog: the fully qualified path of a program to execute on the LSF server node on job allocation. As an LSF queue parameter, it corresponds to the PRE_EXEC parameter in lsb.queues.

  • Epilog: the fully qualified path of a program to execute on the LSF server node on job allocation release. As an LSF queue parameter, it corresponds to the POST_EXEC parameter in lsb.queues

  • Host prolog: the fully qualified path of a program to execute on each node of a job before the job is started. Corresponds to the HOST_PRE_EXEC parameter in lsb.queues. By default it is configured to run the generic prolog

  • Host epilog: the fully qualified path of a program to execute on each node of a job after the job is finished. Corresponds to the HOST_POST_EXEC parameter in lsb.queues. By default it is configured to run the generic epilog.

7.4 Enabling, Disabling, And Monitoring Workload Managers

7.4.0.1 Enabling And Disabling A WLM

A WLM can be disabled for all nodes with cm-wlm-setup. Disabling the WLM means the workload management services are stopped by removing roles, and removing the WLM cluster object.

Alternatively, a WLM can be enabled or disabled by the administrator using role addition and role removal with Base View or cmsh. This is described further on in this section.

Multiple WLM instances of the same type: Versions of NVIDIA Base Command Manager prior to 9.0 already had the ability to have different workload managers run at the same time. However, NVIDIA Base Command Manager version 9.0 introduced the additional ability to run many workload managers of the same kind at the same time.

Example

Two WLM instances, Slurm and LSF, are already running at the same time in the cluster, with each WLM assigned to one category. Then, BCM can start up a third WLM instance, such as another Slurm WLM instance. These WLM instances are alternatively called WLM clusters, because they effectively allow one cluster to function as many separate clusters as far as running WLMs is concerned.

From the Base View or cmsh point of view a WLM consists of

• a WLM server, usually on the head node

• WLM clients, usually on the compute nodes

For the administrator, enabling or disabling the servers or clients is then simply a matter of assigning or unassigning a particular WLM server or client role on the head or compute nodes, as deemed appropriate.

The administrator typically also sets up an appropriate WLM environment module (slurm, lsf), so that it is loaded up for the end user (section 2.2.3).

7.4.1 Enabling And Disabling A WLM With Base View

A particular WLM package may be installed, but the WLM may not be enabled. This can happen, for example, if disabling a WLM that was previously enabled.

If a WLM instance exists, then the WLM client, submission, and server roles can be enabled or disabled from Base View by assigning or removing the appropriate roles to nodes, categories, or configuration overlays. Within the role, the properties of the WLM may be further configured by setting options.

7.4.1.1 Workload Manager Role Assignment To An Individual Node With Base View

Workload Manager Server The following roles are WLM roles that can be assigned to a node:

• server

• submit

• accounting (for the Slurm WLM only, to configure and run the slurmdbd service)

• client

Typically these roles are assigned using configuration overlays.

For example, if the default Slurm configuration is carried out with cm-wlm-setup, then the Slurm server role is assigned to the head node, basecm11, which can be viewed using the navigation path:

Figure 7.16: Workload management role assignment on a head node

Overriding the configuration overlay assignment is possible, at node or category level. For example at node level the role can be defined by assignment at node level, using the navigation path:

This allows parameters to be changed within the role. For example, for the SlurmServerRole the Slurmctld start policy can then be modified at node level.

Workload manager options that are not regarded as server role options can be found under the workload manager object. For Slurm this is in the navigation path:

For example, for a Slurm instance in the Workload Management Clusters pane, a builtin or backfill option can be set for the Scheduler parameter (figure 7.17).

Figure 7.17: Workload management role assignment options on a head node

To have the server start up on non-head nodes (but not for a head node), the imageupdate command (section 5.6.2) can be run. The workload manager server process and any associated schedulers then automatically start up.

Workload Manager Client Similarly, the workload manager client process can be enabled on a node or head node by having the workload manager client role assigned to it at a node level. Some basic options can be set for the client role.

Saving the role, and then running imageupdate (section 5.6.2), automatically starts up the client process with the options chosen, and managed by CMDaemon.

7.4.1.2 Workload Manager Role Assignment To A Category With Base View

It is true that workload manager role assignment can be done as described in the preceding text for individual non-head nodes. However it is usually more efficient to assign roles using categories or configuration overlays, due to the large number of compute nodes in typical clusters.

For example, the case can be considered of all physical on-premises non-head nodes. By default these are in the default category. This means that, by default, roles in the category are automatically assigned to all those non-head nodes, unless, as an exception, an individual node configuration overrides the category setting and uses a role setting instead at node level.

The workload manager roles for the category default can be viewed by using the navigation path: Grouping > Categories[default] > Settings > Roles

If the role has been assigned at category level, then clicking through to the subpane for a role allows role options to be edited and saved.

However,the default configuration using cm-wlm-setup assigns the role at configuration overlay level, which means that the role options (role settings) can

• either be configured using the navigation path from the configuration overlay level:

  Grouping > Configuration Overlays[configuration overlay role] > Settings > Roles > role under configuration overlay role > Role settings

• or they can be configured by adding a role at category level (or node level), with the +Add option. For example, for category level using the default category:

  Grouping > Categories[default] > Settings > Roles > + Add >category-level role

For compute nodes, the role assigned is usually a workload manager client. If the assigned role is that of a workload manager client, then the node with that role can have queues, GPUs, and other parameters specified for it.

For example, queues can then be assigned using the navigation path:
HPC > WLM Management Clusters[Slurm instance ] > Job Queues while GPUs, if using Slurm as the workload manager with default settings, with a default cm-wlm-setup installation, have a configuration overlay slurm-client-gpu that is specified using the navigation path:

The workload manager server role can also be assigned to a non-head node. For example, a Slurm server role can be taken on by a non-head node. This is the equivalent to the --server-nodes option of cm-wlm-setup.

Saving the roles with their options and then running imageupdate (section 5.6.2) automatically starts up the newly-configured workload manager.

7.4.1.3 Workload Manager Role Options With Base View

Each workload manager client role (sometimes loosely called a compute node role) has options that can be set for GPUs, Queues, and Slots. Generally, the value that is set for Slots is the number of jobs expected to run on a node simultaneously. This number can, for example, be set to the number of threads. Threads (virtual cores) in the x86_64 architecture are provided by Intel’s hyper-threading (HT), or by AMD’s simultaneous multithreading (SMT).

The physical CPU, the cores on the CPU, and the threads of a core (HT, SMT) should not be confused with each other. They are distinct concepts, and can all have different values.

• Slots, in a workload manager, corresponds in BCM to:

  • the CPUs setting (a NodeName parameter) in Slurm’s slurm.conf

  • the -n flag, or the total number of processor slots requested, in IBM Spectrum LSF

  • the nproc (or ncpus) setting in PBS,

  In LSF setting the number of slots for the client role to 0 means that the client node does not run jobs on itself, but becomes a submit host, which means it is able to forward jobs to other client nodes.

  The default value for Slots is AUTO, which means that the value for Slots is auto-detected. The parameter may be alternatively be set to a non-negative number. Each WLM has a different implementation on how this is done. For instance, for Slurm, BCM uses Slurm’s own auto-detection implementation.

• Queues with a specified name are available in their associated role after they are created. The creation of queues is described in sections 7.6.2 (using Base View) and 7.7.3 (using cmsh).

All server roles also provide the option to enable or disable the External Server setting. Enabling that means that the server is no longer managed by BCM, but provided by an external device.

The cmsh equivalent of enabling an external server is described here.

7.4.2 Enabling And Disabling A Workload Manager With cmsh

A particular workload manager package may be set up, but not enabled. This can happen, for example, if no WLM server or WLM client role has been assigned.

If a WLM instance exists, then the WLM client, server, or submit roles can be enabled from cmsh by assigning it from within the roles submode. Within the assigned role, the properties of the WLM may be further configured by setting options.

7.4.2.1 Workload Manager Role Assignment To A Configuration Overlay With cmsh

In cmsh, workload manager role assignment to a configuration overlay (section 2.1.5) can be done using configurationoverlay mode. By default cm-wlm-setup run as a TUI session creates some configuration overlays with suggestive names, and assigns roles to the configuration overlays according to what the names suggest. Thus, for example, with the cm-wlm-setup TUI session used to carry out an express setup for Slurm, the configuration overlays that get created are the following:

Example

[basecm11->configurationoverlay]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting     500        yes                                              slurmaccounting
slurm-client         500        no                              default          slurmclient
slurm-server         500        yes                                              slurmserver
slurm-submit         500        no                              default          slurmsubmit
wlm-headnode-submit  600        yes                                              slurmsubmit

Nodes in the default category can take on the slurmclient or slurmsubmit role by setting the nodes for the role using the associated configuration overlays slurm-client or slurm-submit.

The wlm-headnode-submit configuration overlay is a special overlay. It is applied only to the head node, and is shared among all installed workload managers. Setting this overlay means that the head node, by default, has a submit role for a given workload manager.

Example

[basecm11->configurationoverlay]% use slurm-client
[basecm11->configurationoverlay[slurm-client]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             slurm-client
Revision
All head nodes                   no
Priority                         500
Nodes
Categories                       default
Roles                            slurmclient
Customizations                   <0 in submode>
[basecm11->configurationoverlay[slurm-client]]% set nodes
node001  node002  node003  basecm11
[basecm11->configurationoverlay[slurm-client]]% set nodes node001..node002
[basecm11->configurationoverlay*[slurm-client*]]% commit
[basecm11->configurationoverlay[slurm-client]]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting     500        yes                                              slurmaccounting
slurm-client         500        no             node001,node002  default          slurmclient
slurm-server         500        yes                                              slurmserver
slurm-submit         500        no                              default          slurmsubmit
wlm-headnode-submit  600        yes                                              slurmsubmit
[basecm11->configurationoverlay[slurm-client]]%

All the head nodes can also be made to take on the configuration overlay role by setting its All head nodes value to yes. The union set of All head nodes with Nodes is the set of nodes to which the role is applied for that configuration overlay.

Values for the parameters in a role, such as the slurmclient role, can be set within the configuration overlay:

Example

[basecm11->configurationoverlay[slurm-client]]% roles
[basecm11->configurationoverlay[slurm-client]->roles]% use slurmclient
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% show
Parameter                          Value
---------------------------------- -------------------------------------
Name                               slurmclient
Revision
Type                               SlurmClientRole
Add services                       yes
WLM cluster                        slurm
Slots                              0
All queues                         no
Queues                             defq
Features
Sockets                            0
Cores per socket                   0
Threads per core                   0
Boards                             0
Sockets per board                  0
Real memory                        0B
Node address
Weight                             0
Port                               0
Tmp disk                           0
Reason
CPU spec list
Core spec count                    0
Mem spec limit                     0B
GPU auto detect                    BCM
Node customizations                <0 in submode>
Generic resources                  <0 in submode>
Cpu bindings                       None
Slurm hardware probe auto detect   yes
Memory autodetection slack         10.0%
IMEX                               no
Write procs always                 no
Write only Procs                   no
Add to topology                    yes
Nodesets
Power profiles                     <submode>
Nodeset features
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]%

After the workload manager roles are assigned or unassigned, and after running imageupdate (section 5.6.2) for non-head nodes, the associated workload manager services automatically start up or stop as appropriate.

The configuration overlay role values are inherited by categories and nodes, unless the categories and nodes have their own values set. Thus, for role properties, a value set at node level overrides values set at category level, and a value set at configuration overlay level overrides a value set at category level. This is typical of how properties of objects are inherited in BCM levels.

7.4.2.2 Workload Manager Role Assignment To A Category With cmsh

In cmsh, workload manager role assignment to a node category can be done using category mode, using the category name, assigning a role from the roles submode, setting the WLM instance for that role, and committing the modified role:

Example

[root@basecm11 ~]# cmsh
[basecm11]% category
[basecm11->category]% use default
[basecm11->category[default]]% roles
[basecm11->category[default]->roles]% assign slurmclient
[basecm11->category[default]->roles*[slurmclient*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm11     basecm11,node001 node001,node002
[basecm11->category[default]->roles*[slurmclient*]]% set wlmcluster slurm1
[basecm11->category[default]->roles*[slurmclient*]]% commit

Settings that are assigned in the slurmclient role of the category overrule the slurmclient role configuration overlay settings.

The role assignment at category level requires the value for a WLM instance to be specified for wlmcluster before the commit command is successful.

7.4.2.3 Workload Manager Role Assignment To An Individual Node With cmsh

In cmsh, assigning a workload manager role to a head node can be done in device mode. This can be done by using the head node name as the device, assigning the workload manager role to the device, setting the WLM instance value to the role within the role submode, and committing the modified role.

Example

[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% use basecm11
[basecm11->device[basecm11]]% roles
[basecm11->device[basecm11]->roles]% assign slurmserver
[basecm11->category[default]->roles*[slurmserver*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm11     basecm11,node001 node001,node002
[basecm11->category[default]->roles*[slurmserver*]]% set wlmcluster slurm1
[basecm11->device*[basecm11*]->roles*[slurmserver*]]% commit
[basecm11->device[basecm11]->roles[slurmserver]]%

For regular nodes, role assignment is done using device mode, using the node name. Th node name is assigned the workload manager role, the WLM instance value is set for that role in the role submode, and the modified role is committed.

Example

[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% use node001
[basecm11->device[node001]]% roles
[basecm11->device[node001]->roles]% assign slurmclient
[basecm11->device[node001]->roles*[slurmclient*]]% wlm list
Type    Name (key)            Server nodes Submit nodes     Client nodes
------- --------------------- ------------ ---------------- ----------------
slurm   slurm1                basecm11     basecm11,node001 node001,node002
[basecm11->device[node001]->roles*[slurmclient*]]% set wlmcluster slurm1
[basecm11->device[node001]->roles*[slurmclient*]]% commit
[basecm11->device[node001]->roles[slurmclient]]%

The role assignment at node level requires the value for a WLM instance to be specified for wlmcluster before the commit command is successful.

Role assignment values set in device mode have precedence over any role assignment values set in category mode for that node. This means, for example, that if a node is originally in a node category with a slurmclient role and queues set, then when the node is assigned a slurmclient role from device mode, its queue properties are empty by default.

7.4.2.4 Setting Options For Workload Manager Settings With cmsh

In the preceding text, it is explained how the workload manager client or server is assigned a role (such as slurmclient or slurmserver) within the roles submode. It is done from within a main mode of cmsh. The main modes from which role assignment can be done are: configurationoverlay, category or device.

Options for workload managers in general: Whatever main mode is used, the workload manager options for a role can then be set with the usual object commands introduced in section 2.5.3.

• WLM client options: For example, the configuration options of a WLM client, such as the Slurm client, can be seen by using the show command for the role.

  Here it can be seen in the slurmclient role, provided by the default configuration overlay slurm-client:

  Example

  [basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% show
  Parameter                          Value
  ---------------------------------- ------------------------------------------------
  Name                               slurmclient
  Revision
  Type                               SlurmClientRole
  Add services                       yes
  WLM cluster                        slurm
  Slots                              0
  All queues                         no
  Queues                             defq
  Features
  Sockets                            0
  Cores per socket                   0
  Threads per core                   0
  Boards                             0
  Sockets per board                  0
  Real memory                        0B
  Node address
  Weight                             0
  Port                               0
  Tmp disk                           0
  Reason
  CPU spec list
  Core spec count                    0
  Mem spec limit                     0B
  GPU auto detect                    BCM
  Node customizations                <0 in submode>
  Generic resources                  <0 in submode>
  Cpu bindings                       None
  Slurm hardware probe auto detect   yes
  Memory autodetection slack         10.0%
  IMEX                               no
  Write procs always                 no
  Write only Procs                   no
  Add to topology                    yes
  Nodesets
  Power profiles                     <submode>
  Nodeset features

  The Slots option can be set in the role

  Example

  [basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% set slots 2
  [basecm11->configurationoverlay*[slurm-client*]->roles*[slurmclient*]]% commit
  [basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]%

• WLM server options: Similarly, WLM server options can be managed from an assigned server role. For Slurm, the slurmserver role for a device shows:

  Example

  [basecm11->configurationoverlay[slurm-server]->roles[slurmserver]]% show
  Parameter                        Value
  -------------------------------- ------------------------------------------------
  Name                             slurmserver
  Revision
  Type                             SlurmServerRole
  Add services                     yes
  WLM cluster                      slurm
  External server                  no
  Slurmctld start policy           ALWAYS

Option to set an external workload manager: A workload manager can be set to run as an external server from its slurmserver role, within a configuration overlay or within device mode :

Example

[basecm11->device[basecm11]->roles[slurmserver]]% set externalserver on
[basecm11->device[basecm11]->roles[slurmserver*]]% commit

For convenience, setting the external server on the head node is recommended.

The Base View equivalent of configuring externalserver is described here.

7.4.3 Monitoring The State Of Workload Manager Services

By default, the workload manager services are monitored. BCM attempts to restart the services using the service tools (section 3.15), unless the role for that workload manager service is disabled, or the service has been stopped.

Workload manager roles and corresponding services can be disabled using cm-wlm-setup (section 7.3), Base View role configuration (section 7.4.1), or cmsh role configuration (section 7.4.2).

The daemon service states can be viewed for each node using the shell, cmsh, or Base View (section 3.15).

Queue submission and scheduling daemons normally run on the head node.

7.4.3.1 Viewing Services In Base View

From Base View services are viewable using the navigation path to the services running on the node. For example, on a head node basecm11, the navigation path:

leads to a display similar to figure 7.18:

Figure 7.18: Services seen on head node in Base View

For a regular node, a similar navigation path for node001, for example, is:

and leads to a display similar to figure 7.19:

Figure 7.19: Services seen on regular node in Base View

7.4.3.2 Viewing Roles In Base View

Services are often set with BCM roles (section 2.1.5). The navigation path:
Devices > All Devices > basecm11 > Settings > Roles
shows the list of roles. Figure 7.20 shows some of the Slurm roles that result in Slurm services running on the head node:

Figure 7.20: Roles seen on head node in Base View

Similarly, the navigation path:
Devices > All Devices > node001 > Settings > Roles
shows the list of roles for a regular node. Figure 7.21 shows some of the Slurm roles that result in Slurm services running on the regular nodes:

Figure 7.21: Roles seen on regular node in Base View

The roles seen in these figures are from the defaults that cm-wlm-setup provides in an express setup.

For regular nodes, the inheritance of roles from category level or configuration overlay level is indicated by the values in the parentheses alongside the role name. Thus, in figure 7.21, the slurmsubmit and slurmclient roles are decided by the default setting from the configuration overlay.

The assignment of roles can be varied to taste for WLMs. This allows WLM services to run on the head node or on the regular nodes.

7.4.3.3 Viewing Services And Roles In cmsh

From cmsh the services states are viewable from within device mode, using the services command. One-liners from the shell to illustrate this are (output elided):

Example

[root@basecm11 ~]# cmsh -c "device services node001; status"
Service      Status
------------ -----------
...
munge        [   UP   ]
named        [   UP   ]
nslcd        [   UP   ]
...
slurmd       [   UP   ]
[root@basecm11 ~]# cmsh -c "device services basecm11; status"
Service      Status
------------ -----------
...
munge        [   UP   ]
...
slurmctld    [   UP   ]
slurmdbd     [   UP   ]

Roles can be viewed from within the main modes of configurationoverlay, category, or device. One-liners to view these are:

Example

[root@basecm11 ~]# cmsh -c "configurationoverlay; list"
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-client         500        no                              default          slurmclient
slurm-server         500        yes                                              slurmserver
wlm-headnode-submit  600        yes                                              slurmsubmit
[root@basecm11 ~]# cmsh -c "category; use default; roles; list -p"
Name (key)
----------------------------
[overlay:slurm-client:500] slurmclient
[root@basecm11 ~]# cmsh -c "device use node001; roles; list -p"
Name (key)
----------------------------------------
[overlay:slurm-client:500] slurmclient
[root@basecm11 ~]# cmsh -c "device use basecm11; roles; list -p"
Name (key)
----------------------------------------
[750] backup
[750] boot
[750] firewall
[750] headnode
[750] monitoring
[750] provisioning
[750] storage
[overlay:slurm-server:500] slurmserver

The -p|--priority option displays of the list command the priority setting for the roles.

7.5 Configuring And Running Individual Workload Managers

BCM deals with the various choices of workload managers in as generic a way as possible. This means that not all features of a particular workload manager can be controlled, so that fine-tuning must be done through the workload manager configuration files. Workload manager configuration files that are controlled by BCM should normally not be changed directly because BCM overwrites them. However, overwriting by CMDaemon is prevented on setting the directive:

FreezeChangesTo <workload manager>Config = <true|false>

in cmd.conf (Appendix C), where <workload manager> takes the value of Slurm or LSF, as appropriate. The value of the directive defaults to false.

A list of configuration files that are changed by CMDaemon, the items changed, and the events causing such a change are listed in Appendix H.

A very short guide to some specific workload manager commands that can be used outside of the NVIDIA Base Command Manager 11 system is given in Appendix F.

7.5.1 Configuring And Running Slurm

7.5.1.1 Slurm Packages

At the time of writing of this section (July 2026), BCM is integrated with Slurm packages for Slurm versions 25.05, 25.11, and 26.05. Slurm version 25.11 is installed by default.

For Slurm version 25.05, the following packages are available from the BCM repositories for Ubuntu 24.04:

• slurm25.05: Simple Linux Utility for Resource Management, Slurm Workload Management.

• slurm25.05-contribs: contributed utilities for Slurm

• slurm25.05-devel: Development package for Slurm. Includes the header files and static libraries for the SLURM API.

• slurm25.05-example-configs: Example configuration files for Slurm.

• slurm25.05-libpmi: Slurm’s implementation of the pmi libraries.

• slurm25.05-openlava: OpenLava/LSF wrappers for transition from OpenLava/LSF to Slurm.

• slurm25.05-pam: PAM module for restricting access to compute nodes via Slurm.

• slurm25.05-perlapi: Perl API to Slurm.

• slurm25.05-prs: Slurm PRS plugin.

• slurm25.05-sackd: Slurm authentication daemon. Used on login nodes that are not running slurmd daemons to allow authentication to the cluster.

• slurm25.05-slurmctld: Slurm control daemon.

• slurm25.05-slurmd: Slurm compute node daemon.

• slurm25.05-slurmdbd: Slurm database daemon.

• slurm25.05-slurmrestd: Slurm REST API translator.

• slurm25.05-torque: Torque/PBS wrappers for transition from Torque/PBS to Slurm.

For Slurm version 25.11, the value of 25.05 is simply replaced by 25.11 in the preceding list of packages.

Similarly, for Slurm version 26.05, the value of 25.05 is simply replaced by 26.05 in the preceding list of packages.

The distribution version of Slurm (package: slurm) is not integrated with BCM and conflicts with the preceding packages. It should not be used.

From BCM 11.34, BCM Slurm packages include the acct_gather_energy/ipmi plugin alongside the existing energy plugins (gpu, ibmaem, pm_counters, and rapl). This allows administrators to set AcctGatherEnergyType=acct_gather_energy/ipmi in slurm.conf to collect per-node power and energy data from IPMI-based baseboard sensors for job accounting and efficiency reporting. Configuration details can be found in the SchedMD documentation at https://slurm.schedmd.com/slurm.conf.html#OPT_AcctGatherEnergyType.

Important updates from upstream are patched into the BCM repositories. If updating Slurm packages, all the Slurm packages should be updated to the same version, on the compute nodes as well as on the scheduling node.

7.5.1.2 Building Custom Slurm Packages

The BCM Slurm packages can be rebuilt from source by the administrator, on the BCM cluster itself. Rebuilding allows important Slurm patches to be applied sooner than the standard BCM release cycle provides them, and allows the Slurm binaries to be optimized for a particular CPU or hardware configuration.

The build script, along with the RPM and DEB spec files that NVIDIA uses to build the packages, is shipped in a slurm<version>-builder package, for example slurm25.11-builder or slurm26.05-builder. Because the spec files are bundled in the package, no separate download of them is needed. The builder package is available from the BCM repositories, but is not installed by default. Installing it places the build script in /cm/local/apps/slurm/current/builder/, which is not in the standard $PATH.

Only Slurm version 25.11 and higher can be built with this script. The earlier Slurm version that BCM also ships, 25.05, cannot be built with it.

The script can be run on BCM clusters running RHEL 9, RHEL 10, Ubuntu 22.04, Ubuntu 24.04, or Ubuntu 26.04. SLES is not supported.

The script takes a required --version option, in X.Y.Z format:

[root@basecm11 ~]# cd /cm/local/apps/slurm/current/builder
[root@basecm11 builder]# ./create-packages.sh --version 25.11.5

The script downloads the Slurm source tarball and loads the modulefiles that are needed for the build. When the build finishes, the resulting packages are saved to /tmp/slurm<version>-<revision>-<release-tag>/, where <revision> is a build counter that increases with each run, and <release-tag> encodes the BCM version and a checksum of the source tarball.

Validating the build environment:

Adding the --validate option carries out a dry run that checks the prerequisites without building anything. Each check is reported as [PASS] or [FAIL], and the script exits with 0 if every check passed, or 1 if any check failed:

[root@basecm11 builder]# ./create-packages.sh --version 25.11.5 --validate
Slurm version: 25.11.5                                                   [PASS]
Distribution                                                             [PASS]
Host tools                                                               [PASS]
Build tools                                                              [PASS]
Build dependencies                                                       [PASS]
Modules                                                                  [PASS]
Disk space                                                               [PASS]
All checks passed.

The checks cover the Slurm version, the distribution and its major version, the host tools (wget, tar, and jq), the build tool (dpkg-buildpackage on Ubuntu, or rpmbuild on RHEL), the build dependencies, the availability of the required modules, and whether at least 100MB of disk space is free.

Building inside a software image:

Packages that are intended for a particular software image should be built inside that image, so that the build uses the same operating system, libraries, and modules that the image provides. The image should be cloned first, using cmsh or Base View, and the build then run inside the clone with cm-chroot:

[root@basecm11 ~]# cm-chroot /cm/images/slurm-builder \
/cm/local/apps/slurm/current/builder/create-packages.sh --version 25.11.5

The build dependencies are installed automatically, driven by a dependencies.yaml file that ships with the builder package, whether the script is run on the host or inside a cm-chroot image. No manual dependency installation is required beforehand.

Building from a different source tarball:

A source tarball other than the one that the script downloads from upstream can be used by passing its path with --archive:

[root@basecm11 builder]# ./create-packages.sh --version 25.11.5 \
--archive /root/slurm-25.11.5.tar.bz2

The contents of the tarball must be inside a directory that matches the version being built. So, for Slurm 25.11.5, the tarball contents must be inside a directory called slurm-25.11.5, in the same way as the upstream Slurm tarballs.

The packages that are built are installed in the same way as for a Slurm upgrade, except that the packages are taken from the local directory instead of from the package repositories.

7.5.1.3 Upgrading From Earlier Slurm Versions To slurm25.11

Upgrading between major versions of Slurm is generally possible. At the time of writing (December 2025) of this section, the upstream Slurm policy allows upgrades from three previous major release versions since version 24.11. Details on the upgrade compatibility can be seen at the SchedMD site at https://slurm.schedmd.com/upgrades.html#compatibility_window.

It is a good idea to upgrade one version at a time, rather than jumping too many versions ahead, which can require a full wipe of the Slurm configuration.

If Slurm is using Pyxis (section 7.3.3), then upgrading the Slurm version means that Pyxis needs to be reinstalled using cm-wlm-setup. The reinstallation run for Pyxis compiles Pyxis and recreates a plugin directory for the new Slurm version under /cm/local/apps/slurm/.

An upgrade from one major version of Slurm to another can be carried out according to the following example, which is for an update from major version 25.05 to version 25.11, and avoids total reconfiguration of the Slurm configuration:

• It is recommended that no jobs are running. Draining nodes (section 7.7.4) is one way to arrange this over time. No new jobs run on a drained node, but old ones are allowed to finish.

• When all running jobs are finished, then Slurm server services—slurmctld and slurmdbd—should be stopped using cmsh or Base View (section 3.15.2):

  Example

  [basecm11->device[basecm11]->services]% stop slurmctld
  ...Service slurmctld was stopped
  [basecm11->device[basecm11]->services]% stop slurmdbd
  ...Service slurmdbd was stopped

• The old Slurm packages should then be removed. There can be only one version of Slurm at a time, so there will be a package installation conflict if a new version is installed while an old one is still there.

  Removal can be carried out on RHEL-based systems with, for example:

  [root@basecm11 ~]# yum remove slurm25.05*

  In Ubuntu the equivalent is carried out using apt remove slurm25.05*.

  The old packages must also be removed from each software image that uses it:

  Example

  [root@basecm11 ~]# cm-chroot-sw-img /cm/images/<software image>
  ...
          [root@<software image> /]# yum remove slurm25.05*
  ...removal takes place...
  [root@<software image> /]# exit

  The cm-chroot-sw-img wrapper utility is discussed in section 9.4.1.

• The new packages can then be installed. For installation onto the RHEL head node, the installation might be carried out as follows:

  [root@basecm11 ~]# yum install slurm25.11 slurm25.11-contribs slurm25.11-devel \
  slurm25.11-libpmi slurm25.11-openlava slurm25.11-pam slurm25.11-perlapi \
  slurm25.11-prs slurm25.11-sackd slurm25.11-slurmctld slurm25.11-slurmd \
  slurm25.11-slurmdbd slurm25.11-slurmrestd slurm25.11-torque

  The same packages as on the head node are installed by default in the software image when installing with cm-wlm-setup in versions of Slurm from version 24 onward. The installation is carried out manually for the upgrade in each software image with:

  Example

  [root@basecm11 ~]# cm-chroot-sw-img /cm/images/<software image>
  ...
  [root@<software image> /]# yum install slurm25.11-slurmd slurm25.11-contribs slurm25.11-devel \
  slurm25.11-libpmi slurm25.11-openlava slurm25.11-pam slurm25.11-perlapi \
  slurm25.11-prs slurm25.11-sackd slurm25.11-slurmctld \
  slurm25.11-slurmdbd slurm25.11-slurmrestd slurm25.11-torque
  ...installation takes place
  [root@<software image> /]# exit

• The new Slurm version is then set in cmsh or Base View, in the Slurm WLM cluster configuration:

  Example

  [root@basecm11 ~]# cmsh
  [basecm11]% wlm use slurm
  [basecm11->wlm[slurm]]% set version 25.11; commit

• Slurm server services slurmctld and slurmdbd should then be started again using cmsh or Base View:

  Example

  [basecm11->device[basecm11]->services]% start slurmdbd
  [basecm11->device[basecm11]->services]% start slurmctld

• The nodes can then have their new image placed on them, and the new Slurm configuration can then be taken up. This can be done in the following two ways:

  1. The regular nodes can then be restarted to supply the live nodes with the new image and get the new Slurm configuration running.

  2. Alternatively, the imageupdate command (section 5.6.2) can be run on the live nodes to supply them with the image.

     Running the imageupdate command in dry mode (the default) first is recommended. The synclog command can then be run to check there are no unexpected changes that will take place due to the update. If all is well, then imageupdate’s wet mode flag -w can be used in order to really carry out the task.

     For example, the change can be checked, and then actually carried out, for the image on node001 with:

     [root@basecm11 ~]# cmsh
     [basecm11]% device use node001
     [basecm11->device[node001]]% imageupdate
     Performing dry run (use synclog command to review result, then pass -w to perform real update)...
     ...some messages...
     imageupdate [ COMPLETED ]
     [basecm11->device[node001]]% synclog
     ...rsync dry run output...
     [basecm11->device[node001]]% imageupdate -w
     ...same messages as before, but this time it really happens...
     [basecm11->device[node001]]% commit

     The last commit command triggers the reconfiguration of the file /etc/systemd/system/slurmd.service.d/99-cmd.conf on the node. After a short time—around 30 seconds—the file is regenerated. The slurmd service on the node can then be restarted with:

     [root@basecm11 ~]# ssh node001 "systemctl daemon-reload"
     cmsh -c "device services node001; use slurmd; restart"

7.5.1.4 IMEX

NVIDIA IMEX (Internode Memory Exchange Service) is a secure service that facilitates the mapping of GPU memory over NVLink between the GPUs in an NVLIink domain.

The imexctl command in device mode lists the status and other properties of the nodes.

Example

[basecm11->device]% imexctl
Node                IP            Status   Version     Connection        IP            Status
------------------- ------------- -------- ----------- ----------------- ------------- ----------
dgx-gb200-c1        10.140.2.134  READY    570.172.08  dgx-gb200-m06-c1  10.140.2.134  CONNECTED
dgx-gb200-c1        10.140.2.134  READY    570.172.08  dgx-gb200-m06-c2  10.140.2.135  CONNECTED
dgx-gb200-c1-dpu0   0.0.0.0
dgx-gb200-c1-dpu1   0.0.0.0
dgx-gb200-c2        10.140.2.135  READY    570.172.08  dgx-gb200-m06-c1  10.140.2.134  CONNECTED
dgx-gb200-c2        10.140.2.135  READY    570.172.08  dgx-gb200-m06-c2  10.140.2.135  CONNECTED
basecm11            0.0.0.0       NONE
basecm11-k8s-01     0.0.0.0       NONE
basecm11-k8s-02     0.0.0.0       NONE
basecm11-k8s-03     0.0.0.0       NONE
basecm11-slogin-0   0.0.0.0       NONE
[basecm11->device]%

Pinning IMEX to a specific network (FQDN or IP in nodes_config.cfg):

By default, the autogenerated IMEX peer list in /etc/nvidia-imex/nodes_config.cfg uses the short hostname of each node. IMEX can instead be pinned to a particular network. When a network has the imex parameter enabled, the fully qualified domain name (FQDN) on that network is written to nodes_config.cfg for each peer. For example, for an IMEX network called imexnetwork:

[root@basecm11 ~]# cmsh
[basecm11]% network use imexnetwork
[basecm11->network[imexnetwork]]% set -e imex yes
[basecm11->network*[imexnetwork*]]% commit

The resulting config file uses the FQDN for each node:

root@node001:~# cat /etc/nvidia-imex/nodes_config.cfg
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
node001.eth.cluster
node002.eth.cluster
node003.eth.cluster
# END AUTOGENERATED SECTION   -- DO NOT REMOVE

The peers can be written as IP addresses instead of FQDNs by also enabling the imex ip parameter on the same network:

[root@basecm11 ~]# cmsh
[basecm11]% network use imexnetwork
[basecm11->network[imexnetwork]]% set -e imex yes
[basecm11->network*[imexnetwork*]]% set -e "imex ip" yes
[basecm11->network*[imexnetwork*]]% commit

The resulting config file uses the IP address of each node on that network, with the node name retained as a comment:

root@node001:~# cat /etc/nvidia-imex/nodes_config.cfg
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
# node001
10.141.0.1
# node002
10.141.0.2
# node003
10.141.0.3
# END AUTOGENERATED SECTION   -- DO NOT REMOVE

For Slurm, BCM can enable the IMEX daemon either globally, or per job. Enabling the IMEX daemon globally for Slurm:

If IMEX is set globally, then it means that it is set to run on all nodes at runtime.

The advantage of this configuration is that it is easy to set up, and easy to test to see if it is all working.

A disadvantage is that a user running a job on one node can read memory from a job run by another user on another node. It requires overcoming difficult hurdles to carry out, but it is not impossible. The administrator should therefore weigh up if this seems a significant issue for the cluster being administered.

The IMEX global service can be run by setting the values for the service for the compute nodes. For example, nodes in a dgx-gb200 category can have the service added and set to:

[basecm11->category[dgx-gb200]->services[nvidia-imex]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Revision
Service                          nvidia-imex
Run if                           always
Monitored                        yes
Autostart                        yes
Managed                          yes

Per-rack IMEX configuration can be forcibly created, regardless of domains and partitions, with:

[basecm11->category[dgx-gb200]->services[nvidia-imex]]% set -e rack yes

Enabling and disabling the IMEX daemon per job for Slurm:

Configuring per job means that IMEX runs just before the job starts, and that the service runs only on a node that needs it for the job.

A disadvantage is that verifying that it actually works requires a job run.

Running the IMEX daemon per job has the advantage that one user running a job cannot read the memory of a job run by another user on another node.

To run IMEX per job, any global IMEX setting must first be cleared away. This can be done by first:

• removing the nvidia-imex service configuration entirely from CMDaemon

  Example

  [basecm11->category[dgx-gb200]->services[nvidia-imex]]% remove nvidia-imex; commit

• stopping the service on the compute nodes, for example with pdssh or pdexec, or simply carrying out a reboot

The value of imex in the slurmclient role can then be set:

[root@basecm11 ~]# cmsh
[basecm11]% configurationoverlay roles slurm-client-gpu
[basecm11->configurationoverlay[slurm-client-gpu]->roles]% use slurmclient
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]]% set imex yes
[basecm11->configurationoverlay*[slurm-client-gpu*]->roles*[slurmclient*]]% commit
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]]%

Committing that imex setting configures Slurm prolog and epilog scripts in the backend as follows:

• the prolog script configures the IMEX daemon with the nodes allocated to the GPU job and starts the IMEX daemon on the node

• the epilog cleans up the configuration and stops the IMEX daemon on the node

The TCP port used by IMEX for peer communication is defined by SERVER_PORT in /etc/nvidia-imex/config.cfg.

For running IMEX per job, the port in the file is written in as a random port for that job, in the allowed range of 0 to 65535.

To use an IMEX that is not managed by BCM, the SERVER_PORT value needs to be changed manually for all nodes to suit that other IMEX. This can be done with the help of the pdcp command, which comes with the pdsh package (section 14.1). The pdcp command can copy the config.cfg file from the BCM software image to all nodes that no longer use BCM to run IMEX per job.

7.5.1.5 Workload Power Profile Settings

NVIDIA Blackwell GPUs support workload power profile settings (WPPS). The NVIDIA Mission Control System Administration Guide has more background information on WPPS at:

https://docs.nvidia.com/mission-control/docs/systems-administration-guide/2.0.0/wpps/introduction.html

For WPPS to work as expected, a node that has a job allocated to it must have the job allocated exclusively to it, and oversubscribe (here) should not be set.

Each GPU in a job can have a specific workload power profile set for it. The profile can vary per GPU and per user. The gpuworkloadpowerprofiles command (here) can be used to configure the GPU workload power profiles.

BCM allows the user who is running a Slurm job to change the job power profiles in two ways:

1. Statically: the job prolog script switches the power profiles for all GPUs allocated to the job just before the job starts. That is, the power profile is managed at job submission time.

2. Dynamically: the job process sets power profiles for each allocated GPU during job execution. That is, the power profile is managed during the job runtime.

In both cases, when the job is finished, the epilog script resets the GPU power profiles back to the default.

Workload power profiles settings are a part of the slurmclient role, located within the Power profiles submode. They can be reached from within the configuration overlay, category, or node level:

Example

[root@basecm11 ~]# cmsh
[basecm11]% configurationoverlay roles slurm-client-gpu
[basecm11->configurationoverlay[slurm-client-gpu]->roles]% use slurmclient
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]]% powerprofiles
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]->powerprofiles]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Disabled                         yes
Fail on error                    no
Job keyword                      wpps
Jobs profiles directory          /var/run/nvidia/workload-power-profiles
Debug                            no
Debug log directory              /var/spool/cmd/wlm/wpps
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]->powerprofiles]% set disabled no
[basecm11->configurationoverlay*[slurm-client-gpu*]->roles*[slurmclient*]->powerprofiles*]% commit
[basecm11->configurationoverlay[slurm-client-gpu]->roles[slurmclient]]%

With Base View, the equivalent navigation path to the preceding is:

The Power profiles parameters as shown in cmsh are:

1. Disabled: forbids a job from changing workload power profiles (yes by default)

2. Fail on error: allows prolog and epilog to fail if a workload power profiles change fails for any reason

3. Job keyword: a keyword, wpps, in the JSON object, used by the WLM job, to specify a workload power profiles setting in the comment field.

4. Jobs profiles directory: a top directory which includes job ID directories, and used by the job to change the workload power profiles

5. Debug: setting it to yes enables prolog and epilog debug messages The prolog and epilog then write debug logs to the location set by the Debug log directory parameter,

6. Debug log directory: directory where prolog and epilog create debug log files per job. The logs take the format: /var/spool/cmd/wlm/wpps/ <JOB ID>.log by default. The log files are not cleaned up automatically.

If WPPS is enabled then:

• If a job requires static job profiles management, then special BCM prolog and epilog scripts are enabled in Slurm on compute nodes. The scripts parse the SLURM_JOB_COMMENT environment variable that is passed by Slurm. The scripts search for a JSON object that includes the key wpps. The key used can be changed using the Job keyword parameter. The expected format of the JSON object is:

  {"wpps": {"profiles": PROFILES}}

  Here, PROFILES is either

  • a JSON list of strings (workload profile names or their numbers) or

  • a single string (one workload profile name) or

  • a workload profile number.

  The values that PROFILES can take are:

  • max_p

  • max_q

  • llm_training

  • llm_inference

  On matching, the specified profiles are set for all the GPUs on the node that are allocated for the job when the job starts.

  For example, the following job submission command instructs the prolog script to set the workload profiles to max_p and compute:

  $ sbatch --gres=gpu:b100:8 --comment='{"wpps": {"profiles": ["max_p", "compute"]}}' job.sh

  All allocated GPUs on the node get the specified workload profiles, which means that job allocation must be configured to run on that node as a job that excludes other jobs running on it. If the cluster administrator has set oversubscribe (here) for Slurm, then that takes precedence over exclusivity, and must therefore be taken into account.

• If a job requires dynamic profiles management, then the job process can notify BCM to update the power profiles of specific GPUs during job execution. The prolog creates a directory of the form /var/run/nvidia/workload-power-profiles/<JOB_ID>/ per job. The following files are created within the directory:

  1. username: read-only (for users) file that includes the current job user name. Used by BCM.

  2. uuids: read-only (for users) file that includes a list of GPU UUIDs (one per line) allocated for this job.

  3. profiles: file that can be used by the job process to set new profiles. The file is empty by default. An empty file means that BCM does not modify the profiles. If the job process writes two lines:

                             max_p
                             compute

     then BCM reads this file almost immediately, and applies these two workload profiles to the GPUs whose UUIDs are specified in uuids file.

What users are running in each workload power profile can be viewed with a PromQL query, as explained here.

Management of node workload profiles using gpuworkloadpowerprofiles, by regular users: The user, for example John, can be given access to checking and managing GPU power profiles by running the allowgpuworkloadpowerprofiles command.

Example

[basecm11->user[john]]% set allowgpuworkloadpowerprofiles yes

The gpuworkloadpowerprofiles command can then be run to check the current node profiles:

Example

[basecm11->device]% gpuworkloadpowerprofiles show -n node001
Node         GPU      Profiles
------------ -------- ------------
node001      0        COMPUTE(2)
node001      1        MAX_P(0)
node001      2        MAX_P(0)
node001      3        MAX_P(0)
node001      4        MAX_P(0)
node001      5        MAX_P(0)
node001      6        MAX_P(0)
node001      7        MAX_P(0)
[basecm11->device]%

This may be useful for debug purposes.

The BCM script cm-gpu-workload-power-profiles can then be used to change the profiles as follows:

• The user first gets the GPU UUIDs:

  Example

  root@dgx-gb200-n07-c2:~# nvidia-smi --query-gpu=uuid --format=csv,noheader --id=0,1,2,3
  GPU-1e72bc8d-b967-6031-24bd-c5a08ad090e1
  GPU-3a6ac832-3423-68cc-ed1f-64f2b46d248d
  GPU-f40fabb7-eb9b-0253-4bd5-08b4a12916ea

• The user can then use the script to change the profile. For example, just for the GPUs 1 and 2:

  Example

  root@dgx-gb200-n07-c2:~# /cm/local/apps/cmd/sbin/cm-gpu-workload-power-profiles -p compute -u john \
  3a6ac832-3423-68cc-ed1f-64f2b46d248d f40fabb7-eb9b-0253-4bd5-08b4a12916ea

  The script carries the profile change out with the help of CMDaemon in the back end.

  Slurm’s prolog and epilog actually work with the script in BCM when carrying out dynamic profiles management, so that the user does not need to use the script manually.

Management of external users to allow them to use profiles: External users, that is users managed by an external LDAP and not managed by the BCM LDAP, can use the workload power profile settings if their user name or UID is added to the file:

/cm/local/apps/cmd/etc/allow-users-wpps.conf

The file must be created on the head node, or on both head nodes if the cluster has HA. The file should have its mode set to 0600. The user name or UID should be added on its own line in the file.

7.5.1.6 Advanced Slurm Job Accounting

PromQL queries can be run and then filtered by Slurm job labels with BCM. The labels are taken either from a job comment or from an account name used by the job. After the labels for the job are extracted, the labeled entity is stored in the monitoring data. The administrator can then run PromQL queries, and filter results by label. Examples of such PromQL queries are the job_gpu_wasted or job_gpu_utilization queries (section 12.4.1).

Extract accounting info must be set to yes to enable labels to be extracted from job comments or account names:

Example

[root@basecm11 ~]# cmsh
[basecm11]% wlm use slurm
[basecm11->wlm[slurm]]% accounting
[basecm11->wlm[slurm]->show]% accounting
Parameter                        Value
-------------------------------- ------------------------------------------------
Managed hierarchy
Separator                        _
Job comment labels
Extract accounting info          no
[basecm11->wlm[slurm]->accounting]% set extractaccountinginfo yes
[basecm11->wlm*[slurm*]->accounting*]% commit

In addition, the administrator must configure the job comment label format and the format of Slurm account names, if they exist. Without that configuration, BCM cannot parse the labels.

Job comment labels: A user running the job can specify the label for the job comment in a special format. Such a label could be, for example, a tag to indicate the artificial neural network model, where different groups of jobs use different models. The administrator (after having enabled accounting information extraction) can set a regex for the tag in a Job comment field parameter in accounting submode:

Example

[root@basecm11 ~]# cmsh
[basecm11]% wlm use slurm
[basecm11->wlm[slurm]]% accounting
[basecm11->wlm[slurm]->accounting]% set jobcommentlabels model
[basecm11->wlm*[slurm*]->accounting*]% commit