> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Workload Management For clusters that have many users and a significant load, a workload manager (WLM) system allows a more efficient use of resources to be enforced for all users than if there were no such system in place. This is because without resource management, there is a tendency for each individual user to over-exploit common resources. When a WLM is used, the end user can submit a job to it. This can be done interactively, but it is typically done as a non-interactive batch job. The WLM assigns resources to the job, and checks the current availability as well as checking its estimates of the future availability of the cluster resources that the job is asking for. The WLM then schedules and executes the job based on the assignment criteria that the administrator has set for the WLM system. After the job has finished executing, the job output is delivered back to the user. Among the hardware resources that can be used for a job are GPUs. Installing CUDA software to enable the use of GPUs is described in [section 9 of the *Installation Manual*](/installation/nvidiagpus). Configuring GPU settings for BCM is described in section [3.16.2](/admin/configuring#3162-configuring-gpu-settings) of the *Administrator Manual*. Configuring GPU settings for an individual WLM is described in the section on getting that particular WLM up and running. The details of job submission from a user’s perspective are covered in the *User Manual*. Sections [7.1](#71-workload-managers-choices)–[7.5](#75-configuring-and-running-individual-workload-managers) cover the installation procedure to get a WLM up and running. Sections [7.6](#76-using-base-view-with-workload-management) –[7.7](#77-using-cmsh-with-workload-management) describe how Base View and `cmsh` are used to view and handle jobs, queues and node drainage. Section [7.8](#78-examples-of-workload-management-assignment) shows examples of WLM assignments handled by BCM. Section [7.9](#79-power-saving-with-cm-scale) describes the power saving features of WLMs. Section [7.10](#710-cgroups) describes cgroups, a resources limiter, mostly in the context of WLMs. Section [7.11](#711-custom-node-parameters) describes WLM customizations for settings other than the common settings covered by BCM. ## 7.1 Workload Managers Choices ***Installation of new PBS and GE workload managers is no longer offered by BCM. Legacy setups that are already installed continue to work, and documentation for PBS and GE is therefore still available in the manuals.*** Some WLM packages are installed by default, others require registration from the distributor before installation. During cluster installation, a WLM can be chosen ([figure 3.9 of the *Installation Manual*](/installation/installing#fig-workload)) for setting up. The choices are: * **Slurm**: Available as version 23.02 or 23.11. Slurm is a free (GPL) job scheduler, with commercial support. * **LSF v10.1**: IBM Spectrum LSF (Load Sharing Facility) version 10.1, is a further development of what used to be IBM Platform LSF. * **None**: For clusters that need no HPC job-scheduling. The WLMs in the preceding list can also be chosen and set up later using the `cm-wlm-setup` tool (section [7.3](#73-installation-of-workload-managers)). After installation, if there are no major changes in the WLM for updated versions of the workload managers, then * WLMs that are packaged with BCM (Slurm) can have their packages updated using standard package update commands (`yum update` and similar). The installation and configuration of the WLM from the updated packages is carried out as described later on in this chapter. * WLMs that are installed by picking up software from the vendor (LSF) can be updated by following vendor guidelines. ## 7.2 Forcing Jobs To Run In A Workload Management System Another preliminary step is to consider forcing users to run jobs only within the WLM system. Having jobs run via a WLM is normally a best practice. For convenience, BCM defaults to allowing users to log in via `ssh` to a node, using the authorized keys files stored in each users directory in `/home` (section [2.3.2](/admin/clustermanagement#232-logins-using-ssh)). This allows users to run their processes without restriction, that is, outside the WLM system. For clusters with a significant load this policy results in a sub-optimal use of resources, since such unplanned-for jobs disturb any already-running jobs. Disallowing user logins to nodes, so that users have to run their jobs through the WLM system, means that jobs are then distributed to the nodes only according to the planning of the WLM. If planning is based on sensible assignment criteria, then resources use is optimized—which is the entire aim of a WLM in the first place. ### 7.2.1 Disallowing User Logins To Regular Nodes Via `cmsh` The `usernodelogin` setting of `cmsh` restricts direct user logins from outside the WLM, and is thus one way of preventing the user from using node resources in an unaccountable manner. The `usernodelogin` setting is applicable to node categories only, rather than to individual nodes. In `cmsh` the attribute of `usernodelogin` is set from within `category` mode: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% category use default [basecm10->category[default]]% set usernodelogin onlywhenjob [basecm10->category*[default*]]% commit ``` The attributes for `usernodelogin` are: * `always` (the default): This allows all users to `ssh` directly into a node at any time. * `never`: This allows no user other than root to directly `ssh` into the node. * `onlywhenjob`: This allows the user to `ssh` directly into the node when a job is running on it. It typically also prevents other users from doing a direct `ssh` into the same node during the job run, since typically the WLM is set up so that only one job runs per node. However, an `ssh` session that is already running is not automatically terminated after the job is done. * Some cluster administrators may wish to allow some special user accounts to override the `onlywhenjob` setting, for example for diagnostic purposes. Before giving the details of how to override the setting, some background explanation is probably useful: The `onlywhenjob` setting works with the PAM system, and adds the following line to `/etc/pam.d/sshd` on the regular nodes: ```bash account required pam_bright.so ``` Nodes with the `onlywhenjob` restriction can be configured to allow a particular set of users to access them, despite the restriction, by allowing them with the PAM system, as follows: Within the software image \<*node image*> used by the node, that is under `/cm/images/` \<*node image*>, the administrator can add the set of user accounts to the file `etc/security/pam_bright.d/pam_allow.conf`. This file is installed in the software image with a chroot installation (section [11.4](/admin/softwareimagemanagement#114-managing-a-package-in-a-software-image-and-running-it-on-nodes)) of the `cm-libpam` package. Groups of users can be allowed using the file `etc/security/pam_bright.d/pam_allow_group.conf`. Other adjustments to PAM configuration, such as the number of attempted logins and the associated wait time per login, can be carried by editing the `/etc/security/pam_bright.d/cm-check-alloc.conf` file. The image can then be updated to allow the users, by running the `imageupdate` command in `cmsh` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)), or by clicking the `Update node` option in Base View (section [5.6.3](/admin/provisioning#563-updating-running-nodes-with-base-view-using-the-update-node-option)). ### 7.2.2 Disallowing User Logins To Regular Nodes Via Base View In Base View, user node login access is set via a category setting, for example for the `default` category via the navigation path in figure [7.1](#fig-bright-view-workloadmanagement-user-node-login): `Grouping` > `Categories[default]` > `Edit` > `Settings` > `User node login` ![Disallowing user logins to nodes via Base View](/base-command-manager/product-manuals/_fern-img/2eb7e917dc1d95529ef31a762c358ff3dc4ac10a86925d18c40720e03c2570d2.webp) *Figure 7.1: Disallowing user logins to nodes via Base View* ### 7.2.3 Disallowing Other User Processes Outside Of Workload Manager User Processes Besides disabling user logins, administrators may choose to disable interactive jobs in the WLM as an additional measure to prevent users from starting jobs on other nodes. Administrators may also choose to set up scripts that run after job execution. Such scripts can terminate user processes outside the WLM, as part of a policy, or for general administrative hygiene. These are `Epilog` scripts and are part of the WLM. The WLM documentation has more on configuring these options. ### 7.2.4 High Availability By Workload Managers NVIDIA Base Command Manager uses the existing built-in high availability (HA) functionalities of workload managers as much as possible. A double server HA configuration with the existing built-in HA functionality can be carried out using the `cm-wlm-setup` utility (section [7.3](#73-installation-of-workload-managers)), or using the Base View HA wizard (section [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)). The built-in functionality makes use of the primary WLM server and secondary WLM server, which are placed on separate nodes. The HA configuration aspect (section [17.1.3](/admin/ha#1713-services-in-bcm-ha-setups)) in this case means that a WLM server role is assigned to both the WLM primary and the WLM secondary nodes. These primary WLM and secondary WLM servers can then: * both be on head nodes, * both be on regular (compute) nodes It is not possible to configure a mixed setup—that is with one head node and one regular node—for the WLM servers on an HA setup. If the head nodes are configured with the WLM server roles, then in the cluster entity configuration the `primary server` value is empty. This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node. If two regular nodes are configured with the WLM server roles instead, then in the cluster entity configuration the `primary server` parameter is set to one of the compute nodes. The cluster primary head node and the WLM primary server should not be confused. In particular, the active cluster head node and active workload manager server are not necessarily the same for the case of the Slurm, PBS Professional, or LSF workload managers. For these, if the passive cluster head node becomes an active cluster head node without a crash, then this does not trigger a passive WLM server on the newly active head node to also become an active WLM server. Thus, the WLM server can be active on a passive head node, and vice versa, because the WLM primary server is independent of the cluster primary head node. #### 7.2.4.1 Slurm High Availability And `scontrol takeover` Slurm can speed up takeover by running `scontrol takeover` from the Bash shell. This stops the primary `slurmctld` and triggers the Slurm failover algorithm. However, by default BCM automatically restarts `slurmctld`, so the automatic restart of the `slurmctld` service must be disabled manually. There are also some other related housekeeping tasks that may need to be taken care of. Instead of trying to run an `scontrol takeover` from Bash and then having to work around those kind of issues, BCM can be used to automate the takeover. That is to say, to run `scontrol takeover` cleanly from Bash with HA, the following configuration steps can be carried out: 1. Within CMDaemon, for Slurm, the `takeover` parameter is set to `true`. Disabling the automatic restart of the `slurmctld` service is then unnecessary: ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% set --extra takeover true [basecm10->wlm*[slurm*]]% commit ``` The `takeover` setting can be cleared with: ```bash [basecm10->wlm[slurm]]% clear -e takeover [basecm10->wlm*[slurm*]]% commit ``` 2. If the installation of Slurm with `cm-wlm-setup` was carried out on what was a non-HA cluster at that time, then the `prefailoverscript` parameter must be set within the `failover` submode of `partition` mode: ```bash [basecm10->partition[base]->failover]% set prefailoverscript /cm/local/apps/cmd/scripts/slurm.takeover.sh [basecm10->partition*[base*]->failover*]% commit ``` This ensures that the active head node is taken over by Slurm during a change of active head node Table [7.2.4.1](#7241-slurm-high-availability-and-scontrol-takeover) shows what `slurmctld` does when the `takeover` parameter is set. The detailed behavior per head node depends on whether the head nodes are active or primary:
**Node** **`takeover` value** **Active node?** **Slurm primary node?** **`slurmctld` starts?**
Head001 `no` \* \* yes
Head002 `no` \* \* yes
Head001 `yes` yes yes yes
Head002 `yes` no no yes
Head001 `yes` yes no yes
Head002 `yes` no yes no
Head001 `yes` no yes no
Head002 `yes` yes no yes
Head001 `yes` no no yes
Head002 `yes` yes yes yes
\* any value of `yes` or `no` ## 7.3 Installation Of Workload Managers Normally the administrator selects a WLM to be used during BCM installation ([figure 3.9 of the *Installation Manual*](/installation/installing#fig-workload)). A WLM may however also be added and configured after BCM has been installed, using `cm-wlm-setup`, or the Base View WLM wizard. With most other objects, BCM front ends—`cmsh` and Base View—can be used to create a new object from scratch, or can clone a new object from another existing object. However, the front ends cannot do this for a WLM object. An attempt to create or clone a new WLM object via `cmsh` or Base View is prohibited by the front ends, because there are many pitfalls possible in configuration. A new WLM object, and WLM instance, can therefore only be installed via `cm-wlm-setup`, the Base View WLM wizard, or by selecting a WLM during the initial BCM installation. ### 7.3.1 Running `cm-wlm-setup` In CLI Mode The recommended way to run the `cm-wlm-setup` utility is without options or arguments, in which case a TUI dialog starts up. A TUI session run with `cm-wlm-setup` is covered in section [7.3.2](#732-running-cm-wlm-setup-as-a-tui). However, the `cm-wlm-setup` utility can alternatively be used in a non-GUI, command-line, mode, with options and arguments. The utility has the following usage: ```bash [root@basecm10 ~]# cm-wlm-setup -h usage: Workload manager setup cm-wlm-setup [-c ] [--setup | --disable] [--wlm ] [--server-nodes SERVER_NODES] [--server-primary SERVER_PRIMARY] [--server-overlay-name SERVER_OVERLAY_NAME] [--server-overlay-priority SERVER_OVERLAY_PRIORITY] [--client-categories CLIENT_CATEGORIES] [--client-nodes CLIENT_NODES] [--client-overlay-name CLIENT_OVERLAY_NAME] [--client-overlay-priority CLIENT_OVERLAY_PRIORITY] [--client-slots ] [--submit-categories SUBMIT_CATEGORIES] [--submit-nodes SUBMIT_NODES] [--submit-overlay-name SUBMIT_OVERLAY_NAME] [--submit-overlay-priority SUBMIT_OVERLAY_PRIORITY] [--wlm-cluster-name WLM_CLUSTER_NAME] [--reboot] [--reset-cgroups] [--yes-i-really-mean-it] [--archives-location ] [--license ] [--purge] [--accounting-overlay-name ACCOUNTING_OVERLAY_NAME] [--accounting-overlay-priority ACCOUNTING_OVERLAY_PRIORITY] [--with-pyxis] [--add-pyxis] [--reinstall-pyxis] [--remove-pyxis] [--pyxis-data-directory ] [--nvidia-gpus NVIDIA_GPUS] [-v] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action {debug,remotedebug,undo,abort}] [--skip-packages] [--min-reboot-timeout ] [--allow-running-from-secondary] [--dev] [-h] ``` The help output from running `cm-wlm-setup -h` continues on beyond the preceding text output, and presents more options. These options can be grouped as follows: #### 7.3.1.1 Optional Arguments * `--setup`: Helps set up a server, enable roles, and create the default queues/partitions * `--disable`: Disable WLM services * `-h, --help`: Displays the help screen #### 7.3.1.2 Common Arguments * `-c` \<*YAML configuration file*>: Loads a runtime configuration for plugins, from a YAML configuration file. #### 7.3.1.3 Options For Installing Or Managing A WLM * `--wlm` \<*WLM name*>: Specifies which WLM is to be set up. Choices for \<*WLM name*> are: * `openpbs` * `pbspro` * `slurm` * `uge` * `lsf` * `--wlm-cluster-name` \<*WLM cluster name*>: Specifies the name for the new WLM cluster that is to be set up. * `--reboot`: Reboot after install #### 7.3.1.4 Server Role Settings * `--server-nodes` \<*server nodes*>: Sets the server roles of the WLM to the value set for \<*server nodes*>, which is a comma-separated list of nodes. Default value: `HEAD`, which is a reserved name for the head node. * `--server-primary` \<*primary server*>: Sets the hostname used for the primary server to \<*primary server*>. Default name: `HEAD`. * `--server-overlay-name` \<*server overlay name*>: Sets the server role configuration overlay name to \<*server overlay name*>. Default name: \<*WLM name*>`-server`, where \<*WLM name*> is the name specified in the `--wlm` option. * `--server-overlay-priority` \<*server overlay priority*>: Sets the server role configuration overlay priority to \<*server overlay priority*>. Default value: `500`. #### 7.3.1.5 Client Role Settings * `--client-categories` \<*client categories*>: Sets the client roles of the WLM to the value set for \<*client categories nodes*>, which is a comma-separated list of node categories. Default value: `default`. * `--client-nodes` \<*client nodes*>: Sets the client roles of the WLM to the value set for \<*client nodes*>, which is a comma-separated list of nodes. No value set by default. * `--client-overlay-name` \<*client overlay name*>: Sets the client role configuration overlay name to \<*client overlay name*>. Default name: \<*WLM name*>`-client`, where \<*WLM name*> is the name specified in the `--wlm` option. * `--client-overlay-priority` \<*client overlay priority*>: Sets the client role configuration overlay priority to \<*client overlay priority*>. Default value: `500`. * `--client-slots` \<*slots*>: Sets the number of slots on the client to \<*slots*>. #### 7.3.1.6 Submit Role Settings * `--submit-categories` \<*submit categories*>: Sets the submit roles of the WLM to the value set for \<*submit categories nodes*>, which is a comma-separated list of node categories that are submit nodes. Default value: `default`. * `--submit-nodes` \<*submit nodes*>: Sets the submit roles of the WLM to the value set for \<*submit nodes*>, which is a comma-separated list of submit nodes. No value set by default. * `--submit-overlay-name` \<*submit overlay name*>: Sets the submit role configuration overlay name to \<*submit overlay name*>. Default name: \<*WLM name*>`-submit`, where \<*WLM name*> is the name specified in the `--wlm` option. * `--submit-overlay-priority` \<*submit overlay priority*>: Sets the submit role configuration overlay priority to \<*submit overlay priority*>. Default value: `500`. #### 7.3.1.7 Disable Options * `--reset-cgroups`: Disable joining cgroup controllers with systemd setting JoinControllers * `--yes-i-really-mean-it`: Required for additional safety #### 7.3.1.8 Workload Manager Specific Options * `--archives-location` \<*path*>: Set the directory path for the archive files, only for LSF and GE. This parameter is mandatory for LSF installation. * `--license` \<*path*>: Set the path to the PBSPro, LSF, or GE license. * `--purge`: Remove the directories on disable, for GE and LSF. #### 7.3.1.9 Slurm Accounting Role Settings * `--accounting-overlay-name` \<*accounting overlay name*>: Sets the accounting role configuration overlay * `--accounting-overlay-priority` \<*accounting overlay priority*>: Sets the accounting role configuration overlay priority to \<*accounting overlay priority*>. Default value: `500`. #### 7.3.1.10 Slurm Pyxis Settings * `--add-pyxis`: Add Pyxis to an existing cluster * `--reinstall-pyxis`: Reinstall or upgrade Pyxis on all the clusters using it * `--remove-pyxis`: Remove Pyxis from an existing cluster * `--with-pyxis`: Enable Pyxis * `--pyxis-data-directory` \<*path*>: Sets directory where images will be stored #### 7.3.1.11 Slurm GPU Settings * `--nvidia-gpus` \<*NVIDIA GPUs specification*>: Sets the NVIDIA GPUs that will be used in the configuration, specifying type and number of GPUs. For example: `--nvidia-gpus=A100:4` * `--gpu-client-nodes` \<*NVIDIA GPU client nodes*>: Sets a comma-separated list of nodes assigned to the GPU overlay. * `--gpu-client-categories` \<*NVIDIA GPU client categories*>: Sets a comma-separated list of node categories assigned to the GPU overlay. #### 7.3.1.12 Advanced Options * ` -v, --verbose`: This displays a more verbose output. It can be helpful in troubleshooting. * ` --no-distro-checks`: Disables distribution checks based on ds.json. * ` --json`: Use json formatting for logs printed to STDOUT. * ` --output-remote-execution-runner`: Format output for CMDaemon. * ` --on-error-action {debug,remotedebug,undo,abort}`: Upon encountering a critical error, instead of asking the user for choice, the setup will do the selected action. * ` --skip-packages`: Skip the stages which install packages. Requires packages to be already installed. * ` --min-reboot-timeout` \<*timeout*>: How long to wait for nodes to finish reboot, in seconds. Minimum value: `300`. Default value: `300`. * ` --allow-running-from-secondary`: Allow the wizard to be run from the secondary when it is the active head node. * ` --dev`: Enables additional command line arguments for developers. For example, if GE packages are downloaded to `/root/ugedl`, then a GE setup could be carried out with (much output elipsized): **Example** ```bash [basecm10 ~]# cm-wlm-setup -v --setup --wlm uge --archives-location /root/ugedl --client-categories default I 19-12-05 15:29:46 | D 19-12-05 15:29:46 | Please wait... D 19-12-05 15:29:46 | ########################################################################## D 19-12-05 15:29:46 | ######################## START ########################################### D 19-12-05 15:29:46 | ########################################################################## D 19-12-05 15:29:46 | D 19-12-05 15:29:46 | ### Logging Current Runtime environment ... ... I 19-12-05 15:30:20 | Took: 00:33 min. I 19-12-05 15:30:20 | Progress: 100/100 I 19-12-05 15:30:20 | ########### Finished execution for 'Workload manager setup', status: completed I 19-12-05 15:30:20 | I 19-12-05 15:30:20 | Workload manager setup finished! I 19-12-05 15:30:20 | D 19-12-05 15:30:20 | D 19-12-05 15:30:20 | ######################## END ########################################### D 19-12-05 15:30:20 | I 19-12-05 15:30:20 | ``` A check to see if the GE instance (WLM cluster) has been created could be: **Example** ```bash [basecm10 ~]# cmsh -c "wlm; list" Type Name (key) Server nodes Submit nodes Client nodes ------ --------------------- ------------ ------------------------ ---------------- UGE uge basecm10 basecm10,node001,node002 node001,node002 ``` The GE instance can then be removed with: **Example** ```bash [basecm10 ~]# cm-wlm-setup -v --disable --wlm uge --purge --yes-i-really-mean-it --wlm-cluster-name=uge ``` A log file of the setup can be seen at `/var/log/cm-wlm-setup.log`. ### 7.3.2 Running `cm-wlm-setup` As A TUI Running `cm-wlm-setup` with no options and with no arguments brings up a TUI screen (figure [7.2](#fig-cm-wlm-setup-ncurses-initial-screen)). ![cm-wlm-setup TUI initial screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/9dc3627683dd9faacd25c84384e94d8fbc3d9cd5956848efff9dbdab73140f51/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-wlm-setup-ncurses-initial-screen.png) *Figure 7.2: cm-wlm-setup TUI initial screen* #### 7.3.2.1 Express Installation The `Setup (Express)` menu option allows the administrator to select the workload manager in the next screen (figure [7.3](#fig-cm-wlm-setup-ncurses-wlm-selection-screen)), and to install it with a minimal number of configuration steps. If it has already been installed, but disabled via `cm-wlm-setup`, then it can also be re-enabled, instead of installed from scratch. ![cm-wlm-setup TUI WLM selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/2528566c278c3d3382367f728e433e9d21277b1f7b957c5f3eb08077000880af/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-wlm-setup-ncurses-wlm-selection-screen.png) *Figure 7.3: cm-wlm-setup TUI WLM selection screen* #### 7.3.2.2 Step-by-step Installation If the `Setup (Step By Step)` menu option is chosen instead of the express option, then this also allows the administrator to select the workload manager in figure [7.3](#fig-cm-wlm-setup-ncurses-wlm-selection-screen). But after selection, there are a number of extra configuration steps that can be carried out which are not available in the express configuration. Guidance is given for these extra steps, and sensible default values are already filled in for many options. One part of the step-by-step session involves assigning the WLM client role to the compute nodes of the cluster. Only the non-GPU compute nodes should be assigned the (standard, non-GPU) WLM client role. The GPU compute nodes are assigned a *GPU* WLM client role in the section of the TUI wizard that deals with GPU configuration. The WLM client role is assigned to the entire category—the `default` category— of non-head nodes by default (figure [7.4](#fig-urwid-cm-wlm-setup-slurm-config-client-role-category)): ![Slurm with cm-wlm-setup: WLM Client role category configuration screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/8a340d793af816a485872adcf7337ab46334311bb8f22f0d3e71f88f17f75080/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/urwid-cm-wlm-setup-slurm-config-client-role-category.png) *Figure 7.4: Slurm with cm-wlm-setup: WLM Client role category configuration screen* The WLM client role can be assigned to selected nodes only by setting the `default` category checkbox to blank, and then selecting the non-GPU nodes in the following screen. For example, as illustrated in figure [7.4](#fig-urwid-cm-wlm-setup-slurm-config-client-role-category), where the standard, non-GPU nodes are `node002` and `node003`: ![Slurm with cm-wlm-setup: WLM client role node configuration screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/31c97665e8b333a41a8244dbdb402b9fc7256a1dfc910022b521ab856c963ec4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/urwid-cm-wlm-setup-slurm-config-client-role-node.png) *Figure 7.5: Slurm with cm-wlm-setup: WLM client role node configuration screen* #### 7.3.2.3 GPU Configuration Screens The GPU configuration screens are extra steps available during a `Setup (Step By Step)` session. The GPU configuration steps are discussed summarily in the quickstart section for GPUs, [here](/installation/quickstart#14-quickstart-for-gpus). The GPU configuration steps are covered in this section in more detail: After configuring the WLM server, WLM submission and WLM client roles for the nodes of the cluster, a screen that asks if GPU resources should be configured is displayed (figure [1.1](/installation/quickstart#14-quickstart-for-gpus)): ![Slurm With cm-wlm-setup: GPU Configuration Entry Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/14f9a9e8d9b36c585cf9303202bd6af07ba01b2d8ca30e1c06592757a5689694/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/9161a888e3f9ec5cd1868d95b48349a533310e18.png) *Figure 1.1: Slurm With cm-wlm-setup: GPU Configuration Entry Screen* Choosing `yes` means that some extra GPU configuration screens are presented. These are screens that allow: * the configuration overlay (section [2.1.6](/admin/clustermanagement#216-configuration-overlay)) name to be set for the GPU WLM clients. By default the name is set to `slurm-client-gpu`. * a GPU WLM client role to be assigned to a category, if, for example, all the GPU nodes have been given their own category. * the GPU WLM client role to be assigned to individual nodes instead of to a category. * a configuration overlay priority to be set for a GPU WLM client role. By default, this has a value of `450`. * automatic GPU detection, with the following options: * `BCM autodetects GPUs` * `Slurm autodetects NVIDIA GPU details` * `Slurm autodetects AMD GPU details` * `Slurm autodetects Intel GPU details` * `Slurm autodetects AWS Trainium/Inferentia devices` * `Manual GPU configuration in cm-wlm-setup` * `Skip GPU configuration` #### 7.3.2.4 Slurm Accounting Database Configuration In the step-by-step configuration for the Slurm WLM, after the server and client parameters have been set, the Slurm accounting database configuration can be set. This requires setting: * one or two accounting nodes * a primary accounting node All nodes must be reachable from `slurmctld`. The accounting nodes can be set with the `Select accounting nodes` screen (figure [7.7](#fig-cm-wlm-setup-slurm-select-accounting-nodes)): ![cm-wlm-setup selection of accounting nodes](/base-command-manager/product-manuals/_fern-img/0346d19ba9c46bda3bde006a25f0ff370b32d35d01481bfe5c8111890b5143ca.webp) *Figure 7.7: cm-wlm-setup selection of accounting nodes* Head nodes and compute nodes cannot be mixed for the accounting nodes. If two accounting nodes are set, then the `slurmdbd` high availability configuration is automatically set up. The primary accounting node—that is, the node that is `AccountingStorageHost` in `slurm.conf`—can then be set in the `Select the primary accounting server node` screen (figure [7.8](#fig-cm-wlm-setup-slurm-select-primary-accounting-node)): ![cm-wlm-setup selection of primary accounting node](/base-command-manager/product-manuals/_fern-img/db1f7529263bfc5240392ab5121bbb2f576b28948920418dd2d7773a968f6e0d.webp) *Figure 7.8: cm-wlm-setup selection of primary accounting node* The other accounting node is automatically set to be `AccountingStorageBackupHost`. The type of node on which the `slurmdbd` database runs can then be set (figure [7.9](#fig-cm-wlm-setup-slurm-select-slurmdbd-nodes)): ![cm-wlm-setup selection of the type of node on which the Slurm accounting database runs](/base-command-manager/product-manuals/_fern-img/a337103fb20c33ae0a2f5b2ed7faaefae2da9a2718dfc674ae6955981aba4631.webp) *Figure 7.9: cm-wlm-setup selection of the type of node on which the Slurm accounting database runs* * If `Use accounting node` is selected, then the database is stored on the primary accounting node * If `Select cluster node` is selected, then the next screen allows the selection of the BCM node on which the database is to be installed (figure [7.10](#fig-cm-wlm-setup-slurm-storage-server-host)) ![cm-wlm-setup Selection of host where the Slurm accounting database used by slurmdbd is stored](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/375ac7ef818ebed0cb6ea57667148729239bb0c52c9887d4718c9726aad68a38/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-wlm-setup-slurm-storage-server-host.png) *Figure 7.10: cm-wlm-setup Selection of host where the Slurm accounting database used by slurmdbd is stored* * If `Select external node` is selected, then a non-BCM hostname is asked for, and the cluster administrator is responsible for the installation and configuration of the DBMS Modifying the settings after installation is discussed [here](#slurm-accounting-database-configuration). #### 7.3.2.5 Disabling An Installation The `Disable` option in figure [7.2](#fig-cm-wlm-setup-ncurses-initial-screen) allows the administrator to disable an existing instance. #### 7.3.2.6 Summary Screen The screen that appears after the configuration steps are completed, is the `Summary` screen (figure [7.11](#fig-cm-wlm-setup-ncurses-final-summary-screen)). This screen allows the configuration to be viewed, saved, or saved and deployed. ![cm-wlm-setup TUI summary screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/152d325b9474f5fc18b094025489a34fcd1a3dfe2628c4a9f70ab8d293d718fc/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-wlm-setup-ncurses-final-summary-screen.png) *Figure 7.11: cm-wlm-setup TUI summary screen* If deployment is carried out, then several screens of output are displayed. After the deployment is completed, the log file can be viewed at `/var/log/cm-wlm-setup.log`. ### 7.3.3 Installation And Configuration Of Enroot And Pyxis With Slurm To Run Containerized Jobs #### 7.3.3.1 What Is Enroot? As the [README file for Enroot](https://github.com/NVIDIA/enroot/blob/master/README.md) says, Enroot is an open source tool to turn container images into unprivileged sandboxes. Enroot can be thought of as an enhanced unprivileged chroot. It uses user and mount namespaces, as well as other modern kernel features, in order to create such sandboxes. It uses the same underlying technologies as containers, but removes much of the isolation that they inherently provide, while preserving filesystem separation. Further details on Enroot can be found at [https://github.com/NVIDIA/enroot](https://github.com/NVIDIA/enroot). Enroot can be used with different workload managers, but for now only Slurm has been tightly integrated. #### 7.3.3.2 What Is Pyxis? Pyxis ([https://github.com/NVIDIA/pyxis](https://github.com/NVIDIA/pyxis)) is a SPANK plugin for Slurm. SPANK (Slurm Plug-in architecture for Node and job (K)control, `man spank.8`) is a generic interface for job launch code control in Slurm. The Pyxis plugin requires the Enroot utility, and allows the user’s jobs to be executed seamlessly over Enroot in unprivileged containers. The plugin enables the Slurm submission utilities to provide container-related command line options. #### 7.3.3.3 Enroot And Pyxis Packages BCM provides two Enroot package flavors: *standard* and *hardened*. The standard binaries are compiled as follows: * Open file descriptors are inherited * Spectre variant 2 (IBPB/STIBP) mitigations are disabled * Spectre variant 4 (SSBD) mitigations are disabled [As a rule of thumb: the hardened flavor is slightly more secure but suffers a larger overhead.](https://github.com/NVIDIA/enroot/blob/master/doc/installation.md) * The standard packages are already installed by default with BCM, and are: * `enroot`: provides the main utility and helper files. * `enroot+caps`: a nearly empty package which runs a post-installation script to grant extra capabilities to unprivileged users. This allows them to import and convert container images. * `pyxis-sources`: installs a tarball with the Pyxis plugin source files. The tarball is used by `cm-wlm-setup` to compile the Pyxis plugin on a cluster. The version of the package (section [11.1](/admin/softwareimagemanagement#111-nvidia-base-command-manager-packages-their-naming-convention-and-version)), found using `yum info pyxis-sources` or `apt-cache show pyxis-sources`, implies the Pyxis source version that is provided. For example: `pyxis-sources-0.20.0-[...].rpm` provides the Pyxis source for version 0.20.0. * The hardened packages that can be installed from the BCM repositories to replace the standard `enroot` and `enroot+caps` are: * `enroot-hardened`: provides hardened main utility and helper files. * `enroot-hardened+caps`: Provides extra capabilities to unprivileged users so that they can import and convert containers themselves. The package `pyxis-sources` installs the tarball at `/cm/local/apps/slurm/var/pyxis/pyxis-sources.tar.gz`. If the administrator needs a version of Pyxis other than the one provided by the package, then the archive can be replaced with a source tarball of the same name. #### 7.3.3.4 How Are Enroot And Pyxis Set Up In BCM? Pyxis and Enroot can be set up or removed by the administrator by choosing the appropriate options when Slurm is set up. In order to choose the appropriate options, `cm-wlm-setup` can be run in step-by-step mode ([here](#7322-step-by-step-installation)). Alternatively, Pyxis-related command line options can be specified for the `cm-wlm-setup` arguments `--add-pyxis` and `--reinstall-pyxis`. The step-by-step mode eventually presents a screen where the Pyxis setup can be enabled via a plugin: ![cm-wlm-setup Pyxis setup screen](/base-command-manager/product-manuals/_fern-img/6311a12d72eab2043a7886e70d03819f3b930e13aa4f03338b26a3e1e18f26eb.webp) *Figure 7.12: cm-wlm-setup Pyxis setup screen* The Pyxis screen is available for RHEL8-based systems and Ubuntu 20 and beyond. Older systems are not supported. If the plugin is enabled in the Pyxis setup screen, then `cm-wlm-setup` installs the `enroot` and `enroot+caps` packages from [https://github.com/NVIDIA/enroot/releases](https://github.com/NVIDIA/enroot/releases) into the software images where the Slurm client role is to be assigned. The `cm-wlm-setup` uitility also installs them directly on the head node if the head node was selected to run jobs. Pyxis sources are downloaded from GitHub, compiled with the installed Slurm, and installed in the appropriate Slurm directory. If the administrator enables the Pyxis plugin, then a new screen with Enroot settings is shown: ![cm-wlm-setup Enroot settings screen](/base-command-manager/product-manuals/_fern-img/023511e5812325d084c04c0a5a65f1e5bb3fd8f861710dace20f1f619dc2694f.webp) *Figure 7.13: cm-wlm-setup Enroot settings screen* The Enroot settings have reasonable default values. The administrator can change these: * `Share raw images among users and nodes`: If enabled, then the raw container image is shared among users and nodes, and the administrator must ensure that the cache directory is shared among all the compute nodes. Enabling the option disables the creation of a UID subdirectory by the Enroot prolog script. The following points about implementing Enroot cache sharing should be considered by the administrator: * Enabling the checkbox for the option changes the default value of the Enroot cache directory to `/cm/shared/enroot/cache`. The default value can also be changed in the TUI, in the `Cache directory` field, to any shared directory. If the specified directory does not exist, then `cm-wlm-setup` creates it with Unix directory permissions set to `chmod 00777`, which allows all users to share container images in the cache with each other on all the nodes. * Restrictions can be placed on the ability of regular users to share images and to create files in the cache, by setting more restrictive permissions for the directory manually, after the `cm-wlm-setup` run. For example, the permissions can be set to 00770, while the ownership group of the directory is changed to a group with all the Pyxis users in it. In this case, only the users within the Pyxis users group can create the image layers in the cache. * An alternative to cache sharing of images among users and nodes is to save Squashfs images in a shared directory, outside the cache. Then the users can specify a full path to a shared Squashfs image on the cluster in the `srun/sbatch` command line. In this case there is no need to share the cache among the nodes, as the Squashfs images are not copied over to cache before execution. Only the administrator should have access to updating the image. * `Share unpacked container images among nodes`: Enables sharing of unpacked container images (image filesystems) among nodes. A user job can modify the filesystem, so it is not recommended to share this directory among users. Enabling the option disables removal of the Enroot data directory, if the container name is specified. This allows a user to keep the container filesystem between jobs run by the user. It is up to the user or the administrator to clean up the data directory if the option is enabled. If the data directory is shared among nodes, then it is up to a user to ensure that the container filesystem is unpacked at least once before real jobs start. Otherwise, a race condition is possible when the images are extracted simultaneously on several nodes. * `Cache directory`: Path to the cache directory where raw container image layers are stored. If the directory is not shared, then the Slurm prolog script creates a subdirectory with a name that is the job user ID, while epilog cleans up that subdirectory. If the directory is shared, then neither prolog nor epilog touches the directory. * `Data directory`: Directory where the container filesystems (unpackaged images) are stored. If the directory is shared, then the epilog script checks if the container name is specified for the job, and skips removal of the container subdirectory. * `Runtime directory`: A working directory with temporary files created by Enroot. When Pyxis is set up, `cm-wlm-setup` also prepares the following configuration files for the compute nodes: * `/etc/enroot/enroot.conf`: This is a symlink to `/cm/shared/apps/slurm/var/etc/enroot.conf`. The configuration file provides reasonable default settings that allows Enroot to be used by many users. Important settings in the file are: * `ENROOT_RUNTIME_PATH`: working directory for enroot, created per user. Default value: `/run/enroot/runtime/$(id -u)` * `ENROOT_CACHE_PATH`: directory where container layers are stored. Default value: `/run/enroot/cache/$(id -u)` * `ENROOT_DATA_PATH`: directory where the filesystems of running containers are stored. Default value: `/run/enroot/data/$(id -u)` * `ENROOT_SQUASH_OPTIONS`: options passed to `mksquashfs` to produce container images. Default value: `-noI -noD -noF -noX -no-duplicates` * `ENROOT_MOUNT_HOME`: mount the current user’s home directory by default. Default value: `yes`. The administrator can change the symlink, or replace the file with a customized `enroot.conf`, if other values are preferred. * `/etc/sysctl.d/80-enroot.conf`: symlink to `/cm/shared/apps/slurm/var/etc/enroot-sysctl.conf`. The file tunes `sysctl` parameters for Enroot. * `/cm/local/apps/slurm/var/prologs/50-prolog-enroot.sh`: symlink to `/cm/shared/apps/slurm/var/cm/prolog-enroot.sh`. This is the `slurmd` prolog that creates appropriate directories, with appropriate user permissions, that are used by Enroot. * `/cm/local/apps/slurm/var/epilogs/50-epilog-enroot.sh`: symlink to `/cm/shared/apps/slurm/var/cm/epilog-enroot.sh`. Cleans the user directories used by Enroot. When `cm-wlm-setup` finishes its Pyxis configuration run, then there is no need for the nodes to be rebooted. The plugin works immediately. The new Slurm submission command option names start with either `--container` or `--no-container`. The full list of options can be displayed with the `--help` option. For example: **Example** ```bash [user@basecm10 ~]$ srun --help | grep container --container Path to OCI container bundle --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH [pyxis] the image to use for the container --container-mounts=SRC:DST[:FLAGS][,SRC:DST...] [pyxis] bind mount[s] inside the container. Mount --container-workdir=PATH [pyxis] working directory inside the container --container-name=NAME [pyxis] name to use for saving and loading the container on the host. Unnamed containers are containers are not. If a container with this name already exists, the existing container is used and --container-save=PATH [pyxis] Save the container state to a squashfs --container-mount-home [pyxis] bind mount the user's home directory. --no-container-mount-home --container-remap-root [pyxis] ask to be remapped to root inside the container. Does not grant elevated system --no-container-remap-root [pyxis] do not remap to root inside the container --container-entrypoint [pyxis] execute the entrypoint from the container --no-container-entrypoint container image --container-writable [pyxis] make the container filesystem writable --container-readonly [pyxis] make the container filesystem read-only ``` **Simple installation validation:** The simplest way to validate the Pyxis/Enroot setup after Slurm setup is to try out an `srun` command: **Example** ```bash [user@basecm10 ~]$ module load slurm [user@basecm10 ~]$ srun --container-image=ubuntu grep PRETTY /etc/os-release pyxis: importing docker image: ubuntu PRETTY_NAME="Ubuntu 20.04.3 LTS" ``` **A more thorough installation validation:** In order to perform a more thorough test of Pyxis/Enroot, an NCCL-based test can be used. NCCL is the NVIDIA Collective Communications Library ([https://docs.nvidia.com/deeplearning/nccl](https://docs.nvidia.com/deeplearning/nccl)), which is a library of multi-GPU collective communication primitives. The test can be found at [https://github.com/NVIDIA/nccl-tests](https://github.com/NVIDIA/nccl-tests). A prebuilt container image, with the NCCL test already installed, can be started as shown in the example that follows. It should be noted that running a multi-tenant cluster with all the export flags enabled as in the example may compromise security. It is therefore not recommended as a standard configuration. **Example** ```bash [user@basecm10 ~]$ module load slurm [user@basecm10 ~]$ srun --export="NCCL_DEBUG=INFO,NCCL_IB_DISABLE=1,PMIX_MCA_gds=hash" -N 2 \ --ntasks-per-node=1 --gpus-per-task=1 --mpi=pmix --container-image=deepops/mpi-nccl-test \ /nccl_tests/build/all_reduce_perf -b 1M -e 4G -f 2 -g 1 pyxis: imported docker image: deepops/mpi-nccl-test pyxis: imported docker image: deepops/mpi-nccl-test # nThread 1 nGpus 1 minBytes 1048576 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 validation: 1 # # Using devices # Rank 0 Pid 175945 on node001 device 0 [0x00] Tesla V100-SXM3-32GB # Rank 1 Pid 180379 on node002 device 0 [0x00] Tesla V100-SXM3-32GB node001:175945:175945 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.5<0> node001:175945:175945 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation node001:175945:175945 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1. node001:175945:175945 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.5<0> node001:175945:175945 [0] NCCL INFO Using network Socket NCCL version 2.11.4+cuda11.6 node002:180379:180379 [0] NCCL INFO Bootstrap : Using ens3:10.141.0.6<0> node002:180379:180379 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation node002:180379:180379 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1. node002:180379:180379 [0] NCCL INFO NET/Socket : Using [0]ens3:10.141.0.6<0> node002:180379:180379 [0] NCCL INFO Using network Socket node002:180379:182573 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 node001:175945:177561 [0] NCCL INFO Channel 00/02 : 0 1 node001:175945:177561 [0] NCCL INFO Channel 01/02 : 0 1 node001:175945:177561 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 node001:175945:177561 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [receive] via NET/Socket/0 node002:180379:182573 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [receive] via NET/Socket/0 node001:175945:177561 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [receive] via NET/Socket/0 node002:180379:182573 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [receive] via NET/Socket/0 node001:175945:177561 [0] NCCL INFO Channel 00 : 0[60] -> 1[60] [send] via NET/Socket/0 node002:180379:182573 [0] NCCL INFO Channel 00 : 1[60] -> 0[60] [send] via NET/Socket/0 node001:175945:177561 [0] NCCL INFO Channel 01 : 0[60] -> 1[60] [send] via NET/Socket/0 node002:180379:182573 [0] NCCL INFO Channel 01 : 1[60] -> 0[60] [send] via NET/Socket/0 node001:175945:177561 [0] NCCL INFO Connected all rings node002:180379:182573 [0] NCCL INFO Connected all rings node001:175945:177561 [0] NCCL INFO Connected all trees node001:175945:177561 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 node001:175945:177561 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer node002:180379:182573 [0] NCCL INFO Connected all trees node002:180379:182573 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 node002:180379:182573 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer node001:175945:177561 [0] NCCL INFO comm 0x1551f0001000 rank 0 nranks 2 cudaDev 0 busId 60 - Init COMPLETE # # out-of-place in-place # size count type redop time algbw busbw error time algbw busbw error # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) node001:175945:175945 [0] NCCL INFO Launch mode Parallel node002:180379:182573 [0] NCCL INFO comm 0x1551f8001000 rank 1 nranks 2 cudaDev 0 busId 60 - Init COMPLETE 1048576 262144 float sum 1491.5 0.70 0.70 0e+00 1421.3 0.74 0.74 0e+00 2097152 524288 float sum 3077.5 0.68 0.68 0e+00 2568.8 0.82 0.82 0e+00 4194304 1048576 float sum 4617.3 0.91 0.91 0e+00 4622.5 0.91 0.91 0e+00 8388608 2097152 float sum 9483.1 0.88 0.88 0e+00 8911.9 0.94 0.94 0e+00 16777216 4194304 float sum 17516 0.96 0.96 0e+00 18613 0.90 0.90 0e+00 33554432 8388608 float sum 34799 0.96 0.96 0e+00 41837 0.80 0.80 0e+00 67108864 16777216 float sum 99790 0.67 0.67 0e+00 83126 0.81 0.81 0e+00 134217728 33554432 float sum 204614 0.66 0.66 0e+00 199530 0.67 0.67 0e+00 268435456 67108864 float sum 319701 0.84 0.84 0e+00 341630 0.79 0.79 0e+00 536870912 134217728 float sum 608809 0.88 0.88 0e+00 683016 0.79 0.79 0e+00 1073741824 268435456 float sum 1337187 0.80 0.80 0e+00 1247369 0.86 0.86 0e+00 2147483648 536870912 float sum 2638741 0.81 0.81 0e+00 2743454 0.78 0.78 0e+00 4294967296 1073741824 float sum 5996381 0.72 0.72 0e+00 5430549 0.79 0.79 0e+00 # Out of bounds values : 0 OK # Avg bus bandwidth : 0.810575 # ``` The test demonstrates the usage of PMIX, MPI, and GPU in Enroot containers on multiple nodes. In the preceding example, 2 nodes with 1 GPU on each is requested by the job. For better results the cluster administrator can tune the test parameters. It should be noted that the image is quite large, and requires enough free space under `/var`. Also, the transfer timeout (`ENROOT_TRANSFER_TIMEOUT`) in `enroot.conf` must be large enough to download such a large container image to the compute nodes. A value of at least 600 seconds is recommended. If there are issues when executing MPI jobs with PMIX, then to help debug the issues the Pyxis documentation ([https://github.com/NVIDIA/pyxis/wiki/Setup](https://github.com/NVIDIA/pyxis/wiki/Setup)) suggests setting up the following environment variables: * `PMIX_MCA_ptl=^usock` * `PMIX_MCA_psec=none` * `PMIX_SYSTEM_TMPDIR=/var/empty` * `PMIX_MCA_gds=hash` (as configured [here](#7334-how-are-enroot-and-pyxis-set-up-in-bcm)) This configuration change is typically carried out in the software image of the regular nodes. For example, for a node category called `default`: ```bash [root@basecm10 ~]# category=default [root@basecm10 ~]# cat <> /cm/images/${category}/etc/default/slurmd PMIX_MCA_ptl=^usock PMIX_MCA_psec=none PMIX_SYSTEM_TMPDIR=/var/empty PMIX_MCA_gds=hash EOF [root@basecm10 ~]# systemctl restart slurmd [root@basecm10 ~]# cmsh -c "device; imageupdate -c ${category} -w" ``` ### 7.3.4 Prolog And Epilog Scripts #### 7.3.4.1 What Prolog And Epilog Scripts Do The workload manager runs prolog scripts before job execution, and epilog scripts after job execution. The purpose of these scripts can include: * checking if a node is ready before submitting a job execution that may use it * preparing a node in some way to handle the job execution * cleaning up resources after job execution has ended. The administrator can run custom prolog or epilog scripts for the queues from CMDaemon for GE, or LSF, by setting such scripts in the Base View or `cmsh` front ends. **Example** ```bash [basecm10->wlm[uge]->jobqueue]% use all.q [basecm10->wlm[uge]->jobqueue[all.q]]% show | grep . | grep -i epilog Epilog NONE ``` For PBS and Slurm, there are global prolog and epilog scripts, but editing them is not recommended. Indeed, in order to discourage editing them, the scripts cannot be set via the cluster manager front ends. Instead the scripts must be placed by the administrator in the software image, and the relevant nodes updated from the image. #### 7.3.4.2 Detailed Workings Of Prolog And Epilog Scripts Even though it is not recommended, some administrators may nonetheless wish to link and edit the scripts directly for their own needs, outside of the Base View or `cmsh` front ends. A more detailed explanation of how the prolog scripts work therefore follows: When a workload manager is configured via `cm-wlm-setup` or via the Base View setup wizard, then the workload manager is configured to run the generic prolog located in `/cm/local/apps/cmd/scripts/prolog`, and the generic epilog located in `/cm/local/apps/cmd/scripts/epilog`. The generic prolog and epilog scripts call a sequence of scripts for a particular workload manager in special directories. The directories have paths in the format: 1. `/cm/local/apps/` \<*workload manager*>`/var/prologs/` 2. `/cm/local/apps/` \<*workload manager*>`/var/epilogs/` In these directories, scripts are stored with names that have suffixes and prefixes associated with them that make them run in special ways, as follows: * **suffixes used in the prolog/epilog directory:** * `-prejob` script runs prior to all jobs * `-cmjob`: script runs prior to job run in a cloud * **prefixes used in the prolog/epilog directory:** * `00-` to * `99-` Number prefixes determine the order of script execution, with scripts with a lower number running earlier. The script names can therefore look like: **Example** * `01-prolog-prejob` * `10-prolog-cmjob` Return values for the prolog/epilog scripts have these meanings: * `0`: the next script in the directory is run. * *A non-zero return value*: no further scripts are executed from the prolog/epilog directory. Often, the script in a prolog/epilog directory is not a real script but a symlink, with the symlink going to a real file located in a different directory. The general script is then able to take care of what is expected of the symlink. The name of the symlink, and destination file, usually hints at what the script is expected to do. For example, if any health checks are marked to run as prejob checks during `cm-wlm-setup` configuration, then each of the PBS workload manager variants use the symlink `01-prolog-prejob` within the prolog directory `/cm/local/apps/` \<*`workload manager`*>`/var/prologs/`. The symlink links to the script `/cm/local/apps/cmd/scripts/prolog-prejob`. In this case, the script is expected to run prior to the job. **Example** ```bash [root@basecm10 apps]# pwd /cm/local/apps [root@basecm10 apps]# ls -l *pbs*/var/prologs/ openpbs/var/prologs/: total 0 lrwxrwxrwx 1 root root ... 01-prolog-prejob -> /cm/local/apps/cmd/scripts/prolog-prejob pbspro/var/prologs/: total 0 lrwxrwxrwx 1 root root ... 01-prolog-prejob -> /cm/local/apps/cmd/scripts/prolog-prejob ``` Epilog scripts (which run after a job run) have the location `/cm/local/apps/` \<*`workload manager`*>`/var/epilogs/`. Epilog script names follow the same execution sequence pattern as prolog script names. It should be noted that the `01-prolog-prejob` symlink is created and removed by BCM on each compute node where prejob is enabled in the workload manager entity. Each such entity provides a `Enable Prejob` parameter that affects the symlink existence: **Example** ```bash [head->wlm[openpbs]]% get enableprejob yes [head->wlm[openpbs]]% ``` This parameter is set to `yes` by `cm-wlm-setup` when at least one health check is selected as a prejob one. If any healthcheck was configured as a prejob check before `cm-wlm-setup` execution, and the administrator had a checkmark for that health check, then the prejob is considered enabled. #### 7.3.4.3 Workload Manager Configuration For Prolog And Epilog Scripts BCM configures generic prologs and epilogs during workload manager setup with `cm-wlm-setup`. The administrator can configure prologs and epilogs using appropriate parameters in the configuration of the workload managers, by creating the symlinks in the local prologs and epilogs directories. Generic prologs and epilogs are configured by default to run on job compute nodes (one run per each node per job) for Slurm, PBS variants and LSF. However, for GE the generic prolog runs only on main job node due to the limitations of GE configuration. The following parameters for prologs and epilogs can be configured with `cmsh` or Base View: * **Slurm** * `Prolog Slurmctld`: the fully qualified path of a program to execute before granting a new job allocation. The program is executed on the same node where the `slurmserver` role is assigned. The parth corresponds to the `PrologSlurmctld` parameter in `slurm.conf`. * `Epilog Slurmctld`: the fully qualified path of a program to execute upon termination of a job allocation. The program is executed on the same node where the `slurmserver` role is assigned. Corresponds with the `EpilogSlurmctld` parameter in `slurm.conf`. * `Prolog`: the fully qualified path of a program to execute on job compute nodes before granting a new job or step allocation. The program corresponds to the `Prolog` parameter, and by default points to the generic prolog. This prolog runs on every node of the job if the `Prolog flags` parameter contains the flag `Alloc` (the default value), otherwise it is executed only on the first node of the job. * `Epilog`: the fully qualified path of a program to execute on job compute nodes when the job allocation is released. * **LSF** * `Prolog`: the fully qualified path of a program to execute on the LSF server node on job allocation. As an LSF queue parameter, it corresponds to the `PRE_EXEC` parameter in `lsb.queues`. * `Epilog`: the fully qualified path of a program to execute on the LSF server node on job allocation release. As an LSF queue parameter, it corresponds to the `POST_EXEC` parameter in `lsb.queues` * `Host prolog`: the fully qualified path of a program to execute on each node of a job before the job is started. Corresponds to the `HOST_PRE_EXEC` parameter in `lsb.queues`. By default it is configured to run the generic prolog * `Host epilog`: the fully qualified path of a program to execute on each node of a job after the job is finished. Corresponds to the `HOST_POST_EXEC` parameter in `lsb.queues`. By default it is configured to run the generic epilog. * **GE** * `Prolog`: `ugeserver` role parameter that points to the fully qualified path of a program to execute. By default it is the generic prolog. It is executed only on the GE server node. * `Epilog`: `ugeserver` role parameter that is empty by default. It can be configured to run the generic epilog. It is executed only on the GE server node. * **PBS variants** * `Pelogs`: prolog and epilog hooks that emulate the classic PBS *p*`rologue` and *e*`pilogue` scripts located in the `pbs_mom` directory. The `pelogs` are configured in the appropriate workload manager instance, within `pelogs` mode when configuring the PBS cluster entity. For example, in `cmsh`: **Example** ```bash [basecm10]% wlm use openpbs [basecm10->wlm[openpbs]]% pelogs [basecm10->wlm[openpbs]->pelogs]% list Name (key) Enabled Order ---------- -------- -------- cm_epilog yes 99 cm_prolog yes 1 [basecm10->wlm[openpbs]->pelogs]% [basecm10->wlm[openpbs]->pelogs]% show cm_prolog Parameter Value -------------------------------- ------------------------------------------------ Enabled yes Name cm_prolog Events execjob_begin Path /cm/shared/apps/pbspro/var/cm/cm-pelog-hook.py Default action RERUN Enable parallel yes Verbose user output no Torque compatible no Order 1 Alarm 35 Debug no [basecm10->wlm[openpbs]->pelogs]% show cm_epilog Parameter Value -------------------------------- ------------------------------------------------ Enabled yes Name cm_epilog Events execjob_end Path /cm/shared/apps/pbspro/var/cm/cm-pelog-hook.py Default action RERUN Enable parallel yes Verbose user output no Torque compatible no Order 99 Alarm 35 Debug no [basecm10->wlm[openpbs]->pelogs]% ``` The parameters of each pelog correspond to appropriate parameters of PBS hooks: * Name: hook name that will be used in PBS * Enabled: flag to enable the hook in PBS * Events: list of PBS events that the hook will run on * Path: path to the hook script that will be imported to PBS in case the hook is not found. The script will not be imported if a hook with the same name already exists in PBS * Order: the hook execution order * Alarm: the hook alarm time (timeout), in seconds Additional parameters related to the prolog and epilog hooks only are: * Default action: The PBS default action when a prolog or epilog fails * Enable parallel: enable parallel prologues and epilogues, that run on sister moms * Verbose user output: provide verbose hook output to the user’s `.o`/`.e` file * Torque compatible: make torque compatible from prolog/epilog command line arguments point of view By default, two pelogs are added. These are to run the generic prolog and the generic epilog. If needed, the administrator can add more pelog hooks that will run on different events. ## 7.4 Enabling, Disabling, And Monitoring Workload Managers #### 7.4.0.1 Enabling And Disabling A WLM A WLM can be disabled for all nodes with `cm-wlm-setup`. Disabling the WLM means the workload management services are stopped by removing roles, and removing the WLM cluster object. Alternatively, a WLM can be enabled or disabled by the administrator via role addition and role removal with Base View or `cmsh`. This is described further on in this section. **Multiple WLM instances of the same type:** Versions of NVIDIA Base Command Manager prior to 9.0 already had the ability to have different workload managers run at the same time. However, NVIDIA Base Command Manager version 9.0 introduced the additional ability to run many workload managers of the same kind at the same time. **Example** Two WLM instances, Slurm and OpenPBS, are already running at the same time in the cluster, with each WLM assigned to one category. Then, BCM can start up a third WLM instance, such as another Slurm WLM instance. These WLM instances are alternatively called WLM clusters, because they effectively allow one cluster to function as many separate clusters as far as running WLMs is concerned. From the Base View or `cmsh` point of view a WLM consists of * a WLM server, usually on the head node * WLM clients, usually on the compute nodes For the administrator, enabling or disabling the servers or clients is then simply a matter of assigning or unassigning a particular WLM server or client role on the head or compute nodes, as deemed appropriate. The administrator typically also sets up an appropriate WLM environment module (`slurm`, `uge`, `openpbs`, `pbspro`, `lsf`), so that it is loaded up for the end user (section [2.2.3](/admin/clustermanagement#223-setting-up-a-default-environment-for-all-users)). ### 7.4.1 Enabling And Disabling A WLM With Base View A particular WLM package may be installed, but the WLM may not be enabled. This can happen, for example, if disabling a WLM that was previously enabled. If a WLM instance exists, then the WLM client, submission, and server roles can be enabled or disabled from Base View by assigning or removing the appropriate roles to nodes, categories, or configuration overlays. Within the role, the properties of the WLM may be further configured by setting options. #### 7.4.1.1 Workload Manager Role Assignment To An Individual Node With Base View **Workload Manager Server** The following roles are WLM roles that can be assigned to a node: * server * submit * accounting (for the Slurm WLM only, to configure and run the `slurmdbd` service) * client For example, a Slurm server role can be assigned to a head node, `basecm10`, via the navigation path: `Devices` > `Head Nodes[basecm10]` > `Edit` > `Settings` > `Role` > `Role list[ADD]` > `SlurmServerRole` ![Workload management role assignment on a head node](/base-command-manager/product-manuals/_fern-img/33ec793762536567b7b41b26a702a6a4b990a93cd0bb4d1d03de318b29ee6425.webp) *Figure 7.14: Workload management role assignment on a head node* The role window for the server then opens up, and allows role options to be set for the workload manager server. For example, for Slurm, a `builtin` or `backfill` option can be set for the `Scheduler` parameter. The workload manager server role is then saved with the selected options (figure [7.15](#fig-bright-view-workloadmanagement-choice-of-workloadmanager-head-scheduler)). To have the server start up on non-head nodes (but not for a head node), the `imageupdate` command (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) can be run. The workload manager server process and any associated schedulers then automatically start up. ![Workload management role assignment options on a head node](/base-command-manager/product-manuals/_fern-img/421504d8c942d72529beec965ceaa55923e6a4a2b78dcc2a86700a2758dc83ff.webp) *Figure 7.15: Workload management role assignment options on a head node* **Workload Manager Client** Similarly, the workload manager client process can be enabled on a node or head node by having the workload manager client role assigned to it. Some basic options can be set for the client role right away. Saving the role, and then running `imageupdate` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)), automatically starts up the client process with the options chosen, and managed by CMDaemon. #### 7.4.1.2 Workload Manager Role Assignment To A Category With Base View It is true that workload manager role assignment can be done as described in the preceding text for individual non-head nodes. However it is usually more efficient to assign roles using categories or configuration overlays, due to the large number of compute nodes in typical clusters. For example, the case can be considered of all physical on-premises non-head nodes. By default these are in the `default` category. This means that, by default, roles in the category are automatically assigned to all those non-head nodes, unless, as an exception, an individual node configuration overrides the category setting and uses a role setting instead at node level. Viewing the possible workload manager roles for the category `default` is done by using the navigation path: `Grouping` > `Categories[default]` > `Edit` > `Settings` > `Roles` > `Add` Once the role is selected, its options can be edited and saved. For compute nodes, the role assigned is usually a workload manager client. If the assigned role is that of a workload manager client, then the node with that role can have queues, GPUs, and other parameters specified for it. For example, queues can then be assigned via the navigation path:\ `HPC` > `WLM Management Clusters[cluster instance]` > `Job Queues` while GPUs, if using Slurm as the workload manager with default settings, can then be specified via the navigation path: `Configuration Overlays` > `slurm-client-gpu` > `roles` > `slurmclient` > `edit` > `Generic Resources` > `gpu` The workload manager server role can also be assigned to a non-head node. For example, a Slurm server role can be taken on by a non-head node. This is the equivalent to the `--server-nodes` option of `cm-wlm-setup`. Saving the roles with their options and then running `imageupdate` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) automatically starts up the newly-configured workload manager. #### 7.4.1.3 Workload Manager Role Options With Base View Each compute node role (workload manager client role) has options that can be set for `GPUs`, `Queues`, and `Slots`. Generally, the value that is set for `Slots` is the number of jobs expected to run on a node simultaneously. This number can, for example, be set to the number of threads. Threads (virtual cores) in the x86_64 architecture are provided by Intel’s hyper-threading (HT), or by AMD’s simultaneous multithreading (SMT). The physical CPU, the cores on the CPU, and the threads of a core (HT, SMT) should not be confused with each other, they are distinct concepts, and can all have different values. * `Slots`, in a workload manager, corresponds in BCM to: * the `CPUs` setting (a `NodeName` parameter) in Slurm’s `slurm.conf` * the `nproc` setting in PBS, In LSF setting the number of slots for the client role to `0` means that the client node does not run jobs on itself, but becomes a submit host, which means it is able to forward jobs to other client nodes. The default value for `Slots` is `AUTO`, which means that the value for `Slots` is auto-detected. The parameter may be alternatively be set to a non-negative number. Each WLM has a different implementation on how this is done. For instance, for Slurm, BCM uses Slurm’s own auto-detection implementation. On the other hand, for UGE BCM sets its value in the client role configuration overlay to the lowest number of hyperthreads among the nodes. * `Queues` with a specified name are available in their associated role after they are created. The creation of queues is described in sections [7.6.2](#762-queues-display-and-handling-in-base-view) (using Base View) and [7.7.2](#772-job-queue-display-and-handling-in-cmsh-jobqueue-mode) (using `cmsh`). All server roles also provide the option to enable or disable the `External Server` setting. Enabling that means that the server is no longer managed by BCM, but provided by an external device. The `cmsh` equivalent of enabling an external server is described [here](#cmshexternalservertorque). ### 7.4.2 Enabling And Disabling A Workload Manager With `cmsh` A particular workload manager package may be set up, but not enabled. This can happen, for example, if no WLM server or WLM client role has been assigned. If a WLM instance exists, then the WLM client, server, or submit roles can be enabled from `cmsh` by assigning it from within the `roles` submode. Within the assigned role, the properties of the WLM may be further configured by setting options. #### 7.4.2.1 Workload Manager Role Assignment To A Configuration Overlay With `cmsh` In `cmsh`, workload manager role assignment to a configuration overlay (section [2.1.5](/admin/clustermanagement#215-roles)) can be done using `configurationoverlay` mode. By default `cm-wlm-setup` run as a TUI session creates some configuration overlays with suggestive names, and assigns roles to the configuration overlays according to what the names suggest. Thus, for example, with the `cm-wlm-setup` TUI session used to carry out an express setup for Slurm, the configuration overlays that get created are the following: **Example** ```bash [basecm10->configurationoverlay]% list Name (key) Priority All head nodes Nodes Categories Roles -------------------- ---------- -------------- ---------------- ---------------- ---------------- slurm-accounting 500 yes slurmaccounting slurm-client 500 no default slurmclient slurm-server 500 yes slurmserver slurm-submit 500 no default slurmsubmit wlm-headnode-submit 600 yes slurmsubmit ``` Nodes in the default category can take on the `slurmclient` or `slurmsubmit` role by setting the nodes for the role using the associated configuration overlays `slurm-client` or `slurm-submit`. The `wlm-headnode-submit` configuration overlay is a special overlay. It is applied only to the head node, and is shared among all installed workload managers. Setting this overlay means that the head node, by default, has a submit role for a given workload manager. **Example** ```bash [basecm10->configurationoverlay]% use slurm-client [basecm10->configurationoverlay[slurm-client]]% show Parameter Value -------------------------------- ------------------------------------------------ Name slurm-client Revision All head nodes no Priority 500 Nodes Categories default Roles slurmclient Customizations <0 in submode> [basecm10->configurationoverlay[slurm-client]]% set nodes node001 node002 node003 basecm10 [basecm10->configurationoverlay[slurm-client]]% set nodes node001..node002 [basecm10->configurationoverlay*[slurm-client*]]% commit [basecm10->configurationoverlay[slurm-client]]% list Name (key) Priority All head nodes Nodes Categories Roles -------------------- ---------- -------------- ---------------- ---------------- ---------------- slurm-accounting 500 yes slurmaccounting slurm-client 500 no node001,node002 default slurmclient slurm-server 500 yes slurmserver slurm-submit 500 no default slurmsubmit wlm-headnode-submit 600 yes slurmsubmit [basecm10->configurationoverlay[slurm-client]]% ``` All the head nodes can also be made to take on the configuration overlay role by setting its `All head nodes` value to `yes`. The union set of `All head nodes` with `Nodes` is the set of nodes to which the role is applied for that configuration overlay. Values for the parameters in a role, such as the `slurmclient` role, can be set within the configuration overlay: **Example** ```bash [basecm10->configurationoverlay[slurm-client]]% roles [basecm10->configurationoverlay[slurm-client]->roles]% use slurmclient [basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]% show Parameter Value -------------------------------- ------------------------------------------------ Name slurmclient Revision Type SlurmClientRole Add services yes WLM cluster slurm Slots 0 All Queues no Queues defq Features Sockets 0 Cores Per Socket 0 ThreadsPerCore 0 Boards 0 SocketsPerBoard 0 RealMemory 0B NodeAddr Weight 0 Port 0 TmpDisk 0 Reason CPU Spec List Core Spec Count 0 Mem Spec Limit 0B GPU auto detect BCM Node Customizations <0 in submode> Generic Resources <0 in submode> Cpu Bindings None Slurm hardware probe auto detect yes Memory autodetection slack 10.0% IMEX no [basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]% ``` After the workload manager roles are assigned or unassigned, and after running `imageupdate` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) for non-head nodes, the associated workload manager services automatically start up or stop as appropriate. The configuration overlay role values are inherited by categories and nodes, unless the categories and nodes have their own values set. Thus, for role properties, a value set at node level overrides values set at category level, and a value set at configuration overlay level overrides a value set at category level. This is typical of how properties of objects are inherited in BCM levels. #### 7.4.2.2 Workload Manager Role Assignment To A Category With `cmsh` In `cmsh`, workload manager role assignment to a node category can be done using `category` mode, using the category name, assigning a role from the `roles` submode, setting the WLM instance for that role, and committing the modified role: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% category [basecm10->category]% use default [basecm10->category[default]]% roles [basecm10->category[default]->roles]% assign slurmclient [basecm10->category[default]->roles*[slurmclient*]]% wlm list Type Name (key) Server nodes Submit nodes Client nodes ------- --------------------- ------------ ---------------- ---------------- slurm slurm1 basecm10 basecm10,node001 node001,node002 [basecm10->category[default]->roles*[slurmclient*]]% set wlmcluster slurm1 [basecm10->category[default]->roles*[slurmclient*]]% commit ``` Settings that are assigned in the `slurmclient` role of the category overrule the `slurmclient` role configuration overlay settings. The role assignment at category level requires the value for a WLM instance to be specified for `wlmcluster` before the `commit` command is successful. #### 7.4.2.3 Workload Manager Role Assignment To An Individual Node With `cmsh` In `cmsh`, assigning a workload manager role to a head node can be done in `device` mode. This can be done by using the head node name as the device, assigning the workload manager role to the device, setting the WLM instance value to the role within the `role` submode, and committing the modified role. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% use basecm10 [basecm10->device[basecm10]]% roles [basecm10->device[basecm10]->roles]% assign slurmserver [basecm10->category[default]->roles*[slurmserver*]]% wlm list Type Name (key) Server nodes Submit nodes Client nodes ------- --------------------- ------------ ---------------- ---------------- slurm slurm1 basecm10 basecm10,node001 node001,node002 [basecm10->category[default]->roles*[slurmserver*]]% set wlmcluster slurm1 [basecm10->device*[basecm10*]->roles*[slurmserver*]]% commit [basecm10->device[basecm10]->roles[slurmserver]]% ``` For regular nodes, role assignment is done via `device` mode, using the node name. Th node name is assigned the workload manager role, the WLM instance value is set for that role in the `role` submode, and the modified role is committed. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% use node001 [basecm10->device[node001]]% roles [basecm10->device[node001]->roles]% assign slurmclient [basecm10->device[node001]->roles*[slurmclient*]]% wlm list Type Name (key) Server nodes Submit nodes Client nodes ------- --------------------- ------------ ---------------- ---------------- slurm slurm1 basecm10 basecm10,node001 node001,node002 [basecm10->device[node001]->roles*[slurmclient*]]% set wlmcluster slurm1 [basecm10->device[node001]->roles*[slurmclient*]]% commit [basecm10->device[node001]->roles[slurmclient]]% ``` The role assignment at node level requires the value for a WLM instance to be specified for `wlmcluster` before the `commit` command is successful. Role assignment values set in `device` mode have precedence over any role assignment values set in `category` mode for that node. This means, for example, that if a node is originally in a node category with a `slurmclient` role and queues set, then when the node is assigned a `slurmclient` role from `device` mode, its queue properties are empty by default. #### 7.4.2.4 Setting Options For Workload Manager Settings With `cmsh` In the preceding text, it is explained how the workload manager client or server is assigned a role (such as `slurmclient` or `slurmserver`) within the `roles` submode. It is done from within a main mode of `cmsh`. The main modes from which role assignment can be done are: `configurationoverlay`, `category` or `device`. **Options for workload managers in general:** Whatever main mode is used, the workload manager options for a role can then be set with the usual object commands introduced in section [2.5.3](/admin/clustermanagement#253-working-with-objects). * **WLM client options:** For example, the configuration options of a WLM client, such as the PBS Professional client, can be seen by using the `show` command on the role. Here it can be seen at a `category` level, for the default category `default`: **Example** ```bash [basecm10->category[default]->roles[pbsproclient]]% show Parameter Value -------------------------------- ------------------------------------------------ Add services yes All Queues no GPUs 0 Name pbsproclient Properties Queues Revision Slots 1 Type PbsProClientRole WLM cluster Mom Settings Comm Settings Node Customizations <0 in submode> ``` The `Slots` option can be set in the role **Example** ```bash [basecm10->category[default]->roles[pbsproclient]]% set slots 2 [basecm10->category*[default*]->roles*[pbsproclient*]]% commit [basecm10->category[default]->roles[pbsproclient]]% ``` * **WLM server options:** Similarly, WLM server options can be managed from an assigned server role. For PBS, the `pbsproserver` role for a device shows: **Example** ```bash [basecm10->device[basecm10]->roles]% use pbsproserver [basecm10->device[basecm10]->roles[pbsproserver]]% show Parameter Value -------------------------------- ------------------------------------------------ Name pbsproserver Revision Type PbsProServerRole Add services yes WLM cluster External Server no Comm Settings ``` **Option to set an external workload manager:** A workload manager can be set to run as an external server from within a device mode role: **Example** ```bash [basecm10->device[basecm10]->roles[pbsproserver]]% set externalserver on [basecm10->device[basecm10]->roles[pbsproserver*]]% commit ``` For convenience, setting it on the head node is recommended. The Base View equivalent of configuring `externalserver` is described [here](#7413-workload-manager-role-options-with-base-view). ### 7.4.3 Monitoring The Workload Manager Services By default, the workload manager services are monitored. BCM attempts to restart the services using the service tools (section [3.14](/admin/configuring#314-managing-and-configuring-services)), unless the role for that workload manager service is disabled, or the service has been stopped. Workload manager roles and corresponding services can be disabled using `cm-wlm-setup` (section [7.3](#73-installation-of-workload-managers)), Base View role configuration (section [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)), or `cmsh` role configuration (section [7.4.2](#742-enabling-and-disabling-a-workload-manager-with-cmsh)). The daemon service states can be viewed for each node via the shell, `cmsh`, or Base View (section [3.14](/admin/configuring#314-managing-and-configuring-services)). Queue submission and scheduling daemons normally run on the head node. From Base View their states are viewable via the navigation path to the services running on the node. For example, on a head node (figure [7.16](#fig-bright-view-head-services)), via: `Devices > Head Nodes > [basecm10] > Settings > JUMP TO > Services` ![Services seen on head node in Base View](/base-command-manager/product-manuals/_fern-img/ce8eb35510911ea90a496a99129790faa44aa8333b00172691c56acb295eb544.webp) *Figure 7.16: Services seen on head node in Base View* For a regular node, a similar navigation path for `node001`, for example, is: `Devices > Nodes > node001 > Settings > JUMP TO > Services` and leads to a view of services on the regular nodes (figure [7.17](#fig-bright-view-regular-node-services)): ![Services seen on regular node in Base View](/base-command-manager/product-manuals/_fern-img/7b382178268bbf25a7eb9781e693dbe1d2661c6a2445d8e12c643b50d6f07174.webp) *Figure 7.17: Services seen on regular node in Base View* Considering only the WLMs: in figure [7.16](#fig-bright-view-head-services) the `pbsserver` is seen running on the head node, while in figure [7.17](#fig-bright-view-regular-node-services) the `pbsmom` server is seen running on the compute node. The navigation path:\ `Devices > Head Nodes > basecm10 > Settings > JUMP TO > Roles`\ shows the roles that result in the servers running on the head node (figure [7.18](#fig-bright-view-head-wlm-roles)): ![Roles seen on head node in Base View](/base-command-manager/product-manuals/_fern-img/96bc75d50415c921d119186afd5b536817f8ccaee6fa8669b1d26ba29f16adae.webp) *Figure 7.18: Roles seen on head node in Base View* Similarly, the navigation path:\ `Devices > Nodes > node001 > Settings > JUMP TO > Roles`\ shows the roles on a regular node such as `node001` (figure [7.19](#fig-bright-view-regular-node-wlm-roles)): ![Roles seen on regular node in Base View](/base-command-manager/product-manuals/_fern-img/fda7cd32efe849160d3ce0d195148e5fe808bef59d02fde41a2573dda37220bd.webp) *Figure 7.19: Roles seen on regular node in Base View* The roles seen in these figures are from the defaults that `cm-wlm-setup` provides in an express setup. For regular nodes, the inheritance of roles from category level or configuration overlay level is indicated by the values in the `INHERITED` column. Thus, in figure [7.19](#fig-bright-view-regular-node-wlm-roles), the `pbsprosubmit` and `pbsproclient` roles are decided by the default setting from the category level. The assignment of roles can be varied to taste for WLMs. This allows WLM services to run on the head node or on the regular nodes. From `cmsh` the services states are viewable from within `device` mode, using the `services` command. One-liners from the shell to illustrate this are (output elided): **Example** ```bash [root@basecm10 ~]# cmsh -c "device services node001; status" Service Status ------------ ----------- nslcd [ UP ] pbsmom [ UP ] [root@basecm10 ~]# cmsh -c "device services basecm10; status" Service Status ------------ ----------- ... pbsserver [ UP ] ``` Roles can be viewed from within the main modes of `configurationoverlay`, `category`, or `device`. One-liners to view these are: **Example** ```bash [root@basecm10 ~]# cmsh -c "configurationoverlay; list" Name (key) Priority All head nodes Nodes Categories Roles -------------------- ---------- -------------- ---------------- ---------------- ---------------- openpbs-client 500 no default pbsproclient openpbs-server 500 yes pbsproserver openpbs-submit 500 no default pbsprosubmit wlm-headnode-submit 600 yes pbsprosubmit [root@basecm10 ~]# cmsh -c "category; use default; roles; list -p" Name (key) ---------------------------- [overlay:openpbs-client:500] pbsproclient [overlay:openpbs-submit:500] pbsprosubmit [root@basecm10 ~]# cmsh -c "device use node001; roles; list -p" Name (key) ---------------------------------------- [overlay:openpbs-client:500] pbsproclient [overlay:openpbs-submit:500] pbsprosubmit [root@basecm10 ~]# cmsh -c "device use basecm10; roles; list -p" Name (key) ---------------------------------------- [750] backup [750] boot [750] firewall [750] headnode [750] monitoring [750] provisioning [750] storage [overlay:openpbs-server:500] pbsproserver [overlay:openpbs-submit:500] pbsprosubmit ``` The `-p|--priority` option displays of the `list` command the priority setting for the roles. ## 7.5 Configuring And Running Individual Workload Managers BCM deals with the various choices of workload managers in as generic a way as possible. This means that not all features of a particular workload manager can be controlled, so that fine-tuning must be done through the workload manager configuration files. Workload manager configuration files that are controlled by BCM should normally not be changed directly because BCM overwrites them. However, overwriting by CMDaemon is prevented on setting the directive: `FreezeChangesTo` \<*workload manager*>`Config = ` in `cmd.conf` (Appendix [C](/admin/cmdconf)), where \<*workload manager*> takes the value of `Slurm`, `UGE`, `LSF`, or `PBSPro`, as appropriate. The value of the directive defaults to `false`. A list of configuration files that are changed by CMDaemon, the items changed, and the events causing such a change are listed in Appendix [H](/admin/workloadmanagerconfigupdate). A very short guide to some specific workload manager commands that can be used outside of the NVIDIA Base Command Manager 10 system is given in Appendix [F](/admin/workloadmanagementquickref). ### 7.5.1 Configuring And Running Slurm #### 7.5.1.1 Slurm Packages At the time of writing of this section (March 2026), BCM can be integrated with Slurm packages for Slurm versions 23.02, 23.11, 24.05, 24.11, and 25.05. For Slurm version 23.02, the following packages are available from the BCM repositories for RHEL8, RHEL9, and derivatives: * `slurm23.02`: Simple Linux Utility for Resource Management, Slurm Workload Management * `slurm23.02-client`: Spool dir and slurm conf installed on nodes * `slurm23.02-contribs`: Perl tool to print Slurm job state information * `slurm23.02-libpmi`: Slurm’s implementation of the pmi libraries * `slurm23.02-openlava`: openlava/LSF wrappers for transition from OpenLava/LSF to Slurm * `slurm23.02-pam`: PAM module for restricting access to compute nodes via Slurm. * `slurm23.02-perlapi`: Perl API to Slurm. * `slurm23.02-slurmdbd`: Slurm database daemon * `slurm23.02-slurmrestd`: Slurm REST API translator * `slurm23.02-torque`: Torque/PBS wrappers for transition from Torque/PBS to Slurm. For Slurm version 23.11, the value of 23.02 is simply replaced by 23.11 in the preceding list of packages. Slurm version 23.11 also has an extra package: * `slurm23.11-sackd`: Slurm authentication daemon. Used on login nodes that are not running slurmd daemons to allow authentication to the cluster. For Slurm versions 24.05, 24.11, or 25.05, the value of 23.11 in the 23.11 package names is simply replaced by 24.05, 24.11, or 25.05 respectively. In addition, Sharp packages ([here](#7513-slurm-nvidia-sharp-plugin)) are available for 24.05, and 24.11. The distribution version of Slurm (package: `slurm`) is not integrated with BCM and conflicts with the preceding packages. It should not be used. Important updates from upstream are patched into the BCM repositories. If updating Slurm packages, all the Slurm packages should be updated to the same version, on the compute nodes as well as on the scheduling node. #### 7.5.1.2 Updating From Earlier Slurm Versions To `slurm23.11` Upgrading between major versions of Slurm is generally possible. It is a good idea to upgrade one version at a time, rather than jumping 2 or more versions ahead, which requires a full wipe of the Slurm configuration. If Slurm is using Pyxis (section [7.3.3](#733-installation-and-configuration-of-enroot-and-pyxis-with-slurm-to-run-containerized-jobs)), then upgrading the Slurm version means that Pyxis needs to be reinstalled using `cm-wlm-setup`. The reinstallation run for Pyxis compiles Pyxis and recreates a plugin directory for the new Slurm version under `/cm/shared/apps/slurm/`. An upgrade from one major version of Slurm to another can be carried out according to the following example, which is for an update from major version 22.05 to version 23.11, and avoids total reconfiguration of the Slurm configuration: * It is recommended that no jobs are running. Draining nodes (section [7.7.3](#773-nodes-drainage-status-and-handling-in-cmsh)) is one way to arrange this over time. No new jobs run on a drained node, but old ones are allowed to finish. * When all running jobs are finished, then Slurm server services—`slurmctld` and `slurmdbd`—should be stopped using `cmsh` or Base View (section [3.14.2](/admin/configuring#3142-managing-and-configuring-servicesexamples)): **Example** ```bash [basecm10->device[basecm10]->services]% stop slurmctld [basecm10->device[basecm10]->services]% stop slurmdbd ``` * The old Slurm packages should then be removed. There can be only one version of Slurm at a time, so there will be a package installation conflict if a new version is installed while an old one is still there. Removal can be carried out on RHEL-based systems with, for example: ```bash [root@basecm10 ~]# yum remove slurm22.05* ``` The old packages must also be removed from each software image that uses it: ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/ ... [root@ /]# yum remove slurm22.05* ...removal takes place... [root@ /]# exit ``` The `cm-chroot-sw-img` wrapper utility is discussed in section [11.4.1](/admin/softwareimagemanagement#1141-installing-from-head-into-the-image-changing-the-root-directory-into-which-the-packages-are-deployed). * The new packages can then be installed. For installation onto the RHEL head node, the installation might be carried out as follows: ```bash [root@basecm10 ~]# yum install slurm23.11 slurm23.11-client slurm23.11-contribs \ slurm23.11-perlapi slurm23.11-devel slurm23.11-pam slurm23.11-slurmdbd \ slurm23.11-slurmrestd ``` The client package can be installed in each software image with, for example: ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/ ... [root@ /]# yum install slurm23.11-client ...installation takes place [root@ /]# exit ``` Other Slurm packages from the repository may also be installed on the head node and within the software images, as needed. * The new Slurm version is then set in `cmsh` or Base View, in the Slurm WLM cluster configuration: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% set version 23.11; commit ``` * Slurm server services `slurmctld` and `slurmdbd` should then be started again using `cmsh` or Base View: **Example** ```bash [basecm10->device[basecm10]->services]% start slurmdbd [basecm10->device[basecm10]->services]% start slurmctld ``` * The nodes can then have their new image placed on them, and the new Slurm configuration can then be taken up. This can be done in the following two ways: 1. The regular nodes can then be restarted to supply the live nodes with the new image and get the new Slurm configuration running. 2. Alternatively, the `imageupdate` command (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) can be run on the live nodes to supply them with the image. Running the `imageupdate` command in dry mode (the default) first is recommended. The `synclog` command can then be run to check there are no unexpected changes that will take place due to the update. If all is well, then `imageupdate`’s wet mode flag `-w` can be used in order to really carry out the task. For example, the change can be checked, and then actually carried out, for the image on `node001` with: ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% imageupdate Performing dry run (use synclog command to review result, then pass -w to perform real update)... ...some messages... imageupdate [ COMPLETED ] [basecm10->device[node001]]% synclog ...rsync dry run output... [basecm10->device[node001]]% imageupdate -w ...same messages as before, but this time it really happens... [basecm10->device[node001]]% commit ``` The last `commit` command triggers the reconfiguration of the file `/etc/systemd/system/slurmd.service.d/99-cmd.conf` on the node. After a short time—around 30 seconds—the file is regenerated. The `slurmd` service on the node can then be restarted with: ```bash [root@basecm10 ~]# ssh node001 "systemctl daemon-reload" cmsh -c "device services node001; use slurmd; restart" ``` #### 7.5.1.3 Slurm NVIDIA Sharp Plugin NVIDIA’s Sharp packages are packages that offload some operations from CPUs and GPUs to the network. Sharp is the abbreviation used for the Scalable Hierarchical Aggregation and Reduction Protocol. The protocol refers to the reduction in the amount of data traversing the network and the reduction in the time for collective operations. Using Sharp frees up more CPUs and GPU resources for computation. The Sharp binaries are packaged and available for Slurm versions 24.05 and 24.11.Running `apt-cache search slurm| grep sharp` is a way of listing the packages. To use the NVIDIA Sharp plugin, the existing Slurm packages must be substituted by packages with the `-sharp` suffix. This allows the plugin and associated extra options to be used. **Example** The following session on an Ubuntu system illustrates the existing Slurm version 24.05 packages, and then carrying out a replacement of these with the corresponding Sharp versions: ```bash basecm10:~# dpkg --get-selections | grep slurm | cut -f1 | tr "\n" " " ; echo slurm24.05 slurm24.05-client slurm24.05-contribs slurm24.05-devel slurm24.05-perlapi slurm24.05-slurmdbd basecm10:~# apt install slurm24.05-sharp slurm24.05-sharp-client slurm24.05-sharp-contribs\ slurm24.05-sharp-devel slurm24.05-sharp-perlapi slurm24.05-sharp-slurmdbd ``` The change should be treated like the upgrade procedure [here](#7512-updating-from-earlier-slurm-versions-to-slurm2311), and should likewise end with the cluster administrator selecting the `-sharp` Slurm version parameter in `cmsh` and committing it. **Example** ```bash root@basecm10:~# cmsh -c "wlm use slurm; get version" 24.05-sharp ``` This plugin is only useful for a network that has the necessary hardware and services configured. Further information about Sharp can be found at [https://docs.nvidia.com/networking/display/sharpv300](https://docs.nvidia.com/networking/display/sharpv300). #### 7.5.1.4 Configuring Slurm After Slurm setup is configured and installed with `cm-wlm-setup` (section [7.3](#73-installation-of-workload-managers)), the Slurm software components are installed in a symlinked directory `/cm/shared/apps/slurm/current`. Slurm clients and servers can be configured to some extent via role assignment (sections [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view) and [7.4.2](#742-enabling-and-disabling-a-workload-manager-with-cmsh)). Using `cmsh`, advanced option parameters can be set under the `slurmclient` and `slurmserver` roles. The settings for the roles can be done at configuration overlay, category, or node level (sections [2.1.5](/admin/clustermanagement#215-roles), [2.1.6](/admin/clustermanagement#216-configuration-overlay)). By default, the `cm-wlm-setup` utility configures Slurm using configuration overlays. **Example** ```bash [basecm10->configurationoverlay]% list Name (key) Priority All head nodes Nodes Categories Roles -------------------- ---------- -------------- ---------------- ---------------- ---------------- slurm-accounting 500 yes slurmaccounting slurm-client 500 no default slurmclient slurm-server 500 yes slurmserver slurm-submit 500 no default slurmsubmit wlm-headnode-submit 600 yes slurmsubmit ``` The settings within the roles can be viewed and modified. For example, the `slurmclient` role of the `slurm-client` configuration overlay can be viewed: **Example** ```bash [basecm10->configurationoverlay]% roles slurm-client [basecm10->configurationoverlay[slurm-client]->roles]% show slurmclient Parameter Value -------------------------------- ------------------------------------------------ Name slurmclient Revision Type SlurmClientRole Add services yes WLM cluster slurm Slots 0 All Queues no Queues defq Features Sockets 0 Cores Per Socket 0 ThreadsPerCore 0 Boards 0 SocketsPerBoard 0 RealMemory 0B NodeAddr Weight 0 Port 0 TmpDisk 0 Reason CPU Spec List Core Spec Count 0 Mem Spec Limit 0B GPU auto detect BCM Node Customizations <0 in submode> Generic Resources <0 in submode> Cpu Bindings None Slurm hardware probe autodetect yes Memory autodetection slack 0.0% IMEX no ``` **Slurm Hardware Autodetection (non-GPU):** The setting `Slurm hardware probe autodetect` in the Slurm role enables automated hardware detection for Slurm for non-GPU hardware. (Specifically for GPUs is the related Slurm `GPU auto detect` setting ([here](#slurm-hardware-autodetection-for-gpus))). `Slurm hardware probe autodetect` enables detection for the following Slurm parameters: * `corespersocket` * `threadspercore` * `boards` * `socketsperboard` * `sockets` * `realmemory` The autodetected parameters are placed in the `NodeName` line in `slurm.conf`. The Slurm client role parameters can be modified. For example `Core Spec Count`: **Example** ```bash [basecm10->configurationoverlay[slurm-client]->roles]% set slurmclient corespeccount 2 [basecm10->configurationoverlay*[slurm-client*]->roles*]% commit ``` As usual, values set at node level override the values set at categories and configuration overlays level. For example, to set `corespeccount` to 4, only for node001 but not for other nodes, the session might run further as: **Example** ```bash [basecm10->configurationoverlay[slurm-client]->roles]% device use node001 [basecm10->device[node001]]% roles [basecm10->device[node001]->roles]% assign slurmclient [basecm10->device*[node001*]->roles*[slurmclient*]]% set corespeccount 4 [basecm10->device*[node001*]->roles*[slurmclient*]]% commit Field Message ------------------------ ---------------------------------------------------------------- wlmCluster Error: The WLM cluster should be set [basecm10->device*[node001*]->roles*[slurmclient*]]% wlm list Type Name (key) Server nodes Submit nodes Client nodes ------ ------------------------ ------------ ---------------- ---------------- Slurm slurm basecm10 basecm10,node001 node001,node001 [basecm10->device*[node001*]->roles*[slurmclient*]]% set wlmcluster slurm [basecm10->device*[node001*]->roles*[slurmclient*]]% commit ``` In the preceding session, the role needs to be assigned at node level with `assign slurmclient` because it does not initially exist at node level. If it already existed, then `use slurmclient` could have been used to descend into that role. Also in the preceding session, one of the values that the Slurm client needs to know is `wlmcluster`, which decides which WLM it is to work with on the cluster. The value is selected from the list of WLM instance names in `wlm` mode. The level of the active role can be seen with the `list` command. For example, the Slurm client role assignment at node level is seen here: **Example** ```bash [basecm10->device[node001]->roles[slurmclient]]% list Name (key) ----------------------------------- [overlay:slurm-submit] slurmsubmit slurmclient ``` Removing the assignment has the `list` command display the configuration overlay Slurm client role assignment: **Example** ```bash [basecm10->device[node001]->roles[slurmclient]]% unassign slurmclient; commit [basecm10->device[node001]->roles[slurmclient]]% list Name (key) ----------------------------------- [overlay:slurm-submit] slurmsubmit [overlay:slurm-client] slurmclient ``` **Slurm Hardware Autodetection For GPUs:** The Slurm `GPU auto detect` setting is a setting seen in the session output [here](#7514-configuring-slurm). It manages Slurm GRES configuration ([https://slurm.schedmd.com/gres.conf.html](https://slurm.schedmd.com/gres.conf.html)) automatically, and can be set: * globally, for all Slurm compute nodes in a Slurm instance, from within the `wlm` mode of `cmsh` * for a particular role, such as in a device role, a category role, and a configuration overlay role for a Slurm client **Example** ```bash [basecm10->configurationoverlay[slurm-client]->roles[slurmclient]]% set gpuautodetect [TAB][TAB] bcm none nrt nvml off oneapi rsmi ``` If `GPU auto detect` is assigned a value, then a corresponding value is assigned to the `autodetect` parameter in Slurm’s `gres.conf` file. One out of 7 values can be assigned to `GPU auto detect`: 1. `bcm`: to use BCM values, which are the values that CMDaemon automatically puts in for the vendor (NVIDIA, Intel, AWS, AMD) and for the number of GPUs. 2. `none`: to not have the `autodetect` parameter exist in `gres.conf`. In this case, there is actually no corresponding value that can be assigned to the `autodetect` of `gres.conf`. 3. `nrt`: to detect AWS Trainium/Inferentia devices. \<— not yet, in October 2024, trunk, or bcm10.24.09 4. `nvml`: to detect NVIDIA GPUs. 5. `off`: to turn Slurm GPU autodetection off. 6. `oneapi`: to detect Intel GPUs. 7. `rsmi`: to detect AMD GPUs Slurm GPU autodetection is described further [here](#slurmautodetection). **Slurm accounting database configuration in `cmsh`:** After package setup is carried out with `cm-wlm-setup` (section [7.3](#73-installation-of-workload-managers)), `cmsh` can be used to modify the settings for the Slurm accounting database. Changes can be carried out in the `slurm-accounting` configuration overlay: ```bash [basecm10->configurationoverlay[slurm-accounting]]% show Parameter Value -------------------------------- --------------------------- Name slurm-accounting Revision All head nodes no Priority 500 Nodes node001,node002 Categories Roles slurmaccounting Customizations <0 in submode> ``` In the preceding overlay, `slurmdbd` runs on the accounting node(s). It is possible to run `slurmdbd` on both head nodes or on one or two compute nodes, as set in the configuration overlay. It is not possible to mix up head nodes and compute nodes for this configuration, and it is not possible to run `slurmdbd` on more than two nodes. The other Slurm accounting changes can be found within the `slurmaccounting` role. For example, if `node001` has been set during setup with `cm-wlm-setup` to be the primary accounting storage host, and HA has been enabled so that accounting runs on both nodes, then the parameter values may be displayed as follows: **Example** ```bash [basecm10->configurationoverlay[slurm-accounting]->roles[slurmaccounting]]% show Parameter Value -------------------------------- -------------------------- ha true primary node001 Name slurmaccounting Revision Type SlurmAccountingRole Add services yes DbdPort 6819 StorageHost node001 StoragePort 3306 StorageLoc slurm_acct_db StorageUser slurm Slurm Clusters slurm ``` `StorageHost`, `StoragePort`, `StorageLoc` and `StorageUser` are settings for connecting to the MySQL database, If these are for an external host, then they must be configured manually so that they are reachable by the nodes running `slurmdbd`. **Generic resources (gres) configuration in Slurm:** In order to configure generic resources, the `genericresources` mode can be used to set a list of objects. Each object then represents one generic resource available on nodes. Each value of `name` in `genericresources` must already be defined in the list of `GresTypes`. The list of `GresTypes` is defined in the `wlm` role for the instance. **Example** ```bash [basecm10->wlm[slurm]]% get grestypes gpu ``` Several generic resources entries can have the same value for `name` (for example `gpu`), but must have a unique alias. The alias is a string that is used to manage the resource entry in `cmsh` or in Base View. The string is enclosed in square brackets in `cmsh`, and is used instead of the `name` for the object. The alias does not affect Slurm configuration. For example, to add two GPUs for all the nodes in the default category which are of type `k20xm`, and to assign them to different CPU cores, the following `cmsh` commands can be run: **Example** ```bash [basecm10]% configurationoverlay use slurm-client [basecm10->configurationoverlay[slurm-client]]% roles [basecm10->configurationoverlay[slurm-client]->roles*]% use slurmclient [...[slurmclient]]% genericresources [...[slurmclient]->genericresources]% add gpu0 [...[slurmclient*]->genericresources*[gpu0*]]% set name gpu [...[slurmclient*]->genericresources*[gpu0*]]% set file /dev/nvidia0 [...[slurmclient*]->genericresources*[gpu0*]]% set cores 0-7 [...[slurmclient*]->genericresources*[gpu0*]]% set type k20xm [...[slurmclient*]->genericresources*[gpu0*]]% add gpu1 [...[slurmclient*]->genericresources*[gpu1*]]% set name gpu [...[slurmclient*]->genericresources*[gpu1*]]% set file /dev/nvidia1 [...[slurmclient*]->genericresources*[gpu1*]]% set cores 8-15 [...[slurmclient*]->genericresources*[gpu1*]]% set type k20xm [...[slurmclient*]->genericresources*[gpu1*]]% commit [...[slurmclient]->genericresources[gpu1]]% list Alias (key) Name Type Count File ----------- -------- -------- -------- ---------------- gpu0 gpu k20xm /dev/nvidia0 gpu1 gpu k20xm /dev/nvidia1 [...[slurmclient]->genericresources[gpu1]]% ``` Typically this configuration is done automatically during the GPU configuration process as outlined in the GPU Configuration Screens section [here](#7323-gpu-configuration-screens), where by default the configuration overlay is given the name `slurm-client-gpu`. In Base View, the navigation path: `Configuration Overlays` > `slurm-client-gpu` > `Edit` > `Roles` > `slurmclient` > `Edit` > `Generic Resources` > `ADD` provides the equivalent (figure [7.20](#fig-bright-view-workloadmanagement-slurmgres-nvidiaconfig)): ![Base View access to NVIDIA GPU configuration options](/base-command-manager/product-manuals/_fern-img/6bf89aa54af01be6624ab6649517dd2855a1e34d38dd392b2e024635e028388d.webp) *Figure 7.20: Base View access to NVIDIA GPU configuration options* After the generic resources are committed, BCM updates the `gres.conf` file. Since NVIDIA Base Command Manager version 8.2 and higher, a single `gres.conf` configuration file, located at `/cm/shared/apps/slurm/var/etc/slurm/gres.conf` is used. If the category consists of `node001` and `node002`, then the entries to the `gres.conf` file in this case would look like: **Example** ```bash # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia0 Cores=0-7 NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia1 Cores=8-15 # END AUTOGENERATED SECTION -- DO NOT REMOVE [root@basecm10 ~]# ``` **Slurm topology configuration:** Slurm supports topology-aware resource allocation to optimize job performance. For example, on a hierarchical network, the `topology/tree` plugin works so that Slurm allocates resources to jobs to minimize network contention. The network topology configuration for a Slurm instance \<*slurmcluster*> is described by the file: `/cm/shared/apps/slurm/var/etc/` \<*slurmcluster*>`/topology.conf` The file is generated and modified by the `topograph` service ([here](#slurm-topology-generator)) when the cluster administrator modifies the topology. The following parameters can be set to modify the topology: * `topology switches`: A Slurm cluster configuration parameter that defines the list of switches used to write the topology file. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% set topologyswitches switch01 switch02 switch03 [basecm10->wlm[slurm*]]% commit ``` * `switch ports`: The switch port definitions of a node or a switch. This information is used by BCM to construct the topology tree for `topology.conf`. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% set switchports switch01:3 [basecm10->device[node001*]]% commit [basecm10->device[node001]]% use switch01 [basecm10->device[switch01]]% set switchports switch02:15 [basecm10->device[switch01*]]% commit ``` * `SlurmTopology`: an extra node value that allows the Slurm network topology to be configured without an actual need to configure switches and switch ports in BCM. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% set node001 -e SlurmTopology switch02 switch01 [basecm10->device*]% set node002 -e SlurmTopology switch02 switch01 [basecm10->device*]% set node003 -e SlurmTopology switch03 [basecm10->device*]% commit ``` The preceding examples populate the autogenerated section of `topology.conf` as follows: **Example** ```bash SwitchName=switch01 Switches=switch02 SwitchName=switch02 Nodes=node[001,002] SwitchName=switch03 Nodes=node003 ``` The following `AdvancedConfig` directives ([here](/admin/cmdconf#c80-advancedconfig-directive)) can be used to control the topology parameters. * `SlurmStraightExtraTopology`: If set to a value of `0`, then the order of switches is reversed when setting the extra values. For example, when it is set to `0`, then to get the same content of `topology.conf` as in the preceding example, the following lines can be run: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% set node001 -e SlurmTopology switch01 switch02 [basecm10->device*]% set node002 -e SlurmTopology switch01 switch02 [basecm10->device*]% set node003 -e SlurmTopology switch03 [basecm10->device*]% commit ``` In topology/plugin configuration, switches and nodes cannot both be connected to the same switch. This is consistent with the description in `man topology.conf.5`. If the configuration in BCM has both nodes and switches connected to the same switch, then only the nodes-to-switch connection is written to `topology.conf`. * `SlurmConcatTopologySwitchName`: If set to `1`, then it allows concatenation of the switch names in the topology defined via the `SlurmTopology` node extra setting. The `SlurmTopology` setting is described earlier on. The concatenation starts in order of parent switches first. Thus switch `switch02` that is directly connected to switch `switch01` is called in `topology.conf` as `switch01-switch02`. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% set node001 -e SlurmTopology switch01 switch03 [basecm10->device*]% set node002 -e SlurmTopology switch01 switch03 [basecm10->device*]% set node002 -e SlurmTopology switch01 switch03 [basecm10->device*]% set node003 -e SlurmTopology switch01 switch03 [basecm10->device*]% set node003 -e SlurmTopology switch01 switch03 [basecm10->device*]% set node004 -e SlurmTopology switch02 switch03 [basecm10->device*]% set node005 -e SlurmTopology switch02 switch03 [basecm10->device*]% set node006 -e SlurmTopology switch02 switch03 [basecm10->device*]% commit ``` The preceding results in the following content within `topology.conf`: **Example** ```bash SwitchName=switch03 Switches=switch03-switch[01,02] SwitchName=switch03-switch01 Nodes=node[001-003] SwitchName=switch03-switch02 Nodes=node[004-006] ``` **Slurm topology generator:** The `topograph` service discovers and generates a Slurm topology configuration. The `topograph` package is available from the BCM repositories for Ubuntu 22.04 or higher, and for RHEL9-compatible distributions: **Example** ```bash root@basecm10:~# apt install topograph ... Need to get 30.1 MB of archives. After this operation, 0 B of additional disk space will be used. ... ``` After picking up the package, the service should be set up on the nodes with the `slurmserver` role. Typically this is on the head node, and if the cluster is an HA cluster (Chapter [17](/admin/ha)) then it should be set up on both head nodes. In BCM 10, the `topograph` service requires the following `AdvancedConfig` directives to be enabled: * `SlurmTopologyGenerator` ([here](/admin/cmdconf#c96-slurmtopologygenerator-directive)). * `TopographHostname` ([here](/admin/cmdconf#c97-topographhostname-directive)). The service can be set up on the head node(s) in BCM using the `services` submode of `device` mode: **Example** ```bash [basecm10->device[basecm10]]% services [basecm10->device[basecm10]->services]% add topograph [basecm10->device*[basecm10*]->services*[topograph*]]% set autostart yes [basecm10->device*[basecm10*]->services*[topograph*]]% set monitored yes [basecm10->device*[basecm10*]->services*[topograph*]]% commit [basecm10->device[basecm10]->services[topograph]]% status Service Status --------------------------- ----------- topograph [ UP ] repeated for the other head node if using HA ``` The certificates for the `topograph` service are under `/etc/topograph/ssl/`. The cluster administrator should set them up manually; BCM does not manage that configuration. After restarting CMDaemon on nodes with the `slurmserver` role with: `systemctl restart cmd` a `topology.conf` file is generated. ###### For cloud service providers managed by BCM there are some extra considerations for the `topograph` service: * For AWS clouds the compute nodes must be running (exist), and must have a flavor that supports the AWS topology API ([https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-topology.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-topology.html)). * For OCI clouds the nodes must exist in OCI, and the administrator can either: * set the advanced configuration directive ([here](/admin/cmdconf#c80-advancedconfig-directive)): `TopographPassCredentials=1` (default) or can * set the advanced configuration directive: `TopographPassCredentials=0` and specify some named dynamic group ([https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managingdynamicgroups.htm](https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managingdynamicgroups.htm)) policies in the tenancy. Each named dynamic group must have permissions to inspect certain resources. Additionally, the special BCM `thebcmnonprod-topo-dg` dynamic group must have various permissions set to use a variety of resources. The permissions can be set according to the OCI policy syntax ([https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#Writing](https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#Writing)) as follows: ```bash Allow dynamic-group to inspect compute-capacity-topologies in tenancy Allow dynamic-group to inspect compute-bare-metal-hosts in tenancy Allow dynamic-group to inspect compute-hpc-islands in tenancy Allow dynamic-group to inspect compute-network-blocks in tenancy Allow dynamic-group to inspect compute-local-blocks in tenancy Allow dynamic-group 'bcmnonprod-topo-dg' to {COMPUTE_CAPACITY_TOPOLOGY_INSPECT,\ COMPUTE_CAPACITY_TOPOLOGY_READ,COMPUTE_BARE_METAL_HOST_INSPECT,COMPUTE_BARE_METAL_HOST_READ,\ COMPUTE_NETWORK_BLOCK_READ,COMPUTE_LOCAL_BLOCK_READ,COMPUTE_HPC_ISLAND_INSPECT,\ COMPUTE_HPC_ISLAND_READ,COMPUTE_CAPACITY_TOPOLOGY_READ,COMPUTE_NETWORK_BLOCK_INSPECT,\ COMPUTE_NETWORK_BLOCK_READ,COMPUTE_LOCAL_BLOCK_INSPECT} in tenancy ``` **Slurm drain reason policy:** CMDaemon allows nodes to be drained: * either by user request, using the `drain` command (section [7.7.3](#773-nodes-drainage-status-and-handling-in-cmsh)) or * automatically, when a monitoring trigger (section [12.4.5](/admin/monitoring#1245-monitoring-configuration-triggers)) is configured. For example, if some healthcheck fails, then the appropriate node can be drained automatically. The administrator can also manually drain a node outside of `cmsh`, using the Slurm `scontrol` command from the terminal, as described in `man scontrol.1`. In all these cases the drain reason can be specified. By default, if a node is already drained, then a second drain command with a new drain reason replaces the old drain reason. The behavior can be configured with the parameter `Drain reason policy`, within the Slurm cluster settings. The `Drain reason policy` can take one of the following (case-insensitive) values: * `REPLACE`: the old drain reason(s) is replaced by a new one * `APPEND`: the new drain reason is separated by a comma, and appended to the existing one(s). * `SKIP`: * If a drain reason already exists, then setting the new drain reason is skipped. * If no drain reason already exists, then the new drain reason is applied. In the following example, the drain reason policy is set to `APPEND`. This means that CMDaemon always appends a new drain reason to any existing ones. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% set drainreasonpolicy APPEND [basecm10->wlm[slurm*]]% commit ``` It is also possible to temporarily set an append policy in `cmsh` when draining a node, by using the `+` operator as a prefix to the drain reason: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% drain --reason="+this will be appended regardless of the current policy" ``` **Slurm services restart or reconfiguration conditions:** * When parameters in the Slurm configuration files are changed by BCM via CMDaemon, typically via actions of `cmsh` or Base View, then Slurm services automatically restart, and `scontrol reconfigure` is automatically run, as required. * When parameters in the Slurm configuration files are changed directly, then BCM does not detect a change automatically, and therefore does not automatically restart or reconfigure Slurm. Slurm subsystems are restarted and reconfigured based on the following list of conditions: * `slurmctld` and `slurmd` are restarted if: 1. any of the following global `slurm.conf` parameters managed by CMDaemon are changed: * `SelectType` * `SelectTypeParameters` * `GresTypes` * `StateSaveLocation` * `SchedulerType` * `SchedulerParameters` * `SlurmctldParameters` 2. `gres.conf` is changed (by CMDaemon) 3. [`oci.conf` (`https://slurm.schedmd.com/oci.conf.html`)](https://slurm.schedmd.com/oci.conf.html) is changed (by CMDaemon). In `cmsh` this typically happens using the `ocisettings` submode under the `wlm` mode. 4. nodes are changed (by CMDaemon) in `slurm.conf` (nodes are renamed, added, or removed). * An `scontrol reconfigure` is carried out if: 1. any Slurm parameters managed by BCM, that are not mentioned in the preceding list of conditions, are changed. This is true also for the `Partition` and `Node` parameters within the `slurm.conf` file, except for the `NodeName` and `PartitionName` parameters specifically. 2. `cgroups.conf` is changed (by CMDaemon) 3. `topology.conf` is changed (by CMDaemon) A restart or a reconfigure is not done by CMDaemon under other conditions. **Slurm resource consumption monitoring configuration:** Slurm allows the configuration of how consumable resources (core, CPU, memory, etc) are tracked and shared among jobs on a node. At the time of writing of this section (November 2022), `cmsh` or Base View can be used to configure the tracking and sharing of resources as in the following text. * `SelectType`: this plugin configures the algorithm used to select resources for jobs. By default, NVIDIA Base Command Manager configures the `select/cons_tres` plugin. This is an advanced version of `select/cons_res` plugin, and it also allows GPUs to be tracked separately from other consumable resources. Further details on `cons_res` can be found in the Slurm documentation at [https://slurm.schedmd.com/cons_res.html](https://slurm.schedmd.com/cons_res.html) and [https://slurm.schedmd.com/cons_res_share.html](https://slurm.schedmd.com/cons_res_share.html). The `SelectType` parameter is configured in `cmsh` or Base View, from within the Slurm cluster configuration settings: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% set selecttype select/ select/cons_res select/cons_tres select/linear [basecm10->wlm[slurm]]% ``` * `SelectTypeParameters`: allows the configuration of the resource selection plugin that is specified for the `SelectType` parameter. The possible values depend on the plugin type. For `select/cons_tres`, the values follow the pattern: * `CR_Memory` or * `CR_` \<*COMPUTE_UNIT*>`[_Memory]` where \<*COMPUTE_UNIT*> takes one of the following values: * `Core` * `CPU` * `Socket` * `Board` The value of `SelectTypeParameters` specifies if the \<*COMPUTE_UNIT*> resource is tracked, or if the \<*COMPUTE_UNIT*> resource is tracked together with memory, or if only memory is tracked. There are also a few special values that can be applied to `SelectTypeParameters`: * `CR_ONE_TASK_PER_CORE`: allocate one task per core (used only with the `select/linear` plugin) * `CR_CORE_DEFAULT_DIST_BLOCK`: allocate cores within a node using a block distribution * `CR_LLN`: schedule resources to jobs on the least loaded nodes * `CR_Pack_Nodes`: if a job allocation contains more resources than will be used for launching tasks, then rather than distributing a job’s tasks evenly across its allocated nodes, pack them as tightly as possible on these nodes. * `OverSubscribe` ([here](/admin/configuring#31626-gpu-configuration-for-hpc-workload-managers)): * if the `select/linear` plugin is used, then the `OverSubscribe` parameter controls whether or not the nodes are shared among jobs * if the `select/cons_res` or `select/cons_tres` plugins are used, then the `OverSubscribe` parameter controls whether or not the configured consumable resources are shared among jobs. For these plugins, when a consumable resource such as a core, socket, CPU, or other, is shared, it means that more than one job can be assigned to it. The parameter is set per Slurm partition (queue) and accepts the following values: * `EXCLUSIVE`: allocate entire node for a job; * `FORCE`: makes all resources (except for GRES) in the Slurm partition available for oversubscription without any means for users to disable it (may be followed with a colon and maximum number of jobs in running or suspended state); * `YES`: makes all resources (except for GRES) in the Slurm partition available for sharing upon request by the job (may be followed with a colon and maximum number of jobs in running or suspended state); * `NO`: no resource is allocated to more than one job. * `AccountingStorageTRES`: A comma-separated list of resources that the administrator wants to track on the cluster. By default the following resources (TRES) are tracked: * billing * CPU * energy * memory * node * fs/disk * pages * vmem These default TRES cannot be disabled, but only appended to. Resources use is recorded when used on the cluster. If GPUs of different types are tracked, then job requests with matching type specifications are recorded. **Example** If:\ `AccountingStorageTRES=gres/gpu:tesla,gres/gpu:volta` is set\ then:\ `gres/gpu:tesla` and `gres/gpu:volta` track only jobs that explicitly request those two GPU types. **Slurm REST API:** `slurmrestd` is Slurm’s REST API daemon. It is installed with the BCM package `slurm23.02-slurmrestd`, or `slurm23.11-slurmrestd`. The distribution package, `slurm-slurmrestd`, should not be installed, and indeed its installation is not suggested by the package manager in a cluster that is correctly configured. **Slurm REST API package installation and version matching:** If installing the REST API package, then care must be taken to match the versions exactly to its sibling Slurm packages ([here](#7511-slurm-packages)). For example, if the already-installed packages have version 23.02: **Example** ```bash root@basecm10 ~]# rpm -qa | grep slurm #what are the already installed packages? slurm23.02-23.02.6-100806_cm10.0_9c8dc03511.x86_64 slurm23.02-perlapi-23.02.6-100806_cm10.0_9c8dc03511.x86_64 slurm23.02-slurmdbd-23.02.6-100806_cm10.0_9c8dc03511.x86_64 slurm23.02-contribs-23.02.6-100806_cm10.0_9c8dc03511.x86_64 slurm23.02-client-23.02.6-100806_cm10.0_9c8dc03511.x86_64 slurm23.02-devel-23.02.6-100806_cm10.0_9c8dc03511.x86_64 ``` and if the Slurm REST API package version to be installed is as indicated by the following: **Example** ```bash root@basecm10 ~]# yum info slurm23.02-slurmrestd | egrep '(Name|Release|Source)' #what will install? Name : slurm23.02-slurmrestd Release : 100807_cm10.0_b3c3ec09a3 Source : slurm23.02-23.02.7-100807_cm10.0_b3c3ec09a3.src.rpm ``` then it means that all already-installed Slurm packages must be upgraded to release `100807_cm10.0_b3c3ec09a3` too. The reason behind matching versions exactly is that the upstream versions can have significant configuration changes between subminor versions. These changes may result in a non-functioning workload manager. This is true, not only when considering the Slurm REST API package, but also true when considering the other Slurm packages. **Slurm REST API and authentication:** The slurmrestd service can be configured with JWT ([JSON Web Token](https://tools.ietf.org/html/rfc7519)) authentication as follows: 1. The JWT key is generated in the Slurm cluster configuration directory, and its ownership and permissions are then changed: **Example** ```bash [root@basecm10 ~]# module load slurm # Loads SLURM_CONF environment variable from modulefile [root@basecm10 ~]# JWT_KEY=`dirname $SLURM_CONF`/jwt.key [root@basecm10 ~]# install -m 0600 -o slurm -g slurm <(dd if=/dev/random bs=32 count=1) $JWT_KEY ``` The location of the key can differ, but user `slurm` must have read access to the file. 2. The JWT plugin is configured in `slurm.conf` for the Slurm cluster instance. For example, for a Slurm cluster instance \<*cluster name*>: **Example** ```bash AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/cm/shared/apps/slurm/var/etc//jwt.key ``` The `slurmctld` service is restarted to apply the plugin settings. For example, if it is running on the head node: ```bash [root@basecm10 ~]# systemctl restart slurmctld.service ``` 3. A `systemd` drop-in file is created for the new `slurmrestd` service: **Example** ```bash [root@basecm10 ~]# slurmrestapidir="/etc/systemd/system/slurmrestd.service.d" [root@basecm10 ~]# mkdir $slurmrestapidir [root@basecm10 ~]# echo "[Service]" > $slurmrestapidir/99-cmd.conf [root@basecm10 ~]# echo "Environment=SLURM_CONF=$SLURM_CONF" >> $slurmrestapidir/99-cmd.conf ``` Future versions of BCM may automatically create or update the file `99-cmd.conf`. It is therefore recommended to use that file name as a best practice for cluster administration. 4. The `slurmrestd` service is configured within `cmsh` so that it can be monitored by BCM: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device services master [basecm10->device[basecm10]->services]% add slurmrestd [basecm10->device*[basecm10*]->services*[slurmrestd*]]% set monitored yes [basecm10->device*[basecm10*]->services*[slurmrestd*]]% set autostart yes [basecm10->device*[basecm10*]->services*[slurmrestd*]]% commit [basecm10->device[basecm10]->services[slurmrestd]]%. ``` 5. A ping-like `curl` check can be run with custom extra headers to ensure that the Slurm REST API service is running properly, as follows: **Example** ```bash [root@basecm10 ~]# export $(scontrol token username=cmsupport) [root@basecm10 ~]# curl 0.0.0.0:6820/openapi \ -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \ -H "X-SLURM-USER-NAME: cmsupport" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 { "tags": [ { "name": "slurm", "description": "methods that query slurmdbd" }, { "name": "openapi", "description": "methods that query for OpenAPI specifications" } [...] ``` The `slurmrestd` daemon configuration is specified by its command line arguments. These can be made permanent in `/etc/sysconfig/slurmrestd`. **GPU autodetection:** Slurm supports GPU autodetection from version 20 onward, using the `AutoDetect` flag in `gres.conf`. Slurm GPU autodetection allows AMD, Intel, and NVIDIA GPUs to have their GPU parameters be set automatically, if autodetection is also enabled in [`gres.conf`](https://slurm.schedmd.com/gres.conf.html). Slurm GPU autodetection removes the need for explicit GPU configuration in `gres.conf`. The number of GPUs must however still be specified explicitly by the administrator. This is because at the time of writing (October 2024) Slurm has no GPU autodetection by default. In BCM, the Slurm GPU hardware detection setting has 7 options, as covered [here](#slurm-hardware-autodetection-for-gpus). * Setting the `bcm` option for Slurm GPU autodetection removes the need to specify any further GPU autodetect option. * Setting a GPU vendor (NVIDIA, Intel, AMD) option for Slurm GPU autodetection means that the `Gres=...` parameter in `NodeName` lines in `slurm.conf` is still required in order to tell `slurmctld` how many GRES to expect. This means that the administrator must define GPUs in BCM, in the `genericresources` mode of the `slurmclient` role, if that has not already been specified during `cm-wlm-setup`. However there is no need in this case to specify all the details. It is enough to add a single generic resource named `gpu`, and to specify the number of such GPUs. In order to skip adding the generic resources to `gres.conf`, while still allowing GRES information to be added the `slurm.conf` file, the flag `AddToGresConfig` should be set to `no` in the `genericresource` entity. For example: **Example** ```bash [basecm10]% configurationoverlay use slurm-client [basecm10->configurationoverlay[slurm-client]]% roles [basecm10->configurationoverlay[slurm-client]->roles]% use slurmclient [...[slurmclient]]% genericresources [...[slurmclient]->genericresources]% add autodetected-gpus [...[slurmclient*]->genericresources*[autodetected-gpus*]]% set name gpu [...[slurmclient*]->genericresources*[autodetected-gpus*]]% set count 8 [...[slurmclient*]->genericresources*[autodetected-gpus*]]% set addtogresconfig no [...[slurmclient*]->genericresources*[autodetected-gpus*]]% commit [...[slurmclient]->genericresources[autodetected-gpus]]% wlm use slurm [basecm10->wlm[slurm]]% set gpuautodetect nvml [basecm10->wlm*[slurm*]]% commit [basecm10->wlm[slurm]]% ``` Initial generic resources configuration for Slurm GPU autodetection can also be done during Slurm setup by `cm-wlm-setup`. If GPU configuration is chosen in the TUI screen of `cm-wlm-setup` (figure [7.21](#fig-cm-wlm-setup-configure-gpus-screen)): ![The screen prompting for GPU configuration in cm-wlm-setup](/base-command-manager/product-manuals/_fern-img/dd25a82e017ff1e9058253a83d5176e4d6085a0fb1ee151ef782651980850f1d.webp) *Figure 7.21: The screen prompting for GPU configuration in cm-wlm-setup* then the TUI allows Slurm GPU autodetection options to be selected (figure [7.22](#fig-cm-wlm-setup-gpu-autodetect-options-screen)): ![The GPU configuration screen of cm-wlm-setup](/base-command-manager/product-manuals/_fern-img/b52da4ff53650965092f58ca3d25bb05e73717f2460de2163bcfc2336b93e26d.webp) *Figure 7.22: The GPU configuration screen of cm-wlm-setup* The options are the same as those described in the list of 7 options for `GPU auto detect` ([here](#7514-configuring-slurm)). The options result in the following actions: * `BCM autodetects GPUs`: BCM automatically detects numbers and types of GPUs on the selected nodes and configures `slurm.conf` appropriately. The administrator is not asked to configure anything. This option does not work if the nodes have GPUs from a mix of vendors. * `Slurm autodetects` \<*vendor*>` GPU details`: the administrator is prompted to specify numbers and types of GPUs later on. These values are then added to the `Gres` parameter of `NodeName` lines in `slurm.conf`, while Slurm autodetects the rest of the GPU details. * `Manual GPU configuration in cm-wlm-setup`: the administrator is prompted to enter the GPU parameters manually. The manual entries are then converted to appropriate lines in `gres.conf`. **Slurm GPU cluster settings:** When a GPU is configured the following parameters can be configured with BCM:
**Base View** **Slurm Object Value Within** **Description**
**Option** **WLM Mode Of `cmsh`**
`Select Type` `SelectType` Identifies the type of resource selection algorithm to be used. Changing this value causes the `slurmctld` daemon to be restarted. Default: Consumable trackable resource: `select/cons_tres`
`Select Type Parameters` `SelectTypeParameters` Parameters for the resource selection algorithm. Acceptable values for these parameters depend upon the algorithm selected. Default: Core as a consumable resource: `CR_Core`
`Accounting Storage TRES` `AccountingStorageTRES` Comma-separated list of resources to be tracked on the cluster, requested by an `sbatch` / `srun` job on submission. Default: GPU as a generic resource: `gres/gpu`
**Example** ```bash [basecm10->wlm[slurm]]% get selecttype select/cons_tres ``` **MIG and Slurm:** NVIDIA’s Multi-Instance GPU (MIG) technology is a way to optimize GPU use. MIG allows a single GPU to be partitioned into up to 7 separate logical GPU instances, each providing separate GPU resources to user jobs. Slurm is MIG-capable since version 21.08. BCM automates the configuration of MIG profiles as GPU generic resource types. This means that a user can request GPUs by type, and Slurm then tracks the MIG profiles usage. MIG detection on Slurm compute nodes (nodes with the `slurmclient` role assigned) means that the `slurm.conf` and `gres.conf` files are automatically configured by CMDaemon as follows: * `slurm.conf`: 1. For each node, `NodeName` parameter lines have a `Gres` specification (generic resource type specification) with the MIG profile names appended as strings. The strings take the format: ` gpu:` \<*profile*>`:<`*number*> where \<*number*> is the number of instances with the profile name of `gpu:` \<*profile*>. For example: 1. A GPU on `node001` is partitioned into two MIG instances, one with a profile name of `3g.40gb`, and the other with a profile name of `4g.40gb`: **Example** ```bash NodeName=node001 ... Gres=gpu:3g.40gb:1,gpu:4g.40gb:1 ... ``` 2. A GPU on `node001` is partitioned into two MIG instances, both with same profile name `3g.20gb`. Also present is a regular GPU, `a40`, that is configured manually as a generic resource: **Example** ```bash NodeName=node001 ... Gres=gpu:a40:1,gpu:3g.20gb:2 ... ``` 2. The `GresTypes` parameter line has the MIG profiles from all the compute nodes. **Example** ```bash GresTypes=3g.40gb,4g.40gb,gpu ``` 3. The `AccountingStorageTRES` parameter line includes the new generic resource per MIG profile. **Example** ```bash AccountingStorageTRES=gres/gpu,gres/gpu:3g.40gb,gres/gpu:4g.40gb ``` * `gres.conf`: the `AutoDetect` parameter is set to `NVML` for the nodes where MIGs are detected. **Example** ```bash NodeName=node[001-016] AutoDetect=NVML ``` In this case Slurm detects the MIG details (links, file, cores, and so on) automatically. If the GPUs are repartitioned by BCM, then the Slurm configuration is by default automatically updated with the new GPU types according to the new MIG profiles. The default value of `yes` can be modified with the `configuremigs` parameter in Slurm cluster settings: **Example** ```bash [basecm10->wlm[slurm]]% set configuremigs no; commit ``` The administrator can validate that the MIG devices are correctly recognized by Slurm, by checking the `Gres` line output of the `scontrol show node` command: **Example** ```bash [root@basecm10 ~]# scontrol show node node001 | grep "Gres=" Gres=gpu:4g.40gb:1(S:0),gpu:3g.40gb:1(S:0) ``` The presence of a socket value such as `S:0` indicates that Slurm detected the MIG details on the node as expected. **Slurm node settings:** The parameter value for a Slurm option in `slurm.conf` is set by CMDaemon, if its value is not: `0`. A parameter value of `0` means that the default values of Slurm are used. These usually have the value: `0`. The advanced options that CMDaemon manages for Slurm are:
**Base View Option** **Slurm Option** **Description**
`Features` `Feature=<` *string* `>` entry in the file `slurm.conf` Arbitrary strings can be entered to indicate some characteristics of a node, one string per entry. For example: `text1` `text2` and so on. These become part of the: `Feature=text1,text2...` attribute to the `NodeName=<` *`node name`* `>` entry line in `slurm.conf` , as indicated in `man slurm.conf.5` . The strings also become added attributes to the `GresTypes` entry of that file. Default: blank.
`File=<` *device* `>` entries in the file `gres.conf` This is only implemented if the number of GPUs has been set by the administrator using BCM, by setting the basic Slurm client option `GPUs` . The `GPUs` option is seen, for example, in Base View, via the navigation path `Grouping` \> `Node categories[default]` \> `Edit` \> `Settings` `Roles` \>
`slurmclient` \> `GPUs` . Each device detected on the host node can be added as an entry. For example: `/dev/nvidia0` `/dev/nvidia1` and so on. Alternatively, they can be added as a range in one line, for example: `/dev/nvidia[0-1]` These become part of the: `Name=gpu File=<` *device* `>` lines in `gres.conf` , as indicated in `man gres.conf.5` . Default: blank
`CPU` Number of logical processors on the node. For Slurm 20 and beyond, CMDaemon detects the number of CPU cores, and sets `procs=` *`number of cores`* via slots autodetection. Default for Slurm prior to version 20: `0`
`Sockets` `Sockets` Processor chips on node. If this is defined, then `SocketsPerBoard` must not be defined. Default: `0`
`Cores per socket` `CoresPerSocket` Number of cores per socket. Default: `0`
`ThreadsPerCore` `ThreadsPerCore` Number of logical threads for a single core. Default: `0`
`Boards` `Boards` Number of baseboards in a node. Default: `0`
`SocketsPerBoard` `SocketsPerBoard` Number of processor chips on baseboard. If this is defined, then `Sockets` must not be defined. Default: `0`
`RealMemory` `RealMemory` Size of real memory on the node, MB. Default: `0`
`NodeHostname` `NodeHostname` Default: as defined by Slurm’s `NodeName` parameter.
`NodeAddr` `NodeAddr` Default: as set by Slurm’s `NodeHostname` parameter.
`State` `State` State of the node with user jobs. Possible Slurm values are: `DOWN` , `DRAIN` , `FAIL` , `FAILING` , and `UNKNOWN` . Default: `UNKNOWN`
`Weight` `Weight` The priority of the node for scheduling. Default: `0`
`Port` `Port` Port that slurmd listens to on the compute node. Default: as defined by `SlurmdPort` parameter. If `SlurmdPort` is not specified during build: Default: `6818` .
`TmpDisk` `TmpDisk` Total size of Slurm’s temporary filesystem, `TmpFS` , typically `/tmp` , in MB. `TmpFS` is the storage location available to user jobs for temporary storage. Default: `0`
`Options` `extra options` Extra options that are added to `slurm.conf`
Further Slurm documentation is available: * via `man` pages under `/cm/shared/apps/slurm/current/man/` * as HTML documentation in the version dependent directory of the form: `/cm/shared/apps/slurm/current/share/doc/slurm-` \<*version*>`/html`, or * at the Slurm website at [http://slurm.schedmd.com/documentation.html](http://slurm.schedmd.com/documentation.html) Slurm is set up with reasonable defaults, but administrators familiar with Slurm can reconfigure the configuration file using a web browser. The web browser can be used, depending on the Slurm version being used, with one of the following paths: * `/cm/shared/apps/slurm/current/share/doc/slurm-23.02.8/html/` or * `/cm/shared/apps/slurm/current/share/doc/slurm-23.11.11/html/` The choice of JavaScript-based configuration generators is then: * `configurator.easy.html`: for a simplified configurator * `configurator.html`: for a full version of the configurator. If the configuration file becomes mangled beyond repair, the original default can be regenerated once again by re-installing the Slurm package, then running the script `/cm/shared/apps/slurm/var/cm/cm-restore-db-password`, and then running `cm-wlm-setup`. Care must be taken to avoid duplicate parameters being set in the configuration file—`slurmd` may not function correctly in such a configuration. #### 7.5.1.5 Advanced Slurm Job Accounting PromQL queries can be run and then filtered by Slurm job labels with BCM. The labels are taken either from a job comment or from an account name used by the job. After the labels for the job are extracted, the labeled entity is stored in the monitoring data. The administrator can then run PromQL queries, and filter results by label. Examples of such PromQL queries are the `job_gpu_wasted` or `job_gpu_utilization` queries (section [14.4.1](/admin/monitoringaccounting#1441-two-job-gpu-metrics-used-in-promql-queries)). `Extract accounting info` must be set to `yes` to enable labels to be extracted from job comments or account names: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% accounting [basecm10->wlm[slurm]->show]% accounting Parameter Value -------------------------------- ------------------------------------------------ Managed hierarchy Separator _ Job comment fields Extract accounting info no [basecm10->wlm[slurm]->accounting]% set extractaccountinginfo yes [basecm10->wlm*[slurm*]->accounting*]% commit ``` In addition, the administrator must configure the job comment label format and the format of Slurm account names, if they exist. Without that configuration, BCM cannot not parse the labels. **Job comment labels:** A user running the job can specify the label for the job comment in a special format. Such a label could be, for example, a tag to indicate the artificial neural network model, where different groups of jobs use different models. The administrator (after having enabled accounting information extraction) can set a regex for the tag in a `Job comment field` parameter in `accounting` submode: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% accounting [basecm10->wlm[slurm]->accounting]% set jobcommentfields "model=__model:([^\s]+)" [basecm10->wlm*[slurm*]->accounting*]% commit ``` In the preceding example, the label `model` is set in the monitoring data when the job comment includes the string `__model:` \<*name*> where \<*name*> is any word started with the # character. **Account name labels:** Several labels can be assigned by BCM to a job parsed from an associated account name. The labels can be used to represent a hierarchy of organizational entities, such as department, project, team, and so on. The following configuration options allow this representation: 1. Managed hierarchy: representation of the account name as a list of organizational entity labels. 2. Separator: a separator for the labels in the account names. By default this is the underscore character, \_. The last entity is separated with double separator. This means that the last entity (and only the last entity) can have a single separator in the name itself. For instance, if Slurm account names use the following format: ```bash DEPARTMENT_PROJECT_TEAM ``` then `Managed hierarchy` values can be set as follows: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use slurm [basecm10->wlm[slurm]]% accounting [basecm10->wlm[slurm]->accounting]% set managedhierarchy department project team [basecm10->wlm*[slurm*]->accounting*]% commit ``` It is up to the administrator to create the accounting hierarchy in Slurm, and set names as described by the `Managed hierarchy` parameter. **Example** ```bash [root@basecm10 ~]# module load slurm [root@basecm10 ~]# sacctmgr add account NVIDIA Description="NVIDIA account" Organization=NVIDIA parent=root -i [root@basecm10 ~]# sacctmgr add account Department1 Organization=NVIDIA parent=NVIDIA -i [root@basecm10 ~]# sacctmgr add account Department1_Project1 Organization=NVIDIA parent=Department1 -i [root@basecm10 ~]# sacctmgr add account Department1_Project1__Team_1 Organization=NVIDIA parent=Department1 -i [root@basecm10 ~]# sacctmgr add account Department1__Team_2 Organization=NVIDIA parent=Department1 -i ``` In the preceding example two teams have been made. Team 1 is defined according to the full entities hierarchy (department, project, team), while Team 2 is defined only under a department. BCM correctly recognizes such "clipped" hierarchy locations for a team in the account names. Similar to the Slurm command `sshare`, an administrator can display the Slurm account hierarchy with the `fairshare` command in `cmsh`: **Example** ```bash [basecm10->wlm[slurm]]% fairshare -t Account fairshare raw_shares raw_usage norm_shares norm_usage parent ----------------------------- ------------ ---------- ---------- ----------- ---------- ------------------- root (top account) 0.0 0 0 0.0 0.0 nvidia 0.0 1 0 0.5 0.0 root (top account) department1 0.0 1 0 1.0 0.0 nvidia department1__team_2 0.0 1 0 0.333333 0.0 department1 department1_project1 0.0 1 0 0.333333 0.0 department1 department1_project1__team_1 0.0 1 0 0.333333 0.0 department1 root 1.0 1 0 0.5 0.0 root (top account) [basecm10->wlm[slurm]]% fairshare -f "" root ├─nvidia │ └─department1 │ ├─department1__team_2 │ ├─department1_project1 │ └─department1_project1__team_1 └─root ``` The following command options can be used: 1. `-t|--table`: print accounts fairshare parameters as a table 2. `-a|--account `: print information for specific account 3. `-f|--fields `: display only specified fairshare account parameters (comma-delimited) Format: ```bash [%[-]][,[%[-]],...] ; ``` where `MIN` and `MAX` are minimum and maximum value lengths. The parameter can be combined with others. The following example uses labels from both job comment and account names: **Example** ```bash [root@basecm10 ~]# su - alice [alice@basecm10 ~]$ module load slurm [alice@basecm10 ~]$ sbatch -A department1_project1__team_1 --comment="__model:llm123" --wrap="hostname" Submitted batch job 1 [alice@basecm10 ~]$ ``` PromQL queries with these labels are run later on. In `cmsh` the administrator can validate how BCM parses the job labels from both job comment and from the Slurm account name: ```bash [basecm10->wlm[slurm]->jobs]% info 1 | grep "accounting info" -i Accounting info {"department":"department1","model":"llm123","project":"project1","team":"team_1"} [basecm10->wlm[slurm]->jobs]% ``` Jobs information can also be displayed by filtering using the labels (some output elided for clarity): ```bash [basecm10->wlm[slurm]->jobs]% filter -i team="team_1" Job ID Job name User Queue Submit time Start time End time Nodes Exit code ------- ---------- ------ ------- ------------ ----------- --------- --------- ---------- 1 wrap alice 13:28:46 13:28:46 13:28:46 1 2 wrap alice defq 13:28:59 13:28:59 13:30:00 node001 0 3 wrap alice defq 13:30:39 13:30:39 13:31:40 node001 0 [basecm10->wlm[slurm]->jobs]% ``` The following query filters the team: ```bash [basecm10->monitoring->labeledentity]% instantquery "job_gpu_wasted{team=\"team_1\"}[1h]" Name category department group hostname job_id job_name project queue team user wlm ... --------------- -------- ------------ ------ -------- ------- -------- --------- ------ ------- ------ ----- ... job_gpu_wasted default department1 alice node001 1 wrap project1 defq team_1 alice slurm ... job_gpu_wasted default department1 alice node002 2 wrap project1 defq team_1 alice slurm ... ``` The following query filters the model name: ```bash [basecm10->monitoring->labeledentity]% instantquery "job_gpu_wasted{model=\"llm123\"}[1h]" Name category group hostname job_id job_name model queue user wlm ... --------------- -------- ------ -------- -------- -------- -------- -------- ------ ----- ... job_gpu_wasted default alice node001 1 wrap foo defq alice slurm ... job_gpu_wasted default alice node002 2 wrap foo defq alice slurm ... ``` **Slurm NVIDIA Sharp settings:** The Sharp plugin can be enabled by setting the following: * `SelectType` to `select/select/nvidia_sharp` * `SelectTypeParameters` with these mandatory parameters: * CR_CPU or CR_Core or CR_Socket * OTHER_CONS_TRES * CR_NVIDIA_SHARP_V3 * The Topology plugin must be set to `topology/tree` ([https://slurm.schedmd.com/topology.conf.html](https://slurm.schedmd.com/topology.conf.html)) * the value for `Licenses` must be set: **Example** ```bash Licenses=pod01:32,ibcleaf01-01:8,ibcleaf01-02:8,ibcleaf01-03:8,ibcleaf01-04:8 ``` In the preceding, `pod01`, `ibcleaf01-01`, `ibcleaf01-02`, `ibcleaf01-03`, and `ibcleaf01-04` are switches. The number that follows after the switch name is the Sharp allocation, or how many jobs are allowed to use that switch. #### 7.5.1.6 Running Slurm Slurm can be disabled and re-initialized with the `cm-wlm-setup tool` (section [7.3](#73-installation-of-workload-managers)) during package installation itself. Alternatively, role assignment and role removal can be used to adjust what nodes, if any, run Slurm. The assignment and removal of roles can be carried out from Base View (section [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)) or `cmsh` (section [7.4.2](#742-enabling-and-disabling-a-workload-manager-with-cmsh)). The Slurm workload manager runs these daemons: 1. as servers: 1. `slurmdbd`: The database that tracks job accounting. It is part of the `slurmdbd` service. 2. `slurmctld`: The controller daemon. Monitors Slurm processes, accepts jobs, and assigns resources. It is part of the `slurm` service. 3. `munged`: The authentication (client-and-server) daemon. It is part of the `munge` service. 2. as clients: 1. `slurmd`: The compute node daemon that monitors and handles tasks allocated by `slurmctld` to the node. It is part of the `slurm` service. 2. `slurmstepd`: A temporary process spawned by the `slurmd` compute node daemon to handle Slurm job steps. It is not initiated directly by users or administrators. 3. `munged`: The authentication (client-and-server) daemon. It is part of the `munge` service. Logs for the daemons are saved on the node that they run on. Accordingly, the locations are: * `/var/log/slurmdbd` * `/var/log/slurmd` * `/var/log/slurmctld` * `/var/log/munge/munged.log` ### 7.5.2 Installing, Configuring, And Running GE The workload manager, Grid Engine (GE), is a further development of the venerable SGE. Grid Engine should be picked up directly from the Altair website at [https://www.altair.com/grid-engine](https://www.altair.com/grid-engine) The installation and integration of GE into NVIDIA Base Command Manager 10 can be carried out as in the following steps: 1. These GE tar.gz file collections should be downloaded from the Altair website to a directory on the head node: * Binary files: * The 64-bit Linux bundle * `ge-` \<*uge_ver*>`-bin-lx-amd64.tar.gz` * or the 32-bit Linux bundle * `ge-` \<*uge_ver*>`-bin-lx-x86.tar.gz` * Common files: * `ge-` \<*uge_ver*>`-common.tar.gz` Here \<*uge_ver*> is the GE version, for example: `8.6.0` To avoid installation issues, a check should be done to ensure that during the download, the `tar.gz` files have not been renamed, or that their names not been changed to upper case. Both packages must be located in the same directory before installation. If a failover setup already exists, then the installation should be done on the active head node in the steps that follow. 2. The `cm-uge` package should be installed from the BCM repository via the distribution’s package manager. That is, via, `yum`, `zypper`, or `apt`, as appropriate. The package installs, amongst others, the following template files under `/cm/shared/apps/uge/var/cm/`: 1. An environment module template, `uge.module.template` 2. An installation configuration template, `inst_template.conf.template` 3. Some other template files for a default GE configuration, under the directory `templates` The templates decide the configuration of GE (section “*Configuring GE*”, [here](#7521-configuring-ge)). The original files [7.5.2](#752-installing-configuring-and-running-ge) and [7.5.2](#752-installing-configuring-and-running-ge), with the `.template` suffix, should never be modified by the administrator. The administrator can change the parameters affected by the installation configuration from their default values by copying these original files to the same directory without the `.template` suffix, and then editing the `.conf` file: **Example** ```bash [root@basecm10 ~]# cd /cm/shared/apps/uge/var/cm [root@basecm10 cm]# cp inst_template.conf.template inst_template.conf [root@basecm10 cm]# vi inst_template.conf ``` The copied file, `inst_template.conf`, can be changed by the administrator to decide the location of GE software and how GE handles jobs, if needed. The changed file overrides the settings suggested by the `.template` file, when the BCM utility `cm-wlm-setup` runs in the next step of the installation procedure. Some of the values in the key-value pairs in the file are enclosed by percentage signs, %. For example: `CELL_NAME="%CELL_NAME%"`. Such flagged values should not be modified, since they are replaced by BCM values by `cm-wlm-setup` during installation. Values not enclosed by percentage signs are not replaced by `cm-wlm-setup`. Such unflagged values can, if needed, be tuned in the copied file by the administrator. These values are then kept by `cm-wlm-setup` during installation, and used when GE is run. 3. `cm-wlm-setup` is run. The directory where the downloaded GE files are located can be specified with the `--archives-location` option, if using the CLI mode with options. **Example** ```bash root@basecm10:~# ls -al /root/uge/ total 67892 drwxr-xr-x 2 root root 82 Dec 18 16:03 . drwx------ 10 root root 4096 Dec 18 16:03 .. -rw-r--r-- 1 root root 63426260 Dec 18 16:03 ge-8.6.7-demo-bin-lx-amd64.tar.gz -rw-r--r-- 1 root root 6088929 Dec 18 16:03 ge-8.6.7-demo-common.tar.gz root@basecm10:~# cm-wlm-setup --wlm uge --setup --archives-location /root/uge ``` 4. The nodes are rebooted. The GE command `qhost` then displays an output similar to: **Example** ```bash [root@basecm10 ~]# module load uge [root@basecm10 ~]# qhost HOSTNAME ARCH NCPU NSOC NCOR NTHR NLOAD ... -----------------------------------------------... global - - - - - - ... node001 lx-amd64 8 1 1 8 0.01 ... node002 lx-amd64 8 1 1 8 0.01 ... node003 lx-amd64 8 1 1 8 0.01 ... node004 lx-amd64 8 1 1 8 0.01 ... ``` The output in the preceding example has been truncated for this manual, for convenience. #### 7.5.2.1 Configuring GE After installation and initialization, GE has reasonable defaults, with `$SGE_ROOT` set to `/cm/shared/apps/uge/current`. By default, the GE application is installed in `/cm/shared/apps/uge/current`, and job examples are kept in `/cm/shared/examples/workload/sge/jobscripts/`. #### 7.5.2.2 Running GE The GE workload manager runs the following two daemons: 1. an `sge_qmaster` daemon running on the head node. This handles queue submissions and schedules them according to criteria set by the administrator. 2. an `sge_execd` execution daemon running on each compute node. This accepts, manages, and returns the results of the jobs on the compute nodes. Messages from the qmaster daemon are logged in: `/cm/shared/apps/uge/current/` \<*uge instance name*>`/spool/messages`\ On the associated compute nodes the execution log `messages` exists, alongside other job tracking files and directories, at:\ `/cm/local/apps/uge/var/spool/node<`number`>/messages` where `node<`number`>` is the node name, for example: `node001`, `node002` … #### 7.5.2.3 GE Submission, Administrative And Execution Hosts GE is still similar to SGE in many ways, but is evolving. In GE terminology: * A *submission host*, or *submit host*, is a node that is allowed to submit jobs. Submission can be from the user, or from another job run by the user. * An *execution host*, or *exec host*, is a node on which the computational part of the job is executed. * An *administrative host*, or *admin host*, is a node that can carry out administrative Grid Engine commands. **Standard BCM Use** In a standard BCM configuration, a *submission*, *administrative* or *execution* host in GE is configured with the following `ugeclient` role properties: 1. Admin Host – adds the node to a list of administrative hosts in GE, 2. Exec Host – adds the node to a list of execution hosts in GE, 3. Submit Host – adds the node to a list of submission hosts in GE. The nodes that are assigned a `ugeserver` role are always automatically added to the administrative and submission host lists. **Non-standard BCM Use** For non-standard configurations, where the GE service or login service is managed outside of BCM control, advanced configuration directives allow the cluster to be aware of the situation. 1. If the `ugeclient` role cannot be added, for instance due to the GE service being managed outside of BCM, then the additional execution hosts can be specified in the advanced configuration directive, `AdditionalExecHosts`, in the `cmd.conf` file: **Example** ```bash AdvancedConfig = {"AdditionalExecHosts=node002,login01"} ``` 2. If the `ugeclient` role cannot be added, for instance due to the login service being managed outside of BCM, then the execution hosts can be specified in the advanced configuration directive, `AdditionalSubmitHosts`, in the `cmd.conf` file: **Example** ```bash AdvancedConfig = {"AdditionalSubmitHosts=node002,login01"} ``` #### 7.5.2.4 Parallel Environments A parallel environment (PE) allows the configuration of applications that use shared or distributed memory. The main—but not sole—purpose of a PE is to provide a job environment within which MPI applications can run, using various MPI libraries. PEs can be provided by the cluster software or hardware vendor, as well as distributed with GE itself. Default PEs are provided for various MPI libraries, such as Open MPI, MVAPICH, and so on. The default PEs are automatically added to GE when `cm-wlm-setup` installs GE. The CMDaemon front ends, i.e. `cmsh` or Base View, can be used to add, remove, or modify the existing PE. Each GE cluster has a `Parallel Environments` submode. In the `Parallel Environments` submode, each PE is represented as a separate object with associated properties. For example, with `cmsh` the `wlm` mode is used to access the Open MPI properties within a GE cluster (some text elided): **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm [basecm10->wlm]% list Type Name (key) Server nodes Submit nodes Client nodes ------- ----------- ------------ ------------------------------------------------ ----------------- UGE uge basecm10 node001..node003,basecm10 node001..node003 [basecm10->wlm]% use uge [basecm10->wlm[uge]]% parallelenvironments [basecm10->wlm[uge]->parallelenvironments]% list Name (key) Slots Allocation Rule ----------------- -------- ---------------- impi 999999 $round_robin mpich 999999 $round_robin mpich2 999999 $round_robin openmpi 999999 $round_robin openmpi_rankfile 999999 4 [basecm10->wlm[uge]->parallelenvironments]% use openmpi [basecm10->wlm[uge]->parallelenvironments[openmpi]]% show Parameter Value -------------------------------- ------------------------------------------------ Daemon Forks Slaves yes Master Forks Slaves no Revision Type UGEParallelEnvironment Name openmpi Slots 999999 User Lists NONE X User Lists NONE Start Procedure Arguments NONE Stop Procedure Arguments NONE Allocation Rule $round_robin Control Slaves yes Job Is First Task no Urgency Slots min Accounting Summary yes Per PE Task Prolog NONE Per PE Task Epilog NONE Extra Parameters [basecm10->wlm[uge]->parallelenvironments[openmpi]]% ``` The values shown in the preceding example are defaults. #### 7.5.2.5 Database Recovery By default, BCM configures GE to use flat file, or classic spooling as its format for the `sge_qmaster` spool. Typically this can handle thousands of jobs a day without running into performance issues, and it is the current (May 2015) recommendation for stability reasons. For larger clusters it is possible to use Berkeley DB as the spooling format. This comes with a number of utilities, some of which can be useful for GE spool maintenance and debugging, such as `db_recover`, `db_verify`, and others. The Berkeley DB utilities that come with GE are modified versions of the ones that are shipped with with the parent distribution Berkeley DB packages. The GE versions include extra sanity tests for the GE spool database, and it is best to use them instead of the standard versions. The utilities can be found in on the head node under: `/cm/shared/apps/uge/current/utilbin/lx-amd64/` This directory is not added to the `$PATH` when loading the GE environment with `module load uge`. The full Berkeley DB documentation for these tools is part of the GE distribution. A local HTML format copy of the documentation is available at: `/cm/shared/apps/uge/current/doc/berkeleydb/db_*.html` #### 7.5.2.6 GPU Management When managing host resources such as GPUs, GE 8.1.0 and higher allows the accelerators to be configured via a resource map, RSMAP. This is a complex (`man complex.5`) value type defined on the host. RSMAP values restrict how much of a resource is used concurrently from a host, attaches identifiers to the resource used, and assigns the identifiers to the jobs when they get dispatched to that host. BCM automatically configures GE execution hosts with `gpu` RSMAP resources. The `ugeclient` role can configure GPUs by using the following `cmsh` parameters: * `GPU devices`: a list of GPU names that are attached to a job by GE. * `Gpus`: the number of GPUs on a host. The `GPU devices` parameter has a higher priority than `Gpus`, so that if names are set in `GPU devices`, then they are always used. GPUs can also be bound to particular CPU sockets and CPU cores, using a *topology mask*. The mask is defined with a set of characters. In principle, the mask is applied to the computing units available for a device. The computing units correspond to processing units, and are grouped as CPU sockets, cores, and hardware threads. These correspond to the letters S, C, T. In the mask. An uppercase character allows a device to use a computing unit, while a lowercase character forbids it, as indicated by the following table:
*Table [7.5.2.6](#7526-gpu-management): Mask Composition*
**Unit To Mask** **Enable Unit with** **Disable Unit with**
Socket S s
Core C c
Hardware Thread T t
In practice, T and t are ignored. Some mask examples: * `SccSCC`: This is a two-core two-socket system. The first socket cannot be used, while both cores on the second socket can be used. * `SCcCCSccCC`: This is a four-core two-socket system. The first socket has its first, third, and fourth cores available, and the second socket has its third and fourth cores available. Some configuration examples from `cmsh` and the corresponding RSMAP complex attribute values on the host: * Naming 2 GPUs: ```bash [root@basecm10 ~]# cmsh [basecm10]% category roles default [basecm10->device[basecm10]->roles]% assign ugeclient; commit [basecm10->device[basecm10]->roles[ugeclient]] show | grep -i gpu GPU devices gpu0 gpu1 GPUs 2 ``` The RSMAP attribute value is then:\ `GPU=2(gpu0 gpu1)` * Allow GPUs to use only different sockets: ```bash [basecm10->device[basecm10]->roles[ugeclient]]% show | grep -i gpu GPU devices gpu0:SCCCCScccc gpu1:SccccSCCCC GPUs 0 ``` The RSMAP attribute value is then:\ `GPU=2(gpu0:SCCCCScccc gpu1:SccccSCCCC)` Further detailed information about RSMAP and topology mask usage can be found in the *Altair Grid Engine Administrator’s Guide*, at [https://2021.help.altair.com/2021.1/AltairGridEngine/8.7.0/AdminsGuideGE.pdf#subsubsection.1.10.4](https://2021.help.altair.com/2021.1/AltairGridEngine/8.7.0/AdminsGuideGE.pdf#subsubsection.1.10.4) at the time of writing of this paragraph (September 2023). ### 7.5.3 Configuring And Running PBS #### 7.5.3.1 PBS Variants And Versions BCM is integrated with PBS Professional Commercial workload manager version 2022, and with OpenPBS workload manager version 22.05. * **PBS Professional**: This is the commercial variant. It requires a license or a license server in order to run jobs. This information can be provided during a run of the setup wizard, or it can be manually configured after setup. * The packages for version 2022 are available as: * `pbspro2022` for the server, * `pbspro2022-client` for the compute nodes. Pre-2020 major versions of this commercial PBS variant were denoted by 2 digits (such as 18 or 19) to signify the year. Since version 20 the version numbering is denoted by 4 digits (such as 2021 and 2022). * **OpenPBS**: This is the open source variant with community support ([http://openpbs.org](http://openpbs.org)). The community edition packages are available as: * `openpbs22.05` for the server, * `openpbs22.05-client` for the compute nodes. and as * `openpbs23.06` for the server, * `openpbs23.06-client` for the compute nodes. When no particular PBS variant is specified in BCM documentation, then the text is valid for both variants. BCM provides a similar level of integration for the commercial and the community packages. It is up to the cluster administrator to decide which variant is set up. Both variants can be installed as a selection option during NVIDIA Base Command Manager 10 installation, at the point when a workload manager must be selected ([figure 3.9 of the *Installation Manual*](/installation/installing#fig-workload)). Alternatively they can be installed later on, when the cluster has already been set up. The PBS packages that the BCM repositories provide should be used instead of other available versions, such as from the Linux distribution. If the PBS packages themselves have not been picked up, they can be installed and removed with a package manager such as YUM. **Example** ```bash [root@basecm10 ~]# yum install pbspro2022 pbspro2022-client [root@basecm10 ~]# yum install --installroot=/cm/images/default-image pbspro2022-client ``` If BCM has already been set up without PBS, but with PBS packages installed via YUM, then the `cm-wlm-setup` tool (section [7.3](#73-installation-of-workload-managers)) should be used to install and initialize PBS. #### 7.5.3.2 Installing PBS After package installation via the package manager, as described in the preceding section, PBS can be installed and initialized to work with BCM via `cm-wlm-setup`. With no options, a TUI session is started to guide the process. The alternative CLI process with options might take the following forms: **Example** ```bash [root@basecm10 ~]# cm-wlm-setup --wlm pbspro --wlm-cluster-name ppro --license ``` or ```bash [root@basecm10 ~]# cm-wlm-setup --wlm openpbs --wlm-cluster-name opbs ``` The option `--wlm pbspro` installs the commercial version, while `--wlm openpbs` installs the community version. The *license information* is either a path to a license file, or it is a Altair license server address list in the format: \<*port1*>`@<`*host1*>: \<*port2*>`@<`*host2*>: \<*...*>`@<`*...*>: \<*portN*>`@<`*hostN*> This license information can also be set manually for the `pbs_license_info` attribute. For example if there is just one license server, `pbspro-license-server`, serving on port 6200, it could be set with `qmgr` as follows: **Example** ```bash qmgr -c "set server pbs_license_info = 6200@pbspro-license-server" ``` The software components are installed and initialized by default under the `Spool` directory, which is defined by the `PBS_HOME` environment variable. The directory is named after the WLM cluster name, and follows a path of the form:\ `/cm/shared/apps/pbspro/var/spool/` \<*WLM cluster name*>\ or\ `/cm/shared/apps/openpbs/var/spool/` \<*WLM cluster name*>\ as appropriate. The paths `/cm/shared/apps/pbspro` or `/cm/shared/apps/openpbs` are the `Prefix` settings of the WLM, and depend on whether the WLM is running PBS Professional, or OpenPBS. Users must load an environment module associated with the cluster name to set `$PBS_HOME` and other environment variables, in order to use that cluster. **Example** ```bash [root@basecm10 ~]# module load pbspro ``` #### 7.5.3.3 Updating From PBS v2021 To v2022 An old PBS v2021 version can be upgraded to v2022. The release notes in the guide at [https://2022.help.altair.com/2022.1.1/PBS%20Professional/PBS_RN_2022.1.1.pdf](https://2022.help.altair.com/2022.1.1/PBS%20Professional/PBS_RN_2022.1.1.pdf) should be checked to identify possible upgrade issues. To preserve the job history from v2021, steps to carry out the upgrade are as follows: 1. Scheduling should be disabled: **Example** ```bash [root@basecm10 ~]# qmgr -c 'set server scheduling = false' #remember to set to true later ``` A BCM way of preventing nodes from taking on jobs is to drain all the nodes that can run jobs. For example all the compute nodes that have the status `UP`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device drain -s UP #remember to undrain later Engine Node Status Reason -------- ---------------- ---------------- -------------------- pbspro node001 Drained Drained by CMDaemon ``` The cluster administrator should check that there are no running jobs. ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm jobs; list Type Job ID User Queue Running time Status Nodes ------------ ------------ ------------ ------------ ------------ -------- -------------- ``` Queued jobs, where the `Status` column shows the value `Q`, are not an issue. 2. The `autostart` setting for the `pbsserver` service must be set to `no`: ```bash [root@basecm10 ~]# cmsh [basecm10]% device use basecm10; services [basecm10->device[basecm10]->services]% set pbsserver autostart no; commit ``` This is to prevent CMDaemon restarting the PBS server automatically and interfering with the next few housekeeping steps. 3. The `pbsserver` service on the head node is stopped from within `cmsh`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use master [basecm10->device[basecm10]]% services [basecm10->device[basecm10]->services]% stop pbsserver ... [basecm10->device[basecm10]->services]% status pbsserver Service Status ------------ ----------- pbsserver [STOPPED ] ``` 4. The cluster administrator would be wise to make a backup of the PBS instance \<*PBS instance*> at `/cm/shared/apps/pbspro/var/spool/` \<*PBS instance*> directory on the head node, in case of a contingency. 5. The cluster administrator must remove the `pbspro2021` and `pbspro2021-client` packages from the head node and software image(s) using the package manager. 6. The cluster administrator must install the `pbspro2022` and `pbspro22-client` packages on the head node, and the `pbspro2022-client` package in the software image(s) using the package manager. 7. The PBS version must be set to `22` in `cmsh`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% wlm use [basecm10[]% set version 22; commit ``` 8. The PBS service on the head node must be manually restarted (outside of CMDaemon). That means running, for example: ```bash [root@basecm10 ~]# pbs_server Connecting to PBS dataservice...connected to PBS dataservice@basecm10.cm.cluster Using license server at 6200@pbspro-license-server [root@basecm10 ~]# ``` This is to allow for enough time for the PBS database upgrades and any other PBS server upgrades to occur. This may take upwards of 10 minutes to complete. Its status can be checked with: ```bash [root@basecm10 ~]# /etc/init.d/pbs status pbs_server is pid 1045633 pbs_sched is pid 762684 pbs_comm is 762669 ``` or ```bash [root@basecm10 ~]# cmsh -c "device use master; services; status pbsserver" Service Status ------------ ----------- pbsserver [ UP ] ``` Any post-upgrade steps outlined in the release notes documentation can now be carried out ([https://2022.help.altair.com/2022.1.1/PBS%20Professional/PBS_RN_2022.1.1.pdf](https://2022.help.altair.com/2022.1.1/PBS%20Professional/PBS_RN_2022.1.1.pdf)). After the post-upgrade steps, the PBS service on the head node can be stopped again, outside of CMDaemon: ```bash [root@basecm10 ~]# qterm ``` The status can be checked with:\ `/etc/init.d/pbs status`\ or with:\ `cmsh -c "device use master; services; status pbsserver"` 9. The `autostart` setting for the `pbsserver` service can now be set to `yes` again: ```bash [root@basecm10 ~]# cmsh [basecm10]% device use basecm10; services [basecm10->device[basecm10]->services]% set pbsserver autostart yes; commit ``` 10. The `pbserver` service on the head node is restarted: **Example** ```bash [basecm10->device[basecm10]->services]% start pbsserver ... [basecm10->device[basecm10]->services]% status pbsserver Service Status ------------ ----------- pbsserver [ UP ] ``` 11. The `pbsmom` services on the compute nodes are then restarted: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device foreach -l pbsproclient (services; restart pbsmom) ``` 12. Scheduling should be enabled again: **Example** ```bash [root@basecm10 ~]# qmgr -c 'set server scheduling = true' ``` If the nodes were drained, then they can be undrained: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device undrain -s UP Engine Node Status Reason -------- ---------------- ---------------- -------------------- pbspro node001 ``` 13. The administrator can then run the `tracejob` command for PBS to check if the previous job information can still be accessed. #### 7.5.3.4 PBS Configuration PBS documentation is to be found in the PBS Professional Guides, which can be accessed from: [https://community.altair.com/community?id=altair_product_documentation](https://community.altair.com/community?id=altair_product_documentation). By default, PBS examples are available under the directory `/cm/shared/examples/workload/pbspro/jobscripts/` Some PBS configuration under BCM can be done using roles. The roles are the same for the variants of PBS, and are denoted as `pbspro` roles: * In Base View the roles setting allows the configuration of PBS client and server roles. * For the PBS server role, the role is enabled, for example on a head node `basecm10`, via a navigation path of: `Devices` > `Head Nodes` > `basecm10` > `Settings` > `Roles` > `Add` > `PBS pro server role` * Within the role window for the server role, its installation path and server spool path can be specified. * For the PBS client role, the role is enabled along a similar navigation path, just ending at `PBS pro client role`. The number of slots, GPUs and other properties can be specified, and queues can be selected. * In `cmsh` the client and server roles can be managed for the individual nodes in `device` mode, or managed for a node category in `category` mode, or they can be managed for a configuration overlay in `configurationoverlay` mode. For example, if there is a PBS cluster instance called `pbsfast`, then the head node could be assigned a `pbsproserver` role, with the following properties: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device roles basecm10 [basecm10->device[basecm10]->roles]% assign pbsproserver [basecm10->device*[basecm10*]->roles*[pbsproserver*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name pbsproserver Revision Type PbsProServerRole Add services yes WLM cluster External Server no Comm Settings [basecm10->device*[basecm10*]->roles*[pbsproserver*]]% set wlmcluster pbsfast; commit [basecm10->device[basecm10]->roles[pbsproserver]]% ``` Similarly, a category of nodes that can be used by the instance `pbsfast`, and called `fastnodes`, may have a PBS client role assigned and set with the following properties: **Example** ```bash [basecm10->category[fastnodes]->roles[pbsproclient]]% show Parameter Value -------------------------------- ------------------------------------------------ Name pbsproclient Revision Type PbsProClientRole Add services yes WLM cluster pbsfast Slots 1 GPUs 0 All Queues no Queues Properties Mom Settings Comm Settings Node Customizations <0 in submode> ``` Cloudbursting with cluster extension may need special handling for PBS. If this feature is needed, then the administrator should contact BCM support via the website [https://www.nvidia.com/en-us/data-center/bright-cluster-manager/support/](https://www.nvidia.com/en-us/data-center/bright-cluster-manager/support/). Further configuration of PBS is done using its `qmgr` command and is covered in the PBS documentation. #### 7.5.3.5 Running PBS For the WLM cluster instances, PBS runs the following four daemons: 1. a `pbs_server` daemon running, typically on the head node. This handles submissions acceptance, and talks to the execution daemons on the compute nodes when sending and receiving jobs. It writes logs to the `var/spool/` \<*WLM cluster instance*>`/server_logs/` directory, which is a directory that is under `/cm/shared/apps/pbspro` or `/cm/shared/apps/openpbs`. Queues for this service are configured with the `qmgr` command. 2. a `pbs_sched` scheduler daemon, also typically running on the head node. It writes logs to the `var/spool` \<*WLM cluster instance*>`/sched_logs/` directory under `/cm/shared/apps/pbspro` or `/cm/shared/apps/openpbs`. 3. a `pbs_mom` execution daemon running on each compute node. This accepts, manages, and returns the results of jobs on the compute nodes. By default, it writes logs * to the relative directory `var/spool/` \<*WLM cluster instance*>`/mom_logs/`, which is under `/cm/shared/apps/pbspro` or `/cm/shared/apps/openpbs` and * to `/cm/local/apps/pbspro/var/spool/mom_logs/` on nodes with the client role. 4. a `pbs_comm` communication daemon usually running on the head node. This handles communication between PBS daemons, except for server-to-scheduler and server-to-server daemons communications. It writes logs * to the relative directory `var/spool/` \<*WLM cluster instance*>`/comm_logs/`, which is under `/cm/shared/apps/pbspro` or `/cm/shared/apps/openpbs` and * to `/cm/local/apps/pbspro/var/spool/comm_logs/` on nodes with the client role. #### 7.5.3.6 Running PBS On Cluster Extension Running PBS on a cluster extension must not involve any form of NAT, including netmap from iptables. Communication for both the commercial and the community PBS variants fail under the default NAT netmap via OpenVPN. Using a hardware VPN, Direct Connect (for AWS) or ExpressRoute (for Azure) is a workaround for using OpenVPN/netmap. When PBS is set up with `cm-wlm-setup` or during the installation of the head node, then the `pbsproclient` role is assigned by default via a configuration overlay to the `default` node category only. ```bash [root@basecm10 ~]# cmsh -c "category; roles default; list" Name (key) ---------------------------------------- [overlay:openpbs-client] pbsproclient ... ``` In order to add cloud nodes to PBS, the administrator can assign the `pbsproclient` role manually. There are two types of PBS configuration in this case. The configurations can be applied to both the commercial and to the community editions. 1. `pbs_mom` daemons on cloud compute nodes communicate to the `pbs_server` directly. This scenario is suited to a non-VPN setup where cloud nodes have addresses on the same IP subnet for the cloud and for the on-premises parts of the cluster. Usually this kind of setup is used with Amazon DirectConnection or Azure ExpressRoute. In order to add new cloud nodes, the administrator just needs to assign the `pbsproclient` role to the `cloud` node category or the cloud nodes directly. 2. `pbs_mom` daemons on cloud compute nodes communicate to the `pbs_server` via a separate `pbs_comm` server. For BCM the use of the `cloud-director` is recommended for this purpose. This can be useful if cluster extension is configured with a VPN tunnel setup. In this case the `pbs_mom` running on a cloud node communicates with the `pbs_server` by using the VPN connection, and the communication traffic goes via an OpenVPN server. The OpenVPN connection adds overhead on the cloud-director where the OpenVPN daemon runs. If the traffic is routed via `pbs_comm` running on cloud-director, then the OpenVPN server is not used. This is because `pbs_comm` daemon on the cloud director resolves the cloud `pbs_mom` addresses with cloud IP addresses, while `pbs_server` resolves `pbs_comm` on the cloud director by using the VPN tunnel IP. In order to configure PBS to pass the communication traffic via `pbs_comm`, the administrator should assign `pbsproclient` roles to not only the compute cloud nodes, but also to the cloud-director. On the cloud director, the administrator should enable `pbs_comm` daemon to start, and `pbs_mom` daemon to not start, automatically. These actions are done in the `commsettings` and `momsettings` submodes of `pbsproclient` role. For the `pbsproclient` role assigned in the configuration overlay, the settings can be accessed in `cmsh` as in the following: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% configurationoverlay [basecm10->configurationoverlay]% use openpbs-client [basecm10->configurationoverlay[openpbs-client]]% roles [basecm10->configurationoverlay[openpbs-client]->roles]% use pbsproclient [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]]% show Parameter Value -------------------------------- ------------------------------------------------ Name pbsproclient Revision Type PbsProClientRole Add services yes WLM cluster openpbs Slots 1 GPUs 0 All Queues no Queues workq Properties Mom Settings Comm Settings Node Customizations <0 in submode> [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]]% commsettings [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]->commsettings]% show Parameter Value -------------------------------- ------------------------------------------------ Comm Routers Revision Comm Threads 4 Start Comm no [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]->commsettings]% .. [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]]% momsettings [basecm10->configurationoverlay[openpbs-client]->roles[pbsproclient]->momsettings]% show Parameter Value -------------------------------- ------------------------------------------------ Output Hostname Revision Leaf Routers Leaf Name Leaf Management FQDN no Start Mom yes Spool /cm/local/apps/openpbs/var/spool ``` Further configuration that should be carried out is to set the `commrouters` parameter on the cloud director to `master`, and set the `leafrouters` parameter on the compute cloud nodes to the hostname of the cloud director. For example (some text elided): **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% category roles cloud-nodes [basecm10->category[cloud-nodes]->roles]% assign pbsproclient [basecm10->category*[cloud-nodes*]->roles*[pbsproclient*]]% set queues cloudq [basecm10->...*]->roles*[pbsproclient*]->momsettings*]% momsettings [basecm10->...*]->roles*[pbsproclient*]->momsettings*]% set leafrouters director [basecm10->...*]->roles*[pbsproclient*]->momsettings*]% commit [basecm10->...*]->roles[pbsproclient]->momsettings]% device roles director [basecm10->...*]->roles]% assign pbsproclient [basecm10->...*]->roles*[pbsproclient*]]% momsettings [basecm10->...*]->roles*[pbsproclient*]->momsettings*]% set startmom no [basecm10->...*]->roles*[pbsproclient*]->momsettings*]% .. [basecm10->...*]->roles*[pbsproclient*]]% commsettings [basecm10->...*]->roles*[pbsproclient*]->commsettings*]% set startcomm yes [basecm10->...*]->roles*[pbsproclient*]->commsettings*]% set commrouters master [basecm10->...*]->roles*[pbsproclient*]->commsettings*]% commit [basecm10->...]->roles[pbsproclient]->commsettings]% ``` ### 7.5.4 Installing, Configuring, And Running LSF IBM prefers to make LSF available directly from their [Passport Advantage Online website](http://www.ibm.com/software/passportadvantage), which is why it is not available by direct selection in [figure 3.9 of the *Installation Manual*](/installation/installing#fig-workload). IBM provides LSF in 2 ways. Somewhat confusingly, the archived IBM Spectrum LSF Suite version 10.2.0.9 has LSF Standard edition 10.1 as a component. The component is supported, but installation from the suite is not supported by BCM. That is, installation using the official IBM method or by carrying out a manual installation are both not supported by BCM. A cluster administrator should therefore not try to install LSF Standard Edition 10.1 from a IBM Spectrum LSF Suite release. #### 7.5.4.1 Installing LSF The workload manager LSF version 10.1 is installed and integrated into NVIDIA Base Command Manager 10 with the following steps: 1. The following LSF files should be downloaded from the IBM web site into a directory on the head node: * Installation package: `lsf` \<*lsf_ver*>`_lsfinstall_linux_<`*cpu_arch*>.tar.Z * Distribution package: `lsf` \<*lsf_ver*>`_linux<`*kern_ver*>-glibc \<*glibc_ver*>`-<`*cpu_arch*>.tar.Z * Documentation package (optional): `lsf` \<*lsf_ver*>`_documentation.tar.Z` Here: * \<*lsf_ver*> is the LSF version, for example: `10.1` * \<*kern_ver*> is the Linux kernel version, for example: `2.6` * \<*glibc_ver*> is the glibc library version, for example: `2.3` * \<*cpu_arch*> is the CPU architecture, for example: `x86_64` A check should be done to ensure that the `tar.Z` files have not been renamed or had their names changed to lower case during the download, in order to avoid installation issues. All the files must be in the same directory before installation. In case of an existing failover setup, the installation is done on the active head node. A license file for LSF may also be needed for the installation, but it does not have to be in the same directory as the other `tar.Z` files. 2. The `cm-lsf` package must be installed on the head node. The `cm-lsf-client` package must be installed on both the head node and within the software images. By default they should have been already installed. If not then they can be installed manually from the BCM repository. For RHEL-based distributions the procedure looks like: ```bash [root@basecm10 ~]# yum install cm-lsf cm-lsf-client [root@basecm10 ~]# chroot yum install cm-lsf-client ``` For SLES distributions the procedure looks like: ```bash [root@basecm10 ~]# zypper install cm-lsf cm-lsf-client [root@basecm10 ~]# chroot zypper install cm-lsf-client ``` For Ubuntu distributions the procedure looks like: ```bash [root@basecm10 ~]# apt-get install cm-lsf cm-lsf-client [root@basecm10 ~]# cm-chroot-sw-img [root@basecm10 ~]# apt-get install cm-lsf-client ... [root@basecm10 ~]# exit ``` The `cm-lsf` and `cm-lsf-client` packages contain a template for an environment module file, an installation configuration file, and systemd unit files. The installation configuration file may be tuned by the administrator if required. It is passed to the `lsfinstall` script distributed with LSF, which is executed by `cm-wlm-setup` during setup. To change the default values in the installation configuration file, the administrator should change the template file: ```bash [root@basecm10 ~]# cd /cm/shared/apps/lsf/var/cm/ [root@basecm10 cm]# vi install.config.template ... ``` Values enclosed by a percentage sign, ‘%’, are replaced by `cm-wlm-setup` during installation. If such values are replaced by custom values, then `cm-wlm-setup` does not change them, and the custom values are used during installation. If `install.config` is changed instead of the template file, then `cm-wlm-setup` replaces it with the configuration file generated from the template. So, changing `install.config` directly should almost certainly not ever be done. 3. `cm-wlm-setup` is run. The directory where the LSF files were downloaded is specified on one of the setup screens, or with the `--archives-location` option. **Example** ```bash [root@basecm10 ~]# cm-wlm-setup --wlm lsf --setup --archives-location /root/lsf ``` The same can be achieved by executing `cm-wlm-setup` without any command line arguments. The required information can be specified within the TUI configuration screens in this case. Also the same can be achieved with the WLM Wizard in Base View. 4. The nodes are then rebooted, and the LSF command `bhosts` then displays an output similar to: **Example** ```bash [root@basecm10 ~]# module load lsf [root@basecm10 ~]# bhosts HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP ... basecm10 ok - 2 0 0 0 ... head2 ok - 0 0 0 0 ... node001 ok - 1 0 0 0 ... node002 ok - 1 0 0 0 ... node003 unavail - 1 0 0 0 ... node004 closed - 1 0 0 0 ... node005 ok - 1 0 0 0 ... ``` The output in the preceding example has been truncated for this manual, for convenience. The installation status can be checked with the service `lsfd` (some output elided): ```bash [root@basecm10 ~]# systemctl status lsfd lsfd.service - IBM Spectrum LSF Loaded: loaded (/usr/lib/systemd/system/lsfd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2023-01-14 17:48:05 CET; 1 day 15h ago Process: 54334 ExecStart=/cm/shared/apps/lsf/current/etc/lsf_daemons start (code=exited, status=0/SUCCESS) Tasks: 13 (limit: 23269) Memory: 179.5M CGroup: /system.slice/lsfd.service |- 54410 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/res |- 54412 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/sbatchd |- 54427 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/mbatchd -d /cm/shared/apps/l... |-102312 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/eauth -s |-344591 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/lim -d | /cm/shared/apps/l.. |-344595 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/pim -d /cm/shared/apps/... |-344612 /cm/shared/apps/lsf/...10.1/linux2.6-glibc2.3-x86_64/etc/melim |-344647 /cm/shared/apps/lsf/...10.1_linux2.6-glibc2.3-x86_64/10.1... `-457629 /cm/shared/apps/lsf/...10.1_linux2.6-glibc2.3-x86_64/10.1/... Jan 14 23:29:30 basecm10 mbatchd[54427]: Jan 14 23:29:30 2023 54427:54427 3 10.1 ... ... [root@basecm10 ~]# ``` while default queues can be seen by running: ```bash [root@basecm10 ~]# module load lsf [root@basecm10 ~]# bqueues QUEUE_NAME PRIO STATUS MAX JL/U JL/P ... owners 43 Open:Active - - - ... priority 43 Open:Active - - - ... night 40 Open:Active - - - ... chkpnt_rerun_qu 40 Open:Active - - - ... short 35 Open:Active - - - ... license 33 Open:Active - - - ... normal 30 Open:Active - - - ... interactive 30 Open:Active - - - ... idle 20 Open:Active - - - ... ``` The output in the preceding example has been truncated for this manual, for convenience. If more than one instance of an LSF cluster is set up, then the full modulefile name should be specified. For example: ```bash [root@basecm10 ~]# module load lsf/lsf1/10.1 ``` #### 7.5.4.2 Configuring LSF **LSF server configuration:** After LSF is set up with `cm-wlm-setup`, the following CMDaemon settings can be modified for the LSF server role: * `External Server`: a value of `yes` means that LSF server daemons are running on an external server that is not managed by BCM. The following settings are available in LSF cluster settings, and are applied to all the LSF roles of the LSF cluster: * `Prefix`: this sets the path to the root location of the LSF installation. The default value is `/cm/shared/apps/lsf/current`. * `Var`: this sets the path to the `var` directory of LSF. The default value is `/cm/shared/apps/lsf/var`. * `Cgroups`: this is a submode that contains LSF-related cgroups settings. The cgroups settings that affect LSF behavior are available via the LSF instance. For example, for an instance `lsf`, the settings can be accessed within `cmsh` via `cmsh->wlm[lsf]->cgroups`. The settings available are: * `Automount`: if `yes`, then the workload manager tries to mount a subsystem if it is not mounted yet. The default value is `No`. * `Job Cgroup Template`: this is the template relative job cgroup path. The token `$CLUSTER` specified in this template path is replaced with the actual LSF cluster name, and `$JOBID` is replaced by the job ID. The path is used by the BCM monitoring system in order collect job metrics from the cgroups. By default, the path is set to `"lsf/$CLUSTER/job.$JOBID.*"`. * `Process Tracking`: if `yes`, then processes are tracked based on job control functions such as: termination, suspension, resume, and other signaling. These are used on Linux systems that support the freezer subsystem under cgroups. The parameter sets LSF_PROCESS_TRACKING in `lsf.conf`. * `Linux Cgroup Accounting`: if `yes`, then LSF tracks processes based on CPU and memory accounting. This is for Linux systems that support the memory and cpuacct subsystems under cgroups. Once enabled, this parameter takes effect for new jobs. The parameter sets LSF_LINUX_CGROUP_ACCT in `lsf.conf`. If this parameter and `Process Tracking` are both enabled, then they take precedence over the parameters LSF_PIM_LINUX_ENHANCE and EGO_PIM_SWAP_REPORT in `lsf.conf`. * `Mount Point`: specifies a path where cgroups is mounted. It only makes sense to set this when the location is not standard for the operating system. * `Resource Enforce`: If `yes`, then resource enforcement is carried out through the Linux memory and cpuset subsystems under cgroups. This is for Linux systems with cgroup support. The parameter sets LSB_RESOURCE_ENFORCE in `lsf.conf`. The server role settings can be modified as follows: * Within Base View: For example for a head node `basecm10`, via a navigation path of `Devices` > `Head Nodes` > `basecm10` > `Settings` > `Roles` > `LSF server role` * Within `cmsh`: For a particular category in the `category` mode, or a particular device in the `device` mode, the `roles` submode is chosen. Within the `roles` submode, the `lsfserver` object can be assigned or used. The following example shows how to set the LSF `prefix` parameter for the `default` category. **Example** ```bash [root@basecm10~]# cmsh [basecm10]% configurationoverlay use lsf-server; roles [basecm10->configurationoverlay[lsf-server]->roles]% use lsfserver [basecm10->...[lsfserver]]% set externalserver yes [basecm10->...[lsfserver*]]% commit ``` The global LSF cluster settings can be modified as follows: * Within Base View:\ For example, for a head node `basecm10`, via a navigation path of:\ `HPC` > `Wlm Clusters` > `CLUSTER_NAME` > `Settings` * Within `cmsh`:\ A particular LSF cluster instance can be chosen within `wlm` mode, to set its global parameters. The following example shows how to set the LSF `prefix` parameter for the `lsf1` cluster. **Example** ```bash [root@basecm10~]# cmsh [basecm10]% wlm use lsf1 [basecm10->wlm[lsf1]]% set prefix /cm/shared/apps/lsf2 [basecm10->wlm*[lsf1*]]% commit ``` **LSF client configuration:** After installation, the following CMDaemon settings can be specified for the LSF client role: * \*\*Queues \*\* In `cmsh`, queues can be added using `jobqueue` mode. In LSF, the default `queue` is `normal`. More arbitrarily-named queues can be specified with: **Example** ```bash [basecm10->wlm[lsf]->jobqueue]% add abnormal [basecm10->wlm[lsf]->jobqueue*[abnormal*]]% add subnormal [basecm10->wlm[lsf]->jobqueue*[subnormal*]]% add supernormal [basecm10->wlm[lsf]->jobqueue*[supernormal*]]% commit [basecm10->wlm[lsf]->jobqueue*[supernormal]]% .. [basecm10->wlm[lsf]->jobqueue*]% list Name (key) Nodes ------------ ------------------------ abnormal normal node001..node005 subnormal supernormal ``` The `jobqueue` values can be set for the LSF client role: **Example** ```bash [basecm10->wlm[lsf]->jobqueue*]% configurationoverlay [basecm10->configurationoverlay]% use lsf-client [basecm10->configurationoverlay[lsf-client]]% roles [basecm10->...->roles]% use lsfclient [basecm10->...->roles[lsfclient]]% get queues normal [basecm10->...->roles[lsfclient]]% set queues abnormal subnormal supernormal normal [basecm10->...->roles*[lsfclient*]]% commit [basecm10->...->roles*[lsfclient*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name lsfclient Revision Type LSFClientRole Add services yes WLM cluster lsf Slots auto All Queues no Queues subnormal,supernormal,abnormal,normal Server yes Host Model Host Type LINUX GPUs 0 Node Customizations <0 in submode> ``` Alternatively, queues can be managed using Base View (section [7.6.2](#762-queues-display-and-handling-in-base-view)). * \*\*Options \*\* A navigation path of: `Devices` > `Nodes[node001]` > `Edit` > `Settings` > `Roles` > `Add` > `LSF client role`\ lets options be set for a regular node `node001` (figure [7.23](#fig-bright-view-node-roles-lsf)): ![Base View access to LSF configuration options via roles window options](/base-command-manager/product-manuals/_fern-img/6b988ad8a6947f60a37a46cfa332de6361692590087e829e5a96b64da301627d.webp) *Figure 7.23: Base View access to LSF configuration options via roles window options* Available options are: * **Slots**: The number of CPUs per node. By default LSF tries to determine the value automatically with the `auto` setting. If the number of slots is set to `0`, then the node becomes an LSF submit host, so that no jobs can run on the host, but users can submit their jobs from this host using the LSF utilities. * **Queues**: `All` queues can be set, or queues can be selected. * **Server**: Whether the node is an LSF server * **Host Model**, **Host Type**: Possible values for these are defined in `lsf.shared` * **GPUs**: The number of GPUs per node. * **Node Customizations**: LSF node custom properties (section [7.11](#711-custom-node-parameters)). From `cmsh` these properties are accessible from within the appropriate node or category `roles` submode (section [7.4.2.4](#7424-setting-options-for-workload-manager-settings-with-cmsh)). **Submission hosts:** By default, all compute nodes to which the `lsfclient` role is assigned, also become submission hosts. This means that LSF users can submit their jobs from those compute nodes. If the administrator wants to allow users to submit their jobs from a non-compute node—for example from a login node—then the `lsfsubmit` role should be assigned, for example: **Example** ```bash [root@basecm10~]# cmsh [...->configurationoverlay[lsf1-submit]]% roles [...->roles]% assign lsfsubmit [...->roles*[lsfsubmit*]]% append lsfclusters lsf1 [...->roles*[lsfsubmit*]]% commit [...->roles[lsfsubmit]]% ``` Configuring a node according to the preceding steps then allows users to submit their jobs from that node, without the submitted jobs getting scheduled to run on that node. If more than one LSF cluster is set up, and they share the same submit host, then a single `lsfsubmit` role can be used. In this case, all the LSF cluster names should be appended to the `lsfcluster` parameter of the role. **Further configuration:** For further configuration the *Administering Platform LSF* manual provided with the LSF software should be consulted. #### 7.5.4.3 Running LSF Role assignment and role removal enables and disables LSF from Base View (sections [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)) or `cmsh` (section [7.4.2](#742-enabling-and-disabling-a-workload-manager-with-cmsh)). An active LSF master service (typically, but not necessarily on a head node) has the following LSF-related processes running on it:
**Process/Service** **Description**
`res` Remote Execution Server*
`sbatchd` client batch job execution daemon*
`mbatchd` master batch job execution daemon
`eauth` External Authentication method
`lim` Load Information Manager*
`pim` Process Information Manager*
`pem` Process Execution Manager*
`vemkd` Platform LSF Kernel Daemon
`egosc` Enterprise Grid Orchestrator service controller
`mbschd` master batch scheduler daemon
\*These services/processes run on compute nodes. Non-active LSF-masters running as compute nodes run the processes marked with an asterisk only. LSF daemon logs are kept under `/cm/local/apps/lsf/var/log/`, on each of the nodes where LSF services run. ## 7.6 Using Base View With Workload Management Viewing the workload manager services from Base View is described in section [7.4.3](#743-monitoring-the-workload-manager-services). The HPC (High Performance Computing) icon, which looks like a speedometer, is the Base View resource that allows the following items to be accessed: * WLM clusters: for WLM cluster settings for a WLM cluster instance to be viewed and managed * jobs: for jobs to be viewed and managed * queues: for queues to be viewed and managed * job queue stats: for job queue statistics to be viewed These items are described next. ### 7.6.1 Jobs Display And Handling In Base View The navigation path `HPC` > `Jobs` opens up the `Job list` window. This displays a list of recent job IDs, along with the user, status, scheduler used, queue, and nodes allocated to the job (figure [7.24](#fig-bright-view-workloadmanagement-jobs-main)). ![Workload manager job list window](/base-command-manager/product-manuals/_fern-img/61eab38559631b616a95265cedce9738d0a04b15217dca4aa1554be5d2cc877e.webp) *Figure 7.24: Workload manager job list window* Buttons in the last two columns allow the jobs to be examined and managed: * The `Show` button, ![image](/base-command-manager/product-manuals/_fern-img/7fab9dda94b9f3baa427e8f1e31502e1fd0b4ac81e461b6378d34ffa547371d5.webp), is a window-opener button. It opens up a window with further details of the selected job. * The `Actions` button, ![image](/base-command-manager/product-manuals/_fern-img/a0318afa0903d3840c80669dd3b169fb32625bd7936154f0525c0a3033bf000c.webp), brings up a pop-up menu. The menu options allow the selected job to be managed as follows: * The `Hold` option stops selected queued jobs from being considered for running by putting them in a `Hold` state (`H` in the `Status` column). * The `Release` option releases selected queued jobs in the `Hold` state so that they are considered for running again. * The `Suspend` option suspends selected running jobs (`S` in the `Status` column). * The `Resume` option allows selected suspended jobs to run again. * The `Cancel` option removes selected jobs from the queue. ### 7.6.2 Queues Display And Handling In Base View The navigation path `HPC` > `Wlm Clusters` > `Edit` > `Job Queues` displays a list of queues available (figure [7.25](#fig-bright-view-workloadmanagement-queues-main)). ![Workload manager queues](/base-command-manager/product-manuals/_fern-img/a74dd910781c677caab1a26f2ba0c6ffc3c389e092da9c60c412fa048e02813c.webp) *Figure 7.25: Workload manager queues* Queues can be added or deleted. An `Edit` option allows existing queue parameters to be set, for example as in figure [7.26](#fig-bright-view-workloadmanagement-queues-edit-slurm). ![Workload manager queue parameters configuration](/base-command-manager/product-manuals/_fern-img/e2d3a2ffd43be7454ebba33c19f18907e163b35d0882057d95241ca7bb4367c4.webp) *Figure 7.26: Workload manager queue parameters configuration* ## 7.7 Using `cmsh` With Workload Management The `wlm` mode in `cmsh` gives the administrator access to the WLM instances running on the cluster. The `cmsh` tree (Appendix [M](/admin/cmshtree)) has the following submodes structure under the `wlm` mode: ```bash `-- wlm |-- accounting |-- cgroups |-- chargeback |-- jobqueue |-- jobs |-- ocisettings |-- parallelenvironments |-- pelogs `-- placeholders ``` * `accounting`: provides ways to set some advanced Slurm job accounting properties ([here](#7515-advanced-slurm-job-accounting)) * `cgroups`: provides ways to configure WLMs via the cgroup mechanism of the Linux kernel (section [7.10](#710-cgroups)) * `chargeback`: provides ways to measure the costs of requested IT resources for WLM jobs (Chapter [15](/admin/monitoringchargeback)) * `jobqueue`: allows WLM job queues to be managed (section [7.7.2](#772-job-queue-display-and-handling-in-cmsh-jobqueue-mode)) * `jobs`: allows WLM jobs to be viewed and managed (section [7.7.1](#771-the-jobs-submode-in-cmsh)) * `ocisettings`: allows OCI settings ([Chapter 2 of the *Cloudbursting Manual*](/cloudbursting/codcloud)) to be viewed and managed * `parallelenvironments`: allows MPI job environment parameters to be configured ([here](#7524-parallel-environments)) for WLMs * `pelogs`: allows access to prolog and epilog hooks ([here](#7343-workload-manager-configuration-for-prolog-and-epilog-scripts)) for PBS * `placeholders`: allows nodes to be configured in advance, for planning resource use in WLMs (section [8.4.8](/admin/cm-scale#848-queue-node-placeholders)) For basic WLM management via `cmsh`, the administrator is expected to use `wlm` mode with the `jobs` and `jobqueue` submodes. If there is just one WLM instance running on the cluster, then an administrator that accesses `wlm` mode drops straight into the object representing that WLM instance, and can then access the submodes. If there is more than one WLM running, then the administrator must select a WLM instance from the top level `wlm` mode, with the `use` command, in order to access the submodes. While at the top level `wlm` mode, the instances can be listed. The list shows the nodes used by the instance, and the node roles assigned to the nodes. Suppose BCM is being used to manage two WLM instances: * one Slurm WLM, named `slr` * one GE WLM, named `uc` with their nodes having been allocated roles as follows: **Example** ```bash [basecm10->wlm]% list Type Name (key) Server nodes Submit nodes Client nodes ------ ------------------------ ------------ ------------------------- ---------------- Slurm slr node001 basecm10,node001..node003 node002,node003 UGE uc node004 basecm10,node004..node006 node005,node006 ``` Then one way to list all jobs running per WLM instance, queue, and user, is with the `filter` command: ```bash [basecm10->wlm]% filter --running WLM Job ID Job name User Queue Submit time Start time End time Nodes Exit code ---- ------ --------- ----- ------ ----------- ------------ --------- ----------------- ---------- slr 103 sjob maud defq 15:43:02 15:43:03 N/A node002,node003 0 slr 104 sjob maud defq 15:43:02 15:43:25 N/A node002,node003 0 slr 105 sjob maud defq 15:43:02 15:43:33 N/A node002,node003 0 slr 106 sjob maud defq 15:43:02 15:43:36 N/A node002,node003 0 slr 171 sjob maud defq 14:20:09 14:20:23 N/A node002,node003 0 uc 200 ugjob fred all.q 14:22:18 14:23:20 N/A node005,node006 0 ``` The `filter` command can also be run within the `jobs` mode, for a particular WLM instance, and is described in more detail [here](#7714-the-filter-command-within-jobs-submode). An alternate way to list all jobs from `wlm` mode is with a `foreach` command to descend into `jobs` submode to list the jobs for each WLM instance: **Example** ```bash [basecm10->wlm]% foreach * (jobs ; list) Type Job ID User Queue Running time Status Nodes ----- ------- ----- ------ ------------ ---------- ---------------- Slurm 55 maud defq 18m 10s COMPLETED node002,node003 Slurm 56 maud defq 18m 5s COMPLETED node002,node003 Slurm 57 maud defq 2m 11s RUNNING node002,node003 Slurm 58 maud defq 0s PENDING Type Job ID User Queue Running time Status Nodes ----- ------- ----- ------ ------------ ---------- ---------------- UGE 96 maud all.q 2m 30s r node005,node006 UGE 97 maud 0s qw UGE 98 maud 0s qw ``` The `jobs` submode is now discussed further. ### 7.7.1 The `jobs` Submode In `cmsh` Within the `jobs` submode of a WLM instance, the administrator can list jobs that are running or queued up for that particular instance. For example, the running and queued jobs listed for a PBS instance may be displayed as: **Example** ```bash [basecm10->wlm[openpbs]->jobs]% list Type Job ID User Queue Running time Status Nodes --------- ------------- ----------- ------------ ------------ -------- ------------- PBSPro 3117.basecm10 pbuser1 hydroq 1s R node002 PBSPro 3118.basecm10 pbuser2 workq 1s R node001 PBSPro 3119.basecm10 pbuser1 hydroq 0s Q PBSPro 3120.basecm10 pbuser3 hydroq 0s Q PBSPro 3121.basecm10 pbuser1 hydroq 0s Q PBSPro 3122.basecm10 pbuser1 hydroq 0s Q ``` The number of jobs can be very large. The `wait` command in `jobs` mode can be used to provide a shorter running summary. The following `wait` command displays a running summary that regularly displays a tally of the running and pending jobs; waits for all of them to complete; and has the waiting-for-completion condition overruled by a timeout of 10 minutes: **Example** ```bash [basecm10->wlm[slurm]->jobs]% wait --timeout 10m --all 0s| running: 4, pending: 12 1m| running: 3, pending: 12 2m| running: 4, pending: 11 5m| running: 4, pending: 10 7m| running: 4, pending: 9 9m| running: 3, pending: 8 * timeout * ``` By default `wait` runs every minute until the specified timeout. Also, by default, as indicated by the missing 3rd and 4th minutes in the example, the outputs are only displayed with the tally changes. Further options to the `wait` command within `jobs` mode can be found in the help text display output when `help wait` is run. In the `foreach` jobs listings for the Slurm and GE WLM instances shown earlier, the jobs are in a running, queued up, or completed state. They are also all being run by a user `maud`, who happens to be making use of both workload managers. For Slurm, completed jobs are shown for a short time with the status `COMPLETED`. For GE, completed jobs are shown for an even shorter time with a blank for their status. #### 7.7.1.1 Commands That Change The Status Of A Queued Or Running Job Within a `jobs` submode, the following commands, used with a job ID, can change the status of a particular queued or running job: * `hold`: puts a queued job into a `hold` state. This prevents the job from being considered for running. * `release`: releases a job from a `hold` state, putting it back in the queue, so that it can be considered for running. * `suspend`: pauses a running job * `resume`: resumes a suspended job * `remove`: removes a job For example, continuing the case earlier on of the `cmsh` session with the `foreach` listing ([here](#77-using-cmsh-with-workload-management)): The administrator can `suspend` the running job with job ID 57 on the Slurm instance as follows: **Example** ```bash [basecm10->wlm]% jobs slr; suspend 57 [basecm10->wlm[slr]->jobs]% list | head -2; list | grep 57 Type Job ID User Queue Running time Status Nodes ----- ------- ----- ------ ------------ ---------- ---------------- Slurm 57 maud defq 2m 51s SUSPENDED node002,node003 ``` and can `hold` the queued waiting job with job ID 98 on the GE instance as follows: **Example** ```bash [basecm10->wlm]% jobs uc; hold 98 [basecm10->wlm[uc]->jobs]% list | head -2; list | grep 98 Type Job ID User Queue Running time Status Nodes ----- ------- ----- ------ ------------ ---------- ---------------- UGE 98 maud 0s hqw ``` The administrator can then `resume` the suspended job ID 57, and can `release` the held job ID 98 as follows: **Example** ```bash [basecm10->wlm[uc]->jobs]% wlm; jobs uc; release 98 [basecm10->wlm[uc]->jobs]% wlm; jobs slr; resume 57 ``` #### 7.7.1.2 Commands To Inspect Jobs: Within the `jobs` submode, the following commands can be used to inspect a job: * `show` \<*job ID*>: properties are shown for a running or pending job. The properties that are displayed are taken directly from commands associated with particular workload managers: * For Slurm: `scontrol show jobs` * For PBS flavors and Grid Engine: `qstat` * For LSF: `bjobs` * `info` \<*job ID*>: details are shown about a particular job ID. The job can be `pending`, `running`, or `completed` jobs. The data is taken from the BCM database. * `statistics`: statistics are shown for jobs. More detail on the `statistics` command is given [here](#7715-the-statistics-command). * `list`: lists pending and running job IDs along with some other associated properties. Recently completed jobs are also displayed. Completed jobs do not remain listed for longer than a few minutes. The `filter` command (section [7.7.1.4](#7714-the-filter-command-within-jobs-submode)) can be used instead to display all jobs for that WLM instance, using data taken from the BCM database. * `dumpmonitoringdata [`*OPTIONS*`] [` \<*start-time*>` <`*end-time*>] \<*metric*>` <`*job id*>: Displays the monitoring data for a specific metric or healthcheck for a running or completed job. Example: `dumpmonitoringdata CtxtSwitches 170` More detail is given on the `dumpmonitoringdata` command in section [12.6.4](/admin/monitoring#1264-data-values-over-a-periodthe-dumpmonitoringdata-command). * `latestmonitoringdata [`*OPTIONS*`]` \<*job ID*>: Displays the last measured job monitoring data for the job. * `latesthealthdata` \<*job ID*>: The subset of health checks within the last measured monitoring data. * `latestmetricdata` \<*job ID*>: The subset of metrics within the last measured monitoring data. More detail is given on the `latest*data` commands in section [12.6.3](/admin/monitoring#1263-the-latest-data-and-counter-valuesthe-latestdata-and-latestmetriccounters-commands). * `measurables` \<*job ID*>: The measurables used for job monitoring data. * `healthchecks` \<*job ID*>: The subset of health checks within the measurables used for job monitoring data. * `metrics` \<*job ID*>: The subset of metrics within the measurables used for job monitoring data. * `enummetrics` \<*job ID*>: The subset of enummetrics within the measurables used for job monitoring data. #### 7.7.1.3 The `info` Command Within `jobs` Submode Within the `jobs` submode of `wlm`, the `info` \<*job ID*> command shows job information details stored in the BCM database. BCM starts updating the job information in its database as soon as it detects a new job. When the job is completed BCM pulls a final job state and exit code from the workload manager. **Example** ```bash [basecm10->wlm[slr]->jobs]% info 2 Parameter Value -------------------------------- ------------------------------------------------ Job ID 2 Job name hpl Job Array ID Job Task ID User alice Group alice Account Accounting info < not set > WlmCluster slr Queue defq Nodes node001,node002 Submit time 04/12/2024 14:35:00 Start time 04/12/2024 14:35:00 End time N/A Run time N/A Persistent no Exit code 0 Status RUNNING Requested CPUs 32 Requested GPU 8 Requested memory 1.83GiB Monitoring yes Comment Run directory /home/alice Stdin file Stdout file Stderr file Total power usage < not set > Total GPU power usage < not set > Total CPU power usage < not set > Total power under allocation < not set > QOS normal [basecm10->wlm[slurm]->jobs]% ``` #### 7.7.1.4 The `filter` Command Within `jobs` Submode If run at the top-level `wlm` mode, then the `filter` command lists jobs for all WLM instances. It can also be run at the submode-level `jobs` mode, for a particular WLM instance, in which case it lists only the jobs for that instance (section [13.5](/admin/monitoringjob#135-job-monitoring-in-cmsh)). The `filter` command without any options is a bit like an extension of the `list` command in that it lists currently pending and running jobs, although in a different format. However, in addition, it also lists past tasks, with their start and end times. An example to illustrate the output format for a Slurm instance, somewhat simplified for clarity, is: **Example** ```bash [basecm10->wlm[myslurminstance]]% filter Job ID Job name User Queue Submit time Start time End time Nodes Exit code ------ -------------- ----- ----- ----------- ---------- -------- ---------------- --------- 2 hello fred defq 15:00:51 15:00:51 15:00:52 node001 0 3 mpirun fred defq 15:03:17 15:03:17 15:03:18 node001 0 ... 25 slurmhello.sh fred defq 16:17:30 16:17:31 16:17:33 node001..node004 0 26 slurmhello.sh fred defq 16:18:39 16:18:40 16:19:02 node001..node004 0 27 slurmhello.sh fred defq 16:18:41 16:19:03 16:19:25 node001..node004 0 28 slurmhello.sh fred defq 16:18:41 N/A N/A node001..node004 0 29 slurmhello.sh fred defq 16:18:42 N/A N/A node001..node004 0 ``` In the preceding example, the start and end times for jobs 28 and 29 are not yet available because the jobs are still pending. Because all jobs for that WLM instance—historic and present—are displayed by the `filter` command, it means that finding a particular job in the output displayed can be hard. The displayed output can therefore be filtered further. The following options to the `filter` command are therefore available in order to find particular groupings of jobs: * `-w|--wlm`: by WLM instance * `-u|--user`: by user * `-g|--group`: by group * `--ended`: by jobs that ended successfully, with a zero exit status * `--running`: by jobs that are running * `--pending`: by jobs that are pending * `--failed`: by jobs that finished unsuccessfully, with a non-zero exit status It may take about a minute for CMDaemon to become aware of job data. This means that, for example, a job submitted 10s ago may not show up in the output. Further options to `filter` command can be seen by running the command `help filter` within the `wlm` or `jobs` modes. #### 7.7.1.5 The `statistics` Command The `statistics` command without any options displays an overview of states for all past and present jobs: **Example** ```bash [basecm10->wlm[uc]]% statistics Pending Running Finished Error Nodes ---------- ---------- ---------- ---------- ---------- 4 0 154 5 309 ``` The options available include the following: * `-w|--wlm` \<*WLM instance*>: by WLM instance * `-u|--user`: by user * `-g|--group`: by group * `-a|--account`: by account * `-i|--accounting-info [,,...]`: by free text accounting fields * `--hour`: by hour * `--day`: by day * `--week`: by week * `--interval`: by interval The jobs statistics can be split across users. An example of the output format in this case, somewhat simplified for clarity, is: **Example** ```bash [basecm10->wlm[slurm]->jobs]% statistics --user User Pending Running Finished Error Nodes ---------------------- ---------- ---------- ---------- ---------- alice 0 0 2 0 1 bob 0 0 5 0 4 charline 0 1 2 0 2 dennis 0 0 6 0 4 eve 0 0 4 0 1 frank 0 0 1 0 1 ``` The job statistics can be displayed over various time periods, if there are jobs within the associated period. An example of the output format for an interval of 60s is: **Example** ```bash [basecm10->wlm[slurm]->jobs]% statistics --interval 60 Date Pending Running Finished Error Nodes -------------------- ---------- ---------- ---------- ---------- ---------- 2020/05/13 15:00:00 0 0 1 0 1 2020/05/13 15:03:00 0 0 7 0 7 2020/05/13 15:04:00 0 0 1 1 2 2020/05/13 15:06:00 0 0 3 0 7 2020/05/13 15:07:00 0 0 1 0 0 2020/05/13 15:08:00 0 0 6 0 8 2020/05/13 16:17:00 0 0 1 0 4 2020/05/13 16:19:00 0 0 9 0 36 2020/05/13 16:20:00 0 0 6 0 24 2020/05/13 16:21:00 0 0 3 0 12 ``` In the preceding example there are no jobs within the period associated with the time 15:01 and 15:02. For readability the statistics for those intervals are not displayed. #### 7.7.1.6 Job Directives Job directives configure some of the ways in which CMDaemon manages job information processing. The administrator can configure the following directives: * `JobInformationKeepCount`: The maximal number of jobs that are kept in the cache, default 8192, maximal value 65536 ([here](/admin/cmdconf#c114-jobinformationkeepcount-directive)). * `JobInformationKeepDuration`: How long to keep jobs in the CMDaemon database, default 28 days ([here](/admin/cmdconf#c112-jobinformationkeepduration-directive)). * `JobInformationMinimalJobDuration`: Minimal duration for jobs to place them in the cache, default 0s ([here](/admin/cmdconf#c116-jobinformationminimaljobduration-directive)). * `JobInformationFlushInterval`: Over what time period to flush the cache to storage ([here](/admin/cmdconf#c117-jobinformationflushinterval-directive)). * `JobInformationDisabled`: Disables job information processing ([here](/admin/cmdconf#c111-jobinformationdisabled-directive)). ### 7.7.2 Job Queue Display And Handling In `cmsh`: `jobqueue` Mode The `jobqueue` submode under the top level `wlm` mode can be used to manage queues for a WLM instance. Queue properties can be viewed and set. From the top level `wlm` mode, a list of queues can be seen for each WLM instance by descending into the WLM instance with a `foreach`: **Example** ```bash [basecm10->wlm]% foreach * (get name; jobqueue; list) slr Name (key) Nodes ------------ ------------------------ defq node002,node003 uc Name (key) Nodes ------------ ------------------------ all.q node005,node006 [basecm10->wlm]% ``` The usual object manipulation commands (`show`, `get`, `set` and others) of section [2.5.3](/admin/clustermanagement#253-working-with-objects) work in the `jobqueue` submode. #### 7.7.2.1 Job Queue Parameters For GE For example, for a GE instance, the `show` command might display the following properties: **Example** ```bash [basecm10->wlm[uc]->jobqueue]% show all.q Parameter Value -------------------------------- ------------------------------------------------ Epilog NONE Maximal runtime INFINITY Minimal runtime INFINITY Name all.q Nodes node005,node006 Options Parallel environments openmpi,mpich2 Prolog NONE Revision Slots 0 Temp directory /tmp Type UGE WlmCluster uc Resume method NONE Starter method NONE Suspend method NONE Terminate method NONE ``` #### 7.7.2.2 Job Queue Parameters For Slurm Similarly, for a Slurm instance, the `show` command might display the following properties: ```bash [basecm10->wlm[slr]->jobqueue]% show defq Parameter Value -------------------------------- ------------------------------------------------ Name defq Nodes node001..node003 Revision Type Slurm WlmCluster slr Ordering 0 default yes Hidden no Min nodes 1 Max nodes UNLIMITED Default time UNLIMITED Max time UNLIMITED Priority Job Factor 1 Priority Tier 1 OverSubscribe NO Alternate Grace time 0 Preemption mode OFF Require reservation NO Select Type Parameters LLN no TRES Billing Weights Alloc nodes ALL CPU bindings None QOS Default memory per CPU UNLIMITED Max memory per CPU UNLIMITED Default memory per Node UNLIMITED Max memory per Node UNLIMITED Max CPUs per node UNLIMITED Default memory per GPU UNLIMITED Default CPU per GPU UNLIMITED Disable root no Root only no Allow groups ALL Allow accounts ALL Allow QOS ALL Deny Accounts Deny QOS ExclusiveUser no Options ``` #### 7.7.2.3 Job Queue Parameters For PBS Likewise, for a PBS instance, the `show` command might display the following properties: ```bash [basecm10->wlm[pb]->jobqueue]% show workq Parameter Value -------------------------------- ------------------------------------------------ ACL host enable no Default Queue yes Default runtime Enabled yes From Route Only no Maximal Queued 0 Maximal runtime 240:00:00 Minimal runtime 00:00:00 Name workq Nodes node008,node009 Options Priority 0 Queue Type EXECUTION Revision Route Held Jobs no Route Lifetime 0 Route Retry Time 0 Route Waiting Jobs no Routes Started yes Type PBSPro WlmCluster pb ``` ### 7.7.3 Nodes Drainage Status And Handling In `cmsh` Sometimes a node needs to be drained. Draining a node means not allowing new jobs to run on a node, and letting running jobs finish (drain away). Draining nodes can be for the purpose of: * planned maintenance * hardware troubleshooting * preventing new jobs during system changes * isolating problematic nodes Running the `device` mode command `drainstatus` displays if a specified node is in a `Drained` state or not. In a `Drained` state jobs are not allowed to start running on that node. Running the `device` mode command `drain` puts a specified node in a `Drained` state: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% drainstatus Node Queue Status ------------------------ ------------------------ ---------------- node001 PbsPro/pb:workq node002 UGE/uc/all.q [basecm10->device]% drain node001 Node Queue Status ------------------------ ------------------------ ---------------- node001 PbsPro/pb:workq Drained ``` The `undrain` command unsets the `Drained` state so that jobs may start running on the node again. The `drain`, `undrain`, and `drainstatus` commands have the same grouping options. The grouping options can make the command apply to not just one node, but to a list of nodes, a group of nodes, a category of nodes, a rack, a chassis, an overlay, a role, or a status. Continuing the example: ```bash [basecm10->device]% drain -c default; !# for a category of nodes Node Queue Status ------------------------ ------------------------ ---------------- node001 PbsPro/pb:workq Drained node002 UGE/uc/all.q Drained ``` The help text for each command indicates the syntax: **Example** ```bash [root@basecm10 ~]# cmsh -c "device help drain" Name: drain - Drain jobs (not data) on a set of nodes Usage: drain [OPTIONS/node] Options: -n, --nodes List of nodes, e.g. node001..node015,node020..node028,node030 or ^/some/file/containing/hostnames -g, --group Include all nodes that belong to the node group, e.g. testnodes or test01,test03 -c, --category Include all nodes that belong to the category, e.g. default or default,gpu -r, --rack Include all nodes that are located in the given rack, e.g rack01 or rack01..rack04 -h, --chassis Include all nodes that are located in the given chassis, e.g chassis01 or chassis03..chassis05 -e, --overlay Include all nodes that are part of the given overlay, e.g overlay1 or overlayA,overlayC -m, --image Include all nodes that have the given image, e.g default-image or default-image,gpu-image -t, --type Type of devices, e.g node or virtualnode,cloudnode -i, --intersection Calculate the intersection of the above selections -u, --union Calculate the union of the above selections -l, --role Filter all nodes that have the given role -s, --status Only run command on nodes with specified status, e.g. UP, "CLOSED|DOWN", "INST.*" --setactions set drain actions, actions already set will be removed (comma-separated) --appendactions append drain actions to already existing drain actions (comma-separated) --removeactions remove drain actions --listactions list all drain actions --clearactions remove all drain actions Examples: drain Drain the current node drain node001 Drain node001 drain -r rack01 Drain all nodes in rack01 drain --setactions reboot Drain the current node, and reboot when all jobs are completed ``` A useful one-liner to reboot and reprovision a sequence of nodes could be in the following format: **Example** ```bash cmsh -c 'device; drain -n --setactions powerreset ; drain -n --appendactions undrain' ``` This drains the nodes and does a power reset action, which provisions the nodes. After the nodes are up again, they are undrained so that they are ready to accept jobs again. The command allows the sequence of nodes to be rebooted, for example for the purpose of upgrading a kernel, without needing to schedule a downtime for the cluster. The cluster administrator should be aware that using the preceding one-liner is not a universal solution. For example, it is not a good solution if the drain takes a long time, because it means that nodes that drain early could be idle for a long time. ## 7.8 Examples Of Workload Management Assignment ### 7.8.1 Setting Up A New Category And A New Queue For It Suppose a new node with processor optimized to handle Shor’s algorithm is added to a cluster that originally has no such nodes. This merits a new category, `shornodes`, so that administrators can configure more such new nodes efficiently. It also merits a new queue, `shorq`, so that users are aware that they can submit suitably-optimized jobs to this category. Ways to configure the cluster manager for this are described next. #### 7.8.1.1 A New Category And A New Queue With Base View For An Existing Workload Manager Cluster Queues can be added to a WLM cluster by running the WLM wizard (navigation path `HPC > Wlm Wizard`) if the WLM cluster does not yet exist. The wizard provides an option during WLM cluster creation for adding queues. To create a new queue in an existing WLM cluster, the navigation path:\ `HPC > Workload Management Clusters >` \<*WLM cluster name*>` > Job queues > Add`\ is followed. A pop-up menu appears to allow the WLM cluster to be chosen for the queue, and the properties of the new queue can then be set in the dialog window of the queue (figure [7.27](#fig-bright-view-workloadmanagement-add-new-queue-slurm)). At least the queue name must be set for the queue. ![Adding a new queue via Base View](/base-command-manager/product-manuals/_fern-img/2cab43a347dda880d4b70ea74202a51ad11e60bc362c9b6ceb4a1ef95c26777a.webp) *Figure 7.27: Adding a new queue via Base View* The configuration for the queue properties is saved, and then the job queue scheduler configuration is saved. Next, a new category can be added via\ `Grouping > Categories > Add > Category > Settings`\ Parameters settings in the new category can be set—at least the category name should be set—to suit the new machines (figure [7.28](#fig-bright-view-workloadmanager-add-category)). The name `shornodes` can therefore be set here. ![Adding a new category via Base View](/base-command-manager/product-manuals/_fern-img/78b83e449be35a741af1d912fe44f36f7520fe1e1bbe1a3327c5b30f4c5c17e7.webp) *Figure 7.28: Adding a new category via Base View* Another option within the category is to set the queue. The queue, `shorq`, is therefore set here for this new category. Setting a queue for the category means configuring the options of the queue scheduler role, for example the Slurm client role, for the category. Continuing from within the `Node categories` options window of figure [7.28](#fig-bright-view-workloadmanager-add-category), the relative navigation path to set a queue is:\ `JUMP TO > Roles > Slurmclient role > Edit > Queues`\ The appropriate queue can be selected from the queue scheduler menu option (figure [7.29](#fig-bright-view-workloadmanager-category-role-queue)): ![Setting a queue for a new category via Base View](/base-command-manager/product-manuals/_fern-img/a94b3127bc0ab5a0fa38de862b80f7fc2435a8c50d9013be6890b9453837b77e.webp) *Figure 7.29: Setting a queue for a new category via Base View* In this case, the `shorq` created earlier on in figure [7.27](#fig-bright-view-workloadmanagement-add-new-queue-slurm) is presented for selection. After selection, the configuration settings can then be saved by clicking on the `Save` button. Nodes that are to use the queue should be members of the `shornodes` category. The final step is then to allocate such nodes to the category. This can be done, for example for a `node001`, by going into the settings of the node, via `Devices > Nodes > node001 > Edit > Settings > Category` and setting the `category` to `shornodes`. #### 7.8.1.2 A New Category And A New Queue With `cmsh` The preceding example can also be configured in `cmsh` as follows: The new queue can be added from within `jobqueue` mode, for the workload manager. For example, if Slurm is the WLM cluster that is enabled: ```bash [basecm10]% wlm; use slurm; jobqueue; add shorq [basecm10->wlm[slurm]->jobqueue*[shorq*]]% commit ``` The new category, `shornodes`, can be created by cloning an old category, such as `default`: ```bash [basecm10->wlm[slurm]->jobqueue[shorq]]% category [basecm10->category]% clone default shornodes [basecm10->category*[shornodes*]]% commit ``` Then, going into the roles submode, appropriate workload manager roles can be assigned, and appropriate queues can be appended and committed for that category: ```bash [basecm10->category[shornodes]]% roles [basecm10->category[shornodes]->roles]% assign slurmclient; commit [basecm10->category[shornodes]->roles[slurmclient]]% append queues shorq [basecm10->category[shornodes*]->roles*]% commit ``` The nodes belonging to the `shornodes` category can then be placed by going into `device` mode to select the nodes to be placed in that category. For example: ```bash [basecm10->category[shornodes]->roles]% device use node002 [basecm10->device[node002]]% set category shornodes [basecm10->device*[node002*]]% commit ``` ### 7.8.2 Setting Up A Prejob Check #### 7.8.2.1 How It Works Measurables such as health checks (section [12.2.4](/admin/monitoring#1224-health-check)) by default run as scheduled tasks over regular intervals. They can optionally be configured to run on demand instead, or as prejob checks, that is, before a job is run. If a health check is configured with the `prejob` setting, then its response means the same as that of a health check, that is: * If the response to a `prejob` health check is `PASS`, then it shows that the node is displaying healthy behavior for that particular health check. * If the response to a `prejob` health check is `FAIL`, then it implies that the node is unhealthy, at least for that particular health check. The reason a `prejob` health check is treated more carefully is that, if the node it has just run on is unhealthy, then it means that a job submitted to the node may fail, may not be able to start, or may even vanish outright. The way it can vanish in some cases, without any information beyond the job submission “event horizon” leads to this behavior sometimes being called the *Black Hole Node Syndrome*. **Draining On Prejob Health Check Failure:** It can be troublesome for a system administrator to pinpoint the reason for such job failures, since a node may only fail under certain conditions that are hard to reproduce later on. It is therefore a good policy to disallow passing a job to a node which has just been flagged as unhealthy by a health check. Thus, a sensible action (section [12.2.6](/admin/monitoring#1226-action)) taken by a `prejob` health check on receiving a `FAIL` response would be to put the node in a `Drained` state (section [7.7.3](#773-nodes-drainage-status-and-handling-in-cmsh)). The `drain` state means that BCM arranges a rescheduling of the job so that the job runs only on nodes that are believed to be healthy. A node that has been put in a `Drained` state with a health check is not automatically undrained. The administrator must clear such a state manually. The `failedprejob` health check ([here](/admin/metricshealthchecksactions#g21-regular-health-checks)) is enabled by default, and logs any prejob health check passes and failures. By default there are no prejob health checks configured, so by default this health check should always pass. #### 7.8.2.2 Configuration Using `cmsh` Prejob health checks have the parameter `When` set to one of the following values: 1. `pre job` 2. `timed and pre job` The parameter `When` for a health check can be set within the `monitoring setup` mode. **Example** For example, for the `ib` health check: ```bash [basecm10->monitoring->setup]% get ib when Timed [basecm10->monitoring->setup]% set ib when on demand on start out of band pre job timed timed and pre job [basecm10->monitoring->setup]% set ib when pre job [basecm10->monitoring->setup*]% commit ================================= ib ========================================== Field Message ------------------------ ------------------------------------------------------ when warning: Prejob is selected, but no WLM has prejob enabled ``` To allow the healthcheck to run for the prejob with the workload manager, prejob must also be enabled appropriately for the associated workload manager: **Example** ```bash [basecm10->wlm[slurm]]% get enableprejob no [basecm10->wlm[slurm]]% set enableprejob yes [basecm10->wlm*[slurm*]]% commit ``` #### 7.8.2.3 Configuration Using Base View To configure the monitoring of nodes with a prejob check in Base View, the navigation path: `Monitoring>Data Producers>Monitoring data producers>Edit>When>Pre job` is used to set the `When` value for when it runs. In this case, `When` is set to `Pre job`. Prejob should also be enabled appropriately for the associated workload manager with the path: `HPC>Workload Management Clusters>Settings>Enable prejob` #### 7.8.2.4 Configuration With `cm-wlm-setup` To set a prejob health check up via `cm-wlm-setup`, `cm-wlm-setup` is run in step-by-step mode. During one of the steps the administrator is prompted to select, with a tick on a checkbox, which of the available health checks are to run as prejob health checks in BCM (figure [7.30](#fig-cm-wlm-setup-ncurses-prejob-screen)): ![Prejob healthcheck selection in the cm-wlm-setup screen](/base-command-manager/product-manuals/_fern-img/0741f51f1afe11e0212ef7ff9aee9d5bc5a37cbd7307306fd1b2301199409adc.webp) *Figure 7.30: Prejob healthcheck selection in the cm-wlm-setup screen* If a selected health check was already set as a prejob—-that is, with the `When` parameter already set to either `pre job` or `timed and pre job`—then the checkbox is already ticked. If the `When` parameter was set to `timed` before, then selecting the health check sets the `When` parameter to `timed and pre job`. For all other values of he `When` parameter for that health check, the `When` parameter is set to `pre job`. #### 7.8.2.5 Custom Prejob Configuration For more unusual prejob checking requirements, further details on how prologs are configured are given in section [7.3.4](#734-prolog-and-epilog-scripts). ## 7.9 Power Saving With `cm-scale` The `cm-scale` service can be used by an administrator to reduce the energy and storage costs of compute nodes by changing their power state, or their existence state, according to workload demands. That is, `cm-scale` automatically scales a cluster up or down, on-demand, by powering up physical nodes, cloud nodes, or virtual nodes. The scaling is carried out according to settings in the `ScaleServer` role which set the nodes that are to use the `cm-scale` service. The `cm-scale` service is covered extensively in Chapter [8](/admin/cm-scale). ## 7.10 Cgroups Linux system processes and all their future children can be gathered into sets called process aggregations. These sets can be made into hierarchical groups with specialized behavior using the Control Groups (cgroups) mechanism. The behavior is controlled by different subsystems that are attached to the cgroup. A subsystem may, for example, allow particular CPU cores to be allocated, or it may restrict memory, or it may reduce swap usage by processes that belong to the group, and so on. Details about Linux cgroups and their subsystems can be found at [https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt](https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt). As far as workload management is concerned, it makes sense to distinguish between workload manager cgroup settings and system-wide cgroup parameters. The workload manager cgroup settings allow the administrator to configure a workload manager to use cgroups in a particular way, whereas the system-wide cgroup settings allow the administrator to manage cgroups whether a workload manager is used or not. ### 7.10.1 Cgroups Settings For Workload Managers If the workload manager allows cgroups usage, then BCM provides capabilities to manage the cgroup parameters within the workload manager. #### 7.10.1.1 Slurm Cgroups Slurm supports 3 cgroups-related plugins. These are all enabled by default, and are: 1. `proctrack/cgroup`: enables process tracking and suspend/resume capability using cgroups. This plugin is more reliable for tracking and control than the former `proctrack/linux`. 2. `task/cgroup`: provides the ability to * confine jobs and steps to their allocated cpuset * bind tasks to sockets, cores and threads * confine jobs and steps to specific memory resources and gres devices 3. `jobacct_gather/cgroup`: collects accounting statistics for jobs, steps and tasks using the `cpuacct`, `memory`, and `blkio` cgroups subsystems. Slurm uses configuration files to store the parameters and devices used for cgroup support. The file `/cm/shared/apps/slurm/var/etc/` \<*cluster name*>`/cgroup.conf` defines parameters used by Slurm’s Linux cgroup-related plugins. The file contains a section that is autogenerated by CMDaemon, with cgroups-related parameters defined in the SlurmServer role. For Slurm, the administrator can set cgroups parameters using `cmsh` by going into the `cgroups` submode of the WLM instance. Parameters that can be managed include:
**Parameter** **Description** **Configuration Parameter**
**In `cgroup.conf`**
`Auto Mount` \* Force Slurm to mount cgroup subsystems if they are not mounted yet `CgroupAutomount`
`Mount Point` Where cgroup root is mounted `CgroupMountpoint`
`Task Affinity` \* Set a default task affinity to bind each step task to a subset of the allocated cores using `sched_setaffinity` `TaskAffinity`
`Release Agent Dir` Directory containing Slurm cgroup release_agent files `CgroupReleaseAgentDir`
`Constrain Cores` \* Constrain allowed cores to the subset of allocated resources `ConstrainCores`
`Constrain RAM Space` \* Constrain the job’s RAM usage `ConstrainRAMSpace`
`Constrain Swap Space` \* Constrain the job’s swap space usage `ConstrainSwapSpace`
`Constrain Devices` \* Constrain the job’s allowed devices (also constrain GPUs) based on GRES allocated resources `ConstrainDevices`
`Allowed RAM Space` Percentage memory (default is 100%) out of the allocated RAM allowed for the job cgroup RAM. If this percentage is exceeded, then the job steps will be killed and a warning message will be written to standard error. `AllowedRAMSpace`
`Allowed Swap Space` Percent allocated memory allowed for the job cgroup swap space `AllowedSwapSpace`
`Max RAM Percent` Maximum percent of total RAM for a job `MaxRAMPercent`
`Max Swap Percent` Maximum percent of total RAM for the amount of RAM+Swap that may be used for a job `MaxSwapPercent`
`Min RAM Space` Minimum MB for the memory limits defined by `Allowed RAM Space` and `Allowed Swap Space` `MinRAMSpace`
\* Boolean (takes `yes` or `no` as a value) The options are always written in the `cgroup.conf` file. More details on these options can be found in the man page `man cgroup.conf.5`. #### 7.10.1.2 GE Cgroups Altair Grid Engine allows cgroups settings to be defined at two levels. * Globally: The parameters can be set globally in the global configuration. BCM defines cgroups global settings in the `cgroups` submode of the `UGEServer` role. * Per host: The parameters can be overridden in the host configuration for particular hosts. BCM defines cgroups host settings in the `cgroups` submode of the `UGEClient` role. The global and host level cgroups parameters are the same, and are as follows:
**Parameter** **Description** **Configuration Parameter**
**In `qconf`**
`Auto Mount` \* Force GE to mount a cgroup subsystems if they are not mounted yet `mount`
`Cpuset` If true, then core binding is done by the cgroup cpuset subsystem `cpuset`
`Freezer` If true, then it enables the cgroup freezer subsystem for job suspension and resumption `freezer`
`Freeze PE Tasks` If true and the freezer subsystem is turned on, then the master task is suspended, and all slave tasks of the parallel job are also frozen `freeze_pe_tasks`
`Killing` If true then GE signals all processes forked/started by the job until all of them are killed `killing`
`Forced NUMA` If true then on NUMA machines only local memory (memory in the same NUMA zone) is allowed to be used when the job requested memory allocation with `-mbind cores:strict` `forced_numa`
`Virtual Memory Limit` Specifies if virtual memory can be limited with cgroups `h_vmem_limit`
`Memory Free Hard` If true then kernel ensures that the job does not use more main memory than required `h_mem_free_hard`
`Memory Free Soft` If true, and the hard memory limit is turned off, then the requested memory with m_mem_free is a soft limit `h_mem_free_soft`
`Min Memory Limit` A host based minimum memory limit, in bytes or values like 10M, 1G `min_memory_limit`
`Devices` Controls which devices the job can access. It permits wildcard definitions such as `/dev/sda` or `/dev/cpu//` `devices`
`Options` Additional cgroups parameters for GE
\* Boolean (takes `yes` or `no` as a value) The options are updated in GE only when cgroups are enabled in the `ugeserver` or `ugeclient` roles. To enable them, `enabled` must be set to `true` within the `cgroups` submode of the role. By default only global cgroups settings are enabled in GE. #### 7.10.1.3 PBS Cgroups PBS supports cgroups through a special Python hook that is installed by `cm-wlm-setup` by default. The hook name is `pbs_cgroups`. PBS cgroup options can be set to manage cgroups in `cmsh` or Base View. For example, in `cmsh` the options can be shown with: **Example** ```bash [basecm10->wlm[openpbs]->cgroups]% show Parameter Value -------------------------------- ------------------------------------------------ Mount Point /sys/fs/cgroup Revision Type PbsProCgroupsSettings ... Memsw reserve amount 64MiB ``` The following table shows the more important cgroup options that can be set:
**Parameter** **Description** **Default Value**
`Mount Point` Where cgroup root is mounted `/sys/fs/cgroup`
`Revision` Entity revision
`Type` Type of entity `PbsProCgroupsSettings`
`Job cgroup template` Template for job cgroup path ( $ESCAPE_JOBID will be replaced by systemd-escape of job id) pbspro.slice/ pbspro-$ESCAPE_JOBID.slice
`Cgroup prefix` Cgroup prefix used by PBS when the cgroup is created `pbspro`
`Auto Mount` \* If true then workload manager tries to mount a subsystem if it is not mounted yet `no`
`Enabled` \* When set the cgroups hook is enabled (in the hook config: `enabled` ) `yes`
`Nvidia SMI` The location of the `nvidia-smi` command (in the hook config: `nvidia-smi` ) `/usr/bin/nvidia-smi`
`Kill timeout` Maximum number of seconds the hook spends attempting to kill job processes before destroying cgroups (in the hook config: `kill_timeout` ) `10s`
`Server timeout` Maximum number of seconds the hook spends attempting to fetch node info from the server (in the hook config: `server_timeout` ) `15s`
`Use hyperthreads` \* All CPU threads are made available to jobs (in the hook config: `use_hyperthreads` ) `no`
`Ncpus are cores` \* Ncpus of a vnode is the number of cores, and the hook assigns all threads of each core to a job (in the hook config: `ncpus_are_cores` ) `no`
`Cpuacct enabled` \* Enable cpuacct cgroup controller for jobs `yes`
`Cpuset enabled` \* Enable cpuset cgroup controller for jobs `yes`
`Devices enabled` \* Enable devices cgroup controller for jobs `yes`
`Devices allow` Parameter specifies how access to devices will be controlled b *:* rwm,c *:* rwm
`Hugetlb enabled` \* Enable hugetlb cgroup controller for jobs `no`
`Hugetlb default` The amount of huge page memory assigned to the cgroup when the job does not request hpmem `0B`
`Hugetlb reserve percent` The percentage of available huge page memory (hpmem) that is not to be assigned to jobs `0`
`Hugetlb reserve amount` An amount of available huge page memory (hpmem) that is not to be assigned to job `0B`
`Memory enabled` \* Enable memory cgroup controller for jobs `yes`
`Memory soft limit` \* If false, then PBS uses hard memory limits which prevent the processes from ever exceeding their requested memory usage `yes`
`Memory default` Amount of memory assigned to the job if it doesn’t request any memory `64MiB`
`Memory reserve percent` The percentage of available physical memory that is not to be assigned to jobs `0`
`Memory reserve amount` A specific amount of available physical memory that is not to be assigned to jobs `64MiB`
`Memsw enabled` \* Enable memsw cgroup controller for jobs `no`
`Memsw default` Specifies the amount of memory + swap assigned to the job if it doesn’t request any memory `256MiB`
`Memsw reserve percent` Percentage of available swap that is not to be assigned to jobs `0`
`Memsw reserve amount` An amount of available swap that is not to be assigned to jobs `64MiB`
\* Boolean (takes `yes` or `no` as a value) **PBS cgroups, hyperthreading, and `ncpu`:** By default the cluster manager enables the `pbs_cgroups` hook ([here](#71013-pbs-cgroups)). By default, this sets `ncpus_are_cores` and `use_hyperthreading` both to `false`. The PBS Professional 2021.1 Administrator’s Guide says the following about the disabled hyperthreading configuration ([https://2021.help.altair.com/2021.1/PBSProfessional/PBSAdminGuide2021.1.pdf#M19.9.62244.Heading3.15433.Configuring.Hyperthreading.Support](https://2021.help.altair.com/2021.1/PBSProfessional/PBSAdminGuide2021.1.pdf#M19.9.62244.Heading3.15433.Configuring.Hyperthreading.Support)): > In this model PBS makes only the first thread of each core visible to PBS jobs, so if your workload cannot leverage hyperthreading well, you don’t need to disable hyperthreading in the BIOS. The other CPU threads are still usable by the operating system, which means throughput is better than if hyperthreading support is disabled in the BIOS. The value of `resources_available.ncpus` reflects the number of cores associated with a vnode. The value of `ncpus` can be made to match the number of hyperthreads (logical processors) in the node with **Example** ```bash [root@basecm10 ~]# cmsh -c 'wlm; use pbspro; cgroups; set usehyperthreads yes; commit' ``` If `usehyperthreads` remains at its default value of `no`, then it means that a restart of the `pbs_mom` service causes `ncpus` to match the number of cores rather than the full number of logical processors that simultaneous multi-threading provides. Here, the value set for the attribute `slots` in the `pbsproclient` role is ignored. This is a deliberate design choice of PBS Professional, which may seem counter-intuitive without the background information given in this section. #### 7.10.1.4 LSF LSF allows resource enforcement to be controlled with the Linux cgroup memory and cpuset subsystems. By default, when LSF is set up with `cm-wlm-setup`, then both subsystems are enabled for LSF jobs. If job processes on a host use more memory than the defined limit, then the job is immediately killed by the Linux cgroup memory subsystem. The cgroups-related configuration options are available in `cmsh` or Base View, and can be found in the `cgroups` submode of the LSF cluster settings:
**Parameter** **Description** **Configuration Parameter**
**In `lsf.conf`**
`Resource Enforce` Controls resource enforcement through the Linux cgroups memory and cpuset subsystem, on Linux systems with cgroups support. The resource can be either `memory` or `cpu` , or both `cpu` and `memory` , in either order (default: `memory cpu` ) `LSB_RESOURCE_ENFORCE`
`Process Tracking` \* This parameter, when enabled, has LSF track processes based on job control functions such as termination, suspension, resume, and other signals, on Linux systems which support the cgroups freezer subsystem `LSF_PROCESS_TRACKING`
`Linux Cgroup Accounting` \* When enabled, processes are tracked based on CPU and memory accounting for Linux systems that support cgroup’s memory and cpuacct subsystems `LSF_LINUX_CGROUP_ACCT`
\* Boolean (takes `yes` or `no` as a value) ## 7.11 Custom Node Parameters NVIDIA Base Command Manager 10 allows the administrator to specify the most important node parameters when a node is configured in a workload manager. But sometimes the workload manager allows the configuration of other advanced or user-defined settings for the nodes. When an administrator clones a node in BCM, or creates it from scratch, then those advanced settings may need to be configured manually per node in the workload manager. Sometimes the nodes are created or cloned automatically, for example `cm-scale` (Chapter [8](/admin/cm-scale)), in which case node parameters customization is not possible without manual intervention in the workload manager configuration. Since NVIDIA Base Command Manager version 8.2, the administrator can configure the advanced node settings in the node customizations mode of `cmsh` or Base View. The customizations are available in the workload manager client roles, so they can be applied at the node, node category or configuration overlay levels. In order to configure a custom node parameter for the workload manager, the administrator adds the node customization entry and sets its value to `True` or `False`. The entry is enabled by default, that is, an associated parameter `Enabled` is `yes` by default. The workload manager can take a minute to implement the new value. For example, in order to add a new customization entry, for the `resv_enable` setting for PBS, the following `cmsh` commands can be used: **Example** ```bash [basecm10]% configurationoverlay [basecm10->configurationoverlay]% use pbspro-client [basecm10->configurationoverlay[pbspro-client]]% roles [basecm10->configurationoverlay[pbspro-client]-roles]% use pbsproclient ...->roles[pbsproclient]]% nodecustomizations ...->roles[pbsproclient]->nodecustomizations]% list Key (key) Value Enabled ------------ ------------ ------- ...->roles[pbsproclient]->nodecustomizations]% add resv_enable ...->roles*[pbsproclient*]->nodecustomizations*[resv_enable*]]% set value False ...->roles*[pbsproclient*]->nodecustomizations*[resv_enable*]]% show Parameter Value -------------------------------- ------------------------------------------------ Key recv_enable Value False Enabled yes Notes <0 bytes> ...->roles*[pbsproclient*]->nodecustomizations*[resv_enable*]]% commit ...->roles[pbsproclient]->nodecustomizations[resv_enable]]% quit ``` On a cluster with two nodes, the change would show up within a minute: ```bash [root@basecm10 ~]# pbsnodes -av | egrep '(^node|resv)' ; sleep 60 ; pbsnodes -av | egrep '(^node|resv)' node001 resv_enable = True node002 resv_enable = True node001 resv_enable = False node002 resv_enable = False ``` Different workload managers allow different kinds of node settings to be set. For example, for GE, the settings are the so-called complex attributes, as defined by the administrator. But for PBS, the node settings are built-in, and so the administrator can only change their values but not their names. So when the administrator configures node customization entries, BCM configures them differently depending on the workload manager that is used. Thus if the administrator creates a new customization entry with key name \ and value \, then this is applied to the workload managers as follows: * **Slurm**: The node customization entry is appended to the `NodeName` line of the particular nodes in `slurm.conf` in the form \=\. If the entry is removed or disabled in the BCM configuration, then the entry \=\ pair is removed from the `NodeName` line. * **PBS Professional** and **OpenPBS**: The entry is configured with the `qmgr` utility, using this command syntax: `set node NODENAME =`. If the entry is removed or disabled, then it is unset in the `qmgr`. * **GE**: The entry is applied to an exechost complex attribute that must already be defined in the configuration. The complex attribute name is set to \ and its value to \. If the entry is removed or disabled, then it is removed from the exechost settings of the node. * **LSF**: The entry is set in the `lsb.hosts` file, in the `Hosts` section. If the \ of the entry is defined as one of the section columns, then the parameter value is replaced by BCM to \. When the customization entry is removed or disabled, then its value is replaced by `()` in the section. ### 7.11.1 Other PBS Professional Customizations Examples For PBS, node customization can be carried out within the `pbsproclient` role, within the `nodecustomizations` object, as explained earlier [here](#711-custom-node-parameters). The following are other customizations that can be carried out for PBS with the `qmgr` command: **Setting a default value for `ncpus`:** To avoid having to add directives when submitting jobs, the default value for `ncpus` can be configured with the following `qmgr` command: ```bash [root@basecm10 ~]# qmgr -c "set server resources_default.ncpus = 4" ``` **Disallowing users from querying the job status of other users:** To disallow users from querying a job status other than their own, the `query_other_jobs` option can be configured with the following `qmgr` command: ```bash [root@basecm10 ~]# qmgr -c "set server query_other_jobs = False" ``` **Creating and configuring a resource:** A resource `switch` can be created and configured with the following `qmgr` command: ```bash [root@basecm10 ~]# qmgr -c "create resource switch type=string,flag=h" ``` **Setting an ACL for a queue:** The queue `workq` can have its `acl_groups` attribute set to `users` with the following `qmgr` command: ```bash [root@basecm10 ~]# qmgr -c "set q workq acl_groups=users" ```