Grouping > Categories\[default] > > Settings > User node login
*Figure 7.1: Disallowing user logins to nodes via Base View*
### 7.2.3 Disallowing Other User Processes Outside Of Workload Manager User Processes
Besides disabling user logins, administrators may choose to disable interactive jobs in the WLM as an additional measure to prevent users from starting jobs on other nodes.
Administrators may also choose to set up scripts that run after job execution. Such scripts can terminate user processes outside the WLM, as part of a policy, or for general administrative hygiene. These are `Epilog` scripts and are part of the WLM.
The WLM documentation has more on configuring these options.
### 7.2.4 High Availability By Workload Managers
NVIDIA Base Command Manager uses the existing built-in high availability (HA) functionalities of workload managers as much as possible. A double server HA configuration with the existing built-in HA functionality can be carried out using the `cm-wlm-setup` utility (section [7.3](#73-installation-of-workload-managers)), or using the Base View HA wizard (section [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)). The built-in functionality makes use of the primary WLM server and secondary WLM server, which are placed on separate nodes.
The HA configuration aspect (section [15.1.3](/admin/ha#1513-services-in-bcm-ha-setups)) in this case means that a WLM server role is assigned to both the WLM primary and the WLM secondary nodes. These primary WLM and secondary WLM servers can then:
* both be on head nodes,
* both be on regular (compute) nodes
It is not possible to configure a mixed setup—that is with one head node and one regular node—for the WLM servers on an HA setup.
In a non-HA cluster, the WLM `primaryserver` value is set to the head node by default.
**Example**
```bash
[head->wlm[slurm]]% get primaryserver
head
```
This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node.
If both the head nodes are configured with the WLM server roles, then the WLM `primaryserver` value is set to the primary cluster head node. This is because NVIDIA Base Command Manager always configures the primary WLM server on the primary cluster head node, and the backup WLM server on the secondary head node.
**Example**
```bash
[head1->wlm[slurm]]% get primaryserver
head1
[head1->wlm[slurm]]% exit
root@head1:~# ssh head2 "cmha makeactive"
...
...[logging into the newly active head node, and checking where the primary server is:]...
[head2->wlm[slurm]]% get primaryserver
head1
```
If the parameter `primaryserver` is unset in `wlm` mode, and the Slurm server role is assigned to the head nodes, then the `slurmctld` service is always started on both head nodes. During a failover, the Slurm configuration is regenerated and both `slurmctld` services are restarted. The primary `slurmctld` is always on the active head node.
If two regular nodes are configured with the WLM server roles instead, then in the cluster entity configuration the `primaryserver` parameter is set to one of the compute nodes.
The cluster primary head node and the WLM primary server should not be confused. In particular, the active cluster head node and active workload manager server are not necessarily the same for the case of the Slurm or LSF workload managers. For these, if the passive cluster head node becomes an active cluster head node without a crash, then this does not trigger a passive WLM server on the newly active head node to also become an active WLM server. Thus, the WLM server can be active on a passive head node, and vice versa, because the WLM primary server is independent of the cluster primary head node.
#### 7.2.4.1 Slurm High Availability, `scontrol takeover` And `slurmstartpolicy`
This section considers some Slurm high availability issues on running `scontrol takeover` with BCM integration.
**Logical steps during the Slurm takeover between head nodes:** Slurm high availability normally waits a short while with moving its control activity over to a passive head node that has just been made active. After a short delay, the Slurm failover algorithm is followed so that the Slurm controller (`slurmctld`) takes over operations on the newly active head node.
Slurm can speed up takeover by running `scontrol takeover` from the Bash shell on the node with the secondary `slurmctld`. Running `scontrol takeover` stops the primary `slurmctld` and triggers the Slurm failover algorithm.
This can be useful if the administrator wants to carry out maintenance on the usually active head node. Carrying out the takeover lets the Slurm service remain available, and stops the Slurm primary controller following the newly active head node.
The logic on how this works in BCM is:
1. The production state starts with
* an active primary
* a passive secondary
* the primary head node set up as the primary Slurm controller
2. the administrator runs `cmha makeactive` on the secondary passive. This causes the following to happen:
* `scontrol takeover` is run by CMDaemon on the secondary passive
* CMDaemon goes into the passive state on the primary node, and `slurmctld` on the primary node stops
* CMDaemon goes into the active state on the secondary node, and `slurmctld` on the secondary node starts
3. there is no interruption in Slurm services, because with `scontrol takeover` CMDaemon has forced the backup `slurmctld` to take over
4. the administrator can now carry out maintenance on the primary (now passive) head node
5. once maintenance has been completed, the administrator can optionally reboot the primary (passive) head node, and then make it active again with `cmha makeactive` to get back to the original production state
The problem with the preceding logic is that by default BCM automatically restarts a stopped `slurmctld` that it notices. This means that the results of `scontrol takeover` in steps 2 and 3 are unknown if carried out manually. That is, it is not practical to run the `scontrol takeover` command directly from Bash because it is possible to unintentionally configure the primary and backup `slurmctld` with an incorrect status.
To deal with this problem cleanly, CMDaemon provides the `slurmctldstartpolicy` parameter, described next.
**The `slurmctldstartpolicy` parameter and Slurm takeover behavior:** The `slurmctldstartpolicy` parameter can take the following values to decide the `slurmctld` behavior during `scontrol takeover`:
* `TAKEOVER`: The default value on an HA cluster.
* `ALWAYS`: The default value on a non-HA cluster.
* `ACTIVEONLY`: Useful only if `/cm/shared` is not mounted on both head nodes at the same time. For example, as in DAS (Direct Attached Storage), which should never be used anyway for shared storage in a production system.
1. If within NVIDIA Base Command Manager, in the `SlurmServerRole`, the `slurmctldstartpolicy` parameter is set to `TAKEOVER`, then disabling the automatic restart of the `slurmctld` service is unnecessary:
```bash
[root@basecm11 ~]# cmsh
[basecm11-]% configurationoverlay
[basecm11-]>configurationoverlay]% use slurm-server
[basecm11->configurationoverlay[slurm-server]]% roles
[basecm11->configurationoverlay[slurm-server]->roles]% use slurmserver
[basecm11->configurationoverlay[slurm-server]->roles[slurmserver]]% set slurmctldstartpolicy TAKEOVER
[basecm11->configurationoverlay*[slurm-server*]->roles*[slurmserver*]]% commit
```
The setting value can be cleared with:
```bash
[basecm11->configurationoverlay[slurm-server]->roles[slurmserver]]% set slurmctldstartpolicy ALWAYS
[basecm11->configurationoverlay*[slurm-server*]->roles*[slurmserver*]]% commit
```
2. If `cm-wlm-setup` installs Slurm on an HA cluster, then the `slurmctldstartpolicy` parameter can be set in one of the TUI screens during the session. Before HA is configured, the parameter has a default value of `ALWAYS`. After HA is configured it has a default value of `TAKEOVER`. The default value of `TAKEOVER` causes `prefailoverscript` in the `failover` submode (section [15.4.6](/admin/ha#1546-high-availability-parameters)) to take the value `/cm/local/apps/cmd/scripts/slurm.takeover.sh`
3. If `cm-wlm-setup` installs Slurm on a non-HA cluster, then the `prefailoverscript` parameter must be set within the `failover` submode of `partition` mode:
```bash
[basecm11->partition[base]->failover]% set prefailoverscript /cm/local/apps/cmd/scripts/slurm.takeover.sh
[basecm11->partition*[base*]->failover*]% commit
```
This ensures that the active head node is taken over by Slurm during a change of the active head node.
The takeover is temporary. If the head node which has the Slurm primary `slurmctld` on it is restarted, or if the service itself is restarted manually, then the primary `slurmctld` takes back its role, as per Slurm documentation.
For a Slurm cluster with a single `slurmctld`, where the Slurm server role is assigned to a single compute node or the BCM cluster has only one head node, the parameter `slurmctldstartpolicy` should be set to `ALWAYS`, which is the default.
Table [7.2.4.1](#7241-slurm-high-availability-scontrol-takeover-and-slurmstartpolicy) shows what `slurmctld` does when the `slurmctldstartpolicy` parameter is set to `TAKEOVER` and when in `SlurmWlmCluster` the parameter `primaryserver` is set. The detailed behavior per head node depends on whether the head nodes are active or primary:
| **Node pair** | **Is `slurmctldstartpolicy` TAKEOVER?** | **Active node?** | **Slurm primary node?** | **`slurmctld` starts?** |
|---|---|---|---|---|
| Head001 | `no` | \* | \* | yes |
| Head002 | `no` | \* | \* | yes |
| Head001 | `yes` | yes | yes | yes |
| Head002 | `yes` | no | no | yes |
| Head001 | `yes` | yes | no | yes |
| Head002 | `yes` | no | yes | no |
| Head001 | `yes` | no | yes | no |
| Head002 | `yes` | yes | no | yes |
| Head001 | `yes` | no | no | yes |
| Head002 | `yes` | yes | yes | yes |
Devices > All Devices > Head Node
\> Settings > Roles
\> slurmserver
*Figure 7.16: Workload management role assignment on a head node*
Overriding the configuration overlay assignment is possible, at node or category level. For example at node level the role can be defined by assignment at node level, using the navigation path:
Devices > All Devices > Head Node
\> Settings > Roles
\> +Add > SlurmServerRole
This allows parameters to be changed within the role. For example, for the `SlurmServerRole` the `Slurmctld start policy` can then be modified at node level.
Workload manager options that are not regarded as server role options can be found under the workload manager object. For Slurm this is in the navigation path:
HPC > Workload Management Clusters
For example, for a Slurm instance in the Workload Management Clusters pane, a `builtin` or `backfill` option can be set for the `Scheduler` parameter (figure [7.17](#fig-base-view-workloadmanagement-choice-of-workloadmanager-head-scheduler)).
*Figure 7.17: Workload management role assignment options on a head node*
To have the server start up on non-head nodes (but not for a head node), the `imageupdate` command (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) can be run. The workload manager server process and any associated schedulers then automatically start up.
**Workload Manager Client** Similarly, the workload manager client process can be enabled on a node or head node by having the workload manager client role assigned to it at a node level. Some basic options can be set for the client role.
Saving the role, and then running `imageupdate` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)), automatically starts up the client process with the options chosen, and managed by CMDaemon.
#### 7.4.1.2 Workload Manager Role Assignment To A Category With Base View
It is true that workload manager role assignment can be done as described in the preceding text for individual non-head nodes. However it is usually more efficient to assign roles using categories or configuration overlays, due to the large number of compute nodes in typical clusters.
For example, the case can be considered of all physical on-premises non-head nodes. By default these are in the `default` category. This means that, by default, roles in the category are automatically assigned to all those non-head nodes, unless, as an exception, an individual node configuration overrides the category setting and uses a role setting instead at node level.
The workload manager roles for the category `default` can be viewed by using the navigation path: Grouping > Categories\[default] > Settings > Roles
If the role has been assigned at category level, then clicking through to the subpane for a role allows role options to be edited and saved.
However,the default configuration using `cm-wlm-setup` assigns the role at configuration overlay level, which means that the role options (role settings) can
* either be configured using the navigation path from the configuration overlay level:
`Grouping > Configuration Overlays[`*configuration overlay role*`] > Settings > Roles > `*role under configuration overlay role* > Role settings
* or they can be configured by adding a role at category level (or node level), with the `+Add` option. For example, for category level using the `default` category:
Grouping > Categories\[default] > Settings > Roles > + Add >*category-level role*
For compute nodes, the role assigned is usually a workload manager client. If the assigned role is that of a workload manager client, then the node with that role can have queues, GPUs, and other parameters specified for it.
For example, queues can then be assigned using the navigation path:\
`HPC > WLM Management Clusters[`*Slurm instance* ] > Job Queues while GPUs, if using Slurm as the workload manager with default settings, with a default `cm-wlm-setup` installation, have a configuration overlay `slurm-client-gpu` that is specified using the navigation path:
Grouping > Configuration Overlays > slurm-client-gpu
\> Settings > roles
\> slurmclient
\> Generic Resources
\> gpu
The workload manager server role can also be assigned to a non-head node. For example, a Slurm server role can be taken on by a non-head node. This is the equivalent to the `--server-nodes` option of `cm-wlm-setup`.
Saving the roles with their options and then running `imageupdate` (section [5.6.2](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)) automatically starts up the newly-configured workload manager.
#### 7.4.1.3 Workload Manager Role Options With Base View
Each workload manager client role (sometimes loosely called a compute node role) has options that can be set for `GPUs`, `Queues`, and `Slots`. Generally, the value that is set for `Slots` is the number of jobs expected to run on a node simultaneously. This number can, for example, be set to the number of threads. Threads (virtual cores) in the x86_64 architecture are provided by Intel’s hyper-threading (HT), or by AMD’s simultaneous multithreading (SMT).
The physical CPU, the cores on the CPU, and the threads of a core (HT, SMT) should not be confused with each other. They are distinct concepts, and can all have different values.
* `Slots`, in a workload manager, corresponds in BCM to:
* the `CPUs` setting (a `NodeName` parameter) in Slurm’s `slurm.conf`
* the `-n` flag, or the total number of processor slots requested, in IBM Spectrum LSF
* the `nproc` (or `ncpus`) setting in PBS,
In LSF setting the number of slots for the client role to `0` means that the client node does not run jobs on itself, but becomes a submit host, which means it is able to forward jobs to other client nodes.
The default value for `Slots` is `AUTO`, which means that the value for `Slots` is auto-detected. The parameter may be alternatively be set to a non-negative number. Each WLM has a different implementation on how this is done. For instance, for Slurm, BCM uses Slurm’s own auto-detection implementation.
* `Queues` with a specified name are available in their associated role after they are created. The creation of queues is described in sections [7.6.2](#762-queues-display-and-handling-in-base-view) (using Base View) and [7.7.3](#773-job-queue-display-and-handling-in-cmsh-jobqueue-mode) (using `cmsh`).
All server roles also provide the option to enable or disable the `External Server` setting. Enabling that means that the server is no longer managed by BCM, but provided by an external device.
The `cmsh` equivalent of enabling an external server is described [here](#cmshexternalservertorque).
### 7.4.2 Enabling And Disabling A Workload Manager With `cmsh`
A particular workload manager package may be set up, but not enabled. This can happen, for example, if no WLM server or WLM client role has been assigned.
If a WLM instance exists, then the WLM client, server, or submit roles can be enabled from `cmsh` by assigning it from within the `roles` submode. Within the assigned role, the properties of the WLM may be further configured by setting options.
#### 7.4.2.1 Workload Manager Role Assignment To A Configuration Overlay With `cmsh`
In `cmsh`, workload manager role assignment to a configuration overlay (section [2.1.5](/admin/clustermanagement#215-roles)) can be done using `configurationoverlay` mode. By default `cm-wlm-setup` run as a TUI session creates some configuration overlays with suggestive names, and assigns roles to the configuration overlays according to what the names suggest. Thus, for example, with the `cm-wlm-setup` TUI session used to carry out an express setup for Slurm, the configuration overlays that get created are the following:
**Example**
```bash
[basecm11->configurationoverlay]% list
Name (key) Priority All head nodes Nodes Categories Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting 500 yes slurmaccounting
slurm-client 500 no default slurmclient
slurm-server 500 yes slurmserver
slurm-submit 500 no default slurmsubmit
wlm-headnode-submit 600 yes slurmsubmit
```
Nodes in the default category can take on the `slurmclient` or `slurmsubmit` role by setting the nodes for the role using the associated configuration overlays `slurm-client` or `slurm-submit`.
The `wlm-headnode-submit` configuration overlay is a special overlay. It is applied only to the head node, and is shared among all installed workload managers. Setting this overlay means that the head node, by default, has a submit role for a given workload manager.
**Example**
```bash
[basecm11->configurationoverlay]% use slurm-client
[basecm11->configurationoverlay[slurm-client]]% show
Parameter Value
-------------------------------- ------------------------------------------------
Name slurm-client
Revision
All head nodes no
Priority 500
Nodes
Categories default
Roles slurmclient
Customizations <0 in submode>
[basecm11->configurationoverlay[slurm-client]]% set nodes
node001 node002 node003 basecm11
[basecm11->configurationoverlay[slurm-client]]% set nodes node001..node002
[basecm11->configurationoverlay*[slurm-client*]]% commit
[basecm11->configurationoverlay[slurm-client]]% list
Name (key) Priority All head nodes Nodes Categories Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting 500 yes slurmaccounting
slurm-client 500 no node001,node002 default slurmclient
slurm-server 500 yes slurmserver
slurm-submit 500 no default slurmsubmit
wlm-headnode-submit 600 yes slurmsubmit
[basecm11->configurationoverlay[slurm-client]]%
```
All the head nodes can also be made to take on the configuration overlay role by setting its `All head nodes` value to `yes`. The union set of `All head nodes` with `Nodes` is the set of nodes to which the role is applied for that configuration overlay.
Values for the parameters in a role, such as the `slurmclient` role, can be set within the configuration overlay:
**Example**
```bash
[basecm11->configurationoverlay[slurm-client]]% roles
[basecm11->configurationoverlay[slurm-client]->roles]% use slurmclient
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% show
Parameter Value
---------------------------------- -------------------------------------
Name slurmclient
Revision
Type SlurmClientRole
Add services yes
WLM cluster slurm
Slots 0
All queues no
Queues defq
Features
Sockets 0
Cores per socket 0
Threads per core 0
Boards 0
Sockets per board 0
Real memory 0B
Node address
Weight 0
Port 0
Tmp disk 0
Reason
CPU spec list
Core spec count 0
Mem spec limit 0B
GPU auto detect BCM
Node customizations <0 in submode>
Generic resources <0 in submode>
Cpu bindings None
Slurm hardware probe auto detect yes
Memory autodetection slack 10.0%
IMEX no
Write procs always no
Write only Procs no
Add to topology yes
Nodesets
Power profiles
Devices > All Devices > basecm11
\> Settings > Services
leads to a display similar to figure [7.18](#fig-base-view-head-services):
*Figure 7.18: Services seen on head node in Base View*
For a regular node, a similar navigation path for `node001`, for example, is:
Devices > All Devices > node001
\> Settings > Services
and leads to a display similar to figure [7.19](#fig-base-view-regular-node-services):
*Figure 7.19: Services seen on regular node in Base View*
#### 7.4.3.2 Viewing Roles In Base View
Services are often set with BCM roles (section [2.1.5](/admin/clustermanagement#215-roles)). The navigation path:\
Devices > All Devices > basecm11 > Settings > Roles\
shows the list of roles. Figure [7.20](#fig-base-view-head-wlm-roles) shows some of the Slurm roles that result in Slurm services running on the head node:
*Figure 7.20: Roles seen on head node in Base View*
Similarly, the navigation path:\
Devices > All Devices > node001 > Settings > Roles\
shows the list of roles for a regular node. Figure [7.21](#fig-base-view-regular-node-wlm-roles) shows some of the Slurm roles that result in Slurm services running on the regular nodes:
*Figure 7.21: Roles seen on regular node in Base View*
The roles seen in these figures are from the defaults that `cm-wlm-setup` provides in an express setup.
For regular nodes, the inheritance of roles from category level or configuration overlay level is indicated by the values in the parentheses alongside the role name. Thus, in figure [7.21](#fig-base-view-regular-node-wlm-roles), the `slurmsubmit` and `slurmclient` roles are decided by the default setting from the configuration overlay.
The assignment of roles can be varied to taste for WLMs. This allows WLM services to run on the head node or on the regular nodes.
#### 7.4.3.3 Viewing Services And Roles In `cmsh`
From `cmsh` the services states are viewable from within `device` mode, using the `services` command. One-liners from the shell to illustrate this are (output elided):
**Example**
```bash
[root@basecm11 ~]# cmsh -c "device services node001; status"
Service Status
------------ -----------
...
munge [ UP ]
named [ UP ]
nslcd [ UP ]
...
slurmd [ UP ]
[root@basecm11 ~]# cmsh -c "device services basecm11; status"
Service Status
------------ -----------
...
munge [ UP ]
...
slurmctld [ UP ]
slurmdbd [ UP ]
```
Roles can be viewed from within the main modes of `configurationoverlay`, `category`, or `device`. One-liners to view these are:
**Example**
```bash
[root@basecm11 ~]# cmsh -c "configurationoverlay; list"
Name (key) Priority All head nodes Nodes Categories Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-client 500 no default slurmclient
slurm-server 500 yes slurmserver
wlm-headnode-submit 600 yes slurmsubmit
[root@basecm11 ~]# cmsh -c "category; use default; roles; list -p"
Name (key)
----------------------------
[overlay:slurm-client:500] slurmclient
[root@basecm11 ~]# cmsh -c "device use node001; roles; list -p"
Name (key)
----------------------------------------
[overlay:slurm-client:500] slurmclient
[root@basecm11 ~]# cmsh -c "device use basecm11; roles; list -p"
Name (key)
----------------------------------------
[750] backup
[750] boot
[750] firewall
[750] headnode
[750] monitoring
[750] provisioning
[750] storage
[overlay:slurm-server:500] slurmserver
```
The `-p|--priority` option displays of the `list` command the priority setting for the roles.
## 7.5 Configuring And Running Individual Workload Managers
BCM deals with the various choices of workload managers in as generic a way as possible. This means that not all features of a particular workload manager can be controlled, so that fine-tuning must be done through the workload manager configuration files. Workload manager configuration files that are controlled by BCM should normally not be changed directly because BCM overwrites them. However, overwriting by CMDaemon is prevented on setting the directive:
`FreezeChangesTo` \<*workload manager*>`Config =
Grouping > Configuration Overlays > slurm-client-gpu
\> Settings > Roles
\> slurmclient
\> Power profiles
The `Power profiles` parameters as shown in `cmsh` are:
1. `Disabled`: forbids a job from changing workload power profiles (`yes` by default)
2. `Fail on error`: allows prolog and epilog to fail if a workload power profiles change fails for any reason
3. `Job keyword`: a keyword, `wpps`, in the JSON object, used by the WLM job, to specify a workload power profiles setting in the comment field.
4. `Jobs profiles directory`: a top directory which includes job ID directories, and used by the job to change the workload power profiles
5. `Debug`: setting it to `yes` enables prolog and epilog debug messages The prolog and epilog then write debug logs to the location set by the `Debug log directory` parameter,
6. `Debug log directory`: directory where prolog and epilog create debug log files per job. The logs take the format: `/var/spool/cmd/wlm/wpps/` \<*JOB ID*>`.log` by default. The log files are not cleaned up automatically.
If WPPS is enabled then:
* If a job requires static job profiles management, then special BCM prolog and epilog scripts are enabled in Slurm on compute nodes. The scripts parse the `SLURM_JOB_COMMENT` environment variable that is passed by Slurm. The scripts search for a JSON object that includes the key `wpps`. The key used can be changed using the `Job keyword` parameter. The expected format of the JSON object is:
```bash
{"wpps": {"profiles": PROFILES}}
```
Here, `PROFILES` is either
* a JSON list of strings (workload profile names or their numbers) or
* a single string (one workload profile name) or
* a workload profile number.
The values that `PROFILES` can take are:
* `max_p`
* `max_q`
* `llm_training`
* `llm_inference`
On matching, the specified profiles are set for all the GPUs on the node that are allocated for the job when the job starts.
For example, the following job submission command instructs the prolog script to set the workload profiles to `max_p` and `compute`:
```bash
$ sbatch --gres=gpu:b100:8 --comment='{"wpps": {"profiles": ["max_p", "compute"]}}' job.sh
```
All allocated GPUs on the node get the specified workload profiles, which means that job allocation must be configured to run on that node as a job that excludes other jobs running on it. If the cluster administrator has set oversubscribe ([here](/admin/configuring#31726-gpu-configuration-for-hpc-workload-managers)) for Slurm, then that takes precedence over exclusivity, and must therefore be taken into account.
* If a job requires dynamic profiles management, then the job process can notify BCM to update the power profiles of specific GPUs during job execution. The prolog creates a directory of the form `/var/run/nvidia/workload-power-profiles/
Grouping > Configuration Overlays > slurm-client-gpu
\> Settings > Roles
\> slurmclient
\> Generic resources
\> + Add
provides the equivalent (figure [7.22](#fig-base-view-workloadmanagement-slurmgres-nvidiaconfig)):
*Figure 7.22: Base View access to NVIDIA GPU configuration options*
After the generic resources are committed, BCM updates the `gres.conf` file.
Since NVIDIA Base Command Manager version 8.2, a single `gres.conf` configuration file, located at `/cm/shared/apps/slurm/etc/slurm/gres.conf` is used.
If the category consists of `node001` and `node002`, then the entries to the `gres.conf` file in this case would look like:
**Example**
```bash
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia0 Cores=0-7
NodeName=node[001,002] Name=gpu Type=k20xm Count=1 File=/dev/nvidia1 Cores=8-15
# END AUTOGENERATED SECTION -- DO NOT REMOVE
[root@basecm11 ~]#
```
**Slurm topology configuration:**
Slurm supports topology-aware resource allocation to optimize job performance. BCM supports configuration for the `topology/tree` and `topology/block` Slurm plugins using the `topograph` service ([here](#topograph)):
* `topology/tree`: models the cluster network as a hierarchical structure, where Slurm allocates resources to jobs to minimize network contention. Nodes are grouped under leaf switches, which connect to higher-level switches. The plugin helps to allocate nodes that are close together in the network hierarchy. The plugin requires all nodes for a single job be connected using switches. The configuration of the plugin is defined using switches and their relationships.
* `topology/block`: models the network as non-overlapping blocks of nodes. The plugin aims to keep jobs contiguous within blocks for better communication. The plugin does not require all nodes to be part of blocks. The configuration is defined using blocks of nodes and their groupings.
The network topology configuration for a Slurm instance \<*slurmcluster*> is described by:\
`/cm/shared/apps/slurm/etc/` \<*slurmcluster*>`/topology.conf`
The file is modified in an autogenerated section when the cluster administrator modifies the internal topology settings in BCM, or when the topology arrangement record is fetched from Topograph, the topology generation service ([here](#topograph-service-integration)). The topology settings can be modified in BCM via the `topologysettings` submode of the WLM instance in `cmsh`:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% wlm use slurm
[basecm11->wlm[slurm]]% topologysettings
[basecm11->wlm[slurm]->topologysettings]% show
Located Parameter Value
-------------------------------- ------------------------------------------------
Topology plugin Block
Topology source Internal
Parameters | **Base View** | **Slurm Object Value Within** | **Description** |
| **Option** | **WLM Mode Of `cmsh`** | |
| `Select Type` | `SelectType` | Identifies the type of resource selection algorithm to be used. Changing this value causes the `slurmctld` daemon to be restarted. Default: Consumable trackable resource: `select/cons_tres` |
| `Select Type Parameters` | `SelectTypeParameters` | Parameters for the resource selection algorithm. Acceptable values for these parameters depend upon the algorithm selected. Default: Core as a consumable resource: `CR_Core` |
| `Accounting Storage TRES` | `AccountingStorageTRES` | Comma-separated list of resources to be tracked on the cluster, requested by an `sbatch` / `srun` job on submission. Default: GPU as a generic resource: `gres/gpu` |
| **Base View Option** | **Slurm Option** | **Description** |
|---|---|---|
| `Features` | `Feature=<` *string* `>` entry in the file `slurm.conf` | Arbitrary strings can be entered to indicate some characteristics of a node, one string per entry. For example: `text1` `text2` and so on. These become part of the: `Feature=text1,text2...` attribute to the `NodeName=<` *`node name`* `>` entry line in `slurm.conf` , as indicated in `man slurm.conf.5` . The strings also become added attributes to the `GresTypes` entry of that file. Default: blank. |
| `File=<` *device* `>` entries in the file `gres.conf` | This is only implemented if the number of GPUs has been set by the administrator using BCM, by setting the basic Slurm client option `GPUs` . The `GPUs` option is seen, for example, in Base View, using the navigation path `Grouping` \> `Node categories[default]` \> `Edit` \> `Settings` `Roles` \> | |
| `slurmclient` \> `GPUs` . Each device detected on the host node can be added as an entry. For example: `/dev/nvidia0` `/dev/nvidia1` and so on. Alternatively, they can be added as a range in one line, for example: `/dev/nvidia[0-1]` These become part of the: `Name=gpu File=<` *device* `>` lines in `gres.conf` , as indicated in `man gres.conf.5` . Default: blank | ||
| `CPU` | Number of logical processors on the node. For Slurm 20 and beyond, CMDaemon detects the number of CPU cores, and sets `procs=` *`number of cores`* using slots autodetection. Default for Slurm prior to version 20: `0` | |
| `Sockets` | `Sockets` | Processor chips on node. If this is defined, then `SocketsPerBoard` must not be defined. Default: `0` |
| `Cores per socket` | `CoresPerSocket` | Number of cores per socket. Default: `0` |
| `ThreadsPerCore` | `ThreadsPerCore` | Number of logical threads for a single core. Default: `0` |
| `Boards` | `Boards` | Number of baseboards in a node. Default: `0` |
| `SocketsPerBoard` | `SocketsPerBoard` | Number of processor chips on baseboard. If this is defined, then `Sockets` must not be defined. Default: `0` |
| `RealMemory` | `RealMemory` | Size of real memory on the node, MB. Default: `0` |
| `NodeHostname` | `NodeHostname` | Default: as defined by Slurm’s `NodeName` parameter. |
| `NodeAddr` | `NodeAddr` | Default: as set by Slurm’s `NodeHostname` parameter. |
| `State` | `State` | State of the node with user jobs. Possible Slurm values are: `DOWN` , `DRAIN` , `FAIL` , `FAILING` , and `UNKNOWN` . Default: `UNKNOWN` |
| `Weight` | `Weight` | The priority of the node for scheduling. Default: `0` |
| `Port` | `Port` | Port that slurmd listens to on the compute node. Default: as defined by `SlurmdPort` parameter. If `SlurmdPort` is not specified during build: Default: `6818` . |
| `TmpDisk` | `TmpDisk` | Total size of Slurm’s temporary filesystem, `TmpFS` , typically `/tmp` , in MB. `TmpFS` is the storage location available to user jobs for temporary storage. Default: `0` |
| `Options` | `extra options` | Extra options that are added to `slurm.conf` |
Devices > All Devices > Head Node > Settings > Roles > + Add > LsfServerRole
* Within `cmsh`: For a particular category in the `category` mode, or a particular device in the `device` mode, the `roles` submode is chosen. Within the `roles` submode, the `lsfserver` object can be assigned or used. The following example shows how to set the LSF `prefix` parameter for the `default` category.
**Example**
```bash
[root@basecm11~]# cmsh
[basecm11]% configurationoverlay use lsf-server; roles
[basecm11->configurationoverlay[lsf-server]->roles]% use lsfserver
[basecm11->...[lsfserver]]% set externalserver yes
[basecm11->...[lsfserver*]]% commit
```
The global LSF cluster settings can be modified as follows:
* Within Base View:\
For example, for a head node `basecm11`, using a navigation path of:\
`HPC > Workload Management Clusters > `*WLM (LSF) instance* Devices > All Devices > node001 > Settings > Roles > + Add > LSFClientRole\
lets options be set for a regular node `node001` (figure [7.29](#fig-base-view-node-roles-lsf)):
*Figure 7.29: Base View access to LSF configuration options using roles window options*
Available options are:
* **Slots**: The number of CPUs per node. By default LSF tries to determine the value automatically with the `auto` setting. If the number of slots is set to `0`, then the node becomes an LSF submit host, so that no jobs can run on the host, but users can submit their jobs from this host using the LSF utilities.
* **Queues**: `All` queues can be set, or queues can be selected.
* **Server**: Whether the node is an LSF server
* **Host Model**, **Host Type**: Possible values for these are defined in `lsf.shared`
* **GPUs**: The number of GPUs per node.
* **Node Customizations**: LSF node custom properties (section [7.11](#711-custom-node-parameters)).
From `cmsh` these properties are accessible from within the appropriate node or category `roles` submode (section [7.4.2.4](#7424-setting-options-for-workload-manager-settings-with-cmsh)).
**Submission hosts:** By default, all compute nodes to which the `lsfclient` role is assigned, also become submission hosts. This means that LSF users can submit their jobs from those compute nodes. If the administrator wants to allow users to submit their jobs from a non-compute node—for example from a login node—then the `lsfsubmit` role should be assigned, for example:
**Example**
```bash
[root@basecm11~]# cmsh
[...->configurationoverlay[lsf1-submit]]% roles
[...->roles]% assign lsfsubmit
[...->roles*[lsfsubmit*]]% append lsfclusters lsf1
[...->roles*[lsfsubmit*]]% commit
[...->roles[lsfsubmit]]%
```
Configuring a node according to the preceding steps then allows users to submit their jobs from that node, without the submitted jobs getting scheduled to run on that node.
If more than one LSF cluster is set up, and they share the same submit host, then a single `lsfsubmit` role can be used. In this case, all the LSF cluster names should be appended to the `lsfcluster` parameter of the role.
**Further configuration:** For further configuration the *Administering Platform LSF* manual provided with the LSF software should be consulted.
#### 7.5.2.3 Running LSF
Role assignment and role removal enables and disables LSF from Base View (sections [7.4.1](#741-enabling-and-disabling-a-wlm-with-base-view)) or `cmsh` (section [7.4.2](#742-enabling-and-disabling-a-workload-manager-with-cmsh)).
An active LSF master service (typically, but not necessarily on a head node) has the following LSF-related processes running on it:
| **Process/Service** | **Description** |
|---|---|
| `res` | Remote Execution Server* |
| `sbatchd` | client batch job execution daemon* |
| `mbatchd` | master batch job execution daemon |
| `eauth` | External Authentication method |
| `lim` | Load Information Manager* |
| `pim` | Process Information Manager* |
| `pem` | Process Execution Manager* |
| `vemkd` | Platform LSF Kernel Daemon |
| `egosc` | Enterprise Grid Orchestrator service controller |
| `mbschd` | master batch scheduler daemon |
HPC > Workload Management Clusters > Job Queues displays a list of queues available (figure [7.33](#fig-base-view-workloadmanagement-queues-main)).
*Figure 7.33: Workload manager queues*
Queues can be added with the `+ Add` option, or deleted with the ⋮ option. Opening a screen with the ↗ option allows queue parameters for an existing to be set, for example as in figure [7.34](#fig-base-view-workloadmanagement-queues-edit-slurm).
*Figure 7.34: Workload manager queue parameters configuration*
## 7.7 Using `cmsh` With Workload Management
The `wlm` mode in `cmsh` gives the administrator access to the WLM instances running on the cluster.
The `cmsh` tree (Appendix [M](/admin/cmshtree)) has the following submodes structure under the `wlm` mode:
```bash
`-- wlm
|-- accounting
|-- cgroups
|-- chargeback
|-- jobqueue
|-- jobs
|-- ocisettings
`-- placeholders
```
* `accounting`: provides ways to set some advanced Slurm job accounting properties ([here](#7516-advanced-slurm-job-accounting))
* `cgroups`: provides ways to configure WLMs using the cgroup mechanism of the Linux kernel (section [7.10](#710-cgroups))
* `chargeback`: provides ways to measure the costs of requested IT resources for WLM jobs (Chapter [13](/admin/monitoringchargeback))
* `jobqueue`: allows WLM job queues to be managed (section [7.7.3](#773-job-queue-display-and-handling-in-cmsh-jobqueue-mode))
* `jobs`: allows WLM jobs to be viewed and managed (section [7.7.2](#772-the-jobs-submode-in-cmsh))
* `ocisettings`: allows OCI settings ([Chapter 2 of the *Cloudbursting Manual*](/cloudbursting/codcloud)) to be viewed and managed
* `placeholders`: allows nodes to be configured in advance, for planning resource use in WLMs (section [8.4.8](/admin/cm-scale#848-queue-node-placeholders))
In addition to these submodes, `wlm` mode provides several commands at the entity level:
* `activeservers`: displays the active WLM server nodes (section [7.7.1](#771-the-activeservers-command))
* `grid`: shows a visual grid of historic job information (section [7.7.2.6](#7726-the-grid-command))
* `filter`: filters job information across WLM instances ([here](#7724-the-filter-command-within-jobs-submode))
* `statistics`: displays job statistics ([here](#7725-the-statistics-command))
The monitoring commands `healthoverview` ([here](/admin/monitoring#1065-monitoring-data-health-overviewthe-healthoverview-command)), `samplenow` ([here](/admin/monitoring#10621-the-samplenow-command-for-on-demand-measurable-samples)), and `monitoringactionhistory` ([here](/admin/monitoring#1066-monitoring-data-about-the-monitoring-action-historythe-monitoringactionhistory-command)) are also available in `wlm` mode, and described in Chapter [10](/admin/monitoring).
For basic WLM management using `cmsh`, the administrator is expected to use `wlm` mode with the `jobs` and `jobqueue` submodes.
If there is just one WLM instance running on the cluster, then an administrator that accesses `wlm` mode drops straight into the object representing that WLM instance, and can then access the submodes.
If there is more than one WLM running, then the administrator must select a WLM instance from the top level `wlm` mode, with the `use` command, in order to access the submodes.
While at the top level `wlm` mode, the instances can be listed. The list shows the nodes used by the instance, and the node roles assigned to the nodes.
Suppose BCM is being used to manage two WLM instances:
* one Slurm WLM, named `slr`
* one Slurm WLM, named `uc`
with their nodes having been allocated roles as follows:
**Example**
```bash
[basecm11->wlm]% list
Type Name (key) Server nodes Submit nodes Client nodes
------ ------------------------ ------------ ------------------------- ----------------
Slurm slr node001 basecm11,node001..node003 node002,node003
Slurm uc node004 basecm11,node004..node006 node005,node006
```
Then one way to list all jobs running per WLM instance, queue, and user, is with the `filter` command:
```bash
[basecm11->wlm]% filter --running
WLM Job ID Job name User Queue Submit time Start time End time Nodes Exit code
---- ------ --------- ----- ------ ----------- ------------ --------- ----------------- ----------
slr 103 sjob maud defq 15:43:02 15:43:03 N/A node002,node003 0
slr 104 sjob maud defq 15:43:02 15:43:25 N/A node002,node003 0
slr 105 sjob maud defq 15:43:02 15:43:33 N/A node002,node003 0
slr 106 sjob maud defq 15:43:02 15:43:36 N/A node002,node003 0
slr 171 sjob maud defq 14:20:09 14:20:23 N/A node002,node003 0
uc 200 sjob fred batch 14:22:18 14:23:20 N/A node005,node006 0
```
The `filter` command can also be run within the `jobs` mode, for a particular WLM instance, and is described in more detail [here](#7724-the-filter-command-within-jobs-submode).
An alternate way to list all jobs from `wlm` mode is with a `foreach` command to descend into `jobs` submode to list the jobs for each WLM instance:
**Example**
```bash
[basecm11->wlm]% foreach * (jobs ; list)
Type Job ID User Queue Running time Status Nodes
----- ------- ----- ------ ------------ ---------- ----------------
Slurm 55 maud defq 18m 10s COMPLETED node002,node003
Slurm 56 maud defq 18m 5s COMPLETED node002,node003
Slurm 57 maud defq 2m 11s RUNNING node002,node003
Slurm 58 maud defq 0s PENDING
Type Job ID User Queue Running time Status Nodes
----- ------- ----- ------ ------------ ---------- ----------------
Slurm 96 maud batch 2m 30s RUNNING node005,node006
Slurm 97 maud batch 0s PENDING
Slurm 98 maud batch 0s PENDING
```
The `jobs` submode is now discussed further.
### 7.7.1 The `activeservers` Command
The `activeservers` command displays the nodes currently acting as active WLM servers. It helps a cluster administrator check which head or server node is actually handling the workload manager, particularly in high-availability configurations.
**Example**
```bash
[basecm11->wlm]% activeservers
slurm basecm11
```
The output shows the WLM instance name and the hostname of its active server node.
### 7.7.2 The `jobs` Submode In `cmsh`
Within the `jobs` submode of a WLM instance, the administrator can list jobs that are running or queued up for that particular instance. For example, the running and queued jobs listed for a Slurm instance may be displayed as:
**Example**
```bash
[basecm11->wlm[slurm]->jobs]% list
Type Job ID User Queue Running time Status Nodes
--------- ------------- ----------- ------------ ------------ -------- -------------
Slurm 3117 user1 hydroq 1s RUNNING node002
Slurm 3118 user2 workq 1s RUNNING node001
Slurm 3119 user1 hydroq 0s PENDING
Slurm 3120 user3 hydroq 0s PENDING
Slurm 3121 user1 hydroq 0s PENDING
Slurm 3122 user1 hydroq 0s PENDING
```
The number of jobs can be very large. The `wait` command in `jobs` mode can be used to provide a shorter running summary. The following `wait` command displays a running summary that regularly displays a tally of the running and pending jobs; waits for all of them to complete; and has the waiting-for-completion condition overruled by a timeout of 10 minutes:
**Example**
```bash
[basecm11->wlm[slurm]->jobs]% wait --timeout 10m --all
0s| running: 4, pending: 12
1m| running: 3, pending: 12
2m| running: 4, pending: 11
5m| running: 4, pending: 10
7m| running: 4, pending: 9
9m| running: 3, pending: 8
* timeout *
```
By default `wait` runs every minute until the specified timeout. Also, by default, as indicated by the missing 3rd and 4th minutes in the example, the outputs are only displayed with the tally changes. Further options to the `wait` command within `jobs` mode can be found in the help text display output when `help wait` is run.
In the `foreach` jobs listings for the two Slurm WLM instances shown earlier, the jobs are in a running, queued up, or completed state. They are also all being run by a user `maud`, who happens to be making use of both workload managers.
For Slurm, completed jobs are shown for a short time with the status `COMPLETED`.
#### 7.7.2.1 Commands That Change The Status Of A Queued Or Running Job
Within a `jobs` submode, the following commands, used with a job ID, can change the status of a particular queued or running job:
* `hold`: puts a queued job into a `hold` state. This prevents the job from being considered for running.
* `release`: releases a job from a `hold` state, putting it back in the queue, so that it can be considered for running.
* `suspend`: pauses a running job
* `resume`: resumes a suspended job
* `remove`: removes a job
For example, continuing the case earlier on of the `cmsh` session with the `foreach` listing ([here](#77-using-cmsh-with-workload-management)):
The administrator can `suspend` the running job with job ID 57 on the Slurm instance as follows:
**Example**
```bash
[basecm11->wlm]% jobs slr; suspend 57
[basecm11->wlm[slr]->jobs]% list | head -2; list | grep 57
Type Job ID User Queue Running time Status Nodes
----- ------- ----- ------ ------------ ---------- ----------------
Slurm 57 maud defq 2m 51s SUSPENDED node002,node003
```
The administrator can then `resume` the suspended job ID 57 as follows:
**Example**
```bash
[basecm11->wlm[slr]->jobs]% resume 57
```
#### 7.7.2.2 Commands To Inspect Jobs:
Within the `jobs` submode, the following commands can be used to inspect a job:
* `show` \<*job ID*>: properties are shown for a running or pending job.
The properties that are displayed are taken directly from commands associated with particular workload managers:
* For Slurm: `scontrol show jobs`
* For LSF: `bjobs`
* `info` \<*job ID*>: details are shown about a particular job ID. The job can be `pending`, `running`, or `completed` jobs. The data is taken from the BCM database.
* `statistics`: statistics are shown for jobs. More detail on the `statistics` command is given [here](#7725-the-statistics-command).
* `list`: lists pending and running job IDs along with some other associated properties. Recently completed jobs are also displayed. Completed jobs do not remain listed for longer than a few minutes. The `filter` command (section [7.7.2.4](#7724-the-filter-command-within-jobs-submode)) can be used instead to display all jobs for that WLM instance, using data taken from the BCM database.
* `dumpmonitoringdata [`*OPTIONS*`] [` \<*start-time*>` <`*end-time*>] \<*metric*>` <`*job id*>: Displays the monitoring data for a specific metric or healthcheck for a running or completed job. Example: `dumpmonitoringdata CtxtSwitches 170`
More detail is given on the `dumpmonitoringdata` command in section [10.6.4](/admin/monitoring#1064-data-values-over-a-periodthe-dumpmonitoringdata-command).
* `latestmonitoringdata [`*OPTIONS*`]` \<*job ID*>: Displays the last measured job monitoring data for the job.
* `latesthealthdata` \<*job ID*>: The subset of health checks within the last measured monitoring data.
* `latestmetricdata` \<*job ID*>: The subset of metrics within the last measured monitoring data.
More detail is given on the `latest*data` commands in section [10.6.3](/admin/monitoring#1063-the-latest-data-and-counter-valuesthe-latestdata-and-latestmetriccounters-commands).
* `measurables` \<*job ID*>: The measurables used for job monitoring data.
* `healthchecks` \<*job ID*>: The subset of health checks within the measurables used for job monitoring data.
* `metrics` \<*job ID*>: The subset of metrics within the measurables used for job monitoring data.
* `enummetrics` \<*job ID*>: The subset of enummetrics within the measurables used for job monitoring data.
#### 7.7.2.3 The `info` Command Within `jobs` Submode
Within the `jobs` submode of `wlm`, the `info` \<*job ID*> command shows job information details stored in the BCM database. BCM starts updating the job information in its database as soon as it detects a new job. When the job is completed BCM pulls a final job state and exit code from the workload manager.
**Example**
```bash
[basecm11->wlm[slr]->jobs]% info 3
Parameter Value
-------------------------------- ------------------------------------------------------
Job ID 3
Revision
Job name matrix-math---model-other-model-args:3-tasks:1.sbatch
Job array ID
Job task ID
User joe
Group joe
Account math
Accounting info < not set >
Workload manager cluster slurm
Queue defq
Nodes node002
Submit time 27/07/2026 02:35:24
Start time 27/07/2026 02:35:24
End time 27/07/2026 02:36:25
Run time 1m 1s
Persistent no
Exit code 0
Status COMPLETED
Requested CPUs 1
Requested CPU cores 0
Requested GPU 0
Requested memory 3.2GiB
Requested slots 0
Monitoring yes
Comment __model:other-model
Run directory /home/joe/war
Stdin file /dev/null
Stdout file /home/joe/war/matrix-3.out
Stderr file /home/joe/war/matrix-3.err
Total power usage < not set >
Total GPU power usage < not set >
Total CPU power usage < not set >
Total power under allocation < not set >
QOS normal
[basecm11->wlm[slurm]->jobs]%
```
**The power usage fields for WLM jobs in the preceding output:** `Total power usage`, `Total GPU power usage`, `Total CPU power usage`, and `Total power under allocation`, are populated based on metrics configured in the `wlmjobpowerusagesettings` submode in `partition` mode (described later on in this section).
For the associated metrics, power usage is calculated per job, for completed jobs by computing the energy consumption (the sum of power consumed over time) from the job start time until the job end time.
If no metrics are configured, then the power usage fields remain `< not set >`.
When the metrics are configured, the `info` output for a completed job shows its energy consumption:
**Example**
```bash
Total power usage 0 Wh
Total GPU power usage 556.649 Wh
Total CPU power usage 61.2204 Wh
Total power under allocation 0.0%
```
The change from `< not set >` to values only show up some time after the job has completed, to allow values to be properly gathered.
**The `wlmjobpowerusagesettings` submode under `partition` mode:** has power usage parameters that can be associated with metrics:
**Example**
```bash
[basecm11->partition[base]->wlmjobpowerusagesettings]% show
Job age 5m
Timeout 30s
Disabled no
Node power metrics
GPU power metrics
CPU power metrics
Power under allocation metrics
```
* `Job age`: Minimum age of a job before power usage is calculated.
* `Timeout`: Timeout for power data collection.
* `Disabled`: Disables job power usage calculation when set to `yes`.
* `Node power metrics`: Metrics used for total node power usage.
* `GPU power metrics`: Metrics used for GPU power usage.
* `CPU power metrics`: Metrics used for CPU power usage.
* `Power under allocation metrics`: Metrics used for power consumption under allocation.
Metrics can be configured to be part of the power metrics by adding or appending them:
**Example**
```bash
[basecm11->partition*[base*]->wlmjobpowerusagesettings*]% set nodepowermetrics
HPC > Workload Management Clusters
\> Job Queues > + Add
opens up a pop-up menu that allows queues to be added to the WLM cluster. The properties of the new queue can then be set in the dialog window of the queue (figure [7.33](#fig-base-view-workloadmanagement-queues-main))
At least the queue name, `shorq`, must be set for the queue. The configuration for the queue properties is saved, and then the job queue scheduler configuration is saved.
Next, a new category can be added using\
`Grouping > Categories > + Add`\
Parameters settings in the new category can be set—at least the category name should be set—to suit the new machines The name `shornodes` can therefore be set here and it becomes part of the list of categories (figure [7.35](#fig-base-view-workloadmanager-add-shornodes-category)). .
*Figure 7.35: A new shornodes category created using Base View*
Another option within the category is to set a workload manager queue. The queue, `shorq`, is therefore set here for this new category.
Setting a queue for the category means configuring the options of the queue scheduler role, for example the Slurm client role, for the category. So a Slurm client role must be assigned to the category. Continuing from within the `shornodes` category in figure [7.35](#fig-base-view-workloadmanager-add-shornodes-category), the relative navigation path to set a queue is:\
> Settings > Roles > + Add > SlurmClientRole > Queues > shorq\
The appropriate queue can be selected from the queue scheduler menu option (figure [7.36](#fig-base-view-workloadmanager-category-role-queue)):
*Figure 7.36: Setting a queue for a new category using Base View*
In this case, the `shorq` created earlier on is presented for selection. After selection, the category configuration is committed.
Nodes that are to use the queue should be members of the `shornodes` category. The final step is then to allocate such nodes to the category. This can be done, for example for a `node001`, by using the navigation path:
Devices > All devices > node001
\> Settings > Category
and setting the category to `shornodes`.
#### 7.8.1.2 A New Category And A New Queue With `cmsh`
The preceding example can also be configured in `cmsh` as follows:
The new queue can be added from within `jobqueue` mode, for the workload manager. For example, if Slurm is the WLM cluster that is enabled:
```bash
[basecm11]% wlm; use slurm; jobqueue; add shorq
[basecm11->wlm[slurm]->jobqueue*[shorq*]]% commit
```
The new category, `shornodes`, can be created by cloning an old category, such as `default`:
```bash
[basecm11->wlm[slurm]->jobqueue[shorq]]% category
[basecm11->category]% clone default shornodes
[basecm11->category*[shornodes*]]% commit
```
Then, going into the roles submode, appropriate workload manager roles can be assigned, and appropriate queues can be appended and committed for that category:
```bash
[basecm11->category[shornodes]]% roles
[basecm11->category[shornodes]->roles]% assign slurmclient; commit
[basecm11->category[shornodes]->roles[slurmclient]]% append queues shorq
[basecm11->category[shornodes*]->roles*]% commit
```
The nodes belonging to the `shornodes` category can then be placed by going into `device` mode to select the nodes to be placed in that category. For example:
```bash
[basecm11->category[shornodes]->roles]% device use node002
[basecm11->device[node002]]% set category shornodes
[basecm11->device*[node002*]]% commit
```
### 7.8.2 Setting Up A Prejob Or Postjob Check
#### 7.8.2.1 How It Works
Measurables such as health checks (section [10.2.4](/admin/monitoring#1024-health-check)) by default run as scheduled tasks at timed intervals ([here](/admin/monitoring#1041-monitoring-configuration-data-producers)). They can independently be configured to run as prejob or postjob checks.
If a health check is configured as a prejob or postjob check, then its response means the same as that of a health check, that is:
* If the response to a prejob or postjob health check is `PASS`, then it shows that the node is displaying healthy behavior for that particular health check.
* If the response to a prejob or postjob health check is `FAIL`, then it implies that the node is unhealthy at the time of the check, at least for that particular health check.
Sometimes the strain of running a job can itself cause a health issue on the node. If that is the case, then sometimes the issue can be spotted by an immediate check of the node health after a job has completed—which is what the postjob health check does. A postjob health check is otherwise the same as a regular health check.
* If a postjob health check response is `FAIL`, then the node is unhealthy, even though the job completed. What effect the unhealthy node had on the job may need consideration by the cluster administrator.
* If a prejob health check response is `FAIL`, then the node is also unhealthy, but the job is prevented by the WLM from running on that node.
A node that has failed a prejob health check is not allowed to run a job. This is because an unhealthy node means that a job submitted to the node may fail, may not be able to start, or may even vanish outright. The way in which a job can vanish in some cases, without any information beyond the job submission “event horizon” leads to this behavior sometimes being called the *Black Hole Node Syndrome*.
**Draining on a prejob or postjob health check failure:** It can be troublesome for a system administrator to pinpoint the reason for such job failures, since a node may only fail under certain conditions that are hard to reproduce later on. It is therefore a good policy to disallow passing a job to a node which has just been flagged as unhealthy by a health check. Thus, a sensible action (section [10.2.6](/admin/monitoring#1026-action)) taken by a prejob or postbjob health check on receiving a `FAIL` response would be to put the node in a `Drained` state (section [7.7.4](#774-nodes-drainage-status-and-handling-in-cmsh)). The `drain` state means that BCM arranges a rescheduling of the job so that the job runs only on nodes that are believed to be healthy.
A node that has been put in a `Drained` state with a health check is not automatically undrained. The administrator must clear such a state manually.
The `failedprejob` health check ([here](/admin/metricshealthchecksactions#g21-regular-health-checks)) is enabled by default, and logs any prejob health check passes and failures. By default there are no prejob health checks configured, so by default this health check should always pass.
#### 7.8.2.2 Configuration Using `cmsh`
Prejob and postjob health checks can be enabled for a WLM instance:
```bash
[basecm11->wlm[slurm]]% show | grep -i job
Enable pre job no
Enable post job no
[basecm11->wlm[slurm]]% set enableprejob yes; commit
```
Each health check can then be enabled individually within the `monitoring setup` mode.
**Example**
For example, for the `ib` health check:
```bash
[basecm11->monitoring->setup]% get ib prejob
no
[basecm11->monitoring->setup]% set ib prejob | **Parameter** | **Description** | **Configuration Parameter** |
| **In `cgroup.conf`** | ||
| `Auto Mount` \* | Force Slurm to mount cgroup subsystems if they are not mounted yet | `CgroupAutomount` |
| `Mount Point` | Where cgroup root is mounted | `CgroupMountpoint` |
| `Task Affinity` \* | Set a default task affinity to bind each step task to a subset of the allocated cores using `sched_setaffinity` | `TaskAffinity` |
| `Release Agent Dir` | Directory containing Slurm cgroup release_agent files | `CgroupReleaseAgentDir` |
| `Constrain Cores` \* | Constrain allowed cores to the subset of allocated resources | `ConstrainCores` |
| `Constrain RAM Space` \* | Constrain the job’s RAM usage | `ConstrainRAMSpace` |
| `Constrain Swap Space` \* | Constrain the job’s swap space usage | `ConstrainSwapSpace` |
| `Constrain Devices` \* | Constrain the job’s allowed devices (also constrain GPUs) based on GRES allocated resources | `ConstrainDevices` |
| `Allowed RAM Space` | Percentage memory (default is 100%) out of the allocated RAM allowed for the job cgroup RAM. If this percentage is exceeded, then the job steps will be killed and a warning message will be written to standard error. | `AllowedRAMSpace` |
| `Allowed Swap Space` | Percent allocated memory allowed for the job cgroup swap space | `AllowedSwapSpace` |
| `Max RAM Percent` | Maximum percent of total RAM for a job | `MaxRAMPercent` |
| `Max Swap Percent` | Maximum percent of total RAM for the amount of RAM+Swap that may be used for a job | `MaxSwapPercent` |
| `Min RAM Space` | Minimum MB for the memory limits defined by `Allowed RAM Space` and `Allowed Swap Space` | `MinRAMSpace` |
| **Parameter** | **Description** | **Configuration Parameter** |
| **In `lsf.conf`** | ||
| `Resource Enforce` | Controls resource enforcement through the Linux cgroups memory and cpuset subsystem, on Linux systems with cgroups support. The resource can be either `memory` or `cpu` , or both `cpu` and `memory` , in either order (default: `memory cpu` ) | `LSB_RESOURCE_ENFORCE` |
| `Process Tracking` \* | This parameter, when enabled, has LSF track processes based on job control functions such as termination, suspension, resume, and other signals, on Linux systems which support the cgroups freezer subsystem | `LSF_PROCESS_TRACKING` |
| `Linux Cgroup Accounting` \* | When enabled, processes are tracked based on CPU and memory accounting for Linux systems that support cgroup’s memory and cpuacct subsystems | `LSF_LINUX_CGROUP_ACCT` |