Cluster > Settings > BMC Settings >
This opens up a window so that the BMC settings can be managed:
* `User name`: (default: `bright`)
* `Password`: (default: random string generated during head node installation)
* `User ID`: (default: `4`)
* `Power reset delay`: During a reset, this is the time, in seconds, that the machine is off, before it starts powering up again (default: `0`)
* `Extra arguments`: (default: none)
* `privilege`: (default: `administrator`)
The defaults in the preceding are set when the BMC interfaces are configured during head node installation. If the BMC interfaces are not set then, then the defaults are also unset.
BMC configuration is discussed in more detail in section [3.7](#37-configuring-bmc-ipmiilodraccimcredfish-interfaces).
#### 3.1.3.2 Administrator E-mail Setting
By default, the distribution which BCM runs on sends e-mails for the administrator to the root e-mail address. The administrator e-mail address can be changed within BCM so that these e-mails are received elsewhere.
* In Base View, an e-mail address (or space-separated addresses) can be set in the `Administrator e-mail` field via the navigation path `Cluster` > `Settings` > `Administrator e-mail`
* In `cmsh`, the e-mail address (or space-separated addresses) can be set in `partition` mode, using the `base` object as follows:
**Example**
```bash
[basecm11]% partition use base
[basecm11->partition[base]]% set administratore-mail alf@example.com beth@example.com
[basecm11->partition*[base*]]% commit
```
The following critical states or errors cause e-mails to be sent to the e-mail address:
* By default, a month before the cluster license expires, a reminder e-mail is sent to the administrator account by CMDaemon. A daily reminder is sent if the expiry is due within a week.
* A service on the head node that fails on its first ever boot.
* When an automatic failover fails on the head or regular node.
#### 3.1.3.3 SMTP Relay Host Mailserver Setting
The head node uses Postfix as its SMTP server. The default base distribution configuration is a minimal Postfix installation, and so has no value set for the SMTP relay host. To set its value:
* in Base View: the `Relay Host` field, with the navigation path:
`Cluster > Settings > Search domains > Relay Host`
sets the SMTP relay host for the cluster resource.
* in `cmsh`: the `relayhost` property can be set for the `base` object within `partition` mode:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% partition use base
[basecm11-> partition[base]]% set relayhost mail.example.com
[basecm11-> partition[base*]]% commit
```
Postfix on the regular nodes is configured to use the head node as a relay host and is normally left untouched by the administrator.
If the regular node configuration for Postfix is changed due to a `partition` mode change, then a node reboot deploys the change for the node. Setting the `AdvancedConfig` ([here](/admin/cmdconf#c80-advancedconfig-directive)) `SmtpPartitionRelayHostInImages` to `0` disables the changing of the relay host on the regular node.
Further Postfix changes can be done directly to the configuration files as is done in the standard distribution. The changes must be done after the marked auto-generated sections, and should not conflict with the auto-generated sections.
A convenient way to check mail is functioning is to run BCM’s `testemail` command. The command is run from within the `main` mode of `cmsh`. It sends a test e-mail out using CMDaemon:
```bash
[root@basecm11 ~]# mailq; ls -al /var/spool/mail/root
Mail queue is empty
-rw------- 1 root mail 0 Sep 8 11:11 /var/spool/mail/root
[root@basecm11 ~]# cmsh -c "main; testemail"
Mail delivered to postfix
You have new mail in /var/spool/mail/root
[root@basecm11 ~]# mailq; ls -al /var/spool/mail/root
Mail queue is empty
-rw------- 1 root mail 749 Sep 8 11:12 /var/spool/mail/root
```
The test e-mail destination is the administrator e-mail address discussed in the preceding section.
#### 3.1.3.4 Failover Settings
To access options to the high availability (HA) feature of the cluster for head nodes, the administrator can click on the `Failover` option in Base View. This opens up a subwindow that can be used to configure HA-related parameters (section [15.4.6](/admin/ha#1546-high-availability-parameters)).
#### 3.1.3.5 Failover Groups Settings
To access options to the high availability feature of the cluster for groups of regular nodes, the administrator can click on the `Failover groups` option in Base View. This opens up a subwindow that can be used to configure failover-groups-related parameters (section [15.5](/admin/ha#155-ha-for-regular-nodes-and-edge-director-nodes)).
#### 3.1.3.6 Burn Configs
Burning nodes is covered in [Chapter 11 of the *Installation Manual*](/installation/burn). Burn configuration settings for the cluster can be accessed in Base View via the following navigation paths:
* `Cluster > Settings > Default burn configuration`
This allows the `Default burn configuration` for a node burn run to be modified.
* `Cluster > Settings > Burn configs`
This lists the possible burn configuration settings in a subwindow, and allows changes to some of their properties of each item of the list.
#### 3.1.3.7 FIPS Mode
To be compliant with the Federal Information Processing Standards of the USA, Linux-based systems are required to follow some security standards. This is known as FIPS compliance.
FIPS compliance on regular nodes can be set per node or per category.
The existing FIPS status can be checked with the `sysinfo` command at node level:
**Example**
```bash
[head->device]% sysinfo node002
Name Value
------------------------- ---------------------------------
...
FIPS No
```
Or even just with a `get`:
**Example**
```bash
[head->device]% get node002 fips
no
```
FIPS can be enabled per node:
**Example**
```bash
[head->device]% set node002 fips yes
[head->device*]% commit
[head->device]% reboot node002
[time passes]
[node002 finishes reboot]
[head->device]% sysinfo node002 | grep FIPS
FIPS Yes
```
FIPS can be enabled per category:
**Example**
```bash
[head->device]% sysinfo node002
[head->device]% category
[head->category]% set default fips yes
[head->category*]% commit
Successfully committed 1 Categories
[head->category]%
...[notice] head: node001 [ UP ], restart required (fips)
...[notice] head: node002 [ UP ], restart required (fips)
[head->category]% device reboot -n node001..node002
[head->category]% device
[head->device]% get node001 fips; get node002 fips
yes
yes
```
As is usual in BCM, the node-level setting overwrites the category-level setting.
Setting FIPS compliance on the head node itself via CMDaemon is not allowed. This is because FIPS GRUB configuration could, in some unusual cases, result in an unbootable head node. The cluster administrator must consider the existing state of the head node with due care before manually re-configuring it for FIPS.
### 3.1.4 Limiting The Maximum Number Of Open Files
#### 3.1.4.1 Configuring The System Limit On Open Files: The `/proc/sys/fs/file-max` Setting
The maximum number of open files allowed on a running Linux operating system is determined by `/proc/sys/fs/file-max`. To configure this setting so that it is persistent, the Linux operating system uses a `/etc/sysctl.conf` file and `*.conf` files under `/etc/sysctl.d/`. Further information on these files can be found via the man page, `man sysctl.conf.5`. BCM adheres to this standard method, and places a settings file `90-cm-sysctl.conf` in the directory `/etc/sysctl.d`.
By default, the value set for `file-max` by NVIDIA Base Command Manager is 131072. An on-premises head node typically is not used to run applications that exceed this value, and the settings file `90-cm-sysctl.conf` is therefore not available for modification by the CMDaemon front ends (`cmsh` and Base View). For edge and cloud director installations, the installation scripts take care of modifying `file-max` since such installations can have many more open files. Cluster administrators are thus not expected to have a need to modify the value of `file-max` in most use cases.
However, if there is a need to modify the value of `file-max`, then a subsequent, extra, configuration file, such as `/etc/sysctl.d/91-site-sysctl-file-max-tweak.conf` can be added under `/etc/sysctl.d/`, so that the BCM version is not intefered with directly.
#### 3.1.4.2 Configuring The User Limit On Open Files: The `nofile` Setting
The maximum number of open files allowed for a user can be seen on running `ulimit -n`. The value is defined by the `nofile` parameter of ulimit.
By default the value set by BCM is 131072.
Ulimit limits are limits to restrict the resources used by users. If the `pam_limits.so` module is used to apply ulimit limits, then the resource limits can be set via the `/etc/security/limits.conf` file and `*.conf` files in the `/etc/security/limits.d` directory. Further information on these files can be found via the man page, `man limits.conf.5`.
Resource limits that can be set for user login sessions include the number of simultaneous login sessions, the number of open files, and memory limits.
The maximum number of open files for a user is unlimited by default for an operating system that is not managed by BCM. However, it is set to 131072 by default for a system managed by BCM. The `nofile` value is defined by BCM in:
* in `/etc/security/limits.d/91-cm-limits.conf` on the head node
* in `/cm/images/` \<*software image name*>`/etc/security/limits.d/91-cm-limits.conf` in the software image that the regular node picks up.
The values set in `91-cm-limits.conf` are typically sufficient for a user session, unless the user runs applications that are resource hungry and consume a lot of open files.
#### 3.1.4.3 Deciding On Appropriate Ulimit, Limit, And System Limit Values
Decreasing the `nofile` value in `/etc/security/limits.d/91-cm-limits.conf` (but leaving the `/proc/sys/fs/file-max` untouched), or increasing `/proc/sys/fs/file-max` (but leaving the `nofile` value of 131072 per session as is), may help the system stay under the maximum number of open files allowed.
In general, users should not be allowed to use the head node as a compilation server, or as a test bed, before running their applications. This is because user errors can unintentionally cause the head node to run out of resources and crash it.
Depending what is running on the server, and the load on it, the administrator may wish to increase the resource limit values.
A very rough rule-of-thumb that may be useful as a first approximation to set `file-max` optimally is suggested in the kernel source code. The suggestion is to simply multiply the system memory (in MB) by 10 per MB, and make the resulting number the `file-max` value. For example, if the node has 128 GB of memory, then 1280000 can be set as the `file-max` value.
Fine-tuning to try and ensure that the operating system no longer runs out of file handles, and to try and ensure the memory limits for handling the load are not exceeded, is best achieved via an empirical trial-and-error approach.
## 3.2 Network Settings
A simplified quickstart guide to setting up the external head node network configuration on a vendor-prepared cluster is given in [Chapter 6 of the *Installation Manual*](/installation/ipaddresschange). This section ([3.2](#32-network-settings)) covers network configuration more thoroughly.
After the cluster is set up with the correct license as explained in [Chapter 4 of the *Installation Manual*](/installation/licensing), the next configuration step is to define the networks that are present (sections [3.2.1](#321-configuring-networks) and [3.2.2](#322-adding-networks)).
During BCM installation at least three default network objects were created:
`internalnet`: the primary internal cluster network, and the default management network. This is used for booting non-head nodes and for all cluster management communication. In the absence of other internal networks, `internalnet` is also used for storage and communication between compute jobs. Changing the configuration of this network is described [here](#3234-changing-internal-network-parameters-for-the-cluster) under the subheading “Changing Internal Network Parameters For The Cluster”.
`externalnet`: the network connecting the cluster to the outside world (typically a corporate or campus network). Changing the configuration of this network is described [here](#3233-changing-external-network-parameters-for-the-cluster) under the subheading “Changing External Network Parameters For The Cluster”. This is the only network for which BCM also supports IPv6.
`globalnet`: the special network used to set the domain name for nodes so that they can be resolved whether they are cloud nodes or not. This network is described further [here](#3235-changing-the-global-network-parameters-for-the-cluster) under the subheading “Changing The Global Network Parameters For The Cluster”.
For a Type 1 cluster ([section 3.3.9 of the *Installation Manual*](/installation/installing#339-network-topology)) the internal and external networks are illustrated conceptually by figure [3.2](#fig-networksettingsconcepts).
*Figure 3.2: Network Settings Concepts*
The configuration of network settings is completed when, after having configured the general network settings, specific IP addresses are then also assigned to the interfaces of devices connected to the networks.
* Changing the configuration of the head node external interface is described [here](#theipaddressofthecluster) under the subheading “The IP address of the cluster”.
* Changing the configuration of the internal network interfaces is described [here](#theipaddressesandotherinterfacevaluesoftheinternalnetwork) under the subheading “The IP addresses and other interface values of the internal network”.
* How to set a persistent identity for an interface—for example, to ensure that a particular interface that has been assigned the name `eth3` by the kernel keeps this name across reboots—is covered in section [5.8.1](/admin/provisioning#581-node-fails-to-pxe-boot), [here](/admin/provisioning#5811-interface-naming-conventions-post-rhel7-recommended).
* Changing the configuration of `globalnet` is described [here](#3235-changing-the-global-network-parameters-for-the-cluster) under the subheading “Changing The Global Network Parameters For The Cluster”. IP addresses are not configured at the `globalnet` network level itself.
### 3.2.1 Configuring Networks
The `network` mode in `cmsh` gives access to all network-related operations using the standard object commands. Section [2.5.3](/admin/clustermanagement#253-working-with-objects) introduces `cmsh` modes and working with objects.
In Base View, a network can be configured via the navigation path `Networking` > `Networks`, which opens up the `Network list` subwindow (figure [3.3](#fig-base-view-networks)):
*Figure 3.3: Networks*
In the context of the OSI Reference Model, each network object represents a layer 3 (i.e. Network Layer) IP network, and several layer 3 networks can be layered on a single layer 2 network (e.g. routes on an Ethernet segment).
Opening up the subwindow for a network such as `internalnet` or `externalnet` brings up the `Devices Overview` tab window, which lists the devices on the network and associated properties. For example, the `internal` network typically has some nodes and other devices attached to it, each with their IP addresses and statuses (figure [3.4](#fig-base-view-network-overview)).
*Figure 3.4: Network Overview*
Selecting the `Settings` tab window opens a scrollable pane that allows a number of network properties to be changed (figure [3.5](#fig-base-view-network-settings)).
*Figure 3.5: Network Settings*
The properties of figure [3.5](#fig-base-view-network-settings) are introduced in table [3.1](#tablenetwork-configuration-settings):
| **Property** | **Description** |
|---|---|
| `Name` | Name of this network. |
| `Domain Name` | DNS domain associated with the network. |
| `Type` | Menu options to set the network type. Options are `Internal` , `External` , `Tunnel` , `Global` , `Cloud` , or `NetMap` network. |
| `MTU` | Maximum Transmission Unit. The maximum size of an IP packet transmitted without fragmenting. |
| `Allow autosign` | Set if node-installer certificate sign requests are signed automatically. |
| `Write DNS zone` | Set which DNS zones should be written (forward or reverse). |
| `Node booting` |
Enabling means nodes are set to boot from this network (useful in the case of nodes on multiple networks). For an internal subnet called
`<`
*subnet*
`>`
, when node booting is set, CMDaemon adds a subnet configuration
`/etc/dhcpd.<`
*subnet*
`>.conf`
on the head node, which is accessed from
`/etc/dhcpd.conf`
.
\
• It can be set in Base View using the navigation path
Networking > Networks > internalnet > > Settings > Node booting
.
\
• It can be set in cmsh using the network mode, selecting the network for the node, and then setting nodebooting.
|
| `Lock down dhcpd` | Enabling means new nodes are not offered a PXE DHCP IP address from this network, i.e. DHCPD is “locked down”. A DHCP “deny unknown-clients” option is set by this action, so no new DHCP leases are granted to unknown clients for the network. Unknown clients are nodes for which BCM has no MAC addresses associated with the node. \ • It can be set in Base View via the Networking resource, selecting a network, and then setting Lock down dhcpd from within the network settings. \ • It can be set in cmsh via the network mode, selecting a network, and then setting lockdowndhcpd. |
| `IMEX` | Enabling means the network is used for the NVIDIA IMEX peer list. The fully qualified domain name (FQDN) of each node on this network is then written to `/etc/nvidia-imex/nodes_config.cfg` instead of the short hostname (section [7.5.1.4](/admin/workloadmanagement#7514-imex) for the resulting configuration file and further IMEX details). It can be set in cmsh using the network mode, selecting the network, and then setting imex. |
| `IMEX IP` | Enabling means the IMEX peer list in `nodes_config.cfg` is written using the IP address of each node on this network rather than the FQDN. Requires `IMEX` to also be enabled on the network (section [7.5.1.4](/admin/workloadmanagement#7514-imex) ). It can be set in cmsh using the network mode, selecting the network, and then setting imex ip. |
| `Management allowed` | Enabling means that the network has nodes managed by the head node. |
| `Search domain index` | The position of the network domain in the `resolv.conf` file. A value of `0` means the position is determined automatically. |
| `Exclude from search domain` | Enabling means the domain name for the network is not used. |
| `Disable automatic exports` | Enabling means that exports of built-in filesystems are not done automatically for the network. |
| `Base address` | Base address of the network (also known as the *network address* ). |
| `Dynamic range start/end` | Start/end IP addresses of the DHCP range temporarily used by nodes during PXE boot on the internal network. These are addresses that do not conflict with the addresses assigned and used by nodes during normal use. |
| `Netmask bits` | Prefix-length, or number of bits in netmask. The part after the “/” in CIDR notation. |
| `Gateway` | Default route IP address |
| `Gateway metric` | Gateway metric |
| `IPv6 enabled` | Enables IPv6 |
| `IPv6 gateway` | Sets IPv6 gateway |
| `IPv6 base address` | Sets IPv6 base address |
| `IPv6 netmask bits` | Sets IPv6 netmask bits |
| `Cloud Subnet ID` | Cloud ID of subnet |
| `EC2AvailabilityZone` | AWS availability zone in which the subnet exists |
| `Layer3` | If set to `yes` it creates layer 3 connections with /31 netmasks. The following parameters can then be set: \ • Layer3 route: Sets a layer 3 route \ • Layer3 ecmp: Creates a layer 3 network with equal-cost multi-path \ • Layer3 split static route: Create a layer 3 network with split static route |
| `Notes` | Notes can be added for the network |
Devices > All Devices > Head Node > > Settings > Hostname > Commit HeadNode. That is, under the `Devices` resource, the `All Devices` option is selected. The head node is selected from the devices displayed, and its subwindow-opening box in the `Actions` column is then clicked. This opens the device overview pane. The `Settings` option can then be selected. The `Hostname` record can then be modified (figure [3.6](#fig-base-view-headnodesettings)), and saved by clicking on the `Commit HeadNode` icon. When setting a hostname, a domain is not included.
After the change, as suggested by Base View, the head node must be rebooted.
*Figure 3.6: Head Node Settings*
* In `cmsh`, the hostname of the head node can also be changed, via device mode:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11
[basecm11->device[basecm11]]% set hostname foobar
[foobar->device*[foobar*]]% commit
[foobar->device[foobar]]%
Tue Jan 22 17:35:29 2013 [warning] foobar: Reboot required: Hostname changed
[foobar->device[foobar]]% quit
[root@basecm11 ~]# sleep 30; hostname -f foobar.cm.cluster
[root@basecm11 ~]#
```
The prompt string shows the new hostname after a short time, when a new shell is started.
After the hostname has been set, as suggested by `cmsh`, the head node must be rebooted.
#### 3.2.3.2 Adding Hostnames To The Internal Network
Additional hostnames, whether they belong to the cluster nodes or not, can be added as name/value pairs to the `/etc/hosts` file(s) within the cluster. This should be done only outside the specially-marked CMDaemon-managed section. It can be done to the file on the head node, or to the file on the software image for the regular nodes, or to both, depending on where the resolution is required.
However, for hosts that are on the internal network of the cluster, such as regular nodes, it is easier and wiser to avoid adding additional hostnames via `/etc/hosts`.
Instead, it is recommended to let BCM manage host name resolution for devices on the `internalnet` through its DNS server on the `internalnet` interface. The host names can be added to the `additionalhostnames` object, from within `interfaces` submode for the head node. The `interfaces` submode is accessible from the `device` mode. Thus, for the head node, with `eth1` as the interface for `internalnet`:
**Example**
```bash
[basecm11]% device use basecm11
[basecm11->device[basecm11]]% interfaces
[basecm11->device[basecm11]->interfaces]% use eth1
[basecm11->device[basecm11]->interfaces[eth1]]% set additionalhostnames test
[basecm11->device*[basecm11*]->interfaces*[eth1*]]% commit
[basecm11->device[basecm11]->interfaces[eth1]]%
Fri Oct 12 16:48:47 2012 [notice] basecm11: Service named was restarted
[basecm11->device[basecm11]->interfaces[eth1]]% !ping test
PING test.cm.cluster (10.141.255.254) 56(84) bytes of data.
...
```
Multiple hostnames can be added as space-separated entries.
The `named` service automatically restarts within about 20 seconds after committal, implementing the configuration changes. This kind of restart is a feature (section [3.15.1](#3151-why-use-the-cluster-manager-for-services)) of changes made to service configurations by Base View or `cmsh`.
#### 3.2.3.3 Changing External Network Parameters For The Cluster
**The external network parameters of the cluster:** When a cluster interacts with an external network, such as a company or a university network, its connection behavior is determined by the settings of two objects: firstly, the external network settings of the `Networks` resource, and secondly, by the cluster network settings.
1. **The external network object** contains the network settings for all objects configured to connect to the external network, for example, a head node. Network settings are configured in Base View under the `Networking` resource, then under the `Networks` subwindow, then within the `Settings` option for a selected network. Figure [3.5](#fig-base-view-network-settings) shows a settings window for when the `internalnet` item has been selected, but in the current case the `externalnet` item must be selected instead. The following parameters can then be configured:
* the IP network parameters of the cluster (but not the IP address of the cluster):
* `Base address`: the network address of the external network (the “IP address of the external network”). This is not to be confused with the IP address of the cluster, which is described shortly after this.
* `Broadcast address`: the broadcast address of the external network. This should not be confused with the broadcast IP address of the internal network.
* `Dynamic range start` and `Dynamic range end`: Not used by the external network configuration.
* `Netmask bits`: the netmask size, or prefix-length, of the external network, in bits.
* `Gateway`: the default route for the external network.
* the `network name` (what the external network itself is called), by default this is defined as `externalnet` in the base partition on a newly installed Type 1 cluster,
* the `Domain Name`: the network domain (LAN domain, i.e. what domain machines on the external network use as their domain),
* the `External network` checkbox: this is checked for a Type 1 cluster,
* and MTU size (the maximum value for a TCP/IP packet before it fragments on the external network—the default value is 1500).
2. **The cluster object** contains other network settings used to connect to the outside. These are configured in Base View using the navigation path:
`Cluster > Settings`
and are described in section [3.1.1](#311-cluster-configuration-various-cluster-wide-settings):
* e-mail address(es) for the cluster administrator,
* any additional external name servers used by the cluster to resolve external host names. As an aside: by default, only the head node name server is configured, and by default it only serves hosts on the internal network via the internal interface. Enabling the `PublicDNS` directive (Appendix [C.54](/admin/cmdconf#c54-publicdns-directive)) allows the head node name server to resolve queries about internal hosts from external hosts via any interface, including the external interface.
* the DNS search domain (what the cluster uses as its domain),
* and NTP time servers (used to synchronize the time on the cluster with standard time) and time zone settings.
These settings can also be adjusted in `cmsh` in the `base` object under `partition` mode.
Changing the networking parameters of a cluster (apart from the IP address of the cluster) therefore requires making changes in the settings of the two preceding objects.
**Adding custom DNS zones:** BCM appends a BIND DNS `include` directive to the bottom of the head node name server’s generated configuration file, so that administrator-defined DNS zones are preserved across regeneration. The path of both files depends on the distribution:
* on Ubuntu:
* configuration file `/etc/bind/named.conf`
* include file `/etc/bind/named.conf.include`
* on RHEL 9/10 and SLES 15:
* configuration file `/etc/named.conf`
* include file `/etc/named.conf.include`
Custom zone definitions placed in the include file are not touched by BCM. The zone file itself can be placed in any directory the administrator chooses; it is not tied to the distribution-specific `named.conf`/include file paths in the preceding. For example:
**Example**
```bash
zone "runai.superpod-nvidia.local" {
type master;
file "
Networking > Networks > externalnet >
\> Settings
which reaches the window shown in figure [3.7](#fig-base-view-network-settings-externalnet):
*Figure 3.7: Network Settings For External Network*
**The cluster object:**
The cluster object and associated settings are accessed using the navigation path:
`Cluster > Settings`
**The head node IP address:** For a head node `basecm11`, where the IP address is used by the interface `eth0`, the static IP address can be set via the navigation path:
Devices > All Devices > Head Node >
\> Settings > Interfaces >
\> eth0 >
\> IP
as seen in figure [3.8](#fig-base-view-headnode-networksetup-editbox).
*Figure 3.8: Setting The IP Address On A Head Node In Base View*
* ##### With `cmsh`:
The preceding Base View configuration can also be done in `cmsh`, using the `network`, `partition` and `device` modes, as in the following example:
**Example**
```bash
[basecm11]% network use externalnet
[basecm11->network[externalnet]]% set baseaddress 192.168.1.0
[basecm11->network*[externalnet*]]% set netmaskbits 24
[basecm11->network*[externalnet*]]% set gateway 192.168.1.1
[basecm11->network*[externalnet*]]% set gatewaymetric 1
[basecm11->network*[externalnet*]]% commit
[basecm11->network[externalnet]]% partition use base
[basecm11->partition[base]]% set nameservers 192.168.1.1
[basecm11->partition*[base*]]% set searchdomains x.com y.com
[basecm11->partition*[base*]]% append timeservers ntp.x.com
[basecm11->partition*[base*]]% commit
[basecm11->partition[base]]% device use basecm11
[basecm11->device[basecm11]]% interfaces
[basecm11->device[basecm11]->interfaces]% use eth1
[basecm11->device[basecm11]->interfaces[eth1]]% set ip 192.168.1.176
[basecm11->device[basecm11]->interfaces*[eth1*]]% commit
[basecm11->device[basecm11]->interfaces[eth1]]%
```
After changing the external network configurations, a reboot of the head node is necessary to activate the changes.
**Using DHCP to supply network values for the external interface:**
Connecting the cluster via DHCP on the external network is not generally recommended for production clusters. This is because DHCP-related issues can complicate networking troubleshooting compared with using static assignments.
For a Type 1 network, the cluster and head node can be made to use some of the DHCP-supplied external network values as follows:
* In Base View, the DHCP setting of figure [3.8](#fig-base-view-headnode-networksetup-editbox) can be set to `Enabled`
* Alternatively, in `cmsh`, within interfaces mode for the head node interface, the value of the parameter DHCP can be set:
```bash
[basecm11->device[basecm11]->interfaces[eth0]]% set dhcp yes
```
The gateway address, the name server(s), and the external IP address of the head node are then obtained via a DHCP lease. Time server configuration for `externalnet` is not picked up from the DHCP server, having been set during installation ([figure 3.8 in Chapter 3 of the *Installation Manual*](/installation/installing#fig-clustersettings)). The time servers can be changed using Base View using the navigation path:
`Cluster > Settings > Time servers`
(figure [3.1](#fig-base-view-clustersettings)), or using `cmsh` in `partition` mode as in the preceding example. The time zone can be changed similarly.
It is usually sensible to reboot after implementing these changes in order to test the changes are working as expected.
#### 3.2.3.4 Changing Internal Network Parameters For The Cluster
When a cluster interacts with the internal network that the regular nodes and other devices are on, its connection behavior with the devices on that network is determined by settings in:
1. the internal network of the `Networks` resource ([here](#internalnetworkobjectnetworksettings))
2. the cluster network for the internal network ([here](#internalnetworkparametersclusterobject))
3. the individual device network interface ([here](#theipaddressesandotherinterfacevaluesoftheinternalnetwork))
4. the node categories network-related items for the device ([here](#internalnetworknetworkingcategoryvalues)), in the case of the device being a regular node.
In more detail:
**1. The internal network object:** has the network settings for all devices connecting to the internal network, for example, a login node, a head node via its `internalnet` interface, or a managed switch on the internal network. In Base View, the settings can be configured under the `Networking` resource, then under the `Networks` subwindow, then within the `Settings` option for the `internalnet` item (figure [3.5](#fig-base-view-network-settings)). In `cmsh` the settings can be configured by changing the values in the `internalnet` object within `cmsh`’s `network` mode. Parameters that can be changed include:
* the IP network parameters of the internal network (but not the internal IP address):
* the “`domain name`” of the network. This is the LAN domain, which is the domain machines on this network use as their domain. By default, set to a name based on the network interface type used on the internal network, for example `eth.cluster`. In any case, the FQDN must be unique for every node in the cluster.
* the network name, or what the internal network is called. By default, set to `internalnet`.
* The `MTU` size, or the maximum value for a TCP/IP packet before it fragments on this network. By default, set to 1500.
**2. The cluster object:** has other network settings that the internal network in the cluster uses. These particulars are configured in the `Settings` option of the cluster object resource in Base View (figure [3.1](#fig-base-view-clustersettings)). The `cmsh` equivalents can be configured from the `base` object in `partition` mode. Values that can be set include:
* the “`Management network`”. This is the network over which CMDaemon manages the nodes. Management means that CMDaemon on the head node communicates with CMDaemons on the other nodes. The communication includes CMDaemon calls and monitoring data. By default, the management network is set to `internalnet` for Type 1 and Type 2 networks, and `managementnet` in Type 3 networks. It is a partition-level cluster-wide setting by default. Partition-level settings can be overridden by the category level setting, and node-level settings can override category level or partition level settings.
* the “`Node name`” can be set to decide the prefix part of the node name. By default, set to `node`.
* the “`Node digits`” can be set to decide the possible size of numbers used for suffix part of the node name. By default, set to `3`.
* the “`Default category`”. This sets the category the nodes are in by default. By default, it is set to `default`.
* the “`Default software image`”. This sets the image the nodes use by default, for new nodes that are not in a category yet. By default, it is set to `default-image`.
* the “`Name server`”. This sets the name server used by the cluster. By default, it is set to the head node. The default configuration uses the internal network interface and serves only internal hosts. Internal hosts can override using this value at category level ([here](#3234-changing-internal-network-parameters-for-the-cluster)). By default external hosts cannot use this service. To allow external hosts to use the service for resolving internal hosts, the `PublicDNS` directive (Appendix [C.54](/admin/cmdconf#c54-publicdns-directive)) must be set to `True`.
**3. The internal IP addresses and other internal interface values:** In Base View, the network for a node, such as a physical node `node001`, can be configured via a navigation path
Devices > All Devices > Physical Node >
\> Settings > Interfaces > BOOTIF >
The subwindow that opens up then allows network configuration to be done for such nodes in a similar way to figure [3.8](#fig-base-view-headnode-networksetup-editbox). In `cmsh` the configuration can be done by changing properties from the `interfaces` submode that is within the `device` mode for the node.
The items that can be set include:
* the `Network device name`: By default, this is set to `BOOTIF` for a node that boots from the same interface as the one from which it is provisioned.
* the `Network`: By default, this is set to a value of `internalnet`.
* the `IP` address: By default, this is automatically assigned a static value, in the range `10.141.0.1` to `10.141.255.255`, with the first node being given the first address. Using a static IP address is recommended for reliability, since it allows nodes that lose connectivity to the head node to continue operating. The static address can be changed manually, in case there is an IP address or node ID conflict due to an earlier inappropriate manual intervention.
Administrators who want DHCP addressing on the internal network, despite the consequences, can set it via a checkbox.
* `onnetworkpriority`: This sets the priority of DNS resolution queries for the interface on the network. The range of values that it can take is from 0 to 4294967295. Resolution takes place via the interface with the higher value.
The default priority value for a network interface is set according to its type. These defaults are:
| **Type** | **Value** |
| :------- | :-------- |
| Bridge | 80 |
| Bond | 70 |
| Physical | 60 |
| VLAN | 50 |
| Alias | 40 |
| Tunnel | 30 |
| Netmap | 20 |
| BMC | 10 |
* `Additional Hostname`: In the case of nodes this is in addition to the default node name set during provisioning. The node name set during provisioning takes a default form of `node` \<*`3 digit number`*>, as explained earlier [here](#internalnetworkparametersclusterobject) in the section describing the cluster object settings.
For, example, a regular node that has an extra interface, `eth1`, can have its values set as follows:
**Example**
```bash
[basecm11] device interfaces node001
[basecm11->device[node001]->interfaces]% add physical eth1
[basecm11->...->interfaces*[eth1*]]% set network externalnet
[basecm11->...->interfaces*[eth1*]]% set additionalhostnames extra01
[basecm11->...->interfaces*[eth1*]]% set ip 10.141.1.1
[basecm11->...->interfaces*[eth1*]]% commit
[basecm11->...->interfaces[eth1]]% ..
[basecm11->...->interfaces]% ..
[basecm11->device[node001]]% reboot
```
**4. Node category network values:** are settings for the internal network that can be configured for node categories. For example, for the default category called `default` this can be carried out:
* via Base View, using the `Settings` option for the category. This is in the navigation path Grouping > Categories > > Settings
* or via the `category` mode in `cmsh`, for the `default` node category.
If there are individual node settings that have been configured in Base View or `cmsh`, then the node settings override the corresponding category settings for those particular nodes.
The category properties involved in internal networking that can be set include:
* `Default gateway`: The default gateway route for nodes in the node category. If unset, or 0.0.0.0 (the default), then the node default gateway route is decided by the internal network object `Gateway` value. If the default gateway is set as a node category value, then nodes use the node category value as their default gateway route instead.
* `Management network`: The management network is the network used by CMDaemon to manage devices. The default setting is a property of the node object. It can be set as a category property.
* `Name server`, `Time server`, `Search domain`: The default setting for these on all nodes is set by the node-installer to refer to the head node, and is not configurable at the node level using Base View or `cmsh`. The setting can however be set as a category property, either as a space-separated list of entries or as a single entry, to override the default value.
**Application to generic network devices:** The preceding details for the internal network parameters of the cluster, starting from [here](#3234-changing-internal-network-parameters-for-the-cluster), are applicable to regular nodes, but they are often also applicable to generic network devices (section [2.1.1](/admin/clustermanagement#211-devices)). Benefits of adding a generic device to be managed by BCM include that:
* the name given to the device during addition is automatically placed in the internal DNS zone, so that the device is accessible by name
* the device status is automatically monitored via an ICMP ping (Appendix [G.2.1](/admin/metricshealthchecksactions#g21-regular-health-checks)).
* the device can be made to work with the health check and metric framework. The scripts used in the framework will however almost certainly have to be customized to work with the device
After changing network configurations, a reboot of the device is necessary to activate the changes.
#### 3.2.3.5 Changing The Global Network Parameters For The Cluster
The global network `globalnet` is a unique network used to set up a common name space for all nodes in a cluster in BCM. It is required in BCM because of the added cloud extension functionality, described in the *Cloudbursting Manual*. Regular nodes and regular cloud nodes are thus both named under the `globalnet` domain name, which is `cm.cluster` by default. So, for example, if default host names for regular nodes (node001, node002, ...) and regular cloud nodes (cnode001, cnode002, ...) are used, node addresses with the domain are:
* node001.cm.cluster for node001
* cnode001.cm.cluster for cnode001
The only parameter that can be sensibly changed on `globalnet` is the domain name, which is `cm.cluster` by default.
Removing `globalnet` should not be done because it will cause various networking failures, even for clusters deploying no cloud nodes.
Details on how resolution is carried out with the help of `globalnet` are given in [section 6.6.1 of the *Cloudbursting Manual*](/cloudbursting/cloudconsiderations#661-resolution-and-globalnet).
#### 3.2.3.6 Setting Static Routes In The `staticroutes` Submode Of `cmsh`
To route via a specific gateway, the `staticroutes` submode can be used. This can be set for regular nodes and head nodes via the `device` mode, and for node categories via the `category` mode.
On a newly-installed cluster with a type 1 network ([section 3.3.9 of the *Installation Manual*](/installation/installing#339-network-topology)), a node by default routes packets using the head node as the default gateway.
If the administrator would like to configure a regular node to use another gateway to reach a printer on another subnet, as illustrated by figure [3.9](#fig-staticroute-to-a-printer):
*Figure 3.9: Example Of A Static Route To A Printer On Another Subnet*
then an example session for setting the static route is as follows:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11 ]% device use node001
[basecm11->device[node001]]%
[basecm11->device[node001]]% staticroutes
[basecm11->device[node001]->staticroutes]% list
Name (key) Gateway Destination
-------------------------------- ---------------- --------------
[basecm11->device[node001]->staticroutes]% add printerroute
[basecm11->...*[printerroute*]]% set gateway 10.141.1.1
[basecm11->...*[printerroute*]]% set destination 192.168.0.3
[basecm11->...*[printerroute*]]% set networkdevicename bootif
[basecm11->...*[printerroute*]]% commit
[basecm11->...[printerroute]]% show
Parameter Value
-------------------------------- -------------------------------
Destination 192.168.0.3/32
Gateway 10.141.1.1
Name printerroute
Network Device Name bootif
Revision
[basecm11->device[node001]->staticroutes[printerroute]]% exit
[basecm11->device[node001]->staticroutes]% list
Name (key) Gateway Destination
-------------------------------- ---------------- --------------
printerroute 10.141.1.1 192.168.0.3/32
[basecm11->device[node001]->staticroutes]% exit; exit
[basecm11->device]% reboot node001
```
In the preceding example, the regular node `node001`, with IP address 10.141.0.1 can connect to a host 192.168.0.3 outside the regular node subnet using a gateway with the IP addresses 10.141.1.1 and 192.168.0.1. The route is given the arbitrary name `printerroute` for administrator and CMDaemon convenience, because the host to be reached in this case is a print server. The `networkdevicename` is given the interface name `bootif`. If another device interface name is defined in CMDaemon, then that can be used instead. If there is only one interface, then `networkdevicename` need not be set.
After a reboot of the node, the route that packets from the node follow can be seen with a traceroute or similar. The `ping` utility with the `-R` option can often track up to 9 hops, and can be used to track the route:
**Example**
```bash
[root@basecm11 ~]# ssh node001 ping -c1 -R 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(124) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=62 time=1.41 ms
RR: 10.141.0.1
10.141.1.1
192.168.0.1
192.168.0.3
192.168.0.3
192.168.0.1
10.141.1.1
10.141.0.1
--- 192.168.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.416/1.416/1.416/0.000 ms
[root@basecm11->device[node001]->staticroutes]%
```
The routing is achieved by CMDaemon making sure that whenever the network interface is brought up, the OS of the regular node runs a routing command to configure the routing tables. The command executed for the given example is either:
```bash
route add -host 192.168.0.3 gw 10.141.1.1
```
or its modern iproute2 equivalent:
```bash
ip route add 192.168.0.3 via 10.141.1.1
```
If the device `bootif` is `eth0`—which is often the case—then the command is run from the network interface configuration file: `/etc/sysconfig/network-scripts/ifcfg-eth0` (or `/etc/sysconfig/network/ifcfg-eth0` in SUSE).
**If the device is a head node:** then rebooting it to have the new route become active is a clean way to implement the change. However doing this may not be acceptable to cluster users due to the downtime, and the associated interruption of jobs. A quicker update to the interface that should work is using operating system networking scripts. For example, if the interface with the route is `eth0`, then the interface can be brought down and up again with `ifdown eth0 && ifup eth0`. If the various networking configurations have been applied in a way that persists over reboots, then they should be applied automatically to the interface as it starts up.
#### 3.2.3.7 Port Forwarding With `portforward`
**Why port forwarding is useful:** In a typical type 1 network setup ([section 3.3.9 of the *Installation Manual*](/installation/installing#339-network-topology)), only the head nodes are accessible from the external network. So, the compute nodes cannot be accessed directly from the external network.
Accessing a compute node port may however be needed for some workflows. For example, a user on a laptop on the external network may wish to access a compute node port directly.
To allow this direct access to a compute node port, port forwarding rules can be set up from within the `device` mode of `cmsh`. Configuring port forwarding with BCM is designed to be easier than doing port forwarding directly in the operating system.
**The port forwarding help text:** The help text for the `portforward` command summarizes how to use it:
```bash
[basecm11->device]% help portforward
Name:
portforward - Forward a port on the active head node to IP
Usage:
portforward create Devices > All Devices > node001 > Settings > Interfaces > + Add > vlan\
(figure [3.10](#fig-base-view-configuring-vlan)).
*Figure 3.10: Configuring a VLAN via Base View*
A `Network VLAN Interface` window opens up, allowing an IP address, network, and other options to be configured for the VLAN interface.
## 3.5 Configuring Bonded Interfaces
### 3.5.1 Adding A Bonded Interface
The Linux bonding driver allows multiple physical network interfaces that have been configured previously (for example, as [here](#3234-changing-internal-network-parameters-for-the-cluster)) to be bonded as a single logical bond interface. The behavior of such interfaces is determined by their bonding mode. The modes provide facilities such as hot standby or load balancing.
The driver is included by default on head nodes. To configure a non-head node to use a bonded interface, a Linux kernel module called the `bonding` module must be included in the kernel modules list of the software image of the node. A bonding interface can then be created, and its properties set as follows:
**Example**
```bash
[basecm11->device[node001]->interfaces]% add bond bond0
[...->device*[node001*]->interfaces*[bond0*]]% set network internalnet
[...->device*[node001*]->interfaces*[bond0*]]% set ip 10.141.128.1
[...->device*[node001*]->interfaces*[bond0*]]% set mode 3
[...->device*[node001*]->interfaces*[bond0*]]% set interfaces eth1 eth2
```
Each bonded interface has a unique set of object properties, called bonding options, that specify how the bonding device operates. The bonding options are a string containing one or more options formatted as `option=value` pairs, with pairs separated from each other by a space character.
A bonded interface also always has a mode associated with it. By default it is set to `0`, corresponding to a balanced round-robin packet transmission.
The 7 bonding modes are:
* 0 – balance-rr
* 1 – active-backup
* 2 – balance-xor
* 3 – broadcast
* 4 – 802.3ad
* 5 – balance-tlb
* 6 – balance-alb
Technically, outside of Base View or `cmsh`, the bonding mode is just another bonding option specified as part of the options string. However in BCM the bonding mode value is set up using the dedicated `mode` property of the bonded interface, for the sake of clarity. To avoid conflict with the value of the `mode` property, trying to commit a bonding mode value as an `option=value` pair will fail validation.
### 3.5.2 Single Bonded Interface On A Regular Node
A single bonded interface on a node can be configured and coexist in several ways on nodes with multiple network interfaces. Possibilities and restrictions are illustrated by the following:
* The bonded interface may be made up of two member interfaces, and a third interface outside of the bond could be the boot interface. (The boot interface is the node interface used to PXE boot the node before the kernel loads (section [5.1](/admin/provisioning#51-before-the-kernel-loads))).
* The boot interface could itself be a member of the bonded interface. If the boot interface is a member of a bonded interface, then this is the first bonded interface when interfaces are listed as in the example [here](#33-configuring-bridge-interfaces).
* The bonded interface could be set up as the provisioning interface. However, the provisioning interface cannot itself be a member of a bonded interface. (The provisioning interface is the node’s interface that picks up the image for the node after the initial ramdisk is running. Chapter [5](/admin/provisioning) covers this in more detail).
* A bonded interface can be set up as the provisioning interface, while having a member interface which is used for PXE booting.
### 3.5.3 Multiple Bonded Interface On A Regular Node
A node can also have multiple bonded interfaces configured. Possibilities and restrictions are illustrated by the following:
* Any one of the configured bonded interfaces can be configured as the provisioning interface. However, as already mentioned in the case of single bond interfaces (section [3.5.2](#352-single-bonded-interface-on-a-regular-node)), a particular member of a bonded interface cannot be made the provisioning interface.
* When a bonded interface is set as the provisioning interface, then during the node-installer phase of boot, the node-installer brings up the necessary bonded interface along with all its member interfaces so that node provisioning is done over the bonded interface.
### 3.5.4 Bonded Interfaces On Head Nodes And HA Head Nodes
It is also possible to configure bonded interfaces on head nodes.
For a single head node setup, this is analogous to setting up bonding on regular nodes.
For a high availability (HA) setup (Chapter [15](/admin/ha)), bonding is possible for `internalnet` as well as for `externalnet`, but it needs the following extra changes:
* For the bonded interface on `internalnet`, the shared internal IP alias interface name (the value of `networkdevicename`, for example, `eth0:0` in figure [15.1](/admin/ha#1514-failover-network-topology)) for that IP address should be renamed to the bonded alias interface name on `internalnet` (for example, `bond0:0`).
* For the bonded interface on `externalnet`, the shared external IP alias interface name (the value of `networkdevicename`, for example, `eth1:0` in figure [15.1](/admin/ha#1514-failover-network-topology)) for that IP address should be renamed to the bonded alias interface name on `externalnet` (for example, `bond1:0`).
* Additionally, when using a bonded interface name for the internal network, the value of the provisioning network interface name (the value of `provisioninginterface`, for example, `eth0`) for the head nodes, must be changed to the name of the bonded interface (for example, `bond0 [prov]`) on the internal network. The `provisioninginterface` value setting is described further [here](/admin/provisioning#5473-interface-used-to-receive-image-data-provisioninginterface).
**Example**
```bash
[headnode1->device[headnode1]->interfaces]% list
Type Network device name IP Network
------------ -------------------- ---------------- ------------
alias bond0:0 10.141.255.252 internalnet
alias bond1:0 10.150.57.1 externalnet
bond bond0 [prov] 10.141.255.254 internalnet
bond bond1 10.150.57.3 externalnet
physical eth0 (bond0) 0.0.0.0
physical eth1 (bond0) 0.0.0.0
physical eth2 (bond1) 0.0.0.0
physical eth3 (bond1) 0.0.0.0
```
### 3.5.5 Tagged VLAN On Top Of a Bonded Interface
It is possible to set up a tagged VLAN interface on top of a bonded interface. There is no requirement for the bonded interface to have an IP address configured in this case. The IP address can be set to `0.0.0.0`, however a network must be set.
**Example**
```bash
[headnode1->device[node001]->interfaces]% list
Type Network device name IP Network
------------ -------------------- ---------------- ------------
bond bond0 0.0.0.0 internalnet
physical eth0 [prov] 10.141.0.1 internalnet
physical eth1 (bond0) 0.0.0.0
physical eth2 (bond0) 0.0.0.0
vlan bond0.3 10.150.1.1 othernet
```
### 3.5.6 Association Of MAC Address With A Bonded Interface
BCM has MAC settings for interfaces as well as for nodes.
When a node is provisioned via PXE booting (Chapter [5](/admin/provisioning)), the MAC on the NIC that is being booted from is matched with the MAC property of the node. This is how the node is identified.
If a bonded interface is used, then any interface may be used for PXE booting. Each interface has its own MAC address, which means that the interface MAC address may not match the assigned node MAC address. In BCM 11 and later versions, the node-installer recognizes any MAC address in a bonded interface.
### 3.5.7 Further Notes On Bonding
If using bonding to provide failover services, then the kernel module option setting for media independent interface monitoring, `miimon`, which is set to be off by default (set to 0), should be given a non-zero value.
The `miimon` setting is the time period in milliseconds between checks of the interface carrier state. A common value is `miimon=100`.
Its value is set in the bonding device configuration file.
* For RHEL and derivatives, and for SLES distributions, the file is in a network script along a file path such as `/etc/sysconfig/network-scripts/ifcfg-bond0`.
The line within the stanza for the `ifcfg-bond0` file might look like:
**Example**
```bash
BONDING_MODULE_OPTS="mode=active-backup miimon=100"
```
and can be set manually.
* For RHEL and derivatives, it can alternatively be configured with front-end tools such as the `nmcli` tool, `nmtui`, or others.
**Example**
```bash
# nmcli connection add type bond con-name bond0 ifname bond0 bond.options \
"mode=active-backup,miimon=100"
```
* For SLES, YaST can be used as a front-end tool (`YaST` > `System` > `Network Settings`).
* For Ubuntu, the interface definitions are in a file, either `/etc/network/interfaces`, or a file under `/etc/network/interfaces.d/`.
The line to set the value of `miimon` follows a form such as:
* `bond-miimon 100`
instead of
* `miimon=100`
and can be set manually.
Alternatively, Canonical’s `netplan` ([https://netplan.io](https://netplan.io)) utility can be used to set the network configuration files. The `netplan` YAML configuration key to set `bond-miimon` is [`mii-monitor-interval.`](https://netplan.io/reference#properties-for-device-type-bonds)
When listing interfaces in `cmsh`, if an interface is a member of a bond or bridge interface, then the corresponding bonded or bridge interface name is shown in parentheses after the member interface name. Section [3.3](#33-configuring-bridge-interfaces), on configuring bridge interfaces, shows an example of such a listing from within `cmsh` [here](#33-configuring-bridge-interfaces).
More on bonded interfaces (including a detailed description of bonding options and modes) can be found at [http://www.kernel.org/doc/Documentation/networking/bonding.txt](http://www.kernel.org/doc/Documentation/networking/bonding.txt).
## 3.6 Configuring InfiniBand Interfaces
On clusters with an InfiniBand interconnect, the InfiniBand Host Channel Adapter (HCA) in each node must be configured before it can be used.
This section describes how to set up the InfiniBand service on the nodes for regular use. Setting up InfiniBand for booting and provisioning purposes is described in Chapter [5](/admin/provisioning), while setting up InfiniBand for NFS is described in section [3.14.4](#3144-configuring-nfs-volume-exports-and-mounts-over-rdma-with-ofed-drivers).
### 3.6.1 Installing Software Packages
On a standard NVIDIA Base Command Manager cluster, the OFED (OpenFabrics Enterprise Distribution) packages that are part of the Linux base distribution are used. These packages provide RDMA implementations allowing high bandwidth/low latency interconnects on OFED hardware. The implementations can be used by InfiniBand hardware.
By default, all relevant OFED packages are installed on the head node and software images. It is possible to replace the distribution OFED with a non-distribution OFED. The replacement can be for the entire cluster, or only for certain software images. Administrators may choose to switch to a different OFED version if the HCAs used are not supported by the distribution OFED version, or to increase performance by using an OFED version that has been optimized for a particular HCA. Installing a DOCA Mellanox OFED stack onto BCM is covered in [Chapter 10](/installation/nvidiamlnxofed) of the *Installation Manual*.
If the InfiniBand network is enabled during cluster installation, then the `infiniband.conf` and `rdma.conf` modules in the subdirectory `etc/rdma/modules/` are automatically configured for the detected hardware.
The relevant InfiniBand HCA kernel modules are then automatically loaded during the init stage by systemd. Verifying that InfiniBand is active can be done after the cluster is up and running by running `ibstat`:
```bash
auser@head:~$ ibstat
CA 'mlx5_0'
CA type: MT4123
Number of ports: 1
Firmware version: 20.31.2006
Hardware version: 0
Node GUID: 0xb8599f0300e4222a
System image GUID: 0xb
Port 1:
State: Active
Physical state: LinkUp
...
```
### 3.6.2 Mellanox Adapter Settings With `mlxconfig`
The device mode `mlxconfig` command displays and sets Mellanox adapter configuration on nodes. It is useful for checking the runtime state that the NVIDIA `mlxconfig` utility reports, and for applying selected supported settings from BCM.
**Example**
```bash
[basecm11->device]% mlxconfig node001 --adapters
Hostname Interface PCI HCA Card Name Port
-------- --------- ------------ ------ --------------------- ----
node001 ib0 0000:03:00.0 mlx5_0 MCX653105A-HDA_Ax_Bx 1
```
Without `--adapters`, the command expands each matching adapter into the known mlxconfig fields. The output shows the BCM field name, the current value, the pending value, and whether a reboot is required for pending changes to take effect. The output can be limited to an interface, PCI address, port, or detail level:
**Example**
```bash
[basecm11->device]% mlxconfig node001 --interface ib0
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --port 1
[basecm11->device]% mlxconfig node001 --detail-level 1
```
To set a value immediately on a node, use `--set`. A per-port field can be selected either from the interface context or by using a port suffix in the field name:
**Example**
```bash
[basecm11->device]% mlxconfig node001 --interface ib0 --set link_type=ETH
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --port 1 --set link_type=ETH
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --set link_type_p1=ETH
```
Some mlxconfig changes are pending until the node is rebooted. When this is the case, the set result indicates that a reboot is required.
#### 3.6.2.1 Persistent mlxconfig Policy With Extra Values
Persistent Mellanox adapter settings can be declared through `extra_values` entries whose keys start with `MLX_`. The desired settings can be defined on a category, on a node, or on a network interface. When the same field is defined at more than one level, the precedence is:
```bash
category.extra_values < node.extra_values < interface.extra_values
```
Category and node settings apply to all discovered Mellanox adapters on the node. Interface settings apply to the selected interface. For VLAN and bond interfaces, BCM resolves the setting to the underlying physical interface or interfaces before applying it.
The key name after `MLX_` can use a BCM field name, such as `link_type_p1`, or a native mlxconfig field name, such as `KEEP_ETH_LINK_UP_P1`. Values may be strings, booleans, or numbers. Boolean values are applied to mlxconfig as `1` or `0`.
**Example**
```bash
[basecm11->category[default]]% set -e MLX_link_type_p1 ETH
[basecm11->category*[default*]]% commit
[basecm11->device[node001]]% set -e MLX_num_vfs 16
[basecm11->device[node001]->interfaces[ib0]]% set -e MLX_KEEP_ETH_LINK_UP_P1 true
[basecm11->device*[node001*]->interfaces*[ib0*]]% commit
```
The settings are reconciled by the node after it connects to the head node and again during the periodic node update. The apply step is idempotent: BCM does not set a field again if the current or pending mlxconfig value already matches the desired value. If the local mlxconfig binary is missing, no Mellanox adapter is discovered, or mlxconfig rejects a value, BCM raises a warning event.
The `--desired` option shows the declared policy from the cluster configuration. It does not query the node or run mlxconfig, so it can be used to inspect the intended policy even before the node has applied it.
**Example**
```bash
[basecm11->device]% mlxconfig node001 --desired
Hostname Source Interface Field Value
-------- ------------- --------- -------------------- -----
node001 category:gpu * link_type_p1 ETH
node001 node * num_vfs 16
node001 interface:ib0 ib0 KEEP_ETH_LINK_UP_P1 true
```
### 3.6.3 Subnet Managers
Every InfiniBand subnet requires at least one subnet manager to be running. The subnet manager takes care of routing, addressing and initialization on the InfiniBand fabric. Some InfiniBand switches include subnet managers. However, on large InfiniBand networks or in the absence of a switch-hosted subnet manager, a subnet manager needs to be started on at least one node inside of the cluster. When multiple subnet managers are started on the same InfiniBand subnet, one instance will become the active subnet manager whereas the other instances will remain in passive mode. It is recommended to run 2 subnet managers on all InfiniBand subnets to provide redundancy in case of failure.
On a Linux machine that is not running BCM, an administrator sets a subnet manager service (usually `opensm`, but `opensmd` in SLES) to start at boot-time with a command such as:
`systemctl enable opensm.service`
However, for clusters managed by BCM, a subnet manager is best set up using CMDaemon. There are two ways of setting CMDaemon to start up the subnet manager on a node at boot time:
1. by assigning a role.
In `cmsh` this can be done with:
```bash
[root@basecm11 ~]# cmsh -c "device roles
Devices > All Devices > node002
\> Settings > Roles
\> + Add > [SubnetManagerRole]
2. by setting the service up. Services are covered more generally in section [3.15](#315-managing-and-configuring-services).
In `cmsh` this is done with:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device services node001
[basecm11->device[node001]->services]% add opensm
[basecm11->device[node001]->services*[opensm*]]% set autostart yes
[basecm11->device[node001]->services*[opensm*]]% set monitored yes
[basecm11->device[node001]->services*[opensm*]]% commit
[basecm11->device[node001]->services[opensm]]%
```
In Base View the subnet manager service is configured by selecting a head node or regular node from the resources tree, and adding the service to it. The navigation path for this, for a node `node002` for example, is:
Devices > All Devices > node002
\> Settings > Services
\> + Add
When the head node in a cluster is equipped with an InfiniBand HCA, it is a good candidate to run as a subnet manager for smaller clusters.
On large clusters a dedicated node is recommended to run the subnet manager.
### 3.6.4 InfiniBand Network Settings
Although not strictly necessary, it is recommended that InfiniBand interfaces are assigned an IP address (i.e. IP over IB). First, a network object in the cluster management infrastructure should be created. The procedure for adding a network is described in section [3.2.2](#322-adding-networks). The following settings are recommended as defaults:
| **Property** | **Value** |
|---|---|
| `Name` | `ibnet` |
| `Domain name` | `ib.cluster` |
| `Type` | `internal` |
| `Base address` | `10.149.0.0` |
| `Netmask bits` | `16` |
| `MTU` | up to `4k` in datagram mode |
| up to `64k` in connected mode |
| **Property** | **Value** |
|---|---|
| `Name` | `bmcnet` , `ilonet` , `ipminet` , `dracnet` , `cimcnet` , or `rfnet` |
| `Domain name` | `bmc.cluster` , `ilo.cluster` , `ipmi.cluster` , `drac.cluster` , `cimc.cluster` , or `rf.cluster` |
| `Type ` | `Internal` |
| `Base address` | `10.148.0.0` |
| `Netmask bits` | `16` |
| `Broadcast address` | `10.148.255.255` |
| **Property** | **Description** |
|---|---|
| `BMC User ID` | User type. Normally set to `4` for administrator access. |
| `BMC User Name` | User name used when sending a BMC command |
| `BMC Password` | Password for specified user name when sending a BMC command |
| `BMC Power reset delay` | Delay, in seconds, before powering up (default value: `0` ) |
| `BMC extra arguments` | Extra arguments passed to BMC commands |
| `BMC privilege` | Possible options are \ • administrator \ • callback \ • OEMproprietary \ • operator \ • user |
Devices > All Devices > Power Distribution Units
\> Power Control
If it is not set to `apc`, then the list of PDU ports is ignored by default.
### 3.8.2 Configuring SNMP
BCM can use SNMP to manage devices such as rack sensors, power distribution units (PDUs), and switches (including many InfiniBand switches). This requires that SNMP be enabled. If `disablesnmp` is set to `no`, the default, then SNMP is enabled. For example for a switch:
**Example**
```bash
[basecm11]% device add switch myibswitch
[basecm11->device[myibswitch]]% get disablesnmp
no
```
#### 3.8.2.1 Configuring SNMP Community Strings
In order to allow the cluster management software to communicate with a device that is managed using SNMP, SNMP must be enabled on it, and the SNMP community strings should be configured correctly.
By default, the SNMP community strings for switches and PDUs are set to `public` and `private` for read and write access respectively. If different SNMP community strings have been configured in the switch or PDU, the `readstring` and `writestring` properties of the corresponding switch device should be changed.
**Example**
```bash
[basecm11]% device use switch01
[basecm11->device[switch01]]% snmpsettings
[basecm11->device[switch01]->snmpsettings]% get readstring
public
[basecm11->device[switch01]->snmpsettings]% get writestring
private
[basecm11->device[switch01]->snmpsettings]% set readstring public2
[basecm11->device*[switch01*]->snmpsettings*]% set writestring private2
[basecm11->device*[switch01*]->snmpsettings*]% commit
```
Alternatively, these properties can also be set in Base View using the navigation path:
`Devices` > `All Devices` > `Switch` > ↗ > `SNMP Settings` > ↗
#### 3.8.2.2 Configuring SNMP Settings
SNMP settings can be configured in `cmsh` via the `snmpsettings` submode, which is available under `partition` mode as well as under `device` mode. The `partition` mode values are inherited by default by devices using SNMP settings at node level.
The effective (inherited) values can be displayed without entering the submode by using the `-s`|`--show` option, for example `snmpsettings -s` (section [3.7.2.1](#3721-viewing-the-effective-bmc-settings--sshow)).
The submode allows the version to be set to `v1`, `v2c`, or `v3`.
Setting the version to the value `file` is also an option, but is not meant as an option for end users. It is used in SNMP walk emulation for debugging non-standard switches.
The SNMPv3 settings that can be managed in BCM are:
**Example**
```bash
[basecm11]% device use switch01
[basecm11->device[switch01]]% snmpsettings
[basecm11->device[switch01]->snmpsettings]% show
Parameter Value
-------------------------------- ------------------------------------------------
Authentication key < not set >
Authentication protocol MD5
Context
Privacy key < not set >
Privacy protocol DES
Retries -1
Revision
Security level Authentication encrypted
Security name
Timeout 0s
VLAN Timeout 0s
version V3
```
The `set` command can be used, sometimes with tab-completion, to set the SNMP switch parameters. For example, for the SNMPv3 parameters that are set to use cryptographic keys:
**Example**
```bash
[basecm11->device*[switch01*]->snmpsettings*]% set authenticationprotocol Devices > All Devices > Switch > Uplinks > + Add\
(figure [3.11](#fig-base-view-configuring-switches-uplinks)):
*Figure 3.11: Notifying CMDaemon about uplinks with Base View*
The state is saved with the `Commit Switch` button of figure [3.11](#fig-base-view-configuring-switches-uplinks).
* In `cmsh`, the switch is accessed from the `device` mode. The uplink port numbers can be appended one-by-one with the `append` command, or `set` in one go by using space-separated numbers.
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% set switch01 uplinks 15 16
[basecm11->device*]% set switch02 uplinks 01
[basecm11->device*]% commit
successfully committed 3 Devices
```
### 3.9.4 The `showport` MAC Address to Port Matching Tool
The `showport` command can be used in troubleshooting network topology issues, as well as checking and setting up new nodes (section [5.4.2](/admin/provisioning#542-deciding-or-selecting-node-configuration)).
#### 3.9.4.1 Basic Use Of `showport`
In the `device` mode of `cmsh` is the `showport` command, which works out which ports on which switch are associated with a specified MAC address.
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% showport 00:30:48:30:73:92
switch01:12
```
When running `showport`, CMDaemon on the head node queries all switches until a match is found.
If a switch is also specified using the “`-s`” option, then the query is carried out for that switch first. Thus the preceding example can also be specified as:
```bash
[basecm11->device]% showport -s switch01 00:30:49.00:73:92
switch01:12
```
If there is no port number returned for the specified switch, then the scan continues on other switches.
#### 3.9.4.2 Mapping All Port Connections In The Cluster With `showport`
A list indicating the port connections and switches for all connected devices that are up can be generated using this script:
**Example**
```bash
#!/bin/bash
for nodename in $(cmsh -c "device; foreach * (get hostname)")
do
macad=$(cmsh -c "device use $nodename; get mac")
echo -n "$macad $nodename "
cmsh -c "device showport $macad"
done
```
The script may take a while to finish its run. It gives an output like:
**Example**
```bash
00:00:00:00:00:00 switch01: No ethernet switch found connected to this mac address
00:30:49.00:73:92 basecm11: switch01:12
00:26:6C:F2:AD:54 node001: switch01:1
00:00:00:00:00:00 node002: No ethernet switch found connected to this mac address
```
Related utilities to `showport` are:
* `finddevicebyswitchport` (section [3.9.7](#397-the-finddevicebyswitchport-command)) which—as the name suggests—finds the devices for each switch port
* `setmacviaswitchport` (section [3.9.8](#398-the-setmacviaswitchport-command)), running in `--dry-run` mode, which shows the discovered MAC address for a switch
### 3.9.5 Disabling Port Detection
An administrator may wish to disable node identification based on port detection. For example, in the case of switches with buggy firmware, the administrator may feel more comfortable relying on MAC-based identification. Disabling port detection can be carried out by clearing the `switchports` setting of a node, a category, or a group. For example, in `cmsh`, for a node:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use node001
[basecm11->device[node001]]% clear switchports
[basecm11->device*[node001*]]% commit
[basecm11->device[node001]]%
```
Or, for example for the `default` category, with the help of the `foreach` command:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]]% foreach -c default (clear switchports); commit
```
### 3.9.6 The `switchoverview` Command
Also within `device` mode, the `switchoverview` command gives an overview of MAC addresses detected on the ports, and some related properties. The command works using SNMP queries. Output is similar to the following (some lines ellipsized):
```bash
[basecm11->device]% switchoverview dell-switch1
Device: dell-switch1
State : [ UP ]
Model : 24G Ethernet Switch
Port Assignment:
Port Status Assigned Uplink Speed Detected
------ ------ ---------------- ------ -------- --------------------------\
--------------------------------------------------------------------
1 UP false 1 Gb/s -
2 UP false 1 Gb/s -
3 UP false 1 Gb/s 74:86:7A:AD:3F:2F, node3
4 UP false 1 Gb/s 74:86:7A:AD:43:E9, node4
...
11 UP false 1 Gb/s 74:86:7A:AD:44:D8, node11
12 UP false 1 Gb/s 74:86:7A:AD:6F:55, node12
...
23 UP false 1 Gb/s 74:86:7A:E9:3E:85, node23
24 UP false 1 Gb/s 74:86:7A:AD:56:DF, node24
49 UP false 10 Gb/s 74:86:7A:AD:68:FD, node1
50 UP false 10 Gb/s 74:86:7A:AD:41:A0, node2
53 UP node34 false 1 Gb/s 5C:F9:DD:F5:79.0D, node34
54 UP node35 false 1 Gb/s 5C:F9:DD:F5:45:AC, node35
...
179 UP false 1 Gb/s 24:B6:FD:F6:20:6F,\
24:B6:FD:FA:64:2F, 74:86:7A:DF:7E:4C, 90:B1:1C:3F:3D:A9,\
90:B1:1C:3F:51:D1, D0:67:E5:B7:64:0F, D0:67:E5:B7:61:20
180 UP false 100 Mb/s QDR-switch
205 UP true 10 Gb/s -
206 DOWN false 10 Gb/s -
...
[basecm11 ->device]%
```
### 3.9.7 The `finddevicebyswitchport` Command
The `finddevicebyswitchport` command, in `device` mode, is the converse of the `showport` command (section [3.9.4](#394-the-showport-mac-address-to-port-matching-tool)). Instead of finding the switch port for a known MAC address, it finds which device is connected to a specified switch port. The port is specified in the form:
\<*switch name*>`:<`*port number*>
An optional breakout port can be specified with:
\<*switch name*>`:<`*port number*>s \<*breakout port number*>
and several ports can be queried in one command.
**Example**
```bash
[basecm11->device]% finddevicebyswitchport switch01:1
node001
[basecm11->device]% finddevicebyswitchport switch01:1 switch01:2
switch01:1 node001
switch01:2 node002
```
When only one port is queried, just the matching hostname is shown. When several ports are queried, each line is prefixed with the \<*switch name*>`:<`*port number*> that was looked up. As with `showport`, the lookup relies on the port-to-MAC mapping that CMDaemon has discovered for the switch. If no device is associated with a specified port, then no hostname is returned for that port.
### 3.9.8 The `setmacviaswitchport` Command
The `setmacviaswitchport` command, in `device` mode, sets the MAC address of a device based on the MAC address discovered on the switch port to which the device is connected. This is convenient when provisioning hardware whose MAC address is not yet known to CMDaemon, but whose switch port connection is known.
The `--dry-run` option finds the MAC addresses, but does not save the change. Device selection options such as `-t` (type), `-n` (nodes), and `-c` (category), can be used to act on a set of devices. The `-f|--force` option forces an overwrite of devices that already have a MAC set, and the `--index` option selects which MAC to use, in case the switch port has more than one MAC associated with it.
**Example**
```bash
[basecm11->device]% setmacviaswitchport --dry-run -t switch
switch01 no no force and MAC already set
switch02 no no force and MAC already set
switch03 yes 00:00:00:00:00:01
...
```
The second column indicates whether the MAC was (or, for a dry run, would be) saved, while the last column gives the discovered MAC address, or the reason that no change was made. Running the command without the `--dry-run` option commits the discovered MAC addresses.
### 3.9.9 The `checkfactorydefaultpassword` Command
The `checkfactorydefaultpassword` command, in `device` mode, checks whether switches still have their factory default password set, by attempting to authenticate against them. With the `--update` option, it also changes the factory password to the target password that is defined in the `accesssettings` submode of the device. Switches can be selected by name, by a range of names, or with device selection options such as `-t switch`.
**Example**
```bash
[basecm11->device]% checkfactorydefaultpassword switch01..switch04
[basecm11->device]% checkfactorydefaultpassword --update switch01..switch04
[basecm11->device]% checkfactorydefaultpassword -t switch
```
Without the `--update` option the command only reports the status, and does not modify any passwords.
## 3.10 Additional `device` Mode Node Commands And Submodes
Besides the switch-oriented commands of the preceding section, `device` mode provides several commands and submodes for inspecting and configuring devices, including nodes.
### 3.10.1 The `nodeoverview` Command
The `nodeoverview` command displays a brief summary for a node. This includes its uptime, load averages, memory and workload manager (WLM) slot usage, per-interface network throughput, and per-mount disk usage. If no node is specified, then the current node is used.
**Example**
```bash
[basecm11->device]% nodeoverview node001
Device node001
Uptime 3w 3d
Load 0.16 / 0.18 / 0.17
Memory used 64 GiB (3.19%)
Memory unused 1.93 TiB (98.4%)
Memory total 1.96 TiB
WLM slots used 0 (0.0%)
WLM slots unused 144 (100.0%)
WLM slots total 144
enp22s22f0np0 1.69 GiB 3.3 GiB
ibp2p3s0 0 B 0 B
...
/ 21.1 GiB 1.60 TiB 98.7%
```
### 3.10.2 The `additionalnodeinformation` Command
The `additionalnodeinformation` command is run on a set of nodes, and displays administrator-registered information commands for those nodes. The set of registered commands can be listed with the `--list` option. The `-a|--run-on-active-head-node` option runs the command on the active head node instead of on the node itself. Nodes are selected with the usual selection options, such as `-n`, `-c`, or `-r`.
**Example**
```bash
[basecm11->device]% additionalnodeinformation -n node001 --list
info-1
info-2
```
Here `info-1` and `info-2` are the names of administrator-registered information commands. If no commands have been registered, then `--list` returns nothing.
Setting up the commands can be done with the `cm-manipulate-advanced-config.py` command (section [C.80.0.1](/admin/cmdconf#c8001-managing-keyvalue-pairs-with-the-cm-manipulate-advanced-configpy-utility), Managing Key/Value Pairs With The cm-manipulate-advanced-config.py Utility) as follows:
**Example**
```bash
[root@basecm11 ~]# cm-manipulate-advanced-config.py -g "AdditionalNodeInfo=info-1,info-2"
[root@basecm11 ~]# cm-manipulate-advanced-config.py -g "info-1=/cm/shared/info-1.sh"
[root@basecm11 ~]# cm-manipulate-advanced-config.py -g "info-2=/cm/shared/info-2.sh"
```
and then restarting `cmd` with `systemctl restart cmd`.
An example of a script for `info-1.sh` is:
**Example**
```bash
[root@basecm11 ~]# cat /cm/shared/info-1.sh
#!/bin/bash
/cm/local/apps/cmd/bin/cmsh -c "ds"
```
That script can then be run with:
**Example**
```bash
[basecm11->device]% additionalnodeinformation -n basecm11 info-1
name: info-1
path: /cm/shared/info-1.sh
exit code: 0
--- stdout ---
node001 .................. [ UP ]
node002 .................. [ UP ]
node003 .................. [ UP ]
node004 .................. [ UP ]
node005 .................. [ UP ]
basecm11 ................. [ UP ]
```
### 3.10.3 The `minimalconfig` Command
The `minimalconfig` command shows the minimal set of configuration objects that apply to a node. This includes the category, configuration overlays, head nodes, monitoring data producers, and other entities that make up a node configuration. It is useful when troubleshooting why a particular setting or monitoring item for a node may not work. If no node is specified, then the current node is used.
**Example**
```bash
[basecm11->device]% minimalconfig node001
Category default
Configuration overlay slurm-client
Configuration overlay slurm-server
Head node head01
Head node head02
MonitoringDataProducerCPU CPUSampler
MonitoringDataProducerGPU GPUSampler
MonitoringDataProducerDeviceState DeviceState
...
```
### 3.10.4 The `checkremotemount` Command
The `checkremotemount` command checks the status of remote mounts on a set of nodes. The mount to check is given as a path by default. The `--file` option evaluates the path as a file instead of a directory, the `-f|--force` option forces an update of the cached mounts data, and the `--clear` option clears the mount cache data. Nodes are selected with the usual selection options.
**Example**
```bash
[basecm11->device]% checkremotemount -n node001 /cm/shared
node001 1 good
```
The columns show the node, the number of matching mounts found, and the status of the mount.
### 3.10.5 The `proxysettings` Submode
The `proxysettings` submode, within `device` mode, configures the HTTP, HTTPS, and FTP proxy servers that a node uses, along with the list of hosts that should bypass the proxy.
**Example**
```bash
[basecm11->device]% proxysettings node001
[basecm11->device[node001]->proxysettings]% show
Parameter Value
-------------------------------- ------------------------------------------------
HTTP proxy address
HTTP proxy username
HTTP proxy password
HTTPS proxy address
HTTPS proxy username
HTTPS proxy password
FTP proxy address
FTP proxy username
FTP proxy password
Hosts without proxy
```
### 3.10.6 The `selinuxsettings` Submode
The `selinuxsettings` submode, within `device` mode, configures the SELinux behavior for a node, such as the SELinux mode and policy, whether to initialize SELinux, whether to reboot after a context restore, and the context actions to apply for the various install scenarios.
**Example**
```bash
[basecm11->device]% selinuxsettings node001
[basecm11->device[node001]->selinuxsettings]% show
Parameter Value
-------------------------------- ------------------------------------------------
Initialize yes
Reboot after context restore no
Allow NFS home directories yes
Context action auto install auto
Context action full install auto
Context action nosync install always
Mode permissive
Policy targeted
Key value settings Devices > All Devices > node001 > Disk Setup(a tabbed window)`> Devices > `\
and it brings up the window shown in figure [3.12](#fig-ChangingADiskedNodeToADisklessNodeWithbase-view):
*Figure 3.12: Changing a disked node to a diskless node with Base View*
However, in a default installation, the values for the devices in the disk layout are inherited from the category level, which can only be modified at category level. That means the default disk layout values are initially read-only values, with the values that were set up during installation ([section 3.3.16 of the *Installation Manual*](/installation/installing#3316-disk-layout)).
At node level, values can be written so that they override the inherited values. Writing values at node level can be done using the `Select Disk Setup` button in various ways:
1. The `Select Disk Setup` button brings up an option to copy from the category.
If copying over from the category is carried out, then the XML entries in the `Devices > ` screen (figure [3.13](#fig-base-view-configuring-disk-layouts-editing)) can be edited at node level (are no longer read-only):
*Figure 3.13: Editing disk layouts with Base View*
2. The `Select Disk Setup` button also presents a `Select from template` menu option, which shows several possible ready-made configurations that can be loaded up from the CMDaemon database. A selected copy can also be edited to suit the situation.
The XML schema for a node configured for diskless operation is shown in Appendix [D.9](/admin/partitioning#d9-example-diskless). This can often be deployed as is, or it can be modified during deployment using Base View or `cmsh`.
#### 3.13.4.1 Changing A Disk Layout From Disked To Diskless
One of the templates that can be used to switch from the existing disk layout to a diskless one is `slave-diskless.xml`. This XML configuration template can be loaded using the `Select from template` button, and saved to the node or node category.
3. The `Select Disk Setup` button also presents a `Select File` menu option, which can be used to upload a custom configuration via the browser.
### 3.13.5 Changing A Disk Layout Using `cmsh`
To edit an existing disk layout from within `cmsh`, the existing XML configuration is accessed by editing the `disksetup` property in `device` mode for a particular node, or by editing the `disksetup` property in `category` mode for a particular category. Editing is done using the `set` command, which opens up a text editor:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use node001
[basecm11->device[node001]]% set disksetup
```
After editing and saving the XML configuration, the change is then committed to CMDaemon with the `commit` command. It should be understood that a disk layout XML configuration is not stored in a file on the filesystem, but in the CMDaemon database. The XML configurations that exist on a default cluster at\
`/cm/images/default-image/cm/local/apps/cmd/etc/htdocs/disk-setup/`\
and\
`/cm/local/apps/cmd/etc/htdocs/disk-setup/`\
are merely default configurations.
If the `disksetup` setting for a device is deleted, using the `clear` command, then the category level `disksetup` property is used by the device. This is in accordance with the usual behavior for node values that override category values (section [2.1.5](/admin/clustermanagement#215-roles)).
Instead of editing an existing disk layout, another XML configuration can also be assigned. A diskless configuration may be chosen and set as follows:
**Example**
```bash
[basecm11->device[node001]]% set disksetup /cm/local/apps/cmd/\
etc/htdocs/disk-setup/slave-diskless.xml
```
In these preceding Base View and `cmsh` examples, after committing the change and rebooting the node, the node then functions entirely from its RAM, without using its own disk.
However, RAM is usually a scarce resource, so administrators often wish to optimize diskless nodes by freeing up the RAM on them from the OS that is using the RAM. Freeing up RAM can be accomplished by providing parts of the filesystem on the diskless node via NFS from the head node. That is, mounting the regular node with filesystems exported via NFS from the head node. The details of how to do this are a part of section [3.14](#314-configuring-nfs-volume-exports-and-mounts), which covers the configuration of NFS exports and mounts in general.
## 3.14 Configuring NFS Volume Exports And Mounts
NFS allows NFS clients shared access to a filesystem on an NFS server. The accessed filesystem is called an NFS volume by remote machines. The NFS server exports the filesystem to selected hosts or networks, and the clients can then mount the exported volume locally.
An unformatted filesystem cannot be used. The drive must be partitioned beforehand with `fdisk` or similar partitioning tools, and its filesystem formatted with `mkfs` or similar before it can be exported.
In BCM, the head node is typically used to export an NFS volume to the regular nodes, and the regular nodes then mount the volume locally.
* NFS can be made to work at higher speeds with remote direct memory access (RDMA), by bypassing the CPU. If there is RDMA hardware present, and if the `rdma-core` package is installed, then the RDMA service works automatically in RHEL 8 and 9.
The settings that determine client module loading are set in the file `/etc/rdma/modules/rdma.conf` so that the service auto-loads by default.
* An alternative to NFS over RDMA for very fast file systems is [the massively parallel and free (GPLv2) Lustre filesystem](https://whamcloud.com), running over InfiniBand.
#### 3.14.0.1 Automounting With NFS
If auto-mounting is used, then the configuration files for exporting should be set up on the NFS server, and the mount configurations set up on the software images. The service “`autofs`” or the equivalent can be set up using Base View via the “`Services`” option (section [3.15](#315-managing-and-configuring-services)) on the head and regular nodes or node categories. With `cmsh` the procedure to configure auto-mounting on the head and regular nodes could be:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11
[basecm11->device[basecm11]]% services
[basecm11->device[basecm11]->services]% add autofs
[basecm11->device*[basecm11*]->services*[autofs*]]% show
Parameter Value
------------------------------ -------------------------------------
Autostart no
Belongs to role no
Monitored no
Revision
Run if ALWAYS
Service autofs
Sickness check interval 60
Sickness check script
Sickness check script timeout 10
Timeout -1
[basecm11->device*[basecm11*]->services*[autofs*]]% set autostart yes
[basecm11->device*[basecm11*]->services*[autofs*]]% commit
[basecm11->device[basecm11]->services[autofs]]% category use default
[basecm11->category[default]]% services
[basecm11->category[default]->services]% add autofs
[basecm11->category*[default*]->services*[autofs*]]% set autostart yes
[basecm11->category*[default*]->services*[autofs*]]% commit
[basecm11->category[default]->services[autofs]]%
```
Filesystems imported to a regular node via an auto-mount operation must explicitly be excluded in `excludelistupdate` by the administrator, as explained in section [5.6.1.2](/admin/provisioning#5612-the-exclude-list-logic-for-excludelistupdate), [here](/admin/provisioning#automounterbeware).
#### 3.14.0.2 Dropping All Use Of NFS Version 3
At the time of writing, (August 2025), the default NFS protocol versions used in BCM are a mixture of version 3 and 4. This is because some aspects of NFS version 4 clients in BCM have not yet been extensively tested. Migrating the remaining version 3 parts over to version 4 is therefore still experimental. If, despite this, the cluster administrator would still like to migrate over to the use of only NFS version 4 clients, then it can be carried out as follows:
The following configuration files should be modified:
* `/etc/nfs.conf`: On all nodes, changing the default value of `vers3=y` by adding the line
```bash
vers3=n
```
* `/etc/nfsmount.conf`: On the head node(s), changing the line `Nfsvers=3` (the BCM default) to
```bash
Nfsvers=4
```
* `/cm/node-installer*/scripts/node-installer.conf`: On the head node(s), adding the line
```bash
disableNodeInstallerNFSCertificateStore = true
```
* `/cm/node-installer*/scripts/initrd_linuxrc`: On the head node(s), modifying the `nolock` entries:
```bash
root@basecm11:~# grep nolock /cm/node-installer/scripts/initrd_linuxrc
# mount with nolock so we do not have to run portmapper+statd
mount -o nolock,ro $PXE_MASTER_IP:$nodeinstallerpath /installer_root
mount -vo nolock,ro $PXE_MASTER_IP:$nodeinstallerpath /installer_root
```
so that the `vers=4` option is explicitly specified:
```bash
root@basecm11:~# grep nolock /cm/node-installer/scripts/initrd_linuxrc
# mount with nolock so we do not have to run portmapper+statd
mount -o nolock,ro,vers=4 $PXE_MASTER_IP:$nodeinstallerpath /installer_root
mount -vo nolock,ro,vers=4 $PXE_MASTER_IP:$nodeinstallerpath /installer_root
```
The changes can be deployed by rebuilding the ramdisk:
**Example**
```bash
root@basecm11:~# cmsh
[basecm11]% softwareimage foreach * (createramdisk)
```
and then rebooting the head and regular nodes.
The rest of this section describes the configuration of NFS for static mounts, using Base View or `cmsh`.
Sections [3.14.1](#3141-exporting-a-filesystem-using-base-view-and-cmsh) and [3.14.2](#3142-mounting-a-filesystem-using-base-view-and-cmsh) explain how exporting and mounting of filesystems is done in general by an administrator using Base View and `cmsh`, and considers some mounting behavior that the administrator should be aware of.
Section [3.14.3](#3143-mounting-a-filesystem-subtree-for-a-diskless-node-over-nfs) discusses how filesystems in general on a diskless node can be replaced via mounts of NFS exports.
Section [3.14.4](#3144-configuring-nfs-volume-exports-and-mounts-over-rdma-with-ofed-drivers) discusses how OFED InfiniBand or iWarp drivers can be used to provide NFS over RDMA.
### 3.14.1 Exporting A Filesystem Using Base View And `cmsh`
#### 3.14.1.1 Exporting A Filesystem Using Base View
As an example, if an NFS volume exists at “`basecm11:/modeldata`” it can be exported using Base View using the head node navigation path:\
Devices > All Devices > Head Node > > Settings > Filesystem exports >
This shows the list of exports (figure [3.14](#fig-base-view-headnodes-fsexports-list)):
*Figure 3.14: NFS exports from a head node viewed using Base View*
Using the `+ Add` button, a new entry (figure [3.15](#fig-base-view-headnodes-fsexports-add)) can be configured with values as shown:
*Figure 3.15: Setting up an NFS export using Base View*
For this example, the value for “`Name`” is set arbitrarily to “`Fluid Model Data`”, the value for `Path` is set to `/modeldata`, and the value for `Network` is set from the selection menu to allowing access to `internalnet` (by default `10.141.0.0/16` in CIDR notation).
By having the `Write` option disabled, read-only access is kept.
Saving this preceding configuration means the NFS server now provides NFS access to this filesystem for `internalnet`.
The network can be set to other network values using CIDR notation. It can also be set to particular hosts such as just node001 and node002, by specifying a value of “`node001 node002`” instead. Other settings and options are also possible and are given in detail in the man pages for `exports(5)`.
#### 3.14.1.2 Exporting A Filesystem Using `cmsh`
The equivalent to the preceding Base View NFS export procedure can be done in `cmsh` by using the `fsexports` submode on the head node (some output elided):
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11
[basecm11->device[basecm11]]% fsexports
[...->fsexports]% add "Fluid Model Data"
[...->fsexports*[Fluid Model Data*]]% set path /modeldata
[...[Fluid Model Data*]]% set hosts 10.141.0.0/16
[...[Fluid Model Data*]]% commit
[...->fsexports[Fluid Model Data]]% list | grep Fluid
Name (key) Path Hosts Write
------------------- ------------- --------------- ------
Fluid Model Data /modeldata 10.141.0.0/16 no
```
#### 3.14.1.3 General Considerations On Exporting A Filesystem
**Built-in exports:** In versions of NVIDIA Base Command Manager prior to version 9.0, all filesystem exports could be removed from the `fsexports` submode, simply by using the `remove` command with the name of the export.
From version 9.0 onward however, the following filesystem exports:
* `/var/spool/burn`
* `/home`
* `/cm/shared`
are treated as special built-ins.
**Head node role and `disableautomaticexports`:** Built-ins are exported automatically as part of the `headnode` role, also introduced in NVIDIA Base Command Manager 9.0, and cannot simply be removed.
To disable export of the built-in file systems, the `disableautomaticexports` command must be run in the `headnode` role for that node:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11
[basecm11->device[basecm11]]% roles
[basecm11->device[basecm11]->roles]% use headnode
[basecm11->device[basecm11]->roles[headnode]]% show
Parameter Value
-------------------------------- -----------------------
Name headnode
Revision
Type HeadNodeRole
Add services yes
Disable automatic exports no
role use headnode
[basecm11->device[basecm11]->roles[headnode]]% set disableautomaticexports yes ; commit
```
**Disabling exports that are not built-ins:** Exports that are not built-ins can still simply be removed. However, also from NVIDIA Base Command Manager 9.0 onward they can also simply be disabled in the `fsexports` submode. For example there is an export created by the cluster administrator for `/opt`, it can be disabled as follows:
**Example**
```bash
[basecm11->device[basecm11]->fsexports]% list
Name (key) Path Network Disabled
---------------- ------- ... ----------- --------
opt /opt internalnet no
[basecm11->device[basecm11]->fsexports]% set opt disabled yes; commit
```
The reason for automating the export for nodes via a `headnode` role is that NVIDIA Base Command Manager 9.0 onward has multidistro and multiarch capabilities (section [9.7](/admin/softwareimagemanagement#97-creating-images-for-other-distributions-and-architectures-multidistro-and-multiarch)), which would make manual management of exports harder for such nodes. The reason for the extra hurdle of `disableautomaticexports` for built-ins is that disabling these exports can result in an unbootable system.
### 3.14.2 Mounting A Filesystem Using Base View And `cmsh`
Continuing on with the `Fluid Model Data` export example from the preceding section, the administrator decides to mount the remote filesystem over the default category of nodes. Nodes can also mount the remote filesystem individually, but that is usually not a common requirement in a cluster. The administrator also decides not to re-use the exported name from the head node. That is, the remote mount name `modeldata` is not used locally, even though NFS allows this and many administrators prefer to do this. Instead, a local mount name of `/modeldatagpu` is used, perhaps because it avoids confusion about which filesystem is local to a person who is logged in, and perhaps to emphasize the volume is being mounted by nodes with GPUs.
#### 3.14.2.1 Mounting A Filesystem Using Base View
In Base View the navigation path to manage the mount points of nodes in the category of `default` is:
Grouping > Categories[default] > Settings > Extended settings > Filesystem mounts >
A mount point can be added with the `+ Add` button. Values for the remote mount point (`basecm11:/modeldata`), the filesystem type (`nfs`), and the local mount point (`/modeldatagpu`) can then be set for the category, while the remaining options stay at their default values (figure [3.16](#fig-base-view-node-category-fsmounts-add)):
*Figure 3.16: Setting up NFS mounts on a node category using Base View*
Saving the configuration saves the values and creates the local mount point, so that the volume can then be accessed by nodes within that category.
#### 3.14.2.2 Mounting A Filesystem Using `cmsh`
The equivalent to the preceding Base View NFS mount procedure can be done in `cmsh` by using the `fsmounts` submode, for example on the `default` category. The `add` method under the `fsmounts` submode sets the mountpoint path, in this case `/modeldatagpu` (some output elided):
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% category use default
[basecm11->category[default]]% fsmounts
[basecm11->category[default]->fsmounts]% add /modeldatagpu
[basecm11->...*[/modeldatagpu*]]% set device basecm11:/modeldata
[basecm11->...*[/modeldatagpu*]]% set filesystem nfs
[basecm11->category*[default*]->fsmounts*[/modeldatagpu*]]% commit
[basecm11->category[default]->fsmounts[/modeldatagpu]]%
Device Mountpoint (key) Filesystem
--------------------- ------------------ ----------
...
basecm11:/modeldata /modeldatagpu nfs
[basecm11->category[default]->fsmounts[/modeldatagpu]]% show
Parameter Value
------------------- ---------------------
Device basecm11:/modeldata
Dump no
Filesystem nfs
Filesystem Check NONE
Mount options defaults
Mountpoint /modeldatagpu
```
Values can be `set` for `Mount options` other than default. For example, the `noac` flag can be added as follows:
```bash
[basecm11->...[/modeldatagpu]]% set mountoptions defaults,noac; commit
```
Mounting a CIFS might use:
```bash
[basecm11->...[/modeldatagpu]]% set mountoptions gid,users,file_mode=0666,dir_mode=0777,\
iocharset=iso8859-15,credentials=/path/to/credential
[basecm11->...[/modeldatagpu*]]% commit
```
A `_netdev` mount option to make systemd wait until the network is up before it is mounted can be added as follows:
```bash
[basecm11->...[/modeldatagpu]]% append mountoptions ,_netdev; commit
```
#### 3.14.2.3 General Considerations On Mounting A Filesystem
There may be a requirement to segregate the access of nodes. For example, in the case of the preceding, because some nodes have no associated GPUs.
Besides the “`Allowed hosts`” options of NFS exports mentioned earlier in section [3.14.1.1](#31411-exporting-a-filesystem-using-base-view), BCM offers two more methods to fine tune node access to mount points:
* Nodes can be placed in another category that does not have the mount point.
* Nodes can have the mount point set, not by category, but per device within the `Nodes` resource. For this, the administrator must ensure that nodes that should have access have the mount point explicitly set.
Other considerations on mounting are that:
* When adding a mount point object:
* The settings take effect right away by default on the nodes or node categories.
* If `noauto` is set as a mount option, then the option only takes effect on explicitly mounting the filesystem.
* If “`AutomaticMountAll=0`” is set as a CMDaemon AdvancedConfig directive (Appendix [C.104](/admin/cmdconf#c104-automaticmountall-directive), [here](/admin/cmdconf#c104-automaticmountall-directive)), then CMDaemon changes for `/etc/fstab` are written to the file, but the `mount -a` command is not run by CMDaemon. However, the administrator should be aware that since `mount -a` is run by the distribution during booting, a node reboot implements the mount change.
* While a mount point object may have been removed, umount does not take place until reboot, to prevent mount changes outside of the cluster manager. If a umount needs to be to done without a reboot, then it should be done manually, for example, using the `pdsh` or `pexec` command (section [14.1.0.1](/admin/administration#14101-what-pdsh-and-pexec-do)), to allow the administrator to take appropriate action if umounting goes wrong.
* When manipulating mount points, the administrator should be aware which mount points are inherited by category, and which are set for the individual node.
* In Base View, for a node, [inheritance by category](/admin/clustermanagement#213-node-categories) is indicated in the navigation path `Devices` > `Nodes[`*`node name`*`]` > `Edit` > `Settings` > `Filesystem mounts`, under the `INHERITED` column, with the entry `(Category)`.
* In `cmsh`, the category a mount belongs to is displayed in brackets. This is displayed from within the `fsmounts` submode of the `device` mode for a specified node:
**Example**
```bash
[root@basecm11 ~]# cmsh -c "device; fsmounts node001; list"
Device Mountpoint (key) Filesystem
------------------------ -------------------- ----------
[default] none /dev/pts devpts
[default] none /proc proc
[default] none /sys sysfs
[default] none /dev/shm tmpfs
[default] $localnfsserv+ /cm/shared nfs
[default] basecm11:/home /home nfs
basecm11:/cm/shared/exa+ /home/examples nfs
[root@basecm11 ~]#
```
To remove a mount point defined at category level for a node, it must be removed from within the category, and not from the specific node.
#### 3.14.2.4 Mount Order Considerations
Care is sometimes needed in deciding the order in which mounts are carried out.
* For example, if both `/usr/share/doc` and a replacement directory subtree `/usr/share/doc/compat-gcc-34-3.4.6java` are to be used, then the stacking order should be that `/usr/share/doc` is mounted first. This order ensures that the replacement directory subtree overlays the first mount. If, instead, the replacement directory were the first mount, then it would be overlaid, inaccessible, and inactive.
* There may also be dependencies between the subtrees to consider, some of which may prevent the start up of applications and services until they are resolved. In some cases, resolution may be quite involved.
The order in which such mounts are mounted can be modified with the `up` and `down` commands within the `fsmounts` submode of `cmsh`.
### 3.14.3 Mounting A Filesystem Subtree For A Diskless Node Over NFS
#### 3.14.3.1 NFS Vs `tmpfs` For Diskless Nodes
For diskless nodes (Appendix [D.9](/admin/partitioning#d9-example-diskless)), the software image (section [2.1.2](/admin/clustermanagement#212-software-images)) is typically installed from a provisioning node by the node-installer during the provisioning stage, and held as a filesystem in RAM on the diskless node with the `tmpfs` filesystem type.
It can be worthwhile to replace subtrees under the diskless node filesystem held in RAM with subtrees provided over NFS. This can be particularly worthwhile for less frequently accessed parts of the diskless node filesystem. This is because, although providing the files over NFS is much slower than accessing it from RAM, it has the benefit of freeing up RAM for tasks and jobs that run on diskless nodes, thereby increasing the cluster capacity.
An alternative “semi-diskless” way to free up RAM is to use a local disk on the node itself for supplying the subtrees. This is outlined in Appendix [D.10](/admin/partitioning#d10-example-semi-diskless).
#### 3.14.3.2 Moving A Filesystem Subtree Out Of `tmpfs` To NFS
To carry out subtree provisioning over NFS, the subtrees are exported and mounted using the methods outlined in the previous examples in sections [3.14.1](#3141-exporting-a-filesystem-using-base-view-and-cmsh) and [3.14.2](#3142-mounting-a-filesystem-using-base-view-and-cmsh). For the diskless case, the exported filesystem subtree is thus a particular path under `/cm/images/` \<*`image`*> on the provisioning node, and the subtree is mounted accordingly under `/` on the diskless node. By default, \<*image*> is `default-image` on a newly-installed cluster.
While there are no restrictions placed on the paths that may be mounted in NVIDIA Base Command Manager 11, the administrator should be aware that mounting certain paths such as `/bin` is not possible.
When Base View or `cmsh` are used to manage the NFS export and mount of the subtree filesystem, then `tmpfs` on the diskless node is reduced in size due to the administrator explicitly excluding the subtree from `tmpfs` during provisioning.
An example might be to export `/cm/images/default-image` from the head node, and mount the directory available under it, `usr/share/doc`, at a mount point `/usr/share/doc` on the diskless node. In `cmsh`, such an export can be done by creating an FS export object corresponding to the software image object `defaultimage` with the following indicated properties (some prompt output elided):
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11; fsexports
[basecm11->device[basecm11]->fsexports]% add defaultimage
[basecm11...defaultimage*]]% set path /cm/images/default-image
[basecm11...defaultimage*]]% set hosts 10.141.0.0/16
[basecm11...defaultimage*]]% commit
[basecm11...defaultimage]]% list | grep defaultimage
Name (key) Path Hosts Write
---------------- ------------------------ ------------- -----
defaultimage /cm/images/default-image 10.141.0.0/16 no
```
As the output to `list` shows, the NFS export should be kept read-only, which is the default. Appropriate parts of the export can then be mounted by a node or node category. The mount is defined by setting the mount point, the `nfs` filesystem property, and the export device. For example, for a node category (some output elided):
```bash
[br...defaultimage]]% category use default
[basecm11->category[default]]% fsmounts
[basecm11->category[default]->fsmounts]% add /usr/share/doc
[basecm11->...*[/usr/share/doc*]]% set device basecm11:/cm/images/default-image/usr/share/doc
[basecm11->...*[/usr/share/doc*]]% set filesystem nfs
[basecm11->category*[default*]->fsmounts*[/usr/share/doc*]]% commit
[basecm11->category[default]->fsmounts[/usr/share/doc]]% list
Device Mountpoint (key) Filesystem
--------------------- ------------------ ----------
... ... ...
basecm11:/cm/images/default-image/usr/share/doc /usr/share/doc nfs
[basecm11->category[default]->fsmounts[/usr/share/doc]]% show
Parameter Value
---------------- -----------------------------------------------
Device basecm11:/cm/images/default-image/usr/share/doc
Dump no
Filesystem nfs
Filesystem Check 0
Mount options defaults
Mountpoint /usr/share/doc
```
Other mount points can be also be added according to the judgment of the system administrator. Some consideration of mount order may be needed, as discussed [here](#31424-mount-order-considerations) under the subheading “Mount Order Considerations”.
#### 3.14.3.3 An Example Of Several NFS Subtree Mounts
The following mounts save about 440MB from `tmpfs` on a diskless node with Rocky Linux 8, as can be worked out from the following subtree sizes:
```bash
[root@basecm11 ~]# cd /cm/images/default-image/
[root@basecm11 default-image]# du -sh usr/share/locale usr/lib/jvm usr/share/doc usr/src
160M usr/share/locale
118M usr/lib/jvm
88M usr/share/doc
77M usr/src
```
The filesystem mounts can then be created using the techniques in this section. After doing that, the result is then something like (some lines omitted):
```bash
[root@basecm11 default-image]# cmsh
[basecm11]% category use default; fsmounts
[basecm11->category[default]->fsmounts]% list -f device:53,mountpoint:17
device mountpoint (key)
-------------------------------------------------- -----------------
... ...
master:/cm/shared /cm/shared
master:/home /home
basecm11:/cm/images/default-image/usr/share/locale /usr/share/locale
basecm11:/cm/images/default-image/usr/lib/jvm /usr/lib/jvm
basecm11:/cm/images/default-image/usr/share/doc /usr/share/doc
basecm11:/cm/images/default-image/usr/src /usr/src
[basecm11->category[default]->fsmounts]%
```
Diskless nodes that have NFS subtree configuration carried out on them can be rebooted to start them up with the new configuration.
### 3.14.4 Configuring NFS Volume Exports And Mounts Over RDMA With OFED Drivers
If running NFS over RDMA, then at least NFS version 4.0 is recommended. NFS version 3 will also work with RDMA, but uses IPoIB encapsulation instead of native verbs. NFS version 4.1 uses the RDMA Connection Manager (`librdmacm`), instead of the InfiniBand Connection Manager (`ib_cm`) and is thereby also able to provide pNFS.
The administrator can set the version of NFS used by the cluster by setting the value of `Nfsvers` in the file `/etc/nfsmount.conf` on the head node(s).
#### 3.14.4.1 Drivers To Use For NFS Over RDMA
The DOCA Mellanox OFED drivers ([Chapter 10 of the *Installation Manual*](/installation/nvidiamlnxofed)) can support using the RDMA protocol (section [3.6](#36-configuring-infiniband-interfaces)) to provide NFS.
The distribution OFED drivers also support NFS over RDMA.
When using NFS over RDMA, `ibnet`, the IP network used for InfiniBand, should be set. Section [3.6.4](#364-infiniband-network-settings) explains how that can be done.
#### 3.14.4.2 Exporting With Base View And `cmsh` Using NFS Over RDMA
With the drivers installed, a volume export can be carried out using NFS over RDMA.
The procedure using Base View is much the same as done in section [3.14.1](#3141-exporting-a-filesystem-using-base-view-and-cmsh) (“Exporting A Filesystem Using Base View”), except for that the `ibnet` network should be selected instead of the default `internalnet`, and the “`RDMA`” option should be enabled.
The procedure using `cmsh` is much the same as done in section [3.14.1](#3141-exporting-a-filesystem-using-base-view-and-cmsh) (“Exporting A Filesystem Using `cmsh`”), except that the `ibnet` network (normally with a recommended value of 10.149.0.0/16) should be set, and the `rdma` option should be set.
**Example**
(based on the example in section [3.14.1](#3141-exporting-a-filesystem-using-base-view-and-cmsh))
```bash
...
[...->fsexports*[Fluid Model Data*]]% set path /modeldata
[...[Fluid Model Data*]]% set hosts ibnet
[...[Fluid Model Data*]]% set rdma yes
[...[Fluid Model Data*]]% commit
...
```
#### 3.14.4.3 Mounting With Base View And `cmsh` Using NFS Over RDMA
The mounting of the exported filesystems using NFS over RDMA can then be done.
The procedure using Base View is largely like that in section [3.14.2.1](#31421-mounting-a-filesystem-using-base-view), (“Mounting A Filesystem Using Base View”), except that the `Device` entry must point to `master.ib.cluster` so that it resolves to the correct NFS server address for RDMA, and the checkbox for `NFS over RDMA` must be ticked.
The procedure using `cmsh` is similar to that in section [3.14.2.2](#31422-mounting-a-filesystem-using-cmsh), (“Mounting A Filesystem Using `cmsh`”), except that device must be mounted to the `ibnet`, and the `rdma` option must be set, as shown:
**Example**
(based on the example in section [3.14.2.2](#31422-mounting-a-filesystem-using-cmsh))
```bash
...
[basecm11->category[default]->fsmounts]% add /modeldatagpu
[basecm11->...*[/modeldatagpu*]]% set device basecm11.ib.cluster:/modeldata
[basecm11->...*[/modeldatagpu*]]% set filesystem nfs
[basecm11->...*[/modeldatagpu*]]% set rdma yes
[basecm11->category*[default*]->fsmounts*[/modeldatagpu*]]% commit
...
```
## 3.15 Managing And Configuring Services
### 3.15.1 Why Use The Cluster Manager For Services?
Linux administrators should be familiar with managing services from the command line using `systemctl`:
**Example**
```bash
systemctl start Devices > All Devices > Head Node > Settings > Services
* regular nodes, for example for `node001` using a navigation path of Devices > All Devices > node001 > Settings > Services (figure [3.17](#fig-base-view-service-node)):
*Figure 3.17: Operating system service configuration list window for nodes in Base View*
* Node categories, for example using a navigation path of Grouping > Categories > Settings > Services
A service can be added with the `+Add` button, and fields of the service can then be edited.
Figure [3.18](#fig-base-view-service-cups-add) shows CUPS in the services window of Base View (figure [3.18](#fig-base-view-service-cups-add)).
*Figure 3.18: Setting up a service using Base View*
The service command options `start`, `stop`, `restart`, and so on, are displayed as options when clicking on the `⋮` in the `Actions` column (figure [3.19](#fig-base-view-service-cups-node-categories-start-stop)):
*Figure 3.19: Operating system service options, seen after clicking on ⋮ in Actions column in Base View*
#### 3.15.2.6 Monitoring A Service With `cmsh` And Base View
The service is
* in a `DOWN` state if it is not running, and
* in a `FAILING` state if it is unable to run after 10 auto-starts in a row
Event messages are sent during these first 10 auto-starts. After the first 10 auto-starts, no more event messages are sent, but autostart attempts continue.
In case an autostart attempt has not yet restarted the service, the `reset` option may be used to attempt an immediate restart. The `reset` option is not a service option in the regular shell, but is used by CMDaemon (within `cmsh` and Base View) to clear a `FAILING` state of a service, reset the attempted auto-starts count to zero, and attempt a restart of the service.
The monitoring system sets the `ManagedServicesOk` health check (Appendix [G.2.1](/admin/metricshealthchecksactions#g21-regular-health-checks)) to a state of `FAIL` if any of the services it monitors is in the `FAILING` state. In `cmsh`, the statuses of the services are listed by running the `latesthealthdata` command (section [10.6.3](/admin/monitoring#1063-the-latest-data-and-counter-valuesthe-latestdata-and-latestmetriccounters-commands)) from `device` mode.
Standard init.d script behavior is that the script return a non-zero exit code if the service is down, and a zero exit code if the service is up. A non-zero exit code makes BCM decide that the service is down, and should be restarted.
However, some scripts return a non-zero exit code even if the service is up. These services therefore have BCM attempt to start them repetitively, even though they are actually running.
This behavior is normally best fixed by setting a zero exit code for when the service is up, and a non-zero exit code for when the service is down.
## 3.16 Managing And Configuring A Rack
### 3.16.1 Racks
A cluster may have local nodes grouped physically into racks. A rack is 42 units in height by default, and nodes normally take up one unit.
#### 3.16.1.1 Rack List
**Rack list in Base View:** The `Rack list` pane can be opened up in Base View via the navigation path `Datacenter Infrastructure` > `Racks` (figure [3.20](#fig-base-view-racks-overview)):
*Figure 3.20: Rack list using Base View*
Racks can then be added, removed, or edited from the pane.
Within the `Rack list` pane:
* a new rack item can be added with the `+ Add` button. This opens up a rack configuration window where details of the new configuration can be entered and saved (figure [3.21](#fig-base-view-racks-configuration-settings)).
* an existing rack item can be edited with the navigation path > Settings. This opens up the `Settings` tab from within which rack configuration details can be managed.
**Racks overviews in `cmsh`:**
* The `list` command in `rack` mode in `cmsh` allows racks defined in the cluster manager to be listed:
```bash
[basecm11->rack]% list
Name (key) Room x-Coordinate y-Coordinate Height
-------------- ------------- ------------- ------------- ------
rack2 skonk works 2 0 42
racknroll 1 0 42
```
* The `rackoverview` command displays information about the types of entities in a specified rack. It also lists some more detailed information about some of the entity types:
**Example**
```bash
[basecm11->rack]% rackoverview
Devices > All Devices > node001
\> Settings > Rack +
*Figure 3.22: Rack assignment using Base View*
#### 3.16.2.2 Assigning Devices To A Rack Using `cmsh`
* In `device` mode, nodes can be assigned to a position in a rack. For example:
* `node001` to position `1`
* `node002` to position `2`
* `node003` to position `3`
A looping instruction to do this in an existing rack `rack2` is something similar to:
```bash
root@basecm11:~# for i in {001..003} ; do cmsh -c "device use node$i; get hostname; set rack rack2; \
get rack; rackposition; set deviceposition $i; get deviceposition; commit"; done
```
* The `addrackposition` command fills devices into a rack object more implicitly. By default the command fills devices into the first free slots on the rack.
```bash
root@basecm11:~# cmsh-c "device; addrackposition -n node001.node010 --position 1 rack2
```
The first free `deviceposition` value that it uses can be set with the `--position` option. Forcing a device into the the same position as another device is also possible, with the `--force` option.
More options can be found using the `help` text for the command.
```bash
[basecm11->device]% help addrackposition
Name:
addrackposition - Add a rack position information to one or more nodes
...
Examples:
addrackposition -n node001..node010 rack1
addrackposition -n node001..node010 --height 2 --position 4 rack rack2
```
* The `rackposition` submode allows individual nodes to be managed directly:
```bash
[basecm11->device[node002]->rackposition]% show
Parameter Value
-------------------------------- ------------------------------------------------
Rack rack2
Revision
Device position 2
Device height 1
Tray ID
Tray name
```
A similar submode to `rackposition` is `chassisposition` (section [3.16.4](#3164-the-chassisposition-submode)).
#### 3.16.2.3 The Convention Of The Top Of The Device Being Its Position
Since rack manufacturers usually number their racks from top to bottom, the position of a device in a rack (using the parameter `Position` in Base View, and the parameter `deviceposition` in `cmsh`) is always taken to be where the top of the device is located. This is the convention followed even for the less usual case where the rack numbering is from bottom to top.
A position on a rack is 1U of space. Most devices have a height that fits in that 1U, so that the top of the device is located at the same position as the bottom of the device, and no confusion is possible. The administrator should however be aware that for any device that is greater than 1U in height such as, for example, a blade enclosure chassis (section [3.16.3](#3163-assigning-devices-to-a-chassis)), the convention means that it is the position of the top of the device that is where the device is considered to be. The position of the bottom of the device is ignored.
### 3.16.3 Assigning Devices To A Chassis
#### 3.16.3.1 A Chassis As A Physical Part Of A Cluster
In a cluster, several local nodes may be grouped together physically into a chassis. This is common for clusters using blade systems. Clusters made up of blade systems use less space, less hardware, and less electrical power than non-blade clusters with the same computing power. In blade systems, the blades are the nodes, and the chassis is the blade enclosure.
A blade enclosure chassis is typically 6 to 10U in size, and the node density for server blades is typically 2 blades per unit with 2014 technology.
#### 3.16.3.2 Chassis Configuration And Node Assignment
**Chassis in Base View:** The `Chassis` pane can be added in Base View using the navigation path
`Devices > All Devices > + Add > Chassis`
This opens a `Chassis` pane where a new chassis object can be configured. Properties such as name, MAC address, rack location, can be set and saved (figure [3.23](#fig-base-view-chassis-subpane)).
An existing chassis object, for example `chassis01`, can be edited by selecting it from the navigation path Devices > All Devices > chassis01
*Figure 3.23: Base View chassis configuration*
The options that can be set within the `Chassis` pane include the following:
* `Hostname`: a name that can be assigned to the chassis operating system
* `Mac`: the MAC address of the chassis
* `Model`: the hardware model name
* `Rack`: the rack in which the chassis is placed
* `Layout`: how the nodes in a chassis are laid out visually.
* `Network`: which network the chassis is attached to.
* `Access Settings`: Opens up a pane to allow a user name and password to be set to access the chassis operating system, as well as the REST API port.
* `Power control`, `Custom power script`, `Custom power script argument`: power-related items for the chassis.
* `BMC Settings`: Baseboard Management Controller settings for the chassis.
* `Userdefined1`, `Userdefined2`: administrator-defined variables that can be used by CMDaemon.
**Basic chassis configuration and node assignment with `cmsh`:**
The `chassis` type in `cmsh` allows configuration related to a particular chassis. Tab-completion suggestions for a selected chassis with the `set` command show possible parameters that may be set:
**Example**
```bash
[basecm11->device]% add chassis chassis1
[basecm11->device*[chassis1*]]% set
accesssettings defaultgatewaymetric partition serialnumber
bmcsettings hostname partnumber supportsgnss
chassis interfaces powercontrol switchports
custompingscript layout powerdistributionunits userdefined1
custompingscriptargument mac prometheusmetricforwarders userdefined2
custompowerscript managementnetwork provisioninginterface userdefinedresources
custompowerscriptargument model rack
defaultgateway notes revision
```
Whether the suggested parameters are actually supported depends on the chassis hardware. For example, if the chassis has no power control or network interface of its own, then the `powercontrol` value and `mac` address settings may be set, but they cannot function.
The positioning parameters of the chassis within an already-defined rack, `rack2`, can be set as follows with `cmsh`:
**Example**
```bash
[basecm11->device[node001]]% rackposition
[basecm11->device*[node001*]->rackposition*]% set rack rack2
[basecm11->device*[node001*]->rackposition*]% set deviceposition 1; set deviceheight 6
[basecm11->device*[node001*]->rackposition*]% commit
```
The members of the chassis can be set as follows with `cmsh`:
**Example**
```bash
[basecm11->device[chassis1]]% append members basecm11 node001..node005
[basecm11->device*[chassis1*]]% commit
```
### 3.16.4 The `chassisposition` Submode
Section [3.16.2](#3162-assigning-devices-to-a-rack) described positioning attributes available in `device` mode, within the `rackposition` submode, for a device within a rack.
This section, describes positioning attributes available in `device` mode, within the `chassisposition` submode, for a device within a chassis.
Attributes include the chassis that the device belongs to, the slot, position, and width of the device within that chassis.
**Example**
```bash
[basecm11->device]% chassisposition node001
[basecm11->device[node001]->chassisposition]% show
Parameter Value
-------------------------------- ------------------------------------------------
Chassis
Slot
Position 1
Width 1
```
### 3.16.5 Power Management Of Racks
Section [4.2](/admin/power#42-power-operations) of this manual, describes the power operations and power status responses for devices in general, while [section 4.9.4 of the *NVIDIA Mission Control Manual*](/nvidia-mission-control/powershelf#494-power-management-of-racks) goes into some more detail for particular rack operations.
## 3.17 Configuring GPU Settings
### 3.17.1 GPUs And GPU Units
GPUs (Graphics Processing Units) are processors that are heavily optimized for executing certain types of parallel processing tasks. GPUs were originally used for rendering graphics, and one GPU typically has hundreds of cores. When used for general processing, they are sometimes called General Processing GPUs, or GPGPUs. For convenience, the “GP” prefix for General Processing is not used in this manual.
A GPU is typically placed on a PCIe card. GPUs can be physically inside the node that uses them, or they can be physically external to the node that uses them. As far as the operating system on the node making use of the physically external GPUs is concerned, the GPUs are internal to the node.
If the GPUs are physically external to the node, then they are typically in a *GPU unit*. A GPU unit is a chassis that hosts only GPUs. It is typically able to provide GPU access to several nodes, usually via PCIe extender connections. This ability means that external GPUs typically require more configuration than internal GPUs. GPU units are not covered in this manual because they are not very popular due to their greater cost and slowness.
Configuring GPU settings for GPUs—that is, for devices internal to a node—is covered next.
### 3.17.2 Configuring GPU Settings
#### 3.17.2.1 The `gpusettings` Submode In `cmsh`
In `cmsh`, GPUs can be configured for a specified node via `device` mode.
Going into the `gpusettings` submode for that node then allows a type of GPU to be set, from the `amd` or `nvidia` types, and a range to be specified for the GPU slots for that particular node:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% device use node001
[basecm11->device[node001]]% gpusettings
[basecm11->device[node001]->gpusettings]% add nvidia 1-3 ; commit
```
The range can be specified as
* a single number, for a particular slot, for example: `3`
* a range, for a range of slots, for example: `0-2`
* all, for all GPU slots on that node, using:
` all`
or
` *`
GPUs can also be configured for a specified category via `category` mode. For example, using the category `default`, then entering into the `gpusettings` submode allows a type (`nvidia` or `amd`) and a range to be set for the range of GPUs:
**Example**
```bash
[root@basecm11 ~]# cmsh
[basecm11]% category use default
[basecm11->category[default]]% gpusettings
[basecm11->category[default]->gpusettings]% list
GPU range (key) Power limit ECC mode Compute mode Clock speeds
---------------- ------------ ------------ ------------- -------------
[basecm11->category[default]->gpusettings]% add nvidia 1-3 ; commit
[basecm11->category[default]->gpusettings[nvidia:1-3]]% show
Parameter Value
------------------------------ ----------------------------------
Clock speeds
Clock sync boost mode
Compute mode
ECC mode
...
```
As usual, GPU settings for a node override those for a category (section [2.1.3](/admin/clustermanagement#213-node-categories)).
#### 3.17.2.2 GPU Settings With NVIDIA GPUs
The installation of the NVIDIA GPU software driver packages is covered [in section 9.1 of the *Installation Manual*](/installation/nvidiagpus#91-installing-cuda). It should be noted that the `cuda-dcgm` package must be installed to access NVIDIA GPU metrics.
The present section is about configuring NVIDIA GPUs in BCM. The driver itself does not necessarily have to be in place for the configuration to be done, although the configuration only becomes active when the driver is installed.
After a GPU type has been set, the following NVIDIA GPU settings may be specified, if supported, from within the `gpusettings` submode:
* `clockspeeds`: The pair of clock speeds (frequency in MHz) to be set for this parameter can be selected from the list of available speeds. The available speeds can be seen by running the `status` command. The values are specified in the form: \<*number for GPU processor*>,\<*number for memory*>
* `clocksyncboostmode`: GPU boosting. Exceed the maximum core and memory clock speeds if it is safe. Choices are:
* `enabled`
* `disabled`
* `computemode`: Contexts can be computed with the following values for this mode:
* `Default`: Multiple contexts are allowed
* `Exclusive thread`: Only one context is allowed per device, usable from one thread at a time
* `Exclusive process`: Only one context is allowed per device, usable from multiple threads at a time. This mode option is valid for CUDA 4.0 and later. Earlier CUDA versions ran only in this mode.
* `Prohibited`: No contexts are allowed per device
* `eccmode`: Sets the ECC bit error check, with:
* `enabled`
* `disabled`
When ECC is enabled:
* Single bit errors are detected, using the `EccSBitGPU` metric ([here](/admin/metricshealthchecksactions#g11-regular-metrics)), and corrected automatically.
* Double bit errors are also detected, using the `EccDBitGPU` metric ([here](/admin/metricshealthchecksactions#g11-regular-metrics)), but cannot be corrected.
* `GPU range`: range values can be set as follows:
* `all`: The GPU settings apply to all GPUs on the node.
* \<*number*>: The GPU settings apply to an individual GPU, for example: `1`
* \<*number range*>: The GPU settings apply to a range of GPUs, for example: `1,3-5`
* `powerlimit`: The administrator-defined upper power limit for the GPU. Only valid if `powermode` is `Supported`.
* `min`: The minimum upper power limit that the hardware supports.
* `max`: The maximum upper power limit that the hardware supports.
* \<*number*>: An arbitrary upper power limit, specified as a number between `min` and `max`
* `default`: Upper power limit based on the default hardware value.
If no value is specified for a GPU setting, then the hardware default is used.
#### 3.17.2.3 The `updatedevicepowerconfig` Command For Controlling Power Consumption
Above the `gpusettings` submode, within `device` mode, nodes can have their power consumption limited per specified GPU or CPU:
Without arguments, the `updatedevicepowerconfig` command shows the current status for GPUs:
**Example**
```bash
[basecm11->device[node001]]% updatedevicepowerconfig
Node Kind Device Power limit
------------ ------------ ------------ ----------------
node001 GPU GPU0 350 W
node001 GPU GPU1 300 W
```
In the preceding example, the only node with a GPU is node001. The same result is therefore shown if the command is run as `updatedevicepowerconfig -n node001`.
The kind of processing unit that can have its power configuration examined and set can be either a GPU or CPU. A CPU status command can be specified by using the `--kind` option:
**Example**
```bash
[basecm11->device[node001]]% updatedevicepowerconfig -n node001 --kind CPU
Node Kind Device Power limit
------------ ------------ ------------ ----------------
node001 CPU CPU0 200 W
node001 CPU CPU1 200 W
```
The first and second GPUs on `node001` can have their power limits set to the same value with a range syntax and the `--power-limit` option:
```bash
[basecm11->device]% updatedevicepowerconfig node001:0-1 --power-limit 330
Node Kind Device Power limit
------------ ------------ ------------ ----------------
node001 GPU GPU0 330 W
node001 GPU GPU1 330 W
```
For the first GPU on `node001`, and for all CPUs on `node002` the power limit, in watts, is set to specified values:
```bash
[basecm11->device]% updatedevicepowerconfig node001:0:300 node002:CPU*:150
Node Kind Device Power limit
------------ ------------ ------------ ----------------
node001 GPU GPU0 300 W
node001 GPU GPU1 300 W
node002 CPU CPU0 150 W
node002 CPU CPU1 150 W
```
A dry-run option `--dry-run` shows the effect on both the GPUs without execution.
```bash
[basecm11->device]% updatedevicepowerconfig node001:0:300 node002:CPU*:150
Node Kind Device Power limit
------------ ------------ ------------ ----------------
node001 GPU GPU0 300 W
node001 GPU GPU1 300 W
node002 CPU CPU0 150 W
node002 CPU CPU1 150 W
Setting the power limit in both gpusettings values and updatedevicepowerconfigs may cause conflict.
```
#### 3.17.2.4 GPU Settings With AMD GPUs
GPU settings for AMD Radeon GPUs are accessed via `cmsh` in the same way as NVIDIA GPU settings. The AMD GPU setting parameters do differ from the NVIDIA ones.
The AMD GPUs supported are Radeon cards. A list of cards and operating systems compatible with the Linux driver used is at [https://support.amd.com/en-us/kb-articles/Pages/Radeon-Software-for-Linux-Release-Notes.aspx](https://support.amd.com/en-us/kb-articles/Pages/Radeon-Software-for-Linux-Release-Notes.aspx)
AMD GPU driver installation is described in [section 7.4 of the *Installation Manual*](/installation/thirdparty#74-amd-gpu-driver-installation).
The Radeon Instinct MI25 shows the following settings in Ubuntu 16_06 running a Linux 4.4.0-72-generic kernel:
**Example**
```bash
[basecm11->device[node001]->gpusettings]% list
Type GPU range Info
---- --------- ------------------
AMD 0 PowerPlay: manual
[basecm11->device[node001]->gpusettings]% use amd:0
[basecm11->device[node001]->gpusettings[amd:0]]% show
Parameter Value
-------------------------------- -----------------------------
Activity threshold 1
Fan speed 255
GPU clock level 5
GPU range 0
Hysteresis down 0
Hysteresis up 0
Info PowerPlay: manual
Memory clock level 3
Minimum GPU clock 0
Minimum memory clock 0
Overdrive percentage 1
PowerPlay mode manual
Revision
Type AMD
```
The possible values here are:
* `activitythreshold`: Percent GPU usage at a clock level that is required before clock levels change. From 0 to 100.
* `fanspeed`: Maximum fan speed. From 0 to 255
* `gpuclocklevel`: GPU clock level setting. From 0 to 7.
* `gpurange`: The slots used.
* `hysteresisdown`: Delay in milliseconds before a clock level decrease is carried out.
* `hysteresisup`: Delay in milliseconds before a clock level increase is carried out.
* `info`: A compact informative line about the GPU status.
* `memoryclocklevel`: Memory clock speed setting. From 0-3. Other cards can show other values.
* `minimumgpuclock`: Minimum clock frequency for GPU, in MHz. The kernel only allows certain values. Supported values can be seen using the `status` command.
* `minimummemoryclock`: Minimum clock frequency for the memory, in MHz. The kernel only allows certain values. Supported values can be seen using the status command.
* `overdrivepercentage`: Percent overclocking. From 0 to 20%
* `powerplaymode`: Decides how the performance level power setting should be implemented.
* `high`: keep performance high, regardless of GPU workload
* `low`: keep performance low, regardless of GPU workload
* `auto`: Switch clock rates according to GPU workload
* `manual`: Use the memory clock level and GPU clock values.
The `status` command displays supported clock frequencies (some values ellipsized):
**Example**
```bash
[basecm11->device[node001]->gpusettings[amd:0]]% status
Index Name Property Value Supported
----- --------------------- ------------ ------------ -------------------------------------
0 Radeon Instinct MI25 Clock 1399Mhz 852Mhz, 991Mhz, ..., 1440Mhz, 1515Mhz
0 Radeon Instinct MI25 Memory 945Mhz 167Mhz, 500Mhz, 800Mhz, 945Mhz
```
#### 3.17.2.5 GPU Settings In Base View
In Base View, `GPU settings` can be accessed within the settings options for a category or a device. This brings up a GPU settings list.
**GPU settings list in Base View:** A `GPU Settings list` pane can be opened up in Base View for a regular node, for example `node001`, with the navigation path:\
Devices > All Devices > Nodes > Settings > GPU Settings
Similarly, a `GPU Settings list` pane can be opened up in Base View for nodes in a category, for example `gpunodes`, with the navigation path:\
Grouping > Categories\[gpunodes] > Settings > GPU Settings
Within the `GPU Settings list` pane:
* a new GPU settings item can be added with the `+Add` button, and then clicking on either the `AMD` or the `NVIDIA` item in the popup. This opens an `AMDGPUSettings` pane or an `NvidiaGPUSettings` pane, where GPU configuration can be carried out and saved (figure [3.24](#fig-base-view-nodes-gpusettings)).
*Figure 3.24: GPU settings window for a node*
* an existing GPU settings item can be edited with the ↗ option, which opens up the GPU Settings pane where GPU configuration can be managed.
#### 3.17.2.6 GPU Configuration For HPC Workload Managers
**Slurm GPU configuration via direct `slurm.conf` changes:** To configure NVIDIA GPUs for Slurm, changes are made in `slurm.conf` when `cm-wlm-setup` configures GPUs for Slurm (section [7.3](/admin/workloadmanagement#73-installation-of-workload-managers)).
Changes made are kept in the `AUTOGENERATED` section and can be worked out by checking the difference between the `slurm.conf.template` file and the actual `slurm.conf` file. Changes made include defining the GresTypes `gpu` and `mps`, and setting GPU plugins that allow Slurm generic resources to work.
The configured gres options can be seen by running `sbatch --gres=help`:
**Example**
```bash
[fred@basecm11 ~]$ sbatch --gres=help
Valid gres options are:
gpu[[:type]:count]
mps[[:type]:count]
```
This means that a GPU can be requested in a job script with the Slurm gres option:
```bash
#SBATCH --gres=gpu:1
```
Similarly, MPS resources ([https://slurm.schedmd.com/gres.html#MPS_Management](https://slurm.schedmd.com/gres.html#MPS_Management)) can be requested with:
```bash
#SBATCH --gres=mps:100
```
If adding new parameters manually, care must be taken to avoid duplication of parameters already in the file, because `slurmd` is unlikely to work properly with duplicated parameters.
The Slurm client role can be configured at configuration overlay, category, or node level. If configuring the Slurm client role for GPU gres resources manually, then each GPU can be configured within the role:
**Example**
```bash
[basecm11->configurationoverlay]% list
Name (key) Priority All head nodes Nodes Categories Roles
-------------------- ---------- -------------- ---------------- ---------------- ----------------
slurm-accounting 500 yes slurmaccounting
slurm-client 500 no default slurmclient
slurm-server 500 yes slurmserver
slurm-submit 500 no default slurmsubmit
wlm-headnode-submit 600 yes slurmsubmit
[basecm11->configurationoverlay]% use slurm-client
[basecm11->configurationoverlay[slurm-client]]% roles
[basecm11->configurationoverlay[slurm-client]->roles]% use slurmclient
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% genericresources
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]->genericresources]%
[basecm11->...->roles[slurmclient]->genericresources]% add gpu0
[basecm11->...->roles*[slurmclient*]->genericresources*[gpu0*]]% set name gpu
[basecm11->...->roles*[slurmclient*]->genericresources*[gpu0*]]% set file /dev/nvidia0
[basecm11->...->roles*[slurmclient*]->genericresources*[gpu0*]]% commit
[basecm11->...->roles[slurmclient]->genericresources[gpu0]]%
(Repeat similar settings for the other GPUs, gpu1...gpu7)
[basecm11->...->roles[slurmclient]->genericresources]% list
Alias (key) Name Type Count File
----------- -------- -------- -------- ----------------
gpu0 gpu /dev/nvidia0
gpu1 gpu /dev/nvidia1
gpu2 gpu /dev/nvidia2
gpu3 gpu /dev/nvidia3
gpu4 gpu /dev/nvidia4
gpu5 gpu /dev/nvidia5
gpu6 gpu /dev/nvidia6
gpu7 gpu /dev/nvidia7
[basecm11->configurationoverlay[slurm-client]->roles[slurmclient]->genericresources]%
```
By default, Slurm just allows a single job to be executed per node. To change this behavior, it is necessary to allow oversubscription. For example, to allow 8 jobs per node:
**Example**
```bash
[basecm11->wlm[slurm]]% jobqueue
[basecm11->wlm[slurm]->jobqueue]% use defq
[basecm11->wlm[slurm]->jobqueue[defq]]% set oversubscribe yes:8
[basecm11->wlm[slurm]->jobqueue*[defq*]]% commit
[basecm11->wlm[slurm]->jobqueue[defq]]%
```
**Slurm GPU configuration via auto-detection:** Instead of carrying out Slurm configuration by modifying `slurm.conf` by hand, it may be configured via auto-detection. More details on this are to be found starting at [here](/admin/workloadmanagement#slurmautodetection).
**LSF:** Within LSF cluster configuration, GPU devices can be autodetected by setting the `gpuautoconfig` parameter to `yes`. In `cmsh` this can be carried out with:
**Example**
```bash
[basecm11->wlm[lsf]]% set gpuautoconfig yes
[basecm11->wlm*[lsf*]]%
```
The parameter can also be set during LSF configuration via `cm-wlm-setup` (figure [3.25](#fig-cm-wlm-setup-ncurses-lsf-gpu-settings)):
*Figure 3.25: GPU settings screen for LSF in cm-wlm-setup*
GPU resource enforcement can be configured for LSF as follows:
**Example**
```bash
[basecm11->wlm[lsf]->cgroups]% append resourceenforce gpu
[basecm11->wlm*[lsf*]->cgroups*]% commit
[basecm11->wlm[lsf]->cgroups]%
```
### 3.17.3 MIG Configuration
MIG configuration can be carried out for the cluster using the BCM MIG management as described in this section.
An alternative for MIG configuration is to not use the BCM MIG management tool (`cmsh`), and to instead use other MIG management tools, such as the DGX native `nvidia-migmanager.service`, or the GPU operator-provided `nvidia-mig-manager`. If non-BCM MIG management tools are used, then BCM leaves the MIG configuration alone. Using multiple MIG management tools simultaneously to configure MIG should not be done.
#### 3.17.3.1 What Is MIG?
An Ampere NVIDIA GPU is a GPU based on the GA100 microarchitecture. It has compute capability 8.0, which means it can be configured into multiple logical GPU instances if it uses CUDA 11 and NVIDIA driver 450.80.02 or later.
This configuration of multiple logical GPU instances is called Multi-Instance GPU (MIG). The logical GPU instances are MIG devices, that are enabled by setting up the physical GPU to switch to MIG mode.
**As a sanity check to see if MIG is supported:** If the hardware and drivers are in place, then running the `nvidia-smi` command on the node with the physical GPU should display its MIG capability:
```bash
root@basecm11:~# ssh node001 "nvidia-smi" | grep MIG
| | | MIG M. |
| MIG devices: |
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
```
**GPU utilization information changes on enabling MIG:** Once enabled, the full physical GPU is no longer available as a device, and GPU utilization metrics become unavailable by default.
GPU profiling metrics (section [G.1.7](/admin/metricshealthchecksactions#g17-gpu-profiling-metrics)) for the physical GPU can however still be enabled. For example, it can be carried out with `cmsh` as follows, for a GPU on `node001`:
**Example**
```bash
basecm11->device[node001]]% gpuprofiling show
Hostname GPU Major ID Minor ID Field ID Metric Watched
----------- -------- ------------ ------------ ------------ ------------------------------ ----------
node001 0 0 1 1002 gpu_profiling_sm_active no
node001 0 0 1 1003 gpu_profiling_sm_occupancy no
...
basecm11->device[node001]]% gpuprofiling watch 1003
Hostname GPU Major ID Minor ID Field ID Metric Watched
----------- -------- ------------ ------------ ------------ ------------------------------ ----------
...
node001 0 0 1 1003 gpu_profiling_sm_occupancy yes
...
[basecm11->device[node001]]% metrics | grep gpu_profiling
Metric gpu_profiling_sm_occupancy gpu0 GPU GPUSampler
```
Enabling physical GPU profiling after MIG enablement should be done with caution, because:
* it may affect the performance
* newer drivers may support MIG profiling, which may be confusing
#### 3.17.3.2 Overview Of MIG Concepts And Terminology
The logical GPU instances are composed of *slices* of GPU resources. Slices are the smallest fraction possible of the resource that can be allocated in a logical GPU instance. Thus:
* memory slice: this is the smallest fraction of the memory of the physical GPU that can be allocated to the GPU instance. For the Ampere architecture this is 1/8(th) of the total physical GPU memory.
* SM slice: this is the smallest fraction of the streaming multiprocessors (SMs) on the GPU that can be allocated as a logical SM. An SM is composed of multiple cores (streaming processors). For the Ampere architecture an SM slice is 1/7(th) of the total physical GPU SMs.
* GPU slice: this is the smallest fraction of the physical GPU that has a single GPU memory slice and a single GPU SM slice. A maximum of 7 GPU slices can be specified from the original physical GPU.
The preceding fractional slices can be combined in various mixes to compose a logical `GPU instance`:
* GPU instance: a combination of GPU slices and GPU engines. GPU engines are hardware components that execute other work on the GPU, and can be encoders/decoders (NVENCs/NVDECs), shortcut connectors (CE (copy engine) for DMA), etc.
* compute instance: a part of a GPU instance. It consists of a subset of the parent GPU instance’s SM slices and other GPU engines (DMAs, NVENCs ...). The compute instances can share memory and GPU engines with other compute instances within their GPU instance.
Further details on the terminology and how slices can be allocated to GPU instances are given in the NVIDIA documentation at [https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#concepts](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#concepts).
A use case for creating several GPU instances from a full physical GPU is when allocation of the full physical GPU is wasteful.
For example, if a full physical GPU is allocated to a job, but the job only uses a fraction of the full set of GPU cores, then the allocation is wasteful, because no other job can then be processed on the remaining idle cores. Instead, if the physical GPU is split into several instances and the job allocated to an instance with a closer match in resources, then it means that other GPU instances are still available for processing other jobs.
When configuring GPU instances for the cluster, the administrator typically allocates all available slices to all the GPU instances that are being configured. Leaving a slice unallocated means that that slice cannot be available to jobs, and it means that the physical resources of that slice are lying idle. For example, if some of the 7 SM slices from the physical GPU are not used, then they are wasted as their processors are never available to jobs, and so that slice stays idle.
Cluster management of GPU instances is described in the following sections.
#### 3.17.3.3 MIG Status Of Physical GPUs
The MIG status on a GPU can be viewed with the `mig status` command. For example, the following shows 8 physical GPUs on `node001` that have not yet become MIG enabled:
**Example**
```bash
[basecm11->device[node001]]% mig status
Node GPU Active Pending
---------------- -------- -------------------------------- --------------------------------
node001 0 no no
node001 1 no no
node001 2 no no
node001 3 no no
node001 4 no no
node001 5 no no
node001 6 no no
node001 7 no no
```
#### 3.17.3.4 MIG `enable` And `disable` Options To Set Up Pending States For The Physical GPUs
If the `enable` or `disable` options are run, then by default all the physical GPUs are set to a pending state for enabled (`yes`) and disabled (`no`) respectively:
**Example**
```bash
[basecm11->device[node001]]% mig enable
Node GPU Active Pending
---------------- -------- -------------------------------- --------------------------------
node001 0 no yes
node001 1 no yes
node001 2 no yes
node001 3 no yes
node001 4 no yes
node001 5 no yes
node001 6 no yes
node001 7 no yes
```
Individual physical GPUs can also be set to a pending state of enabled or disabled, following the node list syntax (section [2.5.5.10](/admin/clustermanagement#25510-node-list-syntax)):
```bash
[basecm11->device[node001]]% mig disable 3,5-7
Node GPU Active Pending
---------------- -------- -------------------------------- --------------------------------
node001 0 no yes
node001 1 no yes
node001 2 no yes
node001 3 no no
node001 4 no yes
node001 5 no no
node001 6 no no
node001 7 no no
```
#### 3.17.3.5 Rebooting The MIG Instances To Activate/Deactivate Instances According To Pending State Settings
The pending states only become active after the node is rebooted for A100 GPUs:
**Example**
```bash
[basecm11->device[node001]]% reboot
Reboot in progress for: node001
...
node001 [ UP ]
[basecm11->device[node001]]% mig status
Node GPU Active Pending
---------------- -------- -------------------------------- --------------------------------
node001 0 yes yes
node001 1 yes yes
node001 2 yes yes
node001 3 no no
node001 4 yes yes
node001 5 no no
node001 6 no no
node001 7 no no
```
The effect of `enable` and `disable` persists after reboots, until the pending value changes once more.
For H100 GPUs and other more modern GPUs, the changes do not require a reboot.
#### 3.17.3.6 The MIG Profiles
**Listing MIG profiles:** The full list of the existing available MIG profiles for each physical GPU is displayed with the `mig profiles` command for the node. If there are 7 physical GPUs at the node, then the listing might look something like:
**Example**
```bash
[basecm11->device[node001]]% mig profiles
Node GPU ID Name Instances Memory
---------------- -------- -------- ---------- --------- --------
node001 0 1 1g.5gb 7 4.7GiB
node001 0 1 1g.5gb+me 1 4.7GiB
node001 0 2 2g.10gb 3 9.7GiB
node001 0 3 3g.20gb 1 19.6GiB
node001 0 4 4g.20gb 1 19.6GiB
node001 0 7 7g.40gb 1 39GiB
...
node001 7 2 2g.10gb 3 9.7GiB
node001 7 3 3g.20gb 1 19.6GiB
node001 7 4 4g.20gb 1 19.6GiB
node001 7 7 7g.40gb 1 39GiB
```
A list for physical GPU 1 can be displayed with:
**Example**
```bash
[basecm11->device[node001]]% mig profiles 1
Node GPU ID Name Instances Memory
---------------- -------- -------- ---------- --------- --------
node001 1 1 1g.5gb 7 4.7GiB
node001 1 1 1g.5gb+me 1 4.7GiB
node001 1 2 2g.10gb 3 9.7GiB
node001 1 3 3g.20gb 1 19.6GiB
node001 1 4 4g.20gb 1 19.6GiB
node001 1 7 7g.40gb 1 39GiB
```
**MIG profiles naming convention:** The naming format for the profile takes the form
\<*number of GPU slices in the physical GPU*>`g.<`*memory for slice in GB*>gb
The `+me` suffix implies media extensions being active. The number of GPU instances, the number of GPU slices used, and the memory used by the instance can thus be worked out from the name.
For example:
* `1g.5gb` implies that the size of the GPU slice used for the instances is 1. 4.7GiB of memory is used by each of the 7 GPU instances,
* `2g.10gb` implies that the size of the GPU slice used for the instances is 2. 9.7GiB of memory is used by each of the 3 GPU instances.
**Creating GPU instances, by setting the MIG profile for a GPU:** The MIG profile is an attribute that can be set within the GPU settings for its physical GPU. This can be done after having set up a CMDaemon entity for a physical GPU 0 (section [3.17.2](#3172-configuring-gpu-settings)):
**Example**
```bash
[basecm11->device[node001]]% gpusettings
[basecm11->device*[node001]->gpusettings]% list
Type GPU range Info
------ --------- --------
Nvidia 0 default
[basecm11->device[node001]->gpusettings]% use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% show
Parameter Value
-------------------------------- ------------------------------------------------
...
MIG profiles
```
* **A simple existing set of profiles** with 7 GPU instances with 1 GPU slice each, and 5 GB of memory for each slice can be specified with:
**Example**
```bash
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 1g.5gb; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% show
Parameter Value
-------------------------------- ------------------------------------------------
...
MIG profiles 1g.5gb
```
Setting the profiles, and carrying out the `commit` configures the GPU instances, but does not yet apply them:
**Example**
```bash
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig show
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
```
Applying the profile deploys the configuration, and shows the configuration:
**Example**
```bash
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 13 - 1g.5gb 19 6 1
node001 0 13 0 1g.5gb 0 0 1
```
In the preceding, 1 GPU instance has been deployed, with 1 compute instance slice using 5GB. The instance with the `-` represents the hosting GPU instance, while the subsequent row represents the compute instance.
* **Setting a profile with `mig apply --profile`** is an alternative to setting it within `gpusettings`. However, only a profile set within `gpusettings` is persistent. The profile set with the `--profile` option is lost if its node reboots. Multiple profiles can be set using comma-separation (instead of using space-separation). Using multiple `--profile` options is also possible.
**Example**
```bash
[basecm11->device[node001]]% mig apply --profile 1g.5gb,2g.10gb
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 2g.10gb 1 0 2
node001 0 13 - 1g.5gb 19 6 1
node001 0 13 0 1g.5gb 0 0 1
[basecm11->device[node001]]% mig apply --profile 1g.5gb --profile 2g.10gb
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 2g.10gb 1 0 2
node001 0 13 - 1g.5gb 19 6 1
node001 0 13 0 1g.5gb 0 0 1
```
* **Multiple GPU instances** can be specified, if the GPU allows it, using a \<*number*>`*` prefix syntax. So, to deploy 7 GPU instances, each hosting 1 compute instance with 5gb slices, the specification can be:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 7*1g.5gb; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
--------------- -------- -------- -------- -------- -------- -------- --------
node001 0 7 - 1g.5gb 19 0 1
node001 0 7 0 1g.5gb 0 0 1
node001 0 8 - 1g.5gb 19 1 1
node001 0 8 0 1g.5gb 0 0 1
node001 0 9 - 1g.5gb 19 2 1
node001 0 9 0 1g.5gb 0 0 1
node001 0 10 0 1g.5gb 19 3 1
node001 0 10 0 1g.5gb 0 0 1
node001 0 11 - 1g.5gb 19 4 1
node001 0 11 0 1g.5gb 0 0 1
node001 0 12 - 1g.5gb 19 5 1
node001 0 12 0 1g.5gb 0 0 1
node001 0 13 - 1g.5gb 19 6 1
node001 0 13 0 1g.5gb 0 0 1
```
* **GPU slices** can be implied by default by the profile, and subsets of these slices can be specified explicitly.
To deploy 1 GPU instance, with 7 GPU slices of compute instance resources, the specification can be:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 7g.40gb; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 0 - 7g.40gb 0 0 8
node001 0 0 0 7g.40gb 4 0 7
```
If a profile with a more than 1 GPU slice is chosen, then GPU slice subsets can be set up via the following syntax:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 2g.10gb; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 2g.10gb 1 0 2
```
* **GPU slices use a colon syntax** to explicitly specify subsets of GPU slices.
The configuration specification `2g.10gb` can also be specified as `2g.10gb:1`, where that `:1` indicates the number of GPU slices for the compute instance, counting from zero. That means the compute instance has 2 GPU slices. The resulting configuration is exactly the same as `2g.10g`.
If the specification `2g.10gb:0` is used instead, then the compute instance ends up looking like:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 2g.10gb:0; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- ----------- -------- -------- --------
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 1c.2g.10gb 0 0 1
```
Here the `1c` indicates 1 GPU slice (here it is counting from 1).
Adding another slice to a separate compute instance within the same GPU instance can be specified with:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 2g.10gb:0:0; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- ----------- -------- -------- --------
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 1c.2g.10gb 0 0 1
node001 0 5 1 1c.2g.10gb 0 1 1
```
In the preceding `set migprofiles` command, the specification
`2g.10gb:0:0`
can alternatively be expanded out and written in the form:
`2g.10gb:1c.2g.10gb:1c.2g.10gb`
for more clarity, at the expense of more typing.
For a general profile that allows N GPU slices for an instance (with N ≤ 7), the mapping for the colon syntax takes the form:
| **compact colon form** | **expanded form** |
| :--------------------- | :--------------------------------- |
| `:0` | `1c.` \<*profile*> |
| `:1` | `2c.` \<*profile*> |
| ... | ... |
| `:N-1` | `Nc.` \<*profile*> or \<*profile*> |
In practice, there are hardware-based restrictions for what is permitted to be allocated. So for example on the NVIDIA A100-PCIE-40GB:
`7g.40gb:3` is allowed but
`7g.40gb:4` is not.
Details on supported profiles for hardware can be found in the NVIDIA MIG user guide documentation at [https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html), which covers, for example, the [A30](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html#a30-mig-profiles), [A100](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html#a100-mig-profiles), [H100](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html#h100-mig-profiles), and [B200](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html#B200-mig-profiles) profiles.
Increasing the number of GPU instances can also still be done using the earlier \<*number*>`*` prefix syntax together with the colon syntax, if the GPU allows it:
**Example**
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 3*2g.10gb:0:0; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device[node001]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- ----------- -------- -------- --------
node001 0 3 - 2g.10gb 14 0 2
node001 0 3 0 1c.2g.10gb 0 0 1
node001 0 3 1 1c.2g.10gb 0 1 1
node001 0 4 - 2g.10gb 14 2 2
node001 0 4 0 1c.2g.10gb 0 0 1
node001 0 4 1 1c.2g.10gb 0 1 1
node001 0 5 - 2g.10gb 14 4 2
node001 0 5 0 1c.2g.10gb 0 0 1
node001 0 5 1 1c.2g.10gb 0 1 1
```
* **A heterogeneous set of existing profiles** for GPU instances can also be defined for the GPU with MIG profiles.
For example, 2 instances with the MIG profile `1g.5gb`, and 1 instance with the MIG profile `2g.10gb` can be specified with:
```bash
[basecm11->device[node001]]% gpusettings; use nvidia:0
[basecm11->device[node001]->gpusettings[nvidia:0]]% set migprofiles 2*1g.5gb 2g.10gb; commit
[basecm11->device[node001]->gpusettings[nvidia:0]]% show
Parameter Value
-------------------------------- ------------------------------------------------
...
MIG profiles 2*1g.5gb,2g.10gb
[basecm11->device[node001]->gpusettings[nvidia:0]]% ..;..
[basecm11->device*[node001*]]% mig apply
Node GPU MIG Instance Name Profile Start Size
---------------- -------- -------- -------- -------- -------- -------- --------
node001 0 3 - 2g.10gb 14 0 2
node001 0 3 0 2g.10gb 1 0 2
node001 0 11 - 1g.5gb 19 4 1
node001 0 11 0 1g.5gb 0 0 1
node001 0 13 - 1g.5gb 19 6 1
node001 0 13 0 1g.5gb 0 0 1
```
* **heterogeneous sets are useful when trying to use up all the slices available**, to make the maximum resources available. So, while an administrator can carry out the preceding specification:
`2*1g.5gb 2g.10gb`
this is not a good allocation of resources since it only makes 4/7 of the GPU slices available, and 4/8 of the memory slices available. An administrator would more sensibly specify something like, for example:
`5*1g.5gb,2g.10gb`
which uses up the full 7 GPU slices and 35GB (6/8 slices) of memory available from the physical GPU. This makes full use of the SM resources derived from the physical GPU, so that these SM resources are fully available to workloads.
* **Overallocating slices for MIG configuration is not possible**. If there is an attempt to overallocate, then the slices that are allocated too late are simply not allocated. This can lead to unexpected results for the unwary cluster administrator. For example:
* `5*1g.5gb,2g.10gb` allocates the 5 slices of the 1g instance and the 2 slices of the 2g instance.
But
* `6*1g.5gb,2g.10gb` allocates the 6 slices of the 1g instance and none of the 2g instance.
* `7*1g.5gb,2g.10gb` allocates the 7 slices of the 1g instance and none of the 2g instance.
* `70*1g.5gb,2g.10gb` allocates 7 slices of the 1g instance and none of the 2g instance.
## 3.18 Configuring Sampling From A Prometheus Exporter
CMDaemon can be configured to sample a Prometheus exporter, for example from the NVIDIA Unified Fabric Manager (UFM) platform. A CMDaemon front end such as `cmsh` can have a data producer, for example `UFM`, configured within the `monitoring setup` mode (section [10.5.4](/admin/monitoring#1054-the-setup-submode)) to allow sampling of the Prometheus exporter:
**Example**
```bash
[root@basecm11]# cmsh
[basecm11]% monitoring setup
[basecm11->monitoring->setup]% add prometheus UFM #create a data producer of type Prometheus
[basecm11->monitoring->setup*[UFM*]]% set urls http://10.180.217.170:9001/metrics #end point
[basecm11->monitoring->setup*[UFM*]]% set -e NoPostAllowed yes #HTTP GET only, use for older exporters
[basecm11->monitoring->setup*[UFM*]]% nodeexecutionfilters
[basecm11->monitoring->setup*[UFM*]->nodeexecutionfilters]% active #run on active head node only
Added active resource filter
[basecm11->monitoring->setup*[UFM*]->nodeexecutionfilters]% commit
```
In the example, the URL needs to be set to the Prometheus export server endpoint. The value of `NoPostAllowed` only needs to be set to `yes` for some older Prometheus versions that do not work with HTTP POST. The data producer is set to run on only the active head node with the `nodeexecutionfilter` setting ([here](/admin/monitoring#10545-the-setup-submode-deeper-submodes)).
The preceding example configures CMDaemon to sample from a Prometheus exporter. The other way around, that is to have CMDaemon be the exporter of Prometheus data, can be achieved via the `EnablePrometheusExporterService` directive ([here](/admin/cmdconf#c43-enableprometheusexporterservice-directive)).
## 3.19 Configuring Custom Scripts
Some scripts are used for custom purposes. These are used as replacements for certain default scripts, for example, in the case of non-standard hardware where the default script does not do what is expected. The custom scripts that can be set, along with their associated arguments are:
* `custompowerscript` and `custompowerscriptargument` (section [4.1.4](/admin/power#414-custom-power-control))
* `custompingscript` and `custompingscriptargument` (section [3.19.2](#3192-custompingscript))
* `customremoteconsolescript` and `customremoteconsolescriptargument` (section [3.19.3](#3193-customremoteconsolescript))
In addition to the preceding `custom*` scripts, system information scripts can be set that provide extra information to the `sysinfo` command in BCM (section [3.19.4](#3194-sysinfo-custom-scripts)).
The environment variables of CMDaemon ([section 3.3.1 of the *Developer Manual*](/developer/monitoring-actions#331-standard-environment-variables-available-in-action-scripts)) can be used in the scripts. Successful scripts, as is the norm, return 0 on exit.
### 3.19.1 `custompowerscript`
The use of custom power scripts is described in section [4.1.4](/admin/power#414-custom-power-control).
### 3.19.2 `custompingscript`
The following example script:
**Example**
```bash
#!/bin/bash
/bin/ping -c1 $CMD_IP
```
can be defined and set for the cases where the default built-in ping script, cannot be used.
By default, the node device states are detected by the built-in ping script (section [5.5](/admin/provisioning#55-node-states)) using ICMP ping. This results in the statuses that can be seen on running the list command of `cmsh` in device mode. An example output, formatted for convenience, is:
**Example**
```bash
[root@basecm11]# cmsh -c "device; format hostname:15, status:15; list"
hostname (key) status
--------------- --------------
basecm11 [ UP ]
node001 [ UP ]
node002 [ UP ]
```
If some device is added to the cluster that blocks such pings, then the built-in ping can be replaced by the custom ping of the example, which relies on standard ICMP ping.
However, the replacement custom ping script need not actually use a variety of ping at all. It could be a script running web commands to query a chassis controller, asking if all its devices are up. The script simply has to provide an exit status compatible with expected ping behavior. Thus an exit status of 0 means all the devices are indeed up.
### 3.19.3 `customremoteconsolescript`
A custom remote console script can be used to run in the built-in remote console utility. This might be used, for example, to allow the administrator remote console access through a proprietary KVM switch client.
For example, a user may want to run a KVM console access script that is on the head node and with an absolute path on the head node of `/root/kvmaccesshack`. The script is to run on the console, and intended to be used for node `node001`, and takes the argument `1`. This can then be set in `cmsh` as follows:
**Example**
```bash
[root@basecm11]# cmsh
[basecm11]% device use node001
[basecm11->device[node001]]% get customremoteconsolescript; get customremoteconsolescriptargument
[basecm11->device[node001]]% set customremoteconsolescript /root/kvmaccesshack
[basecm11->device[node001]]% set customremoteconsolescriptargument 1
[basecm11->device[node001]]% rconsole
KVM console access session using the 1 argument option is displayed
```
In Base View, the corresponding navigation paths to access these script settings are:\
`Devices` > `Nodes` > `Edit` > `Settings` > `Custom remote console script`\
and\
`Devices` > `Nodes` > `Edit` > `Settings` > `Custom remote console script argument`\
while the remote console can be launched via the navigation path:\
`Devices` > `Nodes` > `Edit` > `Connect` > `Remote console`
### 3.19.4 `sysinfo` Custom Scripts
#### 3.19.4.1 Standard `sysinfo`
The `sysinfo` command in BCM is run from `device` mode in `cmsh` for a node. By default, `sysinfo` returns some basic hardware information for the node, from the OS, from a cache. Direct hardware queries can be carried out with other commands (section [14.11](/admin/administration#1411-querying-hardware-information)).
#### 3.19.4.2 Overview Of Running Custom Scripts In `sysinfo`
A cluster administrator may however wish to extract some additional hardware-related information from the cluster. To do this, custom scripts associated with the `sysinfo` command can be created by the cluster administrator. These `sysinfo` custom scripts run when the `sysinfo` command is executed via `cmsh`, and they pick up the additional information.
**Custom script types:** The scripts can be of three types, with corresponding directory locations. as indicated by the following table:
| **Type of script** | **Script directory path on node** |
|---|---|
| local | `/cm/local/apps/cmd/scripts/sysinfo/local/` |
| director | `/cm/local/apps/cmd/scripts/sysinfo/director/` |
| head | `/cm/local/apps/cmd/scripts/sysinfo/head/` |