Configuring The Cluster

After the NVIDIA Base Command Manager software has been installed on the head node, the cluster must be configured. For convenience, the regular nodes on the cluster use a default software image stored on the head node. The image is supplied to the regular nodes during a process called provisioning (Chapter 5), and both the head node and the regular nodes can have their software modified to suit exact requirements (Chapter 9). This chapter however goes through a number of basic cluster configuration aspects that are required to get all the hardware up and running on the head and regular nodes.

Section 3.1 explains how some of the main cluster configuration settings can be changed.

Section 3.2 details how the internal and external network parameters of the cluster can be changed.

Section 3.3 describes the setting up of network bridge interfaces.

Section 3.4 describes VLAN configuration.

Section 3.5 describes the setting up of network bond interfaces.

Section 3.6 covers how InfiniBand is set up.

Section 3.7 describes how Baseboard Management Controllers such as IPMI, iLO, DRAC, CIMC, and Redfish are set up.

Section 3.8 describes how switches are set up.

Section 3.9 describes how Cumulus switches are configured.

Section 3.11 describes how NetQ can be integrated with BCM.

Section 3.12 describes how UFM can be integrated with BCM.

Section 3.13 explains how disk layouts are configured, as well as how diskless nodes are set up.

Section 3.14 describes how NFS volumes are exported from an NFS server and mounted to nodes using the integrated approach of BCM.

Section 3.15 describes how services can be run from BCM.

Section 3.16 describes how a rack can be configured and managed with BCM.

Section 3.17 describes how GPUs can be configured with BCM.

Section 3.18 describes how sampling from a Prometheus exporter can be configured with BCM.

Section 3.19 describes how custom scripts can replace some of the default scripts in use.

Section 3.20 discusses configuration alternatives that are not based on CMDaemon.

Section 3.21 describes how the configuration files prior to a configuration change can be saved.

More elaborate aspects of cluster configuration such as power management, user management, package management, and workload management are covered in later chapters.

3.1 Main Cluster Configuration Settings

While both front ends—cmsh and Base View—can be used to carry out cluster management (Chapter 2), the BCM manuals often describe configuration with an arbitrary front end rather than for both front ends.

This is because the front ends are usually analogous enough to each other when carrying out a configuration procedure, so that describing the procedure for the other front end in detail as well is mostly wasteful. If the procedures differ significantly, then guidance is typically given on the differences.

Thus, for example, both cmsh and Base View can be used for the global configuration of cluster settings. For the cmsh front end, the configuration is done using partition mode. The analog to global configuration in the Base View front end relies on the navigation path: Cluster > Settings.

This section now continues with the Base View description.

The navigation path: Cluster > Settings brings up Base View’s cluster settings window (figure 3.1):

Figure 3.1: Cluster Settings

The navigational overview (here) indicates how the main cluster Settings window fits into the organizational layout of BCM.

The cluster Settings tab window allows changes to be made to many of the global cluster settings, typically using a grouping configuration that corresponds to the partition-level configuration in cmsh. The values that can be set for these global settings can in some cases be overridden by more specific configuration levels, such as (configuration-)overlay-level, category-level, or node-level configuration. The main cluster settings in figure 3.1 are about name-related settings, cluster networking settings, and some other miscellaneous global cluster settings.

3.1.1 Cluster Configuration: Various Cluster-wide Settings

In the Base View cluster Settings tab window of figure 3.1, the following cluster-wide settings can be viewed and modified:

• Cluster name: Default value: BCM 11 Cluster.

• Administrator email: Default value: Specified during installation (figure 3.8 in Chapter 3 of the Installation Manual).

• Name servers: No default value, or picked up using DHCP.

• Time servers: Default values: 0.pool.ntp.org, 1.pool.ntp.org, 2.pool.ntp.org.

• Search domains: No default value.

• Externally visible IP: Default value: 0.0.0.0.

• Time zone: Default value: Specified during installation (figure 3.8 in Chapter 3 of the Installation Manual).

• BMC settings: This allows BMC and related settings to be managed.

  The following objects already exist by default on a new cluster, and can have their settings modified:

  • BMC (section 3.1.3.1)

  • SNMP (section 3.8.2)

  • Provisioning (Chapter 5, section 5.2.4)

  The following objects do not already exist by default on a new cluster, which means that clicking on them to configure them begins the process of creating the object, prompting the administrator to enter values for the object:

  • SELinux (Chapter 12 of the Installation Manual)

  • Access (section 3.9.1)

  • ZTP (section 3.9)

  • ZTP new switch

  • NetQ (section 3.11)

  • UFM (section 3.12)

  • NMX Manager (section 2.3 of the NVIDIA Mission Control Manual)

  The BMC and related settings are described in their associated sections.

• External network: Default value: as defined during selection of network topology during installation. The Management network can also be configured in this section.

  • Management network: Default value: as defined during selection of network topology during installation.

  Network topology selection during installation is covered in section 3.3.9 of the Installation Manual.

• Default category: Default value: default. This value is inherited as the default object name for category mode.

• ArchOS: No default value. This specifies an architecture and OS.

• Sign installer certificates: Default value: AUTO. Described in section 5.4.1.

• Failover: No default value.

• Failover groups: No default value.

• Default burn configuration: Default value: default-destructive.

• Wlm job power usage settings: No default value.

• Leak action policies: Default value: Default building policy.

• Building Management System: Default value: NVIDIA-conforming BMS.

• Prometheus metric forwarders: No default value.

3.1.2 Cluster Configuration: Some Network-Related Settings

The following network-related settings are also described in the context of external network settings for the cluster object, in section 3.2.3, as well as in the quickstart in Chapter 1 of the Installation Manual.

3.1.2.1 Nameserver And Search Domains Used By Cluster

• If Base View is used, then the settings window can be used to set the IP address of the nameserver and the names of the search domains for the cluster.

  By default, the nameserver is the internal (regular-nodes-facing) IP address of the head node. Multiple nameservers can be added. If the value is set to 0.0.0.0, then the address supplied via DHCP to the head node is used. Alternatively, a static value can be set. Static IP addresses must be used for external addresses in the case of the cluster being configured with high availability.

• If cmsh is used instead of Base View, then the changes to the nameserver and searchdomain values can instead be carried out via partition mode (here).

Limit to the number of search domains: In older versions of glibc (before 2.26), the number of names that can be set as search domains used by the cluster has a hardcoded maximum limit of 6, with a total of 256 characters.

From glibc 2.26 onward, this operating system limit no longer applies to normal lookups. BCM applies its own limit instead, using the MaximalSearchDomains GlobalConfig directive (Appendix C, here), which defaults to 64 search domains, and which can be set to any value from 1 to 1024.

Raising MaximalSearchDomains above 6 only has an effect on a cluster running glibc 2.26 or later. On an older glibc, only the first 6 configured search domains are used, regardless of the directive’s setting.

3.1.2.2 Changing The Order In resolv.conf

For clusters, CMDaemon by default automatically writes the /etc/resolv.conf by using the following sources, and in the following order:

1. Global network

2. Other networks

3. Category search domains

4. Partition search domains

Because older glibc versions only support 6 entries in /etc/resolv.conf, it is sometimes useful to exclude or reorder the preceding sources.

For a network object, there are two fields that control the position of the domain name in the file /etc/resolv.conf:

Example

[basecm11]% network use ibnet
[basecm11->network[ibnet]]% show
...
Exclude from search domain       no
Search domain index              0

If the Exclude from search domain field is set to yes, then the domain name for the network is not used.

The Search domain index field specifies the position of the domain name. A value of 0 means CMDaemon automatically determines its location.

The index of the category and partition search domains can also be changed by appending a number, in the suffix format : <index>, to the domain name:

Example

[basecm11]% partition
[basecm11->partition[base]]% get searchdomains
example.com:1
domain.test:6

If an index is set for one search domain, then setting indices for all search domains is recommended. Search domains without indices are handled automatically by CMDaemon.

CMDaemon sorts all search domains according to index, and writes /etc/resolv.conf with the 6 that have the lowest index, with the lowest index first.

3.1.2.3 Setting The Stub Resolver For Ubuntu Hosts

For Ubuntu, CMDaemon generates a /etc/resolv.conf file (Appendix A), and symlinks it to the resolv.conf managed by systemd at /etc/run/systemd/resolve/resolve.conf.

However, a default Ubuntu without BCM uses a systemd stub resolver service (man systemd-resolved.8) that listens on the IP address 127.0.0.53. Sometimes a stub resolver is also useful in BCM for containers such as enroot containers (section 7.3.3) that mount and use /etc/resolv.conf.

To enable resolution for such containers on a node via the stub resolver instead of the uplink resolver, the resolver can be set using the extra resolve parameter, using the -e|--extra setting:

Example

basecm11->device[node001]]% # to set it to the stub resolver:
basecm11->device[node001]]% set -e resolv stub
basecm11->device*[node001*]]% commit        # wait a bit
basecm11->device[node001]]% # to set it back to the BCM value:
basecm11->device[node001]]% set -e resolv uplink
basecm11->device*[node001*]]% commit       # wait a bit

3.1.2.4 Externally Visible IP Address

The externally visible IP address are public (non-RFC 1918) IP addresses to the cluster. These can be set to be visible on the external network.

• If using Base View, the navigation path is:
  Cluster > Settings > Externally visible IP.

• For cmsh, the parameter externallyvisibleip can be set via partition mode.

3.1.2.5 Time server(s)

Time server hostnames can be specified for the NTP client on the head node.

• If using Base View, the navigation path is via Cluster > Settings > NAME SERVERS.

• For cmsh, the parameter timeservers can be set via partition mode.

3.1.2.6 Time Zone

The time zone setting can be set at various grouping levels:

If applied to the entire cluster, and if it is applied in partition mode, then:

• In Base View, the time zone parameters can be jumped to via the navigation path:
  Cluster > Settings > Time zone

• In cmsh, the time zone can be selected in partition mode, using the base object. Tab-completion prompting after entering “set timezone” displays a list of several hundred possible time zones, from which one can be chosen:

  Example

  [basecm11]% partition use base
  [basecm11->partition[base]]% set timezone america/los_angeles
  [basecm11->partition*[base*]]% commit

A time zone setting can also be applied at the level of a node, category, edge site, and cloud region. As is usual in the BCM hierarchy, the value set for the larger grouping is the default value used by the members of that group, while a value set specifically for the individual members of that group overrides such a default.

3.1.3 Miscellaneous Settings

3.1.3.1 BMC (IPMI/iLO, DRAC, CIMC, Redfish) Settings

This opens up a window so that the BMC settings can be managed:

• User name: (default: bright)

• Password: (default: random string generated during head node installation)

• User ID: (default: 4)

• Power reset delay: During a reset, this is the time, in seconds, that the machine is off, before it starts powering up again (default: 0)

• Extra arguments: (default: none)

• privilege: (default: administrator)

The defaults in the preceding are set when the BMC interfaces are configured during head node installation. If the BMC interfaces are not set then, then the defaults are also unset.

BMC configuration is discussed in more detail in section 3.7.

3.1.3.2 Administrator E-mail Setting

By default, the distribution which BCM runs on sends e-mails for the administrator to the root e-mail address. The administrator e-mail address can be changed within BCM so that these e-mails are received elsewhere.

• In Base View, an e-mail address (or space-separated addresses) can be set in the Administrator e-mail field via the navigation path Cluster > Settings > Administrator e-mail

• In cmsh, the e-mail address (or space-separated addresses) can be set in partition mode, using the base object as follows:

  Example

  [basecm11]% partition use base
  [basecm11->partition[base]]% set administratore-mail alf@example.com beth@example.com
  [basecm11->partition*[base*]]% commit

The following critical states or errors cause e-mails to be sent to the e-mail address:

• By default, a month before the cluster license expires, a reminder e-mail is sent to the administrator account by CMDaemon. A daily reminder is sent if the expiry is due within a week.

• A service on the head node that fails on its first ever boot.

• When an automatic failover fails on the head or regular node.

3.1.3.3 SMTP Relay Host Mailserver Setting

The head node uses Postfix as its SMTP server. The default base distribution configuration is a minimal Postfix installation, and so has no value set for the SMTP relay host. To set its value:

• in Base View: the Relay Host field, with the navigation path:

  Cluster > Settings > Search domains > Relay Host

  sets the SMTP relay host for the cluster resource.

• in cmsh: the relayhost property can be set for the base object within partition mode:

  Example

  [root@basecm11 ~]# cmsh
  [basecm11]% partition use base
  [basecm11-> partition[base]]% set relayhost mail.example.com
  [basecm11-> partition[base*]]% commit

Postfix on the regular nodes is configured to use the head node as a relay host and is normally left untouched by the administrator.

If the regular node configuration for Postfix is changed due to a partition mode change, then a node reboot deploys the change for the node. Setting the AdvancedConfig (here) SmtpPartitionRelayHostInImages to 0 disables the changing of the relay host on the regular node.

Further Postfix changes can be done directly to the configuration files as is done in the standard distribution. The changes must be done after the marked auto-generated sections, and should not conflict with the auto-generated sections.

A convenient way to check mail is functioning is to run BCM’s testemail command. The command is run from within the main mode of cmsh. It sends a test e-mail out using CMDaemon:

[root@basecm11 ~]# mailq; ls -al /var/spool/mail/root
Mail queue is empty
-rw------- 1 root mail 0 Sep  8 11:11 /var/spool/mail/root
[root@basecm11 ~]# cmsh -c "main; testemail"
Mail delivered to postfix
You have new mail in /var/spool/mail/root
[root@basecm11 ~]# mailq; ls -al /var/spool/mail/root
Mail queue is empty
-rw------- 1 root mail 749 Sep  8 11:12 /var/spool/mail/root

The test e-mail destination is the administrator e-mail address discussed in the preceding section.

3.1.3.4 Failover Settings

To access options to the high availability (HA) feature of the cluster for head nodes, the administrator can click on the Failover option in Base View. This opens up a subwindow that can be used to configure HA-related parameters (section 15.4.6).

3.1.3.5 Failover Groups Settings

To access options to the high availability feature of the cluster for groups of regular nodes, the administrator can click on the Failover groups option in Base View. This opens up a subwindow that can be used to configure failover-groups-related parameters (section 15.5).

3.1.3.6 Burn Configs

Burning nodes is covered in Chapter 11 of the Installation Manual. Burn configuration settings for the cluster can be accessed in Base View via the following navigation paths:

• Cluster > Settings > Default burn configuration

  This allows the Default burn configuration for a node burn run to be modified.

• Cluster > Settings > Burn configs

  This lists the possible burn configuration settings in a subwindow, and allows changes to some of their properties of each item of the list.

3.1.3.7 FIPS Mode

To be compliant with the Federal Information Processing Standards of the USA, Linux-based systems are required to follow some security standards. This is known as FIPS compliance.

FIPS compliance on regular nodes can be set per node or per category.

The existing FIPS status can be checked with the sysinfo command at node level:

Example

[head->device]% sysinfo node002
Name                      Value
------------------------- ---------------------------------
...
FIPS                      No

Or even just with a get:

Example

[head->device]% get node002 fips
no

FIPS can be enabled per node:

Example

[head->device]% set node002 fips yes
[head->device*]% commit
[head->device]% reboot node002
        [time passes]
        [node002 finishes reboot]
[head->device]% sysinfo node002 | grep FIPS
FIPS                      Yes

FIPS can be enabled per category:

Example

[head->device]% sysinfo node002
[head->device]% category
[head->category]% set default fips yes
[head->category*]% commit
Successfully committed 1 Categories
[head->category]%
...[notice] head: node001 [   UP   ], restart required (fips)
...[notice] head: node002 [   UP   ], restart required (fips)
[head->category]% device reboot -n node001..node002
[head->category]% device
[head->device]% get node001 fips; get node002 fips
yes
yes

As is usual in BCM, the node-level setting overwrites the category-level setting.

Setting FIPS compliance on the head node itself via CMDaemon is not allowed. This is because FIPS GRUB configuration could, in some unusual cases, result in an unbootable head node. The cluster administrator must consider the existing state of the head node with due care before manually re-configuring it for FIPS.

3.1.4 Limiting The Maximum Number Of Open Files

3.1.4.1 Configuring The System Limit On Open Files: The /proc/sys/fs/file-max Setting

The maximum number of open files allowed on a running Linux operating system is determined by /proc/sys/fs/file-max. To configure this setting so that it is persistent, the Linux operating system uses a /etc/sysctl.conf file and *.conf files under /etc/sysctl.d/. Further information on these files can be found via the man page, man sysctl.conf.5. BCM adheres to this standard method, and places a settings file 90-cm-sysctl.conf in the directory /etc/sysctl.d.

By default, the value set for file-max by NVIDIA Base Command Manager is 131072. An on-premises head node typically is not used to run applications that exceed this value, and the settings file 90-cm-sysctl.conf is therefore not available for modification by the CMDaemon front ends (cmsh and Base View). For edge and cloud director installations, the installation scripts take care of modifying file-max since such installations can have many more open files. Cluster administrators are thus not expected to have a need to modify the value of file-max in most use cases.

However, if there is a need to modify the value of file-max, then a subsequent, extra, configuration file, such as /etc/sysctl.d/91-site-sysctl-file-max-tweak.conf can be added under /etc/sysctl.d/, so that the BCM version is not intefered with directly.

3.1.4.2 Configuring The User Limit On Open Files: The nofile Setting

The maximum number of open files allowed for a user can be seen on running ulimit -n. The value is defined by the nofile parameter of ulimit.

By default the value set by BCM is 131072.

Ulimit limits are limits to restrict the resources used by users. If the pam_limits.so module is used to apply ulimit limits, then the resource limits can be set via the /etc/security/limits.conf file and *.conf files in the /etc/security/limits.d directory. Further information on these files can be found via the man page, man limits.conf.5.

Resource limits that can be set for user login sessions include the number of simultaneous login sessions, the number of open files, and memory limits.

The maximum number of open files for a user is unlimited by default for an operating system that is not managed by BCM. However, it is set to 131072 by default for a system managed by BCM. The nofile value is defined by BCM in:

• in /etc/security/limits.d/91-cm-limits.conf on the head node

• in /cm/images/ <software image name>/etc/security/limits.d/91-cm-limits.conf in the software image that the regular node picks up.

The values set in 91-cm-limits.conf are typically sufficient for a user session, unless the user runs applications that are resource hungry and consume a lot of open files.

3.1.4.3 Deciding On Appropriate Ulimit, Limit, And System Limit Values

Decreasing the nofile value in /etc/security/limits.d/91-cm-limits.conf (but leaving the /proc/sys/fs/file-max untouched), or increasing /proc/sys/fs/file-max (but leaving the nofile value of 131072 per session as is), may help the system stay under the maximum number of open files allowed.

In general, users should not be allowed to use the head node as a compilation server, or as a test bed, before running their applications. This is because user errors can unintentionally cause the head node to run out of resources and crash it.

Depending what is running on the server, and the load on it, the administrator may wish to increase the resource limit values.

A very rough rule-of-thumb that may be useful as a first approximation to set file-max optimally is suggested in the kernel source code. The suggestion is to simply multiply the system memory (in MB) by 10 per MB, and make the resulting number the file-max value. For example, if the node has 128 GB of memory, then 1280000 can be set as the file-max value.

Fine-tuning to try and ensure that the operating system no longer runs out of file handles, and to try and ensure the memory limits for handling the load are not exceeded, is best achieved via an empirical trial-and-error approach.

3.2 Network Settings

A simplified quickstart guide to setting up the external head node network configuration on a vendor-prepared cluster is given in Chapter 6 of the Installation Manual. This section (3.2) covers network configuration more thoroughly.

After the cluster is set up with the correct license as explained in Chapter 4 of the Installation Manual, the next configuration step is to define the networks that are present (sections 3.2.1 and 3.2.2).

During BCM installation at least three default network objects were created:

internalnet: the primary internal cluster network, and the default management network. This is used for booting non-head nodes and for all cluster management communication. In the absence of other internal networks, internalnet is also used for storage and communication between compute jobs. Changing the configuration of this network is described here under the subheading “Changing Internal Network Parameters For The Cluster”.

externalnet: the network connecting the cluster to the outside world (typically a corporate or campus network). Changing the configuration of this network is described here under the subheading “Changing External Network Parameters For The Cluster”. This is the only network for which BCM also supports IPv6.

globalnet: the special network used to set the domain name for nodes so that they can be resolved whether they are cloud nodes or not. This network is described further here under the subheading “Changing The Global Network Parameters For The Cluster”.

For a Type 1 cluster (section 3.3.9 of the Installation Manual) the internal and external networks are illustrated conceptually by figure 3.2.

Figure 3.2: Network Settings Concepts

The configuration of network settings is completed when, after having configured the general network settings, specific IP addresses are then also assigned to the interfaces of devices connected to the networks.

• Changing the configuration of the head node external interface is described here under the subheading “The IP address of the cluster”.

• Changing the configuration of the internal network interfaces is described here under the subheading “The IP addresses and other interface values of the internal network”.

  • How to set a persistent identity for an interface—for example, to ensure that a particular interface that has been assigned the name eth3 by the kernel keeps this name across reboots—is covered in section 5.8.1, here.

• Changing the configuration of globalnet is described here under the subheading “Changing The Global Network Parameters For The Cluster”. IP addresses are not configured at the globalnet network level itself.

3.2.1 Configuring Networks

The network mode in cmsh gives access to all network-related operations using the standard object commands. Section 2.5.3 introduces cmsh modes and working with objects.

In Base View, a network can be configured via the navigation path Networking > Networks, which opens up the Network list subwindow (figure 3.3):

Figure 3.3: Networks

In the context of the OSI Reference Model, each network object represents a layer 3 (i.e. Network Layer) IP network, and several layer 3 networks can be layered on a single layer 2 network (e.g. routes on an Ethernet segment).

Opening up the subwindow for a network such as internalnet or externalnet brings up the Devices Overview tab window, which lists the devices on the network and associated properties. For example, the internal network typically has some nodes and other devices attached to it, each with their IP addresses and statuses (figure 3.4).

Figure 3.4: Network Overview

Selecting the Settings tab window opens a scrollable pane that allows a number of network properties to be changed (figure 3.5).

Figure 3.5: Network Settings

The properties of figure 3.5 are introduced in table 3.1:

Table 3.1: Network Configuration Settings

3.2.1.1 Basic Networking Concepts And The sipcalc Utility

In basic networking concepts, a network is a range of IP addresses. The first address in the range is the base address. The length of the range, i.e. the subnet, is determined by the netmask, which uses CIDR notation. CIDR notation is the so-called / (“slash”) representation, in which, for example, a CIDR notation of 192.168.0.1/28 implies an IP address of 192.168.0.1 with a traditional netmask of 255.255.255.240 applied to the 192.168.0.0 network. The netmask 255.255.255.240 implies that bits 28–32 of the 32-bit dotted-quad number 255.255.255.255 are unmasked, thereby implying a 4-bit-sized host range of 16 (i.e. 2[4]) addresses.

The sipcalc utility can be installed on the head node, and is a useful tool for calculating or checking such IP subnet values (man sipcalc.1 or sipcalc -h for help on this utility):

Example

user@basecm11:~$ sipcalc 192.168.0.1/28
-[ipv4 : 192.168.0.1/28] - 0
[CIDR]
Host address            - 192.168.0.1
Host address (decimal)  - 3232235521
Host address (hex)      - C0A80001
Network address         - 192.168.0.0
Network mask            - 255.255.255.240
Network mask (bits)     - 28
Network mask (hex)      - FFFFFFF0
Broadcast address       - 192.168.0.15
Cisco wildcard          - 0.0.0.15
Addresses in network    - 16
Network range           - 192.168.0.0 - 192.168.0.15
Usable range            - 192.168.0.1 - 192.168.0.14

3.2.1.2 DNS Domains And Zones

Every network has an associated DNS domain which can be used to access a device through a particular network. For internalnet, the default DNS domain is set to eth.cluster, which means that the hostname node001.eth.cluster can be used to access device node001 through the primary internal network. If a dedicated storage network has been added with DNS domain storage.cluster, then node001.storage.cluster can be used to reach node001 through the storage network. Internal DNS zones are generated automatically based on the network definitions and the defined nodes on these networks. For networks marked as external, no DNS zones are generated.

3.2.1.3 Layer 3 Properties

Setting the layer 3 properties of table 3.1 with set layer3 yes configures the nodes in the (main) network range to be routed by a switch with layer 3 features. The default netmask when layer3 is set to yes is /31 (255.255.255.254) for each device and switch port pair in the subnets that are created by this setting, while routing stays defined by the netmask of the (main) network, and the routing is handled by the layer 3 features of the switch.

The netmask for layer 3 networking can be set to /30 with set -e layer3bits 30. This is needed for some DataDirect Networks switches.

3.2.2 Adding Networks

Once a network has been added, it can be used in the configuration of network interfaces for devices.

In Base View the Add button in the Networks subwindow (figure 3.3) can be used to add a new network. After the new network has been added, the Settings pane (figure 3.5) can be used to further configure the newly-added network.

In cmsh, a new network can be added from within network mode using the add or clone commands.

The default assignment of networks (internalnet to Management network and externalnet to External network) can be changed using Base View, via the Settings window of the cluster object, which uses the navigation path:

Cluster > Settings > External network > {External network|Management network}

In cmsh the assignment to Management network and External network can be set or modified from the base object in partition mode:

Example

[root@basecm11 ~]# cmsh
[basecm11]% partition use base
[basecm11->partition[base]]% set managementnetwork internalnet; commit
[basecm11->partition[base]]% set externalnetwork externalnet; commit

3.2.3 Changing Network Parameters

After both internal and external networks are defined, it may be necessary to change network-related parameters from their original or default installation settings.

3.2.3.1 Changing The Head Or Regular Node Hostname

To reach the head node from inside the cluster, the alias master may be used at all times. Setting the hostname of the head node itself to master is not recommended.

The name of a cluster is sometimes used as the hostname of the head node. The cluster name, the head node hostname, and the regular node hostnames, all have their own separate names as a property of their corresponding objects. The name can be changed in a similar manner for each.

For example, to change the hostname of the head node, the device object corresponding to the head node must be modified.

• In Base View, the navigation path for modifying the host name is Devices > All Devices > Head Node > > Settings > Hostname > Commit HeadNode. That is, under the Devices resource, the All Devices option is selected. The head node is selected from the devices displayed, and its subwindow-opening box in the Actions column is then clicked. This opens the device overview pane. The Settings option can then be selected. The Hostname record can then be modified (figure 3.6), and saved by clicking on the Commit HeadNode icon. When setting a hostname, a domain is not included.

  After the change, as suggested by Base View, the head node must be rebooted.

Figure 3.6: Head Node Settings

• In cmsh, the hostname of the head node can also be changed, via device mode:

  Example

  [root@basecm11 ~]# cmsh
  [basecm11]% device use basecm11
  [basecm11->device[basecm11]]% set hostname foobar
  [foobar->device*[foobar*]]% commit
  [foobar->device[foobar]]%
  Tue Jan 22 17:35:29 2013 [warning] foobar: Reboot required: Hostname changed
  [foobar->device[foobar]]% quit
  [root@basecm11 ~]# sleep 30; hostname -f foobar.cm.cluster
  [root@basecm11 ~]#

  The prompt string shows the new hostname after a short time, when a new shell is started.

  After the hostname has been set, as suggested by cmsh, the head node must be rebooted.

3.2.3.2 Adding Hostnames To The Internal Network

Additional hostnames, whether they belong to the cluster nodes or not, can be added as name/value pairs to the /etc/hosts file(s) within the cluster. This should be done only outside the specially-marked CMDaemon-managed section. It can be done to the file on the head node, or to the file on the software image for the regular nodes, or to both, depending on where the resolution is required.

However, for hosts that are on the internal network of the cluster, such as regular nodes, it is easier and wiser to avoid adding additional hostnames via /etc/hosts.

Instead, it is recommended to let BCM manage host name resolution for devices on the internalnet through its DNS server on the internalnet interface. The host names can be added to the additionalhostnames object, from within interfaces submode for the head node. The interfaces submode is accessible from the device mode. Thus, for the head node, with eth1 as the interface for internalnet:

Example

[basecm11]% device use basecm11
[basecm11->device[basecm11]]% interfaces
[basecm11->device[basecm11]->interfaces]% use eth1
[basecm11->device[basecm11]->interfaces[eth1]]% set additionalhostnames test
[basecm11->device*[basecm11*]->interfaces*[eth1*]]% commit
[basecm11->device[basecm11]->interfaces[eth1]]%
Fri Oct 12 16:48:47 2012 [notice] basecm11: Service named was restarted
[basecm11->device[basecm11]->interfaces[eth1]]% !ping test
PING test.cm.cluster (10.141.255.254) 56(84) bytes of data.
...

Multiple hostnames can be added as space-separated entries.

The named service automatically restarts within about 20 seconds after committal, implementing the configuration changes. This kind of restart is a feature (section 3.15.1) of changes made to service configurations by Base View or cmsh.

3.2.3.3 Changing External Network Parameters For The Cluster

The external network parameters of the cluster: When a cluster interacts with an external network, such as a company or a university network, its connection behavior is determined by the settings of two objects: firstly, the external network settings of the Networks resource, and secondly, by the cluster network settings.

1. The external network object contains the network settings for all objects configured to connect to the external network, for example, a head node. Network settings are configured in Base View under the Networking resource, then under the Networks subwindow, then within the Settings option for a selected network. Figure 3.5 shows a settings window for when the internalnet item has been selected, but in the current case the externalnet item must be selected instead. The following parameters can then be configured:

   • the IP network parameters of the cluster (but not the IP address of the cluster):

     • Base address: the network address of the external network (the “IP address of the external network”). This is not to be confused with the IP address of the cluster, which is described shortly after this.

     • Broadcast address: the broadcast address of the external network. This should not be confused with the broadcast IP address of the internal network.

     • Dynamic range start and Dynamic range end: Not used by the external network configuration.

     • Netmask bits: the netmask size, or prefix-length, of the external network, in bits.

     • Gateway: the default route for the external network.

   • the network name (what the external network itself is called), by default this is defined as externalnet in the base partition on a newly installed Type 1 cluster,

   • the Domain Name: the network domain (LAN domain, i.e. what domain machines on the external network use as their domain),

   • the External network checkbox: this is checked for a Type 1 cluster,

   • and MTU size (the maximum value for a TCP/IP packet before it fragments on the external network—the default value is 1500).

2. The cluster object contains other network settings used to connect to the outside. These are configured in Base View using the navigation path:

   Cluster > Settings

   and are described in section 3.1.1:

   • e-mail address(es) for the cluster administrator,

   • any additional external name servers used by the cluster to resolve external host names. As an aside: by default, only the head node name server is configured, and by default it only serves hosts on the internal network via the internal interface. Enabling the PublicDNS directive (Appendix C.54) allows the head node name server to resolve queries about internal hosts from external hosts via any interface, including the external interface.

   • the DNS search domain (what the cluster uses as its domain),

   • and NTP time servers (used to synchronize the time on the cluster with standard time) and time zone settings.

   These settings can also be adjusted in cmsh in the base object under partition mode.

Changing the networking parameters of a cluster (apart from the IP address of the cluster) therefore requires making changes in the settings of the two preceding objects.

Adding custom DNS zones: BCM appends a BIND DNS include directive to the bottom of the head node name server’s generated configuration file, so that administrator-defined DNS zones are preserved across regeneration. The path of both files depends on the distribution:

• on Ubuntu:

  • configuration file /etc/bind/named.conf

  • include file /etc/bind/named.conf.include

• on RHEL 9/10 and SLES 15:

  • configuration file /etc/named.conf

  • include file /etc/named.conf.include

Custom zone definitions placed in the include file are not touched by BCM. The zone file itself can be placed in any directory the administrator chooses; it is not tied to the distribution-specific named.conf/include file paths in the preceding. For example:

Example

zone "runai.superpod-nvidia.local" {
    type master;
    file "<path-to-zone-directory>/runai.superpod-nvidia.local.zone";
};

These and other related BIND DNS files that are managed by CMDaemon on the head node are listed in Appendix A.1.

By default, the head node name server only resolves such zones for queries from the internal network. To resolve them from other networks, the PublicDNS directive (Appendix C.54) must also be set to true.

The IP address of the cluster: The cluster object itself does not contain an IP address value. This is because it is the cluster network topology type that determines whether a direct interface exists from the cluster to the outside world. Thus, the IP address of the cluster in the Type 1, Type 2, and Type 3 configurations (section 3.3.9 of the Installation Manual) is defined by the cluster interface that faces the outside world. For Type 1, this is the interface of the head node to the external network (figure 3.2). For Type 2 and Type 3 interfaces the cluster IP address is effectively that of an upstream router, and thus not a part of BCM configuration. Thus, logically, the IP address of the cluster is not a part of the cluster object or external network object configuration.

For a Type 1 cluster, the head node IP address can be set in BCM, separately from the cluster object settings. This is then the IP address of the cluster according to the outside world.

Setting the network parameters of the cluster and the head node IP address: These values can be set using Base View or cmsh:

• With Base View:

  The cluster network object and associated settings are accessed as follows:

  The external network object:

  The external network object is accessed via the navigation path:

  Networking > Networks > externalnet > > Settings

  which reaches the window shown in figure 3.7:

Figure 3.7: Network Settings For External Network

The cluster object:

The cluster object and associated settings are accessed using the navigation path:

Cluster > Settings

The head node IP address: For a head node basecm11, where the IP address is used by the interface eth0, the static IP address can be set via the navigation path:

as seen in figure 3.8.

Figure 3.8: Setting The IP Address On A Head Node In Base View

• With cmsh:

  The preceding Base View configuration can also be done in cmsh, using the network, partition and device modes, as in the following example:

  Example

  [basecm11]% network use externalnet
  [basecm11->network[externalnet]]% set baseaddress 192.168.1.0
  [basecm11->network*[externalnet*]]% set netmaskbits 24
  [basecm11->network*[externalnet*]]% set gateway 192.168.1.1
  [basecm11->network*[externalnet*]]% set gatewaymetric 1
  [basecm11->network*[externalnet*]]% commit
  [basecm11->network[externalnet]]% partition use base
  [basecm11->partition[base]]% set nameservers 192.168.1.1
  [basecm11->partition*[base*]]% set searchdomains x.com y.com
  [basecm11->partition*[base*]]% append timeservers ntp.x.com
  [basecm11->partition*[base*]]% commit
  [basecm11->partition[base]]% device use basecm11
  [basecm11->device[basecm11]]% interfaces
  [basecm11->device[basecm11]->interfaces]% use eth1
  [basecm11->device[basecm11]->interfaces[eth1]]% set ip 192.168.1.176
  [basecm11->device[basecm11]->interfaces*[eth1*]]% commit
  [basecm11->device[basecm11]->interfaces[eth1]]%

  After changing the external network configurations, a reboot of the head node is necessary to activate the changes.

Using DHCP to supply network values for the external interface:

Connecting the cluster via DHCP on the external network is not generally recommended for production clusters. This is because DHCP-related issues can complicate networking troubleshooting compared with using static assignments.

For a Type 1 network, the cluster and head node can be made to use some of the DHCP-supplied external network values as follows:

• In Base View, the DHCP setting of figure 3.8 can be set to Enabled

• Alternatively, in cmsh, within interfaces mode for the head node interface, the value of the parameter DHCP can be set:

  [basecm11->device[basecm11]->interfaces[eth0]]% set dhcp yes

The gateway address, the name server(s), and the external IP address of the head node are then obtained via a DHCP lease. Time server configuration for externalnet is not picked up from the DHCP server, having been set during installation (figure 3.8 in Chapter 3 of the Installation Manual). The time servers can be changed using Base View using the navigation path:

Cluster > Settings > Time servers

(figure 3.1), or using cmsh in partition mode as in the preceding example. The time zone can be changed similarly.

It is usually sensible to reboot after implementing these changes in order to test the changes are working as expected.

3.2.3.4 Changing Internal Network Parameters For The Cluster

When a cluster interacts with the internal network that the regular nodes and other devices are on, its connection behavior with the devices on that network is determined by settings in:

1. the internal network of the Networks resource (here)

2. the cluster network for the internal network (here)

3. the individual device network interface (here)

4. the node categories network-related items for the device (here), in the case of the device being a regular node.

In more detail:

1. The internal network object: has the network settings for all devices connecting to the internal network, for example, a login node, a head node via its internalnet interface, or a managed switch on the internal network. In Base View, the settings can be configured under the Networking resource, then under the Networks subwindow, then within the Settings option for the internalnet item (figure 3.5). In cmsh the settings can be configured by changing the values in the internalnet object within cmsh’s network mode. Parameters that can be changed include:

• the IP network parameters of the internal network (but not the internal IP address):

• the “domain name” of the network. This is the LAN domain, which is the domain machines on this network use as their domain. By default, set to a name based on the network interface type used on the internal network, for example eth.cluster. In any case, the FQDN must be unique for every node in the cluster.

• the network name, or what the internal network is called. By default, set to internalnet.

• The MTU size, or the maximum value for a TCP/IP packet before it fragments on this network. By default, set to 1500.

2. The cluster object: has other network settings that the internal network in the cluster uses. These particulars are configured in the Settings option of the cluster object resource in Base View (figure 3.1). The cmsh equivalents can be configured from the base object in partition mode. Values that can be set include:

• the “Management network”. This is the network over which CMDaemon manages the nodes. Management means that CMDaemon on the head node communicates with CMDaemons on the other nodes. The communication includes CMDaemon calls and monitoring data. By default, the management network is set to internalnet for Type 1 and Type 2 networks, and managementnet in Type 3 networks. It is a partition-level cluster-wide setting by default. Partition-level settings can be overridden by the category level setting, and node-level settings can override category level or partition level settings.

• the “Node name” can be set to decide the prefix part of the node name. By default, set to node.

• the “Node digits” can be set to decide the possible size of numbers used for suffix part of the node name. By default, set to 3.

• the “Default category”. This sets the category the nodes are in by default. By default, it is set to default.

• the “Default software image”. This sets the image the nodes use by default, for new nodes that are not in a category yet. By default, it is set to default-image.

• the “Name server”. This sets the name server used by the cluster. By default, it is set to the head node. The default configuration uses the internal network interface and serves only internal hosts. Internal hosts can override using this value at category level (here). By default external hosts cannot use this service. To allow external hosts to use the service for resolving internal hosts, the PublicDNS directive (Appendix C.54) must be set to True.

3. The internal IP addresses and other internal interface values: In Base View, the network for a node, such as a physical node node001, can be configured via a navigation path

The subwindow that opens up then allows network configuration to be done for such nodes in a similar way to figure 3.8. In cmsh the configuration can be done by changing properties from the interfaces submode that is within the device mode for the node.

The items that can be set include:

• the Network device name: By default, this is set to BOOTIF for a node that boots from the same interface as the one from which it is provisioned.

• the Network: By default, this is set to a value of internalnet.

• the IP address: By default, this is automatically assigned a static value, in the range 10.141.0.1 to 10.141.255.255, with the first node being given the first address. Using a static IP address is recommended for reliability, since it allows nodes that lose connectivity to the head node to continue operating. The static address can be changed manually, in case there is an IP address or node ID conflict due to an earlier inappropriate manual intervention.

  Administrators who want DHCP addressing on the internal network, despite the consequences, can set it via a checkbox.

• onnetworkpriority: This sets the priority of DNS resolution queries for the interface on the network. The range of values that it can take is from 0 to 4294967295. Resolution takes place via the interface with the higher value.

  The default priority value for a network interface is set according to its type. These defaults are:

  Type	Value
  Bridge	80
  Bond	70
  Physical	60
  VLAN	50
  Alias	40
  Tunnel	30
  Netmap	20
  BMC	10

• Additional Hostname: In the case of nodes this is in addition to the default node name set during provisioning. The node name set during provisioning takes a default form of node <3 digit number>, as explained earlier here in the section describing the cluster object settings.

For, example, a regular node that has an extra interface, eth1, can have its values set as follows:

Example

[basecm11] device interfaces node001
[basecm11->device[node001]->interfaces]% add physical eth1
[basecm11->...->interfaces*[eth1*]]% set network externalnet
[basecm11->...->interfaces*[eth1*]]% set additionalhostnames extra01
[basecm11->...->interfaces*[eth1*]]% set ip 10.141.1.1
[basecm11->...->interfaces*[eth1*]]% commit
[basecm11->...->interfaces[eth1]]% ..
[basecm11->...->interfaces]% ..
[basecm11->device[node001]]% reboot

4. Node category network values: are settings for the internal network that can be configured for node categories. For example, for the default category called default this can be carried out:

• via Base View, using the Settings option for the category. This is in the navigation path Grouping > Categories > > Settings

• or via the category mode in cmsh, for the default node category.

If there are individual node settings that have been configured in Base View or cmsh, then the node settings override the corresponding category settings for those particular nodes.

The category properties involved in internal networking that can be set include:

• Default gateway: The default gateway route for nodes in the node category. If unset, or 0.0.0.0 (the default), then the node default gateway route is decided by the internal network object Gateway value. If the default gateway is set as a node category value, then nodes use the node category value as their default gateway route instead.

• Management network: The management network is the network used by CMDaemon to manage devices. The default setting is a property of the node object. It can be set as a category property.

• Name server, Time server, Search domain: The default setting for these on all nodes is set by the node-installer to refer to the head node, and is not configurable at the node level using Base View or cmsh. The setting can however be set as a category property, either as a space-separated list of entries or as a single entry, to override the default value.

Application to generic network devices: The preceding details for the internal network parameters of the cluster, starting from here, are applicable to regular nodes, but they are often also applicable to generic network devices (section 2.1.1). Benefits of adding a generic device to be managed by BCM include that:

• the name given to the device during addition is automatically placed in the internal DNS zone, so that the device is accessible by name

• the device status is automatically monitored via an ICMP ping (Appendix G.2.1).

• the device can be made to work with the health check and metric framework. The scripts used in the framework will however almost certainly have to be customized to work with the device

After changing network configurations, a reboot of the device is necessary to activate the changes.

3.2.3.5 Changing The Global Network Parameters For The Cluster

The global network globalnet is a unique network used to set up a common name space for all nodes in a cluster in BCM. It is required in BCM because of the added cloud extension functionality, described in the Cloudbursting Manual. Regular nodes and regular cloud nodes are thus both named under the globalnet domain name, which is cm.cluster by default. So, for example, if default host names for regular nodes (node001, node002, …) and regular cloud nodes (cnode001, cnode002, …) are used, node addresses with the domain are:

• node001.cm.cluster for node001

• cnode001.cm.cluster for cnode001

The only parameter that can be sensibly changed on globalnet is the domain name, which is cm.cluster by default.

Removing globalnet should not be done because it will cause various networking failures, even for clusters deploying no cloud nodes.

Details on how resolution is carried out with the help of globalnet are given in section 6.6.1 of the Cloudbursting Manual.

3.2.3.6 Setting Static Routes In The staticroutes Submode Of cmsh

To route via a specific gateway, the staticroutes submode can be used. This can be set for regular nodes and head nodes via the device mode, and for node categories via the category mode.

On a newly-installed cluster with a type 1 network (section 3.3.9 of the Installation Manual), a node by default routes packets using the head node as the default gateway.

If the administrator would like to configure a regular node to use another gateway to reach a printer on another subnet, as illustrated by figure 3.9:

Figure 3.9: Example Of A Static Route To A Printer On Another Subnet

then an example session for setting the static route is as follows:

Example

[root@basecm11 ~]# cmsh
[basecm11 ]% device use node001
[basecm11->device[node001]]%
[basecm11->device[node001]]% staticroutes
[basecm11->device[node001]->staticroutes]% list
Name (key)                       Gateway          Destination
-------------------------------- ---------------- --------------
[basecm11->device[node001]->staticroutes]% add printerroute
[basecm11->...*[printerroute*]]% set gateway 10.141.1.1
[basecm11->...*[printerroute*]]% set destination 192.168.0.3
[basecm11->...*[printerroute*]]% set networkdevicename bootif
[basecm11->...*[printerroute*]]% commit
[basecm11->...[printerroute]]% show
Parameter                        Value
-------------------------------- -------------------------------
Destination                      192.168.0.3/32
Gateway                          10.141.1.1
Name                             printerroute
Network Device Name              bootif
Revision
[basecm11->device[node001]->staticroutes[printerroute]]% exit
[basecm11->device[node001]->staticroutes]% list
Name (key)                       Gateway          Destination
-------------------------------- ---------------- --------------
printerroute                     10.141.1.1       192.168.0.3/32
[basecm11->device[node001]->staticroutes]% exit; exit
[basecm11->device]% reboot node001

In the preceding example, the regular node node001, with IP address 10.141.0.1 can connect to a host 192.168.0.3 outside the regular node subnet using a gateway with the IP addresses 10.141.1.1 and 192.168.0.1. The route is given the arbitrary name printerroute for administrator and CMDaemon convenience, because the host to be reached in this case is a print server. The networkdevicename is given the interface name bootif. If another device interface name is defined in CMDaemon, then that can be used instead. If there is only one interface, then networkdevicename need not be set.

After a reboot of the node, the route that packets from the node follow can be seen with a traceroute or similar. The ping utility with the -R option can often track up to 9 hops, and can be used to track the route:

Example

[root@basecm11 ~]# ssh node001 ping -c1 -R 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(124) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=62 time=1.41 ms
RR:     10.141.0.1
        10.141.1.1
        192.168.0.1
        192.168.0.3
        192.168.0.3
        192.168.0.1
        10.141.1.1
        10.141.0.1
--- 192.168.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 1.416/1.416/1.416/0.000 ms
[root@basecm11->device[node001]->staticroutes]%

The routing is achieved by CMDaemon making sure that whenever the network interface is brought up, the OS of the regular node runs a routing command to configure the routing tables. The command executed for the given example is either:

    route add -host 192.168.0.3 gw 10.141.1.1

or its modern iproute2 equivalent:

    ip route add 192.168.0.3 via 10.141.1.1

If the device bootif is eth0—which is often the case—then the command is run from the network interface configuration file: /etc/sysconfig/network-scripts/ifcfg-eth0 (or /etc/sysconfig/network/ifcfg-eth0 in SUSE).

If the device is a head node: then rebooting it to have the new route become active is a clean way to implement the change. However doing this may not be acceptable to cluster users due to the downtime, and the associated interruption of jobs. A quicker update to the interface that should work is using operating system networking scripts. For example, if the interface with the route is eth0, then the interface can be brought down and up again with ifdown eth0 && ifup eth0. If the various networking configurations have been applied in a way that persists over reboots, then they should be applied automatically to the interface as it starts up.

3.2.3.7 Port Forwarding With portforward

Why port forwarding is useful: In a typical type 1 network setup (section 3.3.9 of the Installation Manual), only the head nodes are accessible from the external network. So, the compute nodes cannot be accessed directly from the external network.

Accessing a compute node port may however be needed for some workflows. For example, a user on a laptop on the external network may wish to access a compute node port directly.

To allow this direct access to a compute node port, port forwarding rules can be set up from within the device mode of cmsh. Configuring port forwarding with BCM is designed to be easier than doing port forwarding directly in the operating system.

The port forwarding help text: The help text for the portforward command summarizes how to use it:

[basecm11->device]% help portforward
Name:
  portforward - Forward a port on the active head node to IP
Usage:
  portforward create <IP> <port> <active-head-node-port>
  portforward destroy [IP] [port] [active-head-node-port]
  portforward bmc [device] [https-interface-port] [console-port]
  portforward list
Examples:
  portforward create 10.148.0.1 443 8443    Create a mapping from head:8443 to 10.148.0.1:443
  portforward create node001 8081 8181      Create a mapping from head:8181 to the management IP of node001:8081
  portforward create node001:rf0 443 8443   Create a mapping from head:8443 to the redfish IP of node001:443
  portforward destroy 10.148.0.1            Destroy all port forwarding to 10.148.0.1
  portforward destroy                       Destroy all port forwarding
  portforward bmc                           Create a mapping to the BMC of the current device
  portforward bmc node001                   Create a mapping to the BMC of node001
  portforward list                          List all active mappings

An illustration of port forwarding working: The following example indicates how BCM port forwarding works. A laptop user on the external network may wish to connect to port 1234 on compute node node001 by connecting to the head node port 4321. There is no port 4321 open on the head node to start with, until a portforward command is run:

Example

[basecm11->device]% connections master | grep 4321
[basecm11->device]% portforward create node001 1234 4321
[basecm11->device]% connections master | grep 4321
basecm11  TCP6     0.0.0.0                    4321     0.0.0.0         0        Listening

The connections command is described here.

In the background on the active head there is a new socat process that is created for each port forward:

[basecm11->device]% !ps x | grep [s]ocat
 159559 ?        S      0:00 /usr/bin/socat tcp-listen:4321,reuseaddr,fork tcp:10.141.0.1:1234

The socat process forwards any packets from external network interface of the head node to the internal network interface of the compute node, as illustrated by the following ASCII diagram:

                 external            internal
company laptop   <------>    head    <------>  node001
                 <-- 4321:10.141.0.1:1234 -->

The forwarding can be tested by setting up a netcat listening process on port 1234 of node001:

[basecm11->device]% ssh node001 "nc -l 1234"

and then typing something to port 4321 on the head node:

root@basecm11:~# nc basecm11 4321
Hello, World!
^C

The Ctrl-C interrupts the session on the head node.

The Hello, World! text entered on the head node shows up on the netcat listener on node001. The session on the compute node ends with the Ctrl-C interrupt on the head node:

[basecm11->device]% ssh node001 "nc -l 1234"
Hello, World!
[basecm11->device]%

A successful session as just illustrated means that port forwarding is working as expected.

If the session is unsuccessful, then it may be because the port is blocked by a firewall. By default, the head node runs a firewall, Shorewall, (here), which may need to have its rules adjusted.

Listing active port forwarding rules: All are listed with the list command:

Example

[basecm11->device]% portforward list
Device       IP           Port         Head port    Sessions
------------ ------------ ------------ ------------ --------
node001      10.141.0.1   8081         8181         1
node002      10.141.0.2   8081         8281         1
node003      10.141.0.3   8081         8381         1
node004      10.141.0.4   8081         8481         1

Sessions denotes the number of cmsh instances that started the exact same rule. This is normally 1. Each request is tracked via a cmsh session unique identifier to prevent killing a port forward rule placed by someone else.

Destroying (removing) every rule: A specific rule can be removed by specifying its compute node IP address and compute node port. All rules are removed with the destroy option used on its own:

Example

[basecm11->device]% portforward destroy
Destroyed: 4 port forward

BMC port forwarding: For BMC port forwarding 2 rules are created.

• A rule for the BMC management interface on 443

• A rule for the BMC console port (5900 or 17900 depending on the type of interface)

3.2.4 Tools For Viewing Cluster Connections And Connectivity

3.2.4.1 Viewing Node Routes With routes In cmsh

The cmsh routes command is a wrapper around the Linux route command, and is designed to run in parallel over nodes. It is intended to get a fast route overview for one or more nodes, and display it for easy comparison.

To get a full overview of the routes for all nodes, the command is run without any options:

Example

[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% routes
Hostname  Destination      Gateway          Netmask          Flags     Metric  Ref  Use  Interface
--------- ---------------- ---------------- ---------------- --------- ------- ---- ---- ---------
basecm11 0.0.0.0          192.168.255.254  0.0.0.0          UG (0003)  0       0    0    eth1
basecm11 10.141.0.0       0.0.0.0          255.255.0.0      U (0001)   0       0    0    eth0
basecm11 169.254.169.254  192.168.241.1    255.255.255.255  UGH (0007) 0       0    0    eth1
basecm11 192.168.240.0    0.0.0.0          255.255.240.0    U (0001)   0       0    0    eth1
node001  0.0.0.0          10.141.255.254   0.0.0.0          UG (0003)  0       0    0    ens3
node001  10.141.0.0       0.0.0.0          255.255.0.0      U (0001)   0       0    0    ens3

To select or filter the output, the grouping options of routes, or the text processing utility awk can be used. Grouping options are options to select nodes, groups, categories, and so on, and are described in the cluster management chapter, here, while a handy link for awk one-liners is http://tuxgraphics.org/~guido/scripts/awk-one-liner.html.

Selection or filtering makes it very easy to detect badly-configured nodes. For example, when the expected gateway is <expected-gateway>, the following command may be used to display only the unexpected gateways:

Example

[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% routes --category default | awk '{if ($3 != "<expected-gateway>") print $0}'
...

3.2.4.2 Viewing IP Addresses With ipaddresses In cmsh

The ipaddresses command, in device mode, lists the configured and detected IP addresses for one or more devices. By default, link-local addresses (169.254.0.0/16 and fe80::/10) are excluded from the listing, since they are rarely of interest to an administrator. The --all option includes them in this default listing; it has no effect on the following --check output, which always excludes link-local and BMC interfaces.

Example

[basecm11->device]% ipaddresses node001
Hostname Name IP         MAC               PCI Bus ID State
-------- ---- ---------- ----------------- ---------- -----
node001  eth0 10.141.0.1 FA:16:3E:F0:C4:EA            up
node001  eth1 10.141.1.1 FA:16:3E:B5:7D:02            up

The output lists one row per configured or detected address. The preceding example omits the Netmask, MTU, Speed, and Duplex columns to fit the page; the full set of columns is:

• Hostname: the node the interface belongs to.

• Name: the interface name.

• IP: the IP address on that interface.

• Netmask: the netmask for the address.

• MAC: the interface’s MAC address.

• PCI Bus ID: the PCI bus address of the NIC that the IP address is assigned to, for example 0000:03:00.0. This identifies which physical network interface corresponds to a given IP address on a node, which is useful on hosts with several NICs. It is blank for interfaces with no discoverable PCI device, such as virtual NICs, as in the preceding example. The REST API exposes the same value as the pciBusId field on the IPAddress entity.

• MTU: the interface’s maximum transmission unit.

• Speed: the negotiated link speed, or - if not available.

• Duplex: the negotiated duplex mode, or unknown if not detected.

• State: whether the interface is up or down.

The --check option compares the IP address configured in BCM for each interface against the address actually detected on the device, and reports any mismatch:

Example

[basecm11->device]% ipaddresses node001 --check
Hostname Interface Configured    Detected       State
-------- --------- ------------- -------------- --------
node001  eth0      10.141.0.1    10.141.0.1     OK
node001  eth1      10.141.128.1  10.141.128.1   OK

If any interface is not OK, the command exits with a non-zero status, though the table is still printed in full:

Example

[basecm11->device]% ipaddresses node001 --check
Hostname Interface Configured    Detected       State
-------- --------- ------------- -------------- ---------------
node001  eth0      10.141.0.1    10.141.0.1     OK
node001  eth1      10.141.128.1  10.141.128.99  Mismatch
node001  eth2      10.141.255.2  -              Not detected
node001  eth3      -             10.2.0.5       Not configured

The State column values are:

• OK: the configured and detected addresses match, or the interface uses DHCP and an address was detected.

• Mismatch: both a configured and a detected address are present, but they differ.

• Not detected: an address is configured, but nothing was detected on the interface.

• Not configured: an address was detected, but nothing is configured in BCM for that interface.

The standard node selection options (-n, -c, -g, -r) can be used with --check to target a set of devices, for example ipaddresses -r rack01 --check. The --check option cannot be combined with --import, and --json is ignored when --check is used.

3.2.4.3 Viewing Connections In cmsh

The connections command can be run from within the device mode of cmsh. It is a parallel wrapper to view active TCP and UDP connections. That is, it runs as a simple command over multiple devices at the same time.

Running connections without options displays a full overview of the currently active TCP and UDP ports on all devices (output truncated):

Example

[basecm11->device]% connections
Node      Type     Source   Port  Destination  Port  State
--------- -------- -------- ----- ------------ ----- ----------
basecm11  TCP      0.0.0.0  22    0.0.0.0      0     Listening
basecm11  TCP      0.0.0.0  25    0.0.0.0      0     Listening
basecm11  TCP      0.0.0.0  111   0.0.0.0      0     Listening
basecm11  TCP      0.0.0.0  636   0.0.0.0      0     Listening
...

Filtering with grep can then show which nodes are listening on which ports. For example, nodes listening for DNS queries (port 53) can be found with:

Example

[basecm11->device]% connections | head -2; connections | grep Listen | grep " 53"
Node      Type     Source             Port     Destination     Port     State
--------- -------- ------------------ -------- --------------- -------- -----------
basecm11  TCP      10.141.255.254     53       0.0.0.0         0        Listening
basecm11  TCP      127.0.0.1          53       0.0.0.0         0        Listening
basecm11  TCP      192.168.200.162    53       0.0.0.0         0        Listening

and shows that only the head node is listening on that port, providing DNS.

Some third party tools require a free port on all nodes for a service to listen on. Filtering and sorting the output of the connections command allows the administrator to find all the existing used ports across all nodes:

Example

[basecm11->device]% connections | awk 'print $4' | sort -un
Port
22
25
53
67
68
69
...

Options to the connections command can be seen by running help connections. Options include node grouping options (such as -n|--nodes, -c|--category, and -t|--type), and filtering out TCP6 and UDP6 connections (--no-tcp6, --no-udp6).

3.2.4.4 Viewing Connections In Base View

The preceding connections command in device mode of cmsh has a Base View version. This is the Network Connections pane display, which is accessible using:

All devices > Bulk actions > Connectivity > Network Connections

3.2.4.5 Viewing Connectivity in cmsh

The connectivity command can be run from within the device mode of cmsh. It runs ICMP pings along each node route on a network of the cluster. By default the network is the management network.

By default the output displays if the connection is OK, the ping sequence ID (counting starts from zero), and the latency between the source and destination:

Example

[basecm11->device]% connectivity
Source    Destination Result       ID   Latency
--------- ----------- ------------ ---- ------------
basecm11  node001     Ok           0    0.4ms
basecm11  node002     Ok           0    0.3ms
basecm11  node003     Ok           0    0.4ms
basecm11  basecm11    Ok           0    0.1ms

Pings where the source and destination have identical names are carried out via identical interfaces.

Further options to the connectivity command can be seen by running help connectivity. Options include the ability to set ping timeouts and the cluster network on which to ping, as well as node grouping options.

For larger clusters, the following type of connectivity check may be a helpful diagnostic:

Example

[basecm11->device]% connectivity --statistics --count 100 --delay 0.01
Name        Value
----------- --------
OK          400
Total       400
Count       2
Average     0.35ms
Minimum     0.35ms
Maximum     0.35ms
Uniformity  100.0%

The preceding shows how uniform nodes are in ping timings. A significant spread in uniformity can indicate network problems.

3.2.4.6 Viewing Connectivity in Base View

The preceding connectivity command in device mode of cmsh has a Base View version. This is the Ping All pane display, which is accessible using:

All devices > Bulk actions > Connectivity > Ping All

3.2.4.7 Viewing Network Performance With cm-iperf.py

A handy tool for checking the network performance of the nodes on a cluster is cm-iperf.py.

The nodes to be checked must be specified with the -n|--node option:

Example

basecm11:~#  /cm/local/apps/cmd/scripts/cm-iperf.py  -n node001..node004
Thread  Port    Server          Client          Target          Bitrate
======= ======= =============== =============== =============== =================
0       5000    node001         node002         node001         7.89 Gbits/sec
0       5001    node001         node003         node001         7.84 Gbits/sec
0       5002    node001         node004         node001         8.22 Gbits/sec
0       5003    node002         node003         node002         7.53 Gbits/sec
0       5004    node002         node004         node002         8.60 Gbits/sec
0       5005    node003         node004         node003         7.59 Gbits/sec

For a Type 1 cluster (section 3.3.9 of the Installation Manual) with n nodes, the total number of connection pairs that can be evaluated is:

For a 4-node cluster this means 6 network connection pairs are displayed, as is seen in the preceding example.

Each pair is checked for about 10s. This means that for a 1000-node cluster, the time for a default run would take about 2 months. The following options are therefore useful:

• --count: specify the number of pairs that are checked

• -r|--random: have the connection pairs chosen in a random sequence

• -p|--parallel: specify the number of parallel threads to runs the process

Example

basecm11:~#  /cm/local/apps/cmd/scripts/cm-iperf.py  -n node001..node004 --count 5 -r -p 2

Further options can be viewed with the -h|--help option.

3.3 Configuring Bridge Interfaces

Bridge interfaces can be used to divide one physical network into two separate network segments at layer 2 without creating separate IP subnets. A bridge thus connects the two networks together at layer 3 in a protocol-independent way.

The network device name given to the bridge interface is of the form br<number>. The following example demonstrates how to set up a bridge interface in cmsh, where the name br0 is stored by the parameter networkdevicename.

Example

[basecm11->device[node001]->interfaces]% add bridge br0
[basecm11->...->interfaces*[br0*]]% set network internalnet
[basecm11->...->interfaces*[br0*]]% set ip 10.141.1.1
[basecm11->...->interfaces*[br0*]]% show
Parameter                      Value
------------------------------ ------------------
Additional Hostnames
DHCP                           no
Forward Delay                  0
IP                             10.141.1.1
Interfaces
MAC                            00:00:00:00:00:00
Network                        internalnet
Network device name            br0
Revision
SpanningTreeProtocol           no
Type                           bridge

A bridge interface is composed of one or more physical interfaces. The IP and network fields of the member interfaces must be empty:

Example

[basecm11->...->interfaces*[br0*]]% set interfaces eth1 eth2; exit
[basecm11->...->interfaces*]% clear eth1 ip; clear eth1 network
[basecm11->...->interfaces*]% clear eth2 ip; clear eth2 network
[basecm11->...->interfaces*]% use br0; commit

The BOOTIF interface is also a valid interface option.

Currently, the following bridge properties can be set:

• SpanningTreeProtocol: sets the spanning tree protocol to be on or off. The default is off.

• Forward Delay: sets the delay for forwarding Ethernet frames, in seconds. The default is 0.

Additional properties, if required, can be set manually using the brctl command in the OS shell.

When listing interfaces in cmsh, if an interface is a member of a bond (or bridge) interface, then the corresponding bond or bridge interface name is shown in parentheses after the member interface name:

Example

[headnode->device[node001]->interfaces]% list
Type         Network device name  IP               Network
------------ -------------------- ---------------- ----------------
bond         bond0 [prov]         10.141.128.1     internalnet
bridge       br0                  10.141.128.2     internalnet
physical     eth0                 10.141.0.1       internalnet
physical     eth1 (bond0)         0.0.0.0
physical     eth2 (bond0)         0.0.0.0
physical     eth3 (br0)           0.0.0.0
physical     eth4 (br0)           0.0.0.0

It is possible to create a bridge interface with no IP address configured, that is, with an IP address of 0.0.0.0. This can be done for security reasons, or when the number of available IP addresses on the network is scarce. As long as such a bridge interface has a network assigned to it, it is properly configured on the nodes and functions as a bridge on the network.

3.4 Configuring VLAN interfaces

A VLAN (Virtual Local Area Network) is an independent logical LAN within a physical LAN network. VLAN tagging is used to segregate VLANs from each other. VLAN tagging is the practice of inserting a VLAN ID tag into a packet frame header, so that each packet can be associated with a VLAN.

The physical network then typically has sections that are VLAN-aware or VLAN-unaware. The nodes and switches of the VLAN-aware section are configured by the administrator to decide which traffic belongs to which VLAN.

A VLAN interface can be configured for an interface within BCM using cmsh or Base View.

3.4.1 Configuring A VLAN Interface Using cmsh

In the following cmsh session, a regular node interface that faces a VLAN-aware switch is made VLAN-aware, and assigned a new interface—an alias interface. It is also assigned a network, and an IP address within that network:

Example

[root@basecm11 ~]# cmsh
[basecm11]% device interfaces node001
[basecm11->device[node001]->interfaces]% list
Type         Network device name  IP             Network
------------ -------------------- -------------- ------------
physical     BOOTIF [prov]        10.141.0.1     internalnet
[basecm11->device[node001]->interfaces]% list -h | tail -4
Arguments:
   type
     alias, bmc, bond, bridge, netmap, physical, tunnel, vlan
[basecm11->device[node001]->interfaces]% add vlan BOOTIF.1
[basecm11->device*[node001*]->interfaces*[BOOTIF.1*]]% commit
[basecm11->device[node001]->interfaces[BOOTIF.1]]% show
Parameter                 Value
------------------------- ------------------------
Additional Hostname
DHCP                      no
IP                        0.0.0.0
MAC                       00:00:00:00:00:00
Network
Network device name       BOOTIF.1
Reorder HDR               no
Revision
Type                      vlan
[basecm11->...[BOOTIF.1]]% set network internalnet
[basecm11->...[BOOTIF.1*]]% set ip 10.141.2.1; commit

The Reorder HDR setting in a VLAN, if enabled, moves the ethernet header around to make it look exactly like a real ethernet device. This setting controls the REORDER_HDR setting in the file at /etc/sysconfig/network-scripts/ifcfg- <interface> on the node.

3.4.2 Configuring A VLAN Interface Using Base View

Within Base View a VLAN interface can be added for a node such as node001 using the navigation path:
Devices > All Devices > node001 > Settings > Interfaces > + Add > vlan
(figure 3.10).

Figure 3.10: Configuring a VLAN via Base View

A Network VLAN Interface window opens up, allowing an IP address, network, and other options to be configured for the VLAN interface.

3.5 Configuring Bonded Interfaces

3.5.1 Adding A Bonded Interface

The Linux bonding driver allows multiple physical network interfaces that have been configured previously (for example, as here) to be bonded as a single logical bond interface. The behavior of such interfaces is determined by their bonding mode. The modes provide facilities such as hot standby or load balancing.

The driver is included by default on head nodes. To configure a non-head node to use a bonded interface, a Linux kernel module called the bonding module must be included in the kernel modules list of the software image of the node. A bonding interface can then be created, and its properties set as follows:

Example

[basecm11->device[node001]->interfaces]% add bond bond0
[...->device*[node001*]->interfaces*[bond0*]]% set network internalnet
[...->device*[node001*]->interfaces*[bond0*]]% set ip 10.141.128.1
[...->device*[node001*]->interfaces*[bond0*]]% set mode 3
[...->device*[node001*]->interfaces*[bond0*]]% set interfaces eth1 eth2

Each bonded interface has a unique set of object properties, called bonding options, that specify how the bonding device operates. The bonding options are a string containing one or more options formatted as option=value pairs, with pairs separated from each other by a space character.

A bonded interface also always has a mode associated with it. By default it is set to 0, corresponding to a balanced round-robin packet transmission.

The 7 bonding modes are:

• 0 – balance-rr

• 1 – active-backup

• 2 – balance-xor

• 3 – broadcast

• 4 – 802.3ad

• 5 – balance-tlb

• 6 – balance-alb

Technically, outside of Base View or cmsh, the bonding mode is just another bonding option specified as part of the options string. However in BCM the bonding mode value is set up using the dedicated mode property of the bonded interface, for the sake of clarity. To avoid conflict with the value of the mode property, trying to commit a bonding mode value as an option=value pair will fail validation.

3.5.2 Single Bonded Interface On A Regular Node

A single bonded interface on a node can be configured and coexist in several ways on nodes with multiple network interfaces. Possibilities and restrictions are illustrated by the following:

• The bonded interface may be made up of two member interfaces, and a third interface outside of the bond could be the boot interface. (The boot interface is the node interface used to PXE boot the node before the kernel loads (section 5.1)).

• The boot interface could itself be a member of the bonded interface. If the boot interface is a member of a bonded interface, then this is the first bonded interface when interfaces are listed as in the example here.

• The bonded interface could be set up as the provisioning interface. However, the provisioning interface cannot itself be a member of a bonded interface. (The provisioning interface is the node’s interface that picks up the image for the node after the initial ramdisk is running. Chapter 5 covers this in more detail).

• A bonded interface can be set up as the provisioning interface, while having a member interface which is used for PXE booting.

3.5.3 Multiple Bonded Interface On A Regular Node

A node can also have multiple bonded interfaces configured. Possibilities and restrictions are illustrated by the following:

• Any one of the configured bonded interfaces can be configured as the provisioning interface. However, as already mentioned in the case of single bond interfaces (section 3.5.2), a particular member of a bonded interface cannot be made the provisioning interface.

• When a bonded interface is set as the provisioning interface, then during the node-installer phase of boot, the node-installer brings up the necessary bonded interface along with all its member interfaces so that node provisioning is done over the bonded interface.

3.5.4 Bonded Interfaces On Head Nodes And HA Head Nodes

It is also possible to configure bonded interfaces on head nodes.

For a single head node setup, this is analogous to setting up bonding on regular nodes.

For a high availability (HA) setup (Chapter 15), bonding is possible for internalnet as well as for externalnet, but it needs the following extra changes:

• For the bonded interface on internalnet, the shared internal IP alias interface name (the value of networkdevicename, for example, eth0:0 in figure 15.1) for that IP address should be renamed to the bonded alias interface name on internalnet (for example, bond0:0).

• For the bonded interface on externalnet, the shared external IP alias interface name (the value of networkdevicename, for example, eth1:0 in figure 15.1) for that IP address should be renamed to the bonded alias interface name on externalnet (for example, bond1:0).

• Additionally, when using a bonded interface name for the internal network, the value of the provisioning network interface name (the value of provisioninginterface, for example, eth0) for the head nodes, must be changed to the name of the bonded interface (for example, bond0 [prov]) on the internal network. The provisioninginterface value setting is described further here.

  Example

  [headnode1->device[headnode1]->interfaces]% list
  Type         Network device name  IP               Network
  ------------ -------------------- ---------------- ------------
  alias        bond0:0              10.141.255.252   internalnet
  alias        bond1:0              10.150.57.1      externalnet
  bond         bond0 [prov]         10.141.255.254   internalnet
  bond         bond1                10.150.57.3      externalnet
  physical     eth0 (bond0)         0.0.0.0
  physical     eth1 (bond0)         0.0.0.0
  physical     eth2 (bond1)         0.0.0.0
  physical     eth3 (bond1)         0.0.0.0

3.5.5 Tagged VLAN On Top Of a Bonded Interface

It is possible to set up a tagged VLAN interface on top of a bonded interface. There is no requirement for the bonded interface to have an IP address configured in this case. The IP address can be set to 0.0.0.0, however a network must be set.

Example

[headnode1->device[node001]->interfaces]% list
Type         Network device name  IP               Network
------------ -------------------- ---------------- ------------
bond         bond0                0.0.0.0          internalnet
physical     eth0 [prov]          10.141.0.1       internalnet
physical     eth1 (bond0)         0.0.0.0
physical     eth2 (bond0)         0.0.0.0
vlan         bond0.3              10.150.1.1       othernet

3.5.6 Association Of MAC Address With A Bonded Interface

BCM has MAC settings for interfaces as well as for nodes.

When a node is provisioned via PXE booting (Chapter 5), the MAC on the NIC that is being booted from is matched with the MAC property of the node. This is how the node is identified.

If a bonded interface is used, then any interface may be used for PXE booting. Each interface has its own MAC address, which means that the interface MAC address may not match the assigned node MAC address. In BCM 11 and later versions, the node-installer recognizes any MAC address in a bonded interface.

3.5.7 Further Notes On Bonding

If using bonding to provide failover services, then the kernel module option setting for media independent interface monitoring, miimon, which is set to be off by default (set to 0), should be given a non-zero value.

The miimon setting is the time period in milliseconds between checks of the interface carrier state. A common value is miimon=100.

Its value is set in the bonding device configuration file.

• For RHEL and derivatives, and for SLES distributions, the file is in a network script along a file path such as /etc/sysconfig/network-scripts/ifcfg-bond0.

  The line within the stanza for the ifcfg-bond0 file might look like:

  Example

      BONDING_MODULE_OPTS="mode=active-backup miimon=100"

  and can be set manually.

  • For RHEL and derivatives, it can alternatively be configured with front-end tools such as the nmcli tool, nmtui, or others.

    Example

        # nmcli connection add type bond con-name bond0 ifname bond0 bond.options \
          "mode=active-backup,miimon=100"

  • For SLES, YaST can be used as a front-end tool (YaST > System > Network Settings).

• For Ubuntu, the interface definitions are in a file, either /etc/network/interfaces, or a file under /etc/network/interfaces.d/.

  The line to set the value of miimon follows a form such as:

  • bond-miimon 100

  instead of

  • miimon=100

  and can be set manually.

  Alternatively, Canonical’s netplan (https://netplan.io) utility can be used to set the network configuration files. The netplan YAML configuration key to set bond-miimon is mii-monitor-interval.

When listing interfaces in cmsh, if an interface is a member of a bond or bridge interface, then the corresponding bonded or bridge interface name is shown in parentheses after the member interface name. Section 3.3, on configuring bridge interfaces, shows an example of such a listing from within cmsh here.

More on bonded interfaces (including a detailed description of bonding options and modes) can be found at http://www.kernel.org/doc/Documentation/networking/bonding.txt.

3.6 Configuring InfiniBand Interfaces

On clusters with an InfiniBand interconnect, the InfiniBand Host Channel Adapter (HCA) in each node must be configured before it can be used.

This section describes how to set up the InfiniBand service on the nodes for regular use. Setting up InfiniBand for booting and provisioning purposes is described in Chapter 5, while setting up InfiniBand for NFS is described in section 3.14.4.

3.6.1 Installing Software Packages

On a standard NVIDIA Base Command Manager cluster, the OFED (OpenFabrics Enterprise Distribution) packages that are part of the Linux base distribution are used. These packages provide RDMA implementations allowing high bandwidth/low latency interconnects on OFED hardware. The implementations can be used by InfiniBand hardware.

By default, all relevant OFED packages are installed on the head node and software images. It is possible to replace the distribution OFED with a non-distribution OFED. The replacement can be for the entire cluster, or only for certain software images. Administrators may choose to switch to a different OFED version if the HCAs used are not supported by the distribution OFED version, or to increase performance by using an OFED version that has been optimized for a particular HCA. Installing a DOCA Mellanox OFED stack onto BCM is covered in Chapter 10 of the Installation Manual.

If the InfiniBand network is enabled during cluster installation, then the infiniband.conf and rdma.conf modules in the subdirectory etc/rdma/modules/ are automatically configured for the detected hardware.

The relevant InfiniBand HCA kernel modules are then automatically loaded during the init stage by systemd. Verifying that InfiniBand is active can be done after the cluster is up and running by running ibstat:

auser@head:~$ ibstat
CA 'mlx5_0'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.31.2006
        Hardware version: 0
        Node GUID: 0xb8599f0300e4222a
        System image GUID: 0xb
        Port 1:
                State: Active
                Physical state: LinkUp
...

3.6.2 Mellanox Adapter Settings With mlxconfig

The device mode mlxconfig command displays and sets Mellanox adapter configuration on nodes. It is useful for checking the runtime state that the NVIDIA mlxconfig utility reports, and for applying selected supported settings from BCM.

Example

[basecm11->device]% mlxconfig node001 --adapters
Hostname  Interface  PCI           HCA     Card Name              Port
--------  ---------  ------------  ------  ---------------------  ----
node001   ib0        0000:03:00.0  mlx5_0  MCX653105A-HDA_Ax_Bx   1

Without --adapters, the command expands each matching adapter into the known mlxconfig fields. The output shows the BCM field name, the current value, the pending value, and whether a reboot is required for pending changes to take effect. The output can be limited to an interface, PCI address, port, or detail level:

Example

[basecm11->device]% mlxconfig node001 --interface ib0
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --port 1
[basecm11->device]% mlxconfig node001 --detail-level 1

To set a value immediately on a node, use --set. A per-port field can be selected either from the interface context or by using a port suffix in the field name:

Example

[basecm11->device]% mlxconfig node001 --interface ib0 --set link_type=ETH
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --port 1 --set link_type=ETH
[basecm11->device]% mlxconfig node001 --pci-address 0000:03:00.0 --set link_type_p1=ETH

Some mlxconfig changes are pending until the node is rebooted. When this is the case, the set result indicates that a reboot is required.

3.6.2.1 Persistent mlxconfig Policy With Extra Values

Persistent Mellanox adapter settings can be declared through extra_values entries whose keys start with MLX_. The desired settings can be defined on a category, on a node, or on a network interface. When the same field is defined at more than one level, the precedence is:

category.extra_values  <  node.extra_values  <  interface.extra_values

Category and node settings apply to all discovered Mellanox adapters on the node. Interface settings apply to the selected interface. For VLAN and bond interfaces, BCM resolves the setting to the underlying physical interface or interfaces before applying it.

The key name after MLX_ can use a BCM field name, such as link_type_p1, or a native mlxconfig field name, such as KEEP_ETH_LINK_UP_P1. Values may be strings, booleans, or numbers. Boolean values are applied to mlxconfig as 1 or 0.

Example

[basecm11->category[default]]% set -e MLX_link_type_p1 ETH
[basecm11->category*[default*]]% commit
[basecm11->device[node001]]% set -e MLX_num_vfs 16
[basecm11->device[node001]->interfaces[ib0]]% set -e MLX_KEEP_ETH_LINK_UP_P1 true
[basecm11->device*[node001*]->interfaces*[ib0*]]% commit

The settings are reconciled by the node after it connects to the head node and again during the periodic node update. The apply step is idempotent: BCM does not set a field again if the current or pending mlxconfig value already matches the desired value. If the local mlxconfig binary is missing, no Mellanox adapter is discovered, or mlxconfig rejects a value, BCM raises a warning event.

The --desired option shows the declared policy from the cluster configuration. It does not query the node or run mlxconfig, so it can be used to inspect the intended policy even before the node has applied it.

Example

[basecm11->device]% mlxconfig node001 --desired
Hostname  Source         Interface  Field                 Value
--------  -------------  ---------  --------------------  -----
node001   category:gpu   *          link_type_p1          ETH
node001   node           *          num_vfs               16
node001   interface:ib0  ib0        KEEP_ETH_LINK_UP_P1   true

3.6.3 Subnet Managers

Every InfiniBand subnet requires at least one subnet manager to be running. The subnet manager takes care of routing, addressing and initialization on the InfiniBand fabric. Some InfiniBand switches include subnet managers. However, on large InfiniBand networks or in the absence of a switch-hosted subnet manager, a subnet manager needs to be started on at least one node inside of the cluster. When multiple subnet managers are started on the same InfiniBand subnet, one instance will become the active subnet manager whereas the other instances will remain in passive mode. It is recommended to run 2 subnet managers on all InfiniBand subnets to provide redundancy in case of failure.

On a Linux machine that is not running BCM, an administrator sets a subnet manager service (usually opensm, but opensmd in SLES) to start at boot-time with a command such as:

systemctl enable opensm.service

However, for clusters managed by BCM, a subnet manager is best set up using CMDaemon. There are two ways of setting CMDaemon to start up the subnet manager on a node at boot time:

1. by assigning a role.

   In cmsh this can be done with:

   [root@basecm11 ~]# cmsh -c "device roles <node>; assign subnetmanager; commit"

   where <node> is the name of a node on which it will run, for example: basecm11, node001, node002…

   In Base View, the subnet manager role is assigned by selecting a head node or regular node from the Devices resource, and assigning it the “Subnet manager role”. The navigation path for this, for a node node002 for example, is:

   Devices > All Devices > node002 > Settings > Roles > + Add > [SubnetManagerRole]

2. by setting the service up. Services are covered more generally in section 3.15.

   In cmsh this is done with:

   Example

   [root@basecm11 ~]# cmsh
   [basecm11]% device services node001
   [basecm11->device[node001]->services]% add opensm
   [basecm11->device[node001]->services*[opensm*]]% set autostart yes
   [basecm11->device[node001]->services*[opensm*]]% set monitored yes
   [basecm11->device[node001]->services*[opensm*]]% commit
   [basecm11->device[node001]->services[opensm]]%

   In Base View the subnet manager service is configured by selecting a head node or regular node from the resources tree, and adding the service to it. The navigation path for this, for a node node002 for example, is:

   Devices > All Devices > node002 > Settings > Services > + Add

When the head node in a cluster is equipped with an InfiniBand HCA, it is a good candidate to run as a subnet manager for smaller clusters.

On large clusters a dedicated node is recommended to run the subnet manager.

3.6.4 InfiniBand Network Settings

Although not strictly necessary, it is recommended that InfiniBand interfaces are assigned an IP address (i.e. IP over IB). First, a network object in the cluster management infrastructure should be created. The procedure for adding a network is described in section 3.2.2. The following settings are recommended as defaults:

By default, an InfiniBand interface is set to datagram mode, because it scales better than connected mode. It can be configured to run in connected mode by setting the connectedmode property:

Example

[basecm11->device[node001]->interfaces[ib0]]% set connectedmode yes

For nodes that are PXE booting or are getting provisioned over InfiniBand, the mode setting in the node-installer script has to be changed accordingly.

Example

[root@basecm11 ~]# echo datagram > /cm/node-installer/scripts/ipoib_mode

Once the network has been created all nodes must be assigned an InfiniBand interface on this network. The easiest method of doing this is to create the interface for one node device and then to clone that device several times.

For large clusters, a labor-saving way to do this is using the addinterface command (section 3.7.1) as follows:

[root@basecm11 ~]# echo "device
addinterface -n node001..node150 physical ib0 ibnet 10.149.0.1
commit" | cmsh -x

When the head node is also equipped with an InfiniBand HCA, it is important that a corresponding interface is added and configured in the cluster management infrastructure.

Example

Assigning an IP address on the InfiniBand network to the head node:

[basecm11->device[basecm11]->interfaces]% add physical ib0
[basecm11->device[basecm11]->interfaces*[ib0*]]% set network ibnet
[basecm11->device[basecm11]->interfaces*[ib0*]]% set ip 10.149.255.254
[basecm11->device[basecm11]->interfaces*[ib0*]]% commit

As with any change to the network setup, the head node needs to be restarted to make the above change active.

3.6.5 Verifying Connectivity

After all nodes have been restarted, the easiest way to verify connectivity is to use the ping utility

Example

Pinging node015 while logged in to node014 through the InfiniBand interconnect:

[root@node014 ~]# ping node015.ib.cluster
PING node015.ib.cluster (10.149.0.15) 56(84) bytes of data.
64 bytes from node015.ib.cluster (10.149.0.15): icmp_seq=1 ttl=64
time=0.086 ms
...

If the ping utility reports that ping replies are being received, the InfiniBand is operational. The ping utility is not intended to benchmark high speed interconnects. For this reason it is usually a good idea to perform more elaborate testing to verify that bandwidth and latency are within the expected range.

The quickest way to stress-test the InfiniBand interconnect is to use the Intel MPI Benchmark (IMB), which is installed by default in /cm/shared/apps/imb/current. The setup.sh script in this directory can be used to create a template in a user’s home directory to start a run.

Example

Running the Intel MPI Benchmark using openmpi to evaluate performance of the InfiniBand interconnect between node001 and node002:

[root@basecm11 ~]# su - cmsupport
[cmsupport@basecm11 ~]$ cd /cm/shared/apps/imb/current/
[cmsupport@basecm11 current]$ ./setup.sh
[cmsupport@basecm11 current]$ cd ~/BenchMarks/imb/2017
[cmsupport@basecm11 2017]$ module load openmpi/gcc
[cmsupport@basecm11 2017]$ module initadd openmpi/gcc
[cmsupport@basecm11 2017]$ make -f make_mpi2
[cmsupport@basecm11 2017]$ mpirun -np 2 -machinefile ../nodes IMB-MPI1 PingPong
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         0.78         0.00
            1         1000         1.08         0.88
            2         1000         1.07         1.78
            4         1000         1.08         3.53
            8         1000         1.08         7.06
           16         1000         1.16        13.16
           32         1000         1.17        26.15
           64         1000         1.17        52.12
          128         1000         1.20       101.39
          256         1000         1.37       177.62
          512         1000         1.69       288.67
         1024         1000         2.30       425.34
         2048         1000         3.46       564.73
         4096         1000         7.37       530.30
         8192         1000        11.21       697.20
        16384         1000        21.63       722.24
        32768         1000        42.19       740.72
        65536          640        70.09       891.69
       131072          320       125.46       996.35
       262144          160       238.04      1050.25
       524288           80       500.76       998.48
      1048576           40      1065.28       938.72
      2097152           20      2033.13       983.71
      4194304           10      3887.00      1029.07
# All processes entering MPI_Finalize

To run on nodes other than node001 and node002, the ../nodes file must be modified to contain different hostnames. To perform other benchmarks, the PingPong argument should be omitted.

3.7 Configuring BMC (IPMI/iLO/DRAC/CIMC/Redfish) Interfaces

BCM can initialize and configure the baseboard management controller (BMC) that may be present on devices. This ability can be set during the installation on the head node (figure 3.15 of the Installation Manual), or it can be set after installation as described in this section. The IPMI, iLO, DRAC, CIMC, or Redfish interface that is exposed by a BMC is treated in the cluster management infrastructure as a special type of network interface belonging to a device. In the most common setup a dedicated network (i.e. IP subnet) is created for BMC communication. The 10.148.0.0/16 network is used by default for BMC interfaces by BCM.

3.7.1 BMC Network Settings

The first step in setting up a BMC is to add the BMC network as a network object in the cluster management infrastructure. The procedure for adding a network is described in section 3.2.2. The following settings are recommended as defaults:

Once the network has been created, all nodes must be assigned a BMC interface, of type bmc, on this network. The easiest method of doing this is to create the interface for one node device and then to clone that device several times.

For larger clusters this can be laborious, and a simple bash loop can be used to do the job instead:

[basecm11 ~]# for ((i=1; i<=150; i++)) do
echo "
device interfaces node$(printf '%03d' $i)
add bmc ipmi0
set network bmcnet
set ip 10.148.0.$i
commit"; done | cmsh -x     # -x usefully echoes what is piped into cmsh

The preceding loop can conveniently be replaced with the addinterface command, run from within the device mode of cmsh:

[basecm11 ~]# echo "
device
addinterface -n node001..node150 bmc ipmi0 bmcnet 10.148.0.1
commit" | cmsh -x

The help text in cmsh gives more details on how to use addinterface.

Most administrators are likely to simply run it as an interactive session in cmsh, running the help addinterface command for reference, and then supplying the options for the nodes and interface settings in device mode. For example, as in the following session:

[root@basecm11 ~]# cmsh
[basecm11]% device
[basecm11->device]% help addinterface
Name:
          addinterface - Add a network interface to one or more nodes
Usage:
          addinterface [OPTIONS] <type> <devicename> <network> <firstip>
Options:
...help text omitted...
Examples:
          addinterface -n node001..node010 physical ib0 ibnet 10.149.0.1
[basecm11->device]% addinterface -n node001..node150 bmc ipmi0 bmcnet 10.148.0.1
[basecm11->device*]% commit

In order to be able to communicate with the BMC interfaces, the head node also needs an interface on the BMC network. Depending on how the BMC interfaces are physically connected to the head node, the head node has to be assigned an IP address on the BMC network one way or another. There are two possibilities for how the BMC interface is physically connected:

• When the BMC interface is connected to the primary internal network, the head node should be assigned an alias interface configured with an IP address on the BMC network.

• When the BMC interface is connected to a dedicated physical network, the head node must also be physically connected to this network. A physical interface must be added and configured with an IP address on the BMC network.

Example

Assigning an IP address on the BMC network to the head node using an alias interface:

[basecm11->device[basecm11]->interfaces]% add alias eth0:0
[basecm11->device[basecm11]->interfaces*[eth0:0*]]% set network bmcnet
[basecm11->device[basecm11]->interfaces*[eth0:0*]]% set ip 10.148.255.254
[basecm11->device[basecm11]->interfaces*[eth0:0*]]% commit
[basecm11->device[basecm11]->interfaces[eth0:0]]%
Mon Dec 6 05:45:05 basecm11: Reboot required: Interfaces have been modified
[basecm11->device[basecm11]->interfaces[eth0:0]]% ..;..
[basecm11->device[basecm11]]% reboot