Skip to navigation

Third Party Software

View as Markdown

In this chapter, several third party software packages included in the BCM repository are described briefly. For all packages, references to the complete documentation are provided.

The packages dashboard at https://service.bcm.nvidia.com/packages-dashboard/ lists the available supported package versions per BCM version, architecture, and distribution.

7.1 Modules Environment

RHEL and derivatives, and SLES BCM package name: env-modules Ubuntu package name: cm-modules

The modules environment package is installed by default on the head node. The home page for the software is at http://modules.sourceforge.net/. The software allows a user of a cluster to modify the shell environment for a particular application, or even for a particular version of an application. Typically, a module file defines additions to environment variables such as PATH, LD_LIBRARY_PATH, and MANPATH.

Cluster users use the module command to load or remove modules from their environment. The module(1) man page has more details about the command, and aspects of the modules environment that are relevant for administrators are discussed in section 2.2 of the Administrator Manual. Also discussed there is Lmod, the Lua-based alternative to the Tcl-based traditional modules environment package.

The modules environment from a user’s perspective is covered in section 2.3 of the User Manual.

7.2 Shorewall

Package name: shorewall

7.2.1 The Shorewall Service Paradigm

BCM provides the Shoreline Firewall (more commonly known as “Shorewall”) package from the BCM repository. The package provides firewall and gateway functionality on the head node of a cluster.

Shorewall is a flexible and powerful high-level interface for the netfilter packet filtering framework. Netfilter is a standard part of Linux kernels. As its building blocks, Shorewall uses iptables and iptables6 commands to configure netfilter. All aspects of firewall and gateway configuration are handled through the configuration files located under /etc/shorewall/.

Shorewall IPv4 configuration is managed with the shorewall command, while IPv6 configuration is managed via the shorewall6 command. IPv4 filtering and IPv6 filtering are treated as separate services in Shorewall. For convenience, only IPv4 Shorewall is described from here onward, because IPv6 management is largely similar.

After modifying Shorewall configuration files, Shorewall must be restarted to have the new configuration take effect. From the shell prompt, this can be carried out with:

service shorewall restart

In NVIDIA Base Command Manager 11, Shorewall is managed by CMDaemon, in order to handle the automation of cloud node access. Restarting Shorewall can thus also be carried out within the services submode (section 3.15 of the Administrator Manual) of cmsh on the head node. For example, on a head node basecm11 a restart of shorewall can be carried out with:

[basecm11->device[basecm11]->services[shorewall]]% restart
restart Successfully restarted service shorewall on: basecm11

System administrators may sometimes need to be aware that Shorewall does not really run as a daemon process. That is, the command to restart the service does not stop and start a shorewall daemon. Instead the command configures netfilter using the iptables settings specified in the shorewall configuration files, and then exits. It exits without leaving a shorewall process up and running, even though service shorewall status implies that such a service is running.

7.2.2 Shorewall Zones, Policies, And Rules

In the default setup, Shorewall provides gateway functionality to the internal cluster network on the first network interface (eth0). This network is known as the nat zone to Shorewall. The external network (i.e. the connection to the outside world) is assumed to be on the second network interface (eth1). This network is known as the net zone in Shorewall.

Letting BCM take care of the network interfaces settings is recommended for all interfaces on the head node (section 3.2 of the Administrator Manual). The file /etc/shorewall/interfaces is generated by the cluster management daemon, and any extra instructions that cannot be added via Base View or cmsh can be added outside of the file section clearly demarcated as being maintained by CMDaemon.

Shorewall is configured by default (through /etc/shorewall/policy) to deny all incoming traffic from the net zone, except for the traffic that has been explicitly allowed in /etc/shorewall/rules. Providing (a subset of) the outside world with access to a service running on a cluster, can be accomplished by creating appropriate rules in /etc/shorewall/rules.

A rule modification is typically needed if Shorewall denies access to a port, on the head node, that needs to be accessed from an external network. The access might be, for example, for carrying out port forwarding from the head node to a compute node when using the portforward command (here).

By default, the cluster responds to ICMP ping packets. Also, during cluster installation, the following ports are open by default, but can be set to be blocked by the administrator:

  • SSH

  • HTTP

  • HTTPS

  • port 8081, which allows access to the cluster management daemon.

7.2.2.1 The cm-cmd-ports Utility

Port 8081 is the default port that CMDaemon listens to when using the HTTPS protocol to manage the nodes. If, for example, a new software needs that port, then cm-cmd-ports utility can be used to set the HTTPS protocol port that CMDaemon listens on to another port, such as 8082.

Example

[root@basecm11 ~]# cm-cmd-ports --https 8082

A restart of CMDaemon has the change take effect, and takes care of opening the firewall on port 8082 for CMDaemon, by adding a line to the rules file of Shorewall. The original port 8081 remains open, but CMDaemon no longer listens to it.

The status of ports used by the cluster manager can be listed with:

[root@basecm11 ~]# cm-cmd-ports -l
type            http      https  firewall rule    path
--------------  ------  -------  ---------------  -------------------------------------------------------
image           8080       8082                   /cm/images/default-image/cm/local/apps/cmd/etc/cmd.conf
image           8080       8082  True             /cm/local/apps/cmd/etc/cmd.conf
node-installer             8082                   /cm/node-installer/scripts/node-installer.conf

7.2.3 Clear And Stop Behavior In service Options, bash Shell Command, And cmsh Shell

To remove all rules, for example for testing purposes, the clear option should be used from the Unix shell. This then allows all network traffic through:

shorewall clear

Administrators should be aware that in the Linux distributions supported by BCM, the service shorewall stop command corresponds to the unix shell shorewall stop command, and not to the unix shell shorewall clear command. The stop option for the service and shell blocks network traffic but allows a pre-defined minimal safe set of connections, and is not the same as completely removing Shorewall from consideration. The stop options discussed so far should not be confused with the equivalent stop option in the cmsh shell under services mode under device mode.

This situation is indicated in the following table:

*Correspondence of stop, clear, and restart in Shorewall vs cmsh*
iptables rulesServiceUnix Shellcmsh shell
keep a safe set:service shorewall stopshorewall stopno equivalent
clear all rules:no equivalentshorewall clearstop shorewall
start with stored rules:service shorewall restartshorewall restartrestart shorewall

7.2.4 Adding To Shorewall Configuration Via A Role In cmsh

This section, about adding to Shorewall configuration via a role in cmsh, requires some familiarity with BCM and Shorewall, and can be skipped at a first reading of the manuals. For convenience, cross-references are provided within this section to the material that the administrator should have some familiarity with before configuring Shorewall via a role.

The command line front end manager to BCM is cmsh (section 2.5 of the Administrator Manual). Within cmsh, services are often implemented as roles (section 2.1.5 of the Administrator Manual).

Administrators that have become familiar with using roles within cmsh can carry out some Shorewall configuration within the firewall role of the head node, via modes (section 2.5.2 of the Administrator Manual) under the firewall role. Firewall modes were introduced in BCM version 9.1.

Shorewall configuration can go far beyond just what can be done within the modes of firewall. These modes are just the BCM way to add changes to some Shorewall firewall configurations, and are aimed at making firewall management for the cluster easier.

BCM provides modes for modifying Shorewall files as indicated by the following table:

modeused for modifying:Shorewall file man pages
policiespolicies in /etc/shorewall/policyhttps://shorewall.org/manpages/shorewall-policy.html
routesroutes /etc/shorewall/routeshttps://shorewall.org/manpages/shorewall-routes.html (not present in Ubuntu clusters by default)
interfacesinterfaces in /etc/shorewall/interfaceshttps://shorewall.org/manpages/shorewall-interfaces.html
zoneszones in /etc/shorewall/zoneshttps://shorewall.org/manpages/shorewall-zones.html
openportsports in /etc/shorewall/ruleshttps://shorewall.org/manpages/shorewall-rules.html

For example, to add a policy to the BCM-managed section of the /etc/shorewall/policy file on the head node, a cmsh session can be run as follows:

Example

[root@head ~]# cmsh
[head]% device use head; roles
[head->device[head]->roles]% use firewall
[head->device[head]->roles[firewall]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             firewall
Revision
Type                             FirewallRole
Add services                     yes
Shorewall                        no
Open ports                       <0 in submode>
Zones                            <0 in submode>
Interfaces                       <0 in submode>
Policies                         <0 in submode>
Routes                           <0 in submode>
[head->device[head]->roles[firewall]]% policies
[head->device[head]->roles[firewall]->policies]% list
Index  Source Dest   Policy       Log       Options
------ ------ ------ ------------ --------- ------------

The preceding output shows no additional policies are currently managed by BCM in that mode. To add some BCM-managed policies, the administrator can check how to do it by typing in the add command (section 2.5.3.2 of the Administrator Manual) without any arguments:

[head->device[head]->roles[firewall]->policies]% add
Name:
          Create a new firewallpolicy with specified policy
Usage:
          add <policy>
          add <source>
          add <source> <dest>
          add <source> <dest> <policy>
          add <source> <dest> <policy> <log>
          add <source> <dest> <policy> <log> <options>
Examples:
          add loc net ACCEPT
          add loc net ACCEPT info

Based on the lookup and some familiarity with Shorewall’s policy file, the administrator can now compose suitable arguments for the add command, and commit (section 2.5.3.2 of the Administrator Manual) the changes:

Example

[head->device[head]->roles[firewall]->policies]% add net fw ACCEPT info
[head->device*[head*]->roles*[firewall*]->policies[0]]% #preceding means accept all traffic from other subnets
[head->device*[head*]->roles*[firewall*]->policies[0]]% list
Index  Source Dest   Policy       Log       Options
------ ------ ------ ------------ --------- ------------
0      net    fw     ACCEPT       info
[head->device*[head*]->roles*[firewall*]->policies[0]]% #next lines show how policy section changes on commit
[head->device*[head*]->roles*[firewall*]->policies[0]]% !head -4 /etc/shorewall/policy
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
# END AUTOGENERATED SECTION   -- DO NOT REMOVE
[head->device*[head*]->roles*[firewall*]->policies[0]]% commit
[head->device[head]->roles[firewall]->policies]% !head -4 /etc/shorewall/policy
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
net     fw      ACCEPT  info
# END AUTOGENERATED SECTION   -- DO NOT REMOVE

After the commit command is run, the additional policy is placed and becomes active in the policy file.

Similarly, to open a port on the head node via the openports mode, a cmsh session can be run as follows:

Example

[head->device[head]->roles[firewall]]% openports
[head->device[head]->roles[firewall]->openports]% add
Name:
          Create a new firewallopenport with specified port
Usage:
          add <port>
          add <description> <port>
          add <action> <network> <dest|port> <protocol> <dest>
          add <action> <network> <dest|port> <protocol> <dest> <cidr>
          add <action> <network> <dest|port> <protocol> <dest> <cidr> <description>
Examples:
          add 8080
          add grafana 3000
          add ACCEPT net 8080
          add ACCEPT net 8080 tcp
          add ACCEPT net 8080 tcp fw
          add ACCEPT net 8080 tcp fw 10.3.9.0/24
          add ACCEPT net 8080 tcp fw 10.3.9.0/24 "nginx proxy"

Based on the lookup, the administrator can compose suitable arguments for the add command, and commit (section 2.5.3.2 of the Administrator Manual) the changes:

Example

[head->device[head]->roles[firewall]->openports]% add ACCEPT net 443 tcp fw
[head->device*[head*]->roles*[firewall*]->openports*[0]]% commit
[head->device[head]->roles[firewall]->openports]% !tail -6 /etc/shorewall/rules
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
ACCEPT   net         fw              TCP     443
# END AUTOGENERATED SECTION   -- DO NOT REMOVE

After the commit command is run, the new port rule is placed in the rules file, and the shorewall service is restarted to apply it.

7.2.5 Further Shorewall Quirks

7.2.5.1 Shorewall Is Normally Only On The Head Node

Shorewall normally runs on the head node.

Shorewall can also run on the regular nodes, but does not run on them by default.

In a type 1 network topology (section 3.3.9), running Shorewall on the regular nodes makes no real sense from a security point of view. This is because the head node controls all the regular nodes, and access to the regular nodes is through the head node firewall.

For type 2 and type 3 topologies, turning Shorewall on for the regular nodes must be considered. It can be turned on by setting the shorewall attribute in the firewall role for the node from off to on.

Example

[basecm11->device*[node001*]->roles*[firewall*]]% get shorewall
no
[basecm11->device*[node001*]->roles*[firewall*]]% !ssh node001 systemctl status shorewall.service
...
Active: inactive (dead)
...
[basecm11->device*[node001*]->roles*[firewall*]]% set shorewall on
[basecm11->device*[node001*]->roles*[firewall*]]% commit ; show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             firewall
Revision
Type                             FirewallRole
Add services                     yes
Shorewall                        yes
...
[basecm11->device[node001]->roles[firewall]]% !systemctl status shorewall.service
ssh: connect to host node001 port 22: Connection refused

7.2.5.2 Standard Distribution Firewall Should Be Disabled

Administrators should also be aware that RHEL and its derivatives run their own set of high-level iptables setup scripts if the standard distribution firewall is enabled. To avoid conflict, the standard distribution firewall, Firewalld, must stay disabled in RHEL8 and 9 and their derivatives. This is because BCM requires Shorewall for regular functioning. Shorewall can be configured to set up whatever iptables rules are installed by the standard distribution script instead.

7.2.5.3 Shorewall Stopped Outside Of BCM Considered Harmful

System administrators wishing to stop Shorewall should note that BCM by default has the autostart setting (section 3.15 of the Administrator Manual) set to on. With such a value, CMDaemon attempts to restart a stopped Shorewall if the service has been stopped from outside of cmsh or Base View.

Stopping Shorewall outside of cmsh or Base View is considered harmful, because it can trigger a failover. This is because stopping Shorewall blocks the failover prevention monitoring tests. These tests are the ping and failoverping checks (both based on SYN connections), and the CMDaemon status (based on REST calls) (section 15.4.2.1 of the Administrator Manual). In most cases, with default settings, Shorewall is not restarted in time, even when autostart is on, so that a failover then takes place.

A failover procedure is quite a sensible option when Shorewall is stopped from outside of cmsh or Base View, because besides the failover monitoring tests failing, other failures also make the head node pretty useless. The blocking of ports means that, amongst others, workload managers and NFS shares are also unable to connect. Ideally, therefore, Shorewall should not be stopped outside cmsh or Base View in the first place.

Full documentation on the specifics of Shorewall is available at http://www.shorewall.net.

7.3 The GCC Compiler

The GCC compiler may be installed through yum, zypper, or apt (section 9.2 of the Administrator Manual).

The package names are

  • gcc-recent for RHEL and derivatives, and SLES

  • cm-gcc for Ubuntu

The GCC suite that the distribution provides is also present by default.

An alternative to the GCC compiler is the Intel compiler, which is part of the oneAPI toolkit, available from https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html. Earlier versions of BCM had the Intel compiler components available from the BCM repositories; this is no longer the case for BCM 10.0 and onward.

7.4 AMD GPU Driver Installation

AMD GPUs require drivers to be installed in order to work. The DKMS system, which is used to compile kernel modules that are not part of the mainline kernel, is used for creating AMD GPU kernel modules on the image to be used by the nodes that have GPUs.

7.4.1 AMD GPU Hardware Check

A hardware check to see if the node has an AMD GPU is:

Example

[root@node001 ~]# lspci | grep AMD
    00:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10\
     [Instinct MI25/MI25x2/V340/V320]

If the kernel has detected the AMD GPU, then a grep amdgpu of the dmesg output shows all sorts of messages that indicate that:

Example

[root@node001 ~]# dmesg | grep amdgpu
[  157.942430] [drm] amdgpu kernel modesetting enabled.
[  157.945225] amdgpu: CRAT table not found
[  157.945241] amdgpu: Virtual CRAT table created for CPU
[  157.945265] amdgpu: Topology: Add CPU node
[  158.211025] amdgpu 0000:00:06.0: amdgpu: Fetched VBIOS from ROM
[  158.211065] amdgpu: ATOM BIOS: 113-D0513600-109
[  158.226062] amdgpu 0000:00:06.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[  159.253402] amdgpu 0000:00:06.0: amdgpu: MEM ECC is active.
[  159.253418] amdgpu 0000:00:06.0: amdgpu: SRAM ECC is not presented.
[  159.253465] amdgpu 0000:00:06.0: amdgpu: VRAM: 16368M 0x000000F400000000 - 0x000000F7FEFFFFFF (16368M used)
...

7.4.2 AMD GPU Driver Installation Per Supported Distribution

Across the distributions—RHEL and derivatives, Ubuntu, and SLES—the procedure to install the GPU drivers is very similar.

7.4.2.1 Cloning An Image For AMD GPU Nodes

In all distributions, the default-image can first be cloned to an image that is to be the AMD GPU image, for example am:

Example

[root@basecm11:~]# cmsh
[basecm11]% softwareimage
[basecm11->softwareimage]% clone default-image am
[basecm11->softwareimage*[am*]]% commit
[basecm11->softwareimage[am]]%
[notice] basecm11: Started to copy: /cm/images/default-image -> /cm/images/am (4117)
...
[notice] basecm11: Initial ramdisk for image am was generated successfully
[basecm11->softwareimage[am]]% quit

7.4.2.2 AMD Instructions For Installing The Driver Packages On AMD GPU Nodes

Instructions from AMD describe how to install the driver packages. At the time of writing (February 2024) the instructions are at:
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
The instructions describe two ways of installing the driver:

  1. configuring the AMD driver repository for the package manager, and then installing the driver via standard package manager commands and OS changes

  2. installing an AMDGPU installer that takes care of the installation

In either case, the instructions that are followed must be the ones for the OS release version of the software image that is on the AMD GPU nodes. The release version can be found with:

Example

root@basecm11:~# grep PRETTY /etc/os-release
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"

7.4.2.3 Installing The Driver Packages On AMD GPU Nodes On The Node Images Via Chroot

The installation must be done in the image. This is done using a chroot into the image, and using a bind mount to have some special filesystem directories (/proc, /sys, and similar) be available during the package installation. This is needed for the DKMS installation.

Bind mounting the filesystems and then chrooting is a little tedious, so the cm-chroot-sw-img utility (here) is used to automate the job.

The following session output illustrates (with much text elided) the procedure for the second of the two ways of installing the driver. That is, installing and running the AMDGPU installer on a software image:

The chrooted image created earlier (am) is first entered:

Example

[root@basecm11 ~]# cm-chroot-sw-img /cm/images/am/
mounted /cm/images/default-image/dev
mounted /cm/images/default-image/dev/pts
...

The amdgpu-install package can then be installed, as per the AMD instructions. The package provides the AMDGPU installer (amdgpu-install), which is run in a chrooted image to install the ROCm stack. Installation takes a while, with much of the time taken up by the compilation steps that are carried out for the DKMS parts.

  • For RHEL 9.2 and derivatives the procedure looks like:

    Example

    [root@am /]# urldomain=https://repo.radeon.com
    [root@am /]# urlpath=/amdgpu-install/6.0.2/rhel/9.2/amdgpu-install-6.0.60002-1.el9.noarch.rpm
    [root@am /]# yum install $urldomain$urlpath
    ...
    [root@am /]# amdgpu-install --usecase=graphics,rocm
    ...

    The exact path depends on which version is used.

  • For Ubuntu 22.04 (Jammy Jellyfish) the procedure looks like:

    Example

    root@am:/# urldomain=https://repo.radeon.com
    root@am:/# urlpath=/amdgpu-install/6.0.2/ubuntu/jammy/amdgpu-install_6.0.60002-1_all.deb
    root@am:/# wget $urldomain$urlpath
    ...
    root@am:/# apt install ./amdgpu-install_6.0.60002-1_all.deb
    ...
    root@am:/# amdgpu-install --usecase=rocm
    ...

  • For SLES15 the procedure for SLES15sp5 looks like:

    Example

    am:/ # urldomain=https://repo.radeon.com
    am:/ # urlpath=/amdgpu-install/latest/sle/15.5/amdgpu-install-6.0.60000-1.noarch.rpm
    am:/ # zypper --no-gpg-checks install $urldomain$urlpath
    ...
    am:/ # amdgpu-install --usecase=graphics,rocm
    ...

    The exact path depends on which service pack (sp) version is in use on the node.

7.4.2.4 Replacing The rocm-smi Package With A Package From BCM On The AMD GPU Nodes Image

Running the AMDGPU installer installs the rocm-smi command, which is used for some monitoring tasks related to AMD GPUs. BCM requires its own version of this command to collect AMD GPU statistics. If such statistics are needed, then the BCM cm-rocm-smi package should be installed in the image too, to provide the BCM version:

    [root@am /]# yum install cm-rocm-smi   for RHEL and derivatives

For Ubuntu, the equivalent installation command is apt install cm-rocm-smi, while for SLES it is zypper install cm-rocm-smi.

The chroot for the prepared image can now be exited, which automatically unmounts the bind mounts:

Example

[root@am /]# exit
umounted /cm/images/am/dev/pts
umounted /cm/images/am/dev
umounted /cm/images/am/proc
umounted /cm/images/am/sys
umounted /cm/images/am/run

7.4.2.5 Using The Image For The AMD GPU Nodes

The nodes that are to use the driver should then be set to use the new image, and should be rebooted:

Example

[root@basecm11 ~]# cmsh
[basecm11]% device use node001
[basecm11->device[node001]]% set softwareimage am
[basecm11->device*[node001*]]% commit
[basecm11->device[node001]]% reboot node001

Normal nodes without the AMD GPU also boot up without crashing if they are set to use this image, but will not be able to run OpenCL programs.

After the configuration file has been placed, the ldconfig command is run, still within chroot, to link the library in the image(s).