> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Third Party Software In this chapter, several third party software packages included in the BCM repository are described briefly. For all packages, references to the complete documentation are provided. The packages dashboard at [https://service.bcm.nvidia.com/packages-dashboard/](https://service.bcm.nvidia.com/packages-dashboard/) lists the available supported package versions per BCM version, architecture, and distribution. ## 7.1 Modules Environment RHEL and derivatives, and SLES BCM package name: `env-modules` Ubuntu package name: `cm-modules` The *modules environment* package is installed by default on the head node. The home page for the software is at [http://modules.sourceforge.net/](http://modules.sourceforge.net/). The software allows a user of a cluster to modify the shell environment for a particular application, or even for a particular version of an application. Typically, a module file defines additions to environment variables such as `PATH`, `LD_LIBRARY_PATH`, and `MANPATH`. Cluster users use the `module` command to load or remove modules from their environment. The `module(1)` man page has more details about the command, and aspects of the modules environment that are relevant for administrators are discussed in [section 2.2 of the *Administrator Manual*](/admin/clustermanagement#22-modules-environment). Also discussed there is Lmod, the Lua-based alternative to the Tcl-based traditional modules environment package. The modules environment from a user’s perspective is covered in [section 2.3 of the *User Manual*](/user/clusterusage#23-environment-modules). ## 7.2 Shorewall Package name: `shorewall` ### 7.2.1 The Shorewall Service Paradigm BCM provides the Shoreline Firewall (more commonly known as “Shorewall”) package from the BCM repository. The package provides firewall and gateway functionality on the head node of a cluster. Shorewall is a flexible and powerful high-level interface for the netfilter packet filtering framework. Netfilter is a standard part of Linux kernels. As its building blocks, Shorewall uses `iptables` and `iptables6` commands to configure netfilter. All aspects of firewall and gateway configuration are handled through the configuration files located under `/etc/shorewall/`. Shorewall IPv4 configuration is managed with the `shorewall` command, while IPv6 configuration is managed via the `shorewall6` command. IPv4 filtering and IPv6 filtering are treated as separate services in Shorewall. For convenience, only IPv4 Shorewall is described from here onward, because IPv6 management is largely similar. After modifying Shorewall configuration files, Shorewall must be restarted to have the new configuration take effect. From the shell prompt, this can be carried out with: ```bash service shorewall restart ``` In NVIDIA Base Command Manager 10, Shorewall is managed by CMDaemon, in order to handle the automation of cloud node access. Restarting Shorewall can thus also be carried out within the `services` submode ([section 3.14 of the *Administrator Manual*](/admin/configuring#314-managing-and-configuring-services)) of `cmsh` on the head node. For example, on a head node `basecm10` a restart of `shorewall` can be carried out with: ```bash [basecm10->device[basecm10]->services[shorewall]]% restart restart Successfully restarted service shorewall on: basecm10 ``` System administrators may sometimes need to be aware that Shorewall does not really run as a daemon process. That is, the command to restart the service does not stop and start a `shorewall` daemon. Instead the command configures netfilter using the iptables settings specified in the shorewall configuration files, and then exits. It exits without leaving a `shorewall` process up and running, even though `service shorewall status` implies that such a service is running. ### 7.2.2 Shorewall Zones, Policies, And Rules In the default setup, Shorewall provides gateway functionality to the internal cluster network on the first network interface (`eth0`). This network is known as the `nat` zone to Shorewall. The external network (i.e. the connection to the outside world) is assumed to be on the second network interface (`eth1`). This network is known as the `net` zone in Shorewall. Letting BCM take care of the network interfaces settings is recommended for all interfaces on the head node ([section 3.2 of the *Administrator Manual*](/admin/configuring#32-network-settings)). The file `/etc/shorewall/interfaces` is generated by the cluster management daemon, and any extra instructions that cannot be added via Base View or `cmsh` can be added outside of the file section clearly demarcated as being maintained by CMDaemon. Shorewall is configured by default (through `/etc/shorewall/policy`) to deny all incoming traffic from the `net` zone, except for the traffic that has been explicitly allowed in `/etc/shorewall/rules`. Providing (a subset of) the outside world with access to a service running on a cluster, can be accomplished by creating appropriate rules in `/etc/shorewall/rules`. By default, the cluster responds to ICMP ping packets. Also, during cluster installation, the following ports are open by default, but can be set to be blocked by the administrator: * SSH * HTTP * HTTPS * port 8081, which allows access to the cluster management daemon. #### 7.2.2.1 The `cm-cmd-ports` Utility Port 8081 is the default port that CMDaemon listens to when using the HTTPS protocol to manage the nodes. If, for example, a new software needs that port, then the `cm-cmd-ports` utility can be used to move the port that CMDaemon listens on, to another port ([here](/admin/cmdconf#c401-changing-cmdaemon-ports)). ### 7.2.3 Clear And Stop Behavior In `service` Options, `bash` Shell Command, And `cmsh` Shell To remove all rules, for example for testing purposes, the `clear` option should be used from the Unix shell. This then allows all network traffic through: ```bash shorewall clear ``` Administrators should be aware that in the Linux distributions supported by BCM, the `service shorewall stop` command corresponds to the unix shell `shorewall stop` command, and not to the unix shell `shorewall clear` command. The `stop` option for the service and shell blocks network traffic but allows a pre-defined minimal safe set of connections, and is not the same as completely removing Shorewall from consideration. The stop options discussed so far should not be confused with the equivalent stop option in the `cmsh` shell under services mode under`device` mode. This situation is indicated in the following table:
*Correspondence of `stop`, `clear`, and `restart` in Shorewall vs `cmsh`*
**iptables rules** **Service** **Unix Shell** **`cmsh` shell**
keep a safe set: service shorewall stop shorewall stop *no equivalent*
clear all rules: *no equivalent* shorewall clear stop shorewall
start with stored rules: service shorewall restart shorewall restart restart shorewall
### 7.2.4 Adding To Shorewall Configuration Via A Role In `cmsh` This section, about adding to Shorewall configuration via a role in `cmsh`, requires some familiarity with BCM and Shorewall, and can be skipped at a first reading of the manuals. For convenience, cross-references are provided within this section to the material that the administrator should have some familiarity with before configuring Shorewall via a role. The command line front end manager to BCM is `cmsh` ([section 2.5 of the *Administrator Manual*](/admin/clustermanagement#25-cluster-management-shell)). Within `cmsh`, services are often implemented as roles ([section 2.1.5 of the *Administrator Manual*](/admin/clustermanagement#215-roles)). Administrators that have become familiar with using roles within `cmsh` can carry out some Shorewall configuration within the `firewall` role of the head node, via modes ([section 2.5.2 of the *Administrator Manual*](/admin/clustermanagement#252-levels-modes-help-and-commands-syntax-in-cmsh)) under the `firewall` role. Firewall modes were introduced in BCM version 9.1. Shorewall configuration can go far beyond just what can be done within the modes of `firewall`. These modes are just the BCM way to add changes to some Shorewall firewall configurations, and are aimed at making firewall management for the cluster easier. BCM provides modes for modifying Shorewall files as indicated by the following table:
**mode** **used for modifying:** **Shorewall file man pages**
`policies` policies in `/etc/shorewall/policy` [https://shorewall.org/manpages/shorewall-policy.html](https://shorewall.org/manpages/shorewall-policy.html)
`routes` routes `/etc/shorewall/routes` [https://shorewall.org/manpages/shorewall-routes.html](https://shorewall.org/manpages/shorewall-routes.html) (not present in Ubuntu clusters by default)
`interfaces` interfaces in `/etc/shorewall/interfaces` [https://shorewall.org/manpages/shorewall-interfaces.html](https://shorewall.org/manpages/shorewall-interfaces.html)
`zones` zones in `/etc/shorewall/zones` [https://shorewall.org/manpages/shorewall-zones.html](https://shorewall.org/manpages/shorewall-zones.html)
`openports` ports in `/etc/shorewall/rules` [https://shorewall.org/manpages/shorewall-rules.html](https://shorewall.org/manpages/shorewall-rules.html)
For example, to add a policy to the BCM-managed section of the `/etc/shorewall/policy` file on the head node, a `cmsh` session can be run as follows: **Example** ```bash [root@head ~]# cmsh [head]% device use head; roles [head->device[head]->roles]% use firewall [head->device[head]->roles[firewall]]% show Parameter Value -------------------------------- ------------------------------------------------ Name firewall Revision Type FirewallRole Add services yes Shorewall no Open ports <0 in submode> Zones <0 in submode> Interfaces <0 in submode> Policies <0 in submode> Routes <0 in submode> [head->device[head]->roles[firewall]]% policies [head->device[head]->roles[firewall]->policies]% list Index Source Dest Policy Log Options\nopagebreak ------ ------ ------ ------------ --------- ------------ ``` The preceding output shows no additional policies are currently managed by BCM in that mode. To add some BCM-managed policies, the administrator can check how to do it by typing in the `add` command ([section 2.5.3.2 of the *Administrator Manual*](/admin/clustermanagement#2532-working-with-objects-add-commit-remove)) without any arguments: ```bash [head->device[head]->roles[firewall]->policies]% add Name: Create a new firewallpolicy with specified policy Usage: add add add add add add Examples: add loc net ACCEPT add loc net ACCEPT info ``` Based on the lookup and some familiarity with Shorewall’s policy file, the administrator can now compose suitable arguments for the `add` command, and `commit` ([section 2.5.3.2 of the *Administrator Manual*](/admin/clustermanagement#2532-working-with-objects-add-commit-remove)) the changes: **Example** ```bash [head->device[head]->roles[firewall]->policies]% add net fw ACCEPT info [head->device*[head*]->roles*[firewall*]->policies[0]]% #preceding means accept all traffic from other subnets [head->device*[head*]->roles*[firewall*]->policies[0]]% list Index Source Dest Policy Log Options ------ ------ ------ ------------ --------- ------------ 0 net fw ACCEPT info [head->device*[head*]->roles*[firewall*]->policies[0]]% #next lines show how policy section changes on commit [head->device*[head*]->roles*[firewall*]->policies[0]]% !head -4 /etc/shorewall/policy # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE # END AUTOGENERATED SECTION -- DO NOT REMOVE [head->device*[head*]->roles*[firewall*]->policies[0]]% commit [head->device[head]->roles[firewall]->policies]% !head -4 /etc/shorewall/policy # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE net fw ACCEPT info # END AUTOGENERATED SECTION -- DO NOT REMOVE ``` After the `commit` command is run, the additional policy is placed and becomes active in the `policy` file. ### 7.2.5 Further Shorewall Quirks #### 7.2.5.1 Shorewall Is Normally Only On The Head Node Shorewall always runs on the head node. Shorewall can also run on the regular nodes, but does not run on them by default. In a type 1 network topology (section [3.3.9](/installation/installing#339-network-topology)), running Shorewall on the regular nodes makes no real sense from a security point of view. This is because the head node controls all the regular nodes, and access to the regular nodes is through the head node firewall. For type 2 and type 3 topologies, turning Shorewall on for the regular nodes must be considered. It can be turned on by setting the `shorewall` attribute in the `firewall` role for the node from `off` to `on`. **Example** ```bash [basecm10->device*[node001*]->roles*[firewall*]]% get shorewall no [basecm10->device*[node001*]->roles*[firewall*]]% !ssh node001 systemctl status shorewall.service ... Active: inactive (dead) ... [basecm10->device*[node001*]->roles*[firewall*]]% set shorewall on [basecm10->device*[node001*]->roles*[firewall*]]% commit ; show Parameter Value -------------------------------- ------------------------------------------------ Name firewall Revision Type FirewallRole Add services yes Shorewall yes ... [basecm10->device[node001]->roles[firewall]]% !systemctl status shorewall.service ssh: connect to host node001 port 22: Connection refused ``` #### 7.2.5.2 Standard Distribution Firewall Should Be Disabled Administrators should also be aware that RHEL and its derivatives run their own set of high-level iptables setup scripts if the standard distribution firewall is enabled. To avoid conflict, the standard distribution firewall, Firewalld, must stay disabled in RHEL8 and 9 and their derivatives. This is because BCM requires Shorewall for regular functioning. Shorewall can be configured to set up whatever iptables rules are installed by the standard distribution script instead. #### 7.2.5.3 Shorewall Stopped Outside Of BCM Considered Harmful System administrators wishing to stop Shorewall should note that BCM by default has the `autostart` setting ([section 3.14 of the *Administrator Manual*](/admin/configuring#314-managing-and-configuring-services)) set to `on`. With such a value, CMDaemon attempts to restart a stopped Shorewall if the service has been stopped from outside of `cmsh` or Base View. Stopping Shorewall outside of `cmsh` or Base View is considered harmful, because it can trigger a failover. This is because stopping Shorewall blocks the failover prevention monitoring tests. These tests are the `ping` and `failoverping` checks (both based on TCP SYN connections), and the CMDaemon status (based on REST calls) ([section 17.4.2.1 of the *Administrator Manual*](/admin/ha#17421-cmha-status-querying-ha-status)). In most cases, with default settings, Shorewall is not restarted in time, even when `autostart` is on, so that a failover then takes place. A failover procedure is quite a sensible option when Shorewall is stopped from outside of `cmsh` or Base View, because besides the failover monitoring tests failing, other failures also make the head node pretty useless. The blocking of ports means that, amongst others, workload managers and NFS shares are also unable to connect. Ideally, therefore, Shorewall should not be stopped outside `cmsh` or Base View in the first place. Full documentation on the specifics of Shorewall is available at [http://www.shorewall.net](http://www.shorewall.net). ## 7.3 The GCC Compiler The GCC compiler may be installed through `yum`, `zypper`, or `apt` ([section 11.2 of the *Administrator Manual*](/admin/softwareimagemanagement#112-managing-packages-on-the-head-node)). The package names are * `gcc-recent` for RHEL and derivatives, and SLES * `cm-gcc` for Ubuntu The GCC suite that the distribution provides is also present by default. An alternative to the GCC compiler is the Intel compiler, which is part of the oneAPI toolkit, available from [https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html](https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html). Earlier versions of BCM had the Intel compiler components available from the BCM repositories; this is no longer the case for 10.0 and onward. ## 7.4 AMD GPU Driver Installation AMD GPUs require drivers to be installed in order to work. The DKMS system, which is used to compile kernel modules that are not part of the mainline kernel, is used for creating AMD GPU kernel modules on the image to be used by the nodes that have GPUs. ### 7.4.1 AMD GPU Hardware Check A hardware check to see if the node has an AMD GPU is: **Example** ```bash [root@node001 ~]# lspci | grep AMD 00:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10\ [Instinct MI25/MI25x2/V340/V320] ``` If the kernel has detected the AMD GPU, then a `grep amdgpu` of the `dmesg` output shows all sorts of messages that indicate that: **Example** ```bash [root@node001 ~]# dmesg | grep amdgpu [ 157.942430] [drm] amdgpu kernel modesetting enabled. [ 157.945225] amdgpu: CRAT table not found [ 157.945241] amdgpu: Virtual CRAT table created for CPU [ 157.945265] amdgpu: Topology: Add CPU node [ 158.211025] amdgpu 0000:00:06.0: amdgpu: Fetched VBIOS from ROM [ 158.211065] amdgpu: ATOM BIOS: 113-D0513600-109 [ 158.226062] amdgpu 0000:00:06.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported [ 159.253402] amdgpu 0000:00:06.0: amdgpu: MEM ECC is active. [ 159.253418] amdgpu 0000:00:06.0: amdgpu: SRAM ECC is not presented. [ 159.253465] amdgpu 0000:00:06.0: amdgpu: VRAM: 16368M 0x000000F400000000 - 0x000000F7FEFFFFFF (16368M used) ... ``` ### 7.4.2 AMD GPU Driver Installation Per Supported Distribution Across the distributions—RHEL and derivatives, Ubuntu, and SLES—the procedure to install the GPU drivers is very similar. #### 7.4.2.1 Cloning An Image For AMD GPU Nodes In all distributions, the `default-image` can first be cloned to an image that is to be the AMD GPU image, for example `am`: **Example** ```bash [root@basecm10:~]# cmsh [basecm10]% softwareimage [basecm10->softwareimage]% clone default-image am [basecm10->softwareimage*[am*]]% commit [basecm10->softwareimage[am]]% [notice] basecm10: Started to copy: /cm/images/default-image -> /cm/images/am (4117) ... [notice] basecm10: Initial ramdisk for image am was generated successfully [basecm10->softwareimage[am]]% quit ``` #### 7.4.2.2 AMD Instructions For Installing The Driver Packages On AMD GPU Nodes Instructions from AMD describe how to install the driver packages. At the time of writing (February 2024) the instructions are at:\ [https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html)\ The instructions describe two ways of installing the driver: 1. configuring the AMD driver repository for the package manager, and then installing the driver via standard package manager commands and OS changes 2. installing an AMDGPU installer that takes care of the installation In either case, the instructions that are followed must be the ones for the OS release version of the software image that is on the AMD GPU nodes. The release version can be found with: **Example** ```bash root@basecm10:~# grep PRETTY /etc/os-release PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)" ``` #### 7.4.2.3 Installing The Driver Packages On AMD GPU Nodes On The Node Images Via Chroot The installation must be done in the image. This is done using a chroot into the image, and using a bind mount to have some special filesystem directories (`/proc`, `/sys`, and similar) be available during the package installation. This is needed for the DKMS installation. Bind mounting the filesystems and then chrooting is a little tedious, so the `cm-chroot-sw-img` utility ([here](/admin/softwareimagemanagement#11412-change-root-with-chroot-then-running-the-package-manager-commands)) is used to automate the job. The following session output illustrates (with much text elided) the procedure for the second of the two ways of installing the driver. That is, installing and running the AMDGPU installer on a software image: The chrooted image created earlier (`am`) is first entered: **Example** ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/am/ mounted /cm/images/default-image/dev mounted /cm/images/default-image/dev/pts ... ``` [The `amdgpu-install` package can then be installed, as per the AMD instructions](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html#rocm-amdgpu-quick). The package provides the AMDGPU installer (`amdgpu-install`), which is run in a chrooted image to install the ROCm stack. Installation takes a while, with much of the time taken up by the compilation steps that are carried out for the DKMS parts. * For RHEL 9.2 and derivatives the procedure looks like: **Example** ```bash [root@am /]# urldomain=https://repo.radeon.com [root@am /]# urlpath=/amdgpu-install/6.0.2/rhel/9.2/amdgpu-install-6.0.60002-1.el9.noarch.rpm [root@am /]# yum install $urldomain$urlpath ... [root@am /]# amdgpu-install --usecase=graphics,rocm ... ``` The exact path depends on which version is used. * For Ubuntu 22.04 (Jammy Jellyfish) the procedure looks like: **Example** ```bash root@am:/# urldomain=https://repo.radeon.com root@am:/# urlpath=/amdgpu-install/6.0.2/ubuntu/jammy/amdgpu-install_6.0.60002-1_all.deb root@am:/# wget $urldomain$urlpath ... root@am:/# apt install ./amdgpu-install_6.0.60002-1_all.deb ... root@am:/# amdgpu-install --usecase=rocm ... ``` If the node is using Ubuntu 20.04 (Focal Fossa) then the `urlpath` needs to be changed from `jammy` to `focal`. * For SLES15 the procedure for SLES15sp5 looks like: **Example** ```bash am:/ # urldomain=https://repo.radeon.com am:/ # urlpath=/amdgpu-install/latest/sle/15.5/amdgpu-install-6.0.60000-1.noarch.rpm am:/ # zypper --no-gpg-checks install $urldomain$urlpath ... am:/ # amdgpu-install --usecase=graphics,rocm ... ``` The exact path depends on which service pack (sp) version is in use on the node. #### 7.4.2.4 Replacing The `rocm-smi` Package With A Package From BCM On The AMD GPU Nodes Image Running the AMDGPU installer installs the `rocm-smi` command, which is used for some monitoring tasks related to AMD GPUs. BCM requires its own version of this command to collect AMD GPU statistics. If such statistics are needed, then the BCM `cm-rocm-smi` package should be installed in the image too, to provide the BCM version: ```bash [root@am /]# yum install cm-rocm-smi for RHEL and derivatives ``` For Ubuntu, the equivalent installation command is `apt install cm-rocm-smi`, while for SLES it is `zypper install cm-rocm-smi`. The chroot for the prepared image can now be exited, which automatically unmounts the bind mounts: **Example** ```bash [root@am /]# exit umounted /cm/images/am/dev/pts umounted /cm/images/am/dev umounted /cm/images/am/proc umounted /cm/images/am/sys umounted /cm/images/am/run ``` #### 7.4.2.5 Using The Image For The AMD GPU Nodes The nodes that are to use the driver should then be set to use the new image, and should be rebooted: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% set softwareimage am [basecm10->device*[node001*]]% commit [basecm10->device[node001]]% reboot node001 ``` Normal nodes without the AMD GPU also boot up without crashing if they are set to use this image, but will not be able to run OpenCL programs. After the configuration file has been placed, the `ldconfig` command is run, still within chroot, to link the library in the image(s).