Quickstart Installation Guide | NVIDIA Base Command Manager

Download PDF

This chapter describes a basic and quick installation of NVIDIA Base Command Manager (BCM) on “bare metal” cluster hardware as a step-by-step process, and gives very little explanation of the steps. Following these steps should allow a moderately experienced cluster administrator to get a cluster up and running in a fairly standard configuration as quickly as possible. This would be without even having to read the introductory Chapter 2 of this manual, let alone the entire manual. References to chapters and sections are provided where appropriate.

Some asides, before getting on with the steps themselves:

If the cluster has already been installed, tested, and configured, but only needs to be configured now for a new network, then the administrator should only need to look at Chapter 6. Chapter 6 lays out how to carry out the most common configuration changes that usually need to be done to make the cluster work in the new network.
For administrators that are very unfamiliar with clusters, reading the introduction (Chapter 2) and then the more detailed installation walkthrough for a bare metal installation (Chapter 3, sections 3.1, 3.2, and 3.3) is recommended. Having carried out the head node installation, the administrator can then return to this quickstart chapter (Chapter 1), and continue onward with the quickstart process of regular node installation (section 1.3).
The configuration and administration of the cluster after it has been installed is covered in the BCM Administrator Manual. The Administrator Manual should be consulted for further background information as well as guidance on cluster administration tasks, after the introduction (Chapter 2) of the Installation Manual has been read.
If all else fails, administrator-level support is available via https://www.nvidia.com/en-us/data-center/bright-cluster-manager/support/. Section 16.2 of the Administrator Manual has further details on how to brief the support team, so that the issue can be resolved as quickly as possible.

The quickstart steps now follow:

1.1 Installing The Head Node

The head node does not need to be connected to the regular nodes at this point, though it helps to have the wiring done beforehand so that how things are connected is known.

The BIOS of the head node should have the local time set.
The head node should be booted from the BCM installation DVD or from a USB flash drive with the DVD ISO on it.
The option: Start Base Command Manager Graphical Installer, or Start Base Command Manager Text Installer, should be selected in the text boot menu. The graphical installation is recommended, and brings up the GUI installation welcome screen. The text installer provides a minimal ncurses-based version of the GUI installation.

Only the GUI installation is discussed in the rest of this quickstart for convenience.
At the GUI welcome screen, the Start installation button should be clicked.
At the EULA screens:
- At the NVIDIA EULA screen, the acceptance checkbox should be ticked. Next should then be ticked.
- At the Linux base distribution EULA screen, the acceptance checkbox should be ticked. Next should then be clicked.
At the Kernel modules screen, Next should be clicked.
At the Hardware info screen, the detected hardware should be reviewed. If additional kernel modules are required, then the administrator should go back to the Kernel Modules screen. Once all the relevant hardware (Ethernet interfaces, hard drive and DVD drive) is detected, Next should be clicked.
At the Installation source screen, the DVD drive containing the BCM DVD, or the USB flash drive containing the DVD ISO, should be selected, then Next clicked.
At the General cluster settings screen, one or more nameservers and one or more domains can be set, if they have not already been automatically filled. The remaining settings can usually be left as is.
At the Workload management screen, an HPC workload manager can be selected. The choice can be made later on too, after BCM has been installed.
For the Network topology screen, a Type 1 network is the most common.
For the Head node settings screen, the head is given a name and a password.
For the Compute nodes settings screen, the head is given a name and a password.
- The number of racks and regular nodes are specified
- The base name for the regular nodes is set. Accepting the default of node means nodes names are prefixed with node, for example: node001, node002…
- The number of digits to append to the base name is set. For example, accepting the default of 3 means nodes from node001 to node999 are possible names.
- The correct hardware manufacturer is selected
For the BMC configuration screen, the use of IPMI/iLO/DRAC/CIMC/Redfish BMCs is carried out. Adding an IPMI/iLO/DRAC/CIMC/Redfish network is needed to configure IPMI/iLO/DRAC/CIMC/Redfish interfaces in a different IP subnet, and is recommended.
At the Networks screen, the network parameters for the head node should be entered for the interface facing the network named externalnet:
- If using DHCP on that interface, the parameters for IP Address, Netmask and Gateway as suggested by the DHCP server on the external network can be accepted.
- If not using DHCP on that interface, static values put in instead.
The network parameters for externalnet that can be set include the:
- base address (also called the network address)
- netmask
- domain name
The network externalnet corresponds to the site network that the cluster resides in (for example, a corporate or campus network). The IP address details are therefore the details of the head node for a type 1 externalnet network (figure 3.10). A domain name should be entered to suit the local requirements.
For the Head node interfaces screen, the head node network interfaces are assigned networks and IP addresses. The assigned values can be reviewed and changed.
At the Compute node interfaces screen, the compute node interfaces are assigned networks and IP addresses. The assigned values can be reviewed and changed.
At the Disk layout screen, a drive should be selected for the head node. The installation will be done onto this drive, overwriting all its previous content.
At the Disk layout Settings screen, the administrator can modify the disk layout for the head node by selecting a pre-defined layout.

For hard drives that have less than about 500GB space, the XML file master-one-big-partition.xml in the installer file system is used by default:

Partition Space Mounted At Filesystem Type
1 1G /boot ext2
0 100M /boot/efi fat
2 16G - swap
3 rest / xfs

Default layout for up to 500GB: One big partition.

For hard drives that have about 500GB or more of space, the XML file master-standard.xml is used by default:

Partition Space Mounted At Filesystem Type
1 1G /boot ext2
0 100M /boot/efi fat
2 16G - swap
3 20G /tmp xfs
4 180G /var xfs
5 rest / xfs

Default layout for more than 500GB: Several partitions.

The layouts indicated by these tables may be fine-tuned by editing the XML partitioning definition during this stage. The “max” setting in the XML file corresponds to the “rest” entry in these tables, and means the rest of the drive space is used up for the associated partition, whatever the leftover space is.

There are layout templates available from a menu, other than the preceding default layouts.
At the Additional software screen, extra software options can be chosen for installation if these were selected for the installation ISO. The extra software options are:
- CUDA
- Ceph
- OFED stack
The Summary screen should be reviewed. A wrong entry can still be fixed at this point. The Next button then starts the installation.
The Deployment screen should eventually complete. Clicking on Reboot reboots the head node.

Partition	Space	Mounted At	Filesystem Type
1	1G	/boot	ext2
0	100M	/boot/efi	fat
2	16G	-	swap
3	rest	/	xfs

Partition	Space	Mounted At	Filesystem Type
1	1G	/boot	ext2
0	100M	/boot/efi	fat
2	16G	-	swap
3	20G	/tmp	xfs
4	180G	/var	xfs
5	rest	/	xfs

1.2 First Boot

The DVD should be removed or the boot-order altered in the BIOS to ensure that the head node boots from the first hard drive.
Once the machine is fully booted, a login should be done as root with the password that was entered during installation.
A check should be done to confirm that the machine is visible on the external network. Also, it should be checked that the second NIC (i.e. eth1) is physically connected to the external network.
If the parent distribution for BCM is RHEL and SUSE then registration (Chapter 5) should usually be done.
The license parameters should be verified to be correct:
cmsh -c "main licenseinfo"

If the license being used is a temporary license (see End Time value), a new license should be requested well before the temporary license expires. The procedure for requesting and installing a new license is described in Chapter 4.
The head node software should be updated via its package manager (yum, dnf, apt, zypper) so that it has the latest packages (sections 11.2 -11.3 of the Administrator Manual).

1.3 Booting Regular Nodes

A check should be done to make sure the first NIC (i.e. eth0) on the head node is physically connected to the internal cluster network.
The BIOS of regular nodes should be configured to boot from the network. The regular nodes should then be booted. No operating system is expected to be on the regular nodes already. If there is an operating system there already, then by default, it is overwritten by a default image provided by the head node during the next stages.
If everything goes well, the node-installer component starts on each regular node and a certificate request is sent to the head node.

If a regular node does not make it to the node-installer stage, then it is possible that additional kernel modules are needed. Section 5.8 of the Administrator Manual contains more information on how to diagnose problems during the regular node booting process.
To identify the regular nodes (that is, to assign a host name to each physical node), several options are available. Which option is most convenient depends mostly on the number of regular nodes and whether a (configured) managed Ethernet switch is present.

Rather than identifying nodes based on their MAC address, it is often beneficial (especially in larger clusters) to identify nodes based on the Ethernet switch port that they are connected to. To allow nodes to be identified based on Ethernet switch ports, section 3.9 of the Administrator Manual should be consulted.

If a node is unidentified, then its node console displays an ncurses message to indicate it is an unknown node, and the net boot keeps retrying its identification attempts. Any one of the following methods may be used to assign node identities when nodes start up as unidentified nodes:
1. Identifying each node on the node console: To manually identify each node, the “Manually select node” option is selected for each node. The node is then identified manually by selecting a node-entry from the list, choosing the Accept option. This option is easiest when there are not many nodes. It requires being able to view the console of each node and keyboard entry to the console.
2. Identifying nodes using cmsh: In cmsh the newnodes command in device mode (here) can be used to assign identities to nodes from the command line. When called without parameters, the newnodes command can be used to verify that all nodes have booted into the node-installer and are all waiting to be assigned an identity.
3. Identifying nodes using Base View: The node identification resource (here) in Base View automates the process of assigning identities so that manual identification of nodes at the console is not required.
Example

To verify that all regular nodes have booted into the node-installer:
```
[root@mycluster ~]# cmsh
[mycluster]% device newnodes
MAC                First appeared           Detected on switch port
-----------------  ------------------------ -----------------------
00:0C:29:D2:68:8D  05 Sep 2011 13:43:13 PDT [no port detected]
00:0C:29:54:F5:94  05 Sep 2011 13:49:41 PDT [no port detected]
..
[mycluster]% device newnodes | wc -l
MAC                First appeared      Detected on switch port
-----------------  ------------------- -----------------------
32
[mycluster]% exit
[root@mycluster ~]#
```
Example

Once all regular nodes have been booted in the proper order, the order of their appearance on the network can be used to assign node identities. To assign identities node001 through node032 to the first 32 nodes that were booted, the following commands may be used:
```
[root@mycluster ~]# cmsh
[mycluster]% device newnodes -s -n node001..node032
MAC                First appeared                Hostname
-----------------  ----------------------------- ---------
00:0C:29:D2:68:8D  Mon, 05 Sep 2011 13:43:13 PDT node001
00:0C:29:54:F5:94  Mon, 05 Sep 2011 13:49:41 PDT node002
..
[mycluster]% exit
[root@mycluster ~]#
```
Each regular node is now provisioned and eventually fully boots. In case of problems, section 5.8 of the Administrator Manual should be consulted.
Optional: To configure power management, Chapter 4 of the Administrator Manual should be consulted.
To update the software on the nodes, a package manager is used to install to the node image filesystem that is on the head node.

The node image filesystem should be updated via its package manager (yum, dnf, apt, zypper) so that it has the latest packages (sections 11.4 -11.5 of the Administrator Manual).

1.4 Quickstart For GPUs

This assumes that the head nodes and regular nodes are up and running. Getting GPUs set up right can be done as follows for a typical installation:

GPU Hardware And Sanity Check

If the hardware is not detected, then that must first be fixed. Checking for the hardware can be done as follows:
1. NVIDIA GPU hardware should be detected on the nodes that use it. This is true for NVIDIA GPU units (separate from the nodes) as well as for on-board NVIDIA GPUs. The lspci command can be used for detection. For example, for a GPU used by node001:
  
  Example
```
[root@basecm10 ~]# ssh node001 lspci | grep NVIDIA
00:07.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40c] (rev a1)
```
2. AMD CPUs, which have a GPU integrated with the CPU, the CPU chip can similarly be identified with lscpu:
  
  Example
```
[root@basecm10 ~]# ssh node001 lscpu | grep "Model name:"
Model name:          AMD Ryzen 5 2500U with Radeon Vega Mobile Gfx
```
  The AMD chips can then be checked against the list of AMD chips with AMD GPUs, as listed at https://www.amd.com/en/support/kb/release-notes/rn-prorad-lin-18-20
Installing The Software
1. Details of AMD GPU software installation are given in section 7.4.
2. For NVIDIA GPUs, assuming the GPU is on the regular node node001, and that the hardware is supported by CUDA 12.6, then software installation is carried out at the head node as follows:
  1. The software components are installed for the head node itself with:
```
[root@basecm10 ~]# yum install cuda12.6-toolkit cuda12.6-sdk cuda12.6-visual-tools
```
  2. Components are installed into the image used by the nodes that have the GPUs, for example the image default-image, with:
```
[root@basecm10 ~]# cm-chroot-sw-img /cm/images/default-image
        ...enters chrooted image...
[root@default-image /]# yum install cuda-driver cuda-dcgm
```
  3. The nodes with GPUs can then simply be rebooted to compile the CUDA drivers as the node boots, and to start the CUDA driver up:
```
[root@basecm10 ~]# cmsh -c 'device reboot -n node001..node015'
```
    Further details on the basic installation of CUDA for NVIDIA GPUs are given in Chapter 9.
Configuring GPUs For The Cluster Manager

After the GPU software has been installed, the gpusettings submode can be used under device mode for the nodes that have GPUs. This submode allows the cluster administrator to modify GPU values. Keeping the default settings in the submode should be fine, which means configuration of GPU settings can simply be skipped.

However, if needed, then the settings can be modified as described in section 3.16.2 of the Administrator Manual,
1. for NVIDIA GPUs
2. for AMD GPUs.
Configuring GPUs For The Workload Manager

A workload manager can be set up from the head node by running:
```
[root@basecm10 ~]# cm-wlm-setup
```
This starts up a TUI configuration. An NVIDIA GPU can be configured via for Slurm using the Setup (Step by Step) option for Slurm (section 7.3.2 of the Administrator Manual).

After configuring the WLM server, WLM submission and WLM client roles for the nodes of the cluster, a screen that asks if GPU resources should be configured is displayed (figure 1.1):

Slurm With cm-wlm-setup: GPU Configuration Entry Screen

Figure 1.1: Slurm With cm-wlm-setup: GPU Configuration Entry Screen

Following through brings up a GPU device settings configuration screen (figure 1.2):

Slurm With cm-wlm-setup: GPU Device Settings Configuration Screen

Figure 1.2: Slurm With cm-wlm-setup: GPU Device Settings Configuration Screen

The help text option in the screen gives hints based on the descriptions at https://slurm.schedmd.com/gres.conf.html, and also as seen in Slurm’s man gres.conf.1.

Figure 1.2 shows 2 physical GPUs on the node being configured. The type is an arbitrary string for the GPU, and each CPU core is allocated an alias GPU device.

The next screen (figure 1.3) allows the NVIDIA CUDA MPS (Multi-Process Service) to be configured:

Configuring An NVIDIA GPU For Slurm With cm-wlm-setup: MPS Settings Configuration Screen

Figure 1.3: Configuring An NVIDIA GPU For Slurm With cm-wlm-setup: MPS Settings Configuration Screen

The help text for this screen gives hints on how the fields can be filled in. The number of GPU cores (figure 1.3) for a GPU device can be set.

The rest of the cm-wlm-setup procedure can then be completed.

The regular nodes that had a role change during cm-wlm-setup can then be rebooted to pick up the workload manager (WLM) services. A check via the cmsh command ds should show what nodes need a restart.

Example

[root@basecm10 ~]# cmsh -c 'ds'
node001 ... [ UP ], restart required (cm-wlm-setup: compute role assigned)
node002 ... [ UP ], restart required (cm-wlm-setup: compute role assigned)
...

If, for example, the range from node001 to node015 needs to be restarted to get the WLM services running, then it could be carried out with:

Example

[root@basecm10 ~]# cmsh -c 'device; reboot -n node001..node015'

NVIDIA configuration for Slurm and other workload managers is described in further detail in section 7.5 of the Administrator Manual

Running A GPU Slurm Job

A user can be created with, for example, an unimaginative name such as auser:

[root@basecm10 ~]# cmsh
[basecm10]% user add auser
[basecm10->user*[auser*]]% set password
enter new password:
retype new password:
[basecm10->user*[auser*]]% ..
[basecm10->user*]% commit

The “Hello World” helloworld.cu script from section 8.5.4 of the User Manual can be saved in auser’s directory, and then compiled for a GPU with nvcc:

[auser@basecm10 ~]$ module load shared cuda12.3/toolkit
[auser@basecm10 ~]$ nvcc helloworld.cu -o helloworld

A slurmgpuhello.sh batch script can be built as follows:

[auser@basecm10 ~]$ cat slurmgpuhello.sh
#!/bin/sh
#SBATCH -o my.stdout
#SBATCH --time=30     #time limit to batch job
#SBATCH --ntasks-per-node=1
#SBATCH -p defq       #assuming gpu is in defq
#SBATCH --gpus=1
echo -n "hostname is "; hostname
module clear -f
./helloworld

It can be submitted as a batch job from the head node:

[auser@basecm10 ~]$ module load slurm
[auser@basecm10 ~]$ sbatch slurmgpuhello.sh
Submitted batch job 35

The output from submission to a node with a GPU can then be seen:

[auser@basecm10 ~]$ cat my.stdout
hostname is node001
String for encode/decode: Hello World!
String encoded on CPU as: Uryy|-d|yq.
String decoded on GPU as: Hello World!
[auser@basecm10 ~]$

More about Slurm batch scripts and GPU compilation can be found in Chapter 8 of the User Manual.

1.5 Optional: Upgrading Python

The version of Python provided by the Linux-based OS distributors typically lags significantly behind the latest upstream version. This is normally a good thing, since the distributors provide integration, and carry out testing to make sure that it works well with the rest of the OS.

Some BCM tools and packages rely on some of the later features of Python, and by default BCM installs its own later Python version via a package dependency when needed. So explicit upgrades to Python are not needed for BCM itself. For example, for Rocky Linux 9.3, the default (distribution) version of Python at the time of writing (March 2024) is 3.9, while BCM installs and deploys a version 3.11 when needed.

However, some cluster administrators would also like to have various Python versions available on their cluster for user applications. For example, because users too would like to use later versions for their nicer features. BCM therefore makes these various versions of Python available for the end user in an integrated manner, as an optional installation.

For example, for NVIDIA Base Command Manager version 10.0, the Python packages that can be installed to provide their associated features are:

cm-python39, for Python 3.9
cm-python311, for Python 3.11

Users can use the modules command to switch the environment to the appropriate Python version. For example, to switch to Python 3.11:

[root@basecm10 ~]# python -V
Python 3.9.18
[root@basecm10 ~]# module load python39
[root@basecm10 ~]# python -V
Python 3.9.10

If the change is carried out correctly, then support is not available for Python-related bugs, but is available for BCM-related features.

The Jupyter module is an example of a module that automatically loads its own version of Python. This may cause confusion if the system administrator is not paying attention to which version of Python is active. For example, in BCM 10.0, if the user has loaded python311, and then loads Jupyter, the active version of Python is 3.9:

Example

[root@basecm10 ~]# module list
Currently Loaded Modulefiles:
 1) shared   2) cmsh   3) cmd   4) cluster-tools/10.0   5) cm-setup/10.0   6) cm-image/10.0
    7) openmpi4/gcc/4.1.5   8) python311
[root@basecm10 ~]# python -V
Python 3.11.3
[root@basecm10 ~]# module load jupyter
Loading jupyter/15.3.0
  Loading requirement: python39
[root@basecm10 ~]# python -V
Python 3.9.18

1.6 Running Base View

To run the Cluster Management GUI (Base View) on the cluster from a workstation running X11: A recent web browser should be used, and pointed to
https:// <head node address>:8081/base-view/
A suitable web browser is the latest Chrome from Google, but Opera, Firefox, Chromium, and similar should all also just work. The hardware on which the browser runs must be fast enough, and for a reasonable experience, should be roughly equivalent to that of a mid- to high-end desktop of 2016.

The cluster should now be ready for running compute jobs.

For more information:

This manual, the Installation Manual, has more details and background on the installation of the cluster in the next chapters.
The Upgrade Manual describes upgrading from earlier versions of NVIDIA Base Command Manager.
The Administrator Manual describes the general management of the cluster.
The User Manual describes the user environment and how to submit jobs for the end user
The Cloudbursting Manual describes how to deploy the cloud capabilities of the cluster.
The Developer Manual has useful information for developers who would like to program with BCM.
The Machine Learning Manual describes how to install and configure machine learning capabilities with BCM.
The Containerization Manual describes how to manage containers with BCM.

Download this manual as a PDF