Multinode Testing

For multinode testing NVIDIA recommends using nvbandwidth application. The nvbandwidth application, when used with the Message Passing Interface (MPI), connects to each GPU instance and tracks test data copies across a full mesh of cluster GPUs.

Installing Required Packages

The nvbandiwdth test requires git, libboost-program-options-dev, sshpass, openmpi-bin, make, and libopenmpi-dev and net-tools. Install them using Apt.

apt install git libboost-program-options-dev sshpass openmpi-bin make libopenmpi-dev net-tools

Important

Install these packages on every MNNVL compute node.

Configure SSH

Each compute node must generate an SSH key, copy it to all other compute nodes and change the SSH Server configuration before running nvbandwidth.

Creating and Copy SSH Keys

The nvbandwidth application relies on mpirun to connect to each node through passwordless SSH to run tests and provide client-server connectivity.

Create SSH keys on each compute node with the ssh-keygen command. A passphrase isn’t required.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/nvidia/.ssh/id_rsa):
/home/nvidia/.ssh/id_rsa already exists.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/nvidia/.ssh/id_rsa
Your public key has been saved in /home/nvidia/.ssh/id_rsa.pub

Copy the /home/nvidia/.ssh/id_rsa.pub to each compute node using ssh-copy-id.

$ ssh-copy-id nvidia@10.28.238.236
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/nvidia/.ssh/id_rsa.pub"

Number of key(s) added: 1

Tip

Generate an SSH key on every compute node, then copy the id_rsa.pub key to every other compute node.

Every compute node copies their keys to every other compute node.

Verify that you can log in with SSH without a password from each compute node to all other compute nodes.

Configuring the SSH Server

To accommodate the number of SSH connections that mpirun requires, increase the number of allowed SSH connections.

On every compute node create the file /etc/ssh/sshd_config.d/mpi.conf with the following contents:

MaxSessions 550
MaxStartups 10:30:100

On every compute node, restart the SSH service and apply the new configuration.

systemctl restart sshd.service

Build nvbandwidth

Clone the nvbandwidth repository to a compute node.

git clone https://github.com/NVIDIA/nvbandwidth/tree/v0.7

Enter the cloned nvbandwidth directory and run the build script and copy the nvbandwidth binary to the /usr/local/bin directory.

cd nvbandwidth
sudo ./debian_install.sh
sudo cp nvbandwidth /usr/local/bin/

Copy the newly created nvbandwidth binary to every compute node.

Tip

Use scp and the IMEX configuration file to automate the file copy process.

while IFS= read -r dest; do
  scp nvbandwidth "$dest:/usr/local/bin/"
done </etc/nvidia-imex/nodes_config.cfg

Run nvbandwidth

The mpirun application requires an interface name to connect to the other compute trays and run the test application.

From a compute tray, get the name of the default interface with the ip route show command.

$ ip route show default
default via 10.114.160.1 dev enP5p9s0 proto dhcp src 10.114.160.219 metric 100

Tip

This example assumes all compute trays can communicate over the default interface. Use the appropiate interface for your environment.

Use mpirun to run the nvbandwidth application between all GPUs on the compute nodes, providing the Ethernet interface name and full path to the nvbandwidth binary. The -np parameter specifies the number of GPUs.

Note

Running nvbandwidth requires all compute nodes to have IMEX installed and running.

$ mpirun --allow-run-as-root --map-by ppr:4:node --bind-to core -np  72 --report-bindings -mca btl_tcp_if_include enP5p9s0 --hostfile /etc/nvidia-imex/nodes_config.cfg /usr/local/bin/nvbandwidth -t multinode_device_to_device_memcpy_read_ce

Figure 11 Example NVBandwidth Read Test

$ mpirun --allow-run-as-root --map-by ppr:4:node --bind-to core -np 72 --report-bindings -mca btl_tcp_if_include enP5p9s0 --hostfile /etc/nvidia-imex/nodes_config.cfg /usr/local/bin/nvbandwidth -t multinode_device_to_device_memcpy_write_ce

Figure 12 Example NVBandwidth Write Test