Enabling GPUDirect RDMA

There are two parts to enabling RDMA for Holoscan:

• Enabling RDMA on the ConnectX SmartNIC
• Enabling GPUDirect RDMA

Enabling RDMA on the ConnectX SmartNIC

Skip to the next section if you do not plan to leverage a ConnectX SmartNIC.

The NVIDIA IGX Orin developer kit comes with an embedded ConnectX Ethernet adapter to offer advanced hardware offloads and accelerations. You can also purchase an individual ConnectX adapter and install it on other systems, such as x86_64 workstations.

The following steps are required to ensure your ConnectX can be used for RDMA over Converged Ethernet (RoCE):

1. Install MOFED drivers

Ensure the Mellanox OFED drivers version 23.10 or above are installed:

$
cat /sys/module/mlx5_core/version

If not installed, or an older version is installed, you should install the appropriate version from the MLNX_OFED download page, or use the script below:

$
# You can choose different versions/OS or download directly from the
$
# Download Center in the webpage linked above
$
MOFED_VERSION="24.07-0.6.1.0"
$
OS="ubuntu22.04"
$
MOFED_PACKAGE="MLNX_OFED_LINUX-${MOFED_VERSION}-${OS}-$(uname -m)"
$
wget --progress=dot:giga https://www.mellanox.com/downloads/ofed/MLNX_OFED-${MOFED_VERSION}/${MOFED_PACKAGE}.tgz
$
tar xf ${MOFED_PACKAGE}.tgz
$
sudo ./${MOFED_PACKAGE}/mlnxofedinstall
$
# add the --force flag to force uninstallation if necessary:
$
# sudo ./${MOFED_PACKAGE}/mlnxofedinstall --force
$
rm -r ${MOFED_PACKAGE}*

2. Load MOFED drivers

Ensure the drivers are loaded:

$
sudo lsmod | grep ib_core

If nothing appears, run the following command:

$
sudo /etc/init.d/openibd restart

3. Switch the board Link Layer to Ethernet

The ConnectX SmartNIC can function in two separate modes (called link layer):

• Ethernet (ETH)
• Infiniband (IB)

Holoscan does not support IB at this time (as it is not tested), so the ConnectX will need to use the ETH link layer.

To identify the current mode, run ibstat or ibv_devinfo and look for the Link Layer value. In the example below, the mlx5_0 interface is in Ethernet mode, while the mlx5_1 interface is in Infiniband mode. Do not pay attention to the transport value which is always InfiniBand.

1
$ ibstat
2
CA 'mlx5_0'
3
        CA type: MT4129
4
        Number of ports: 1
5
        Firmware version: 28.37.0190
6
        Hardware version: 0
7
        Node GUID: 0x48b02d0300ee7a04
8
        System image GUID: 0x48b02d0300ee7a04
9
        Port 1:
10
                State: Down
11
                Physical state: Disabled
12
                Rate: 40
13
                Base lid: 0
14
                LMC: 0
15
                SM lid: 0
16
                Capability mask: 0x00010000
17
                Port GUID: 0x4ab02dfffeee7a04
18
                Link layer: Ethernet
19
CA 'mlx5_1'
20
        CA type: MT4129
21
        Number of ports: 1
22
        Firmware version: 28.37.0190
23
        Hardware version: 0
24
        Node GUID: 0x48b02d0300ee7a05
25
        System image GUID: 0x48b02d0300ee7a04
26
        Port 1:
27
                State: Active
28
                Physical state: LinkUp
29
                Rate: 100
30
                Base lid: 0
31
                LMC: 0
32
                SM lid: 0
33
                Capability mask: 0x00010000
34
                Port GUID: 0x4ab02dfffeee7a05
35
                Link layer: InfiniBand

If no results appear after ibstat and sudo lsmod | grep ib_core returns a result like this:

1
ib_core               425984  1 ib_uverbs

Consider running the following command or rebooting:

$
sudo /etc/init.d/openibd restart

To switch the link layer mode, there are two possible options:

1. On IGX Orin developer kits, you can switch that setting through the BIOS: see IGX Orin documentation.

2. On any system with a ConnectX (including IGX Orin developer kits), you can run the command below from a terminal (this will require a reboot). sudo ibdev2netdev -v is used to identify the PCI address of the ConnectX (any of the two interfaces is fine to use), and mlxconfig is used to apply the changes.

   $
   mlx_pci=$(sudo ibdev2netdev -v | awk '{print $1}' | head -n1)
   $
   sudo mlxconfig -d $mlx_pci set LINK_TYPE_P1=ETH LINK_TYPE_P2=ETH

   Note: LINK_TYPE_P1 and LINK_TYPE_P2 are for mlx5_0 and mlx5_1 respectively. You can choose to only set one of them. You can pass ETH or 2 for Ethernet mode, and IB or 1 for InfiniBand.

   This is the output of the command above:

   1
   Device #1:
   2
   ----------

  Device type: ConnectX7 Name: P3740-B0-QSFP_Ax Description: NVIDIA Prometheus P3740 ConnectX-7 VPI PCIe Switch Motherboard; 400Gb/s; dual-port QSFP; PCIe switch5.0 X8 SLOT0 ;X16 SLOT2; secure boot; Device: 0005:03:00.0   Configurations: Next Boot New LINK_TYPE_P1 ETH(2) ETH(2) LINK_TYPE_P2 IB(1) ETH(2)   Apply new Configuration? (y/n) [n] :

`Next Boot` is the current value that was expected to be used at the next reboot, while `New` is the value you're about to set to override `Next Boot`.
Apply with `y` and reboot afterwards:
```bash
Applying... Done!
-I- Please reboot machine to load new configurations.

4. Configure the IP addresses of the Ethernet interfaces

First, identify the logical names of your ConnectX interfaces. Connecting a cable in just one of the interfaces on the ConnectX will help you identify which port is which (in the example below, only mlx5_1 – i.e., eth3 – is connected):

$
$ sudo ibdev2netdev
$
mlx5_0 port 1 ==> eth2 (Down)
$
mlx5_1 port 1 ==> eth3 (Up)

1
$ sudo ibdev2netdev -v
2
0005:03:00.0 mlx5_0 (MT4129 - P3740-0002 ) NVIDIA IGX, P3740-0002, 2-port QSFP up to 400G, InfiniBand and Ethernet, PCIe5                                                                                                         fw 28.37.0190 port 1 (DOWN  ) ==> eth2 (Down)
3
0005:03:00.1 mlx5_1 (MT4129 - P3740-0002 ) NVIDIA IGX, P3740-0002, 2-port QSFP up to 400G, InfiniBand and Ethernet, PCIe5                                                                                                         fw 28.37.0190 port 1 (DOWN  ) ==> eth2 (Down)

1
$ sudo dmesg | grep -w mlx5_core
2
...
3
[   11.512808] mlx5_core 0005:03:00.0 eth2: Link down
4
[   11.640670] mlx5_core 0005:03:00.1 eth3: Link down
5
...
6
[ 3712.267103] mlx5_core 0005:03:00.1: Port module event: module 1, Cable plugged

The next step is to set a static IP on the interface you’d like to use so you can refer to it in your Holoscan applications (e.g., Emergent cameras, distributed applications…).

First, check if you already have an address setup. We’ll use the eth3 interface in this example for mlx5_1:

$
ip -f inet addr show eth3

If nothing appears, or you’d like to change the address, you can set an IP and MTU (Maximum Transmission Unit) through the Network Manager user interface, CLI (nmcli), or other IP configuration tools. In the example below, we use ip (ifconfig is legacy) to configure the eth3 interface with an address of 192.168.1.1/24 and a MTU of 9000 (i.e., “jumbo frame”) to send Ethernet frames with a payload greater than the standard size of 1500 bytes:

$
sudo ip link set dev eth3 down
$
sudo ip addr add 192.168.1.1/24 dev eth3
$
sudo ip link set dev eth3 mtu 9000
$
sudo ip link set dev eth3 up

Enabling GPUDirect RDMA

Follow the instructions below to enable GPUDirect RDMA:

$
  nv_driver_version=$(modinfo nvidia | awk '/^version:/ {print $2}' | cut -d. -f1)
$
  sudo dpkg-reconfigure nvidia-dkms-$nv_driver_version # or nvidia-dkms-${nv_driver_version}-server

$
  sudo modprobe nvidia-peermem

$
  echo nvidia-peermem | sudo tee -a /etc/modules

Testing with Rivermax

The instructions below describe the steps to test GPUDirect using the Rivermax SDK. The test applications used by these instructions, generic_sender and generic_receiver, can then be used as samples in order to develop custom applications that use the Rivermax SDK to optimize data transfers.

Running the Rivermax sample applications requires two systems, a sender and a receiver, connected via ConnectX network adapters. If two Developer Kits are used, then the onboard ConnectX can be used on each system. However, if only one Developer Kit is available, then it is expected that another system with an add-in ConnectX network adapter will need to be used. Rivermax supports a wide array of platforms, including both Linux and Windows, but these instructions assume that another Linux based platform will be used as the sender device while the Developer Kit is used as the receiver.

1
rivermax_sdk=$HOME/Documents/Rivermax/1.31.10

1
rivermax_sdk=$DOWNLOAD_PATH/1.31.10

1. Determine the logical name for the ConnectX devices that are used by each system. This can be done by using the lshw -class network command, finding the product: entry for the ConnectX device, and making note of the logical name: that corresponds to that device. For example, this output on a Developer Kit shows the onboard ConnectX device using the enp9s0f01 logical name (lshw output shortened for demonstration purposes).

   $
      $ sudo lshw -class network
   $
      *-network:0
   $
           description: Ethernet interface
   $
           product: MT28908 Family [ConnectX-6]
   $
           vendor: Mellanox Technologies
   $
           physical id: 0
   $
           bus info: pci@0000:09:00.0
   $
           <b>logical name: enp9s0f0</b>
   $
           version: 00
   $
           serial: 48:b0:2d:13:9b:6b
   $
           capacity: 10Gbit/s
   $
           width: 64 bits
   $
           clock: 33MHz
   $
           capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd autonegotiation
   $
           configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.4-1.0.3 duplex=full firmware=20.27.4006 (NVD0000000001) ip=10.0.0.2 latency=0 link=yes multicast=yes
   $
           resources: iomemory:180-17f irq:33 memory:1818000000-1819ffffff

 The instructions that follow will use the `enp9s0f0` logical name for
 `ifconfig` commands, but these names should be replaced with the
 corresponding logical names as determined by this step.
2. Run the `generic_sender` application on the sending system.
 a. Bring up the network:
    ```bash
       sudo ifconfig enp9s0f0 up 10.0.0.1

b. Build the sample apps:

$
   cd ${rivermax_sdk}/apps
$
   make

e. Launch the generic_sender application:

$
   sudo ./generic_sender -l 10.0.0.1 -d 10.0.0.2 -p 5001 -y 1462 -k 8192 -z 500 -v

which gives

   +#############################################
   | Sender index: 0
   | Thread ID: 0x7fa1ffb1c0
   | CPU core affinity: -1
   | Number of streams in this thread: 1
   | Memory address: 0x7f986e3010
   | Memory length: 59883520[B]
   | Memory key: 40308
   +#############################################
   | Stream index: 0
   | Source IP: 10.0.0.1
   | Destination IP: 10.0.0.2
   | Destination port: 5001
   | Number of flows: 1
   | Rate limit bps: 0
   | Rate limit max burst in packets: 0
   | Memory address: 0x7f986e3010
   | Memory length: 59883520[B]
   | Memory key: 40308
   | Number of user requested chunks: 1
   | Number of application chunks: 5
   | Number of packets in chunk: 8192
   | Packet's payload size: 1462
   +**********************************************

3. Run the generic_receiver application on the receiving system.

   a. Bring up the network:

   $
      sudo ifconfig enp9s0f0 up 10.0.0.2

   b. Build the generic_receiver app with GPUDirect support from the Rivermax GitHub Repo. Before following the instructions to build with CUDA Toolkit support, apply the changes to the file generic_receiver/generic_receiver.cpp in this PR. This was tested on the NVIDIA IGX Orin Developer Kit with Rivermax 1.31.10.

   c. Launch the generic_receiver application from the build directory:

   $
      $ sudo ./generic_receiver -i 10.0.0.2 -m 10.0.0.2 -s 10.0.0.1 -p 5001 -g 0
   $
      ...
   $
      Attached flow 1 to stream.
   $
      Running main receive loop...
   $
      Got 5877704 GPU packets | 68.75 Gbps during 1.00 sec
   $
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
   $
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
   $
      Got 5877704 GPU packets | 68.75 Gbps during 1.00 sec
   $
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
   $
      ...

With both the generic_sender and generic_receiver processes active, the receiver will continue to print out received packet statistics every second. Both processes can then be terminated with <ctrl-c>.