Rivermax SDK

Clara Developer Kits can be used along with the NVIDIA Rivermax SDK to provide an extremely efficient network connection using the onboard ConnectX network adapter that is further optimized for GPU workloads by using GPUDirect. This technology avoids unnecessary memory copies and CPU overhead by copying data directly to or from pinned GPU memory, and supports both the integrated GPU or the discrete GPU.

The instructions below describe the steps to test the Rivermax SDK with the Developer Kits. The test applications used by these instructions, generic_sender and generic_receiver, can then be used as samples in order to develop custom applications that use the Rivermax SDK to optimize data transfers using GPUDirect.

Running the Rivermax sample applications requires two systems, a sender and a receiver, connected via ConnectX network adapters. If two Developer Kits are used then the onboard ConnectX can be used on each system, but if only one Developer Kit is available then it is expected that another system with an add-in ConnectX network adapter will need to be used. Rivermax supports a wide array of platforms, including both Linux and Windows, but these instructions assume that another Linux based platform will be used as the sender device while the Developer Kit is used as the receiver.

1. Determine the logical name for the ConnectX devices that are used by each system. This can be done by using the lshw -class network command, finding the product: entry for the ConnectX device, and making note of the logical name: that corresponds to that device. For example, this output on a Developer Kit shows the onboard ConnectX device using the enp9s0f01 logical name (lshw output shortened for demonstration purposes).

   Copy

   Copied!

               
   
               
      $ sudo lshw -class network
      *-network:0
           description: Ethernet interface
           product: MT28908 Family [ConnectX-6]
           vendor: Mellanox Technologies
           physical id: 0
           bus info: pci@0000:09:00.0
           <b>logical name: enp9s0f0</b>
           version: 00
           serial: 48:b0:2d:13:9b:6b
           capacity: 10Gbit/s
           width: 64 bits
           clock: 33MHz
           capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd autonegotiation
           configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.4-1.0.3 duplex=full firmware=20.27.4006 (NVD0000000001) ip=10.0.0.2 latency=0 link=yes multicast=yes
           resources: iomemory:180-17f irq:33 memory:1818000000-1819ffffff
           

   The instructions that follow will use the enp9s0f0 logical name for ifconfig commands, but these names should be replaced with the corresponding logical names as determined by this step.

2. Run the generic_sender application on the sending system.

   a. Bring up the network:

   Copy

   Copied!

               
   
               
      $ sudo ifconfig enp9s0f0 up 10.0.0.1
           

   b. Build the sample apps:

   Copy

   Copied!

               
   
               
      $ cd 1.8.21/apps
      $ make
           

   Note

   The 1.8.21 path above corresponds to the path where the Rivermax SDK package was installed. If the Rivermax SDK was installed via SDK Manager, this path will be $HOME/Documents/Rivermax/1.8.21.

   e. Launch the ```generic_sender` application:

   Copy

   Copied!

               
   
               
      $ sudo ./generic_sender -l 10.0.0.1 -d 10.0.0.2 -p 5001 -y 1462 -k 8192 -z 500 -v
      ...
      +#############################################
      | Sender index: 0
      | Thread ID: 0x7fa1ffb1c0
      | CPU core affinity: -1
      | Number of streams in this thread: 1
      | Memory address: 0x7f986e3010
      | Memory length: 59883520[B]
      | Memory key: 40308
      +#############################################
      | Stream index: 0
      | Source IP: 10.0.0.1
      | Destination IP: 10.0.0.2
      | Destination port: 5001
      | Number of flows: 1
      | Rate limit bps: 0
      | Rate limit max burst in packets: 0
      | Memory address: 0x7f986e3010
      | Memory length: 59883520[B]
      | Memory key: 40308
      | Number of user requested chunks: 1
      | Number of application chunks: 5
      | Number of packets in chunk: 8192
      | Packet's payload size: 1462
      +**********************************************
           

3. Run the generic_receiver application on the receiving system.

   a. Bring up the network:

   Copy

   Copied!

               
   
               
      $ sudo ifconfig enp9s0f0 up 10.0.0.2
           

   b. Build the sample apps with GPUDirect support (```CUDA=y`):

   Copy

   Copied!

               
   
               
      $ cd 1.8.21/apps
      $ make CUDA=y
           

   Note

   The 1.8.21 path above corresponds to the path where the Rivermax SDK package was installed. If the Rivermax SDK was installed via SDK Manager, this path will be $HOME/Documents/Rivermax/1.8.21.

   c. Launch the generic_receiver application:

   Copy

   Copied!

               
   
               
      $ sudo ./generic_receiver -i 10.0.0.2 -m 10.0.0.2 -s 10.0.0.1 -p 5001 -g 0
      ...
      Attached flow 1 to stream.
      Running main receive loop...
      Got 5877704 GPU packets | 68.75 Gbps during 1.00 sec
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
      Got 5877704 GPU packets | 68.75 Gbps during 1.00 sec
      Got 5878240 GPU packets | 68.75 Gbps during 1.00 sec
      ...
           

With both the generic_sender and generic_receiver processes active, the receiver will continue to print out received packet statistics every second. Both processes can then be terminated with <ctrl-c>