Networking#

Vera-Rubin NVL72 uses separate network planes for scale-out GPU traffic, north-south data center connectivity, and management. Tuning should start by identifying which plane carries the application traffic being measured.

Network Planes#

In the reference topology, east-west scale-out traffic uses ConnectX-9, with eight CX9 NIC ASICs per compute tray and two NIC ASICs logically associated with each Rubin GPU package for up to 1.6 Tb/s of east-west bandwidth per GPU package. Confirm the per-GPU association and supported bandwidth against the installed platform’s topology. The NICs are implemented as four ConnectX-9 adapter boards, two CX9 ASICs per board, in two front-panel I/O card assemblies.

North-south traffic uses BlueField-4. The DPU provides 800 Gb/s per tray through dual 400 Gb/s QSFP112 links and can operate in DPU or NIC-oriented modes depending on platform policy and software support.

Out-of-band management, tray control, NVLink switch management, BMC/HMC/SMM traffic, and data plane traffic should be kept conceptually separate in tuning notes and benchmark logs.

Discovery#

Collect a fresh network inventory after every driver, firmware, OS, or platform management update:

ibdev2netdev
rdma link
ethtool -i <netdev>
ethtool -k <netdev>
ethtool -l <netdev>
ethtool -S <netdev> | grep -E 'rx|tx|err|drop|pause|ecn|roce'
nvidia-smi topo -m
lspci -vv -s <pci-address>

Record PCIe locality, NUMA locality, GPU locality, link speed, firmware version, MTU, RoCE/InfiniBand mode, OSFP/QSFP port mode, and error counters with benchmark results. These values matter more on Vera-Rubin than on smaller systems because one tray has many high-bandwidth NICs sharing the same CPU and GPU locality map.

GPU/NIC Affinity#

Start from the discovered topology rather than fixed mlx5 device names. Device numbering can change across software updates, service actions, and platform configurations.

For GPU communication libraries, validate:

  • Each rank is bound to the intended GPU.

  • The rank’s CPU mask is local to the GPU and NIC path.

  • UCX and NCCL select the intended HCA or rail set.

  • GPUDirect RDMA is enabled and actually used.

  • NIC-sharing tests are run separately from dedicated-NIC tests.

Use the software and firmware qualified for the installed platform release. Validate buffer registration and GPUDirect RDMA behavior for the application, especially when multiple ranks share NICs.

Operating System And Driver Checks#

Recommended checks before performance runs:

# CPU locality and IRQ placement.
lscpu
numactl -H
cat /proc/irq/*/smp_affinity_list 2>/dev/null | head

# IOMMU and ACS state.
cat /proc/cmdline
lspci -vv | grep -Ei 'ACS|IOMMU|ATS|PRI|PASID'

# Driver modules; module presence alone does not prove GPUDirect RDMA use.
lsmod | grep -E 'nvidia|peermem|mlx5'
nvidia-smi topo -m

GPUDirect RDMA can use DMA-BUF with a supported kernel and NVIDIA open GPU driver, or the nvidia-peermem module supplied with a compatible NVIDIA driver. DMA-BUF does not require that module. Confirm the supported path for the installed stack and verify its use in transport logs and measurements; module inventory and topology alone are insufficient. See the public NCCL GPU-to-NIC troubleshooting guidance.

Follow the platform-qualified IOMMU, ATS, and PCIe ACS configuration. Consider iommu.passthrough=1 on bare metal only when platform guidance requires it and deployment security policy permits it. Verify the effective IOMMU mode after changes. ACS settings must follow the qualified topology, not a universal disable rule. Virtualized and isolated deployments must retain their required DMA isolation and access controls.

NIXL#

NVIDIA Inference Xfer Library (NIXL) provides point-to-point data movement for distributed AI inference across GPU memory, host memory, and storage through pluggable transfer backends. Validate the backend’s network plane, GPU/NIC locality, and supported GPUDirect RDMA path. Reuse registered memory and peer metadata where the backend supports it. See the NIXL architecture guide for backend, API, and configuration details.

DOCA Perftest#

Use DOCA Perftest to characterize RDMA bandwidth, message rate, and latency, starting with isolated CX9 paths before multi-node and multi-rail scenarios. Record affinity, software and firmware versions, transport, memory type, traffic pattern, message size, and duration. Follow the DOCA Perftest documentation for the installed release’s prerequisites and scenario configuration.

Benchmarking#

Measure networking at three levels:

  • Per-NIC health with platform-approved perftest or fabric validation tools.

  • Per-GPU communication with UCX, NCCL, and NVSHMEM microbenchmarks.

  • Application-level throughput and tail latency with the production launch stack.

Run one-tray, partial-rack, and full-rack tests. Use the same rail mapping, message sizes, process placement, and power profile for each comparison.