Optimizing IO Performance#

NUMA Affinity#

Important

On Vera-Rubin platforms, it’s critical that you use numactl to bind your IO applications to the local CPU and memory that are in the same NUMA domain as your IO devices.

To check your network adapter’s NUMA domain, run following commands:

cat /sys/class/net/<ethernet interface>/device/numa_node
cat /sys/class/net/<ethernet interface>/device/local_cpulist

To check your NVMe drive’s NUMA domain, run following commands:

cat /sys/class/nvme/<NVMe device>/device/numa_node
cat /sys/class/nvme/<NVMe device>/device/local_cpulist

After you determine the NUMA node for your IO devices, use numactl to bind your IO applications to the CPU and memory in the same NUMA domain as your IO devices.

Refer to CPU NUMA Affinity for more information about using numactl.

PCIe Peer-to-Peer Bandwidth#

A PCIe peer-to-peer (P2P) transfer that crosses the socket boundary travels over the C2C die-to-die link between the two Vera sockets. Cross-socket P2P writes are subject to a bandwidth limit that local P2P traffic is not.

The maximum remote socket P2P write bandwidth is 54 GB/s per destination x16 PCIe instance. This is the cumulative bandwidth of the remote socket P2P writes that arrive at that destination from all PCIe sources on the other socket.

_images/vera-pcie-p2p-remote-write-shared-dest.png

Remote P2P writes from two Socket_0 sources converge on one Socket_1 destination. The 54 GB/s limit applies to the cumulative write bandwidth that arrives at that destination.#

The limit applies per destination and is not shared between destinations. Each x16 PCIe destination can simultaneously sink up to 54 GB/s of remote socket P2P writes.

_images/vera-pcie-p2p-remote-write-separate-dest.png

Remote P2P writes from Socket_0 to two different Socket_1 destinations, and each destination has its own 54 GB/s limit.#

Remote socket P2P reads are not affected and stream at the full x16 Gen6 PCIe rate to each destination. Local socket P2P reads and writes stay within one socket, never cross the C2C link, and also run at the full x16 Gen6 rate.

_images/vera-pcie-p2p-local-write.png

Local P2P write inside Socket_1, from PCIE_5 OCU through UCF_SOC to PCIE_0 EPOC. The traffic never crosses C2C and runs at full x16 Gen6 rate.#

The following table summarizes the peer-to-peer bandwidth limits:

PCIe Peer-to-Peer Bandwidth Limits#

Traffic

Scope

Limit

P2P write

Remote socket

54 GB/s per destination x16 instance, cumulative from all remote socket sources

P2P read

Remote socket

Full x16 Gen6 rate per destination

P2P write

Local socket

Full x16 Gen6 rate

P2P read

Local socket

Full x16 Gen6 rate

PCIe Relaxed Ordering#

Enabling PCIe Relaxed Ordering (RO=1) on your PCIe devices improves the performance of device traffic to system memory and to GPU memory. The benefit applies to local traffic and to traffic that crosses the socket boundary, and it grows with the distance that the traffic travels.

PCIe ordering rules are maintained across the CPU fabric and, for cross-socket traffic, across the C2C link between the two Vera sockets. When Relaxed Ordering is disabled (RO=0), which the PCIe specification calls Default Ordering, all writes must maintain global visibility of write-after-write across addresses and destinations which adds additional overhead. Relaxed Ordering allows better overlap of writes because global visibility does not need to be strictly maintained across different addresses. The additional overhead of RO=0 write-after-write ordering might increase the observed latency for Read transactions.

Important

Systems that run workloads with sustained cross-socket traffic to NVMe drives are required to enable PCIe Relaxed Ordering.

This setting helps optimize PCIe transaction handling under high-bandwidth traffic conditions and reduces the likelihood of completion timeouts (CTO) and downstream port containment (DPC) that might be reported by NVMe and ConnectX-9 devices during stress testing.

Enabling Relaxed Ordering depends on the endpoint:

  • Network adapters: Relaxed Ordering is requested by the application or the driver on each memory region that it registers. On NVIDIA network adapters, this is the per_mkey default, and NVIDIA recommends that you keep it. To check the setting, set ADVANCED_PCI_SETTINGS to True and run the following command:

    sudo mlxconfig -d <dev> query | grep PCI_WR_ORDERING
    PCI_WR_ORDERING per_mkey(0)
    

    Verify that your workload requests Relaxed Ordering on the memory regions that it registers. The GPU communication libraries, and the benchmarks that are distributed with them, already do this.

  • NVMe drives: Relaxed Ordering is controlled by the drive firmware. Contact your SSD vendor for the firmware version and the configuration that enable it on your drives.

Note

Relaxed Ordering is not a substitute for NUMA affinity. Bind your IO applications to the socket that owns the device first, as described in NUMA Affinity, and enable Relaxed Ordering for the workloads that must cross the socket boundary.

Networking#

NVIDIA recommends that you download the latest driver and firmware for your network adapter. Before making any changes, contact your network adapter’s vendor for information about whether the tuning options in this guide are applicable.

Prerequisites for System Tuning#

Before you tune at the network level, apply the following system-level tunings, which directly affect networking workloads:

  • Power and thermals: Networking workloads, especially at high packet rates, are sensitive to CPU frequency and throttling. Keeping the system out of power-saving and thermal-throttling states gives consistent per-packet processing time, higher throughput, and lower, more stable latency. Refer to Power and Thermals for more information.

  • NUMA affinity: Run network threads on cores, and allocate their buffers in memory, that are local to the NIC. This avoids cross-NUMA traffic, which adds latency and reduces achievable bandwidth. Refer to NUMA Affinity for more information.

GPU-NIC Affinity#

Configure GPU networking workloads to use the NICs that are assigned to each GPU in the applicable NVIDIA reference architecture (RA). Completing these assignments provides redundancy and load balancing. To see the platform topology and to check that your assignment matches it, run nvidia-smi topo -m.

doca-perftest detects GPU-NIC affinity automatically. For other tools, or when you select devices manually, use the following rules.

Identifying the NIC Type#

The node has two kinds of NICs, and only one kind is meant for GPU traffic:

  • ConnectX-9 SuperNIC (east-west): The Spectrum-X or InfiniBand NIC. Most GPU traffic goes through these NICs.

  • BlueField-4 DPU (north-south): Management and remote storage. Do not use these devices for GPU benchmarks.

To see which mlx5_* device is which, run ibdev2netdev -v. The description names the model, ConnectX-9 SuperNIC or BlueField-4.

Selecting the NIC for Each GPU#

In the nvidia-smi topo -m matrix, select a ConnectX-9 NIC whose affinity to the GPU shows NODE, which provides the best performance. A NIC that shows SYS also works, but its traffic crosses the inter-socket link, so expect lower performance. The IB device name for each NIC is in the NIC Legend at the bottom of the output, for example, NIC0: mlx5_0.

Note

NODE is the recommended pairing from the system point of view. Determining which NIC is best out of the set of NODE NICs depends on the InfiniBand or Spectrum-X reference architecture. When you select a NIC, follow the reference architecture for your deployment.

Run the following command:

nvidia-smi topo -m
_images/vr-nvl72_nvidia-smi_topo.png

Expected nvidia-smi topo -m output on a healthy Vera-Rubin NVL72 node.#

In the output on a healthy Vera-Rubin NVL72 node:

  • GPU0 and GPU1 are on the first socket, CPU cores 0-87 and 176-263. GPU2 and GPU3 are on the second socket, CPU cores 88-175 and 264-351. All GPUs are connected with NV36.

  • The ConnectX-9 NICs are mlx5_0, mlx5_1, mlx5_2, mlx5_5, mlx5_6, mlx5_7, mlx5_8, and mlx5_11. The BlueField-4 devices, which are two dual-port DPUs, are mlx5_3, mlx5_4, mlx5_9, and mlx5_10.

  • GPU0 and GPU1 have NODE affinity to NIC0-NIC2, and NIC5, which are the ConnectX-9 devices mlx5_0, mlx5_1, mlx5_2, and mlx5_5. GPU2 and GPU3 have NODE affinity to NIC6-NIC8, and NIC11, which are the ConnectX-9 devices mlx5_6, mlx5_7, mlx5_8, and mlx5_11.

  • NODE is the closest GPU-NIC relationship on this platform and provides the best performance. You can use SYS NICs, with lower expected performance. PIX appears only between the two ports of each BlueField-4, NIC3-NIC4 and NIC9-NIC10.

  • Pin processes to the CPU cores that are listed for the GPU’s socket, as described in NUMA Affinity.

Storage/Filesystem#

This section provides information about performance tunings that are related to storage and the filesystem. The performance metrics for storage are:

  • Throughput: For large requests, throughput is reported using units such as MB/s or GB/s. For small requests, the unit is IOPS (Input/Output Operations Per Second).

  • Latency: Time to complete a request.

Preconditioning an SSD#

SSD performance depends on its previous write history. For reproducible, device-level results, establish a defined initial state and condition the test range before collecting measurements. Here is a typical sequence for a write benchmark:

  • Perform two full sequential writes across the test range.

  • Run the target workload, such as random writes, until performance reaches steady state.

If fio v3.41 or later is available, you can use sprandom to reach steady state more quickly for a random write workload. In the following example, the fio job uses sprandom to precondition for a 4kB Random Write workload:

[global]
ioengine=io_uring
direct=1
bs=4k
iodepth=256
numjobs=1
rw=randwrite
norandommap
randrepeat=0
size=100%
sprandom=1
spr_num_regions=400
spr_op=0.18

[nvme0n1]
filename=/dev/nvme0n1

Refer to Sample fio Job Files for NVMe SSDs for more information about using fio.

Note

The spr_op=0.18 setting applies generally to TLC drives. For QLC drives, NVIDIA recommends that you use spr_op=0.22 or consult the Overprovisioning specifications published by the SSD vendor.

Warning

Preconditioning overwrites data in the test range, so use only a device or namespace whose contents can be destroyed.

Refer to the SNIA Solid State Storage Performance Test Specification for more information about standardized preconditioning and steady-state criteria.

Drop Page Cache#

When files are read from storage into memories on a Linux system, they are cached in unused memory areas called page cache. To drop the page cache, for example, because you want to benchmark the storage subsystem, you might need to drop the page cache before every test to see true storage performance.

To drop the page cache, run the following command:

# Write out dirty pages to disk(s)
sudo sync
# Drop page cache, dentries, and inodes
echo 3 | sudo tee /proc/sys/vm/drop_caches

To compare how much memory area has been released from dropping the page cache, compare the Cached: line in the output of following command before and after invoking the previous command:

cat /proc/meminfo | grep Cached

Sample fio Job Files for NVMe SSDs#

This section provides sample fio job files for benchmarking performance of NVMe SSDs using the Flexible IO Tester (fio) benchmark.

The sample files are configured to use the io_uring engine and allow pinning the requesting cores to the local socket using the cpus_allowed parameter. The job performs random reads of 4K blocks on two NVMe SSDs concurrently.

The CPU and NUMA node values in the following job file match the sample dual-socket topology in Checking the Non-Uniform Memory Access Settings. Before you run the job, replace these values with the topology reported for your system and devices.

The number of IO requests in flight can be tuned by adjusting the iodepth and numjobs parameters. Typically, for small requests (for example, 4K) higher values for iodepth and numjobs are used to keep the device busy. In the following example, the fio job uses four threads per SSD and 256 requests per thread:

[global]
ioengine=io_uring
direct=1
bs=4k
iodepth=256
numjobs=4
thread=1
group_reporting=1
runtime=60
time_based=1
ramp_time=5
rw=randread
norandommap=1
randrepeat=0
cpus_allowed_policy=split

[nvme0n1]
# nvme0 is attached to socket 0
filename=/dev/nvme0n1
cpus_allowed=0-3
numa_mem_policy=bind:0

[nvme1n1]
# nvme1 is attached to socket 1
filename=/dev/nvme1n1
cpus_allowed=88-91
numa_mem_policy=bind:1

Tuning Performance for NVMe SSDs#

To ensure that SSDs hit peak performance on Vera, consider the following factors:

  • Minimize the overhead of interrupt handling by coalescing interrupts for NVMe devices or by using polling.

  • Minimize the overhead of IOMMU translations by using lazy invalidations.

  • To keep the device busy, ensure that there are a sufficient number of requests in flight.

  • Minimize cross-socket traffic by pinning the cores performing IO operations to the local socket(s). Refer to the cpus_allowed and numa_mem_policy fields in the sample fio job file for more information.

Interrupt Coalescing for NVMe SSDs#

For NVMe SSDs, interrupt coalescing reduces the interrupt overhead by generating an interrupt after a batch of requests have been completed. As per the NVMe specification:

  • Aggregation Time (TIME): Specifies the recommended maximum time in 100 microsecond increments that a controller might delay an interrupt due to interrupt coalescing. A value of 0h means there is no delay. The controller might apply this time to each interrupt vector or across all interrupt vectors. The reset value of this setting is 0h.

  • Aggregation Threshold (THR): Specifies the recommended minimum number of completion queue entries to aggregate each interrupt vector before signaling an interrupt to the host. This is an 0-based value, and the reset value of this setting is 0h.

To request an aggregation threshold of eight completion queue entries per interrupt vector, and a maximum aggregation time of 100 microseconds, run the following command:

# For NVMe drive nvme0n1
sudo nvme set-feature /dev/nvme0n1 -f 0x8 -V 0x00000107

# Check the setting
sudo nvme get-feature /dev/nvme0n1 -f 0x8 -H

get-feature:0x08 (Interrupt Coalescing), Current value:0x00000107
    Aggregation Time     (TIME): 100 usec
    Aggregation Threshold (THR): 8

Lazy IOMMU Invalidations#

Lazy IOMMU invalidation is a kernel feature that amortizes the overhead of IOMMU cache flushes over multiple IO requests, and it applies only when the IOMMU translates DMA. In the IOMMU passthrough mode, DMA translation and its associated invalidations are bypassed, so iommu.strict does not provide an additional performance benefit.

To check the active IOMMU mode, run the following command:

sudo dmesg | grep -i iommu | grep -i 'passthrough\|strict\|lazy'

If the system uses translated DMA, boot the kernel with the iommu.strict=0 kernel parameter to request lazy invalidation. This can improve IO throughput at the cost of reduced device isolation. The IOMMU driver falls back to strict mode if it does not support lazy invalidation.

Refer to Input-Output Memory Management Unit Passthrough for more information about selecting the IOMMU mode.

Enabling the Polling-Based Completion on NVMe SSDs#

To enable polling-based completion:

  1. Configure the Linux nvme kernel module to use polling for a subset of IO queues:

    # Edit /etc/modprobe.d/nvme.conf - each NVMe device will have 16 queues with polling enabled (no interrupts)
    options nvme poll_queues=16
    
  2. Update the initramfs. For example, on Ubuntu 26.04:

    sudo dracut --force
    
  3. Reboot the system.

  4. Add the following lines to the global section of the fio job file:

    hipri=1
    fixedbufs=1
    

I/O Scheduler for NVMe SSDs#

For raw NVMe benchmarking, use the Linux none I/O scheduler to minimize block-layer scheduling overhead. NVMe devices already perform their own queueing, and none ensures that fio results reflect device and platform performance instead of the scheduler behavior.

To check the current scheduler:

cat /sys/block/<nvme_device>/queue/scheduler

The active scheduler is shown in brackets, for example:

[none] mq-deadline

To set the scheduler to none before running benchmarks:

echo none | sudo tee /sys/block/<nvme_device>/queue/scheduler

This setting is per device and does not persist across reboots unless configured through system policy, such as a udev rule.

NUMA Affinity for RAID Array#

As mentioned in NUMA Affinity section, on Vera-Rubin platforms, it is critical for your IO applications to bind to local CPU and memory. To create a software RAID array for your SSDs on a Vera-Rubin platform, it’s important that you create a separate RAID array for the SSDs attached to each CPU socket.

When accessing these RAID arrays, bind your applications using numactl to the CPU and memory that are on the same NUMA domain as each RAID array.

Refer to CPU NUMA Affinity for more information about using numactl.