16. Displaying GDS NVIDIA FS Driver Statistics

GDS exposes the IO statistics information on the procfs file system.

1. To display driver statistics, run the following command.

   $ cat /proc/driver/nvidia-fs/stats
   

2. Review the output, for example:

   GDS Version: 1.0.0.71
   NVFS statistics(ver: 4.0)
   NVFS Driver(version: 2:7:47)
   Mellanox PeerDirect Supported: True
   IO stats: Enabled, peer IO stats: Enabled
   Logging level: info
   
   Active Shadow-Buffer (MiB): 0
   Active Process: 0
   Reads                           : n=0 ok=0 err=0 readMiB=0 io_state_err=0
   Reads                           : Bandwidth(MiB/s)=0 Avg-Latency(usec)=0
   Sparse Reads : n=6 io=0 holes=0 pages=0
   Writes                          : n=0 ok=0 err=0 writeMiB=0 io_state_err=0
           pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
   Writes                          : Bandwidth(MiB/s)=0 Avg-Latency(usec)=0
   Mmap : n=183 ok=183 err=0 munmap=183
   Bar1-map                        : n=183 ok=183 err=0 free=165 callbacks=18
           active=0
   Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
   Ops : Read=0 Write=0
   GPU 0000:be:00.0  uuid:87e5c586-88ed-583b-df45-fcee0f1e7917 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:e7:00.0  uuid:029faa3b-cb0d-2718-259c-6dc650c636eb : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:5e:00.0  uuid:39eeb04b-1c52-81cc-d76e-53d03eb6ed32 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:57:00.0  uuid:a99a7a93-7801-5711-258b-c6aca4fe6d85 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:39:00.0  uuid:d22b0bc4-cdb1-65ac-7495-3570e5860fda : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:34:00.0  uuid:e11b33d9-60f7-a721-220a-d14e5b15a52c : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=128 cross_root_port(%)=0
   GPU 0000:b7:00.0  uuid:e8630cd2-5cb7-cab7-ef2e-66c25507c119 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:e5:00.0  uuid:b3d46477-d54f-c23f-dc12-4eb5ea172af6 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:e0:00.0  uuid:7a10c7bd-07e0-971b-a19c-61e7c185a82c : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:bc:00.0  uuid:bb96783c-5a46-233a-cbce-071aeb308083 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:e2:00.0  uuid:b6565ee8-2100-7009-bcc6-a3809905620d : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=2 cross_root_port(%)=0
   GPU 0000:5c:00.0  uuid:5527d7fb-a560-ab42-d027-20aeb5512197 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:59:00.0  uuid:bb734f6b-24ad-2f83-86c3-6ab179bce131 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:3b:00.0  uuid:0ef0b9ee-bb8f-cdae-4535-c0d790b2c663 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:b9:00.0  uuid:ad59f685-5836-c2ea-2c79-3c95bea23f0d : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   GPU 0000:36:00.0  uuid:fda65234-707b-960a-d577-18c519301848 : Registered_MiB=0 Cache_MiB=0
           max_pinned_MiB=1 cross_root_port(%)=0
   

16.1. nvidia-fs Statistics

The following table describes nvidia-fs statistics.

Table 16.1 NVIDIA-FS Statistics

Type	Statistics	Description
Reads	n	Total number of read requests.
	ok	Total number of successful read requests.
	err	Total number of read errors.
	Readmb (mb)	Total data read into the GPUs.
	io_state_err	Read errors that were seen. Some pages might have been in the page cache.
Reads	Bandwidth (MB/s)	Active Read Bandwidth when IO is in flight. This is the period from when IO was submitted to the GDS kernel driver until the IO completion was received by the GDS kernel driver. There was no userspace involved.
	Avg-Latency (usec)	Active Read latency when IO is in flight. This is from the period from when IO was submitted to the GDS kernel driver until the IO completion is received by the GDS kernel driver. There was no userspace involved.
Sparse Reads	n	Total number of sparse read requests.
	holes	Total number of holes that were observed during reads.
	pages	Total number of pages that span the holes.
Writes	n	Total number of write requests.
	ok	Total number of successful write requests.
	err	Total number of write errors.
	Writemb (mb)	Total data that was written from the GPUs to the disk.
	io_state_err	Write errors that were seen. Some pages might have been in the page cache.
	pg-cache	Total number of write requests that were found in the page cache.
	pg-cache-fail	Total number of write requests that were found in the page cache but could not be flushed.
	pg-cache-eio	Total number of write requests that were found in the page-cache, but could not be flushed after multiple retries, and IO failed with EIO.
Writes	Bandwidth (MB/s)	Active Write Bandwidth when IO is in flight. This is the period from when IO is submitted to the GDS kernel driver until the IO completion is received by the GDS kernel driver. There was no userspace involved.
	Avg-Latency (\(\mu\)sec)	Active Write latency when IO is in flight. This is the period from when IO is submitted to the GDS kernel driver until the IO completion is received by the GDS kernel driver. There was no userspace involved.
Mmap	n	Total number of mmap system calls that were issued.
	ok	Total number of successful mmap system calls.
	err	Errors that were observed through the mmap system call.
	munmap	Total number of munmap that were issued.
Bar-map	n	Total number of times the GPU BAR memory was pinned.
	ok	Total number of times the successful GPU BAR memory was pinned.
	err	Total errors that were observed during the BAR1 pinning.
	free	Total number of times the BAR1 memory was unpinned.
	callbacks	Total number of times the NVIDIA kernel driver invoked callback to the GDS driver. This is invoked on the following instances: When the process crashes or was abruptly killed. When cudaFree is invoked on memory, which is pinned through cuFileBufRegister, but cuFileBufDeregister is not invoked.
	active	Active number of BAR1 memory that was pinned. (This value is the total number and not the total memory.)
Error	cpu-gpu-pages	Number of IO requests that had a mix of CPU-GPU pages when nvfs_dma_map_sg_attrs is invoked.
	sg-ext	Scatterlist that could not be expanded because the number of GPU pages is greater than blk_nq_nr_phys_segments.
	dma-map	A DMA map error.
Ops	Read	Total number of Active Read IO in flight.
	Write	Total number of Active Write IO in flight.

16.2. Analyze Statistics for Each GPU

You can analyze the statistics for each GPU to better understand what is happening in that GPU.

Consider the following example output:

GPU 0000:5e:00:0  uuid:dc87fe99-4d68-247b-b5d2-63f96d2adab1 : pinned_MB=0 cache_MB=0 max_pinned_MB=79
GPU 0000:b7:00:0  uuid:b3a6a195-d08c-09d1-bf8f-a5423c277c04 : pinned_MB=0 cache_MB=0 max_pinned_MB=76
GPU 0000:e7:00:0  uuid:7c432aed-a612-5b18-76e7-402bb48f21db : pinned_MB=0 cache_MB=0 max_pinned_MB=80
GPU 0000:57:00:0  uuid:aa871613-ee53-9a0c-a546-851d1afe4140 : pinned_MB=0 cache_MB=0 max_pinned_MB=80

In this sample output, 0000:5e:00:0, is the PCI BDF of the GPU with the Dc87fe99-4d68-247b-b5d2-63f96d2adab1 UUID. This is the same UUID that can be used to observe nvidia-smi statistics for this GPU.

Here is some additional information about the statistics:

• pinned-MB shows the active GPU memory that is pinned by using nvidia_p2p_get_pages from the GDS driver in MB across all active processes.

• cache_MB shows the active GPU memory that is pinned by using nvidia_p2p_get_pages, but this memory is used as the internal cache by GDS across all active processes.

• max_pinned_MB shows the max GPU memory that is pinned by GDS at any point in time on this GPU across multiple processes.

  This value indicates that the max BAR size and administrator can be used for system sizing purposes.

16.3. Resetting the nvidia-fs Statistics

To reset the nvidia-fs statistics, run the following commands:

$ sudo bash
$ echo 1 >/proc/driver/nvidia-fs/stats

16.4. Checking Peer Affinity Stats for a Kernel File System and Storage Drivers

The following proc files contain information about peer affinity DMA statistics via nvidia-fs callbacks:

• nvidia-fs/stats

• nvidia-fs/peer_affinity

• nvidia-fs/peer_distance

To enable the statistics, run the following command:

$ sudo bash
$ echo 1 > /sys/module/nvidia_fs/parameters/peer_stats_enabled

To view consolidated statistics as a regular user, run the following command:

$ cat /proc/driver/nvidia-fs/stats

Sample output:

GDS Version: 1.0.0.71
NVFS statistics(ver: 4.0)
NVFS Driver(version: 2:7:47)
Mellanox PeerDirect Supported: True
IO stats: Enabled, peer IO stats: Enabled
Logging level: info

Active Shadow-Buffer (MiB): 0
Active Process: 0
Reads                           : n=0 ok=0 err=0 readMiB=0 io_state_err=0
Reads                           : Bandwidth(MiB/s)=0 Avg-Latency(usec)=0
Sparse Reads                    : n=6 io=0 holes=0 pages=0
Writes                          : n=0 ok=0 err=0 writeMiB=0 io_state_err=0 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
Writes                          : Bandwidth(MiB/s)=0 Avg-Latency(usec)=0
Mmap                            : n=183 ok=183 err=0 munmap=183
Bar1-map                        : n=183 ok=183 err=0 free=165 callbacks=18 active=0
Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
Ops                             : Read=0 Write=0
GPU 0000:be:00.0  uuid:87e5c586-88ed-583b-df45-fcee0f1e7917 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:e7:00.0  uuid:029faa3b-cb0d-2718-259c-6dc650c636eb : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:5e:00.0  uuid:39eeb04b-1c52-81cc-d76e-53d03eb6ed32 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:57:00.0  uuid:a99a7a93-7801-5711-258b-c6aca4fe6d85 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:39:00.0  uuid:d22b0bc4-cdb1-65ac-7495-3570e5860fda : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:34:00.0  uuid:e11b33d9-60f7-a721-220a-d14e5b15a52c : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=128 cross_root_port(%)=0
GPU 0000:b7:00.0  uuid:e8630cd2-5cb7-cab7-ef2e-66c25507c119 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:e5:00.0  uuid:b3d46477-d54f-c23f-dc12-4eb5ea172af6 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:e0:00.0  uuid:7a10c7bd-07e0-971b-a19c-61e7c185a82c : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:bc:00.0  uuid:bb96783c-5a46-233a-cbce-071aeb308083 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:e2:00.0  uuid:b6565ee8-2100-7009-bcc6-a3809905620d : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=2 cross_root_port(%)=0
GPU 0000:5c:00.0  uuid:5527d7fb-a560-ab42-d027-20aeb5512197 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:59:00.0  uuid:bb734f6b-24ad-2f83-86c3-6ab179bce131 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:3b:00.0  uuid:0ef0b9ee-bb8f-cdae-4535-c0d790b2c663 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:b9:00.0  uuid:ad59f685-5836-c2ea-2c79-3c95bea23f0d : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0
GPU 0000:36:00.0  uuid:fda65234-707b-960a-d577-18c519301848 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=1 cross_root_port(%)=0

The cross_root_port (%) port is the percentage of total DMA traffic through nvidia-fs callbacks, and this value spans across PCIe root ports between GPU and its peers such as HCA.

• This can be a major reason for low throughput on certain platforms.

• This does not consider the DMA traffic that is initiated via cudaMemcpyDeviceToDevice or cuMemcpyPeer with the specified GPU.

16.5. Checking the Peer Affinity Usage for a Kernel File System and Storage Drivers

1. To get the peer affinity usage for a kernel file system and storage drivers, run the following command:

   $ cat /proc/driver/nvidia-fs/peer_affinity
   

2. Review the sample output, for example:

   GPU P2P DMA distribution based on pci-distance
   
   (last column indicates p2p via root complex)
   GPU :0000:bc:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:e0:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:e5:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:57:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:59:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:be:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:34:00.0 :0 0 1274489 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:e7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:b7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:36:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:3b:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:39:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:b9:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:5c:00.0 :0 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:e2:00.0 :0 0 39434 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   GPU :0000:5e:00.0 :0 0 513889 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   

Each row represents a GPU entry, and the columns indicate the peer ranks in ascending order. The lower the rank, the better the affinity. Each column entry is the total number of DMA transactions that occurred between the specified GPU and the peers that belong to the same rank.

For example, the row with GPU 0000:34:00.0 has 2621440 IO operations through the peer with rank 3. Non-zero values in the last column indicate that the IO is routed through the root complex.

Here are some examples:

Run the following command:

$ /usr/local/cuda-x.y/gds/samples /mnt/lustre/test 0
$ cat /proc/driver/nvidia-fs/stats

Here is the output:

GDS Version: 1.0.0.71
NVFS statistics(ver: 4.0)
NVFS Driver(version: 2:7:47)
Mellanox PeerDirect Supported: True
IO stats: Enabled, peer IO stats: Enabled
Logging level: info


Active Shadow-Buffer (MB): 0
Active Process: 0
Reads                           : n=0 ok=0 err=0 readmb=0 io_state_err=0
Reads                           : Bandwidth(MB/s)=0 Avg-Latency(usec)=0
Sparse Reads                    : n=0 io=0 holes=0 pages=0
Writes                          : n=1 ok=1 err=0 writemb=0 io_state_err=0 pg-cache=0 pg-cache-fail=0
pg-cache-eio=0
Writes                          : Bandwidth(MB/s)=0 Avg-Latency(usec)=0
Mmap                            : n=1 ok=1 err=0 munmap=1
Bar1-map                        : n=1 ok=1 err=0 free=1 callbacks=0 active=0
Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0
Ops                             : Read=0 Write=0
GPU 0000:34:00:0  uuid:98bb4b5c-4576-b996-3d84-4a5d778fa970 : pinned_MB=0 cache_MB=0 max_pinned_MB=0 cross_root_port(%)=100

Run the following command:

$ cat /proc/driver/nvidia-fs/peer_affinity

Here is the output:

GPU P2P DMA distribution based on pci-distance

(last column indicates p2p via root complex)

GPU :0000:b7:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:b9:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:bc:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:be:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e0:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e2:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e5:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e7:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:34:00:0 :0 0 0 0 0 0 0 0 0 0 0 2
GPU :0000:36:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:39:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:3b:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:57:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:59:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5c:00:0 :0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5e:00:0 :0 0 0 0 0 0 0 0 0 0 0 0

In the above example, there are DMA transactions between the GPU (34:00.0) and one of its peers. The peer device has the highest possible rank which indicates it is farthest away from the respective GPU pci-distance wise.

To check the percentage of traffic, check the cross_root_ port % in /proc/driver/nvidia-fs/stats. In the third example above, this value is 100%, which means that the peer-to peer-traffic is happening over QPI links.

16.6. Display the GPU-to-Peer Distance Table

The peer_distance table displays the device-wise IO distribution for each peer with its rank for the specified GPU, and it complements the rank-based stats.

The peer_distance table displays the device-wise IO distribution for each peer with its rank for the specified GPU. It complements the rank-based stats.

The ranking is done in the following order:

1. Primary priority given to p2p distance (upper 2 bytes).

2. Secondary priority is given to the device bandwidth (lower 2 bytes)

For peer paths that cross the root port, a fixed cost for p2p distance (127) is added. This is done to induce a preference for paths under one CPU root port relative to paths that cross the CPU root ports.

Issue the following command:

$ cat /proc/driver/nvidia-fs/peer_distance

Sample output:

gpu             peer            peerrank                p2pdist np2p    link    gen     class
0000:af:00.0    0000:86:00.0    0x820088                0x82    0       0x8     0x3     network
0000:af:00.0    0000:18:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:af:00.0    0000:86:00.1    0x820088                0x82    0       0x8     0x3     network
0000:af:00.0    0000:19:00.1    0x820088                0x82    0       0x8     0x3     network
0000:af:00.0    0000:87:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:af:00.0    0000:19:00.0    0x820088                0x82    0       0x8     0x3     network
0000:3b:00.0    0000:86:00.0    0x820088                0x82    0       0x8     0x3     network
0000:3b:00.0    0000:18:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:3b:00.0    0000:86:00.1    0x820088                0x82    0       0x8     0x3     network
0000:3b:00.0    0000:19:00.1    0x820088                0x82    0       0x8     0x3     network
0000:3b:00.0    0000:87:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:3b:00.0    0000:19:00.0    0x820088                0x82    0       0x8     0x3     network
0000:5e:00.0    0000:86:00.0    0x820088                0x82    0       0x8     0x3     network
0000:5e:00.0    0000:18:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:5e:00.0    0000:86:00.1    0x820088                0x82    0       0x8     0x3     network
0000:5e:00.0    0000:19:00.1    0x820088                0x82    0       0x8     0x3     network
0000:5e:00.0    0000:87:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:5e:00.0    0000:19:00.0    0x820088                0x82    0       0x8     0x3     network
0000:d8:00.0    0000:86:00.0    0x820088                0x82    0       0x8     0x3     network
0000:d8:00.0    0000:18:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:d8:00.0    0000:86:00.1    0x820088                0x82    0       0x8     0x3     network
0000:d8:00.0    0000:19:00.1    0x820088                0x82    0       0x8     0x3     network
0000:d8:00.0    0000:87:00.0    0x820088                0x82    0       0x8     0x3     nvme
0000:d8:00.0    0000:19:00.0    0x820088                0x82    0       0x8     0x3     network

16.7. The GDSIO Tool

GDSIO is a synthetic IO benchmarking tool that uses cufile APIs for IO. The tool can be found in the /usr/local/cuda-x.y/tools directory. For more information about how to use this tool, run /usr/local/cuda-x.y/tools/gdsio -h or review the gdsio section in the /usr/local/cuda-x.y/tools/README file. In the examples below, the files are created on an ext4 file system.

Issue the following command:

# ./gdsio -f /root/sg/test -d 0 -w 4 -s 1M -x 0 -i 4K:32K:1K -I 1

Sample output:

IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 671/1024(KiB)
IOSize: 4-32-1(KiB) Throughput: 0.044269 GiB/sec, Avg_Latency:
996.094925 usecs ops: 60 total_time 0.014455 secs

This command does a write IO (-I 1) on a file named test of size 1MiB (-s 1M) with an IO size that varies between 4KiB to 32 KiB in steps of 1KiB (-i 4K:32K:1K). The transfer is performed using GDS (-x 0) using 4 threads (-w 4) on GPU 0 (-d 0).

Some additional features of the tool are:

• Support for read/write at random offsets in a file.

  The gdsio tool provides options to perform a read and write to a file at random offsets.

  • Using -I 2 and -I 3 options does a file read and write operation at random offset respectively but the random offsets are always 4KiB aligned.

    # ./gdsio -f /root/sg/test -d 0 -w 4 -s 1M -x 0 -i 4K:32K:1K -I 3
    IoType: RANDWRITE XferType: GPUD Threads: 4 DataSetSize: 706/1024(KiB) IOSize: 4-32-1(KiB) Throughput: 0.079718 GiB/sec, Avg_Latency: 590.853274 usecs ops: 44 total_time 0.008446 secs
    

  • To perform a random read and write at unaligned 4KiB offsets, the -U option can be used with -I 0 or -I 1 for read and write, respectively.

    # ./gdsio -f /root/sg/test -d 0 -w 4 -s 1M -x 0 -i 4K:32K:1K -I 1 -U
    
    IoType: RANDWRITE XferType: GPUD Threads: 4 DataSetSize: 825/1024(KiB) IOSize: 4-32-1(KiB) Throughput: 0.055666 GiB/sec, Avg_Latency: 919.112500 usecs ops: 49 total_time 0.014134 secs
    

  • Random buffer fill for dedupe and compression.

    Using the -R option fills the io size buffer (-i) with random data. This random data is then written to the file onto different file offsets.

    # ./gdsio -f /root/sg/test -d 0 -w 4 -s 1M -x 0 -i 4K:32K:1K -I 1 -R
    
    IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 841/1024(KiB) IOSize: 4-32-1(KiB) Throughput: 0.059126 GiB/sec, Avg_Latency: 788.884580 usecs ops: 69 total_time 0.013565 secs
    

  • Using the -F option will fill the entire file with random data.

    # ./gdsio -f /root/sg/test -d 0 -w 4 -s 1M -x 0 -i 4K:32K:1K -I 1 -F
    
    IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 922/1024(KiB) IOSize: 4-32-1(KiB) Throughput: 0.024376 GiB/sec, Avg_Latency: 1321.104532 usecs ops: 73 total_time 0.036072 secs
    

    This is useful for file systems that use dedupe and compression algorithms to minimize disk access. Using random data increases the probability that these file systems will hit the backend disk more often.

• Variable block size.

  To perform a read or a write on a file, you can specify the block size (- i), which says that IO would be performed in chunks of block sized lengths. To check the stats for what block sizes are used use the gds_stats tool. Ensure the the /etc/cufile.json file has cufile_stats is set to 3:

  # ./gds_stats -p <pid of the gdsio process> -l 3
  

  Sample output:

  0-4(KiB): 0  0
  4-8(KiB): 0  17205
  8-16(KiB): 0  45859
  16-32(KiB): 0  40125
  32-64(KiB): 0  0
  64-128(KiB): 0  0
  128-256(KiB): 0  0
  256-512(KiB): 0  0
  512-1024(KiB): 0  0
  1024-2048(KiB): 0  0
  2048-4096(KiB): 0  0
  4096-8192(KiB): 0  0
  8192-16384(KiB): 0  0
  16384-32768(KiB): 0  0
  32768-65536(KiB): 0  0
  65536-...(KiB): 0  0
  

  The highlighted counters show that, for the command above, the block sizes that are used for file IO are in the 4-32 KiB range.

• Verification mode usage and limitations.

  To ensure data integrity, there is an option to perform IO in a Write and Read in verify mode using the -V option. Here is an example:

  # ./gdsio -V -f /root/sg/test -d 0 -w 1 -s 2G -o 0 -x 0 -k 0 -i 4K:32K:1K -I 1
  IoType: WRITE XferType: GPUD Threads: 1 DataSetSize: 2097144/2097152(KiB) IOSize: 4-32-1(KiB) Throughput: 0.074048 GiB/sec, Avg_Latency: 231.812570 usecs ops: 116513 total_time 27.009349 secs
  Verifying data
  IoType: READ XferType: GPUD Threads: 1 DataSetSize: 2097144/2097152(KiB) IOSize: 4-32-1(KiB) Throughput: 0.103465 GiB/sec, Avg_Latency: 165.900663 usecs ops: 116513 total_time 19.330184 secs
  

  The command above will perform a write followed by a read verify test.

  While using the verify mode, remember the following points:

  • read test (-I 0) with verify option (-V) should be used with files written (-I 1) with the -V option

  • read test (-I 2) with verify option (-V) should be used with files written (-I 3) with the -V option and using same random seed (-k) using same number of threads, offset, and data size

  • write test (-I 1/3) with verify option (-V) will perform writes followed by read.

  • Verify mode cannot be used in timed mode (-T option).

    If Verify mode is used in a timed mode, it will be ignored.

• The configuration file

  GDSIO has an option to configure the parameters that are needed to perform an IO in a configuration file and run the IO using those configurations. The configuration file gives the option of performing multiple jobs, where each job has some different configurations.

  The configuration file has global parameters and job specific parameter support. For example, with a configuration file, you can configure each job to perform on a GPU and with a different number of threads. The global parameters, such as IO Size and transfer mode, remain the same for each job. For more information, refer to /usr/local/cuda-x.y/tools/README and /usr/local/cuda-x.y/tools/rw-sample.gdsio files. After configuring the parameters, to perform the IO operation using the configuration file, run the following command:

  # ./gdsio <config file name>
  

See Tabulated Fields for a list of the tabulated fields.

16.8. Tabulated Fields

The following table describes the tabulated fields in the output of the #./gdsio <config file name> command.

Table 16.2 Tabulated Fields

Global Option	Description
xfer_type	GDSIO Transfer types: 0 : Storage->GPU 1 : Storage->CPU 2 : Storage->CPU->GPU 3 : Storage->CPU->GPU_ASYNC 4 : Storage->PAGE_CACHE->CPU->GPU 5 : Storage->GPU_ASYNC_STREAM 6: Storage->GPU_BATCH 7: Storage->GPU_BATCH_STREAM
rw	IO type, rw=read, rw=write, rw=randread, rw=randwrite
bs	block size, for example, bs=1M, for variable block size can specify range, for example, bs=1M:4M:1M, (1M : start block size, 4M : end block size, 1M :steps in which size is varied).
size	File-size, for example, size=2G.
runtime	Duration in seconds.
do_verify	Use 1 for enabling verification
skip_bufregister	Skip cufile buffer registration, ignored in cpu mode.
enable_nvlinks	Set up NVlinks. This field is recommended if p2p traffic is cross node.
random_seed	Use random seed, for example, 1234.
refill_buffer	Refill io buffer after every write.
fill_random	Fill request buffer with random data.
unaligned_random	Use random offsets which are not page-aligned.
start_offset	File offset to start read/write from.
Per-Job Options	Description
numa_node	NUMA node.
gpu_dev_id	GPU device index (check nvidia-smi).
num_threads	Number of IO Threads per job.
directory	Directory name where files are present. Each thread will work on a per file basis.
filename	Filename for single file mode, where threads share the same file. (Note: directory mode and filemode should not be used in a mixed manner across jobs).
mem_type	Memory types to be used. Supported values: 0 - (cudaMalloc), 1 - (cuMem), 2 - (cudaMallocHost) 3 - malloc 4 - mmap.
fd_type	File Descriptor mode. 0 - O_DIRECT (default) 1 - non-O_DIRECT

16.9. GDSCHECK Tool

The /usr/local/cuda-x.y/tools/gdscheck.py tool is used to perform a GDS platform check and has other options that can be found by using -h option.

$ ./gdscheck.py -h
usage: gdscheck.py [-h] [-p] [--object] [-f FILE] [-v] [-V] [-t] [-e EXPORT_TOPOLOGY] [-o OUTPUT_FILE] [-d DURATION] [-l {0,1,2}] [-i]

GPUDirectStorage platform checker

options:
  -h, --help          show this help message and exit
  -p                  gds platform check
  --object            with -p, show the cuObject platform and configuration
  -f FILE             gds file check
  -v                  gds version checks
  -V                  gds fs checks
  -t                  print optimal GPU/device pairings (quick view)
  -e EXPORT_TOPOLOGY  export full topology to JSON file (modes: sysfs, config, latency, cufile)
  -o OUTPUT_FILE      output JSON file path (required with -e)
  -d DURATION         duration for latency measurements in seconds (default: 30, max: 3600, use with -e latency)
  -l {0,1,2}          logging level: 0=error, 1=warning (default), 2=debug (use with -e)
  -i                  show detailed info about topology export mode (use with -e)

Examples:
  gdscheck.py -p
  gdscheck.py -p --object
  gdscheck.py -f gds_latency.bin
  gdscheck.py -v
  gdscheck.py -V
  gdscheck.py -t
  gdscheck.py -e cufile -o /etc/topology.json
  gdscheck.py -e sysfs -o /etc/topology.json -l 2
  sudo gdscheck.py -e latency -o /etc/topology.json -d 120 -l 2

To perform a GDS platform check, issue the following command and expect the output in the following format:

# ./gdscheck.py -p
 GDS release version: 1.20.0
 nvidia_fs version:  2.31 nvidia_fs minimum version: 2.12
 Platform: aarch64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : c2c, nvfs, compat
 NVMeOF             : compat
 SCSI               : compat
 ScaleFlux CSD      : compat
 NVMesh             : compat
 DDN EXAScaler      : compat
 IBM Spectrum Scale : compat
 NFS                : compat
 BeeGFS             : compat
 ScaTeFS            : compat
 VIRTIOFS           : compat
 PanFS              : compat
 WekaFS             : compat
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect       : Disabled
 --rdma library              : Not Loaded (libcufile_rdma.so)
 --rdma devices              : Not configured
 --rdma_multipath            : Disabled
 --rdma_health_monitor       : Not initialized
 --rdma_dev_addr_list_status : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : true
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 256
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 393216
 properties.per_buffer_cache_size_kb : 16384
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 64
 properties.gpu_bounce_buffer_slab_size_kb : 1024 8192 16384 65536
 properties.gpu_bounce_buffer_slab_count : 128 8 4 2
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 properties.rdma_multipath_enabled : false
 properties.rdma_max_backup_devices : 2
 properties.rdma_io_retry_count : 3
 properties.rdma_io_retry_delay_ms : 100
 properties.rdma_failback_enabled : true
 properties.rdma_failback_delay_ms : 100
 properties.rdma_health_check_interval_ms : 100
 properties.rdma_async_event_monitoring : true
 properties.rdma_unhealthy_threshold : 3
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.lustre.use_pci_p2pdma : false
 properties.vanilla_posix_io_mode : false
 properties.gds_fallback_io : false
 fs.beegfs.posix_gds_min_kb: 0
 fs.beegfs.use_pci_p2pdma : false
 fs.scatefs.posix_gds_min_kb: 0
 fs.scatefs.use_pci_p2pdma : false
 fs.weka.rdma_write_support: false
 fs.nfs.use_pci_p2pdma : false
 fs.gpfs.gds_write_support: true
 fs.gpfs.gds_async_support: true
 fs.gpfs.use_pci_p2pdma : false
 fs.virtiofs.use_pci_p2pdma : false
 fs.panfs.use_pci_p2pdma : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 sparse.p2p_compat_enable : true
 sparse.min_p2pdma_threshold_kb : 16
 miscellaneous.enable_static_routing : false
 miscellaneous.static_routing_filepath : /etc/cufile-static-topology.json
 miscellaneous.enable_static_config : false
 miscellaneous.static_config_filepath : /etc/cufile-static-config.json
 properties.allow_rdma_token_reset : false
 miscellaneous.rdma_token_reset_timeout_secs : 30
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 block.raid.use_pci_p2pdma : true
 block.raid1.use_pci_p2pdma : false
 block.raid10.use_pci_p2pdma : false
 block.nvme.use_pci_p2pdma : true
 block.nvmeof.use_pci_p2pdma : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA GH200 144G HBM3e bar:disabled bar size (MiB):N/A supports GDS, IOMMU State: Pass-through or Enabled
 GPU index 1 NVIDIA GH200 144G HBM3e bar:disabled bar size (MiB):N/A supports GDS, IOMMU State: Pass-through or Enabled
 ==============
 PLATFORM INFO:
 ==============
 Found ACS enabled for switch 0009:00:00.0
 Found ACS enabled for switch 0019:00:00.0
 IOMMU: Pass-through or enabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  13040
 Platform: ARS-221GL-NHIR, Arch: aarch64(Linux 7.0.0-15-generic)
 Platform verification succeeded

Starting with cuObject client v1.4.0 in CUDA 13.5, use -p --object to perform the platform check with cuObject configuration selection:

/usr/local/cuda/gds/tools/gdscheck.py -p --object

To check a custom configuration file:

export CUOBJ_ENV_PATH_JSON=/path/to/cuobj.json
/usr/local/cuda/gds/tools/gdscheck.py -p --object

Use the tool from the matching CUDA installation and the same configuration environment as the application.

The --object option requires -p. Without --object, -p checks the cuFile platform and configuration.

Each invocation initializes its own diagnostic process. Its output describes the configuration selected for that invocation; it does not inspect or change the configuration already active in another application process.

# ./gdscheck.py -p --object
 Using legacy GPU bounce buffer configuration
 GDS release version: 1.20.0
 nvidia_fs version:  2.29 nvidia_fs minimum version: 2.12
 Platform: aarch64
 =====================
 CUOBJECT ENVIRONMENT:
 =====================
 cuObject client release version: 1.4.0
 CUOBJ_ENV_PATH_JSON : cuobj.json
 Userspace RDMA     : Supported
 --Mellanox PeerDirect       : Disabled
 --DmaBuf support            : Enabled
 --rdma library              : Loaded (libcufile_rdma.so)
 --rdma devices              : Configured
 --rdma_multipath            : Disabled
 --rdma_health_monitor       : Not initialized
 --rdma_dev_addr_list_status : Up: 1 Down: 0
 =======================
 CUOBJECT CONFIGURATION:
 =======================
 properties.gpu_bounce_buffer_mode : legacy
 properties.max_device_cache_size_kb : 131072
 properties.per_buffer_cache_size_kb : 1024
 properties.max_device_pinned_mem_size_kb : 18014398509481980
 properties.rdma_peer_type : dmabuf
 properties.rdma_transport_type : DC_V1
 properties.rdma_dev_addr_list : mlx5_0
 properties.rdma_load_balancing_policy : RoundRobin
 properties.rdma_multipath_enabled : false
 properties.rdma_max_backup_devices : 2
 properties.rdma_io_retry_count : 3
 properties.rdma_io_retry_delay_ms : 100
 properties.rdma_failback_enabled : true
 properties.rdma_failback_delay_ms : 100
 properties.rdma_health_check_interval_ms : 100
 properties.rdma_async_event_monitoring : true
 properties.rdma_unhealthy_threshold : 3
 execution.parallel_io : false
 profile.cufile_stats : 0
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA GH200 480GB bar:disabled bar size (MiB):N/A supports GDS, IOMMU State: Pass-through or Enabled
 ==============
 PLATFORM INFO:
 ==============
 Found ACS enabled for switch 0009:00:00.0
 IOMMU: Pass-through or enabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  13030
 Platform: QuantaGrid S74G-2U 1S7GZ9Z0002, Arch: aarch64(Linux 6.8.0-111-generic)
 Platform verification succeeded

16.10. NFS Support with GPUDirect Storage

This section provides information about NFS support with GDS.

16.10.1. Install Linux NFS server with RDMA Support on MLNX_OFED 5.3 or Later

To install a standard Linux kernel-based NFS server with RDMA support, complete the following steps:

Note

The server must have a Mellanox connect-X4/5 NIC with MLNX_OFED 5.3 or later installed.

1. Issue the following command:

   $ ofed_info -s MLNX_OFED_LINUX-5.3-1.0.5.1:
   

2. Review the output to ensure that the server was installed.

   $ sudo apt-get install nfs-kernel-server
   $ mkfs.ext4 /dev/nvme0n1
   $ mount -o data=ordered /dev/nvme0n1 /mnt/nvme
   $ cat /etc/exports
     /mnt/nvme *(rw,async,insecure,no_root_squash,no_subtree_check)
   $ service nfs-kernel-server restart
   $ modprobe rpcrdma
   $ echo rdma 20049 > /proc/fs/nfsd/portlist
   

16.10.2. Install GPUDirect Storage Support for the NFS Client

To install a NFS client with GDS support complete the following steps:

Note

The client must have a Mellanox connect-X4/5 NIC with MLNX_OFED 5.3 or later installed.

1. Issue the following command:

   $ ofed_info -s MLNX_OFED_LINUX-5.3-1.0.5.0:
   

2. Review the output to ensure that the support exists.

   $ sudo apt-get install nfs-common
   $ modprobe rpcrdma
   $ mkdir -p /mnt/nfs_rdma_gds
   $ sudo mount -v -o proto=rdma,port=20049,vers=3 172.16.0.101:/ /mnt/nfs_rdma_gds
   To mount with nconnect using VAST nfs client package:
   Eg: client IB interfaces 172.16.0.17 , 172.16.0.18, 172.16.0.19, 172.16.0.20, 172.16.0.21,172.16.0.22,172.16.0.23 172.16.0.24
   $ sudo mount -v -o proto=rdma,port=20049,vers=3,nconnect=20,localports=172.16.0.17-172.16.0.24,remoteports=172.16.0.101-172.16.0.120 172.16.0.101:/ /mnt/nfs_rdma_gds
   

16.11. NFS GPUDirect Storage Statistics and Debugging

NFS IO can be observed using regular Linux tools that are used for monitoring IO, such as iotop and nfsstat.

• To enable NFS RPC stats debugging, run the following command:

  $ rpcdebug -v
  

• To observer GDS-related IO stats, run the following command:

  $ cat /proc/driver/nvidia-fs/stats
  

• To determine GDS statistics per process, run the following command:

  $ /usr/local/cuda-x.y/tools/gds_stats -p <PID> -l 3
  

16.12. GPUDirect Storage IO Behavior

This section provides information about IO behavior in GDS.

16.12.1. Read/Write Atomicity Consistency with GPUDirect Storage Direct IO

In GDS, the max_direct_io_size_kb property controls the IO unit size in which the limitation is issued to the underlying file system. By default, this value is 16MB. This implies that from a Linux VFS perspective, the atomicity of size is limited to the max_direct_io_size_kb size and not the original request size. This limitation exists in the standard GDS path and in compatible mode.

16.12.2. Write with File a Opened in O_APPEND Mode (cuFileWrite)

For a file that is opened in O_APPEND mode with concurrent writers, if the IO size that is used is larger than the max_direct_io_size_kb property, because of the write atomicity limitations, the file might have interleaved data from multiple writers. This cannot be prevented even if the underlying file-system has locking guarantees.

16.12.3. GPU to NIC Peer Affinity

The library maintains a peer affinity table that is a pci-distance-based ranking for a GPU and the available NICs in the platform for RDMA. Currently, the limitation in the ranking does not consider NUMA attributes for the NICs. For a NIC that does not share a common root port with a GPU, the P2P traffic might get routed cross socket over QPI links even if there is a NIC that resides on the same CPU socket as the GPU.

16.12.4. Compatible Mode with Unregistered Buffers

Currently, in compatible mode, the IO path with non-registered buffers does not have optimal performance and does buffer allocation and deallocation in every cuFileRead or cuFileWrite.

16.12.5. Unaligned writes with Non-Registered Buffers

For unaligned writes, using unregistered buffers performance may not be optimal as compared to registered buffers.

16.12.6. Process Hang with NFS

A process hang is observed in NFS environments when the application crashes.

16.12.7. Tools Support Limitations for CUDA 9 and Earlier

The gdsio binary has been built against CUDA runtime 10.1 and has a dependency on the CUDA runtime environment to be equal to version 10.1 or later. Otherwise, a driver dependency error will be reported by the tool.

16.13. GDS Statistics for Dynamic Routing

Dynamic Routing decisions are performed at I/O operation granularity. The GDS User-space Statistics contain a per-GPU counter to indicate the number of I/Os that have been routed using Dynamic Routing.

Table 16.3 cuFile Dynamic Routing Counter

Entry	Description
dr	Number of cuFileRead/cuFileWrite for which I/O was routed using Dynamic Routing for a given GPU.

There are existing counters in the PER_GPU POOL BUFFER STATS and PER_GPU POSIX POOL BUFFER STATS from which a user can infer the GPUs that are chosen by dynamic routing for use as the bounce buffers.

• Platform has GPUs (0 and 1) not sharing the same PCIe host bridge as the NICs:

  "rdma_dev_addr_list": [ "192.168.0.12", "192.168.1.12" ],
  "rdma_dynamic_routing": true,
  "rdma_dynamic_routing_order": [ "GPU_MEM_NVLINKS", "GPU_MEM", "SYS_MEM" ]
  

  $ gds_stats -p <process id> -l 3
  

  GPU 0 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=3.37598 util(%)=532 n=6629 posix=0 unalign=0 dr=6629 err=0 MiB=6629 BufRegister: n=4 err=0 free=0 MiB=4
  GPU 1 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=3.29297 util(%)=523 n=6637 posix=0 unalign=0 dr=6637 err=0 MiB=6637 BufRegister: n=4 err=0 free=0 MiB=4
  
  PER_GPU POOL BUFFER STATS:
  GPU : 6 pool_size_MiB : 7 usage : 1/7 used_MiB : 1
  GPU : 7 pool_size_MiB : 7 usage : 0/7 used_MiB : 0
  GPU : 8 pool_size_MiB : 7 usage : 2/7 used_MiB : 2
  GPU : 9 pool_size_MiB : 7 usage : 2/7 used_MiB : 2
  
  PER_GPU POSIX POOL BUFFER STATS:
  
  PER_GPU RDMA STATS:
  GPU 0000:34:00.0 :   mlx5_3(138:48):0  mlx5_6(265:48):0
  GPU 0000:36:00.0 :   mlx5_3(138:48):0  mlx5_6(265:48):0
  GPU 0000:39:00.0 :   mlx5_3(138:48):0  mlx5_6(265:48):0
  GPU 0000:3b:00.0 :   mlx5_3(138:48):0  mlx5_6(265:48):0
  GPU 0000:57:00.0 :   mlx5_3(7:48):0  mlx5_6(265:48):0
  GPU 0000:59:00.0 :   mlx5_3(7:48):0  mlx5_6(265:48):0
  GPU 0000:5c:00.0 :   mlx5_3(3:48):3318  mlx5_6(265:48):0
  GPU 0000:5e:00.0 :   mlx5_3(3:48):3318  mlx5_6(265:48):0
  GPU 0000:b7:00.0 :   mlx5_6(3:48):3316  mlx5_3(265:48):0
  GPU 0000:b9:00.0 :   mlx5_6(3:48):3317  mlx5_3(265:48):0
  GPU 0000:bc:00.0 :   mlx5_6(7:48):0  mlx5_3(265:48):0
  GPU 0000:be:00.0 :   mlx5_6(7:48):0  mlx5_3(265:48):0
  GPU 0000:e0:00.0 :   mlx5_6(138:48):0  mlx5_3(265:48):0
  GPU 0000:e2:00.0 :   mlx5_6(138:48):0  mlx5_3(265:48):0
  GPU 0000:e5:00.0 :   mlx5_6(138:48):0  mlx5_3(265:48):0
  GPU 0000:e7:00.0 :   mlx5_6(138:48):0  mlx5_3(265:48):0
  

• Platform configuration that has no GPUs sharing the same PCIe host bridge as the NICs and no NVLinks between the GPUs. For such configurations, an admin can set a policy to use system memory other than the default P2P policy.

  "rdma_dev_addr_list": [ "192.168.0.12", "192.168.1.12" ],
  "rdma_dynamic_routing": true,
  "rdma_dynamic_routing_order": [ "SYS_MEM" ]
  

  PER_GPU STATS:
  GPU 4 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=1.11GiB util(%)=0 n=1023 posix=1023 unalign=1023 dr=1023 err=0 MiB=1023 BufRegister: n=0 err=0 free=0 MiB=0
  GPU 8 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=1.11GiB util(%)=0 n=1023 posix=1023 unalign=1023 dr=1023 err=0 MiB=1023 BufRegister: n=0 err=0 free=0 MiB=0
  PER_GPU POSIX POOL BUFFER STATS:
  GPU 4 4(KiB) :0/0 1024(KiB) :0/1 16384(KiB) :0/0
  GPU 8 4(KiB) :0/0 1024(KiB) :1/1 16384(KiB) :0/0
  

16.13.1. Peer Affinity Dynamic Routing

Dynamic Routing decisions are performed at I/O operation granularity. The GDS User-space Statistics contain a per-GPU counter to indicate the number of I/Os that have been routed using Dynamic Routing.

Table 16.4 cuFile Dynamic Routing Counter

Entry	Description
dr	Number of cuFileRead/cuFileWrite for which I/O was routed using Dynamic Routing for a given GPU.

There are existing counters in the PER_GPU POOL BUFFER STATS and PER_GPU POSIX POOL BUFFER STATS from which a user can infer the GPUs that are chosen by dynamic routing for use as the bounce buffers.

// "rdma_dev_addr_list": [ "192.168.4.12", "192.168.5.12", "192.168.6.12", "192.168.7.12" ],

cufile.log:
--------------
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:133 Computing GPU->NIC affinity table:
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:34:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:36:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:39:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:3b:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:57:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:59:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:5c:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:5e:00.0  RDMA dev: mlx5_6 mlx5_8 mlx5_7 mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:b7:00.0  RDMA dev: mlx5_6
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:b9:00.0  RDMA dev: mlx5_6
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:bc:00.0  RDMA dev: mlx5_7
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:be:00.0  RDMA dev: mlx5_7
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:e0:00.0  RDMA dev: mlx5_8
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:e2:00.0  RDMA dev: mlx5_8
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:e5:00.0  RDMA dev: mlx5_9
 23-02-2021 10:17:49:641 [pid=22436 tid=22436] INFO   curdma-ldbal:139 GPU: 0000:e7:00.0  RDMA dev: mlx5_9

A sample from gds_stats showing the GPU to NIC binding during a sample IO test:

PER_GPU RDMA STATS:
   GPU 0000:34:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:36:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:39:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:3b:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:57:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:59:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:5c:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:5e:00.0 :   mlx5_6(265:48):0  mlx5_8(265:48):0  mlx5_7(265:48):0  mlx5_9(265:48):0
   GPU 0000:b7:00.0 :   mlx5_6(3:48):22918  mlx5_7(7:48):0  mlx5_8(138:48):0  mlx5_9(138:48):0
   GPU 0000:b9:00.0 :   mlx5_6(3:48):22949  mlx5_7(7:48):0  mlx5_8(138:48):0  mlx5_9(138:48):0
   GPU 0000:bc:00.0 :   mlx5_7(3:48):22945  mlx5_6(7:48):0  mlx5_8(138:48):0  mlx5_9(138:48):0
   GPU 0000:be:00.0 :   mlx5_7(3:48):22942  mlx5_6(7:48):0  mlx5_8(138:48):0  mlx5_9(138:48):0
   GPU 0000:e0:00.0 :   mlx5_8(3:48):22937  mlx5_9(7:48):0  mlx5_6(138:48):0  mlx5_7(138:48):0
   GPU 0000:e2:00.0 :   mlx5_8(3:48):22930  mlx5_9(7:48):0  mlx5_6(138:48):0  mlx5_7(138:48):0
   GPU 0000:e5:00.0 :   mlx5_9(3:48):22922  mlx5_8(7:48):0  mlx5_6(138:48):0  mlx5_7(138:48):0
   GPU 0000:e7:00.0 :   mlx5_9(3:48):22920  mlx5_8(7:48):0  mlx5_6(138:48):0  mlx5_7(138:48):0

For kernel-based DFS, DDN-Lustre and VAST-NFS, nvidia-fs driver provides a callback to determine the best NIC given a target GPU. The nvidia-fs peer_affinity can be used to track end-to-end IO affinity behavior.

For example, with a routing policy of GPU_MEM_NVLINK, one should not see cross-port traffic as shown in the statistics snippet below:

$ cat /proc/driver/nvidia-fs/peer_affinity
GPU P2P DMA distribution based on pci-distance

(last column indicates p2p via root complex)
GPU :0000:bc:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e0:00.0 :0 0 205305577 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e5:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:57:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:59:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:be:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:34:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:b7:00.0 :0 0 205279892 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:36:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:3b:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:39:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:b9:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5c:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e2:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5e:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

With routing policy of P2P, one can expect to see cross-port traffic as shown in the following statistics snippet:

dgxuser@e155j-dgx2-c6-u04:~/ssen$ cat /proc/driver/nvidia-fs/peer_affinity
GPU P2P DMA distribution based on pci-distance

(last column indicates p2p via root complex)
GPU :0000:bc:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e0:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e5:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:57:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:59:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:be:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:34:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9186359
GPU :0000:e7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:b7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:36:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9191164
GPU :0000:3b:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9194318
GPU :0000:39:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9188836
GPU :0000:b9:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5c:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:e2:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GPU :0000:5e:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

16.13.2. cuFile Log Related to Dynamic Routing

The following log shows the routing table with possible GPUS to be used for IP addresses:

 /"rdma_dev_addr_list": [ "192.168.0.12", "192.168.1.12", "192.168.2.12", "192.168.3.12", "192.168.4.12", "192.168.5.12", "192.168.6.12", "192.168.7.12" ],

 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:141 Computing NIC->GPU affinity table for rdma devices available in config:
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib3 bdf:0000:5d:00.0 ip: 192.168.3.12 best gpus: 6 7 4 5
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib9 bdf:0000:e6:00.0 ip: 192.168.5.12 best gpus: 14 15 12 13
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib2 bdf:0000:58:00.0 ip: 192.168.2.12 best gpus: 4 5 6 7
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib6 bdf:0000:b8:00.0 ip: 192.168.6.12 best gpus: 8 9 10 11
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib1 bdf:0000:3a:00.0 ip: 192.168.1.12 best gpus: 3 2 0 1
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib8 bdf:0000:e1:00.0 ip: 192.168.4.12 best gpus: 12 13 14 15
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib0 bdf:0000:35:00.0 ip: 192.168.0.12 best gpus: 0 1 3 2
 22-02-2021 21:16:27:776 [pid=90794 tid=90794] INFO   cufio-route:156 netdev:ib7 bdf:0000:bd:00.0 ip: 192.168.7.12 best gpus: 10 11 8 9

22-02-2021 21:16:27:776 [pid=90794 tid=90794] DEBUG  cufio:1218 target gpu: 4 best gpu: 4 selected based on dynamic routing