3. GPUDirect Storage Benchmarking on Direct Attached Storage#
This section covers benchmarking GDS on storage directly attached to the server, typically in the form of NVMe SSD devices on the PCIe bus. The specific examples on DGX-2 and DGX A100 can be used as guidelines for any server configuration. Note that in the following examples, the output of various command line tools and utilities is included. In some cases, rows or columns are deleted to improve readability and clarity.
3.1. GPUDirect Storage Performance on DGX-2 System#
Currently, GDS supports NVMe devices as direct attached storage, where NVMe SSDs are plugged directly into the PCIe bus. The DGX-2 system comes configured with up to 16 of these devices that are typically configured as a large RAID metadevice. As per the previous section, the DGX-2 system used to execute these examples was very specifically configured, such that pairs of NVMe SSDs on the same PCIe switch are in a RAID0 group, and the gdsio command line intentionally selects GPUs that share the same upstream PCIe switch.
A Simple Example: Writing to large files with a large IO size using the GDS path.
This example uses a RAID0 device configured with nvme2 and nvme3 with an ext4 file system (mounted as
/nvme23, with a gds_dir subdirectory to hold the generated files).
dgx2> gdsio -D /nvme23/gds_dir -d 2 -w 8 -s 500M -i 1M -x 0 -I 0 -T 120
IoType: READ XferType: GPUD Threads: 8 DataSetSize: 818796544/4096000(KiB) IOSize: 1024(KiB) Throughput: 6.524658 GiB/sec, Avg_Latency: 1197.370995 usecs ops: 799606 total_time 119.679102 secs
Here is some additional information about the options in the example:
-D /nvme23/gds_dir, the target directory.-d 2, selects GPU # 2 for data target/destination.-w 8, 8 workers (8 IO threads)-s 500M, the target file size.-i 1M, IO size (important for assessing throughput).-x 0, the IO data path, in this case GDS.See Table 2 in gdsio Utility for more information.
-I 0, writes the IO load (0 is for reads, 1 is for writes)-T 120, runs for 120 seconds.
The results generated by gdsio show expected performance, given that the storage IO target is a RAID 0
configuration of the two NVMe SSDs, where each SSD is configured with around 3.4GB/sec large read
performance. The average sustained throughput was 6.5GB/sec, with a 1.2ms average latency. We can look
at system data during the gdsio execution for additional data points on data rates and movement. This
is often useful for validating results reported by load generators, as well as ensuring the data path
is as expected. Using the Linux iostat utility (iostat -cxzk 1):
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.42 7.87 0.00 91.68
Device r/s rkB/s r_await rareq-sz w/s wkB/s . . . %util
md127 54360.00 6958080.00 0.00 128.00 0.00 0.00 . . . 0.00
nvme2n1 27173.00 3478144.00 1.03 128.00 0.00 0.00 . . . 100.00
nvme3n1 27179.00 3478912.00 0.95 128.00 0.00 0.00 . . . 100.00
Also, data from the nvidia-smi dmon command:
dgx2> nvidia-smi dmon -i 2 -s putcm
# gpu pwr gtemp mtemp sm mem enc dec rxpci txpci mclk pclk fb bar1
# Idx W C C % % % % MB/s MB/s MHz MHz MB MB
2 63 37 37 0 4 0 0 8923 0 958 345 326 15
2 63 37 37 0 4 0 0 8922 0 958 345 326 15
2 63 37 37 0 4 0 0 8930 0 958 345 326 15
2 63 37 37 0 4 0 0 8764 0 958 345 326 15
This data is consistent with the results reported by gdsio. The iostat data shows just over 3.4GB/sec from
each of the two NVMe drives, and close to 1ms latency per device. Note each drive sustained about 27k
writes-per-second (IOPS). The second data set from the dmon subcommand of nvidia-smi, note the
rxpci column. Recall our gdsio command line initiated GPUDirect Storage reads, so reads from the
storage to the GPU. We see the selected GPU, 2, receiving over 8GB/sec over PCIe. This is GPUDirect
Storage in action - the GPU reading (PCIe receive) directly from the NVMe drives over PCIe.
While the preceding information is important to enable an optimal configuration, the GDS software will
always attempt to maintain an optimal data path, in some cases via another GPU that has better affinity
to the storage targets. By monitoring PCIe traffic with either nvidia-smi or dcgmi (the command
line component of DCGM), we can observe data rates in and out of the GPUs.
Using a previous example, running on the same RAID0 metadevice comprised of two NVMe drives on the same PCIe switch, but specifying GPU 12 this time, and capturing GPU PCIe traffic with dcgmi, we’ll observe sub-optimal performance as GPU 12 is not on the same downstream PCIe switch as our two NVMe drives.
dgx2> gdsio -D /nvme23/gds_dir -d 12 -w 8 -s 500M -i 1M -x 0 -I 0 -T 120
IoType: READ XferType: GPUD Threads: 8 DataSetSize: 491438080/4096000(KiB) IOSize: 1024(KiB) Throughput: 3.893747 GiB/sec, Avg_Latency: 2003.091575 usecs ops: 479920 total_time 120.365276 secs
Note throughput dropped from 6.5GB/sec to 3.9GB/sec, and latency almost doubled to 2ms. The PCIe traffic data tells an interesting story:
dgx2> dcgmi dmon -e 1009,1010 -d 1000
# Entity PCITX PCIRX
Id
GPU 0 4712070210 5237742827
GPU 1 435418 637272
. . .
GPU 11 476420 739272
GPU 12 528378278 4721644934
GPU 13 481604 741403
GPU 14 474700 736417
GPU 15 382261 611617
Note we observe PCIe traffic on GPU 12, but also traffic on GPU 0. This is GDS in action once again. The cuFile library will select a GPU for the data buffers (GPU memory) that is on the same PCIe switch as the storage. In this case, GPU 0 was selected by the library, as it is the first of four GPUs on the same PCIe switch as the NVMe devices. The data is then moved to the target GPU (12).
The net effect of a sub-optimal device selection is an overall decrease in throughput, and increase in latency. With GPU 2, the average throughput was 6.5GB/sec, average latency 1ms. With GPU 12, the average throughput was 3.9GB/sec, average latency 2ms. Thus we observe a 40% decrease in throughput and a 2X increase in latency when a non-optimal configuration is used.
Not all workloads are about throughput. Smaller IO sizes and random IO patterns are an attribute of many production workloads, and assessing IOPS (IO Operations Per Second) performance is a necessary component to the storage benchmarking process.
A critical component to determining what peak performance levels can be achieved is ensuring there is sufficient load. Specifically, for storage benchmarking, the number of processes/threads generating IO is critical to determining maximum performance.
The gdsio tool provides for specifying random reads or random writes (-I flag). In the examples below,
once again we’re showing an optimal combination of GPU (0) and NVMe devices, generating a small (4k) random
read low with an increasing number of threads (-w).
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 4 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 4 DataSetSize: 11736528/4194304(KiB) IOSize: 4(KiB) Throughput: 0.093338 GiB/sec, Avg_Latency: 163.478958 usecs ops: 2934132 total_time 119.917332 secs
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 8 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 8 DataSetSize: 23454880/8388608(KiB) IOSize: 4(KiB) Throughput: 0.187890 GiB/sec, Avg_Latency: 162.422553 usecs ops: 5863720 total_time 119.049917 secs
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 16 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 16 DataSetSize: 48209436/16777216(KiB) IOSize: 4(KiB) Throughput: 0.385008 GiB/sec, Avg_Latency: 158.918796 usecs ops: 12052359 total_time 119.415992 secs
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 32 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 32 DataSetSize: 114100280/33554432(KiB) IOSize: 4(KiB) Throughput: 0.908862 GiB/sec, Avg_Latency: 139.107219 usecs ops: 28525070 total_time 119.726070 secs
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 64 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 64 DataSetSize: 231576720/67108864(KiB) IOSize: 4(KiB) Throughput: 1.848647 GiB/sec, Avg_Latency: 134.554997 usecs ops: 57894180 total_time 119.465109 secs
dgx2> gdsio -D /nvme23/gds_dir -d 0 -w 128 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 128 DataSetSize: 406924776/134217728(KiB) IOSize: 4(KiB) Throughput: 3.243165 GiB/sec, Avg_Latency: 151.508258 usecs ops: 101731194 total_time 119.658960 secs
We can compute the IOPS by dividing the ops value by the total time. Note in all cases the total time was just over 119 seconds (120 seconds was specified as the run duration on the command line).
Thread Count (-w) |
IOPS (ops / total_time) |
|---|---|
4 |
24,468 |
8 |
49,255 |
16 |
100,928 |
32 |
238,245 |
64 |
484,612 |
128 |
850,240 |
It is interesting to observe the average latency on each run (Avg_Latency) actually gets better as the number
of threads and IOPS increases, with 134.5us average latency at 484,612 IOPS running 64 threads. Increasing
the thread count to 128, we observe a slight uptick in latency to 151.51us while sustaining 850,240 random
reads per second. Tracking latency with throughput (or, in this case, IOPS) is important in characterizing
delivered performance. In this example, the specification for the NVMe drives that make up the RAID0 device
indicates a random read capability of about 800k IOPS per drive. Thus, even with 128 threads generating load,
the latency is excellent as the load is well within drive specifications, as each of the two drives in the
RAID0 device sustained about 425,000 IOPS. This was observed with the iostat utility:
avg-cpu: %user %nice %system %iowait %steal %idle
16.03 0.00 6.52 76.97 0.00 0.48
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz . . . %util
md127 856792.00 3427172.00 0.00 0.00 0.00 4.00 . . . 0.00
nvme2n1 425054.00 1700216.00 0.00 0.00 0.13 4.00 . . . 100.80
nvme3n1 431769.00 1727080.00 0.00 0.00 0.13 4.00 . . . 100.00
We observe the row showing the RAID0 metadevice, md127, displaying total reads per second (r/s) reflects the sum of the two underlying NVMe drives.
Extending this example to demonstrate delivered performance when a GPU target is specified that is not part of the same PCIe segment:
dgx2> gdsio -D /nvme23/gds_dir -d 10 -w 64 -s 1G -i 4K -x 0 -I 2 -k 0308 -T 120
IoType: RANDREAD XferType: GPUD Threads: 64 DataSetSize: 13268776/67108864(KiB) IOSize: 4(KiB) Throughput: 0.105713 GiB/sec, Avg_Latency: 2301.201214 usecs ops: 3317194 total_time 119.702494 secs
In this example we specified GPU 10 as the data read target. Note the dramatic difference in performance. With 64 threads generating random reads, latency went from 151.51us to 2.3ms, and IOPS dropped from 850k IOPS to about 28k IOPS. This is due to the overhead of GDS using a GPU on the same PCIe segment for the primary read buffer, then moving that data to the specified GPU. Again, this can be observed when monitoring GPU PCIe traffic:
dgx2> dcgmi dmon -e 1009,1010 -d 1000
# Entity PCITX PCIRX
Id
GPU 0 108216883 122481373
GPU 1 185690 61385
. . .
GPU 9 183268 60918
GPU 10 22110153 124205217
. . .
We observe PCIe traffic on both GPU 10, which was specified in the gdsio command line, and GPU 0, which
was selected by GDS as the primary read buffer due to its proximity to the NVMe devices. Using gds_stats,
we can see the buffer allocation on GPU 0:
dgx2> gds_stats -p 1545037 -l 3
cuFile STATS VERSION : 3
GLOBAL STATS:
Total Files: 64
Total Read Errors : 0
Total Read Size (MiB): 4996
Read BandWidth (GiB/s): 0.126041
Avg Read Latency (us): 2036
Total Write Errors : 0
Total Write Size (MiB): 0
Write BandWidth (GiB/s): 0
Avg Write Latency (us): 0
READ-WRITE SIZE HISTOGRAM :
0-4(KiB): 0 0
4-8(KiB): 1279109 0
8-16(KiB): 0 0
. . .
65536-...(KiB): 0 0
PER_GPU STATS:
GPU 0 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 MiB=0 BufRegister: n=0 err=0 free=0 MiB=0
. . .
GPU 9 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 MiB=0 BufRegister: n=0 err=0 free=0 MiB=0
GPU 10 Read: bw=0.124332 util(%)=6387 n=1279109 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=4996 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 MiB=0 BufRegister: n=64 err=0 free=0 MiB=0
GPU 11 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 r_sparse=0 r_inline=0 err=0 MiB=0 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 MiB=0 BufRegister: n=0 err=0 free=0 MiB=0
. . .
PER_GPU POOL BUFFER STATS:
GPU : 0 pool_size_MiB : 64 usage : 63/64 used_MiB : 63
The output from gds_stats shows Read activity on GPU 10 (specified on the gdsio command line), and POOL BUFFER
activity on GPU 0, with 63 of 64 1MB buffers in use. Recall GDS selected GPU 0 because it’s the first GPU on the same
PCIe segment as the NVMe drives. This illustrates one of the uses of the GPU POOL BUFFER (see section on gds_stats).
There are two key points to consider based on these results. First, for small, random IO loads, a large number of threads generating load are necessary to assess peak performance capability. Second, for small, random IO loads, the performance penalty of a sub-optimal configuration is much more severe than was observed with large throughput-oriented IO loads.
3.2. GPUDirect Storage Performance on a DGX A100 System#
GDS is also supported on DGX A100 system, the world’s first 5 peta FLOPS AI system built with a new generation of GPUs, NVMe drives and network interfaces. Please refer to the DGX A100 product page for details. In this section, we will use the same test methodology we used on the DGX-2 example to benchmark GDS performance on a DGX A100 system.
First, we map out the GPU and NMVe drive affinity:
Checking the NVMe drive name and PICe BFD values:
dgxuser@dgxa100:~$ ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:08:00.0-nvme-1 -> ../../nvme0n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:09:00.0-nvme-1 -> ../../nvme1n1 lrwxrwxrwx 1 root root 13 Oct 26 10:51 pci-0000:22:00.0-nvme-1 -> ../../nvme2n1 lrwxrwxrwx 1 root root 15 Oct 26 10:51 pci-0000:22:00.0-nvme-1-part1 -> ../../nvme2n1p1 lrwxrwxrwx 1 root root 15 Oct 26 10:51 pci-0000:22:00.0-nvme-1-part2 -> ../../nvme2n1p2 lrwxrwxrwx 1 root root 13 Oct 26 10:51 pci-0000:23:00.0-nvme-1 -> ../../nvme3n1 lrwxrwxrwx 1 root root 15 Oct 26 10:51 pci-0000:23:00.0-nvme-1-part1 -> ../../nvme3n1p1 lrwxrwxrwx 1 root root 15 Oct 26 10:51 pci-0000:23:00.0-nvme-1-part2 -> ../../nvme3n1p2 lrwxrwxrwx 1 root root 9 Oct 26 10:51 pci-0000:25:00.3-usb-0:1.1:1.0-scsi-0:0:0:0 -> ../../sr0 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:52:00.0-nvme-1 -> ../../nvme4n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:53:00.0-nvme-1 -> ../../nvme5n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:89:00.0-nvme-1 -> ../../nvme6n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:8a:00.0-nvme-1 -> ../../nvme7n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:c8:00.0-nvme-1 -> ../../nvme8n1 lrwxrwxrwx 1 root root 13 Oct 26 10:52 pci-0000:c9:00.0-nvme-1 -> ../../nvme9n1
Checking the GPU index numbering correlated to the PCIe BFD:
dgxuser@gpu01:~$ nvidia-smi --query-gpu=index,name,pci.domain,pci.bus, --format=csv
index, name, pci.domain, pci.bus
0, A100-SXM4-40GB, 0x0000, 0x07
1, A100-SXM4-40GB, 0x0000, 0x0F
2, A100-SXM4-40GB, 0x0000, 0x47
3, A100-SXM4-40GB, 0x0000, 0x4E
4, A100-SXM4-40GB, 0x0000, 0x87
5, A100-SXM4-40GB, 0x0000, 0x90
6, A100-SXM4-40GB, 0x0000, 0xB7
7, A100-SXM4-40GB, 0x0000, 0xBD
Checking the NVMe drive and GPU PCIe slot relationship:
dgxuser@dgxa100:~$ lspci -tv | egrep -i "nvidia|NVMe" | +-01.1-[b1-cb]----00.0-[b2-cb]--+-00.0-[b3-b7]----00.0-[b4-b7]----00.0-[b5-b7]----00.0-[b6-b7]----00.0-[b7]----00.0 NVIDIA Corporation Device 20b0 | | | \-10.0-[bb-bd]----00.0-[bc-bd]----00.0-[bd]----00.0 NVIDIA Corporation Device 20b0 | | +-08.0-[be-ca]----00.0-[bf-ca]--+-00.0-[c0-c7]----00.0-[c1-c7]--+-00.0-[c2]----00.0 NVIDIA Corporation Device 1af1 | | | | +-01.0-[c3]----00.0 NVIDIA Corporation Device 1af1 | | | | +-02.0-[c4]----00.0 NVIDIA Corporation Device 1af1 | | | | +-03.0-[c5]----00.0 NVIDIA Corporation Device 1af1 | | | | +-04.0-[c6]----00.0 NVIDIA Corporation Device 1af1 | | | | \-05.0-[c7]----00.0 NVIDIA Corporation Device 1af1 | | | +-04.0-[c8]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | | +-08.0-[c9]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | +-01.1-[81-95]----00.0-[82-95]--+-00.0-[83-8a]----00.0-[84-8a]--+-00.0-[85-88]----00.0-[86-88]--+-00.0-[87]----00.0 NVIDIA Corporation Device 20b0 | | | +-10.0-[89]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | | \-14.0-[8a]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | | \-10.0-[8e-91]----00.0-[8f-91]--+-00.0-[90]----00.0 NVIDIA Corporation Device 20b0 | +-01.1-[41-55]----00.0-[42-55]--+-00.0-[43-48]----00.0-[44-48]----00.0-[45-48]----00.0-[46-48]--+-00.0-[47]----00.0 NVIDIA Corporation Device 20b0 | | | \-10.0-[4c-4f]----00.0-[4d-4f]--+-00.0-[4e]----00.0 NVIDIA Corporation Device 20b0 | | +-08.0-[50-54]----00.0-[51-54]--+-00.0-[52]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | | +-04.0-[53]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | +-03.2-[22]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 | +-03.3-[23]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 +-01.1-[01-13]----00.0-[02-13]--+-00.0-[03-09]----00.0-[04-09]--+-00.0-[05-07]----00.0-[06-07]----00.0-[07]----00.0 NVIDIA Corporation Device 20b0 | | +-10.0-[08]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | \-14.0-[09]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM173X | | \-10.0-[0d-0f]----00.0-[0e-0f]----00.0-[0f]----00.0 NVIDIA Corporation Device 20b0 dgxuser@dgxa100:~$
Then mapping the NVMe and GPU affinities:
GPU # |
GPU PCIe Bus # |
NVMe # |
NVMe PCIe Bus # |
|---|---|---|---|
0, 1 |
0x07, 0x0F |
nvme0, nvme1 |
0x08, 0x09 |
2, 3 |
0x47, 0x4e |
nvme4, nvme5 |
0x52, 0x53 |
4, 5 |
0x87, 0x90 |
nvme6, nvme7 |
0x89, 0x8a |
6, 7 |
0xb7, 0xbd |
nvme8, nvme9 |
0xc8, 0xc9 |
By default, all NVMe drives in DGX systems are configured as a RAID0 storage array /dev/md1,
but they are connected with different PCIe switches, as shown below.