5. Troubleshooting Performance#

This section covers issues related to performance.

5.1. Running Performance Benchmarks with GDS#

You can run performance benchmarks with GDS and compare the results with CPU numbers.

GDS has a homegrown benchmarking utility, /usr/local/cuda-x.y/gds/tools/gdsio, which helps you compare GDS IO throughput numbers with CPU IO throughput. Run gdsio -h for detailed usage information.

Here are some examples:

GDS: Storage –> GPU Memory

$ /usr/local/cuda-x.y/tools/gdsio -f /mnt/ai200/fio-seq-writes-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 0

Storage –> CPU Memory

$ /usr/local/cuda-x.y/tools/gdsio -f /mnt/ai200/fio-seq-writes-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 1

Storage –> CPU Memory –> GPU Memory

$ /usr/local/cuda-x.y/tool/gdsio -f /mnt/ai200/fio-seq-writes-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 2

Storage –> GPU Memory using batch mode

$ /usr/local/cuda-x.y/tool/gdsio -f /mnt/ai200/fio-seq-read-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 6

Storage –> GPU Memory using async stream mode

$ /usr/local/cuda-x.y/tool/gdsio -f /mnt/ai200/fio-seq-read-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 5

5.2. Tracking Whether GPUDirect Storage is Using an Internal Cache#

You can determine whether GDS is using an internal cache.

Prerequisite: Before you start, read the GPUDirect Storage Best Practices Guide.

GDS Stats has per-GPU stats, and each piece of the GPU bus device function (BDF) information is displayed. If the cache_MB field is active on a GPU, GDS is using the cache internally to complete the IO.

GDS might use the internal cache when one of the following conditions are true:

The file_offset that was issued in cuFileRead/cuFileWrite is not 4K aligned.
The size in cuFileRead/cuFileWrite calls are not 4K aligned.
The devPtr_base that was issued in cuFileRead/cuFileWrite is not 4K aligned.
The devPtr_base+devPtr_offset that was issued in cuFileRead/cuFileWrite is not 4K aligned.

5.3. Tracking when IO Crosses the PCIe Root Complex and Impacts Performance#

You can track when the IO crosses the PCIe root complex and affects performance.

Refer to Checking Peer Affinity Stats for a Kernel File System and Storage Drivers for more information.

5.4. Using GPUDirect Statistics to Monitor CPU Activity#

Although you cannot use GDS statistics to monitor CPU activity, you can use the following Linux tools to complete this task:

htop
perf
mpstat

5.5. Monitoring Performance and Tracing with cuFile-* APIs#

You can monitor performance and tracing with the cuFile-* APIs.

You can use the FTrace, the Perf, or the BCC-BPF tools to monitor performance and tracing. Ensure that you have the symbols that you can use to track and monitor the performance with a standard Linux IO tool.

5.6. Example: Using Linux Tracing Tools#

The cuFileBufRegister function makes the pages that underlie a range of GPU virtual memory accessible to a third-party device. This process is completed by pinning the GPU device memory in the BAR space, which is an expensive operation and can take up to a few milliseconds.

You can using the BCC/BPF tool to trace the cuFileBufRegister API, understand what is happening in the Linux kernel, and understand why this process is expensive.

Scenario

You are running a workload with 8 threads where each thread is issuing cuFileBufRegister to pin to the GPU memory.
```
$ ./gdsio -f /mnt/ai200/seq-writes-1 -d 0 -w 8 -s 10G -i 1M -I 0 -x 0
```
When IO is in progress, use a tracing tool to understand what is going on with cuFileBufRegister:
```
$ /usr/share/bcc/tools# ./funccount -Ti 1 nvfs_mgroup_pin_shadow_pages
```
Review the sample output:
```
15:04:56
FUNC                                    COUNT
nvfs_mgroup_pin_shadow_pages            8
```
As you can see, the nvfs_mgroup_pin_shadow_pages function has been invoked 8 times in one per thread.

To see the latency for that function, run:

$ /usr/share/bcc/tools# ./funclatency -i 1 nvfs_mgroup_pin_shadow_pages

Review the output:

Tracing 1 functions for "nvfs_mgroup_pin_shadow_pages"... Hit Ctrl-C to end.

     nsecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 0        |                                        |
      4096 -> 8191       : 0        |                                        |
      8192 -> 16383      : 1        |*****                                   |
     16384 -> 32767      : 7        |****************************************|

Seven calls of the nvfs_mgroup_pin_shadow_pages function took about 16-32 microseconds. This is probably coming from the Linux kernel get_user_pages_fast that is used to pin shadow pages.

cuFileBufRegister invokes nvidia_p2p_get_pages NVIDIA driver function to pin GPU device memory in the BAR space. This information is obtained by running $ perf top -g and getting the call graph of cuFileBufRegister.

The following example the overhead of the nvidia_p2p_get_pages:

$ /usr/share/bcc/tools# ./funclatency -Ti 1 nvidia_p2p_get_pages

45:19
nsecs               : count     distribution
-> 1          : 0        |                                        |
-> 3          : 0        |                                        |
-> 7          : 0        |                                        |
-> 15         : 0        |                                        |
-> 31         : 0        |                                        |
-> 63         : 0        |                                        |
-> 127        : 0        |                                        |
-> 255        : 0        |                                        |
-> 511        : 0        |                                        |
-> 1023       : 0        |                                        |
-> 2047       : 0        |                                        |
-> 4095       : 0        |                                        |
-> 8191       : 0        |                                        |
-> 16383      : 0        |                                        |
-> 32767      : 0        |                                        |
-> 65535      : 0        |                                        |
-> 131071     : 0        |                                        |
131072 -> 262143     : 0        |                                        |
262144 -> 524287     : 2        |*************                           |
524288 -> 1048575    : 6        |****************************************|

5.7. Tracing the cuFile* APIs#

You can use nvprof/NVIDIA Nsight to trace the cuFile* APIs.

NVTX static tracepoints are available for the public interfaces in the libcufile.so library, and NVTX tracing is enabled by default. After these static tracepoints are enabled, you can view these traces in NVIDIA Nsight just like any other CUDA symbols.

5.8. Improving Performance using Dynamic Routing#

On platforms where the IO transfers between GPU(s) and the storage NICs involve PCIe traffic across PCIe-host bridge, GPUDirect Storage IO may not see a great throughput especially for writes. Also, certain chipsets may support only P2P read traffic for host bridge traffic. In such cases, the dynamic routing feature can be enabled to debug and identify what routing policy is deemed best for such platforms. This can be illustrated with a single GPU write test with the gdsio tool, where there is one Storage NIC and 10 GPUs with NVLINKs access enabled between the GPUS. With dynamic routing enabled, even though the GPU and NIC might be on different sockets, GDS can still achieve the maximum possible write throughput.

$ cat /etc/cufile.json | grep rdma_dev
         "rdma_dev_addr_list": [ "192.168.0.19" ],

Dynamic Routing OFF:

$ cat /etc/cufile.json | grep routing
         "rdma_dynamic_routing": false
$ for i in 0 1 2 3 4 5 6 7 8 9 10;
    do
 ./gdsio -f /mnt/nfs/file1 -d $i -n 0 -w 4 -s 1G -i 1M -x 0 -I 1 -p -T 15 ;
    done

IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 45792256/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.873560 GiB/sec, Avg_Latency: 1359.280174 usecs ops: 44719 total_time 15.197491 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 45603840/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.867613 GiB/sec, Avg_Latency: 1363.891220 usecs ops: 44535 total_time 15.166344 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 42013696/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.848411 GiB/sec, Avg_Latency: 1373.154082 usecs ops: 41029 total_time 14.066573 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 43517952/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.880763 GiB/sec, Avg_Latency: 1358.207427 usecs ops: 42498 total_time 14.406582 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 34889728/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.341907 GiB/sec, Avg_Latency: 1669.108902 usecs ops: 34072 total_time 14.207836 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 36955136/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.325239 GiB/sec, Avg_Latency: 1680.001220 usecs ops: 36089 total_time 15.156790 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 37075968/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.351491 GiB/sec, Avg_Latency: 1661.198487 usecs ops: 36207 total_time 15.036584 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 35066880/4194304(KiB) IOSize: 1024(KiB) Throughput: 2.235654 GiB/sec, Avg_Latency: 1748.638950 usecs ops: 34245 total_time 14.958656 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 134095872/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.940253 GiB/sec, Avg_Latency: 436.982682 usecs ops: 130953 total_time 14.304269 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 135974912/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.932070 GiB/sec, Avg_Latency: 437.334849 usecs ops: 132788 total_time 14.517998 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 174486528/4194304(KiB) IOSize: 1024(KiB) Throughput: 11.238476 GiB/sec, Avg_Latency: 347.603610 usecs ops: 170397 total_time 14.806573 secs

Dynamic Routing ON (nvlinks enabled):

$ cat /etc/cufile.json | grep routing
         "rdma_dynamic_routing": true
         "rdma_dynamic_routing_order": [ "GPU_MEM_NVLINKS"]

$ for i in 0 1 2 3 4 5 6 7 8 9 10;
do
./gdsio -f  /mnt/nfs/file1 -d $i -n 0 -w 4 -s 1G -i 1M -x 0 -I 1 -p -T 15 ;
done

IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 134479872/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.885214 GiB/sec, Avg_Latency: 437.942083 usecs ops: 131328 total_time 14.434092 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 138331136/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.891407 GiB/sec, Avg_Latency: 437.668104 usecs ops: 135089 total_time 14.837118 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 133800960/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.897250 GiB/sec, Avg_Latency: 437.305565 usecs ops: 130665 total_time 14.341795 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 133990400/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.888714 GiB/sec, Avg_Latency: 437.751327 usecs ops: 130850 total_time 14.375893 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 141934592/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.905190 GiB/sec, Avg_Latency: 437.032919 usecs ops: 138608 total_time 15.200055 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 133379072/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.892493 GiB/sec, Avg_Latency: 437.488259 usecs ops: 130253 total_time 14.304222 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 142271488/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.892426 GiB/sec, Avg_Latency: 437.660016 usecs ops: 138937 total_time 15.258004 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 134951936/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.890496 GiB/sec, Avg_Latency: 437.661177 usecs ops: 131789 total_time 14.476154 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 132667392/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.930203 GiB/sec, Avg_Latency: 437.420830 usecs ops: 129558 total_time 14.167817 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 137982976/4194304(KiB) IOSize: 1024(KiB) Throughput: 8.936189 GiB/sec, Avg_Latency: 437.123356 usecs ops: 134749 total_time 14.725608 secs
    url index :0, urlname :192.168.0.2 urlport :18515
IoType: WRITE XferType: GPUD Threads: 4 DataSetSize: 170469376/4194304(KiB) IOSize: 1024(KiB) Throughput: 11.231479 GiB/sec, Avg_Latency: 347.818052 usecs ops: 166474 total_time 14.474698 secs