17. GDS Library Tracing#
The GDS Library has USDT (static tracepoints), which can be used with Linux tools such as lttng,
bcc/bpf, perf. This section assumes familiarity with these tools.
The examples in this section show tracing by using the `bcc/bpf <iovisor/bcc>`__
tracing facility. GDS does not ship these tracing tools. Refer to Installing BCC
for more information about installing bcc/bpf tools. Users must have root privileges to install.
Note
The user must also have sudo access to use these tools.
17.1. Example: Display Tracepoints#
To display tracepoints, run the following command:
# ./tplist -l /usr/local/gds/lib/libcufile.so
Review the output, for example:
/usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_px_read /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_rdma_read /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_gds_read /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_gds_read_async /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_px_write /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_gds_write /usr/local/cuda-x.y/lib/libcufile.so cufio:cufio_gds_write_async /usr/local/cuda-x.y/lib/libcufile.so cufio-internal:cufio-internal-write-bb /usr/local/cuda-x.y/lib/libcufile.so cufio-internal:cufio-internal-read-bb /usr/local/cuda-x.y/lib/libcufile.so cufio-internal:cufio-internal-bb-done /usr/local/cuda-x.y/lib/libcufile.so cufio-internal:cufio-internal-io-done /usr/local/cuda-x.y/lib/libcufile.so cufio-internal:cufio-internal-map
17.1.1. Example: Tracepoint Arguments#
Here are examples of tracepoint arguments.
cufio_px_read
This tracepoint tracks POSIX IO reads and takes the following arguments:
Arg1: File descriptor
Arg 2: File offset
Arg 3: Read size
Arg 4: GPU Buffer offset
Arg 5: Return value
Arg 6: GPU ID for which IO is done
cufio_rdma_read
This tracepoint tracks IO reads for through WEKA file system and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Read size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_rdma_write
This tracepoint tracks IO reads for through WEKA file system and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Write size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_gds_read
This tracepoint tracks IO reads going through the GDS kernel drive and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Read size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_gds_read_async
This tracepoint tracks iO reads going through the GDS kernel driver and poll mode is set and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Read size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_px_write
This tracepoint tracks POSIX IO writes and takes the following arguments:
Arg1: File descriptor
Arg 2: File offset
Arg 3: Write size
Arg 4: GPU Buffer offset
Arg 5: Return value
Arg 6: GPU ID for which IO is done
cufio_gds_write
This tracepoint tracks IO writes going through the GDS kernel driver and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Write size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_gds_unaligned_write
This tracepoint tracks IO writes going through the GDS kernel driver if the IO was unaligned and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Write size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio_gds_write_async
This tracepoint tracks IO writes going through the GDS kernel driver, and poll mode is set and takes the following arguments:
Arg1: File descriptor
Arg2: File offset
Arg3: Write size
Arg4: GPU Buffer offset
Arg5: Return value
Arg6: GPU ID for which IO is done
Arg7: Is the IO done to GPU Bounce buffer
cufio-internal-write-bb
This tracepoint tracks IO writes going through internal GPU Bounce buffers and is specific to the EXAScaler file system and block device-based file systems. This tracepoint is in hot IO-path tracking in every IO and takes the following arguments:
Arg1: Application GPU (GPU ID)
Arg2: GPU Bounce buffer (GPU ID)
Arg3: File descriptor
Arg4: File offset
Arg5: Write size
Arg6: Application GPU Buffer offset
Arg7: Size is bytes transferred from application GPU buffer to target GPU bounce buffer.
Arg8: Total Size in bytes transferred so far through bounce buffer.
Arg9: Pending IO count in this transaction
cufio-internal-read-bb
This tracepoint tracks IO reads going through internal GPU Bounce buffers and is specific to the EXAScaler file system and block device-based file systems. This tracepoint is in hot IO-path tracking every IO and takes the following arguments:
Arg1: Application GPU (GPU ID)
Arg2: GPU bounce buffer (GPU ID)
Arg3: File descriptor
Arg4: File offset
Arg5: Read size
Arg6: Application GPU Buffer offset
Arg7: Size is bytes transferred from the GPU bounce buffer to application GPU buffer.
Arg8: Total Size in bytes transferred so far through bounce buffer.
Arg9: Pending IO count in this transaction.
cufio-internal-bb-done
This tracepoint tracks all IO going through bounce buffers and is invoked when IO is completed through bounce buffers. The tracepoint can be used to track all IO going through bounce buffers and takes the following arguments:
Arg1: IO-type READ - 0, WRITE - 1
Arg2: Application GPU (GPU ID)
Arg3: GPU Bounce buffer (GPU ID)
Arg4: File descriptor
Arg5: File offset
Arg6: Read/Write size
Arg7: GPU buffer offset
Arg8: IO is unaligned (1 - True, 0 - False)
Arg9: Buffer is registered (1 - True, 0 - False)
cufio-internal-io-done
This tracepoint tracks all IO going through the GDS kernel driver. This tracepoint is invoked when the IO is completed and takes the following arguments:
Arg1: IO-type READ - 0, WRITE - 1
Arg2: GPU ID for which IO is done
Arg3: File descriptor
Arg4: File offset
Arg5: Total bytes transferred
cufio-internal-map
This tracepoint tracks GPU buffer registration using cuFileBufRegister and takes the following arguments:
Arg1: GPU ID
Arg2: GPU Buffer size for which registration is done
Arg3: max_direct_io_size that was used for this buffer.
The shadow memory size is set in the /etc/cufile.json file.
Arg4: boolean value indicating whether buffer is pinned.
Arg5: boolean value indicating whether this buffer is a GPU bounce buffer.
Arg6: GPU offset.
The data type of each argument in these tracepoints can be found by running the following command:
# ./tplist -l /usr/local/cuda-x.y/lib/libcufile.so -vvv | grep cufio_px_read -A 7
cufio:cufio_px_read [sema 0x0]
Here is the output:
# ./tplist -l /usr/local/cuda-x.y/lib/libcufile.so -vvv | grep cufio_px_read -A 7
cufio:cufio_px_read [sema 0x0]
location #1 /usr/local/cuda-x.y/lib/libcufile.so 0x16437c
argument #1 4 signed bytes @ dx
argument #2 8 signed bytes @ cx
argument #3 8 unsigned bytes @ si
argument #4 8 signed bytes @ di
argument #5 8 signed bytes @ r8
argument #6 4 signed bytes @ ax
17.2. Example: Track the IO Activity of a Process that Issues cuFileRead/ cuFileWrite#
This example provides information about how you can track the IO activity of a process that
issues the cuFileRead or the cuFileWrite API.
Run the folloiwng command.
# ./funccount u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_* -i 1 -T -p 59467 Tracing 7 functions for "u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_*"... Hit Ctrl-C to end.
Review the output, for example:
cufio_gds_write 1891 16:21:13 FUNC COUNT cufio_gds_write 1852 16:21:14 FUNC COUNT cufio_gds_write 1865 ^C 16:21:14 FUNC COUNT cufio_gds_write 1138 Detaching...
17.3. Example: Display the IO Pattern of all the IOs that Go Through GDS#
This example provides information about how you can display and understand the IO pattern of all IOs that go through GDS.
Run the following command:
# ./argdist -C 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():size_t:arg3# Size Distribution'
Review the output, for example:
[16:38:22] IO Size Distribution COUNT EVENT 4654 arg3 = 1048576 7480 arg3 = 131072 9029 arg3 = 65536 13561 arg3 = 8192 14200 arg3 = 4096 [16:38:23] IO Size Distribution COUNT EVENT 4682 arg3 = 1048576 7459 arg3 = 131072 9049 arg3 = 65536 13556 arg3 = 8192 14085 arg3 = 4096 [16:38:24] IO Size Distribution COUNT EVENT 4678 arg3 = 1048576 7416 arg3 = 131072 9018 arg3 = 65536 13536 arg3 = 8192 14082 arg3 = 4096
The 1M, 128K, 64K, 8K, and 4K IOs are all completing reads through GDS.
17.4. Understand the IO Pattern of a Process#
You can review the output to understand the IO pattern of a process.
Run the following command.
# ./argdist -C 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():size_t:arg3#IO Size Distribution' -p 59702
Review the output.
[16:40:46] IO Size Distribution COUNT EVENT 20774 arg3 = 4096 [16:40:47] IO Size Distribution COUNT EVENT 20727 arg3 = 4096 [16:40:48] IO Size Distribution COUNT EVENT 20713 arg3 = 4096
Process 59702 issues 4K IOs.
17.5. IO Pattern of a Process with the File Descriptor on Different GPUs#
Run the following command.
# ./argdist -C 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size:arg1, arg6,arg3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize' -p `pgrep -n gdsio`
Review the output, for example:
[17:00:03] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 5482 arg1 = 87, arg6 = 2, arg3 = 131072 7361 arg1 = 88, arg6 = 1, arg3 = 65536 9797 arg1 = 89, arg6 = 0, arg3 = 8192 11145 arg1 = 74, arg6 = 3, arg3 = 4096 [17:00:04] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 5471 arg1 = 87, arg6 = 2, arg3 = 131072 7409 arg1 = 88, arg6 = 1, arg3 = 65536 9862 arg1 = 89, arg6 = 0, arg3 = 8192 11079 arg1 = 74, arg6 = 3, arg3 = 4096 [17:00:05] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 5490 arg1 = 87, arg6 = 2, arg3 = 131072 7402 arg1 = 88, arg6 = 1, arg3 = 65536 9827 arg1 = 89, arg6 = 0, arg3 = 8192 11131 arg1 = 74, arg6 = 3, arg3 = 4096
gdsio issues READS to 4 files with fd=87, 88,89, 74 to GPU 2, 1, 0, and 3 and with IO-SIZE of 128K, 64K, 8K, and 4K.
17.6. Determine the IOPS and Bandwidth for a Process in a GPU#
You can determine the IOPS and bandwidth for each process in a GPU.
Run the following command.
#./argdist -C 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1, arg6,arg3:arg6==0||arg6==3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize' -p `pgrep -n gdsio`
Review the output.
[17:49:33] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3:arg6==0||arg6==3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 9826 arg1 = 89, arg6 = 0, arg3 = 8192 11168 arg1 = 86, arg6 = 3, arg3 = 4096 [17:49:34] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3:arg6==0||arg6==3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 9815 arg1 = 89, arg6 = 0, arg3 = 8192 11141 arg1 = 86, arg6 = 3, arg3 = 4096 [17:49:35] u:/usr/local/cuda-x.y/lib/libcufile.so:cufio_gds_read():int,int,size_t:arg1,arg6,arg3:arg6==0||arg6==3#IO Size Distribution arg1=fd, arg6=GPU# arg3=IOSize COUNT EVENT 9914 arg1 = 89, arg6 = 0, arg3 = 8192 11194 arg1 = 86, arg6 = 3, arg3 = 4096
gdsio is doing IO on all 4 GPUs, and the output is filtered for GPU 0 and GPU 3.
Bandwidth per GPU is GPU 0 - 9826 IOPS of 8K block size, and the bandwidth = ~80MB/s .
17.7. Display the Frequency of Reads by Processes that Issue cuFileRead#
You can display information about the frequency of reads by process that issue the cuFileRead API.
Run the following command.
#./argdist -C 'r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID'
Review the output, for example:
[17:58:01] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID COUNT EVENT 31191 $PID = 60492 31281 $PID = 60593 [17:58:02] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID COUNT EVENT 11741 $PID = 60669 30447 $PID = 60593 30670 $PID = 60492 [17:58:03] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID COUNT EVENT 29887 $PID = 60593 29974 $PID = 60669 30017 $PID = 60492 [17:58:04] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID COUNT EVENT 29972 $PID = 60593 30062 $PID = 60492 30068 $PID = 60669
17.8. Display the Frequency of Reads when cuFileRead Takes More than 0.1 ms#
You can display the frequency of reads when the cuFileRead API takes more than 0.1 ms.
Run the following command.
#./argdist -C 'r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000'
Review the output, for example:
[18:07:35] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000 COUNT EVENT 17755 $PID = 60772 [18:07:36] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000 COUNT EVENT 17884 $PID = 60772 [18:07:37] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000 COUNT EVENT 17748 $PID = 60772 [18:07:38] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000 COUNT EVENT 17898 $PID = 60772 [18:07:39] r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead():u32:$PID:$latency > 100000 COUNT EVENT 17811 $PID = 60772
17.9. Displaying the Latency of cuFileRead for Each Process#
You can display the latency of the the cuFileRead API for each process.
Run the following command.
#./funclatency /usr/local/cuda-x.y/lib/libcufile.so:cuFileRead -i 1 -T -u
Review the output, for example:
Tracing 1 functions for "/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead"... Hit Ctrl-C to end.
Here are two process with PID 60999 and PID 60894 that are issuing cuFileRead:
18:12:11 Function = cuFileRead [60999] usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 17973 |****************************************| 128 -> 255 : 13383 |***************************** | 256 -> 511 : 27 | | Function = cuFileRead [60894] usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 17990 |****************************************| 128 -> 255 : 13329 |***************************** | 256 -> 511 : 19 | | 18:12:12 Function = cuFileRead [60999] usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 18209 |****************************************| 128 -> 255 : 13047 |**************************** | 256 -> 511 : 58 | | Function = cuFileRead [60894] usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 18199 |****************************************| 128 -> 255 : 13015 |**************************** | 256 -> 511 : 46 | | 512 -> 1023 : 1 |
17.10. Example: Tracking the Processes that Issue cuFileBufRegister#
This example shows you can track processes that issue the cuFileBufRegister API.
Run the following command:
# ./trace 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio-internal-map "GPU %d Size %d Bounce-Buffer %d",arg1,arg2,arg5'
Review the output, for example:
PID TID COMM FUNC - 62624 62624 gdsio_verify cufio-internal-map GPU 0 Size 1048576 Bounce-Buffer 1 62659 62726 fio cufio-internal-map GPU 0 Size 8192 Bounce-Buffer 0 62659 62728 fio cufio-internal-map GPU 2 Size 131072 Bounce-Buffer 0 62659 62727 fio cufio-internal-map GPU 1 Size 65536 Bounce-Buffer 0 62659 62725 fio cufio-internal-map GPU 3 Size 4096 Bounce-Buffer 0
gdsio_verify issued an IO, but it did not register GPU memory using cuFileBufRegister. As a result, the GDS library pinned 1M of a bounce buffer on GPU 0. FIO, on the other hand, issued a cuFileBufRegister of 128K on GPU 2.
17.11. Example: Tracking Whether the Process is Constant when Invoking cuFileBufRegister#
You can track whether the process is constant when invoking the cuFileBufRegister API.
Run the following command:
# ./trace 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio-internal-map (arg5 == 0) "GPU %d Size %d",arg1,arg2'
Review the output, for example:
PID TID COMM FUNC - 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576 444 472 cufile_sample_0 cufio-internal-map GPU 0 Size 1048576
As seen in this example, there is one thread in a process that continuously issues 1M of cuFileBufRegister on GPU 0. This might mean that the API is called in a loop and might impact performance.
Note
cuFileBufRegister involves pinning GPU memory, which is an expensive operation.
17.12. Example: Monitoring IOs that are Going Through the Bounce Buffer#
This example shows how you can monitor whether IOs are going through the bounce buffer.
Run the following command:
# ./trace 'u:/usr/local/cuda-x.y/lib/libcufile.so:cufio-internal-bb-done "Application GPU %d Bounce-Buffer GPU %d Transfer Size %d Unaligned %d Registered %d", arg2,arg3,arg8,arg9,arg10'
Review the output, for example:
PID TID COMM FUNC -
1013 1041 gdsio App-GPU 0 Bounce-Buffer GPU 0 Transfer Size 1048576 Unaligned 1 Registered 0
1013 1042 gdsio App-GPU 3 Bounce-Buffer GPU 3 Transfer Size 1048576 Unaligned 1 Registered 0
1013 1041 gdsio App-GPU 0 Bounce-Buffer GPU 0 Transfer Size 1048576 Unaligned 1 Registered 0
1013 1042 gdsio App-GPU 3 Bounce-Buffer GPU 3 Transfer Size 1048576 Unaligned 1 Registered 0
The ``gdsio`` app has 2 threads and both are doing unaligned IO on GPU 0 and GPU 3. Since the IO is unaligned, bounce buffers are also from the same application GPU.
17.13. Example: Tracing cuFileRead and cuFileWrite Failures, Print, Error Codes, and Time of Failure#
This example shows you how to trace the cuFileRead and cuFileWrite failures, print, error codes, and time of failure.
Run the following command:
# ./trace 'r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileRead ((int)retval < 0) "cuFileRead failed: %d", retval' 'r:/usr/local/cuda-x.y/lib/libcufile.so:cuFileWrite ((int)retval < 0) "cuFileWrite failed: %d", retval' -T
Review the output, for example:
TIME PID TID COMM FUNC - 23:22:16 4201 4229 gdsio cuFileRead cuFileRead failed: -5 23:22:42 4237 4265 gdsio cuFileWrite cuFileWrite failed: -5
In this example, two failures were observed with EIO (-5) as the return code with the timestamp.
17.14. Example: User-Space Statistics for Each GDS Process#
The cuFile library exports user-level statistics in the form of API level counters for each process.
In addition to the regular GDS IO path, there are paths for user-space file-systems and IO compatibility
modes that use POSIX read/writes, which do not go through the nvidia-fs driver. The user-level statistics
are more useful in these scenarios.
There is a verbosity level for the counters which users can specify using JSON configuration file to enable and set the level. The following describes various verbosity levels.
Level |
Description |
|---|---|
Level 0 |
cuFile stats will be disabled. |
Level 1 |
cuFile stats will report only Global Counters like overall throughput, average latency and error counts. |
Level 2 |
With the Global Counters, an IO Size histogram will be reported for information on access patterns. |
Level 3 |
At this level, per GPU counters are reported and also live usage of cuFile internal pool buffers. |
The following is the JSON configuration key to enable GDS statistics by using the /etc/cufile.json file:
"profile": {
// cufile stats level(0-3)
"cufile_stats": 3
},
17.15. Example: Viewing GDS User-Level Statistics for a Process#
This example shows how you can use the gds_stats tool to display user-level statistics for a process.
Prerequisite: Before you run the tool, ensure that the IO application is active, and the gds_stats
has the same user permissions as the application.
The gds_stats tool can be used to read statistics that are exported by libcufile.so.
The output of the statistics is displayed in the standard output. If the user permissions are not the same,
there might not be sufficient privilege to view the stats. A future version of gds_stats will integrate
nvidia-fs kernel level statistics into this tool.
To use the tool, run the following command:
$ /usr/local/cuda-x.y/tools/gds_stats -p <pidof application> -l <stats_level(1-3)>
When specifying the statistics level, ensure that the corresponding level (profile.cufile_stats)
is also enabled in the /etc/cufile.json file.
The GDS user level statistics are logged once to cufile.log file when the library is shut down,
or the cuFileDriverClose API is run. To view statistics in the log file, set the log level to INFO.
17.16. Example: Displaying Sample User-Level Statistics for Each GDS Process#
This example shows how to display sample user-level statistics for each GDS process.
Run the following command:
$ ./gds_stats -p 23198 -l 3
Review the output, for example:
cuFile STATS VERSION : 8 GLOBAL STATS: Read: ok = 215814 err = 0 Write: ok = 0 err = 0 HandleRegister: ok = 1 err = 0 HandleDeregister: ok = 0 err = 0 BufRegister: ok = 128 err = 0 BufDeregister: ok = 0 err = 0 BatchSubmit: ok = 0 err = 0 BatchComplete: ok = 0 err = 0 BatchSetup: ok = 0 err = 0 BatchCancel: ok = 0 err = 0 BatchDestroy: ok = 0 err = 0 BatchEnqueued: ok = 0 err = 0 PosixBatchEnqueued: ok = 0 err = 0 BatchProcessed: ok = 0 err = 0 PosixBatchProcessed: ok = 0 err = 0 Total Read Size (MiB): 107907 Read BandWidth (GiB/s): 2.50343 Avg Read Latency (us): 49731 Total Write Size (MiB): 0 Write BandWidth (GiB/s): 0 Avg Write Latency (us): 0 Total Batch Read Size (MiB): 0 Total Batch Write Size (MiB): 0 Batch Read BandWidth (GiB/s): 0 Batch Write BandWidth (GiB/s): 0 Avg Batch Submit Latency (us): 0 Avg Batch Completion Latency (us): 0 READ-WRITE SIZE HISTOGRAM : 0-4(KiB): 0 0 4-8(KiB): 0 0 8-16(KiB): 0 0 16-32(KiB): 0 0 32-64(KiB): 0 0 64-128(KiB): 0 0 128-256(KiB): 0 0 256-512(KiB): 0 0 512-1024(KiB): 0 0 1024-2048(KiB): 107907 0 2048-4096(KiB): 0 0 4096-8192(KiB): 0 0 8192-16384(KiB): 0 0 16384-32768(KiB): 0 0 32768-65536(KiB): 0 0 65536-...(KiB): 0 0 PER_GPU STATS: GPU 0(UUID: ce4dfa044611339ca1e22bf10a772fe) Read: bw=2.50531 util(%)=12791 n=107907 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0 err=0 MiB=107907 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 err=0 MiB=0 BufRegister: n=128 err=0 free=0 MiB=128 PER_GPU POOL BUFFER STATS: PER_GPU POSIX POOL BUFFER STATS: PER_GPU RDMA STATS: GPU 0000:43:00.0(UUID: ce4dfa044611339ca1e22bf10a772fe) : RDMA MRSTATS: peer name nr_mrs mr_size(MiB) mlx5_0 1 2 mlx5_1 1 2 PER GPU THREAD POOL STATS: gpu node: 0 enqueues:0 completes:0 pending suspends:0 pending yields:0 active:0 suspends:0