18. User-Space Counters in GPUDirect Storage

The following tables provide information about user-space counters in GDS.

Table 18.1 Global cuFile Counters

Counter Name	Description
Total Files	Total number of files registered successfully with cuFileHandleRegister. This is a cumulative counter. cuFileHandleDeregister does not change this counter.
Total Read Errors	Total number of cuFileRead errors.
Total Read Size	Total number of bytes read in MB using cuFileRead.
Read Bandwidth	Average overall read throughput in GiB/s over one second time period.
Avg Read Latency	Overall average read latency in microseconds over one second time period.
Total Write Errors	Total number of cuFileWrite errors.
Total Write Size	Total number of bytes written in MB using cuFileWrite.
Write Bandwidth	Overall average write throughput in GiB/s over one second time period.
Avg Write Latency	Overall average read latency in microseconds over one second time period.
Total Batch Read Size	Total number of bytes read in MB using cuFile Batch mode.
Total Batch Write Size	Total number of bytes written in MB using cuFile Batch mode.
Batch Read BandWidth	Average overall read throughput in GiB/s over one second time period for cuFile Batch Mode.
Batch Write BandWidth	Average overall write throughput in GiB/s over one second time period for cuFile Batch Mode.
Avg Batch Submit Latency	Overall average cuFile Batch IO submit latency in microseconds over one second time period using cuFileBatchIOSubmit.
Avg Batch Completion Latency	Overall average cuFile Batch IO completion latency in microseconds over one second time period. This includes submission and completion times.

Table 18.2 IO-Size Histogram

Counter Name	Description
Read	Distribution of number of cuFileRead requests based on IO size. Bin Size uses a 4K log scale.
Write	Distribution of number of cuFileWrite requests based on IO size. Bin Size uses a 4K log scale.

Table 18.3 Per-GPU Counters

Counter Name	Description
Read.bw/Write.bw	Average GPU read/write bandwidth in GiB/s per GPU.
Read.util/Write.util	Average per GPU read/write utilization in %. If A is the total length of time the resource was busy in a time interval T, then utilization is defined as A/T. Here the utilization is reported over one second period.
Read.n/Write.n	Number of cuFileRead/cuFileWrite requests per GPU.
Read.posix/Write.posix	Number of cuFileRead/cuFileWrite using POSIX read/write APIs per GPU.
Read.dr/Write.dr	Number of cuFileRead/cuFileWrites for a GPU have been issued using dynamic routing. If the routing policy uses SYS_MEM, GPU posix counters for read/writes will be incrementing in addition to the dr counter. Note: This counter does not tell which GPU was actually being used for routing the IO. For the latter information, one needs to observe the PER_GPU POOL BUFFER STATS/PER_GPU POSIX POOL BUFFER STATS.
Read.unalign/Write.unalign	Number of cuFileRead/cuFileWrite per GPU which have at least one IO parameter not 4K aligned. This can be either size, file offset or device pointer.
Read.error/Write.error	Number of cuFileRead/cuFileWrite errors per GPU.
Read.mb/Write.mb	Total number of bytes in MB read/written using cuFileRead/cuFileWrite per GPU.
BufRegister.n	Total number of cuFileBufRegister calls per GPU.
BufRegister.err	Total number of errors per GPU seen with cuFileBufRegister.
BufRegister.free	Total number of cuFileBufRegister calls per GPU.
BufRegister.mb	Total number of bytes in MB currently registered per GPU.

Table 18.4 Bounce Buffer Counters Per GPU

Counter Name	Description
pool_size_mb	Total size of buffers allocated for per GPU bounce buffers in MB.
used_mb	Total size of buffers currently used per GPU for bounce buffer based IO.
usage	Fraction of bounce buffers used currently.

Table 18.5 Register Calls

Counter Name	Description
HandleRegister HandleDeregister	ok: Number of cuFileHandleRegister calls that have been issued and completed successfully. err: Number of cuFileHandleRegister calls that have been issued and completed with errors. ok: Number of cuFileHandleDeregister calls that have been issued and completed successfully. err: Number of cuFileHandleDeregister calls that have been issued and completed with errors.
BufRegister	ok: Number of cuFileBufRegister calls that have been issued and completed successfully. err: Number of cuFileBufRegister calls that have been issued and completed with errors.
BufDeregister	ok: Number of cuFileBufDeregister calls that have been issued and completed successfully. err: Number of cuFileBufDeregister calls that have been issued and completed with errors.
BatchSubmit	ok: Number of cuFileBatchIOSubmit calls that have been issued and completed successfully. err: Number of cuFileBatchIOSubmit calls that have been issued and completed with errors.
BatchComplete	ok: Number of cuFileBatchIOGetStatus calls that have been issued and completed successfully. err: Number of cuFileBatchIOGetStatus calls that have been issued and completed with errors.
BatchSetup	ok: Number of cuFileBatchIOSetUp calls that have been issued and completed successfully. err: Number of cuFileBatchIOSetUp calls that have been issued and completed with errors.
BatchCancel	ok: Number of cuFileBatchIOCancel calls that have been issued and completed successfully. err: Number of cuFileBatchIOCancel calls that have been issued and completed with errors.
BatchDestroy	ok: Number of cuFileBatchIODestroy calls that have been issued and completed successfully. err: Number of cuFileBatchIODestroy calls that have been issued and completed with errors.
BatchEnqueued	For batch entries that have unaligned size/offset, the entry can have an aligned(GDS path) and unaligned(Posix Path) portion. This entry indicates the number of GDS path IOs in such scenarios. These IOs are not directly submitted but enqueued in a threadpool. ok: Number of GDS path IOs successfully enqueued to the threadpool. err: Number of GDS path IOs that could not be enqueued to threadpool.
PosixBatchEnqueued	Similar to BatchEnqueued but used for the number of Posix IOs enqueued. ok: Number of Posix path IOs successfully enqueued to the threadpool. err: Number of Posix path IOs that could not be enqueued to threadpool.
BatchProcessed	This counter indicates the number of IOs processed for the IOs that are tracked using BatchEnqueued. ok: Number of IOs successfully processed. err: Number of IOs that completed with errors.
PosixBatchProcessed	This counter indicates the number of IOs processed for the IOs that are tracked using PosixBatchEnqueued. ok: Number of IOs successfully processed. err: Number of IOs that completed with errors.

18.1. Distribution of IO Usage in Each GPU

The cuFile library has a metric for IO utilization per GPU by application. This metric indicates the amount of time, in percentage, that the cuFile resource was busy in IO.

To run a single-threaded gdsio test, run the following command:

$./gdsio -f /mnt/md1/test -d 0 -n 0 -w 1 -s 10G -i 4K -x 0 -I 1

Here is the sample output:

PER_GPU STATS
GPU 0 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 mb=0 Write: bw=0.154598
util(%)=89 n=510588 posix=0 unalign=0 err=0 mb=1994 BufRegister: n=1 err=0 free=0 mb=0

The util metric says that the application was completing IO on GPU 0 89% of the time.

To run a gdsio test using two-threads, run the following command:

$./gdsio -f /mnt/md1/test -d 0 -n 0 -w 2 -s 10G -i 4K -x 0 -I 1

Here is the sample output:

PER_GPU STATS
GPU 0 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 err=0 mb=0 Write: bw=0.164967 util(%)=186 n=140854 posix=0 unalign=0 err=0 mb=550 BufRegister: n=2 err=0 free=0 mb=0

Now the utilization is ~186%, which indicates the amount of parallelism in the way each GPU is used for IO.

18.2. User-space Statistics for Dynamic Routing

The PER_GPU section of gds_stats has a dr counter which indicates how many cuFileRead/ cuFileWrites for a GPU have been issued using dynamic routing.

$ ./gds_stats -p <pidof application> -l 3

    GPU 0 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0
 err=0 MiB=0 Write: bw=3.37598 util(%)=532 n=6629 posix=0 unalign=0 dr=6629 err=0
 MiB=6629 BufRegister: n=4 err=0 free=0 MiB=4
    GPU 1 Read: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0
 err=0 MiB=0 Write: bw=3.29297 util(%)=523 n=6637 posix=0 unalign=0 dr=6637 err=0
 MiB=6637 BufRegister: n=4 err=0 free=0 MiB=4