19. User-Space RDMA Counters in GPUDirect Storage#
The library provides counters to monitor the RDMA traffic at a per-GPU level and requires that cuFile starts verbosity with a value of 3.
Table cuFile RDMA IO Counters (PER_GPU RDMA STATS) provides the following information:
Each column stores the total number of bytes that are sent/received between a GPU and a NIC.
Each row shows the distribution of RDMA load with regards to a GPU across all NICS.
Each row reflects the order of affinity that a GPU has with a NIC.
Ideally, all traffic should be routed through the NIC with the best affinity or is closest to the GPU as shown in Example 1.
In the annotation of each NIC entry in the table, the major number is the pci-distance in terms of the number
of hops between the GPU and the NIC, and the minor number indicates the current bandwidth of the NIC (link_width
multiplied by pci-generation). The NICs that the GPUs use for RDMA are loaded from the rdma_dev_addr_list cufile.json property:
"rdma_dev_addr_list": [
"172.172.1.240",
"172.172.1.241",
"172.172.1.242",
"172.172.1.243",
"172.172.1.244",
"172.172.1.245",
"172.172.1.246",
"172.172.1.247" ],
Each IP address corresponds to an IB device that appear as column entries in the RDMA counter table.
19.1. cuFile RDMA IO Counters (PER_GPU RDMA STATS)#
The following tables list cuFile RDMA IO counters.
Entry |
Description |
|---|---|
GPU |
Bus device function |
NIC |
+)Bus device function
+)Device Attributes
++)pci-distance between GPU and NIC
++)device bandwidth indicator
+)Send/Receive bytes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
19.2. cuFile RDMA Memory Registration Counters (RDMA MRSTATS)#
The following tables list cuFile RDMA memory registeration counters.
Entry |
Description |
|---|---|
peer name |
System name of the NIC. |
nr_mrs |
Count of active memory registration per NIC. |
mr_size(mb) |
Total size |
peer name |
nr_ms |
mr_size (mb) |
|---|---|---|
mlx5_3 |
128 |
128 |
mlx5_5 |
128 |
128 |
mlx5_11 |
0 |
0 |
mlx5_1 |
0 |
0 |
mlx5_15 |
128 |
128 |
mlx5_19 |
128 |
128 |
mlx5_17 |
128 |
128 |
mlx5_9 |
128 |
128 |
mlx5_13 |
128 |
128 |
mlx5_7 |
128 |
128 |