7. EXAScaler File System LNet Troubleshooting#
This section describes how to troubleshoot issues with the EXAScaler file system.
7.1. Determining the EXAScaler File system Client Module Version#
To check the EXAScaler file system Client version, check dmesg after you install the EXAScaler file system.
Note
The EXAScaler server version should be EXA-5.2.
This table provides a list of the client kernel module versions that have been tested with DDN AI200 and DDN AI400 systems:
DDN Client Version |
Kernel Version |
MLNX_OFED version |
|---|---|---|
2.12.3_ddn28 |
4.15.0 |
MLNX_OFED 4.7 |
2.12.3_ddn29 |
4.15.0 |
MLNX_OFED 4.7 |
2.12.3_ddn39 |
4.15.0 |
MLNX_OFED 5.1 |
2.12.5_ddn4 |
5.4.0 |
MLNX_OFED 5.1 |
2.12.6_ddn19 |
5.4.0 |
MLNX_OFED 5.3 |
To verify the client version, run the following command:
$ sudo lctl get_param version
Sample output:
Lustre version: 2.12.3_ddn39
7.2. Checking the LNet Network Setup on a Client#
To check the LNet network setup on the client:
Run the following command.
$ sudo lnetctl net show:
Review the output, for example:
net: - net type: lo
7.3. Checking the Health of the Peers#
An Lnet health value of 1000 is the best possible value that can be reported for a network interface. Anything less than 1000 indicates that the interface is running in a degraded mode and has encountered some errors.
Run the following command;
$ sudo lnetctl net show -v 3 | grep health
Review the output, for example:
health stats: health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000
7.4. Checking for Multi-Rail Support#
To verify whether multi-rail is supported:
Run the following command:
$ sudo lnetctl peer show | grep -i Multi-Rail:
Review the output, for example:
Multi-Rail: True
7.5. Checking GDS Peer Affinity#
For peer affinity, you need to check whether the expected interfaces are being used for the associated GPUs.
The code snippet below is a description of a test that runs load on a specific GPU. The test validates whether the interface that is performing the send and receive is the interface that is the closest, and is correctly mapped, to the GPU. See Resetting the nvidia-fs Statistics and Checking Peer Affinity Stats for a Kernel File System and Storage Drivers for more information about the metrics that are used to check peer affinity.
You can run a gdsio test for the tools section and monitor the LNET stats. See the readme file for more information.
In the gdsio test, a write test has been completed on GPU 0. The expected NIC interface for GPU 0 is ib0 on the
NVIDIA DGX-2 platform. The lnetctl net show statistics were previously captured, and after the gdsio
test, you can see that the RPC send and receive have happened over the IB0.
Run the
gdsiotest.Review the output, for example:
$ sudo lustre_rmmod $ sudo mount -t lustre 192.168.1.61@o2ib,192.168.1.62@o2ib:/ai200 /mnt/ai200/ $ sudo lnetctl net show -v 3 | grep health health stats: health value: 0 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 health stats: health value: 1000 $ sudo lnetctl net show -v 3 | grep -B 2 -i 'send_count\|recv_count' status: up statistics: send_count: 0 recv_count: 0 -- 0: ib0 statistics: send_count: 3 recv_count: 3 -- 0: ib2 statistics: send_count: 3 recv_count: 3 -- 0: ib3 statistics: send_count: 2 recv_count: 2 -- 0: ib4 statistics: send_count: 13 recv_count: 13 -- 0: ib5 statistics: send_count: 12 recv_count: 12 -- 0: ib6 statistics: send_count: 12 recv_count: 12 -- 0: ib7 statistics: send_count: 11 recv_count: 11 $ echo 1 > /sys/module/nvidia_fs/parameters/peer_stats_enabled $ /usr/local/cuda-x.y/tools/gdsio -f /mnt/ai200/test -d 0 -n 0 -w 1 -s 1G -i 4K -x 0 -I 1 IoType: WRITE XferType: GPUD Threads: 1 DataSetSize: 1073741824/1073741824 IOSize: 4(KB),Throughput: 0.004727 GB/sec, Avg_Latency: 807.026154 usecs ops: 262144 total_time 211562847.000000 usecs $ sudo lnetctl net show -v 3 | grep -B 2 -i 'send_count\|recv_count' status: up statistics: send_count: 0 recv_count: 0 -- 0: ib0 statistics: send_count: 262149 recv_count: 524293 -- 0: ib2 statistics: send_count: 6 recv_count: 6 -- 0: ib3 statistics: send_count: 6 recv_count: 6 -- 0: ib4 statistics: send_count: 33 recv_count: 33 -- 0: ib5 statistics: send_count: 32 recv_count: 32 -- 0: ib6 statistics: send_count: 32 recv_count: 32 -- 0: ib7 statistics: send_count: 32 recv_count: 32 $ cat /proc/driver/nvidia-fs/peer_affinity GPU P2P DMA distribution based on pci-distance (last column indicates p2p via root complex) GPU :0000:be:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:3b:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e5:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e0:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:57:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:39:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:36:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e2:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:59:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:b7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:b9:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:bc:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:34:00.0 :0 0 23872512 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:5e:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:5c:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7.6. Checking for LNet-Level Errors#
The errors impact the health of individual NICs and affect how the EXAScaler file system selects the best peer, which impacts GDS performance.
Note
To run these commands, you must have sudo priveleges.
Run the following command:
$ cat /proc/driver/nvidia-fs/peer_affinity
Review the ouput, for example:
GPU P2P DMA distribution based on pci-distance (last column indicates p2p via root complex) GPU :0000:be:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:3b:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e5:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e0:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:57:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1276417 (Note : if peer traffic goes over Root-Port, one of the reasons might be that health of nearest NIC might be affected) GPU :0000:39:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:36:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:e2:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:59:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:b7:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:b9:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:bc:00.0 :0 0 7056141 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:34:00.0 :0 0 8356175 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:5e:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GPU :0000:5c:00.0 :0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 $ sudo lnetctl stats show statistics: msgs_alloc: 1 msgs_max: 126 rst_alloc: 25 errors: 0 send_count: 243901 resend_count: 1 response_timeout_count: 1935 local_interrupt_count: 0 local_dropped_count: 208 local_aborted_count: 0 local_no_route_count: 0 local_timeout_count: 1730 local_error_count: 0 remote_dropped_count: 0 remote_error_count: 0 remote_timeout_count: 0 network_timeout_count: 0 recv_count: 564436 route_count: 0 drop_count: 0 send_length: 336176013248 recv_length: 95073248 route_length: 0 drop_length: 0 lnetctl net show -v 4 net: - net type: o2ib local NI(s): - nid: 192.168.1.71@o2ib status: up interfaces: 0: ib0 statistics: send_count: 171621 recv_count: 459717 drop_count: 0 sent_stats: put: 119492 get: 52129 reply: 0 ack: 0 hello: 0 received_stats: put: 119492 get: 0 reply: 340225 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 32 peer_buffer_credits: 0 credits: 256 peercredits_hiw: 16 map_on_demand: 1 concurrent_sends: 64 fmr_pool_size: 512 fmr_flush_trigger: 384 fmr_cache: 1 ntx: 512 conns_per_peer: 1 lnd tunables: dev cpt: 0 tcp bonding: 0 CPT: "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]" - nid: 192.168.2.71@o2ib status: up interfaces: 0: ib1 statistics: send_count: 79 recv_count: 79 drop_count: 0 sent_stats: put: 78 get: 1 reply: 0 ack: 0 hello: 0 received_stats: put: 78 get: 0 reply: 1 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 979 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 1 error: 0 tunables: peer_timeout: 180 peer_credits: 32 peer_buffer_credits: 0 credits: 256 peercredits_hiw: 16 map_on_demand: 1 concurrent_sends: 64 fmr_pool_size: 512 fmr_flush_trigger: 384 fmr_cache: 1 ntx: 512 conns_per_peer: 1 lnd tunables: dev cpt: 0 tcp bonding: 0 CPT: "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]" - nid: 192.168.2.72@o2ib status: up interfaces: 0: ib3 statistics: send_count: 52154 recv_count: 52154 drop_count: 0 sent_stats: put: 25 get: 52129 reply: 0 ack: 0 hello: 0 received_stats: put: 25 get: 52129 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 66 interrupts: 0 dropped: 208 aborted: 0 no route: 0 timeouts: 1735 error: 0 tunables: peer_timeout: 180 peer_credits: 32 peer_buffer_credits: 0 credits: 256 peercredits_hiw: 16 map_on_demand: 1 concurrent_sends: 64 fmr_pool_size: 512 fmr_flush_trigger: 384 fmr_cache: 1 ntx: 512 conns_per_peer: 1 lnd tunables: dev cpt: 0 tcp bonding: 0 CPT: "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]"
If you see incrementing error stats, capture the net logging and provide this information for debugging:
$ lctl set_param debug=+net
# reproduce the problem
$ lctl dk > logfile.dk
7.7. Resolving LNet NIDs Health Degradation from Timeouts#
With large machines, such as DGX that have multiple interfaces, if Linux routing is not correctly set up, there might be connection failures and other unexpected behavior.
A typical network setting that is used to resolve local connection timeouts is:
sysctl -w net.ipv4.conf.all.accept_local=1
There are also generic pointers for resolving LNet Network issues. Refer to MR Cluster Setup for more information.
7.8. Configuring LNet Networks with Multiple OSTs for Optimal Peer Selection#
When there are multiple OSTs (Object Storage Targets), and each OST is dual interface, to need to have one interface on each of the LNets for which the client is configured.
For example, you have the following two LNet Subnets on the client side:
o2ibo2ib1
The server has only one Lnet subnet, o2ib. In this situation, the routing is not optimal, because you
are restricting the ib selection logic to a set of devices, which may not be closest to the GPU. There is
no way to reach OST2 except over the LNet to which it is connected.
The traffic that goes to this OST will never be optimal, and this configuration might affect overall
throughput and latency. If, however, you configure the server to use two networks, o2ib0 and o2ib1,
then OST1 and OST2 can be reached over both networks. When the selection algorithm runs, it will determine
that the best path is, for example, OST2 over o2ib1.
To configure the client-side LNET, run the following command:
$ sudo lnetctl net show
Review the output, for example:
net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: o2ib local NI(s): - nid: 192.168.1.71@o2ib status: up interfaces: 0: ib0 - nid: 192.168.1.72@o2ib status: up interfaces: 0: ib2 - nid: 192.168.1.73@o2ib status: up interfaces: 0: ib4 - nid: 192.168.1.74@o2ib status: up interfaces: 0: ib6 - net type: o2ib1 local NI(s): - nid: 192.168.2.71@o2ib1 status: up interfaces: 0: ib1 - nid: 192.168.2.72@o2ib1 status: up interfaces: 0: ib3 - nid: 192.168.2.73@o2ib1 status: up interfaces: 0: ib5 - nid: 192.168.2.74@o2ib1 status: up interfaces: 0: ib7
For an optimal configuration, the LNet peer should show two LNet subnets.
In this case, the primary nid is only one o2ib:
$ sudo lnetctl peer show
Sample output:
peer:
- primary nid: 192.168.1.62@o2ib
Multi-Rail: True
peer ni:
- nid: 192.168.1.62@o2ib
state: NA
- nid: 192.168.2.62@o2ib1
state: NA
- primary nid: 192.168.1.61@o2ib
Multi-Rail: True
peer ni:
- nid: 192.168.1.61@o2ib
state: NA
- nid: 192.168.2.61@o2ib1
state: NA
From the server side, here is an example of sub-optimal LNet configuration:
[root@ai200-090a-vm01 ~]# lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: o2ib (o2ib1 is not present)
local NI(s):
- nid: 192.168.1.62@o2ib
status: up
interfaces:
0: ib0
- nid: 192.168.2.62@o2ib
status: up
interfaces:
0: ib1
Here is an example of an IB configuration for a non-optimal case, where a file is stripped over two OSTs, and there are sequential reads:
$ ibdev2netdev -v
0000:b8:00.1 mlx5_13 (MT4123 - MCX653106A-ECAT) ConnectX-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and 100GbE), dual-port QSFP56 fw 20.26.4012 port 1 (ACTIVE) ==> ib4 (Up) (o2ib)
ib4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 192.168.1.73 netmask 255.255.255.0 broadcast 192.168.1.255
0000:bd:00.1 mlx5_15 (MT4123 - MCX653106A-ECAT) ConnectX-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and 100GbE), dual-port QSFP56 fw 20.26.4012 port 1 (ACTIVE) ==> ib5 (Up) (o2ib1)
ib5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 192.168.2.73 netmask 255.255.255.0 broadcast 192.168.2.255
$ cat /proc/driver/nvidia-fs/peer_distance | grep 0000:be:00.0 | grep network
0000:be:00.0 0000:58:00.1 138 0 network
0000:be:00.0 0000:58:00.0 138 0 network
0000:be:00.0 0000:86:00.1 134 0 network
0000:be:00.0 0000:35:00.0 138 0 network
0000:be:00.0 0000:5d:00.0 138 0 network
0000:be:00.0 0000:bd:00.0 3 0 network
0000:be:00.0 0000:b8:00.1 7 30210269 network (ib4) (chosen peer)
0000:be:00.0 0000:06:00.0 134 0 network
0000:be:00.0 0000:0c:00.1 134 0 network
0000:be:00.0 0000:e6:00.0 138 0 network
0000:be:00.0 0000:3a:00.1 138 0 network
0000:be:00.0 0000:e1:00.0 138 0 network
0000:be:00.0 0000:bd:00.1 3 4082933 network (ib5) (best peer)
0000:be:00.0 0000:e6:00.1 138 0 network
0000:be:00.0 0000:86:00.0 134 0 network
0000:be:00.0 0000:35:00.1 138 0 network
0000:be:00.0 0000:e1:00.1 138 0 network
0000:be:00.0 0000:0c:00.0 134 0 network
0000:be:00.0 0000:b8:00.0 7 0 network
0000:be:00.0 0000:5d:00.1 138 0 network
0000:be:00.0 0000:3a:00.0 138 0 network
Here is an example of an optimal LNet configuration:
[root@ai200-090a-vm00 ~]# lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: o2ib
local NI(s):
- nid: 192.168.1.61@o2ib
status: up
interfaces:
0: ib0
- net type: o2ib1
local NI(s):
- nid: 192.168.2.61@o2ib1
status: up
interfaces:
0: ib1