13. Setting Up and Troubleshooting VAST Data (NFSoRDMA+MultiPath)#
This section provides information about how to set up and troubleshoot VAST data (NFSoRDMA+MultiPath).
13.1. Installing MLNX_OFED and VAST NFSoRDMA+Multipath Packages#
13.1.1. Client Software Requirements#
The following table lists the minimum client software requirements for using MLNX_OFED and VAST NFSoRDMA+Multipath packages.
NFS Connection Type |
Linux Kernel |
MLNX_OFED |
|---|---|---|
NFSoRDMA + Multipath |
The following kernel versions are supported:
|
The following MLNX_OFED versions are supported:
|
For the most up to date supportability matrix and client configuration steps and package downloads, refer to: https://kb.vastdata.com/docs/gpu-direct-storage-installation-and-operations-guide-2025.
MLNX_OFED must be installed for the VAST NFSoRDMA+Multipath package to function optimally. It is also important to download the correct VAST software packages to match your kernel+MLNX_OFED version combination. Refer to NVMe and NVMe-oF Troubleshooting support for information about how to install MLNX_OFED with GDS support.
To verify the current version of MLNX_OFED, issue the following command:
$ ofed_info -s MLNX_OFED_LINUX-5.3-0.6.6.01:
To verify the currently installed Linux kernel version, issue the following command:
$ uname -r -v
After you verify that your system has the correct combination of kernel and MLNX_OFED, you can install the VAST Multipath package.
13.1.2. Install the VAST Multipath Package#
Although the VAST Multipath with NFSoRDMA package has been submitted upstream for inclusion in a future kernel release, it is currently only available as a download from: https://support.vastdata.com/hc/en-us/articles/360016813140-NFSoRDMA-with-Multipath.
Be sure to download the correct .deb file that is based on your kernel and MLNX_OFED. version.
Install the VAST NFSoRDMA+Multipath package:
$ sudo apt-get install mlnx-nfsrdma-*.deb
Generate a new initramfs image:
$ sudo update-initramfs -u -k `uname -r`
Verify that the package is installed, and the version is the number that you expected:
$ dpkg -l | grep mlnx-nfsrdma ii mlnx-nfsrdma-dkms 5.3-OFED.5.1.0.6.6.0 all DKMS support for NFS RDMA kernel module
Reboot the host and run the following commands to verify that the correct version is loaded:
Note
The versions shown by each command should match.
$ cat /sys/module/sunrpc/srcversion 4CC8389C7889F82F5A59269 $ modinfo sunrpc | grep srcversion srcversion: 4CC8389C7889F82F5A59269
13.2. Set Up the Networking#
This section provides information about how to set up client networking for VAST for GDS.
To ensure optimal GPU-to-storage performance while leveraging GDS, you need to configure VAST and client networking in a balanced manner.
13.2.1. VAST Network Configuration#
VAST is a multi-node architecture. Each node has multiple high-speed (IB-HDR100 or 100GbE) interfaces, which can host client-facing Virtual IPs. Refer to VAST-Managing Virtual IP (VIP) Pools for more information.
Here is the typical workflow:
Multiply the number of VAST-Nodes * 2 (one per Interface).
Create a VIP Pool with the resulting IP count.
Place the VAST-VIP Pool on the same IP-subnet as the client.
13.2.2. Client Network Configuration#
The following is information about client network configuration.
Typically, GPU optimized clients (such as the NVIDIA DGX-2 and DGX-A100) are configured with multiple high speed network interface cards (NICs). In the following example, the system contains 8 separate NICs that were selected for optimal balance for NIC –>GPU and NIC –>CPU bandwidth.
$ sudo ibdev2netdev
mlx5_0 port 1 ==> ibp12s0 (Up)
mlx5_1 port 1 ==> ibp18s0 (Up)
mlx5_10 port 1 ==> ibp225s0f0 (Down)
mlx5_11 port 1 ==> ibp225s0f1 (Down)
mlx5_2 port 1 ==> ibp75s0 (Up)
mlx5_3 port 1 ==> ibp84s0 (Up)
mlx5_4 port 1 ==> ibp97s0f0 (Down)
mlx5_5 port 1 ==> ibp97s0f1 (Down)
mlx5_6 port 1 ==> ibp141s0 (Up)
mlx5_7 port 1 ==> ibp148s0 (Up)
mlx5_8 port 1 ==> ibp186s0 (Up)
mlx5_9 port 1 ==> ibp202s0 (Up)
Not all interfaces are connected, and this is to ensure optimal bandwidth.
When using the aforementioned VAST NFSoRDAM+Multipath package, it is recommended to assign static IP’s to each interface on the same subnet, which should also match the subnet configured on the VAST VIP Pool. If using GDS with NVIDIA DGX-A100s, a simplistic netplan is all that is required, for example:
ibp12s0:
addresses: [172.16.0.17/24]
dhcp4: no
ibp141s0:
addresses: [172.16.0.18/24]
dhcp4: no
ibp148s0:
addresses: [172.16.0.19/24]
dhcp4: no
However, if you are using other systems, or non-GDS code, you need to apply the following code to ensure that the proper interfaces are used to traverse from Client–>VAST.
Note
See the routes section for each interface in the following sample.
$ cat /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
enp226s0:
dhcp4: yes
ibp12s0:
addresses: [172.16.0.25/24]
dhcp6: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.25
table: 101
routing-policy:
- from: 172.16.0.25
table: 101
ibp18s0:
addresses: [172.16.0.26/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.26
table: 102
routing-policy:
- from: 172.16.0.26
table: 102
ibp75s0:
addresses: [172.16.0.27/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.27
table: 103
routing-policy:
- from: 172.16.0.27
table: 103
ibp84s0:
addresses: [172.16.0.28/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.28
table: 104
routing-policy:
- from: 172.16.0.28
table: 104
ibp141s0:
addresses: [172.16.0.29/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.29
table: 105
routing-policy:
- from: 172.16.0.29
table: 105
ibp148s0:
addresses: [172.16.0.30/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.30
table: 106
routing-policy:
- from: 172.16.0.30
table: 106
ibp186s0:
addresses: [172.16.0.31/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.31
table: 107
routing-policy:
- from: 172.16.0.31
table: 107
ibp202s0:
addresses: [172.16.0.32/24]
dhcp4: no
routes:
- to: 172.16.0.0/24
via: 172.16.0.32
table: 108
routing-policy:
- from: 172.16.0.32
table: 108
After making changes to the netplan, before issuing the following command, ensure that you have a IPMI/console connection to the client:
$ sudo netplan apply
13.2.3. Verify Network Connectivity#
Once the proper netplan is applied, verify connectivity between all client interfaces and all VAST-VIPs with a ping loop:
# Replace with appropriate interface names
$ export IFACES="ibp12s0 ibp18s0 ibp75s0 ibp84s0 ibp141s0 ibp148s0 ibp186s0 ibp202s0"
# replace with appropriate VAST-VIPs
$ export VIPS=$(echo 172.16.0.{101..116})
$ echo "starting pingtest" > pingtest.log
$ for i in $IFACES;do for v in $VIPS; do echo $i >> pingtest.log; ping -c 1 $v -W 0.2 -I $i|grep loss >> pingtest.log;done;done;
# Verify no failures:
$ grep '100%' pingtest.log
You should also verify that one of the following conditions are met:
All client interfaces are directly cabled to the same IB switches as VAST.
There are sufficient InterSwitch Links (ISLs) between client-switches, and switches to which VAST is connected.
To verify the current IB switch topology, issue the following command:
$ sudo ibnetdiscover
<output trimmed>
[37] "H-b8599f0300c3f4cb"[1](b8599f0300c3f4cb) # "vastraplab-cn1 HCA-2" lid 55 2xHDR # <-- example of Vast-Node
[43] "S-b8599f0300e361f2"[43] # "MF0;RL-QM87-C20-U33:MQM8700/U1" lid 1 4xHDR # <-- example of ISL
[67] "H-1c34da030073c27e"[1](1c34da030073c27e) # "rl-dgxa-c21-u19 mlx5_9" lid 23 4xHDR # <-- example of client
13.3. Mount VAST NFS#
To fully utilize available VAST VIPs, you must mount the file system by issuing the following command:
$ sudo mount -o proto=rdma,port=20049,vers=3 \
-o noidlexprt,nconnect=40 \
-o localports=172.16.0.25-172.16.0.32 \
-o remoteports=172.16.0.101-172.16.0.140 \
172.16.0.101:/ /mnt/vast
The options are:
- proto
RDMA must be specified.
- port=20049
Must be specified, this is RDMA control port.
- noidlexprt
Do not disconnect idle connections. This is to detect and recover failing connections when there are no pending I/Os.
- nconnect
Number of concurrent connections. Should be divisible evenly by the number of remoteports specified below for best balance.
- localports
A list of IPv4 addresses for the local ports to bind.
- remoteports
A list of NFS server IPv4 ports to bind.
For both localports and remoteports you can specify an inclusive range with the - delimiter, for example, FIRST-LAST. Multiple ranges or individual IP addresses can be separated by ~ (a tilde)
13.4. Debugging and Monitoring VAST Data#
Typically, mountstats under /proc shows xprt statistics. However, instead of modifying it in a non-compatible
way with the nfsstat utility, the VAST Multipath package extends mountstats with extra state reporting, to be
exclusively accessed from /sys/kernel/debug.
The stats node was added for each RPC client, and the RPC client 0 shows the mount that is completed:
$ sudo cat /sys/kernel/debug/sunrpc/rpc_clnt/0/stats
The added information is multipath IP address information per xprt and xprt state in string format.
For example:
xprt: rdma 0 0 1 0 24 3 3 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
172.25.1.101 -> 172.25.1.1, state: CONNECTED BOUND
xprt: rdma 0 0 1 0 24 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
172.25.1.102 -> 172.25.1.2, state: CONNECTED BOUND
xprt: rdma 0 0 1 0 23 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
172.25.1.103 -> 172.25.1.3, state: CONNECTED BOUND
xprt: rdma 0 0 1 0 22 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
172.25.1.104 -> 172.25.1.4, state: CONNECTED BOUND
xprt: rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
172.25.1.101 -> 172.25.1.5, state: BOUND
xprt: rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
172.25.1.102 -> 172.25.1.6, state: BOUND
xprt: rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
172.25.1.103 -> 172.25.1.7, state: BOUND
xprt: rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
172.25.1.104 -> 172.25.1.8, state: BOUND