10. WekaFS Troubleshooting#
This section provides troubleshooting and FAQ information about the WekaIO file system.
10.1. Downloading the WekaIO Client Package#
To download the WekaIO client package, run the following command:
$ curl http://<IP of one of the WekaIO hosts' IB interface>:14000/dist/v1/install | sh
For example, $ curl http://172.16.8.1:14000/dist/v1/install | sh.
10.2. Determining Whether the WekaIO Version is Ready for GDS#
To determine whether the WekaIO version is ready for GDS:
Run the following command:
$ weka version
Review the output, for example:
* 3.6.2.5-rdma-beta
Note
Currently, the only WekaIO FS version that supports GDS is
* 3.6.2.5-rdma-beta
10.3. Mounting a WekaIO File System Cluster#
The WekaIO file system can take a parameter to reserve a fixed number of cores for the user space process.
To mount a
server_ip 172.16.8.1with two dedicated cores, run the following command:$ mkdir -p /mnt/weka $ sudo mount -t wekafs -o num_cores=2 -o net=ib0,net=ib1,net=ib2,net=ib3,net=ib4,net=ib5,net=ib6,net=ib7 172.16.8.1/fs01 /mnt/weka
Review the output, for example:
Mounting 172.16.8.1/fs01 on /mnt/weka Creating weka container Starting container Waiting for container to join cluster Container "client" is ready (pid = 47740) Calling the mount command Mount completed successfully
10.4. Resolving a Failing Mount#
Before you use the IB interfaces in the mount options, verify that the interfaces are set up for
net=<interface>:$ sudo mount -t wekafs -o num_cores=2 -o net=ib0,net=ib1,net=ib2,net=ib3,net=ib4,net=ib5,net=ib6,net=ib7 172.16.8.1/fs01 /mnt/weka
Review the output, for example:
Mounting 172.16.8.1/fs01 on /mnt/weka Creating weka container Starting container Waiting for container to join cluster error: Container "client" has run into an error: Resources assignment failed: IB/MLNX network devices should have pre-configured IPs and ib4 has none
Remove interfaces that do not have network connectivity from the mount options.
$ ibdev2netdev mlx5_0 port 1 ==> ib0 (Up) mlx5_1 port 1 ==> ib1 (Up) mlx5_2 port 1 ==> ib2 (Up) mlx5_3 port 1 ==> ib3 (Up) mlx5_4 port 1 ==> ib4 (Down) mlx5_5 port 1 ==> ib5 (Down) mlx5_6 port 1 ==> ib6 (Up) mlx5_7 port 1 ==> ib7 (Up) mlx5_8 port 1 ==> ib8 (Up) mlx5_9 port 1 ==> ib9 (Up)
10.5. Resolving 100% Usage for WekaIO for Two Cores#
If you have two cores, and you are experiencing 100% CPU usage:
Run the following command.
$ top
Review the output, for example:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 54816 root 20 0 11.639g 1.452g 392440 R 94.4 0.1 781:06.06 wekanode 54825 root 20 0 11.639g 1.452g 392440 R 94.4 0.1 782:00.32 wekanode
When the
num_cores=2parameter is specified, two cores are used for the user mode poll driver for WekaIO FE networking. This process improves the latency and performance. Refer to the WekaIO documentation for more information.
10.6. Checking for an Existing Mount in the Weka File System#
To check for an existing mount in the WekaIO file system:
Run the following command:
$ mount | grep wekafs
Review the output, for example:
172.16.8.1/fs01 on /mnt/weka type wekafs ( rw,relatime,writecache,inode_bits=auto,dentry_max_age_positive=1000, dentry_max_age_negative=0)
10.7. Checking for a Summary of the WekaIO File System Status#
To check for a summary of the WekaIO file system status.
Run the following command:
$ weka status
Review the output, for example:
WekaIO v3.6.2.5-rdma-beta (CLI build 3.6.2.5-rdma-beta) cluster: Nvidia (e4a4e227-41d0-47e5-aa70-b50688b31f40) status: OK (12 backends UP, 72 drives UP) protection: 8+2 hot spare: 2 failure domains (62.84 TiB) drive storage: 62.84 TiB total, 819.19 MiB unprovisioned cloud: connected license: Unlicensed io status: STARTED 1 day ago (1584 buckets UP, 228 io-nodes UP) link layer: InfiniBand clients: 1 connected reads: 61.54 GiB/s (63019 IO/s) writes: 0 B/s (0 IO/s) operations: 63019 ops/s alerts: 3 active alerts, use `Wekaalerts` to list them
10.8. Displaying the Summary of the WekaIO File System Statistics#
To display a summary of the status of the WekaIO file system:
Run the following command.
$ cat /proc/wekafs/stat
Review the output, for example:
IO type: UM Average UM Longest KM Average KM Longest IO count -------------------------------------------------------------------------------------------------------------------------------- total: 812 us 563448 us 9398 ns 10125660 ns 718319292 (63260 IOPS, 0 MB/sec) lookup: 117 us 3105 us 6485 ns 436709 ns 4079 (12041) readdir: 0 us 0 us 0 ns 0 ns 0 mknod: 231 us 453 us 3970 ns 6337 ns 96 open: 0 us 0 us 0 ns 0 ns 0 (3232) release: 0 us 0 us 0 ns 0 ns 0 (2720) read: 0 us 0 us 0 ns 0 ns 0 write: 18957 us 563448 us 495291 ns 920127 ns 983137 (983041) getattr: 10 us 10 us 6771 ns 6771 ns 1 (9271) setattr: 245 us 424 us 4991 ns 48222 ns 96 rmdir: 0 us 0 us 0 ns 0 ns 0 unlink: 0 us 0 us 0 ns 0 ns 0 rename: 0 us 0 us 0 ns 0 ns 0 symlink: 0 us 0 us 0 ns 0 ns 0 readlink: 0 us 0 us 0 ns 0 ns 0 hardlink: 0 us 0 us 0 ns 0 ns 0 statfs: 4664 us 5072 us 38947 ns 59618 ns 7 SG_release: 0 us 0 us 0 ns 0 ns 0 SG_allocate: 1042 us 7118 us 2161 ns 110282 ns 983072 falloc: 349 us 472 us 4184 ns 10239 ns 96 atomic_open: 0 us 0 us 0 ns 0 ns 0 flock: 0 us 0 us 0 ns 0 ns 0 backcomm: 0 us 0 us 0 ns 0 ns 0 getroot: 19701 us 19701 us 57853 ns 57853 ns 1 trace: 0 us 0 us 0 ns 0 ns 0 jumbo alloc: 0 us 0 us 0 ns 0 ns 0 jumbo release: 0 us 0 us 0 ns 0 ns 0 jumbo write: 0 us 0 us 0 ns 0 ns 0 jumbo read: 0 us 0 us 0 ns 0 ns 0 keepalive: 46 us 1639968 us 1462 ns 38996 ns 184255 ioctl: 787 us 50631 us 8732 ns 10125660 ns 717328710 setxattr: 0 us 0 us 0 ns 0 ns 0 getxattr: 0 us 0 us 0 ns 0 ns 0 listxattr: 0 us 0 us 0 ns 0 ns 0 removexattr: 0 us 0 us 0 ns 0 ns 0 setfileaccess: 130 us 3437 us 6440 ns 71036 ns 3072 unmount: 0 us 0 us 0 ns 0 ns 0
10.9. Why WekaIO Writes Go Through POSIX#
For the WekaIO file system, GDS supports RDMA based reads and writes. You can use the fs:weka:rdma_write_support JSON
property to enable writes on supported Weka file systems. This option is disabled by default. If this option is set to false,
writes will be internally staged through system memory, and the cuFile library will use pwrite POSIX calls internally for writes.
10.10. Checking for nvidia-fs.ko Support for Memory Peer Direct#
To check for nvidia-fs.ko support for memory peer direct:
Run the following command:
$ lsmod | grep nvidia_fs | grep ib_core && echo "Ready for Memory Peer Direct"
Review the output, for example:
ib_core 319488 16 rdma_cm,ib_ipoib,mlx4_ib,ib_srp,iw_cm,nvidia_fs,ib_iser,ib_umad, rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,ib_ucm "Ready for Memory Peer Direct"
10.11. Checking Memory Peer Direct Stats#
To to check memory peer statistics:
Run the following script, which shows the counter for memroy peer direct statistics:
list=`ls /sys/kernel/mm/memory_peers/nvidia-fs/`. for stat in $list . do echo "$stat value: " $(cat /sys/kernel/mm/memory_peers/nvidia-fs/$stat). done
Review the output.
num_alloc_mrs value: 1288 num_dealloc_mrs value: 1288 num_dereg_bytes value: 1350565888 num_dereg_pages value: 329728 num_free_callbacks value: 0 num_reg_bytes value: 1350565888 num_reg_pages value: 329728 version value: 1.0
10.12. Checking for Relevant nvidia-fs Statistics for the WekaIO File System#
To check for relevant nvida-fs statistics for the WekaIO file system:
Run the following command:
$ cat /proc/driver/nvidia-fs/stats | egrep -v 'Reads|Writes|Ops|Error'
Review the output, for example:
GDS Version: 1.0.0.80 NVFS statistics(ver: 4.0) NVFS Driver(version: 2.7.49) Active Shadow-Buffer (MB): 256 Active Process: 1 Mmap : n=2088 ok=2088 err=0 munmap=1832 Bar1-map : n=2088 ok=2088 err=0 free=1826 callbacks=6 active=256 GPU 0000:34:00.0 uuid:12a86a5e-3002-108f-ee49-4b51266cdc07 : Registered_MB=32 Cache_MB=0 max_pinned_MB=1977 GPU 0000:e5:00.0 uuid:4c2c6b1c-27ac-8bed-8e88-9e59a5e348b5 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:b7:00.0 uuid:b224ba5e-96d2-f793-3dfd-9caf6d4c31d8 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:39:00.0 uuid:e8fac7f5-d85d-7353-8d76-330628508052 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:5c:00.0 uuid:2b13ed25-f0ab-aedb-1f5c-326745b85176 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:e0:00.0 uuid:df46743a-9b22-30ce-6ea0-62562efaf0a2 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:bc:00.0 uuid:c4136168-2a1d-1f3f-534c-7dd725fedbff : Registered_MB=32 Cache_MB=0 max_pinned_MB=32 GPU 0000:57:00.0 uuid:54e472f2-e4ee-18dc-f2a1-3595fa8f3d33 : Registered_MB=32 Cache_MB=0 max_pinned_MB=32
Note
Reads, Writes, Ops, and Error counters are not available through this interface for the WekaIO file system, so the value will be zero. See Displaying the Summary of the WekaIO File System Statistics about using the Weka status for reads and writes.
10.13. Conducting a Basic WekaIO File System Test#
To conduct a basic WekaIO file system test:
Run the following command:
$ /usr/local/cuda-x.y/tools/gdsio_verify -f /mnt/weka/gdstest/tests/reg1G -n 1 -m 0 -s 1024 -o 0 -d 0 -t 0 -S -g 4K
Review the output, for example:
gpu index :0,file :/mnt/weka/gdstest/tests/reg1G, RING buffer size :0, gpu buffer alignment :4096, gpu buffer offset :0, file offset :0, io_requested :1024, bufregister :false, sync :0, nr ios :1,fsync :0, address = 0x564ffc5e76c0 Data Verification Success
10.14. Unmounting a WekaIO File System Cluster#
To unmount a WekaIO file system cluster:
Run the following command.
$ sudo umount /mnt/weka
Review the output, for example:
Unmounting /mnt/weka Calling the umount command umount successful, stopping and deleting client container Umount completed successfully
10.15. Verify the Installed Libraries for the WekaIO File System#
The following table summarizes the tasks and command oputput for verifying the installled libraries for the WekaIO file systems.
Task |
Output |
|---|---|
Check the WekaIO version. |
$ weka status
WekaIO v3.6.2.5-rdma-beta (CLI build 3.6.2.5-rdma-beta)
|
Check whether GDS support for WekaFS is present. |
$ gdscheck -p
[...]
WekaFS : dmabuf, compat
Userspace RDMA: Supported
[...]
|
Check for MLNX_OFED information. |
Check with Currently supported with: MLNX_OFED_LINUX-5.1-0.6.6.0 $ ofed_info -s MLNX_OFED_LINUX-5.1-0.6.6.0:
|
Check for the |
$ lsmod | grep nvidia_fs | grep ib_core && echo "Ready for Memory Peer Direct"
|
Check for |
$ dpkg -s libibverbs-dev
Package: libibverbs-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 1151
Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org>
Architecture: amd64
Multi-Arch: same
Source: rdma-core
Version: 47mlnx1-1.47329
|
10.16. GDS Configuration File Changes to Support the WekaIO File System#
By default, the configuration for Weka RDMA-based writes is disabled.
"fs": {
"weka": {
// enable/disable WekaFs rdma write
"rdma_write_support" : false
}
}
To support the WekaIO file system, change the configuration to add a new property, rdma_dev_addr_list:
"properties": {
// allow compat mode,
// this will enable use of cufile posix read/writes
//"allow_compat_mode": true,
"rdma_dev_addr_list": [
"172.16.8.88" , "172.16.8.89",
"172.16.8.90" , "172.16.8.91",
"172.16.8.92" , "172.16.8.93",
"172.16.8.94", "172.16.8.95"
]
}
Note
For CUDA 11.5.1 and later, if you plan to use Weka FS then you must run:
modprobe nvidia_peermem
This will load the module that supports PeerDirect capabilities. It is necessary to run this command after reboot of the system.
In order to load the module automatically after every reboot, run the following command:
echo "nvidia-peermem" | sudo tee /etc/modules-load.d/nvidia-peermem.conf
10.17. Check for Relevant User-Space Statistics for the WekaIO File System#
To check for relevant user-space statistics for the WekaIO file system, issue the following command:
$ ./gds_stats -p <pid> -l 3 | grep GPU
Refer to User-Space RDMA Counters in GPUDirect Storage for more information about statistics.
10.18. Check for WekaFS Support#
If WekaFS support does not exist, the following issues are possible:
Issue |
Action |
|---|---|
MLNX_OFED peer direct is not enabled. |
Check whether MLNX_OFED is installed ( This issue can occur if the nvidia-fs Debian package was installed before MLNX_OFED was installed. When this issue occurs, uninstall and reinstall the nvidia-fs package. |
RDMA devices are not populated in the |
Add IP addresses to |
None of the configured RDMA devices are UP. |
Check IB connectivity for the interfaces. |