15. NVMe and NVMe-oF Troubleshooting#

This section provides troubleshooting information for NVME and NVMeOF support.

15.1. Storage Transport Requirements#

To enable GDS support for NVMe, NVMe over Fabrics, or NFS over RDMA, install the required storage transport packages by using one of the following options:

Do not install both MLNX_OFED and DOCA storage transport packages on the same system unless instructed by NVIDIA support.

15.2. MLNX_OFED Requirements and Installation#

Use this section only if MLNX_OFED is already installed in your environment or if your deployment specifically requires MLNX_OFED.

To enable GDS support for NVMe and NVMe over Fabrics, install MLNX_OFED 5.3 or later with GDS support.

Run the following command to install MLNX_OFED with the required options:

$ sudo ./mlnxofedinstall --with-nvmf --with-nfsrdma --enable-gds --add-kernel-support --dkms

Note

With MLNX_OFED 5.3 and later, the --enable-gds flag is no longer required.

After the installation is complete, update initramfs and reboot the system for the changes to take effect.

$ sudo update-initramfs -u -k `uname -r`
$ sudo reboot

The Linux kernel versions tested with MLNX_OFED 5.3-1.0.5.01 are 4.15.0-x and 5.4.0-x.

15.3. DOCA Requirements and Installation#

For new GDS installations, NVIDIA recommends using DOCA to install the required storage transport packages for NVMe and NVMe over Fabrics.

This section is not required if you already installed GDS support by using MLNX_OFED as described in MLNX_OFED Requirements and Installation.

For general DOCA host installation instructions, see Installing Software on Host.

If your system does not use a supported host OS or kernel, install the required DOCA extra packages as described in the DOCA installation guide. Then install the NVMe and NVMe over Fabrics packages for your operating system.

For Ubuntu:

$ sudo apt install doca-ofed mlnx-fw-updater mlnx-nvme-dkms
$ sudo update-initramfs -u -k `uname -r`
$ sudo reboot

For RHEL:

$ sudo dnf install doca-ofed mlnx-fw-updater kmod-mlnx-nvme
$ sudo dracut -f
$ sudo reboot

Note

For supported host operating systems and installation profiles, see Supported Host OS and Features per DOCA-Host Installation Profile.

15.4. Determining Whether the NVMe device is Supported for GDS#

NVMe devices must be compatible with GDS; the device cannot have the block device integrity capability.

For device integrity, the Linux block layer completes the metadata processing based on the payload in the host memory. This is a deviation from the standard GDS IO path and, as a result, cannot accommodate these devices. The cuFile file registration will fail when this type of underlying device is detected with appropriate error log in the cufile.log file.

$ cat /sys/block/<nvme>/integrity/device_is_integrity_capable

15.5. RAID Support in GDS#

Currently, GDS only supports RAID 0.

15.6. Mounting a Local File System for GDS#

Currently, EXT4 and XFS are the only block device based file system that GDS supports. Because of Direct IO semantics, the ext4 file system must be mounted with the journaling mode set to data=ordered. This has to be explicitly part of the mount options so that the library can recognize it:

$ sudo mount -o data=ordered /dev/nvme0n1 /mnt

If the EXT4 journaling mode is not in the expected mode, the cuFileHandleRegister will fail, and an appropriate error message will be logged in the log file. For instance, in the following case, /mnt1 is mounted with writeback, and GDS returns an error:

$ mount | grep /mnt1
/dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)

$ ./cufile_sample_001 /mnt1/foo 0
opening file /mnt1/foo
file register error:GPUDirect Storage not supported on current file

15.7. Check for an Existing EXT4 Mount#

To check for an existing EXT4 mount:

$ mount | grep ext4
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/nvme1n1 on /mnt type ext4 (rw,relatime,data=ordered)
/dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)

Note

A similar check can be used to check for an existing XFS mount, for example:

mount | grep xfs

15.8. Check for IO Statistics with Block Device Mount#

The following command and partial log show you how to obtain the IO statistics:

$ sudo iotop
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     193.98 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  881  be/3  root      0.00 B/s   15.52 K/s  0.00 %  0.01 % [jbd2/sda2-8]
    1  be/4  root      0.00 B/s    0.00 B/s  0.00 %  0.00 % init splash

15.9. RAID Group Configuration for GPU Affinity#

Creating one RAID group from the available NVMe devices might not be optimal for GDS performance. You might need to create RAID groups that consist of devices that have a pci-affinity with the specified GPU. This is required to prevent and cross-node P2P traffic between the GPU and the NVMe devices.

If affinity is not enforced, GDS will use an internal mechanism of device bounce buffers to copy data from the NVMe devices to an intermediate device that is closest to the drives and copy the data back to the actual GPU. If NVLink is enabled, this will speed up these transfers.

15.10. Conduct a Basic EXT4 File System Test#

To conduct a basic EXT4 file system test, issue the following command:

$ /usr/local/cuda-x.y/gds/tools/gdsio_verify  -f /mnt/nvme/gdstest/tests/reg1G -n 1 -m 0 -s 1024 -o 0  -d 0 -t 0 -S -g 4K

Sample output:

gpu index :0,file :/mnt/weka/gdstest/tests/reg1G, RING buffer size :0, gpu buffer alignment :4096, gpu buffer offset :0, file offset :0, io_requested :1024, bufregister :false, sync :0, nr ios :1,fsync :0,
address = 0x564ffc5e76c0
Data Verification Success

15.11. Unmount a EXT4 File System#

To unmount an EXT4 file system, issue the following command:

$ sudo umount /mnt/

15.12. Udev Device Naming for a Block Device#

The library has a limitation when identifying the NVMe-based block devices in that it expects device names to have the nvme prefix as part of the naming convention.

15.13. BATCH I/O Performance#

It has been observed that m separate batches with n/m entries each, showed better performance than 1 batch with n entries especially in case of NVMe based storage.

On this page