14. NVMe P2PDMA Troubleshooting#

This section provides troubleshooting information for NVMe Support using Linux PCI P2PDMA.

14.1. Linux Kernel Requirements#

Check that the Linux kernel version is newer than 6.2 and above on Ubuntu distributions.

For other distributions please check the PCI P2PDMA feature is compiled. If the PCI P2PDMA is not enabled, GDS-specific NVMe patches can be installed from MLNX_OFED to support GDS with nvidia-fs.ko.

$ cat /proc/kallsyms | grep -i p2pdma_pgmap_ops
0000000000000000 d p2pdma_pgmap_ops

Check if you have a system with NVIDIA GPU newer than or equal to NVIDIA Ampere-based architecture.

14.2. P2PDMA Supported GPUs#

A100, A40, L4, L40S, H100, H200, B100, B200, B300 are supported.

14.3. Setting the Driver Registries for Enabling PCI P2PDMA#

Note

Required only on x86 platforms.

Disable multipathing support

NVMe Multipathing is currently not enabled with PCI P2PDMA in the upstream driver and will not work without a specialized patch.

$ cat /etc/modprobe.d/nvme.conf
options nvme_core multipath=N

#RH/SLES:
$ sudo dracut -f

#Ubuntu :
$ sudo update-initramfs -u -k `uname -r`
$ sudo reboot
$ cat /sys/module/nvme_core/parameters/multipath
N

pci_p2pdma support ( Enable static BAR1 and force disable write combine)

Set the following parameters in driver modprobe conf settings to persist beyond reboots:

$ cat /etc/modprobe.d/nvidia-p2pdma.conf
options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1;"

Note

For pre-Hopper GPUs (L4, L40, A100, A40), the following additional setting of ForceP2P=0 should be applied:

$ cat /etc/modprobe.d/nvidia-p2pdma.conf
options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;\
ForceP2P=0;\
RmForceDisableIomapWC=1;"

#RH/SLES:
$ sudo dracut -f

#ubuntu :
$ sudo update-initramfs -u -k `uname -r`

# reboot
$ sudo reboot

# check the settings
$ cat /proc/driver/nvidia/params | grep -i static
RegistryDwords: "RMForceStaticBar1=1;RmForceDisableIomapWC=1;"

14.4. cufile.json Settings#

Add the following config parameter to the /etc/cufile.json or app-specific JSON file:

{
    "properties": {
        "use_pci_p2pdma": true
    }
     "block": {
         "nvme": {
               "use_pci_p2pdma": true
         }
   }
}

14.5. Verify P2P Mode is Supported by GDS#

/usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p

GDS release version: 1.13.0.7
nvidia_fs version:  2.24 libcufile version: 2.12
Platform: x86_64
============
ENVIRONMENT:
============
=====================
DRIVER CONFIGURATION:
=====================
NVMe               : p2pdma, compat

Note

NVMe P2PDMA mode takes precedence if NVMe is supported by both PCI P2PDMA and nvidia-fs.

14.6. RAID Support#

GDS P2PDMA for RAID devices is currently supported only on ARM platforms; x86 platforms are not supported.

14.7. Mounting a Local File System for GDS#

Currently, EXT4 and XFS are the only block device based file systems that GDS supports. Because of Direct IO semantics, the EXT4 file system must be mounted with the journaling mode set to data=ordered. This has to be explicitly part of the mount options so that the library can recognize it:

$ sudo mount -o data=ordered /dev/nvme0n1 /mnt

If the EXT4 journaling mode is not in the expected mode, the cuFileHandleRegister will fail, and an appropriate error message will be logged in the log file. For instance, in the following case, /mnt1 is mounted with writeback, and GDS returns an error:

$ mount | grep /mnt1
/dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)
$ ./cufile_sample_001 /mnt1/foo 0
opening file /mnt1/foo
file register error:GPUDirect Storage not supported on current file

14.8. Check for an Existing EXT4 Mount#

To check for an existing EXT4 mount:

$ mount | grep ext4
   /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
   /dev/nvme1n1 on /mnt type ext4 (rw,relatime,data=ordered)
   /dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)

Note

A similar check can be used to check for an existing XFS mount, for example:

mount | grep xfs

14.9. Check for IO Statistics with Block Device Mount#

The following command and partial log show you how to obtain the IO statistics:

$ sudo iotop
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 193.98 K/s
 TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
 881 be/3 root 0.00 B/s 15.52 K/s 0.00 % 0.01 % [jbd2/sda2-8]
 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init splash

14.10. Conduct a Basic EXT4 File System Test#

To conduct a basic EXT4 file system test, issue the following command:

$ /usr/local/cuda-x.y/gds/tools/gdsio_verify -f /mnt/nvme/gdstest/tests/reg1G -n 1 -m 0 -s 1024 -o 0 -d 0 -t 0 -S -g 4K

Sample output:

gpu index :0,file :/mnt/weka/gdstest/tests/reg1G, RING buffer size :0, gpu buffer alignment :4096, gpu buffer offset :0, file offset :0, io_requested :1024, bufregister :false, sync :0, nr ios :1,fsync :0,
   address = 0x564ffc5e76c0
   Data Verification Success

14.11. Unmount an EXT4 File System#

To unmount an EXT4 file system, issue the following command:

$ sudo umount /mnt/

14.12. Udev Device Naming for a Block Device#

The library has a limitation when identifying the NVMe-based block devices in that it expects device names to have the nvme prefix as part of the naming convention.

14.13. BATCH I/O Performance#

It has been observed that m separate batches with n/m entries each, showed better performance than 1 batch with n entries especially in case of NVMe based storage.

14.14. Statistics#

There are no separate statistics for differentiating between PCI P2PDMA mode and nvidia-fs mode. The GDS mode stats are common for both these modes and should be differentiated based on the mode the application is operating on.

On this page