Setting the Driver Registries for Enabling PCI P2PDMA

Disable multipathing support

NVMe Multipathing is currently not enabled with PCI P2PDMA in the upstream driver and will not work without a specialized patch.

$ cat /etc/modprobe.d/nvme.conf
options nvme_core multipath=N

#RH/SLES:

     $ sudo dracut -f

#ubuntu:

     $ sudo update-initramfs -u -k `uname -r`

$ sudo reboot

$ cat /sys/module/nvme_core/parameters/multipath
N

pci_p2pdma support (Enable static BAR1 and force disable write combine)

Set these params in driver modprobe conf settings to persist beyond reboots:

$ cat /etc/modprobe.d/nvidia-temp.conf

options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1;"

Note

For pre-Hopper GPUs (L4, L40, A100, A40), the following additional setting of ForceP2P=0 should be applied:

$ cat /etc/modprobe.d/nvidia-p2pdma.conf

options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;ForceP2P=0;RmForceDisableIomapWC=1;"

#RH/SLES:

   $ sudo dracut -f

#ubuntu :

   $ sudo update-initramfs -u -k `uname -r`

# reboot

   $ sudo reboot

# check the settings

   $ cat /proc/driver/nvidia/params  | grep -i static
   RegistryDwords: "RMForceStaticBar1=1;RmForceDisableIomapWC=1;"