Setting cufile.json Configuration#

Add the following config parameter into the /etc/cufile.json or App-specific JSON file:

cufile.json
{
   "properties": {
      "use_pci_p2pdma": true
   }
}

Before You Test GDS#

To install GDS on a non-DGX platform, complete the following steps:

Run the following command to check the current status of IOMMU.

$ dmesg | grep -i iommu

On x86_64 based platforms, if IOMMU is enabled, complete step 3 to disable it; otherwise continue to step 4.

Disable IOMMU
1. Run the following command:
```
$ sudo vi /etc/default/grub
```
2. Add one of the following options to the GRUB_CMDLINE_LINUX_DEFAULT option:
  - If you have an AMD CPU, add amd_iommu=off.
  - If you have an Intel CPU, add intel_iommu=off.
3. If there are already other options, enter a space to separate the options, for example, GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 amd_iommu=off"
4. In our experience, iommu=off works the best in terms of functionality and performance. On certain platforms such as DGX H100, DGX A100, iommu=pt is supported. iommu=on is not guaranteed to be either functional or performant.

 $ /usr/local/cuda/gds/tools/gdscheck -p

 GDS release version: 1.12.0.25
 libcufile version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe P2PDMA        : Supported
 NVMe               : Unsupported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Unsupported
 BeeGFS             : Unsupported
 YanRong YRCloudFile: Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : true

 . . .
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA L4 bar:1 bar size (MiB):32768 supports GDS, IOMMU State: Disabled
==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  12070
 Platform: ProLiant DL360 Gen10, Arch: x86_64(Linux 6.2.0-31-generic)
 Platform verification succeeded

Note

Make sure ACS is disabled and IOMMU is pt or off.

Create Local Filesystem on NVMe#

$ sudo mkfs.ext4 /dev/nvme1n1

Mount the NVME drive#

$ mkdir -p  /mnt/nvme
$ sudo mount -o data=ordered /dev/nvme1n1 /mnt/nvme1
$ sudo chmod -R 777 /mnt/nvme