1. Installing GPUDirect Storage#

This section includes GDS installation, uninstallation, and configuration information.

Note

For NVAIE and vGPU environments, please follow steps from their respective documents.

1.1. Before You Install GDS#

To install GDS on a non-DGX platform, complete the following steps.

Check the current IOMMU status on x86 platforms.

Note

This step can be skipped on ARM platforms.

$ journalctl -k -b | grep -i iommu

On a system where IOMMU is enabled, you will see similar lines below:

iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: lazy mode
pci 0000:XX:YY.Z: AMD-Vi: IOMMU performance counters supported
pci 0000:XX:YY.Z: Adding to iommu group 0
pci 0000:XX:YY.Z: Adding to iommu group 1
pci 0000:XX:YY.Z: Adding to iommu group 2

If IOMMU is enabled, disable it as described in the next step. Otherwise, continue to the storage transport requirements.

Disable IOMMU on x86 platforms, if required.

Note

In our experience, iommu=off provides the best functionality and performance. On certain platforms, such as DGX A100 and DGX-2, iommu=pt is supported. iommu=on is not guaranteed to work functionally or performantly.

Open the GRUB configuration file.
```
$ sudo vi /etc/default/grub
```
Add one of the following options to the GRUB_CMDLINE_LINUX_DEFAULT option:
- For AMD CPUs, add amd_iommu=off.
- For Intel CPUs, add intel_iommu=off.
If other options are already present, separate the options with a space. For example:
```
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 amd_iommu=off"
```
Update GRUB and reboot the system.
```
$ sudo update-grub
$ sudo reboot
```
After the system reboots, verify that the change took effect.
```
$ journalctl -k -b | grep -i iommu
```
On a system where IOMMU is disabled, you will see similar lines below:
```
Command line: ... amd_iommu=off ...
Kernel command line: ... amd_iommu=off ...
iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: lazy mode
(...no groups)
```
Review the storage transport requirements.

Additional software is required only if you need to enable support for NVMe, NVMe over Fabrics, or NFS over RDMA.
- To install MLNX_OFED, see MLNX_OFED Requirements and Installation.
- To install DOCA, see DOCA Requirements and Installation.
This step is not required for DGX OS 6.x or later.
Disable PCIe ACS, if required.

For best GPUDirect Storage (GDS) performance, disable PCIe ACS at boot. Add the following parameters to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, preserving any existing parameters:
```
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 pci=disable_acs_redir=pci:0:0 dyndbg=\"file drivers/pci/pci.c +p\""
```
After editing the file, regenerate the GRUB configuration using the command appropriate for the operating system, then reboot.
```
$ sudo update-grub
$ sudo reboot
```
After the system reboots, verify that the change took effect.
```
$ journalctl -k -b | grep -i 'acs'
```
You should observe the following lines:
```
Command line: ... pci=disable_acs_redir=pci:0:0 ...
Kernel command line: ... pci=disable_acs_redir=pci:0:0 ...
pcieport 0000:XX:YY.Z: Configured ACS to 0x0000
```

1.2. Installing GDS#

GDS installation is supported in two ways:

using package managers such as Debian and RPMs

For installation on DGX platforms, refer to:

For installation on non-DGX platforms:

Starting in CUDA Toolkit 13.4, nvidia-fs must be installed separately by following the platform-specific instructions on the NVIDIA nvidia-fs downloads page.
The cuFile user-space libraries and GDS tools are still distributed as part of the CUDA Toolkit.

Additional information about CUDA Toolkit installation is in the CUDA Installation Guide for Linux.

Throughout this document, in cuda-<x>.<y>, x refers to the CUDA major version and y refers to the minor version.

1.2.1. Configuring File System Settings for GDS#

Before proceeding, please refer to the File System specific section in this document for necessary configurations needed to support GDS:

Lustre-Lnet: Configuring LNet Networks with Multiple OSTs for Optimal Peer Selection
WekaIO: GDS Configuration File Changes to Support the WekaIO File System
IBM Spectrum Scale: GDS Configuration to Support IBM Spectrum Scale
BeeGFS: BeeGFS Client Configuration for GDS
VAST: Set Up the Networking
NVMe: RAID Group Configuration for GPU Affinity
EXT4/XFS: Mounting a Local File System for GDS

1.2.2. Verifying a Successful GDS Installation#

To verify that GDS installation was successful, run gdscheck:

$ /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p

Note

The gdscheck command expects python3 to be present on the system. If it fails because of python3 not being available, then you can invoke the command with the explicit path to where python (i.e. python3) is installed. For example:

$ /usr/bin/python /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p

The output of this command shows whether a supported file system or device installed on the system supports GDS. The output also shows whether PCIe ACS is enabled on any of the PCI switches.

Sample output:

GDS release version: 1.16.0.49
nvidia_fs version:  2.27 nvidia_fs minimum version: 2.12
Platform: x86_64
============
ENVIRONMENT:
============
=====================
DRIVER CONFIGURATION:
=====================
NVMe               : nvfs, compat
NVMeOF             : compat
SCSI               : compat
ScaleFlux CSD      : compat
FSx Lustre         : Supported
NVMesh             : compat
DDN EXAScaler      : compat
IBM Spectrum Scale : dmabuf, compat
NFS                : compat
BeeGFS             : compat
ScaTeFS            : compat
VIRTIOFS           : compat
WekaFS             : dmabuf, compat
Userspace RDMA     : Supported
--Mellanox PeerDirect : Disabled
--DmaBuf support : Enabled
--rdma library        : Loaded (libcufile_rdma.so)
--rdma devices        : Configured
--rdma_device_status  : Up: 1 Down: 0
=====================
CUFILE CONFIGURATION:
=====================
properties.use_pci_p2pdma : false
properties.use_compat_mode : true
properties.force_compat_mode : false
properties.gds_rdma_write_support : true
properties.use_poll_mode : false
properties.poll_mode_max_size_kb : 4
properties.max_batch_io_size : 128
properties.max_batch_io_timeout_msecs : 5
properties.max_direct_io_size_kb : 16384
properties.max_device_cache_size_kb : 131072
properties.per_buffer_cache_size_kb : 1024
properties.max_device_pinned_mem_size_kb : 33554432
properties.posix_pool_slab_size_kb : 4 1024 16384
properties.posix_pool_slab_count : 128 64 64
properties.rdma_peer_affinity_policy : RoundRobin
properties.rdma_dynamic_routing : 0
fs.generic.posix_unaligned_writes : false
fs.lustre.posix_gds_min_kb: 0
fs.lustre.use_pci_p2pdma : false
fs.beegfs.posix_gds_min_kb: 0
fs.beegfs.use_pci_p2pdma : false
fs.scatefs.posix_gds_min_kb: 0
fs.scatefs.use_pci_p2pdma : false
fs.weka.rdma_write_support: false
fs.nfs.use_pci_p2pdma : false
fs.gpfs.gds_write_support: false
fs.gpfs.gds_async_support: true
fs.gpfs.use_pci_p2pdma : false
fs.virtiofs.use_pci_p2pdma : false
profile.cufile_stats : 0
miscellaneous.api_check_aggressive : false
execution.max_io_threads : 4
execution.max_io_queue_depth : 128
execution.parallel_io : true
execution.min_io_threshold_size_kb : 8192
execution.max_request_parallelism : 4
properties.force_odirect_mode : false
properties.prefer_iouring : false
block.raid.use_pci_p2pdma : false
block.nvme.use_pci_p2pdma : false
block.nvmeof.use_pci_p2pdma : false
=========
GPU INFO:
=========
GPU index 0 NVIDIA L4 bar:1 bar size (MiB):32768 supports GDS, IOMMU State: Disabled
==============
PLATFORM INFO:
==============
IOMMU: disabled
Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
Cuda Driver Version Installed:  13010
Platform: ProLiant DL360 Gen10, Arch: x86_64(Linux 5.14.0-570.25.1.el9_6.x86_64)
Platform verification succeeded

Note

There are READMEs provided in /usr/local/cuda-<x>.<y>/gds/tools and /usr/local/cuda-<x>.<y>/gds/samples to show usage.

1.3. Installed GDS Libraries and Tools#

GPUDirect Storage userspace libraries are located in the /usr/local/cuda-<X>.<Y>/targets/x86_64-linux/lib/ directory.

Note

GPUDirect Storage packages are installed at /usr/local/cuda-X.Y/gds, where X is the major version of the CUDA toolkit, and Y is the minor version.

$ ls -1 /usr/local/cuda-X.Y/targets/x86_64-linux/lib/*cufile*
cufile.h
libcufile.so
libcufile.so.0
libcufile.so.1.0.0
libcufile_rdma.so
libcufile_rdma.so.0
libcufile_rdma.so.1.0.0

GPUDirect Storage tools and samples are located in the /usr/local/cuda-X.Y/gds directory.

$ ls -lh /usr/local/cuda-X.Y/gds/
total 20K
-rw-r--r-- 1 root root 8.4K Mar 15 13:01 README
drwxr-xr-x 2 root root 4.0K Mar 19 12:29 samples
drwxr-xr-x 2 root root 4.0K mar 19 10:28 tools

For this release, GPUDirect Storage is providing an additional libcufile-dev package (cuFile library developers package) . This is primarily intended for the developer’s environment. Essentially the libcufile-dev package contains a static version of cuFile library (libcufile_static.a, libcufile_rdma_static.a) and cufile.h header file which may be required by the applications that use cuFile library APIs.

1.4. Uninstalling GPUDirect Storage#

To uninstall GDS from Ubuntu and DGX OS:

$ sudo apt-get remove --purge "*libcufile*" "*gds-tools*" "*nvidia-fs*"

To uninstall from RHEL:

$ sudo dnf remove "nvidia-gds*"

1.5. Environment Variables Used by GPUDirect Storage#

The following table lists environment variables used by cuFile. cuObject client environment variables are described after this table.

GDS uses the following environment variables.

Table 1.1 GDS Environment Variables#
CUFILE_ENV Variable	Description
`CUFILE_CQ_DEPTH`	Completion queue depth for the DC target.
`CUFILE_EXPERIMENTAL_FS=1`	Controls whether cufile checks for supporting file systems. When set to 1, allows testing with new file systems that are not yet officially enabled with cuFile.
`CUFILE_ENV_PATH_JSON=/home/user/cufile.json`	The path to an additional `cufile.json` configuration file; any parameters provided in this file take precedence over the system defaults in `/etc/cufile.json`. This can be used for container environments and applications that require non-default settings.
`CUFILE_ETH_SL`	Sets QOS level on RoCEv2 device QP for userspace RDMA targets (WekaFS and GPFS).
`CUFILE_IB_SL=\[0-15\]`	Sets QOS level on IB device QP for userspace RDMA targets (WekaFS and GPFS).
`CUFILE_LOGFILE_PATH=/etc/log/cufile_\$\$.log`	Controls the path for cuFile log information. Specifies the default log path, which is the current working directory of the application. Useful for containers or logging.
`CUFILE_LOGGING_LEVEL=TRACE`	Controls the tracing level and can override the trace level for a specific application without requiring a new configuration file.
`CUFILE_MIN_RNR_TIMER`	Minimum RNR value for QP after which the QP will error out with RNR timeout if no Work Request is posted on the remote end. Default value is 16 (2.56ms).
`CUFILE_NVTX`	DEPRECATED
`CUFILE_RDMA_DC_KEY="0XABABCDEF"`	Controls the DC_KEY for userspace RDMA DC targets for WekaFS and GPFS.
`CUFILE_RDMA_HOP_LIMIT`	Maximum number of hops before the packet is discarded on the network. Prevents indefinite looping of the packet. Default is 64.
`CUFILE_RDMA_PKEY_INDEX`	Partition key index.
`CUFILE_RDMA_SR_MAX_WR`	Maximum number of Work requests supported by the Shared Request Queue.
`CUFILE_RDMA_SR_MAX_SGE`	Maximum number of Scatter Gather Entries supported per Work Request.
`CUFILE_SKIP_TOPOLOGY_DETECTION`	Setting this environment variable to true will skip topology detection in compat mode. This will reduce the high startup latency seen in compat mode on systems with multiple PCI devices.
`CUFILE_FORCE_COMPAT_MODE`	Overrides cufile.json settings and forces I/O to go through compatible mode instead of GDS mode.
`CUFILE_ALLOW_COMPAT_MODE`	This does exactly what the `allow_compat_mode` tag in `cufile.json` file does.
`CUFILE_USE_PCIP2PDMA`	When set to `true`, the IO preference is set to the PCI P2PDMA path over the traditional nvidia-fs path if the kernel and filesystem support it. Additionally, the JSON configuration for the filesystem must be enabled; otherwise, the traditional path via nvidia-fs is used.
`CUFILE_LOGFILE_MAX_SIZE_MB`	Maximum `cufile.log` size in MB before truncation. The default is 32; -1 means no limit.
`CUFILE_VISIBLE_DEVICES`	Restricts which GPUs cuFile operates on, indexed relative to `CUDA_VISIBLE_DEVICES`.
`CUFILE_STATS_LEVEL`	Sets the cuFile stats/profiling level (0-3).

cuObject Client Environment Variables

Starting with cuObject client v1.4.0 in CUDA 13.5, the following environment variables configure the cuObject client. Set these variables before client initialization.

For the individual settings listed below, a CUOBJ_* variable takes precedence over its corresponding CUFILE_* variable. The legacy variable is used only when the CUOBJ_* variable is absent. Leave the CUOBJ_* variable unset to use this fallback. CUOBJ_ENV_PATH_JSON has separate file-selection behavior and no legacy fallback.

Table 1.2 cuObject Client Environment Variables#
Environment variable	Legacy fallback	Description
`CUOBJ_ENV_PATH_JSON`	None	Selects the cuObject JSON configuration file. If unset or empty, the client selects `/etc/cuobj.json`. A custom file replaces the default file selection; it is not an overlay.
`CUOBJ_DMABUF_ENABLE`	`CUFILE_DMABUF_ENABLE`	`TRUE` selects the dmabuf GPU memory registration backend. `FALSE` selects `nvidia_peermem`.
`CUOBJ_RDMA_DC_KEY`	`CUFILE_RDMA_DC_KEY`	Sets a nonzero 32-bit RDMA DC key as a hexadecimal value, for example, `0xffeeddcc`.
`CUOBJ_RDMA_ALLOW_RDMA_TOKEN_RESET`	`CUFILE_RDMA_ALLOW_RDMA_TOKEN_RESET`	Enables or disables RDMA token/queue-pair reset handling. Accepts `TRUE`/`FALSE` or `1`/`0`.
`CUOBJ_RDMA_TOKEN_RESET_TIMEOUT_SECS`	`CUFILE_RDMA_TOKEN_RESET_TIMEOUT_SECS`	Sets the timeout for draining outstanding I/O during RDMA queue-pair reset. Accepts an integer from 1 through 86400, in seconds.
`CUOBJ_IO_PRIORITY`	`CUFILE_IO_PRIORITY`	Sets the runtime I/O stream priority. Named values include `low`, `med`, `high`, and `default`.
`CUOBJ_STREAM_MEMOPS_BYPASS`	`CUFILE_STREAM_MEMOPS_BYPASS`	`TRUE` bypasses the CUDA stream memory-operations support check. `FALSE` uses the normal support check.
`CUOBJ_MAX_DEVICE_CACHE_SIZE`	`CUFILE_MAX_DEVICE_CACHE_SIZE`	Sets the maximum device cache size, in KB.
`CUOBJ_PER_BUFFER_CACHE_SIZE`	`CUFILE_PER_BUFFER_CACHE_SIZE`	Sets the per-buffer cache size, in KB. The value must be positive and a multiple of 64 KB.
`CUOBJ_STATS_LEVEL`	`CUFILE_STATS_LEVEL`	Sets the statistics collection level from 0 through 3; 0 disables collection. Overrides the JSON field `profile.cufile_stats`.
`CUOBJ_VISIBLE_DEVICES`	`CUFILE_VISIBLE_DEVICES`	Restricts the runtime to selected GPU IDs, supplied as a comma-separated list. When `CUDA_VISIBLE_DEVICES` contains numeric IDs, select IDs from that list.
`CUOBJ_LOGGING_LEVEL`	`CUFILE_LOGGING_LEVEL`	Sets logging verbosity, such as `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. Overrides the JSON logging level.
`CUOBJ_LOGFILE_MAX_SIZE_MB`	`CUFILE_LOGFILE_MAX_SIZE_MB`	Sets the maximum log-file size from 1 through 4096 MB. A value of `-1` disables the size limit.

Only the listed variables have these cuObject spellings. Other CUFILE_* variables must not be converted to CUOBJ_* names by changing the prefix.

1.6. JSON Config Parameters Used by GPUDirect Storage#

Refer to GPUDirect Storage Parameters for details about the JSON Config parameters used by GDS.

Consider compat_mode for systems or mounts that are not yet set up with GDS support. To learn more about Compatibility Mode, refer to cuFile Compatibility Mode.

cuObject Client Configuration File

Starting with cuObject client v1.4.0 in CUDA 13.5, the default cuObject configuration path is /etc/cuobj.json. To select a different file, set:

export CUOBJ_ENV_PATH_JSON=/path/to/cuobj.json

cuObject initializes built-in defaults, loads the selected JSON file, and then applies supported environment-variable overrides.

When CUOBJ_ENV_PATH_JSON is nonempty, only the specified JSON file is loaded. /etc/cuobj.json is not loaded first. If the variable is unset or empty, /etc/cuobj.json is selected.

cuObject does not select its configuration through CUFILE_ENV_PATH_JSON or /etc/cufile.json. Applications that previously used those locations for cuObject settings must move the settings to /etc/cuobj.json or a file selected by CUOBJ_ENV_PATH_JSON.

Use the cuObject client library and its matching libcufile from the same release. Combining the new cuObject client with an older libcufile is unsupported and can cause linking or loading failures.

1.7. GDS Configuration File Changes to Support Dynamic Routing#

For dynamic routing to support multiple file systems and mount points, configure the global per-file system rdma_dev_addr_list property for a single mount or the rdma_dev_addr_list property for a per file system mount table. This feature is to tune and restrict which NICs are available for cuFile to utilize.

"fs": {
      "lustre": {
      // if using a single lustre mount, provide the ip addresses
      // here (use : sudo lnetctl net show)
      //"rdma_dev_addr_list" : []

      // if using multiple lustre mounts, provide ip addresses
      // used by respective mount here
      //"mount_table" : {
      // "/lustre/ai200_01/client" : {
      //    "rdma_dev_addr_list" : ["172.172.1.40",
                          "172.172.1.42"]
      // },

      // "/lustre/ai200_02/client" : {
      //    "rdma_dev_addr_list" : ["172.172.2.40",
                          "172.172.2.42"]
      //}
     },

     "nfs": {
        //"rdma_dev_addr_list" : []

        //"mount_table" : {
        //  "/mnt/nfsrdma_01/" : {
        //           "rdma_dev_addr_list" : []
        //},

        //  "/mnt/nfsrdma_02/" : {
        //           "rdma_dev_addr_list" : []
        //}
        //}
        },
     },

1.8. Determining Which Version of GDS is Installed#

To determine which version of GDS you have, run the following command:

$ gdscheck.py -v

Example output:

GDS release version: 1.0.0.78
 nvidia_fs version:  2.7 libcufile version: 2.4