1. Software Settings#

This section describes the settings required for GDS.

For the best performance, multiple software settings are required across the entire system, and some settings are specific to the filesystem that you are using.

For more information, refer to the GPUDirect Storage Installation and Troubleshooting Guide.

1.1. System Settings#

For GDS p2p support on the Grace CPU based DGX™ (Grace Hopper) platform, IOMMU should be enabled and passthrough settings should be disabled.

The following are system settings that we recommend for the best performance on a bare metal x86_64 based platform.

  • PCIe Access Control Services (ACS).

    ACS forces P2P PCIe transactions to go up through the PCIe Root Complex, which does not enable GDS to bypass the CPU on paths between a network adapter or NVMe and the GPU in systems that include a PCIe switch.

    For optimal GDS performance, disable ACS.

    Note

    To list all of the PCI switches that have ACS enabled, issue /usr/local/cuda/gds/tools/gdscheck -p.

  • IOMMU

    When the IOMMU setting is enabled, PCIe traffic will be routed through the CPU root ports. This routing limits the maximum achievable throughput for configurations where the GPU and NIC are under the same PCIe switch. Before you install GDS, you must disable IOMMU. Refer to Installing GPUDirect Storage for more information.

    Note

    To determine whether the IOMMU setting is enabled, check the output from cat /proc/cmdline or use the gdscheck command..

    As an example, the following output shows IOMMU is enabled on this system:

    $ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=UUID=fb2a25a8-9d2e-4e1c-9d8a-efabdf165adc ro rootflags=data=ordered amd_iommu=on

    Similarly, using gdscheck you should see the following output if the IOMMU is disabled on the system:

    $ /usr/local/cuda/gds/tools/gdscheck -p
 IOMMU: disabled
 Platform verification succeeded

  • NIC affinity

    For the P2P DMA to function efficiently, NICs, NVMes and GPUs should be under a PCIe switch when possible. For the P2P DMA to function efficiently on NVIDIA DGX™ based platforms, ensure at least one NIC is in the same CPU socket as the GPU.

  • Avoid configurations where the NICs are assigned across the CPU sockets that require PCIe traffic to cross the CPU root ports or go across CPU sockets that use QPI.

  • NIC versions

    • When using Mellanox ConnectX-5 or later, the HCAs must be configured in InfiniBand or RoCE v2 mode.

    • For GDS support, MLNX_OFED 5.4 or later, or DOCA 2.9.0 or later is required.

1.2. Use of CUDA Context in GPU Kernels and Storage IO#

There are scenarios where the GDS workload data can be posted through intermediate buffers called bounce buffers. Hence a D2D copy is involved to/from these GPU bounce buffers to/from the application’s GPU buffers. The cuFile library posts these IOs on a stream created on the primary CUDA context. If a heavy compute job or application kernel is running in the background in the form of GPU kernels on a separate context (not the primary context), it can interfere with the D2D copies and increase the D2D copy launch times. This problem does not happen if the compute kernels are running in the primary context, so it is recommended that the application launch GPU kernels on the primary context instead of using a separate context.

Note

If the application uses CUDA runtime API, the kernel launches will happen in the primary context by default.

1.3. cuFile Configuration Settings#

The cuFile configuration settings in GDS are stored in the /etc/cufile.json file.

Edit the file for best performance for your application as shown below. For more information, refer to the GPUDirect Storage Parameters section.

To display the configuration setting, run the following command:

$ cat /etc/cufile.json

A portion of the sample output:

"properties": {
            // max IO size issued by cuFile to nvidia-fs driver (in KB)
            "max_direct_io_size_kb" : 16384,
            ...
    }

For the requested IO size, GDS issues IO requests sequentially in chunks of reads/writes based on the max_direct_io_size parameter. Larger values of max_direct_io_size will result in a reduced number of calls to the IO stack and might result in higher throughput.

The max_direct_io_size_kb parameter can be set to a value that is a multiple of 64K. This process defines the additional system memory that is used for each buffer during cuFileBufRegister up to a maximum value for the properties:max_direct_io_size_kb parameter of 16MB. This value can be reduced to 1MB to lower the amount of system memory that is used per buffer.

The total system memory that is used can be obtained from nvidia-fs stats.

In this example, each of 256 threads register a 1MB buffer for GDS.

  1. Run the following command:

    $ cat /proc/driver/nvidia-fs/stats

  2. Review the output:

    NVFS statistics(ver:1.0)
Active Shadow-Buffer (MB): 256...

    There are many tunables available in cufile.json. Refer to GPUDirect Storage Parameters.

On this page