11. Enabling IBM Spectrum Scale Support with GDS#

GDS is supported starting with IBM Spectrum Scale 5.1.2.

After reviewing the NVIDIA GDS documentation, refer to IBM Spectrum Scale 5.1.2. Please see especially the GDS sections in the Planning and Installation guides.

For troubleshooting see https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=troubleshooting-gpudirect-storage-issues.

11.1. IBM Spectrum Scale Limitations with GDS#

Refer to the following documentation for IBM Spectrum Scale Limitations with GDS:

https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=architecture-gpudirect-storage-support-spectrum-scale

11.2. Checking nvidia-fs.ko Support for Mellanox PeerDirect#

Use the following command to check support for memory peer direct:

$ cat /proc/driver/nvidia-fs/stats | grep -i "Mellanox PeerDirect Supported"

Mellanox PeerDirect Supported: True

In the above example, False means that MLNX_OFED was not installed with GPUDirect Storage support prior to installing nvidia-fs.

The other option to check for Mellanox PeerDirect Support is via gdscheck -p output. If it’s enabled, you should be able to see something as below.

--Mellanox PeerDirect : Enabled

11.3. Verifying Installed Libraries for IBM Spectrum Scale#

The following tasks, shown with sample output, can be peformed to show installed libraries for IBM Spectrum Scale:

  • Check whether GDS support for IBM Spectrum Scale is present:

    [~]# /usr/local/cuda/gds/tools/gdscheck -p | egrep -e "Spectrum Scale|PeerDirect|rdma_device_status"
 IBM Spectrum Scale : dmabuf, compat
  --Mellanox PeerDirect : Disabled
  --DmaBuf support : Enabled
  --rdma_device_status  : Up: 2 Down: 0

  • Check for MLNX_OFED information:

    $ ofed_info -s
MLNX_OFED_LINUX-5.4-1.0.3.0:

  • Check for nvidia-fs.ko driver:

    [~]# cat /proc/driver/nvidia-fs/stats
 GDS Version: 1.0.0.82
  NVFS statistics(ver: 4.0)
  NVFS Driver(version: 2.7.49)
  Mellanox PeerDirect Supported: True
  IO stats: Disabled, peer IO stats: Disabled
  Logging level: info

  Active Shadow-Buffer (MiB): 0
  Active Process: 0
  Reads                           : err=0 io_state_err=0
  Sparse Reads                    : n=230 io=0 holes=0 pages=0
  Writes                          : err=0 io_state_err=237 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
  Mmap                            : n=27 ok=27 err=0 munmap=27
  Bar1-map                        : n=27 ok=27 err=0 free=27 callbacks=0 active=0
  Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
  Ops                             : Read=0 Write=0
  GPU 0000:2f:00.0  uuid:621f7d17-5e7d-8f79-be27-d2f4256ddd88 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=2

  • To check for libibverbs.so on Ubuntu:

    $ dpkg -s libibverbs-dev
root@fscc-sr650-59:~# dpkg -s libibverbs-dev
Package: libibverbs-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 1428
Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org>
Architecture: amd64
Multi-Arch: same
Source: rdma-core
Version: 54mlnx1-1.54103

  • To check for libibverbs.so on RHEL:

    []# rpm -qi libibverbs
    Name        : libibverbs
    Version     : 54mlnx1
    Release     : 1.54103
    Architecture: x86_64
    Install Date: Tue 13 Jul 2021 10:21:18 AM CEST
    Group       : System Environment/Libraries
    Size        : 535489
    License     : GPLv2 or BSD
    Signature   : DSA/SHA1, Fri 02 Jul 2021 08:14:44 PM CEST, Key ID c5ed83e26224c050
    Source RPM  : rdma-core-54mlnx1-1.54103.src.rpm
    Build Date  : Fri 02 Jul 2021 06:59:01 PM CEST
    Build Host  : c-141-24-1-005.mtl.labs.mlnx
    Relocations : (not relocatable)
    URL         : https://github.com/linux-rdma/rdma-core
    Summary     : A library and drivers for direct userspace use of RDMA (InfiniBand/iWARP/RoCE) hardware
    Description :
    libibverbs is a library that allows userspace processes to use RDMA
    "verbs" as described in the InfiniBand Architecture Specification and
    the RDMA Protocol Verbs Specification.  This includes direct hardware
    access from userspace to InfiniBand/iWARP adapters (kernel bypass) for
    fast path operations.

    Device-specific plug-in ibverbs userspace drivers are included:

- libmlx5: Mellanox ConnectX-4+ InfiniBand HCA

11.4. Checking PeerDirect Stats#

To check memory peer statistics, run the following script:

list=`ls /sys/kernel/mm/memory_peers/nvidia-fs/`; for stat in $list;do echo  "$stat value: " $(cat /sys/kernel/mm/memory_peers/nvidia-fs/$stat); done

Sample output:

num_alloc_mrs value:  1288
num_dealloc_mrs value:  1288
num_dereg_bytes value:  1350565888
num_dereg_pages value:  329728
num_free_callbacks value:  0
num_reg_bytes value:  1350565888
num_reg_pages value:  32972
version value:  1.0

11.5. Checking for Relevant nvidia-fs Stats with IBM Spectrum Scale#

Use the following steps to check for relevant nvidia-fs statistics for the IBM Spectrum Scale file system.

  1. Enable nvidia-fs statistics:

    # echo 1 > /sys/module/nvidia_fs/parameters/rw_stats_enabled

  2. $ cat /proc/driver/nvidia-fs/stats

  3. Review the output:

      [~]# cat /proc/driver/nvidia-fs/stats
    GDS Version: 1.0.0.82
    NVFS statistics(ver: 4.0)
    NVFS Driver(version: 2.7.49)
    Mellanox PeerDirect Supported: True
    IO stats: Disabled, peer IO stats: Disabled
    Logging level: info

    Active Shadow-Buffer (MiB): 0
    Active Process: 0
    Reads                           : err=0 io_state_err=0
    Sparse Reads                    : n=230 io=0 holes=0 pages=0
    Writes                          : err=0 io_state_err=237 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
    Mmap                            : n=27 ok=27 err=0 munmap=27
    Bar1-map                        : n=27 ok=27 err=0 free=27 callbacks=0 active=0
    Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
    Ops                             : Read=0 Write=0
GPU 0000:2f:00.0  uuid:621f7d17-5e7d-8f79-be27-d2f4256ddd88 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=2

11.6. GDS User Space Stats for IBM Spectrum Scale for Each Process#

To check GDS user space level stats, make sure the cufile_stats property in cufile.json is set to 3. Run the following command to check the user space stats for a specific process:

$ /usr/local/cuda-<x>.<y>/gds/tools/gds_stats -p <pid> -l 3
cuFile STATS VERSION : 4
GLOBAL STATS:
Total Files: 1
Total Read Errors : 0
Total Read Size (MiB): 7302
Read BandWidth (GiB/s): 0.691406
Avg Read Latency (us): 6486
Total Write Errors : 0
Total Write Size (MiB): 0
Write BandWidth (GiB/s): 0
Avg Write Latency (us): 0
READ-WRITE SIZE HISTOGRAM :
0-4(KiB): 0  0
4-8(KiB): 0  0
8-16(KiB): 0  0
16-32(KiB): 0  0
32-64(KiB): 0  0
64-128(KiB): 0  0
128-256(KiB): 0  0
256-512(KiB): 0  0
512-1024(KiB): 0  0
1024-2048(KiB): 0  0
2048-4096(KiB): 3651  0
4096-8192(KiB): 0  0
8192-16384(KiB): 0  0
16384-32768(KiB): 0  0
32768-65536(KiB): 0  0
65536-...(KiB): 0  0
PER_GPU STATS:
GPU 0 Read: bw=0.690716 util(%)=199 n=3651 posix=0 unalign=0 dr=0 r_sparse=0 r_inline=0 err=0 MiB=7302 Write: bw=0 util(%)=0 n=0 posix=0 unalign=0 dr=0 err=0 MiB=0 BufRegister: n=2 err=0 free=0 MiB=4
PER_GPU POOL BUFFER STATS:
PER_GPU POSIX POOL BUFFER STATS:

PER_GPU RDMA STATS:
GPU 0000:43:00.0 :   mlx5_0(130:64):Reads: 3594 Writes: 0  mlx5_1(130:64):Reads: 3708 Writes: 0
RDMA MRSTATS:
peer name   nr_mrs      mr_size(MiB)
mlx5_0      1           2
mlx5_1      1           2

In the example above, 3954 MiB of IBM Spectrum Scale Read-IO went through mlx5_0 and 3708 MiB MiB of IBM Spectrum Scale Read went through mlx5_1. The RDMA MRSTATS value shows the number of RDMA memory registrations and size of those registrations.

11.7. GDS Configuration to Support IBM Spectrum Scale#

  1. Configure the DC key.

    The DC key for the IBM Spectrum Scale client can be configured in the following ways:

    • Set the environment variable CUFILE_RDMA_DC_KEY. This should be set to a 32-bit hex value. This can be set as shown in the following example:

      export CUFILE_RDMA_DC_KEY = 0x11223344

    • Set the property rdma_dc_key in cufile.json. This property is a 32-bit value and can be set as shown in the following example:

      "rdma_dc_key": "0xffeeddcc",

    In case both the environment variable and the cufile.json have the property set, the environment variable CUFILE_RDMA_DC_KEY will take precedence over the rdma_dc_key property set in cufile.json.

    In case none of the above is set, the default DC Key configured is 0xffeeddcc.

  2. Configure the IP addresses in cufile.json.

    The >rdma_dev_addr_list property should be set in cufile.json with the IP address of the RDMA devices to be used for IO.

    "properties": {
    -------
    "rdma_dev_addr_list": [
           "172.16.8.88" , "172.16.8.89",
           "172.16.8.90" , "172.16.8.91",
           "172.16.8.92" , "172.16.8.93",
           "172.16.8.94", "172.16.8.95" ]
     }
     ---------
}

  3. Configure the max_direct_io_size_kb property in cufile.json.

    You can change the IO size with the following property:

    "properties": {
    -------
    "max_direct_io_size_kb" : 1024
    ---------
}

  4. Configure the rdma_access_mask property in cufile.json.

    This property is a performance tunable. Refer to IBM Spectrum Scale documentation in https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=configuring-gpudirect-storage-spectrum-scale for optimal configuration of this property.

    "properties": {
    -------
    "rdma_access_mask": "0x1f",
    ---------
}

Note

For CUDA 11.5.1 and later, if you plan to use IBM Spectrum Scale then you must run:

modprobe nvidia_peermem

This will load the module that supports PeerDirect capabilities. It is necessary to run this command after reboot of the system.

In order to load the module automatically after every reboot, run the following command:

echo "nvidia-peermem" | sudo tee /etc/modules-load.d/nvidia-peermem.conf

11.8. Scenarios for Falling Back to Compatibility Mode#

There are couple of scenarios that will cause the IBM Spectrum Scale IOs to go through compatibility mode, irrespective of the allow_compat_mode property’s value in cufile.json. For a full list of these cases please refer to http://www.ibm.com/support/pages/node/6444075.

11.9. GDS Limitations with IBM Spectrum Scale#

The current maximum of RDMA memory registrations for a GPU buffer is 16. Hence, the maximum size of memory that can be registered with RDMA per GPU buffer is 16 * max_direct_to_size_kb (set in cufile.json). Any GDS IO with IBM Spectrum Scale beyond this offset will go through bounce buffers and might have a performance impact.

On this page