20. Cheat Sheet for Diagnosing Problems#

The following tables can help users diagnose GDS problems.

Make sure to go through following variables and observe if performance is where it needs to be.

Variable impacting performance

Description

Steps to take enable/disable the functionality (“How to”)

Compat mode

Disable compat mode in cufile.json

Set allow_compat_mode: false in cufile.json.

Or

Set CUFILE_FORCE_COMPAT_MODE environment variable to false.

Log level

Set log level to ERROR in cufile.json

Following setting in cufile.json will set the logging level to ERROR.

"logging": {
         // log

directory, if not enabled will create log file under current working directory

//"dir": "/home/<xxxx>",
        // NOTICE|ERROR|WARN|INFO|DEBUG|TRACE (in decreasing order of severity)
       "level": "ERROR"
        },

Nvidia-fs stats

Make sure nvidia-fs read/write stats are disabled. These can have a performance impact for small IO sizes. By default these are disabled.

To check the current state of the stats, use the following command.

#cat /sys/module/nvidia_fs/parameters/rw_stats_enabled

0 - Disabled

1 - Enabled

To disable them,

echo 0 > /sys/module/nvidia_fs/parameters/rw_stats_enabled

GDR stats/RDMA stats (CQE errors)

Relax Ordering

For distributed file systems, make sure NICs have relax ordering is enabled

set “MAX_ACC_OUT_READ=44” on ConnectX-6.

Set “MAX_ACC_OUT_READ=0” for ConnectX-7.

sudo mlxconfig -y -d <NIC> set ADVANCED_PCI_SETTINGS=1

sudo mlxconfig -y -d <NIC> set MAX_ACC_OUT_READ=44|0

sudo reboot

Persistent mode

Enable persistent mode

Clock speed

Set clock speed to maximum

BAR size

Make sure BAR size is enabled to maximum possible value

Numa affinity

Set numa affinity of the process where NIC-GPU are in the same switch

MRRS

Sets the PCIe Max Read Request Size for the NIC/NVMe Specifying Max Read request size enables the Requestor (NIC/NVME) to read data from the GPU memory upto the specified size to improve Writes from GPU to storage performance.

Check the setting using:

# lspci -vvv -s <B:D.F> | grep -i MaxReadReq

Read the current value:

#setpci -v -s <B:D.F> cap_exp+8.w

To set to 4K:

#setpci -v -s <B:D.F> cap_exp+8.w= ``5`` 000:7000

To set to 512 bytes:

#setpci -v -s <B:D.F> cap_exp+8.w= ``2`` 000:7000

The acceptable values are: 0 - 128B, 1 - 256B, 2 - 512B, 3 - 1024B, 4 - 2048B and 5 - 4096B.

Caution: Specifying selector indexes outside this range might cause the system to crash.

For ROCE setups, consider additional following items:

CPU Governor

Performance

#cpupower frequency-set -g performance

RX/TX ring

Set them to maximum

#ethtool -G $adapter rx $(ethtool -g $adapter | awk '/RX:/ {print $NF; exit}')#ethtool -G $adapter tx $(ethtool -g $adapter | awk '/TX:/ {print $NF; exit}')

RX/TX channels

Set to max allowed

#ethtool -L $adapter combined 15

LRO

Turn on large receive offload

#ethtool -K $adapter lro on

IRQ affinity

Set IRQ affinity to the affine NUMA node

IRQ balance

Turn off IRQ balancer

#systemctl stop irqbalance

TX queue length

Increase TX queue length

#ifconfig $adapter $addr/$netmask mtu 9000 txqueuelen 20000 up

If the above steps do not help, collect the following information and share it with us.

Measure GDR performance

For RDMA connectivity and performance issues run ib_read and ib_write tests with cuda and GPU enabled

Follow instructions at linux-rdma/perftest

In tests use the option

--use_cuda=<gpu index>

Use Nsight systems

For RDMA connectivity and performance issues run ib_read and ib_write tests with cuda and GPU enabled

cat /etc/cufile.json | grep nvtx

// nvtx profiling on/off

"nvtx": true,

/usr/local/cuda/bin/nsys profile <command>

Describe env

Virtual (Docker or actual VM) or BM

# printenv

# lsb_release -a

# dmidecode

# docker info

# docker container inspect <container id>

Collect gds logs

Collect cufile.log

/usr/local/cuda/gds/tools/gds_log_collection.py

# /usr/local/cuda/gds/tools/gds_log_collection.py

Debugging:

Dmesg errors?

Check for kernel errors

# dmesg

# /var/log/kern.log

MiG mode enabled?

Check if MIG is enabled

# nvidia-smi mig -lgi

FM enabled or not

For NVSwitch based systems. Check if fabric manager is running and active without any errors

# systemctl status nvidia fabricmanager