Known Issues

Refer to the following known issues with NVIDIA DGX OS 6 and NVIDIA DGX Systems.

Reduced Network Communication Speeds on DGX H100 System

Issue

On DGX H100 Systems running DGX OS 6, the MAX_ACC_OUT_READ configuration for the ConnectX-7 controllers can be too low and result in reduced data transfer rates.

This issue is fixed in DGX OS 6.1.

Explanation

For DGX H100 Systems, the value is set to 44 by the nvidia-mlnx-config service.

You can check the value by using the mlxconfig or mstconfig command to query each device and view the value of the MAX_ACC_OUT_READ configuration.

Workaround

Perform the following steps to set the value to 128 and disable the nvidia-mlnx-config service.

Stop the service:
```
sudo systemctl stop nvidia-mlnx-config
```
Disable the service so that it does not start after the next boot and revert the configuration:
```
sudo systemctl disable nvidia-mlnx-config
```

Set the configuration based on whether your system uses the MLNX_OFED drivers or the Inbox drivers.

MLNX_OFED drivers

for dev in $(ls /sys/class/infiniband/); do \
  sudo mlxconfig -y -d ${dev} set ADVANCED_PCI_SETTINGS=1; \
  sudo mlxconfig -y -d ${dev} set MAX_ACC_OUT_READ=128; \
done

Inbox OFED drivers

for bdf in $(ls /sys/bus/pci/devices); do \
  if [[ -e "/sys/bus/pci/devices/${bdf}/infiniband" ]]; then \
    sudo mstconfig -y -d "${bdf}" set ADVANCED_PCI_SETTINGS=1; \
    sudo mstconfig -y -d "${bdf}" set MAX_ACC_OUT_READ=128; \
  fi \
done

NVSM Raises Alerts for Missing Devices on DGX H100 System

Issue

NVSM reports the following missing devices on NVIDIA DGX H100 Systems:

/systems/localhost/pcie/alerts/alert0
    message_details = Device is missing on b1:00.1.
    ...

/systems/localhost/pcie/alerts/alert1
    message_details = Device is missing on b1:00.0.
    ...

 /systems/localhost/pcie/alerts/alert2
    message_details = Device is missing on 0b:00.1.

Explanation

NVSM version 22.12.02 is configured with the preceding devices as resources in the DGX H100 System configuration file and the devices are not present in the system at the reported PCI IDs.

You can ignore the alerts for these PCI IDs. The alerts are false positives.

Workaround

After upgrading to a newer version of NVSM, the alerts can persist in local storage. Perform the following steps to remove the NVSM alert database from local storage:

Stop the NVSM service:
```
sudo systemctl stop nvsm
```
Delete the alert database from local storage:
```
sudo rm /var/lib/nvsm/sqlite/nvsm.db
```
Start the NVSM service:
```
sudo systemctl start nvsm
```

DGX A800 Station/Server: mig-parted config

Issue

DGX Station A800 is not currently supported in the all-balanced configuration of the default mig-parted config file.

Workaround

To add the A800 device ID to the all-balanced configuration:

Make a copy of the default configuration.
Add device ID 0x20F310DE to the device-filter of the all-balanced config.
Point mig-parted apply at this new file when selecting a config.

Erroneous Insufficient Power Error May Occur for PCIe Slots

Issue

Reported in release 4.99.9.

The DGX A100 server reports “Insufficient power” on PCIe slots when network cables are connected.

Explanation

This may occur with optical cables and indicates that the calculated power of the card + 2 optical cables is higher than what the PCIe slot can provide.

The message can be ignored.

Applications that call the cuCTXCreate API Might Experience a Performance Drop

Issue

Reported in release 5.0.

When some applications call cuCtxCreate, cuGLCtxCreate, or cut Destroy, there might be a drop in performance.

Explanation

This issue occurs with Ubuntu 22.04, but not with previous versions. The issue affects applications that perform graphics/compute interoperations or have a plugin mechanism for CUDA, where every plugin creates its own context, or video streaming applications where computations are needed. Examples include ffmpeg, Blender, simpleDrive Runtime, and cuSolverSp_LinearSolver.

This issue is not expected to impact deep learning training.

Incorrect nvidia-container-toolkit version after upgrade from 5.X to 6.0

Issue

Reported in release 6.0.

Following a release upgrade from DGX OS 5.X to DGX OS 6.X the version of nvidia-container-toolkit is not updated to the 6.X version.

Explanation

During the release upgrade process the older version of nvidia-container-toolkit has a higher priority than the newer version so it doesn’t get updated.

The workaround is to perform an additional sudo apt update and sudo apt upgrade following the nvidia-release-upgrade.

UBSAN error and mstconfig stack dump in kernel logs at boot

Issue

Reported in release 6.0.

On boot an error message similar to UBSAN: shift-out-of-bounds in /build/linux-nvidia-s96GJ3/linux-nvidia-5.15.0/debian/build/build-nvidia may appear, followed by a stack trace.

Explanation

This warning is generate when mstconfig closes a device file. The warning can be safely ignored, and will be fixed in a future release.

The BMC Redfish interface is not active on first boot after installation

Issue

Reported in release 6.0.

When the system is first booted after installation the BMC Redfish network interface will be in a DOWN state and will not appear when running the ifconfig command.

Explanation

The interface is reconfigured after the system has already brought up interfaces, so it doesn’t get automatically started.

Running the command sudo netplan apply will cause the interface to be started. It will also automatically be started on all subsequent boots even without running the additional sudo netplan apply