Known Issues

Incorrect DCGM Version After Upgrade from 5.X to 6.2.0
Drop IQ2M Usage to Push QMDs
Errors Occur When Loading Mirrored Repositories on Air-Gapped Systems
Reduced Network Communication Speeds on DGX H100 System
NVSM Raises Alerts for Missing Devices on DGX H100 System
DGX A800 Station/Server: mig-parted config
Erroneous Insufficient Power Error May Occur for PCIe Slots
Applications that call the cuCTXCreate API Might Experience a Performance Drop
Incorrect nvidia-container-toolkit version after upgrade from 5.X to 6.0
UBSAN error and mstconfig stack dump in kernel logs at boot
The BMC Redfish interface is not active on first boot after installation

Incorrect DCGM Version After Upgrade from 5.X to 6.2.0

Issue

During the upgrade process from DGX OS 5.X to DGX OS 6.2.0, the version of the NVIDIA® Data Center GPU Manager (DCGM) is not updated to the latest version.

Workaround

When you upgrade from Base OS 5.x to Base OS 6.2.0, DCGM might not be upgraded automatically. To ensure that the latest DCGM version is installed, run the following commands after the upgrade and reboot are complete:

sudo apt update
sudo apt upgrade

Drop IQ2M Usage to Push QMDs

Issue

A potential corruption can occur due to launching kernels on H100 GPUs. The issue is more likely to happen when the GPU is shared among multiple processes. This symptom might manifest in XID 13 error messages, such as Graphics Exception: SKEDCHECK11_TOTAL_THREADS. Currently, this issue has no user-controllable workaround and will be resolved in the next release.

Note

At the time of the 6.2.0 release, the software stack does not contain the latest R535TeslaRD8 GPU driver, which will be available in an upcoming release shortly.

Reduced Network Communication Speeds on DGX H100 System

Issue

On DGX H100 Systems running DGX OS 6.0, the MAX_ACC_OUT_READ configuration for the ConnectX-7 controllers can be too low and result in reduced data transfer rates.

This issue is fixed for DGX H100 systems that ship from NVIDIA with DGX OS 6.1 preinstalled.

For DGX H100 systems that were initially preinstalled with DGX OS 6.0, the workaround in the following section is required even if the system is upgraded or re-imaged to DGX OS 6.1.

Explanation

For DGX H100 Systems, the value is set to 44 by the nvidia-mlnx-config service.

You can check the value by using the mlxconfig or mstconfig command to query each device and view the value of the MAX_ACC_OUT_READ configuration.

Workaround

Perform the following steps to set the value to 0. For DGX OS 6.0, disable the nvidia-mlnx-config service.

On systems that were upgraded or re-imaged with DGX OS 6.1, the following systemctl commands might return an error message that begins with Failed to stop.... This message can occur if the nvidia-mlnx-config service is not running or not installed. Continue to perform the remaining steps.

For DGX OS 6 systems, stop and disable the nvidia-mlnx-config service.
1. Stop the service:
```
sudo systemctl stop nvidia-mlnx-config
```
2. Disable the service so that it does not start after the next boot and revert the configuration:
```
sudo systemctl disable nvidia-mlnx-config
```

For all systems, set the configuration based on whether your system uses the MLNX_OFED drivers or the Inbox drivers.

MLNX_OFED drivers

for dev in $(ls /sys/class/infiniband/); do \
  sudo mlxconfig -y -d ${dev} set ADVANCED_PCI_SETTINGS=1; \
  sudo mlxconfig -y -d ${dev} set MAX_ACC_OUT_READ=0; \
done

Inbox OFED drivers

for bdf in $(ls /sys/bus/pci/devices); do \
  if [[ -e "/sys/bus/pci/devices/${bdf}/infiniband" ]]; then \
    sudo mstconfig -y -d "${bdf}" set ADVANCED_PCI_SETTINGS=1; \
    sudo mstconfig -y -d "${bdf}" set MAX_ACC_OUT_READ=0; \
  fi \
done

Perform a chassis power cycle so the changes take effect.

NVSM Raises Alerts for Missing Devices on DGX H100 System

Issue

NVSM reports the following missing devices on NVIDIA DGX H100 Systems:

/systems/localhost/pcie/alerts/alert0
    message_details = Device is missing on b1:00.1.
    ...

/systems/localhost/pcie/alerts/alert1
    message_details = Device is missing on b1:00.0.
    ...

 /systems/localhost/pcie/alerts/alert2
    message_details = Device is missing on 0b:00.1.

Explanation

NVSM version 22.12.02 is configured with the preceding devices as resources in the DGX H100 System configuration file and the devices are not present in the system at the reported PCI IDs.

You can ignore the alerts for these PCI IDs. The alerts are false positives.

Workaround

After upgrading to a newer version of NVSM, the alerts can persist in local storage. Perform the following steps to remove the NVSM alert database from local storage:

Stop the NVSM service:
```
sudo systemctl stop nvsm
```
Delete the alert database from local storage:
```
sudo rm /var/lib/nvsm/sqlite/nvsm.db
```
Start the NVSM service:
```
sudo systemctl start nvsm
```

DGX A800 Station/Server: mig-parted config

Issue

DGX Station A800 is not currently supported in the all-balanced configuration of the default mig-parted config file.

Workaround

To add the A800 device ID to the all-balanced configuration:

Make a copy of the default configuration.
Add device ID 0x20F310DE to the device-filter of the all-balanced config.
Point mig-parted apply at this new file when selecting a config.

Erroneous Insufficient Power Error May Occur for PCIe Slots

Issue

Reported in release 4.99.9.

The DGX A100 server reports “Insufficient power” on PCIe slots when network cables are connected.

Explanation

This may occur with optical cables and indicates that the calculated power of the card + 2 optical cables is higher than what the PCIe slot can provide.

The message can be ignored.

Applications that call the cuCTXCreate API Might Experience a Performance Drop

Issue

Reported in release 5.0.

When some applications call cuCtxCreate, cuGLCtxCreate, or cut Destroy, there might be a drop in performance.

Explanation

This issue occurs with Ubuntu 22.04, but not with previous versions. The issue affects applications that perform graphics/compute interoperations or have a plugin mechanism for CUDA, where every plugin creates its own context, or video streaming applications where computations are needed. Examples include ffmpeg, Blender, simpleDrive Runtime, and cuSolverSp_LinearSolver.

This issue is not expected to impact deep learning training.

Incorrect nvidia-container-toolkit version after upgrade from 5.X to 6.0

Issue

Reported in release 6.0.

Following a release upgrade from DGX OS 5.X to DGX OS 6.X the version of nvidia-container-toolkit is not updated to the 6.X version.

Explanation

During the release upgrade process the older version of nvidia-container-toolkit has a higher priority than the newer version so it doesn’t get updated.

The workaround is to perform an additional sudo apt update and sudo apt upgrade following the nvidia-release-upgrade.

UBSAN error and mstconfig stack dump in kernel logs at boot

Issue

Reported in release 6.0.

On boot an error message similar to UBSAN: shift-out-of-bounds in /build/linux-nvidia-s96GJ3/linux-nvidia-5.15.0/debian/build/build-nvidia may appear, followed by a stack trace.

Explanation

This warning is generate when mstconfig closes a device file. The warning can be safely ignored, and will be fixed in a future release.

The BMC Redfish interface is not active on first boot after installation

Issue

Reported in release 6.0.

When the system is first booted after installation the BMC Redfish network interface will be in a DOWN state and will not appear when running the ifconfig command.

Explanation

The interface is reconfigured after the system has already brought up interfaces, so it doesn’t get automatically started.

Running the command sudo netplan apply will cause the interface to be started. It will also automatically be started on all subsequent boots even without running the additional sudo netplan apply

Known Issues

Incorrect DCGM Version After Upgrade from 5.X to 6.2.0

Issue

Workaround

Drop IQ2M Usage to Push QMDs

Issue

Errors Occur When Loading Mirrored Repositories on Air-Gapped Systems

Issue

Explanation

Workaround

Reduced Network Communication Speeds on DGX H100 System

Issue

Explanation

Workaround

NVSM Raises Alerts for Missing Devices on DGX H100 System

Issue

Explanation

Workaround

DGX A800 Station/Server: mig-parted config

Issue

Workaround

Erroneous Insufficient Power Error May Occur for PCIe Slots

Issue

Explanation

Applications that call the cuCTXCreate API Might Experience a Performance Drop

Issue

Explanation

Incorrect nvidia-container-toolkit version after upgrade from 5.X to 6.0

Issue

Explanation

UBSAN error and mstconfig stack dump in kernel logs at boot

Issue

Explanation

The BMC Redfish interface is not active on first boot after installation

Issue

Explanation