Is this page helpful?

Known Issues#

Upgrade Reports XPMEM DKMS Build Failures and Missing Packages#

Issue#

During an upgrade from DGX OS 7.6.0 to DGX OS 8.0.x, xpmem/2510.0.21 might fail to build against the target kernels. These failures prevent kernel packages from being configured correctly. The upgrade might also report that the dgx-system-utils and nvidia-system-extra-localization packages cannot be located.

Although xpmem/2604.0.2 might subsequently build successfully, the upgrade finishes with an error summary and exit status 1.

Workaround#

Before you start the upgrade, purge the XPMEM DKMS package:

sudo apt purge xpmem-dkms

Then start the upgrade:

sudo nvidia-release-upgrade --baseos 9 -d

Packages Kept Back after Upgrading#

Issue#

After an upgrade, the nvidia-fs and nvidia-fs-dkms packages might be kept back.

This issue occurs because the dkms package from the DOCA repository has a higher installation priority than the version from the CUDA repository. However, the latest nvidia-fs and nvidia-fs-dkms packages require a newer version of dkms than the version available from the DOCA repository.

Workaround#

To resolve this issue, explicitly install the required dkms package before the upgrade:

sudo apt update
sudo apt install dkms=1:3.4.3-1ubuntu1
sudo apt dist-upgrade

nvidia_fs Kernel Module Fails to Load During Upgrade#

Issue#

After upgrading DGX systems running DGX OS 7.5.0, the GPUDirect Storage (GDS) kernel module nvidia_fs might fail to load. Affected systems report validation failures because the module is absent from the running kernel.

This issue results from a Dynamic Kernel Module Support (DKMS) package ordering conflict during the upgrade process. Specifically, nvidia-dkms-580-open is configured before mlnx-ofed-kernel-dkms, preventing it from finding certain required symbol files. As a result, module compilation fails during the build phase and dependent packages (including nvidia-fs-dkms) remain unconfigured for the pending kernel.

Affected environments:

  • Releases: DGX OS 7.5.0 and later

  • Platforms:

    • DGX B300 and DGX B200

    • DGX H100/H200/H800

    • DGX A100/A800

    • DGX GB300 and DGX GB200

Workaround#

To prevent build failures when upgrading from DGX OS 7.5.0, run the following commands in sequence:

  1. Refresh the package list:

    sudo apt update

  2. Upgrade the Mellanox OFED kernel driver package first:

    sudo apt install --only-upgrade mlnx-ofed-kernel-dkms

  3. Proceed with the standard distribution upgrade:

    sudo apt dist-upgrade

    Note

    Any apt dist-upgrade configuration errors encountered during this step can be safely ignored, provided step 4 is executed immediately after.

  4. Repair and complete unconfigured package builds:

    sudo dpkg --configure -a

  5. Reboot:

    sudo reboot

  6. Rebuild and load nvidia-fs:

    nvidia_fs_module=$(dkms status nvidia-fs -k $(uname -r) | cut -d, -f1)
sudo dkms remove "$nvidia_fs_module" -k "$(uname -r)"
sudo dkms install "$nvidia_fs_module" -k "$(uname -r)"
sudo modprobe nvidia-fs

Full Power Cycle Might Be Required to Apply MAX_ACC_OUT_READ Changes#

Issue#

On any NVIDIA DGX platform equipped with NVIDIA ConnectX adapters and running a DGX OS release earlier than 7.6.0, a change to the Mellanox MAX_ACC_OUT_READ setting might not take effect after a standard reboot. The nvidia-mlnx-config package sets MAX_ACC_OUT_READ to the value appropriate for the platform type in which the ConnectX adapter is installed. Alternatively, you can modify MAX_ACC_OUT_READ manually by using the mlxconfig command. Although mlxconfig indicates that a reboot is required, the adapter’s Current value can remain different from its Next Boot value until a full power cycle is performed.

The expected MAX_ACC_OUT_READ value depends on the platform:

  • 44 on DGX A100, DGX A800, DGX Station A100, and DGX Station A800 systems.

  • 0 on all other DGX platform types.

For platforms whose expected value is 0, 0 in the Next Boot column means that the adapter uses its default value. For example, Current and Default can both show 128 while Next Boot shows 0. This is the expected result after a reboot and does not require a power cycle.

After a standard reboot, you can determine whether a full power cycle is required by running following command to display the values for each ConnectX interface:

#!/bin/sh

for devpath in /sys/class/infiniband/mlx5_*; do
  [ -e "${devpath}" ] || continue

  dev=${devpath##*/}
  bdf=$(basename "$(readlink -f "${devpath}/device")")

  if lspci -D -s "${bdf}" | grep -q "ConnectX-"; then
    echo "${dev} (${bdf})"

    output=$(sudo mlxconfig -d "${dev}" --enable_verbosity query 2>&1)
    rc=$?

    if [ "${rc}" -ne 0 ]; then
      printf 'rc: %s %s\n' "${rc}" "${output}"
    else
      printf '%s\n' "${output}" |
        grep -E 'Configurations:|MAX_ACC_OUT_READ'
    fi
    printf '\n'
  fi
done

In the output, compare the Current and Next Boot columns with the expected value for the platform:

  • For platforms whose expected value is 44, the setting is correct when both columns contain 44 for every interface.

  • For platforms whose expected value is 0, the setting is correct when Next Boot contains 0 and Current matches Default for every interface.

  • For platforms whose expected value is 44, if Next Boot contains 44 but Current does not, the change did not take effect during the reboot and a full power cycle is required.

  • If Next Boot does not contain the expected value, the change was not staged correctly. Restart nvidia-mlnx-config.service and repeat the check:

    sudo systemctl restart nvidia-mlnx-config.service

    Verify that Next Boot now contains the expected value. For platforms whose expected value is 44, perform a full power cycle if Current still differs.

Workaround#

If a full power cycle is required, perform it through the baseboard management controller (BMC) instead of issuing another standard reboot. For example, run the following IPMI command:

ipmitool power cycle

After the system starts, repeat the check. For platforms whose expected value is 44, verify that both Current and Next Boot contain 44. For platforms whose expected value is 0, verify that Next Boot contains 0 and Current matches Default for every ConnectX interface.

Upgrading the Kernel May Cause DKMS Errors#

Issue#

When upgrading the kernel on DGX systems running DGX OS 7.5.0 and DOCA versions earlier than 3.2.1-044418, the following DKMS errors may occur:

Error message:

x86_64:

Autoinstall on 6.8.0-110-generic succeeded for module(s) kernel-mft-dkms knem mlnx-ofed-kernel nvidia srp xpmem nvidia-fs.
Autoinstall on 6.8.0-110-generic failed for module(s) iser(10) isert(10) mlnx-nfsrdma(10) mlnx-nvme(10).

arm64:

Autoinstall on 6.17.0-1017-nvidia-64k succeeded for module(s) kernel-mft-dkms knem xpmem.
Autoinstall on 6.17.0-1017-nvidia-64k failed for module(s) nvidia(10).

If the kernel upgrade has not yet been performed, the issue can be avoided by updating the DOCA components before upgrading the kernel.

Workaround#

Run the following commands:

  1. Refresh the package list.

    sudo apt update

  2. Install doca-ofed.

    sudo apt install doca-ofed mlnx-nfsrdma-dkms mlnx-nvme-dkms

  3. Retry the upgrade.

    sudo apt full-upgrade -y

DOCA Repository GPG Key Error for Systems Installed Using the NVIDIA DGX Software Stack#

Issue#

The repo definition and keys installed when following the steps in Installing DGX System Configurations and Tools need to be updated due to a new repo location and GPG key for the DOCA repository.

When running the sudo apt update command, you might encounter the following GPG error:

arm64:

GPG error: https://linux.mellanox.com/public/repo/doca/DGX_GBxx_latest_DOCA/ubuntu24.04/arm64-sbsa
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY DC726C5E41B9CC50

x86_64:

GPG error: https://linux.mellanox.com/public/repo/doca/DGX_latest_DOCA/ubuntu24.04/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY DC726C5E41B9CC50

Workaround#

This procedure installs the updated repository definitions and missing GPG keys needed to verify the DOCA repository signatures.

  1. Install the new repository definitions and key packages.

    • For x86_64 systems:

      curl https://repo.download.nvidia.com/baseos/ubuntu/noble/x86_64/dgx-repo-files.tgz | sudo tar xzf - -C /

    • For ARM64 systems:

      curl https://repo.download.nvidia.com/baseos/ubuntu/noble/arm64/dgx-repo-files.tgz | sudo tar xzf - -C /

  2. Remove the old DOCA repository definition file.

    sudo rm /etc/apt/sources.list.d/doca.sources

  3. Refresh your package list to include the newly added DOCA repository.

    sudo apt update

    The GPG error no longer appears.

DOCA Repository GPG Key Error#

Issue#

When running the sudo apt update command, you might encounter the following GPG error:

GPG error: https://linux.mellanox.com/public/repo/doca/3.2.1-044413/ubuntu24.04/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY DC726C5E41B9CC50

Workaround#

This procedure installs the missing GPG keys needed to verify the DOCA repository signatures.

  1. Install the required repository and key packages.

    sudo apt install doca-bos8-latest-repo nvidia-repo-keys -y

  2. Refresh your package list to include the newly added DOCA repository.

    sudo apt update

    The GPG error no longer appears.

DGX OS Upgrade Fails on linux-generic and linux-nvidia Packages#

Issue#

When upgrading DGX OS, the upgrade process might fail when installing the linux-generic and linux-nvidia packages. This failure occurs due to some missing dependencies in the iser-dkms and isert-dkms packages.

Error message:

x86_64:

Errors were encountered while processing:
   linux-headers-6.8.0-101-generic
   linux-headers-generic
   linux-generic

arm64:

Errors were encountered while processing:
   linux-headers-6.17.0-1017-nvidia-64k
   linux-headers-nvidia-64k-hwe-24.04
   linux-nvidia-64k-hwe-24.04

Workaround#

To resolve this issue and complete the upgrade, run the following command immediately after the error occurs:

sudo apt -f install -y

This completes the linux-generic-6.8.0-xxx or linux-nvidia-64k-hwe-24.04 package configuration.

NVMe Opcode 0x2 (I/O Cmd) QID 256 Timeout#

Issue#

On DGX B300 systems with Micron E1.S drives, the following non-destructive issue appears on every boot. This occurs because the NVSM tool periodically checks drive health by running the mdadm --examine command, which triggers an error message similar to the following example approximately every 30 seconds. Over time, this might fill up the kernel log.

2025-12-22T21:42:38.413108+08:00 53A kernel: nvme nvme2: I/O tag 641 (3281) opcode 0x2 (I/O Cmd) QID 256 timeout, aborting req_op:DRV_IN(34) size:512
2025-12-22T21:42:38.413129+08:00 53A kernel: nvme nvme2: Abort status: 0x4002
2025-12-22T21:43:08.621095+08:00 53A kernel: nvme nvme2: I/O tag 641 (3281) opcode 0x2 (I/O Cmd) QID 256 timeout, reset controller
2025-12-22T21:43:08.734094+08:00 53A kernel: nvme nvme2: D3 entry latency set to 10 seconds
2025-12-22T21:43:08.755075+08:00 53A kernel: nvme nvme2: 256/0/0 default/read/poll queues

This issue is related to the Micron E1.S drive firmware. Currently, there is no temporary solution available.

Upgrade Error During Package Configuration#

Issue#

When upgrading DGX OS, the upgrade process might fail during package configuration. This failure occurs because the nvidia-fs DKMS module has a dependency on the nvidia module, which might not be built until later in the upgrade process.

Error message:

dpkg: error processing package linux-nvidia-64k-6.14 (--configure):
  dependency problems - leaving unconfigured

Workaround#

To resolve this issue and complete the upgrade, run the following command immediately after the error occurs:

sudo apt -f install -y

This command will complete the configuration of the linux-nvidia-64k-6.14 package.

Error Messages Reported by GPU Driver#

Issue#

The GPU driver emits error messages about failing to allocate an NvKmsKapiDevice:

[   41.633844] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00080100] Failed to allocate NvKmsKapiDevice
[   41.656934] [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00080100] Failed to load device

These messages can be safely disregarded. A resolution will be provided in a future release.

DGX GB200 System Failure During Upgrade#

Issue#

Performing a DGX OS upgrade to version 7.2.1 and later can result in DOCA DKMS build errors on DGX GB200 systems. After attempting to reboot, the system cannot boot to the new kernel.

Workaround#

Note

If you have rebooted before applying this workaround, the system might not boot to the new kernel. In this situation, use the GRUB menu’s Advanced Options to select and boot to an older kernel, and then apply the workaround.

To successfully upgrade to DGX OS 7.2.1 on DGX GB200 systems, run the following commands:

  1. Re-run DKMS build with the --force option against the newly installed kernel.

    Be sure to specify your new kernel version, as the one in the example might be outdated if you have recently upgraded. To help identify the new kernel version, use the ls -v /boot/vmlinuz-* command to list installed kernels.

    sudo dkms autoinstall --force -k 6.8.0-1032-nvidia-64k

  2. Re-configure broken packages.

    sudo apt -f install -y

RShim Devices Not Created After Starting rshim.service#

Issue#

RShim devices might not appear under /dev/rshim* because the RShim driver might be in use by the DPU BMC.

Workaround#

Option 1

  1. Connect to the BlueField-3 (BF3) BMC and stop the RShim service.

  2. On the host system, enable and restart the RShim service.

    sudo systemctl enable rshim
sudo systemctl restart rshim

  3. Verify that the service is running.

    sudo systemctl status rshim

Option 2

  1. Edit /etc/rshim.conf, and uncomment the line that says #FORCE_MODE     1

  2. Restart the RShim service.

    sudo systemctl restart rshim.service

Firmware Crash on DGX GB200 During Boot#

Issue#

On DGX GB200 systems, you might observe a firmware crash during some reboots, resulting in an extra reboot. This can be visible over the serial console.

Log of events seen for this issue:

INFO:    th500_ras_intr_handler: External Abort reason=0 syndrome=0xbe000411 flags=0x1
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ERROR:   spmd_ffa_direct_message failed (4294967292) on CPU121
ASSERT: plat/nvidia/tegra/soc/th500/plat_ras.c:408
BACKTRACE: START: assert
0: EL3: 0x78732b2648
1: EL3: 0x78732b0294
2: EL3: 0x78732c45fc
3: EL3: 0x78732c481c
4: EL3: 0x78732bda18
5: EL3: 0x78732b9f18
6: EL3: 0x78732b1258
BACKTRACE: END: assert

Workaround#

Currently, there is no temporary solution available.

The nvidia_peermem Module Does Not Load After an OTA Update#

Issue#

After you perform an OTA update, the nvidia-peermem-loader package is installed, but the nvidia_peermem module is not loaded. As a result, the following error message appears in the dmesg log:

nvidia_peermem: disagrees about version of symbol ib_register_peer_memory_client
nvidia_peermem: Unknown symbol ib_register_peer_memory_client (err -22)

Workaround#

This issue occurs because updating the mlnx-ofed-kernel package requires rebuilding the nvidia-peermem DKMS module to use the new version of the mlnx-ofed-kernel package.

The following commands will rebuild the necessary modules:

MODULE_VERSION=`dkms status nvidia -k $(uname -r) | cut -d "," -f1` || true
if [ ! -z "${MODULE_VERSION}" ]; then
         sudo dkms remove -m ${MODULE_VERSION} -k $(uname -r) || true
         sudo dkms install -m ${MODULE_VERSION} -k $(uname -r) || true
fi

The systemd-modules-load Service Failed to Insert the nvidia_peermem Module#

Issue#

On a DGX Station A100 or DGX Station A800, when you install a Base OS version earlier than 7.0.2, the nvidia-peermem-loader package might have been installed on the system. As a result, the following error message occurs:

$ sudo systemctl status systemd-modules-load.service
(code=exited, status=1/FAILURE)
...
systemd-modules-load[2143]: Failed to insert module 'nvidia_peermem': Invalid argument

Workaround#

To avoid this failure, remove the nvidia-peermem-loader package.

sudo apt purge nvidia-peermem-loader

BMC Redfish Interface Not Active on the First Boot After Installation#

Issue#

Reported in DGX OS 7.0.0.

When the DGX system was booted the first time after the DGX OS 7.0.0 installation, the BMC Redfish network interface was not renamed or autoconfigured correctly, as reported by the ip a command.

Workaround#

During the installation of DGX OS 7.0.x, the Redfish interface might not be configured correctly with the proper interface name or IP address. To resolve this issue, run the following command to reconfigure the interface:

sudo /usr/sbin/configure-redfish-intf.bash

No Permissions to Access /var/run/nvidia-fabricmanager for Non-Root Users on DGX B200#

Issue#

After DGX OS 7.0.x is installed on the DGX B200 system, the required access to the /var/run/nvidia-fabricmanager directory by the Fabric Manager service is not set for a non-root account. This can cause failures running the HPL benchmark and NCCL test as a non-root user.

Workaround#

Change the permissions on the /var/run/nvidia-fabricmanager/ directory by running the following commands:

# If Fabric Manager is running, stop it.
sudo systemctl stop nvidia-fabricmanager.service

# Change the permission setting to 755.
sudo chmod 755 /var/run/nvidia-fabricmanager/

# Start Fabric Manager.
sudo systemctl start nvidia-fabricmanager.service

DGX B200 Display Goes Blank Shortly After GRUB Menu#

Issue#

During the DGX B200 system boot process, the NVIDIA splash screen and the POST display properly. However, when the DGX OS Linux messages start to appear, the screen goes blank. The system is connected to a standard HD monitor with a resolution of 1920 x1080 pixels through the VGA port. This issue occurs due to an incompatibility between the DGX B200 monitor and the supported resolutions of the connected monitor.

Workaround#

When a low-resolution monitor is attached to the system being installed, the GRUB video= parameter should be set to the resolution of the monitor that is attached. For this specific issue, where a monitor with a resolution of 1920 x 1080 pixels is connected through the VGA port, add the video=1920x1080 parameter to the GRUB menu entry at boot time.

  1. On the GRUB boot screen, select the e key to edit the boot entry.

  2. Using the down arrow key, navigate to the linux line that contains the boot parameters.

  3. Append the following parameter to the linux line.

    video=1920x1080

  4. Select Ctrl+x to boot the system with the added parameter.

    The system finishes booting and remains resolution at 1920x1080 pixels.

nv-disk-encrypt Failed on Pre-Owned NVMe Drives#

Issue#

Using the nv-disk-encrypt tool to initialize the system for NVMe drive encryption failed with the following error messages:

takeOwnership failed
SED takeownership failed on /dev/nvme0n1

Workaround#

To resolve the issue, recover from lost keys and erase the drives as shown in the following single-drive example:

  1. Take ownership of all drives one at a time.

    sudo sedutil-cli --takeownership <your-sid-password> /dev/nvme6n1

  2. If step 1 fails, specify the PSID to reset the drive using the sedutil-cli command.

    Caution

    Before performing this step, back up your important data to another location because it will delete everything on the drive.

    You can obtain the PSID, which is printed on the label, by physically examining the drive.

    For example,

    sudo sedutil-cli --yesIreallywanttoERASEALLmydatausingthePSID <your-drive-psid> /dev/nvme6n1

  3. Check taking ownership.

    sudo sedutil-cli --takeownership <your-sid-password> /dev/nvme6n1

  4. Revert ownership before the initialization process for drive encryption.

    sudo sedutil-cli --reverttper <your-sid-password> /dev/nvme6n1

  5. Initialize the system for drive encryption using the nv-disk-encrypt init command.

    sudo nv-disk-encrypt init [-k <your-vault-password>] [-f <path/to/json-file>] [-g] [-r]

GPUs Cannot Be Reset During MIG Configurations on DGX A100 and A800 Systems#

Issue#

When you run the nvidia-mig-parted tool to apply the MIG configurations on DGX A100 or DGX A800 systems, the following error message might occur:

The following GPUs could not be reset:
  GPU 00000000:01:00.0: In use by another client
  GPU 00000000:47:00.0: In use by another client
  GPU 00000000:81:00.0: In use by another client
  GPU 00000000:C2:00.0: In use by another client

Workaround#

To recover from this error, reboot the server to apply the most recent nvidia-mig-parted configuration.

For future updates using the nvidia-mig-parted command, ensure to run the following command before any additional nvidia-mig-parted commands:

$ sudo rmmod nvidia_drm nvidia_modeset

After the nvidia-mig-parted command is complete, reload the nvidia_drm and nvidia_modeset modules.

DGX System Device ID Not Found in /usr/share/misc/pci.ids#

Issue#

When you run the following command to apply the default mig-parted configuration, the nvidia-mig-parted tool issues warnings about failing to find the device ID for the DGX system:

$ sudo nvidia-mig-parted apply -f /etc/nvidia-mig-manager/config-default.yaml -c all-balanced -k /etc/nvidia-mig-manager/hooks-default.yaml

2024/09/05 01:00:00 WARNING: unable to get device name: [failed to find device with id '22a3']
2024/09/05 01:00:00 WARNING: unable to get device name: [failed to find device with id '22a3']
2024/09/05 01:00:00 WARNING: unable to get device name: [failed to find device with id '22a3']

Workaround#

Update the system with the current version of the PCI ID list by running the update-pciids command:

sudo update-pciids

Missing the nvidia-system-station Metapackage on DGX Station A100 and DGX Station A800#

Issue#

During the installation using the DGX OS 7.0.0 ISO on a DGX Station A100 or DGX Station A800, the nvidia-system-station metapackage was incorrectly removed.

Workaround#

This issue occurs only on the DGX Station A100 and DGX Station A800. To resolve this issue, install the nvidia-system-station metapackage manually:

sudo apt install nvidia-system-station

Virtualization Not Supported#

Issue#

Virtualization technology, such as ESXi hypervisors or kernel-based virtual machines (KVM), is not an intended use case on DGX systems and has not been tested.

On this page