Known Issues#

Upgrade Error During Package Configuration#

Issue#

When you upgrade DGX OS, the upgrade process might fail during package configuration. This failure occurs because some DOCA DKMS packages recommend the linux-headers virtual package. This dependency can cause APT to install linux-headers-7.0.0-1005-gke and linux-gke-headers-7.0.0-1005. These packages also pull in the 7.0.0-1005-gke kernel, which causes the upgrade to fail.

Error message:

Autoinstall on 7.0.0-1005-gke failed for module(s) mlnx-nvme(10).

Error! One or more modules failed to install during autoinstall.
 Refer to previous errors for more information.
run-parts: /etc/kernel/header_postinst.d/dkms exited with return code 1
dpkg: error processing package linux-headers-7.0.0-1005-gke (--configure):

Error! Bad return status for module build on kernel: 7.0.0-1005-gke (x86_64)
 run-parts: /etc/kernel/header_postinst.d/dkms exited with return code 1
dpkg: error processing package linux-headers-7.0.0-1005-gke (--configure)
Error: Sub-process /usr/bin/dpkg returned an error code (1)

Errors were encountered while processing:
 linux-headers-7.0.0-1005-gke
needrestart is being skipped since dpkg has failed
Error: Sub-process /usr/bin/dpkg returned an error code (1)

Workaround#

To avoid this issue, run the following command before you run apt full-upgrade:

sudo apt install linux-nvidia -y

After you run this command, proceed with running apt full-upgrade.

If the error has already occurred, run the following command to resolve it:

sudo apt purge linux-headers-7.0.0-1005-gke linux-gke-headers-7.0.0-1005 -y

Error Messages Reported by GPU Driver#

Issue#

The GPU driver emits error messages about failing to allocate an NvKmsKapiDevice:

[   41.633844] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00080100] Failed to allocate NvKmsKapiDevice
[   41.656934] [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00080100] Failed to load device

These messages can be safely disregarded. A resolution will be provided in a future release.

RShim Devices Not Created After Starting rshim.service#

Issue#

RShim devices might not appear under /dev/rshim* because the RShim driver might be in use by the DPU BMC.

Workaround#

Option 1

  1. Connect to the BlueField-3 (BF3) BMC and stop the RShim service.

  2. On the host system, enable and restart the RShim service.

    sudo systemctl enable rshim
sudo systemctl restart rshim

  3. Verify that the service is running.

    sudo systemctl status rshim

Option 2

  1. Edit /etc/rshim.conf, and uncomment the line that says #FORCE_MODE     1.

  2. Restart the RShim service.

    sudo systemctl restart rshim.service

DGX B200 Display Goes Blank Shortly After GRUB Menu#

Issue#

During the DGX B200 system boot process, the NVIDIA splash screen and the POST display properly. However, when the DGX OS Linux messages start to appear, the screen goes blank. The system is connected to a standard HD monitor with a resolution of 1920 x 1080 pixels through the VGA port. This issue occurs due to an incompatibility between the DGX B200 monitor and the supported resolutions of the connected monitor.

Workaround#

When a low-resolution monitor is attached to the system being installed, the GRUB video= parameter should be set to the resolution of the monitor that is attached. For this specific issue, where a monitor with a resolution of 1920 x 1080 pixels is connected through the VGA port, add the video=1920x1080 parameter to the GRUB menu entry at boot time.

  1. On the GRUB boot screen, select the e key to edit the boot entry.

  2. Using the down arrow key, navigate to the linux line that contains the boot parameters.

  3. Append the following parameter to the linux line.

    video=1920x1080

  4. Select Ctrl+x to boot the system with the added parameter.

    The system finishes booting and remains at a resolution of 1920x1080 pixels.

nv-disk-encrypt Failed on Pre-Owned NVMe Drives#

Issue#

Using the nv-disk-encrypt tool to initialize the system for NVMe drive encryption failed with the following error messages:

takeOwnership failed
SED takeownership failed on /dev/nvme0n1

Workaround#

To resolve the issue, recover from lost keys and erase the drives as shown in the following single-drive example:

  1. Take ownership of all drives one at a time.

    sudo sedutil-cli --takeownership <your-sid-password> /dev/nvme6n1

  2. If step 1 fails, specify the PSID to reset the drive using the sedutil-cli command.

    Caution

    Before performing this step, back up your important data to another location because it will delete everything on the drive.

    You can obtain the PSID, which is printed on the label, by physically examining the drive.

    For example,

    sudo sedutil-cli --yesIreallywanttoERASEALLmydatausingthePSID <your-drive-psid> /dev/nvme6n1

  3. Check taking ownership.

    sudo sedutil-cli --takeownership <your-sid-password> /dev/nvme6n1

  4. Revert ownership before the initialization process for drive encryption.

    sudo sedutil-cli --reverttper <your-sid-password> /dev/nvme6n1

  5. Initialize the system for drive encryption using the nv-disk-encrypt init command.

    sudo nv-disk-encrypt init [-k <your-vault-password>] [-f <path/to/json-file>] [-g] [-r]

GPUs Cannot Be Reset During MIG Configurations on DGX A100 and A800 Systems#

Issue#

When you run the nvidia-mig-parted tool to apply the MIG configurations on DGX A100 or DGX A800 systems, the following error message might occur:

The following GPUs could not be reset:
  GPU 00000000:01:00.0: In use by another client
  GPU 00000000:47:00.0: In use by another client
  GPU 00000000:81:00.0: In use by another client
  GPU 00000000:C2:00.0: In use by another client

Workaround#

To apply a MIG configuration without rebooting, unload the nvidia_drm and nvidia_modeset modules if they are loaded. This releases their GPU client references and allows nvidia-mig-parted to reset the GPUs rather than reporting that they are in use by another client.

First check whether either module is loaded:

lsmod | grep -E '^nvidia(_drm|_modeset)?\b'

If either of the modules is loaded, stop any GPU or display clients that are currently active with appropriate commands.

Then, unload the nvidia_modeset and nvidia_drm modules before running nvidia-mig-parted:

sudo rmmod nvidia_drm nvidia_modeset

Run the desired nvidia-mig-parted configuration command(s).

sudo nvidia-mig-parted <desired_arguments>

Then, reload the nvidia_modeset and nvidia_drm modules:

sudo modprobe nvidia_modeset nvidia_drm

If the GPUs still cannot be reset, reboot the server.

Note

On headless DGX systems, the nvidia_drm and nvidia_modeset modules might not be loaded, indicating that something other than nvidia_drm or nvidia_modeset is holding the GPUs open. In this case, it will be necessary to reboot to resolve the inability to reset GPUs.

Virtualization Not Tested#

Issue#

Virtualization technology, such as ESXi hypervisors or kernel-based virtual machines (KVM), is not an intended use case on DGX systems and has not been tested.