General

• UVM may fail to initialize correctly when both MIG and HMM are used on systems with a large number of GPUs containing large amounts of GPU memory. To workaround this issue HMM may be disabled by passing uvm_disable_hmm=1 when loading the nvidia-uvm.ko kernel module. Doing so will prevent usage of HMM features such as system allocated memory. Alternatively MIG may be disabled. 4435278
• "Change ECC State" and "Enable Error Correction Code" do not change synchronously when ECC state changes. 3838953
• CUDA kernels that use the sparsity feature of tensor cores through the mma.sp PTX instruction on Hopper architecture GPUs may intermittently experience silent data corruption resulting in incorrect results. NVIDIA libraries currently do not provide access to tensor cores with sparsity so only kernels directly developed using the mma.sp PTX instruction are impacted. This issue will be fixed in an upcoming release. 4254491

• cpufreq-info is not reporting correct core frequency on Grace. 4253549

  The CPU frequency reported by the Linux kernel might vary significantly from the actual value.

  WorkaroundThere is currently no workaround, and this issue will be resolved in a future kernel version.

• Grace is doing a seemingly unnecessary munmap of the untouched host memory when CUDA context is active. 4052424

  Customers who use older stable/longterm trees should update to the latest upstream subversion, and customers who are using another tree should cherry-pick the subversion from one of the trees (versions linux-6.5.y, linux-6.1.y, or linux-5.15.y).

  Workaround

  There is currently no workaround.

• Support for 4k page size in nv-p2p. – 4316020

  The NVIDIA driver's kernel mode GPUDirect RDMA APIs that are used for Peer-direct support in MLNX_OFED and GPUDirect Storage are not supported on GH200 platforms when used with Linux kernels that are configured with the 4K page size. These APIs are not functional and might lead to a kernel memory corruption.

  Users are strongly encouraged to move their software stack to the dma-buf APIs, which requires the open-source GPU driver, Linux kernel 5.12 or later, and NVIDIA Turing™ + GPU. Since the dma-buf APIs work correctly on 4K page kernels, using the APIs is ideal mitigation for this issue.
• Graphics functionalities, such as EGL, GLX, and Vulcan, are currently not supported under the 4K OS page size – 4333780

  Workaround

  We recommend that you use a 64K page size.

• The SLES driver RPM package might have issues with 64k kernels. – 4251209

  WorkaroundUse the 64k kernel support packages from the distro package maintainer (https://en.opensuse.org/SDB:NVIDIA_drivers) or use the .run file install method.

• Set_declare SMBPBI Master caps will return success. – 3977576

  The undocumented SMBPBI opcode 0x8 does not return ERR_OPCODE in conformance with the SMBPBI specification. This opcode is not supported on the Hopper HGX 8-GPU baseboard and should not be used.

• All Mellanox ports are shown by command "nvidia-smi topo -m" after dual-port NICs are bonded.
• When polling the H100 GPU via SMBPBI using GPU Performance Monitoring metrics, driver reloads or GPU resets can result in driver errors that manifest as PID (X62) errors on Linux. NVIDIA is investigating this issue.
• On NVIDIA H800, monitoring software such as DCGM or NVML might report lower double-precision (FP64) utilization metrics. This is expected as per the NVIDIA H800 product configuration. Refer to the NVIDIA H800 product brief for more details.
• The GPU driver build system might not pick the Module.symvers file, produced when building the ofa_kernel module from MLNX_OFED, from the right subdirectory. Because of that, nvidia_peermem.ko does not have the right kernel symbol versions for the APIs exported by the IB core driver, and therefore it does not load correctly. That happens when using MLNX_OFED 5.5 or newer on a Linux Arm64 or ppc64le platform.
  To work around this issue, perform the following:

  1. Verify that nvidia_peermem.ko does not load correctly.
  2. Uninstall old MLNX_OFED if one was installed.
  3. Manually remove /usr/src/ofa_kernel/default if one exists.
  4. Install MLNX_OFED 5.5 or newer.
  5. Manually create a soft link:

      /usr/src/ofa_kernel/default -> /usr/src/ofa_kernel/$(uname -m)/$(uname -r)

  6. Reinstall the GPU driver.
• If you encounter an error on RHEL7 when installing with cuda-drivers-fabricmanager packages, use the following alternate instructions. For example:

  If you are upgrading from a different branch, for example to driver 515.65.01:

  new_version=515.65.01
  sudo yum swap nvidia-driver-latest-dkms nvidia-driver-latest-dkms-${new_version}
  sudo yum install nvidia-fabric-manager-${new_version}

• When installing a driver on SLES15 or openSUSE15 that previously had an R515 driver installed, users need to run the following command afterwards to finalize the installation:

  sudo zypper install --force nvidia-gfxG05-kmp-default 

  Without doing this, users may see the kernel objects as missing.

• nvidia-release-upgrade

  may report that not all updates have been installed and exit.
  When running the

  nvidia-release-upgrade

  command on DGX systems running DGX OS 4.99.x, it may exit and tell users: "Please install all available updates for your release before upgrading" even though all upgrades have been installed.
  Users who see this can run the following command:

  sudo apt install -y nvidia-fabricmanager-450/bionic-updates --allow-downgrades

  After running this, proceed with the regular upgrade steps:

  sudo apt update
  sudo apt full-upgrade -y 
  sudo apt install -y nvidia-release-upgrade 
  sudo nvidia-release-upgrade

• By default, Fabric Manager runs as a systemd service. If using

  DAEMONIZE=0

  in the Fabric Manager configuration file, then the following steps may be required.
  1. Disable FM service from auto starting.

     systemctl disable nvidia-fabricmanager

  2. Once the system is booted, manually start FM process.

     /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

     Note, since the process is not a daemon, the SSH/Shell prompt will not be returned (use another SSH shell for other activities or run FM as a background task).
• Important correctness fix for H100 GPU instructions used by cuBLAS, other CUDA libraries, and user CUDA code

  An issue was discovered recently with H100 GPUs (H100 PCIe and HGX H100) where certain operations put the GPU in an invalid state that allowed some GPU instructions to operate at unsupported frequency that can result in incorrect computation results and faster than expected performance. The affected GPU instructions are used by cuBLAS, other CUDA libraries, and can also be used for user CUDA code.

  The operations that allow the GPU to enter an invalid state are the following:

  • Enabling MIG
  • Deinitialize and reinitialize the GPU (for example, turn off persistence mode and turn it back on or reload the nvidia.ko driver)
  • Any Compute Engine error (for example, MMU fault, Out of Range warp error, and so on)
  Once the GPU enters the invalid state, the performance for some GPU instructions is increased by 7-10%, but the computation results may be incorrect.

  The current release fixes this issue, and it is no longer possible to enter the invalid GPU state. This issue has been present in all drivers since the H100 launch, and we recommend that you upgrade to the current release as soon as possible. If upgrading is not immediately possible, a GPU reset can restore the GPU back to the correct operational state, except for when MIG is being used. For MIG, the new driver is required, and there is no workaround available.

• Uninstalling the driver fails, and the system reboots automatically.
  On Windows 2019 and 2022 servers, uninstalling the driver causes the system to restart automatically before the uninstallation is completed. The issue also occurs when you upgrade the driver from an older version to a new version, even after selecting the Perform Clean Installation option in the installer UI.

  Note: This issue does not occur in Linux.

  Workaround

  We strongly recommend that you always install, uninstall, and upgrade drivers from Safe mode.

• In Shared Switch virtualization mode, the guest VM GPU driver load and unload stress test fails after certain iteration

  In the Shared Switch virtualization mode, the stress test to load and unload the GPU driver on Guest VM in every 30 second interval runs into issues approximately after three hours of the test.

  Workaround

  Do not run the stress reload driver cycle at this time.

• A few Async SMBPBI commands do not function as intended when the driver is unloaded.

  When the driver is unloaded, the following Async SMBPBI commands do not operate as specified:

  • Arg1 0x00: Reads total GPU power limit control data.
  • Arg1 0x01: Sets the total GPU power limit.
  • Arg1 0x02: Reads the total GPU power limit policy information.
  Due to this issue, some properties of the following Redfish URIs are impacted:

  • PowerLimitWatts.SetPoint:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]/EnvironmentMetrics

  • SpeedLimitMHz, SpeedLocked:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]

  The Patch operation of the following URIs are impacted:

  • PowerLimitWatts.SetPoint:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]/EnvironmentMetrics

  • Oem.Nvidia.PowerMode "MaxP" or "MaxQ":

    /redfish/v1/Chassis/HGX_Chassis_0/EnvironmentMetrics

  • SpeedLimitMHz, SpeedLocked:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]

  Workaround

  Load the driver for these URIs to work properly.

• Fabric Manager state is not reported accurately on NVSwitch OOB query

  The NVSwitch SMPBI query that reports Fabric Manager state (Manager State) is not reporting the actual FM state.

• Instructions to reset all GPUs Using the nvidia-smi -r Command

  When resetting all GPUs using the nvidia-smi command with the -r option instead of a resetting specific GPU using the -i <gpu_index> option, all the NVSwitches will also be reset. This process wipes out the NVSwitch routing entries, and subsequent CUDA application launches will fail. The Fabric Manager service will also show interaction errors with the NVSwitch device via the switch driver.

  Workaround

  1. Stop the Fabric Manager service.
  2. To reset all GPUs, run nvidia-smi -r.
  3. After the reset is finished, start the Fabric Manager service.

GPU Performance Counters

The use of developer tools from NVIDIA that access various performance counters requires administrator privileges. See this note for more details. For example, reading NVLink utilization metrics from nvidia-smi (nvidia-smi nvlink -g 0) would require administrator privileges.

NoScanout Mode

NoScanout mode is no longer supported on NVIDIA Data Center GPU products. If NoScanout mode was previously used, then the following line in the “screen” section of /etc/X11/xorg.conf should be removed to ensure that X server starts on data center products:

Option         "UseDisplayDevice" "None"

NVIDIA Data Center GPU products now support one display of up to 2560x1600 resolution.

Unified Memory Support

CUDA and unified memory is not supported when used with Linux power management states S3/S4.

IMPU FRU for Volta GPUs

OpenCL 3.0 Known Issues

Device side enqueue

• Device-Side-Enqueue related queries may return 0 values, although corresponding built-ins can be safely used by kernel. This is in accordance with conformance requirements described at https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_API.html#opencl-3.0-backwardscompatibility

• Shared virtual memory - the current implementation of shared virtual memory is limited to 64-bit platforms only.

Confidential Compute

• Confidential Compute Early Access has concluded, and CC support is removed in this, and all future r535 releases.
• Confidential Compute General Access is now enabled in the r550 version and future releases.
• For more information, please visit https://docs.nvidia.com/nvtrust.