General

• OpenCL is not supported in the confidential compute mode available with Hopper (https://www.nvidia.com/en-in/data-center/solutions/confidential-computing/) . Any attempt to run OpenCL apps in this mode will result in clGetPlatformIDs returning an error CL_INVALID_DEVICE.
• All Mellanox ports are shown by command "nvidia-smi topo -m" after dual-port NICs are bonded.
• When polling the H100 GPU via SMBPBI using GPU Performance Monitoring metrics, driver reloads or GPU resets can result in driver errors that manifest as PID (X62) errors on Linux. NVIDIA is investigating this issue and more information will be updated here soon.
• On NVIDIA H800, monitoring software such as DCGM or NVML might report lower double-precision (FP64) utilization metrics. This is expected as per the NVIDIA H800 product configuration. Refer to the NVIDIA H800 product brief for more details.
• For some SKUs of GH100 the MIG profile name reported by cuDeviceGetName, particularly the number of compute instances, might be incorrect. Use nvidia-smi to query the actual loaded MIG profile names. Only cuDeviceGetName is affected; developers are recommended to query the precise SM information for precise configuration. This will be fixed in a subsequent driver release.
• "Change ECC State" and "Enable Error Correction Code" do not change synchronously when ECC state changes.
• The GPU driver build system might not pick the Module.symvers file, produced when building the ofa_kernel module from MLNX_OFED, from the right subdirectory. Because of that, nvidia_peermem.ko does not have the right kernel symbol versions for the APIs exported by the IB core driver, and therefore it does not load correctly. That happens when using MLNX_OFED 5.5 or newer on a Linux Arm64 or ppc64le platform.
  To work around this issue, perform the following:

  1. Verify that nvidia_peermem.ko does not load correctly.
  2. Uninstall old MLNX_OFED if one was installed.
  3. Manually remove /usr/src/ofa_kernel/default if one exists.
  4. Install MLNX_OFED 5.5 or newer.
  5. Manually create a soft link:

      /usr/src/ofa_kernel/default -> /usr/src/ofa_kernel/$(uname -m)/$(uname -r)

  6. Reinstall the GPU driver.
• If you encounter an error on RHEL7 when installing with cuda-drivers-fabricmanager packages, use the following alternate instructions. For example:

  If you are upgrading from a different branch, for example to driver 515.65.01:

  new_version=515.65.01
  sudo yum swap nvidia-driver-latest-dkms nvidia-driver-latest-dkms-${new_version}
  sudo yum install nvidia-fabric-manager-${new_version}

• When installing a driver on SLES15 or openSUSE15 that previously had an R515 driver installed, users need to run the following command afterwards to finalize the installation:

  sudo zypper install --force nvidia-gfxG05-kmp-default 

  Without doing this, users may see the kernel objects as missing.

• nvidia-release-upgrade

  may report that not all updates have been installed and exit.
  When running the

  nvidia-release-upgrade

  command on DGX systems running DGX OS 4.99.x, it may exit and tell users: "Please install all available updates for your release before upgrading" even though all upgrades have been installed.
  Users who see this can run the following command:

  sudo apt install -y nvidia-fabricmanager-450/bionic-updates --allow-downgrades

  After running this, proceed with the regular upgrade steps:

  sudo apt update
  sudo apt full-upgrade -y 
  sudo apt install -y nvidia-release-upgrade 
  sudo nvidia-release-upgrade

• By default, Fabric Manager runs as a systemd service. If using

  DAEMONIZE=0

  in the Fabric Manager configuration file, then the following steps may be required.
  1. Disable FM service from auto starting.

     systemctl disable nvidia-fabricmanager

  2. Once the system is booted, manually start FM process.

     /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

     Note, since the process is not a daemon, the SSH/Shell prompt will not be returned (use another SSH shell for other activities or run FM as a background task).
• Important correctness fix for H100 GPU instructions used by cuBLAS, other CUDA libraries, and user CUDA code

  An issue was discovered recently with H100 GPUs (H100 PCIe and HGX H100) where certain operations put the GPU in an invalid state that allowed some GPU instructions to operate at unsupported frequency that can result in incorrect computation results and faster than expected performance. The affected GPU instructions are used by cuBLAS, other CUDA libraries, and can also be used for user CUDA code.

  The operations that allow the GPU to enter an invalid state are the following:

  • Enabling MIG
  • Deinitialize and reinitialize the GPU (for example, turn off persistence mode and turn it back on or reload the nvidia.ko driver)
  • Any Compute Engine error (for example, MMU fault, Out of Range warp error, and so on)
  Once the GPU enters the invalid state, the performance for some GPU instructions is increased by 7-10%, but the computation results may be incorrect.

  The current release fixes this issue, and it is no longer possible to enter the invalid GPU state. This issue has been present in all drivers since the H100 launch, and we recommend that you upgrade to the current release as soon as possible. If upgrading is not immediately possible, a GPU reset can restore the GPU back to the correct operational state, except for when MIG is being used. For MIG, the new driver is required, and there is no workaround available.

• Uninstalling the driver fails, and the system reboots automatically.
  On Windows 2019 and 2022 servers, uninstalling the driver causes the system to restart automatically before the uninstallation is completed. The issue also occurs when you upgrade the driver from an older version to a new version, even after selecting the Perform Clean Installation option in the installer UI.

  Note: This issue does not occur in Linux.

  Workaround

  We strongly recommend that you always install, uninstall, and upgrade drivers from Safe mode.

• In Shared Switch virtualization mode, the guest VM GPU driver load and unload stress test fails after certain iteration

  In the Shared Switch virtualization mode, the stress test to load and unload the GPU driver on Guest VM in every 30 second interval runs into issues approximately after three hours of the test.

  Workaround

  Do not run the stress reload driver cycle at this time.

• A few Async SMBPBI commands do not function as intended when the driver is unloaded.

  When the driver is unloaded, the following Async SMBPBI commands do not operate as specified:

  • Arg1 0x00: Reads total GPU power limit control data.
  • Arg1 0x01: Sets the total GPU power limit.
  • Arg1 0x02: Reads the total GPU power limit policy information.
  Due to this issue, some properties of the following Redfish URIs are impacted:

  • PowerLimitWatts.SetPoint:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]/EnvironmentMetrics

  • SpeedLimitMHz, SpeedLocked:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]

  The Patch operation of the following URIs are impacted:

  • PowerLimitWatts.SetPoint:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]/EnvironmentMetrics

  • Oem.Nvidia.PowerMode "MaxP" or "MaxQ":

    /redfish/v1/Chassis/HGX_Chassis_0/EnvironmentMetrics

  • SpeedLimitMHz, SpeedLocked:

    /redfish/v1/Systems/HGX_Baseboard_0/Processors/GPU_SXM_[1-8]

  Workaround

  Load the driver for these URIs to work properly.

• Fabric Manager state is not reported accurately on NVSwitch OOB query

  The NVSwitch SMPBI query that reports Fabric Manager state (Manager State) is not reporting the actual FM state.

• Instructions to reset all GPUs Using the nvidia-smi -r Command

  When resetting all GPUs using the nvidia-smi command with the -r option instead of a resetting specific GPU using the -i <gpu_index> option, all the NVSwitches will also be reset. This process wipes out the NVSwitch routing entries, and subsequent CUDA application launches will fail. The Fabric Manager service will also show interaction errors with the NVSwitch device via the switch driver.

  Workaround

  1. Stop the Fabric Manager service.
  2. To reset all GPUs, run nvidia-smi -r.
  3. After the reset is finished, start the Fabric Manager service.

GPU Performance Counters

The use of developer tools from NVIDIA that access various performance counters requires administrator privileges. See this note for more details. For example, reading NVLink utilization metrics from nvidia-smi (nvidia-smi nvlink -g 0) would require administrator privileges.

NoScanout Mode

NoScanout mode is no longer supported on NVIDIA Data Center GPU products. If NoScanout mode was previously used, then the following line in the “screen” section of /etc/X11/xorg.conf should be removed to ensure that X server starts on data center products:

Option         "UseDisplayDevice" "None"

NVIDIA Data Center GPU products now support one display of up to 4K resolution.

Unified Memory Support

CUDA and unified memory is not supported when used with Linux power management states S3/S4.

IMPU FRU for Volta GPUs

OpenCL 3.0 Known Issues

Device side enqueue

• Device-Side-Enqueue related queries may return 0 values, although corresponding built-ins can be safely used by kernel. This is in accordance with conformance requirements described at https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_API.html#opencl-3.0-backwardscompatibility

• Shared virtual memory - the current implementation of shared virtual memory is limited to 64-bit platforms only.

NVML