General

• Added support for the following NVIDIA GPU products:

  • NVIDIA A100X
  • NVIDIA A30X
  • NVIDIA A10M
• OpenCL Vulkan Interop

  New external memory and semaphore sharing extensions provide a generic framework that enables OpenCL to import external memory and semaphore handles to synchronize with the external runtime, coordinating the use of shared memory. Refer to “NVIDIA OpenCL Vulkan Interop” on page 4 for details.

• NVIDIA OpenCL Compiler Upgrade

  The embedded OpenCL Just-In-Time compiler will offer an opt-in version utilizing CLANG 7.0 and NVVM 7.0 components, providing support for 16-bit floating point and 128-bit integer data types. Refer to “NVIDIA OpenCL Compiler Upgrade” on page 5 for details.

OpenCL Vulkan Interop

Starting with R510 driver version 511.65, the NVIDIA OpenCL driver has added support for the following new provisional extension specifications released by Khronos. The specifications are for OpenCL external semaphore and external memory.

• https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_Ext.html#cl_khr_semaphore
• https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_Ext.html#cl_khr_external_semaphore
• https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_Ext.html#cl_khr_external_memory

NVIDIA is seeking developer feedback on this new extension support.

OpenCL External semaphore and memory extensions

The set of new External Memory and Semaphore Sharing extensions provides a generic framework that enables OpenCL to import external memory and semaphore handles exported by external APIs—using a methodology that will be familiar to Vulkan developers—and then use those semaphores to synchronize with the external runtime, coordinating the use of shared memory.

The following key features are supported as part of these extensions:

• Importing memory into buffers using FD, Win32 KMT and NT handles
• Importing memory into images using FD, Win32 KMT and NT handles
• Importing binary semaphores using FD, Win32 KMT and NT handles
• Synchronizing using Wait and Signal on imported semaphores
• Using buffers and images imported in OpenCL kernels and other APIs such as other regular cl_mem.

Limitations of the Current Implementation

• Support for importing external memory and semaphores using FD handles on Linux and Win32 NT and KMT handles on Windows. No other handle types are currently available.
• Support for binary semaphores only.
• No support for exporting semaphore or memory from OpenCL.
• clEnqueueAcquireExternalMemObjectsKHR and clEnqueueReleaseExternalMemObjectsKHR APIs are currently not required as execution hand-off can be managed through semaphore wait and signal. But, these may be required in the future for correct functionality.

NVIDIA OpenCL Compiler Upgrade

The NVIDIA OpenCL driver uses an embedded OpenCL Just-In-Time (JIT) compiler based on the legacy 3.4 versions of the NVVM optimizer component and CLANG compiler front-end. NVIDIA plans to upgrade its OpenCL JIT compiler to use version 7.0 of the CLANG front-end and NVVM optimizer component.

NVIDIA is releasing this new OpenCL compiler as an opt-in feature in this driver release, with the default OpenCL compiler remaining the same. In a subsequent driver release, the default compiler will be switched to use the new components, replacing the old compiler. This approach provides an opportunity to solicit feedback from OpenCL customers and work to ensure this feature meets functional and performance expectations for key use-cases prior to the final release.

Enabling the OpenCL NVVM 7.0 Compiler

As part of this driver install, a new compiler library should be visible in the system folder as “libnvidia-compiler-next” along with the old “libnvidia-compiler”. The driver by default will pick the old compiler unless the following environment variable is set: “NVCL_USE_NVVM70_COMPILER=1”

NVVM 7.0 New Compiler Features

The new NVVM 7.0 based compiler takes advantage of years of development in the Clang+LLVM framework. In addition to several minor bug fixes and diagnostic improvements, this compiler introduces the following noteworthy features:

• 16-bit floating point (half) type

  16-bit floating point types or “half” type is available as a native data type in the new compiler.

  This type is enabled by the “cl_khr_fp16” feature guard pragma

  Example:

  #pragma OPENCL EXTENSION cl_khr_fp16 : enable
  
  half scalar_arith(half n, half k) {
    half w = n + k; 8
    half x = n - k; -1
    half y = w * x; -8
    half z = y / x; 8
    return -z;
  }
  __kernel void foo(__global int* x) {
    half a = 3.5H, b = 4.5H;
    if (scalar_arith(a, b) == -8.0)
    	*x = 1;
    return;
  }

• 128-bit integer type

  128-bit integer types or “(un)signed long long” is available as a native data type in the new compiler. This type is enabled by default and does not require any macros to be defined

  Example:

  typedef unsigned long long ULL;
  typedef long long LL;
  
  LL scalar_arith(ULL n, ULL k) {
    LL w = n + k;
    LL x = n - k;
    LL y = w * x;
    LL z = y / x;
    return -z;
  }
  
  __kernel void foo(__global int* x) {
    ULL a = 0x123456789ABCDEF0ULL;
    ULL b = 0xFEDCBA9876543210ULL;
    if (scalar_arith(a, b) < 0)
    	*x = 1;
    return;
  }

• Upgraded math libraries

  The built-in standard math functions (e.g. sin(), cos()) have been upgraded to be on par with CUDA C++. This ensures that your application benefits from high-performance math routines optimized for the latest GPU architectures

Known Issues with NVVM 7.0-based Compiler

The new Clang/NVVM 7.0 based compiler has stricter error checking compared to the previous compiler. The following use-cases which were allowed with the older compiler may now throw an error.

• Updating const variables after they have been assigned
• Using address spaces other than global for kernel pointer parameters
• Using variadic arguments in functions and blocks

General

• nvidia-release-upgrade may report that not all updates have been installed and exit.

  When running the nvidia-release-upgrade command on DGX systems running DGX OS 4.99.x, it may exit and tell users: "Please install all available updates for your release before upgrading" even though all upgrades have been installed.

  Users who see this can run the following command:

  sudo apt install -y nvidia-fabricmanager-450/bionic-updates --allow-downgrades

  After running this, proceed with the regular upgrade steps:

  sudo apt update
  sudo apt full-upgrade -y 
  sudo apt install -y nvidia-release-upgrade 
  sudo nvidia-release-upgrade

• By default, Fabric Manager runs as a systemd service. If using DAEMONIZE=0 in the Fabric Manager configuration file, then the following steps may be required.

  1. Disable FM service from auto starting. (systemctl disable nvidia-fabricmanager)

  2. Once the system is booted, manually start FM process. (/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg). Note, since the process is not a daemon, the SSH/Shell prompt will not be returned (use another SSH shell for other activities or run FM as a background task).

• On NVSwitch systems with Windows Server 2019 in shared NVSwitch virtualization mode, the host may hang or crash when a GPU is disabled in the guest VM. This issue is under investigation.

GPU Performance Counters

The use of developer tools from NVIDIA that access various performance counters requires administrator privileges. See this note for more details. For example, reading NVLink utilization metrics from nvidia-smi (nvidia-smi nvlink -g 0) would require administrator privileges.

NoScanout Mode

NoScanout mode is no longer supported on NVIDIA Data Center GPU products. If NoScanout mode was previously used, then the following line in the “screen” section of /etc/X11/xorg.conf should be removed to ensure that X server starts on data center products:

Option         "UseDisplayDevice" "None"

NVIDIA Data Center GPU products now support one display of up to 4K resolution.

Unified Memory Support

Some Unified Memory APIs (for example, CPU page faults) are not supported on Windows in this version of the driver. Review the CUDA Programming Guide on the system requirements for Unified Memory

CUDA and unified memory is not supported when used with Linux power management states S3/S4.

IMPU FRU for Volta GPUs

OpenCL 3.0 Known Issues

• Device-Side-Enqueue related queries may return 0 values, although corresponding built-ins can be safely used by kernel. This is in accordance with conformance requirements described at https://www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_API.html#opencl-3.0-backwardscompatibility
• Shared virtual memory - the current implementation of shared virtual memory is limited to 64-bit platforms only.