1. What's New
Welcome to the 23.11 version of the NVIDIA HPC SDK, a comprehensive suite of compilers and libraries enabling developers to program the entire HPC platform, from the GPU foundation to the CPU and out through the interconnect. The 23.11 release of the HPC SDK includes new features as well as important functionality and performance improvements.
- The HPC Compilers have implemented the -gpu=unified option to support coherent memory on systems that use the Grace Hopper superchip. The -gpu=unified option also works on x86_64 systems with the HMM driver. The -gpu=unified option is supported for Standard Parallelism and OpenACC modes for both C++ and Fortran, and CUDA Fortran.
- Version 23.11 of nvfortran introduces support for Fortran 2018 C descriptors. Together with library functions that have standard C prototypes, this feature provides a means for defining and manipulating Fortran data objects from within a C function.
-
HPC-X is now the default MPI library for x86_64 and aarch64 architectures.
OpenMPI 4 is now the default for the ppc64le architecture.
- Previously, OpenMPI 3 was the default on all platforms, and it can be obtained by loading the nvhpc-openmpi3 environment module.
- The MPI wrappers in comm_libs/mpi/bin automatically detect the CUDA driver and select the matching MPI library from comm_libs/X.Y. Applications that require a full MPI directory hierarchy (e.g., bin, include, lib) should bypass the MPI wrappers by loading the nvhpc-hpcx-cuda11 or the nvhpc-hpcx-cuda12 environment module, depending on the installed CUDA driver version.
- Starting with version 23.11, code written using C++ standard language parallelism can be profiled using NVTX and can be visualized on nsys timelines when NVTX tracing is enabled. This feature can be disabled with the compiler option -DNVHPC_STDPAR_DISABLE_NVTX_RANGES.
- The HPC SDK ships with support for CUDA 11.8 and 12.3 libraries and tools. Support for other CUDA 11 and 12 versions is available using CUDA Compatibiility or directly using NVHPC_CUDA_HOME.
- C++20 coroutines are now supported by nvc++ for execution on the host CPU. This feature can be enabled with the -fcoroutines option.
2. Release Component Versions
The NVIDIA HPC SDK 23.11 release contains the following versions of each component:
| Linux_x86_64 | Linux_ppc64le | Linux_aarch64 | ||||
|---|---|---|---|---|---|---|
| CUDA 11.8 | CUDA 12.3 | CUDA 11.8 | CUDA 12.3 | CUDA 11.8 | CUDA 12.3 | |
| nvc++ | 23.11 | 23.11 | 23.11 | |||
| nvc | 23.11 | 23.11 | 23.11 | |||
| nvfortran | 23.11 | 23.11 | 23.11 | |||
| nvcc | 11.8.89 | 12.3.52 | 11.8.89 | 12.3.52 | 11.8.89 | 12.3.52 |
| NCCL | 2.18.5 | 2.18.5 | 2.18.5 | 2.18.5 | 2.18.5 | 2.18.5 |
| NVSHMEM | 2.10.1 | 2.10.1 | 2.10.1 | 2.10.1 | N/A | 2.10.1 |
| cuBLAS | 11.11.4.17 | 12.3.2.9 | 11.11.3.6 | 12.3.2.9 | 11.11.3.6 | 12.3.2.9 |
| cuFFT | 10.9.0.58 | 11.0.11.19 | 10.9.0.58 | 11.0.11.19 | 10.9.0.58 | 11.0.11.19 |
| cuFFTMp | 11.0.14 | 11.0.14 | 11.0.5 | N/A | N/A | 11.0.14 |
| cuRAND | 10.3.0.86 | 10.3.4.52 | 10.3.0.86 | 10.3.4.52 | 10.3.0.86 | 10.3.4.52 |
| cuSOLVER | 11.4.1.48 | 11.5.3.52 | 11.4.1.48 | 11.5.3.52 | 11.4.1.48 | 11.5.3.52 |
| cuSOLVERMp | 0.4.2 | N/A | N/A | N/A | 0.4.2 | N/A |
| cuSPARSE | 11.7.5.86 | 12.1.3.153 | 11.7.5.86 | 12.1.3.153 | 11.7.5.86 | 12.1.3.153 |
| cuTENSOR | 1.7.0 | 1.7.0 | 1.7.0 | 1.7.0 | 1.7.0 | 1.7.0 |
| Nsight Compute | 2023.3.0 | 2023.3.0 | 2023.3.0 | |||
| Nsight Systems | 2023.3.1 | 2023.3.1 | 2023.3.1 | |||
| OpenMPI | 3.1.5 | 3.1.5 | 3.1.5 | |||
| HPC-X | 2.14 | 2.16 | N/A | N/A | 2.14 | 2.16 |
| OpenBLAS | 0.3.23 | 0.3.23 | 0.3.23 | |||
| Scalapack | 2.2.0 | 2.2.0 | 2.2.0 | |||
| Thrust | 1.15.1 | 2.2.0 | 1.15.1 | 2.2.0 | 1.15.1 | 2.2.0 |
| CUB | 1.15.1 | 2.2.0 | 1.15.1 | 2.2.0 | 1.15.1 | 2.2.0 |
| libcu++ | 1.8.1 | 2.2.0 | 1.8.1 | 2.2.0 | 1.8.1 | 2.2.0 |
3. Supported Platforms
3.1. Platform Requirements for the HPC SDK
| Architecture | Linux Distributions | Minimum gcc/glibc Toolchain | Minimum CUDA Driver |
|---|---|---|---|
| x86_64 |
RHEL/CentOS 7.3 - 7.9 |
C99: 4.8 |
450.36.06 |
| ppc64le |
RHEL 7.3 - 7.7 |
C99: 4.8 |
450.36.06 |
| aarch64 |
RHEL/CentOS/Rocky 8.0 - 8.7 |
C99: 4.8 |
450.36.06 |
Programs generated by the HPC Compilers for x86_64 processors require a minimum of AVX instructions, which includes Sandy Bridge and newer CPUs from Intel, as well as Bulldozer and newer CPUs from AMD. POWER 8 and POWER 9 CPUs from the POWER architecture are supported. The HPC SDK includes support for v8.1+ Server Class Arm CPUs that meet the requirements appendix E specified in the SBSA 7.1 specification.
The HPC Compilers are compatible with gcc and g++ and use the GCC C and C++ libraries; the minimum compatible versions of GCC are listed in Table 2. The minimum system requirements for CUDA and NVIDIA Math Library requirements are available in the NVIDIA CUDA Toolkit documentation.
3.2. Supported CUDA Toolchain Versions
The NVIDIA HPC SDK uses elements of the CUDA toolchain when building programs for execution with NVIDIA GPUs. Every HPC SDK installation package puts the required CUDA components into an installation directory called [install-prefix]/[arch]/[nvhpc-version]/cuda.
An NVIDIA CUDA GPU device driver must be installed on a system with a GPU before you can run a program compiled for the GPU on that system. The NVIDIA HPC SDK does not contain CUDA Drivers. You must download and install the appropriate CUDA Driver from NVIDIA , including the CUDA Compatibility Platform if that is required.
The nvaccelinfo tool prints the CUDA Driver version in its output. You can use it to find out which version of the CUDA Driver is installed on your system.
- CUDA 11.8
- CUDA 12.3
4. Known Limitations
- Passing an internal procedure as an actual argument to a Fortran subprogram is supported by nvfortran provided that the dummy argument is declared as an interface block or as a procedure dummy argument. Nvfortran does not support internal procedures as actual arguments to dummy arguments declared external.
- nvfortran only supports the Fortran 2003 standard maximum of 7 dimensions for arrays (Fortran 2008 raised the standard maximum dimensions to 15). This limit is defined in the standard CFI_MAX_RANK macro in the ISO_Fortran_binding.h C header file.
- Section “15.5.2.4 Ordinary dummy variables”, constraint C1540 and Note 5 in the Fortran 2018 Standard allow Fortran compilers to avoid copy-in/copy-out argument passing provided that the actual and corresponding dummy arguments have the ASYNCHRONOUS/VOLATILE attribute, and the dummy arguments do not have the VALUE attribute. This feature is fully supported in nvfortran with BIND(C) interfaces (i.e., Fortran calling C). Copy-in/copy-out avoidance with asynchronous/volatile attributes may not be available in other cases with nvfortran.
- Some applications may see failures on Haswell and Broadwell with MKL version 2023.1.0 when running certain workloads with 4 or more OpenMP threads. The issue is resolved in MKL version 2023.2.0.
- cuSolverMp has two new dependencies on UCC and UCX libraries in the HPC-X directory. To execute a program linked against cuSolverMP, please use the “nvhpc-hpcx-cuda11” environment module for the HPC-X library, or set the environment variable LD_LIBRARY_PATH as follows: LD_LIBRARY_PATH=${NVHPCSDK_HOME}/comm_libs/11.8/hpcx/latest/ucc/lib:${NVHPCSDK_HOME}/comm_libs/11.8/hpcx/latest/ucx/lib:$LD_LIBRARY_PATH
- To use HPC-X, please use the provided environment module files or take care to source the hpcx-init.sh script: $ . /[install-path]/Linux_x86_64/dev/comm_libs/X.Y/hpcx/latest/hpcx-init.sh Then, run the hpcx_load function defined by this script: hpcx_load. These actions will set important environment variables that are needed when running HPC-X. The following warning from HPC-X while running an MPI job – “WARNING: Open MPI tried to bind a process but failed. This is a warning only; your job will continue, though performance may be degraded” – is a known issue, and may be suppressed as follows: export OMPI_MCA_hwloc_base_binding_policy=""
- Fortran derived type objects with zero-size derived type allocatable components that are used in sourced allocation or allocatable assignment may result in a runtime segmentation violation.
- When using -stdpar to accelerate C++ parallel algorithms, the algorithm calls cannot include virtual function calls or function calls through a function pointer, cannot use C++ exceptions, and must use random access iterators (raw pointers as iterators work best). When unified memory is not enabled, the algorithm calls can only dereference pointers that point to the heap. See the c++ parallel algorithms documentation for more details.
5. Deprecations and Changes
- The GNU extension macros linux and unix are no longer defined when in ANSI mode (e.g., -std=c++17 or -std=c99). If your code is compiled in ANSI mode and you rely on either of these macros, you will need to use one of the ANSI compliant macros __linux__ or __unix__.
- Arm (aarch64) only: The 23.9 version of nvfortran changes the calling/return sequence for Fortran complex functions to match GNU's gfortran convention. Prior to the 23.9 release, nvfortran functions returned complex values via the stack using a "hidden" pointer as the first parameter. Now, complex values are returned following the gforran convention via the floating-point registers. All libraries released with NVIDIA HPC SDK for Arm have been updated to follow the "gfortran" method. Users linking against Arm's performance libraries will need to use the "gcc" version instead of the "arm" version. All Fortran code, including libraries, that uses complex numbers must be recompiled when using nvfortran on Arm systems.
- Support for CUDA Fortran textures is deprecated in CUDA 11.0 and 11.8, and has been removed from CUDA 12. The 23.9 release is the last version of the HPC Compilers to include support for CUDA Fortran texture.
- The OpenMPI 3 library will be removed from the HPC SDK in a future release.
- The -Minfo=intensity option is no longer supported.
- The CUDA_HOME environment variable is ignored by the HPC Compilers. It is replaced by NVHPC_CUDA_HOME.
- The -Mipa option has been disabled starting with the 23.3 version of the HPC Compilers.
- The -ta=tesla, -Mcuda, -Mcudalib options for the HPC Compilers have been deprecated.
- Support for the RHEL 7-based operating systems will be removed in the HPC SDK version 23.7, corresponding with the upstream end-of-life (EOL).
- In an upcoming release the HPC SDK will bundle only CUDA 11.8 and the latest version of the CUDA 12.x series. Codepaths in the HPC Compilers that support CUDA versions older than 11.0 will no longer be tested or maintained.
- Support for the Ubuntu 18.04 operating system will be removed in the HPC SDK version 23.5, corresponding with the upstream end-of-life (EOL).
- cudaDeviceSynchronize() in CUDA Fortran has been deprecated, and support has been removed from device code. It is still supported in host code.
- Starting with the 21.11 version of the NVIDIA HPC SDK, the HPC-X package is no longer shipped as part of the packages made available for the POWER architecture.
- Starting with the 21.5 version of the NVIDIA HPC SDK, the -cuda option for NVC++ and NVFORTRAN no longer automatically links the NVIDIA GPU math libraries. Please refer to the -cudalib option.
- HPC Compiler support for the Kepler architecture of NVIDIA GPUs was deprecated starting with the 21.3 version of the NVIDIA HPC SDK.
Notices
Notice
ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, "MATERIALS") ARE BEING PROVIDED "AS IS." NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.
Information furnished is believed to be accurate and reliable. However, NVIDIA Corporation assumes no responsibility for the consequences of use of such information or for any infringement of patents or other rights of third parties that may result from its use. No license is granted by implication of otherwise under any patent rights of NVIDIA Corporation. Specifications mentioned in this publication are subject to change without notice. This publication supersedes and replaces all other information previously supplied. NVIDIA Corporation products are not authorized as critical components in life support devices or systems without express written approval of NVIDIA Corporation.
Trademarks
NVIDIA, the NVIDIA logo, CUDA, CUDA-X, GPUDirect, HPC SDK, NGC, NVIDIA Volta, NVIDIA DGX, NVIDIA Nsight, NVLink, NVSwitch, and Tesla are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.
