Skip to navigation

The NVIDIA HPC SDK

View as Markdown

The NVIDIA HPC software development kit (https://developer.nvidia.com/hpc-sdk) is a suite of compilers, libraries, and other tools for HPC.

Features include:

  • The NVIDIA HPC SDK C, C++, and Fortran compilers that support GPU acceleration of HPC modeling and simulation applications with standard C++ and Fortran, OpenACC directives, and CUDA.

  • GPU-accelerated math libraries that maximize performance on common HPC algorithms, and optimized communications libraries enable standards-based multi-GPU and scalable systems programming.

  • Performance profiling and debugging tools that simplify porting and optimization of HPC applications

  • Containerization tools that enable easy deployment on-premises or in the cloud.

  • Support for ARM, OpenPOWER, x86-64 CPUs, as well as NVIDIA GPUs, running Linux

8.1 Packages And Versions

The main NVIDIA HPC SDK package is available as a package cm-nvhpc. The cm-nvhpc package, in turn, depends on a package with the naming format cm-nvhpc-cuda <number>. Here, <number> is a CUDA version number.

The latest CUDA version number that is compatible with the cm-nvhpc package version at the time of release is set as the package dependency. Older CUDA packages are optional but can also be installed.

A command line way to check the release versions and availability is as indicated by the following outputs for a Rocky Linux 9.3 cluster:

Example

[root@basecm11 ~]# yum info cm-nvhpc | grep ^Version
Version      : 25.3
[root@basecm11 ~]# yum install cm-nvhpc
...
Installing dependencies:
 cm-nvhpc-cuda12.8
..
[root@basecm11 ~]# yum search cm-nvhpc-cuda*
...
cm-nvhpc-cuda11.8.x86_64 : NVIDIA HPC SDK
cm-nvhpc-cuda12.6.x86_64 : NVIDIA HPC SDK
cm-nvhpc-cuda12.8.x86_64 : NVIDIA HPC SDK

The preceding output is what was available at the time of writing (June 2025). The output can be expected to change.

A browser-based way to check the cm-nvhpc versions and CUDA availability situation for BCM versions, distributions and architecture is to use cm-nvhpc as a string in the distributed packages list for BCM at https://service.bcm.nvidia.com/packages-dashboard

8.2 Compiler Modules

The cm-nvhpc package makes several environment modules available for compiling:

[root@basecm11 ~]# module avail | grep -o -P 'nvhpc.*?/'
nvhpc-hpcx-cuda11/
nvhpc/
nvhpc-hpcx-cuda12/
nvhpc-byo-compiler/
nvhpc-hpcx/
nvhpc-hpcx-2.20-cuda12/
nvhpc-nompi/

The nvhpc-hpcx-cuda11 or nvhpc-hpcx-cuda12 environment module sets up the HPC-X library environment with the selected CUDA version.

The nvhpc-hpcx environment module sets up the HPC-X library environment. This is an alternative to the OpenMPI 3.x library.

The nompi tag implies that paths to the MPI binaries and MPI libraries that come with cm-nvhpc are not set, so that no MPI library is used from the package. An external MPI library can then be used with the nvhpc-nompi compiler.

The nvhpc environment module is the standard HPC SDK.

The byo tag is an abbreviation for ’bring-your-own’, and means that the general compiler environment for C, C++ and Fortran are not set.

8.3 Viewing Installed Available CUDA Versions, And The Running CUDA Version

The installed available CUDA versions for nvhpc can be viewed with:

Example

[root@basecm11] ~# module load shared nvhpc
[root@basecm11] ~# basename -a $(ls -1d $NVHPC_ROOT/cuda/[0-9]*)
12.6
12.8

The running CUDA version for nvhpc can be viewed with:

Example

[root@basecm11 ~]# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

8.4 Changing The Running CUDA Version

In the preceding example, CUDA version 12.8 is the default version, while CUDA version 12.6 is also seen to be available.

The CUDA version that is used can be changed on the nodes where nvhpc and the CUDA versions have been installed.

For example, the version can be changed to version 12.6, as follows, for:

  • nvhpc cluster-wide:

    [root basecm11] # makelocalrc ${NVHPC_ROOT}/compilers/bin -cuda 12.6 -x

  • nvhpc on a specific head or compute node, as specified by hostname -s:

    [root basecm11] # makelocalrc ${NVHPC_ROOT}/compilers/bin -cuda 12.6 -o > \
        ${NVHPC_ROOT}/compilers/bin/localrc.$(hostname -s)

  • nvhpc for a specific user on a specific head or compute node, as specified by hostname -s:

    [root basecm11] # makelocalrc ${NVHPC_ROOT}/compilers/bin -cuda 12.6 -o > \
        ${HOME}/localrc.$(hostname -s)

If the nvhpc compiler is run then:

  1. the ${NVHPC_ROOT}/compilers/bin/localrc configuration file is read,

  2. followed by a second configuration file, which—if it exists—is either:

    • the ${NVHPC_ROOT}/compilers/bin/localrc.$(hostname -s) configuration file
      or

    • the ${HOME}/localrc.$(hostname -s) configuration file

The second configuration file overwrites any settings set with ${NVHPC_ROOT}/compilers/bin/localrc

If the ${NVHPC_ROOT}/compilers/bin/localrc.$(hostname -s) configuration file exists, then a ${HOME}/localrc.$(hostname -s) is ignored.