Skip to navigation

CUDA For GPUs

View as Markdown

The optional CUDA packages should be deployed in order to take advantage of the computational capabilities of NVIDIA GPUs. The packages may already be in place, and ready for deployment on the cluster, depending on the particular BCM software that was obtained. If the CUDA packages are not in place, then they can be picked up from the BCM repositories, or a local mirror.

9.1 Installing CUDA

9.1.1 CUDA Packages Available

At the time of writing of this section (April 2026), the CUDA 13.1 package is the most recent version available in the BCM YUM, zypper, and APT repositories. The available versions are updated typically in the next point release version of NVIDIA Base Command Manager after the upstream changes are made available. An up-to-date list of packages available for a particular distribution and architecture can be viewed at https://service.bcm.nvidia.com/packages-dashboard/.

Thus, for example, at the time of writing, Rocky Linux 9.6 has many CUDA versions available, in the range from CUDA 11.7 to CUDA 13.1 for the x86_64 architecture, while Ubuntu 24.04 just has CUDA 12.5, CUDA 12.6, CUDA 12.8, CUDA 12.9, CUDA-13.0, and CUDA 13.1 for x86_64.

9.1.1.1 Example: CUDA Packages Available For Ubuntu 24.04

At the time of writing of this section (April 2026), a cluster administrator can manage these packages for Ubuntu 24.04:

PackageTypeDescription
cuda12.5-toolkitsharedCUDA math libraries and utilities
cuda12.6-toolkitsharedCUDA math libraries and utilities
cuda12.8-toolkitsharedCUDA math libraries and utilities
cuda12.9-toolkitsharedCUDA math libraries and utilities
cuda13.0-toolkitsharedCUDA math libraries and utilities
cuda13.1-toolkitsharedCUDA math libraries and utilities
cuda12.5-visual-toolssharedCUDA visual toolkit
cuda12.6-visual-toolssharedCUDA visual toolkit
cuda12.8-visual-toolssharedCUDA visual toolkit
cuda12.9-visual-toolssharedCUDA visual toolkit
cuda13.0-visual-toolssharedCUDA visual toolkit
cuda13.1-visual-toolssharedCUDA visual toolkit
cuda12.5-sdksharedCUDA software development kit
cuda12.6-sdksharedCUDA software development kit
cuda12.8-sdksharedCUDA software development kit
cuda12.9-sdksharedCUDA software development kit
cuda13.0-sdksharedCUDA software development kit
cuda13.1-sdksharedCUDA software development kit
nvidia-driver**, †localCUDA GPU driver and libraries.
cuda-dcgmlocalCUDA Data Center GPU Manager (DCGM).
This includes the dcgmi CLI tool.
nvidia-fabricmanager,*localFabric Manager tools
nvidia-fabricmanager-dev,*localFabric Manager tools

* optional, intended only for hardware containing an NVSwitch, such as a DGX system

** installed by default on software images, not on head nodes

† the driver typically has a version number tag such as -590.


  • The packages of type shared in the preceding table should be installed on the head nodes of a cluster using CUDA-compatible GPUs.

  • The packages of type local should be installed to all nodes that access the GPUs. In most cases this means that the

    • nvidia-driver (NVIDIA driver metapackage)

    • cuda-dcgm (NVIDIA Data Center GPU Manager)

    • nvidia-fabricmanager (Fabric Manager for NVSwitch based systems)

    • nvidia-fabricmanager-dev (Fabric Manager API headers and associated library)

    packages should be installed in a software image (section 2.1.2 of the Administrator Manual). By default, BCM software images already have these packages, which means that an explicit installation by the cluster administrator should not be required.

    If a head node also accesses GPUs, then the nvidia-driver and cuda-dcgm packages should be installed on it, too.

For packages of type shared, the particular CUDA version that is run on the node can be selected via a modules environment command. For example, to select CUDA SDK version 13.0, the user can run:

Example

user@basecm11:~$ module add shared cuda13.0

Release notes and features list for the CUDA packages:

CUDA packages that the cluster administrator normally does not manage: As an aside, there are also the additional CUDA DCGM packages:

PackageTypeDescriptionInstallation on GPU node
cuda-dcgm-libslocalNVIDIA DCGM librariesinstalled by default
cuda-dcgm-devellocalNVIDIA DCGM development filesnot installed by default
cuda-dcgm-nvvslocalNVIDIA DCGM validation suiteinstalled by default, except for in RHEL8, and except for in a DGX node

At the time of writing of this paragraph (April 2026) the latest DCGM version supported by BCM was 4.5.2. Running a

  • dnf info cuda-dcgm; dnf list cuda-dcgm for RHEL-family distributions

  • apt show -a cuda-dcgm for Ubuntu distributions

should reveal the currently installed and available versions for the cluster.

9.1.1.2 CUDA Package That The Cluster Administrator May Wish To Install For CUDA Programming

CUB is a CUDA programming library that developers may wish to access. It is available for some distributions for some older CUDA versions, and provided by the package cm-cub-cuda.

9.1.1.3 CUDA Package Installation Basic Sanity Check

  • The NVIDIA GPU hardware should be detectable by the kernel, otherwise the GPUs cannot be used by the drivers. Running the lspci command on the device with the GPU before the CUDA package driver installation is a quick check that should make it clear if the NVIDIA hardware is detected in the first place:

    Example

    running lspci on node001. Here 4 GPUs are detected:

    [root@node001]# { lspci | grep '3D controller: NVIDIA';} && echo "NVIDIA GPU hardware detected by kernel"
    0008:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1)
    0009:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1)
    0018:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1)
    0019:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1)
    NVIDIA GPU hardware detected by kernel
    [root@node001]#

    If the hardware is not detected by the kernel already, then the administrator should reassess the situation.

    The Device 2941 part of the string can be tallied against the identifiers in the README.txt.gz file of the nvidia-driver-580 package series, at the node where it has been installed.

    Example

    root@basecm11:~# zgrep 2941 /cm/images/default-image/usr/share/doc/nvidia-driver-580/README.txt.gz
     NVIDIA GB200                                              2941 10DE 2046  K
     NVIDIA GB200                                              2941 10DE 20CA  K
     NVIDIA GB200                                              2941 10DE 20D5  K
     NVIDIA GB200                                              2941 10DE 21C9  K
     NVIDIA GB200                                              2941 10DE 21CA  K

    The output of the lspci command can be more directly informative for some hardware:

    Example

    root@basecm11:~# ssh node001 "lspci | grep 3D controller: NVIDIA"
    00:06.0 3D controller: NVIDIA Corporation GA100 [A100 PCIe 40GB] (rev a1)
    root@basecm11:~# zgrep A100 /cm/images/default-image/usr/share/doc/nvidia-driver-580/README.txt.gz \
    | grep -i PCIE | grep 40GB
     NVIDIA A100-PCIE-40GB                                     20F1 10DE 145F  J

  • Only after CUDA package installation has taken place, and after rebooting the node with the GPU, and allowing CMDaemon some time (a few minutes) to detect the change, are GPU details visible using the sysinfo command:

    Example

    running sysinfo on node001, which is where the GPUs are, via cmsh on the head node, while cuda-dcgm is not yet ready:

    [root@basecm11 ~]# cmsh
    [basecm11]% device use node001
    [basecm11->device[node001]]% sysinfo | grep GPU

    Example

    running sysinfo on node001, via cmsh on the head node, after cuda-dcgm is ready. 4 GPUs are detected

    [basecm11->device[node001]]% sysinfo | grep GPU
    [maple->device[dgx-gb200-n01-c1]]% sysinfo  | grep GPU
    Motherboard Name          GB200 1CPU:2GPU Board PC
    Number of GPUs            4
    GPU[0-3] name             NVIDIA Graphics Device
    GPU[0-3] power limit      1.20KW
    GPU[0-3] BIOS             97.00.6C.00.03
    GPU[0-3] driver version   570.82
    GPU[0-3] nvlink           2 up

9.1.1.4 CUDA Package Installation Guidelines For Compilation With Login Nodes

  • CUDA compilation should take place on a node that uses NVIDIA GPUs during compilation.

    • Using a workload manager to allocate this task to GPU nodes is recommended.

  • Cross compilation of CUDA software with a CPU is generally not a best practice due to resource consumption, which can even lead to crashes.

    • If, despite this, cross compilation with a CPU is done, then the nvidia-driver package should be installed on the node on which the compilation is done, and the GPU-related services on the node, such as:

      • nvidia-persistenced.service

      • nvidia-fabricmanager.service

      • nvidia-dcgm.service

      should be disabled.

9.1.2 CUDA Package And Modules

The CUDA packages require access to the BCM repositories and to the distribution repositories.

The nvidia-driver packages replace the older cuda-driver packages that were used in versions of BCM prior to version 11. The newer packages do not compile the driver modules source at boot time, but during package installation. The newer packages are also open source.

Example

For example, on a cluster where (some of) the nodes access GPUs, but the head node does not access a GPU, the following commands install the CUDA 13.0 packages on the head using APT:

root@mycluster:~# apt install cuda13.0-toolkit cuda13.0-sdk cuda13.0-visual-tools

The nvidia-driver-, cuda-dcgm, cuda-dcgm-nvvs, and nvidia-fabricmanager are already installed by default on the default software image, which is default-image on a newly-installed default cluster.

Carrying out an explicit installation or imageupdate for these is therefore normally not needed.

In cases where it is required, then manual installation into a software image is described in section 9.4 of the Administrator Manual, while running the imageupdate command (section 5.6 of the Administrator Manual), for example on nodes 1 to 10, can be carried out with:

Example

[root@mycluster ~]# cmsh
[mycluster]% device
[mycluster->device]% imageupdate -n node0[01-10] -w

If the CUDA packages have been set up on the cluster, then the end user typically just switches between CUDA environments with a module change (section 2.2 of the Administrator Manual and section 2.3 of the User Manual).

Example

module purge && module load shared cuda13.0/toolkit

9.2 Verifying CUDA

The cuda*-sdk packages provide CUDA SDK sources that can be used to compile libraries and tools that are not part of the CUDA toolkit, but used by CUDA software developers, such as the deviceQuery binary (section 9.2).

An extensive method to verify that CUDA is working is to run the verify_cuda <version>.sh script, located in the CUDA SDK directory. The value of <version> can be 13.0, for example.

This script first copies the CUDA SDK source to a local directory under /tmp or /local. It then builds CUDA test binaries. To build the makefiles, the compilation may need cmake, installed from the cm-make package into the image (section 9.4 of the Administrator Manual).

These binaries clutter up the disk and are not intended for use as regular tools, so the administrator is urged to remove them after they are built and run.

A help text showing available script options is displayed when “verify_cuda <version>.sh -h” is run.

A typical run output is as follows (some output elided):

Example

root@node001:~# module load shared cuda13.0/toolkit
root@node001:~# cd $CUDA_SDK
root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2# ./verify_cuda13.0.sh
Copy cuda13.0 sdk files to "/tmp/cuda13.0" directory.
cmake
make (may take a while)
Run multiple tests non-interactive ? (y/N)?
Executing: /tmp/cuda13.0/build/CMakeFiles/3.26.3/CMakeDetermineCompilerABI_CUDA.bin
80cmd: /tmp/cuda13.0/build/CMakeFiles/3.26.3/CMakeDetermineCompilerABI_CUDA.bin - rc : 0
Executing: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_C.bin
INFO:OpenMP-date[201511]
cmd: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_C.bin - rc : 0
Executing: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_CXX.bin
INFO:OpenMP-date[201511]
cmd: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_CXX.bin - rc : 0
Executing: /tmp/cuda13.0/build/Samples/0_Introduction/clock/clock
...
Test passed
cmd: /tmp/cuda13.0/build/Samples/6_Performance/cudaGraphsPerfScaling/cudaGraphsPerfScaling - rc : 0
All cuda13.0 compiled test programs can be found in the "/tmp/cuda13.0/build/" directory
Alternatively the "run_tests.py" can be run in the "/tmp/cuda13.0" directory.
For example:
  cd /tmp/cuda13.0; python3 run_tests.py --output ./test --dir ./build/Samples --config test_args.json
The "/tmp/cuda13.0" directory may take up a lot of diskspace.
Use "rm -rf /tmp/cuda13.0" to remove the data.

One of the binaries compiled is the deviceQuery command. This can be run on a node accessing one or more GPUs. The deviceQuery command lists all CUDA-capable GPUs that a device can access, along with several of their properties (some output elided). It can be run in CUDA 13 as follows:

root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2# /tmp/cuda13.0/Samples/1_Utilities/deviceQuery/deviceQuery
/tmp/cuda13.0/Samples/1_Utilities/deviceQuery/deviceQuery Starting...
 CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "NVIDIA A100-PCIE-40GB"
  CUDA Driver Version / Runtime Version          13.0 / 13.0
  CUDA Capability Major/Minor version number:    8.0
  Total amount of global memory:                 40441 MBytes (42405855232 bytes)
  (108) Multiprocessors, (064) CUDA Cores/MP:    6912 CUDA Cores
  GPU Max Clock rate:                            1410 MHz (1.41 GHz)
  Memory Clock rate:                             1215 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                 41943040 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        167936 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 6
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.0, CUDA Runtime Version = 13.0, NumDevs = 1
Result = PASS
root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2#

The query can detect if there is more than 1 GPU, and display output for each GPU that it finds.

The CUDA programming guide (https://docs.nvidia.com/cuda/cuda-programming-guide/) has further information on how to run compute jobs using CUDA.

9.2.0.1 Further Information On CUDA Verification

More on verification can be found in the NVIDIA CUDA INSTALLATION GUIDE FOR LINUX at https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf.

9.3 Verifying OpenCL

CUDA 12.8 contains an OpenCL compatible interface. To verify that OpenCL is working, or failing, the binaries can be compiled from the provided OpenCL source and run. For example:

Example

root@node001:~# module load shared cuda12.8/toolkit
root@node001 ~# cd $CUDA_SDK
root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1# cd opencl/OpenCL/ ; ls
bin  common  doc  Makefile  releaseNotesData  Samples.html  src
root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL# make
a - obj/release/oclUtils.cpp.o
make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/common'
make[1]: Entering directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/shared'
ar: `u' modifier ignored since `D' is the default (see `U')
r - obj/x86_64/release/shrUtils.cpp.o
r - obj/x86_64/release/rendercheckGL.cpp.o
r - obj/x86_64/release/cmd_arg_reader.cpp.o
make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/shared'
make[1]: Entering directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/src/oclBandwidthTest'
make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/src/oclBandwidthTest'
...

The binaries made using the Makefile can then be run from their directory:

root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL# cd bin/linux/release/
root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/bin/linux/release# ls
oclBandwidthTest
oclBandwidthTest.txt
oclBlackScholes
oclConvolutionSeparable
oclCopyComputeOverlap
oclDCT8x8
oclDeviceQuery
oclDotProduct
oclDXTCompression
oclFDTD3d
oclHiddenMarkovModel
oclHistogram
oclInlinePTX
oclMatrixMul
oclMatVecMul
oclMersenneTwister
oclMultiThreads
oclQuasirandomGenerator
oclRadixSort
oclReduction
oclScan
oclSimpleMultiGPU
oclSortingNetworks
oclTranspose
oclTridiagonal
oclVectorAdd
root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/bin/linux/release# for i in ocl*; do ./$i ; done
[oclBandwidthTest] starting...
./oclBandwidthTest Starting...
Running on...
Tesla V100-SXM3-32GB
Quick Mode
Host to Device Bandwidth, 1 Device(s), Paged memory, direct access
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     5528.1
Device to Host Bandwidth, 1 Device(s), Paged memory, direct access
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     3915.9
Device to Device Bandwidth, 1 Device(s)
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     724307.1
[oclBandwidthTest] test results...
PASSED
> exiting in 3 seconds: 3...2...1...done!

…And so on. The other binaries listed should also run and pass.…

9.4 NVIDIA Validation Suite (Package: cuda-dcgm-nvvs)

The NVIDIA Validation Suite (NVVS) runs diagnostics to validate the NVIDIA software components. Running the nvvs binary directly is deprecated. The diagnostics for CUDA 12.8 are run instead as part of the dcgmi diag command.

The package for NVVS is not installed by default. It is also not available for RHEL8 and derivatives.

If the package is not installed, then an attempt to run the dcgmi utility fails:

Example

[root@basecm11 ~] module load cuda-dcgm
[root@basecm11 ~]# dcgmi diag -r 1
Error: Diagnostic could not be run because the Tesla recommended driver is not being used.

The package can be installed with:

Example

[root@basecm11 ~] module load cuda-dcgm
[root@basecm11 ~] yum install cuda-dcgm-nvvs

After it is installed, the node on which the installation is done must be rebooted.

Running the diagnostic after the reboot should display output similar to:

Example

[root@basecm11 ~] module load cuda-dcgm
[root@basecm11 ~]# dcgmi diag -r 3
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 4.1.1                                          |
| Driver Version Detected   | 570.82                                         |
| GPU Device IDs Detected   | 2941, 2941, 2941, 2941                         |
| Number of CPUs Detected   | 2                                              |
|-----  Deployment  --------+------------------------------------------------|
| software                  | Pass                                           |
|                           | GPU0: Pass                                     |
|                           | GPU1: Pass                                     |
|                           | GPU2: Pass                                     |
|                           | GPU3: Pass                                     |
+-----  Hardware  ----------+------------------------------------------------+
| memory                    | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
| diagnostic                | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
| nvbandwidth               | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
+-----  Integration  -------+------------------------------------------------+
| pcie                      | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
+-----  Stress  ------------+------------------------------------------------+
| memory_bandwidth          | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
| targeted_stress           | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
| targeted_power            | Skip                                           |
|                           | GPU0: Skip                                     |
|                           | GPU1: Skip                                     |
|                           | GPU2: Skip                                     |
|                           | GPU3: Skip                                     |
+---------------------------+------------------------------------------------+

9.5 Further NVIDIA Configuration Via The Cluster Manager

After the installation and verification has been carried out for CUDA drivers, further configuration as part of CMDaemon management can be carried out, as explained in section 3.17.2 of the Administrator Manual.