CUDA For GPUs

The optional CUDA packages should be deployed in order to take advantage of the computational capabilities of NVIDIA GPUs. The packages may already be in place, and ready for deployment on the cluster, depending on the particular BCM software that was obtained. If the CUDA packages are not in place, then they can be picked up from the BCM repositories, or a local mirror.

9.1 Installing CUDA

9.1.1 CUDA Packages Available

At the time of writing of this section (December 2025), the CUDA 13.0 package is the most recent version available in the BCM YUM, zypper, and APT repositories. An up-to-date list of packages available for a particular distribution and architecture can be viewed at https://service.bcm.nvidia.com/packages-dashboard/.

9.1.1.1 Example: CUDA Packages Available For Rocky Linux 9.3

At the time of writing of this section (December 2025), a cluster administrator can manage these packages for Rocky Linux 9.5:

* optional, intended only for hardware containing an NVSwitch, such as a DGX system

The packages of type shared in the preceding table should be installed on the head nodes of a cluster using CUDA-compatible GPUs. The packages of type local should be installed to all nodes that access the GPUs. In most cases this means that the cuda-driver and cuda-dcgm packages should be installed in a software image (section 2.1.2 of the Administrator Manual).

If a head node also accesses GPUs, then the cuda-driver and cuda-dcgm packages should be installed on it, too.

For packages of type shared, the particular CUDA version that is run on the node can be selected via a modules environment command:

Example

module add shared cuda12.1

Release notes and features list for the CUDA packages:

• Release notes for the NVIDIA toolkit can be found at https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html.

• The features list for the NVIDIA toolkit can be found at https://docs.nvidia.com/cuda/cuda-features-archive/index.html.

CUDA packages that the cluster administrator normally does not manage: As an aside, there are also the additional CUDA DCGM packages:

At the time of writing of this paragraph (December 2025) the latest DCGM version supported was 4.1.1.1. A yum info cuda-dcgm or apt show cuda-dcgm should reveal the current latest available version for the cluster.

• Release notes for the DCGM versions are available at https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html.

9.1.1.2 CUDA Package That The Cluster Administrator May Wish To Install For CUDA Programming

CUB is a CUDA programming library that developers may wish to access. It is provided by the package cm-cub-cuda, from the Machine Learning (cm-ml) repository.

9.1.1.3 CUDA Package Installation Basic Sanity Check

• The NVIDIA GPU hardware should be detectable by the kernel, otherwise the GPUs cannot be used by the drivers. Running the lspci command on the device with the GPU before the CUDA package driver installation is a quick check that should make it clear if the NVIDIA hardware is detected in the first place:

  Example

  running lspci on node001 which is where the GPU should be:

  [root@node001]# lspci | grep NVIDIA
  00:06.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM3 32GB] (rev a1)

  If the hardware is not detected by the kernel already, then the administrator should reassess the situation.

• Only after CUDA package installation has taken place, and after rebooting the node with the GPU, and allowing CMDaemon some time (a few minutes) to detect the change, are GPU details visible using the sysinfo command:

  Example

  running sysinfo on node001, which is where the GPU is, via cmsh on the head node, while cuda-dcgm is not yet ready:

  [root@basecm10 ~]# cmsh
  [basecm10]% device use node001
  [basecm10->device[node001]]% sysinfo | grep GPU

  Example

  running sysinfo on node001, which is where the GPU is, via cmsh on the head node, after cuda-dcgm is ready:

  [basecm10->device[node001]]% sysinfo | grep GPU
  Number of GPUs            1
  GPU0 Name                 NVIDIA Tesla V100-SXM3-32GB
  GPU0 Power Limit          350 W
  GPU0 BIOS                 88.00.6B.00.01
  GPU Driver Version        530.23.08
  GPU0 nvlink               0 up, 6 down

9.1.1.4 CUDA Package Installation Guidelines For Compilation With Login Nodes

• CUDA compilation should take place on a node that uses NVIDIA GPUs during compilation.

  • Using a workload manager to allocate this task to GPU nodes is recommended.

• Cross compilation of CUDA software is generally not a best practice due to resource consumption, which can even lead to crashes.

  • If, despite this, cross compilation with a CPU is done, then the cuda-driver package should be installed on the node on which the compilation is done, and the GPU-related services on the node, such as:

    • cuda-driver.service

    • nvidia-persistenced.service

    • cuda-dcgm.service

    should be disabled.

9.1.2 CUDA Package Dependencies

The CUDA packages require access to the BCM repositories and to the distribution repositories.

In particular, the freeglut, freeglut-devel, and xorg-x11-util-macros packages are required. The installation ISO/DVD that Red Hat provides contains packages from the main repository, and does not contain these packages. For convenience, these packages are provided with a BCM installation ISO/DVD for Red Hat. Updates must however come from a subscription to the Red Hat supplementary/optional channels. Packages that are needed for a working BCM cluster, and which are provided by the BCM installation ISO/DVD, but which are not provided in the Red Hat installation DVD, are discussed in general in section 11.6.2 of the Administrator Manual, where the problems that such package dependencies can cause when creating a software image for a cluster with cm-create-image are discussed.

As a separate issue:

yum -q deplist cuda-driver | grep freeglut-devel
  dependency: freeglut-devel
   provider: freeglut-devel-3.0.0-8.el8.i686
   provider: freeglut-devel-3.0.0-8.el8.x86_64

shows that one of the dependencies of the cuda-driver package in RHEL8 (not in RHEL9) is the freeglut-devel package, so for RHEL8 and derivatives it should be installed on a node that accesses a GPU. If the CUDA SDK source is to be compiled on the head node (with the head node not accessing a GPU, and with the cuda-driver package not installed) then the freeglut, freeglut-devel, and libXi-devel packages should be installed on the head node.

The cuda-driver package is used to compile the kernel drivers which manage the GPU. Therefore, when installing cuda-driver with yum, several other X11-related packages are installed too, due to package dependencies.

The cuda*-sdk packages can be used to compile libraries and tools that are not part of the CUDA toolkit, but used by CUDA software developers, such as the deviceQuery binary (section 9.3).

The cuda-xorg package is optional, and contains the driver and libraries for an X server

Example

For example, on a cluster where (some of) the nodes access GPUs, but the head node does not access a GPU, the following commands can be issued on the head node to install the CUDA 12.2 packages using YUM:

[root@mycluster ~]# yum install cuda12.2-toolkit cuda12.2-sdk cuda12.2-visual-tools
[root@mycluster ~]# yum --installroot=/cm/images/default-image install cuda-driver cuda-dcgm

The --installroot command installs to the image used by the nodes. Here the image used by the nodes is assumed to be default-image. To ensure the software is installed from the image to the nodes, the imageupdate command can be run from within cmsh for the appropriate nodes.

[root@mycluster ~]# cmsh
[mycluster]% device
[mycluster->device]% imageupdate -n node0[01-10] -w

9.1.2.1 Compiling And Loading CUDA Drivers On The Fly

The cuda-driver package provides a systemd script that is executed at boot-time to load the CUDA driver. Because the CUDA driver depends on the running kernel, the script compiles the CUDA driver on the fly, and subsequently loads the module into the running kernel, and creates the nvidia* devices (/dev/nvidia0, /dev/nvidia1…).

The actions of compiling the cuda-driver, loading the module, and creating the nvidia* devices, can also be carried out during normal running by manually running the script:

Example

[root@mycluster ~]# systemctl start cuda-driver

The status of the driver and some event messages can be seen with:

Example

[root@node001 ~]# systemctl status cuda-driver
* cuda-driver.service - build, install and load the nvidia kernel modules
     Loaded: loaded (/usr/lib/systemd/system/cuda-driver.service; enabled; vendor preset: disabled)
     Active: active (exited) since Sun 2023-05-28 07:06:12 CEST; 4h 32min ago
   Main PID: 1816 (code=exited, status=0/SUCCESS)
        CPU: 3min 29.287s
May 24 07:04:01 node001 systemd[1]: Starting build, install and load the nvidia kernel modules...
May 24 07:04:15 node001 cuda-driver[1816]: compile nvidia kernel modules
May 24 07:06:09 node001 cuda-driver[1816]: loading kernel modules
May 24 07:06:12 node001 cuda-driver[1816]: create devices
May 24 07:06:12 node001 cuda-driver[1816]: [  OK  ]
May 24 07:06:12 node001 systemd[1]: Finished build, install and load the nvidia kernel modules.

If there is a failure in compiling the CUDA module, it is usually indicated by a message saying “Could not make module”, “NVRM: API mismatch:”, or “Cannot determine kernel version”. Such a failure typically occurs because compilation is not possible due to missing the correct kernel development package from the distribution. Section 9.2 explains how to check for, and install, the appropriate missing package.

9.2 Installing Kernel Development Packages

This section can be skipped if there is no CUDA compilation problem.

Typically, a CUDA compilation problem (section 9.1) is due to a missing or mismatched kernel package and kernel-devel package.

To check the head node and software images for the installation status of the kernel-devel package, the BCM utility kernel-devel-check is used (section 11.3.5.1 of the Administrator Manual).

Alternatively, if a standard kernel is in use by the image, then simply upgrading CUDA, the standard kernel, and kernel-devel, to their latest versions may be a good tactic to fix a CUDA compilation problem, because the kernel and kernel-devel package versions become synchronized during such an upgrade.

9.3 Verifying CUDA

An extensive method to verify that CUDA is working is to run the verify_cuda <version>.sh script, located in the CUDA SDK directory. The value of <version> is 12.2, for example.

This script first copies the CUDA SDK source to a local directory under /tmp or /local. It then builds CUDA test binaries and runs them. It is possible to select which of the CUDA test binaries are run. These binaries clutter up the disk and are not intended for use as regular tools, so the administrator is urged to remove them after they are built and run.

A help text showing available script options is displayed when “verify_cudaX.sh -h” is run.

The script can be run as follows on the head or regular node (some output elided):

Example

[root@node001 ~]# module load shared cuda12.1/toolkit
[root@node001 ~]# cd $CUDA_SDK
[root@node001 12.1.0]# ./verify_cuda12.1.sh
Copy cuda12.1 sdk files to "/tmp/cuda12.1" directory.
make clean
make (may take a while)
Run all tests? (y/N)? y
Executing: /tmp/cuda12.1/bin/x86_64/linux/release/alignedTypes
[/tmp/cuda12.1/bin/x86_64/linux/release/alignedTypes] - Starting...
GPU Device 0: "Volta" with compute capability 7.0
[Tesla V100-PCIE-32GB] has 80 MP(s) x 64 (Cores/MP) = 5120 (Cores)
> Compute scaling value = 1.00
> Memory Size = 49999872
Allocating memory...
Generating host input data array...
Uploading input data to GPU memory...
Testing misaligned types...
uint8...
Avg. time: 1.234375 ms / Copy throughput: 37.724362 GB/s.
    TEST OK
uint16...
...
...
All cuda12.1 just compiled test programs can be found in the
"/tmp/cuda12.1/bin/x86_64/linux/release/" directory
They can be executed from the "/tmp/cuda12.1" directory.
The "/tmp/cuda12.1" directory may take up a lot of diskspace.
Use "rm -rf /tmp/cuda12.1" to remove the data.

Another method to verify that CUDA is working, is to build and use the deviceQuery command on a node accessing one or more GPUs. The deviceQuery command lists all CUDA-capable GPUs that a device can access, along with several of their properties (some output elided):

Example

[root@node001 ~]# module load shared cuda12.2/toolkit
[root@node001 ~]# cd $CUDA_SDK
[root@node001 12.2.2]# cd  Samples/1_Utilities/deviceQuery
[root@node001 deviceQuery ]# make
...
mkdir -p ../../bin/x86_64/linux/release
cp deviceQuery ../../bin/x86_64/linux/release
[root@node001 deviceQuery ]# ./deviceQuery
./deviceQuery Starting...
 CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Tesla V100-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.2 / 12.2
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                 32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
  GPU Max Clock rate:                            1380 MHz (1.38 GHz)
  Memory Clock rate:                             877 Mhz
...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.2, CUDA Runtime Version = 12.2, NumDevs
= 1
Result = PASS

The CUDA user manual has further information on how to run compute jobs using CUDA.

9.3.0.1 Further Information On CUDA Verification

More on verification can be found in the NVIDIA CUDA INSTALLATION GUIDE FOR LINUX at https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf.

9.4 Verifying OpenCL

CUDA also contains an OpenCL compatible interface. To verify that OpenCL is working, or failing, the verify_opencl.sh script can be run on the node with the GPU (some output elided):

Example

[root@cuda-test ~]# module load shared cuda12.1/toolkit
[root@cuda-test ~]# cd $CUDA_SDK
[root@cuda-test 12.1.0]# ./verify_opencl.sh
Copy opencl files to "/tmp/opencl" directory.
make clean
make (may take a while)
Run all tests? (y/N)? y
Executing: /tmp/opencl/OpenCL/bin/linux/release/oclBandwidthTest
[oclBandwidthTest] starting...
/tmp/opencl/OpenCL/bin/linux/release/oclBandwidthTest Starting...
Running on...
Tesla V100-SXM3-32GB
Quick Mode
Host to Device Bandwidth, 1 Device(s), Paged memory, direct access
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     3480.5
Device to Host Bandwidth, 1 Device(s), Paged memory, direct access
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     3386.6
Device to Device Bandwidth, 1 Device(s)
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     691078.1
[oclBandwidthTest] test results...
PASSED
> exiting in 3 seconds: 3...2...1...done!

…many more tests, all of which should end with PASSED…

All opencl just compiled test programs can be found in the
"/tmp/opencl/OpenCL/bin/linux/release/" directory
They can be executed from the "/tmp/opencl/OpenCL" directory.
The "/tmp/opencl" directory may take up a lot of diskspace.
Use "rm -rf /tmp/opencl" to remove the data.

9.5 Configuring The X Server For SLES15

To support the X server, X.org, in SLES15, the cuda-driver, and cuda-xorg packages need to be installed.

The default configuration file for X.org is /etc/X11/xorg.conf. If there is no xorg.conf file, then a template can be generated with the help of Xorg -configure.

The following file pathname lines may need to be added to the Files section of the X configuration file:

    ModulePath "/usr/lib64/xorg/modules/extensions/nvidia"
    ModulePath "/usr/lib64/xorg/modules/extensions"
    ModulePath "/usr/lib64/xorg/modules"

The following dynamic module loading line may need to be added to the Module section of the X configuration:

    Load       "glx"

The following graphics device description lines need to be replaced in the Device section of the X configuration:

    Driver      "nvidia"

The BusID line may need to be replaced with the ID shown for the GPU by the lspci command.

Example

Section "ServerLayout"
    Identifier     "Default Layout"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
EndSection
Section "Files"
    ModulePath     "/usr/lib64/xorg/modules/extensions/nvidia"
    ModulePath     "/usr/lib64/xorg/modules/extensions"
    ModulePath     "/usr/lib64/xorg/modules"
EndSection
Section "Module"
    Load           "glx"
EndSection
Section "InputDevice"
    Identifier     "Keyboard0"
    Driver         "kbd"
    Option         "XkbModel" "pc105"
    Option         "XkbLayout" "us"
EndSection
Section "Device"
    Identifier     "Videocard0"
    Driver         "nvidia"
    BusID          "PCI:14:0:0"
EndSection
Section "Screen"
    Identifier     "Screen0"
    Device         "Videocard0"
    DefaultDepth   24
    SubSection     "Display"
        Viewport   0 0
        Depth      24
    EndSubSection
EndSection

Using GPUs for an X server as well as for general computing with CUDA can have some issues. These are considered in https://nvidia.custhelp.com/app/answers/detail/a_id/3029/~/using-cuda-and-x.

9.6 NVIDIA Validation Suite (Package: cuda-dcgm-nvvs)

The NVIDIA Validation Suite (NVVS) runs diagnostics to validate the NVIDIA software components. Running the nvvs binary directly is now deprecated. The diagnostics are now run instead as part of the dcgmi diag command.

The package for NVVS is not installed by default.

If the package is not installed, then an attempt to run the dcgmi utility fails:

Example

[root@basecm10 ~] module load cuda-dcgm
[root@basecm10 ~]# dcgmi diag -r 1
Error: Diagnostic could not be run because the Tesla recommended driver is not being used.

The package can be installed with:

Example

[root@basecm10 ~] module load cuda-dcgm
[root@basecm10 ~] yum install cuda-dcgm-nvvs

After it is installed, the node on which the installation is done must be rebooted.

Running the diagnostic after the reboot should display output similar to:

Example

[root@basecm10 ~] module load cuda-dcgm
[root@basecm10 ~]# dcgmi diag -r 3
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Deployment  --------+------------------------------------------------|
| Blacklist                 | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement           | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
| Diagnostic                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| SM Stress                 | Pass - All                                     |
| Targeted Stress           | Pass - All                                     |
| Targeted Power            | Pass - All                                     |
| Memory Bandwidth          | Pass - All                                     |
+---------------------------+------------------------------------------------+

9.7 Further NVIDIA Configuration Via The Cluster Manager

After the installation and verification has been carried out for CUDA drivers, further configuration as part of CMDaemon management can be carried out, as explained in section 3.16.2 of the Administrator Manual.