> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # CUDA For GPUs The optional CUDA packages should be deployed in order to take advantage of the computational capabilities of NVIDIA GPUs. The packages may already be in place, and ready for deployment on the cluster, depending on the particular BCM software that was obtained. If the CUDA packages are not in place, then they can be picked up from the BCM repositories, or a local mirror. ## 9.1 Installing CUDA ### 9.1.1 CUDA Packages Available At the time of writing of this section (December 2025), the CUDA 13.0 package is the most recent version available in the BCM YUM, zypper, and APT repositories. An up-to-date list of packages available for a particular distribution and architecture can be viewed at [https://service.bcm.nvidia.com/packages-dashboard/](https://service.bcm.nvidia.com/packages-dashboard/). #### 9.1.1.1 Example: CUDA Packages Available For Rocky Linux 9.3 At the time of writing of this section (December 2025), a cluster administrator can manage these packages for Rocky Linux 9.5:
**Package** **Type** **Description**
`cuda11.7-toolkit` shared CUDA math libraries and utilities
`cuda11.8-toolkit` shared CUDA math libraries and utilities
`cuda12.0-toolkit` shared CUDA math libraries and utilities
`cuda12.1-toolkit` shared CUDA math libraries and utilities
`cuda12.2-toolkit` shared CUDA math libraries and utilities
`cuda12.3-toolkit` shared CUDA math libraries and utilities
`cuda12.4-toolkit` shared CUDA math libraries and utilities
`cuda12.5-toolkit` shared CUDA math libraries and utilities
`cuda12.6-toolkit` shared CUDA math libraries and utilities
`cuda12.8-toolkit` shared CUDA math libraries and utilities
`cuda12.9-toolkit` shared CUDA math libraries and utilities
`cuda13.0-toolkit` shared CUDA math libraries and utilities
`cuda11.7-visual-tools` shared CUDA visual toolkit
`cuda11.8-visual-tools` shared CUDA visual toolkit
`cuda12.0-visual-tools` shared CUDA visual toolkit
`cuda12.1-visual-tools` shared CUDA visual toolkit
`cuda12.2-visual-tools` shared CUDA visual toolkit
`cuda12.3-visual-tools` shared CUDA visual toolkit
`cuda12.4-visual-tools` shared CUDA visual toolkit
`cuda12.5-visual-tools` shared CUDA visual toolkit
`cuda12.6-visual-tools` shared CUDA visual toolkit
`cuda12.8-visual-tools` shared CUDA visual toolkit
`cuda12.9-visual-tools` shared CUDA visual toolkit
`cuda13.0-visual-tools` shared CUDA visual toolkit
`cuda11.7-sdk` shared CUDA software development kit
`cuda11.8-sdk` shared CUDA software development kit
`cuda12.0-sdk` shared CUDA software development kit
`cuda12.1-sdk` shared CUDA software development kit
`cuda12.2-sdk` shared CUDA software development kit
`cuda12.3-sdk` shared CUDA software development kit
`cuda12.4-sdk` shared CUDA software development kit
`cuda12.5-sdk` shared CUDA software development kit
`cuda12.6-sdk` shared CUDA software development kit
`cuda12.8-sdk` shared CUDA software development kit
`cuda12.9-sdk` shared CUDA software development kit
`cuda13.0-sdk` shared CUDA software development kit
`cuda-driver` local CUDA GPU driver and libraries.
`cuda-dcgm` local CUDA Data Center GPU Manager (DCGM).
This includes the `dcgmi` CLI tool.
`cuda-fabric-manager` \* local Fabric Manager tools
`cuda-fabric-manager-development` \* local Fabric Manager tools
\* optional, intended only for hardware containing an NVSwitch, such as a DGX system \ The packages of type `shared` in the preceding table should be installed on the head nodes of a cluster using CUDA-compatible GPUs. The packages of type `local` should be installed to all nodes that access the GPUs. In most cases this means that the `cuda-driver` and `cuda-dcgm` packages should be installed in a software image ([section 2.1.2 of the *Administrator Manual*](/admin/clustermanagement#212-software-images)). If a head node also accesses GPUs, then the `cuda-driver` and `cuda-dcgm` packages should be installed on it, too. For packages of type `shared`, the particular CUDA version that is run on the node can be selected via a modules environment command: **Example** `module add shared cuda12.1` **Release notes and features list for the CUDA packages:** * Release notes for the NVIDIA toolkit can be found at [https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html). * The features list for the NVIDIA toolkit can be found at [https://docs.nvidia.com/cuda/cuda-features-archive/index.html](https://docs.nvidia.com/cuda/cuda-features-archive/index.html). **CUDA packages that the cluster administrator normally does not manage:** As an aside, there are also the additional CUDA DCGM packages:
**Package** **Type** **Description** **Installation on GPU node**
`cuda-dcgm-libs` local NVIDIA DCGM libraries installed by default
`cuda-dcgm-devel` local NVIDIA DCGM development files not installed by default
`cuda-dcgm-nvvs` local NVIDIA DCGM validation suite installed by default
At the time of writing of this paragraph (December 2025) the latest DCGM version supported was 4.1.1.1. A `yum info cuda-dcgm` or `apt show cuda-dcgm` should reveal the current latest available version for the cluster. * Release notes for the DCGM versions are available at [https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html](https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html). #### 9.1.1.2 CUDA Package That The Cluster Administrator May Wish To Install For CUDA Programming CUB is a CUDA programming library that developers may wish to access. It is provided by the package `cm-cub-cuda`, from the Machine Learning (`cm-ml`) repository. #### 9.1.1.3 CUDA Package Installation Basic Sanity Check * The NVIDIA GPU hardware should be detectable by the kernel, otherwise the GPUs cannot be used by the drivers. Running the `lspci` command on the device with the GPU before the CUDA package driver installation is a quick check that should make it clear if the NVIDIA hardware is detected in the first place: **Example** *running `lspci` on `node001` which is where the GPU should be:* ```bash [root@node001]# lspci | grep NVIDIA 00:06.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM3 32GB] (rev a1) ``` If the hardware is not detected by the kernel already, then the administrator should reassess the situation. * Only after CUDA package installation has taken place, and after rebooting the node with the GPU, and allowing CMDaemon some time (a few minutes) to detect the change, are GPU details visible using the `sysinfo` command: **Example** *running `sysinfo` on `node001`, which is where the GPU is, via `cmsh` on the head node, while `cuda-dcgm` is not yet ready:* ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% sysinfo | grep GPU ``` **Example** *running `sysinfo` on `node001`, which is where the GPU is, via `cmsh` on the head node, after `cuda-dcgm` is ready:* ```bash [basecm10->device[node001]]% sysinfo | grep GPU Number of GPUs 1 GPU0 Name NVIDIA Tesla V100-SXM3-32GB GPU0 Power Limit 350 W GPU0 BIOS 88.00.6B.00.01 GPU Driver Version 530.23.08 GPU0 nvlink 0 up, 6 down ``` #### 9.1.1.4 CUDA Package Installation Guidelines For Compilation With Login Nodes * CUDA compilation should take place on a node that uses NVIDIA GPUs during compilation. * Using a workload manager to allocate this task to GPU nodes is recommended. * Cross compilation of CUDA software is generally not a best practice due to resource consumption, which can even lead to crashes. * If, despite this, cross compilation with a CPU is done, then the `cuda-driver` package should be installed on the node on which the compilation is done, and the GPU-related services on the node, such as: * `cuda-driver.service` * `nvidia-persistenced.service` * `cuda-dcgm.service` should be disabled. ### 9.1.2 CUDA Package Dependencies The CUDA packages require access to the BCM repositories and to the distribution repositories. In particular, the `freeglut`, `freeglut-devel`, and `xorg-x11-util-macros` packages are required. The installation ISO/DVD that Red Hat provides contains packages from the main repository, and does not contain these packages. For convenience, these packages are provided with a BCM installation ISO/DVD for Red Hat. Updates must however come from a subscription to the Red Hat supplementary/optional channels. Packages that are needed for a working BCM cluster, and which are provided by the BCM installation ISO/DVD, but which are not provided in the Red Hat installation DVD, are discussed in general in [section 11.6.2 of the *Administrator Manual*](/admin/softwareimagemanagement#1162-creating-the-software-image-with-cm-create-image), where the problems that such package dependencies can cause when creating a software image for a cluster with `cm-create-image` are discussed. As a separate issue: ```bash yum -q deplist cuda-driver | grep freeglut-devel dependency: freeglut-devel provider: freeglut-devel-3.0.0-8.el8.i686 provider: freeglut-devel-3.0.0-8.el8.x86_64 ``` shows that one of the dependencies of the `cuda-driver` package in RHEL8 (not in RHEL9) is the `freeglut-devel` package, so for RHEL8 and derivatives it should be installed on a node that accesses a GPU. If the CUDA SDK source is to be compiled on the head node (with the head node not accessing a GPU, and with the `cuda-driver` package not installed) then the `freeglut`, `freeglut-devel`, and `libXi-devel` packages should be installed on the head node. The `cuda-driver` package is used to compile the kernel drivers which manage the GPU. Therefore, when installing `cuda-driver` with `yum`, several other X11-related packages are installed too, due to package dependencies. The `cuda*-sdk` packages can be used to compile libraries and tools that are not part of the CUDA toolkit, but used by CUDA software developers, such as the `deviceQuery` binary (section [9.3](#93-verifying-cuda)). The `cuda-xorg` package is optional, and contains the driver and libraries for an X server **Example** For example, on a cluster where (some of) the nodes access GPUs, but the head node does not access a GPU, the following commands can be issued on the head node to install the CUDA 12.2 packages using YUM: ```bash [root@mycluster ~]# yum install cuda12.2-toolkit cuda12.2-sdk cuda12.2-visual-tools [root@mycluster ~]# yum --installroot=/cm/images/default-image install cuda-driver cuda-dcgm ``` The `--installroot` command installs to the image used by the nodes. Here the image used by the nodes is assumed to be `default-image`. To ensure the software is installed from the image to the nodes, the `imageupdate` command can be run from within `cmsh` for the appropriate nodes. ```bash [root@mycluster ~]# cmsh [mycluster]% device [mycluster->device]% imageupdate -n node0[01-10] -w ``` #### 9.1.2.1 Compiling And Loading CUDA Drivers On The Fly The `cuda-driver` package provides a systemd script that is executed at boot-time to load the CUDA driver. Because the CUDA driver depends on the running kernel, the script compiles the CUDA driver on the fly, and subsequently loads the module into the running kernel, and creates the `nvidia*` devices (`/dev/nvidia0`, `/dev/nvidia1`...). The actions of compiling the `cuda-driver`, loading the module, and creating the `nvidia*` devices, can also be carried out during normal running by manually running the script: **Example** ```bash [root@mycluster ~]# systemctl start cuda-driver ``` The status of the driver and some event messages can be seen with: **Example** ```bash [root@node001 ~]# systemctl status cuda-driver * cuda-driver.service - build, install and load the nvidia kernel modules Loaded: loaded (/usr/lib/systemd/system/cuda-driver.service; enabled; vendor preset: disabled) Active: active (exited) since Sun 2023-05-28 07:06:12 CEST; 4h 32min ago Main PID: 1816 (code=exited, status=0/SUCCESS) CPU: 3min 29.287s May 24 07:04:01 node001 systemd[1]: Starting build, install and load the nvidia kernel modules... May 24 07:04:15 node001 cuda-driver[1816]: compile nvidia kernel modules May 24 07:06:09 node001 cuda-driver[1816]: loading kernel modules May 24 07:06:12 node001 cuda-driver[1816]: create devices May 24 07:06:12 node001 cuda-driver[1816]: [ OK ] May 24 07:06:12 node001 systemd[1]: Finished build, install and load the nvidia kernel modules. ``` If there is a failure in compiling the CUDA module, it is usually indicated by a message saying “`Could not make module`”, “`NVRM: API mismatch:`”, or “`Cannot determine kernel version`”. Such a failure typically occurs because compilation is not possible due to missing the correct kernel development package from the distribution. Section [9.2](#92-installing-kernel-development-packages) explains how to check for, and install, the appropriate missing package. ## 9.2 Installing Kernel Development Packages This section can be skipped if there is no CUDA compilation problem. Typically, a CUDA compilation problem (section [9.1](#91-installing-cuda)) is due to a missing or mismatched `kernel` package and `kernel-devel package`. To check the head node and software images for the installation status of the `kernel-devel` package, the BCM utility `kernel-devel-check` is used ([section 11.3.5.1 of the *Administrator Manual*](/admin/softwareimagemanagement#11351-kernel-driver-modules-kernel-devel-check-compilation-check)). Alternatively, if a standard kernel is in use by the image, then simply upgrading CUDA, the standard kernel, and `kernel-devel`, to their latest versions may be a good tactic to fix a CUDA compilation problem, because the `kernel` and `kernel-devel` package versions become synchronized during such an upgrade. ## 9.3 Verifying CUDA An extensive method to verify that CUDA is working is to run the `verify_cuda` \<*version*>`.sh` script, located in the CUDA SDK directory. The value of \<*version*> is `12.2`, for example. This script first copies the CUDA SDK source to a local directory under `/tmp` or `/local`. It then builds CUDA test binaries and runs them. It is possible to select which of the CUDA test binaries are run. These binaries clutter up the disk and are not intended for use as regular tools, so the administrator is urged to remove them after they are built and run. A help text showing available script options is displayed when “`verify_cuda`*X*`.sh -h`” is run. The script can be run as follows on the head or regular node (some output elided): **Example** ```bash [root@node001 ~]# module load shared cuda12.1/toolkit [root@node001 ~]# cd $CUDA_SDK [root@node001 12.1.0]# ./verify_cuda12.1.sh Copy cuda12.1 sdk files to "/tmp/cuda12.1" directory. make clean make (may take a while) Run all tests? (y/N)? y Executing: /tmp/cuda12.1/bin/x86_64/linux/release/alignedTypes [/tmp/cuda12.1/bin/x86_64/linux/release/alignedTypes] - Starting... GPU Device 0: "Volta" with compute capability 7.0 [Tesla V100-PCIE-32GB] has 80 MP(s) x 64 (Cores/MP) = 5120 (Cores) > Compute scaling value = 1.00 > Memory Size = 49999872 Allocating memory... Generating host input data array... Uploading input data to GPU memory... Testing misaligned types... uint8... Avg. time: 1.234375 ms / Copy throughput: 37.724362 GB/s. TEST OK uint16... ... ... All cuda12.1 just compiled test programs can be found in the "/tmp/cuda12.1/bin/x86_64/linux/release/" directory They can be executed from the "/tmp/cuda12.1" directory. The "/tmp/cuda12.1" directory may take up a lot of diskspace. Use "rm -rf /tmp/cuda12.1" to remove the data. ``` Another method to verify that CUDA is working, is to build and use the `deviceQuery` command on a node accessing one or more GPUs. The `deviceQuery` command lists all CUDA-capable GPUs that a device can access, along with several of their properties (some output elided): **Example** ```bash [root@node001 ~]# module load shared cuda12.2/toolkit [root@node001 ~]# cd $CUDA_SDK [root@node001 12.2.2]# cd Samples/1_Utilities/deviceQuery [root@node001 deviceQuery ]# make ... mkdir -p ../../bin/x86_64/linux/release cp deviceQuery ../../bin/x86_64/linux/release [root@node001 deviceQuery ]# ./deviceQuery ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "Tesla V100-PCIE-32GB" CUDA Driver Version / Runtime Version 12.2 / 12.2 CUDA Capability Major/Minor version number: 7.0 Total amount of global memory: 32510 MBytes (34089730048 bytes) (80) Multiprocessors, ( 64) CUDA Cores/MP: 5120 CUDA Cores GPU Max Clock rate: 1380 MHz (1.38 GHz) Memory Clock rate: 877 Mhz ... deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.2, CUDA Runtime Version = 12.2, NumDevs = 1 Result = PASS ``` The CUDA user manual has further information on how to run compute jobs using CUDA. #### 9.3.0.1 Further Information On CUDA Verification More on verification can be found in the *NVIDIA CUDA INSTALLATION GUIDE FOR LINUX* at [https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf](https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf). ## 9.4 Verifying OpenCL CUDA also contains an OpenCL compatible interface. To verify that OpenCL is working, or failing, the `verify_opencl.sh` script can be run on the node with the GPU (some output elided): **Example** ```bash [root@cuda-test ~]# module load shared cuda12.1/toolkit [root@cuda-test ~]# cd $CUDA_SDK [root@cuda-test 12.1.0]# ./verify_opencl.sh Copy opencl files to "/tmp/opencl" directory. make clean make (may take a while) Run all tests? (y/N)? y Executing: /tmp/opencl/OpenCL/bin/linux/release/oclBandwidthTest [oclBandwidthTest] starting... /tmp/opencl/OpenCL/bin/linux/release/oclBandwidthTest Starting... Running on... Tesla V100-SXM3-32GB Quick Mode Host to Device Bandwidth, 1 Device(s), Paged memory, direct access Transfer Size (Bytes) Bandwidth(MB/s) 33554432 3480.5 Device to Host Bandwidth, 1 Device(s), Paged memory, direct access Transfer Size (Bytes) Bandwidth(MB/s) 33554432 3386.6 Device to Device Bandwidth, 1 Device(s) Transfer Size (Bytes) Bandwidth(MB/s) 33554432 691078.1 [oclBandwidthTest] test results... PASSED > exiting in 3 seconds: 3...2...1...done! ``` *...many more tests, all of which should end with* `PASSED`*...* ```bash All opencl just compiled test programs can be found in the "/tmp/opencl/OpenCL/bin/linux/release/" directory They can be executed from the "/tmp/opencl/OpenCL" directory. The "/tmp/opencl" directory may take up a lot of diskspace. Use "rm -rf /tmp/opencl" to remove the data. ``` ## 9.5 Configuring The X Server For SLES15 To support the X server, X.org, in SLES15, the `cuda-driver`, and `cuda-xorg` packages need to be installed. The default configuration file for X.org is `/etc/X11/xorg.conf`. If there is no `xorg.conf` file, then a template can be generated with the help of `Xorg -configure`. The following file pathname lines may need to be added to the `Files` section of the X configuration file: ```bash ModulePath "/usr/lib64/xorg/modules/extensions/nvidia" ModulePath "/usr/lib64/xorg/modules/extensions" ModulePath "/usr/lib64/xorg/modules" ``` The following dynamic module loading line may need to be added to the `Module` section of the X configuration: ```bash Load "glx" ``` The following graphics device description lines need to be replaced in the `Device` section of the X configuration: ```bash Driver "nvidia" ``` The BusID line may need to be replaced with the ID shown for the GPU by the `lspci` command. **Example** ```bash Section "ServerLayout" Identifier "Default Layout" Screen 0 "Screen0" 0 0 InputDevice "Keyboard0" "CoreKeyboard" EndSection Section "Files" ModulePath "/usr/lib64/xorg/modules/extensions/nvidia" ModulePath "/usr/lib64/xorg/modules/extensions" ModulePath "/usr/lib64/xorg/modules" EndSection Section "Module" Load "glx" EndSection Section "InputDevice" Identifier "Keyboard0" Driver "kbd" Option "XkbModel" "pc105" Option "XkbLayout" "us" EndSection Section "Device" Identifier "Videocard0" Driver "nvidia" BusID "PCI:14:0:0" EndSection Section "Screen" Identifier "Screen0" Device "Videocard0" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 24 EndSubSection EndSection ``` Using GPUs for an X server as well as for general computing with CUDA can have some issues. These are considered in [https://nvidia.custhelp.com/app/answers/detail/a_id/3029/\~/using-cuda-and-x](https://nvidia.custhelp.com/app/answers/detail/a_id/3029/~/using-cuda-and-x). ## 9.6 NVIDIA Validation Suite (Package: `cuda-dcgm-nvvs`) The NVIDIA Validation Suite (NVVS) runs diagnostics to validate the NVIDIA software components. Running the `nvvs` binary directly is now deprecated. The diagnostics are now run instead as part of the `dcgmi diag` command. The package for NVVS is not installed by default. If the package is not installed, then an attempt to run the `dcgmi` utility fails: **Example** ```bash [root@basecm10 ~] module load cuda-dcgm [root@basecm10 ~]# dcgmi diag -r 1 Error: Diagnostic could not be run because the Tesla recommended driver is not being used. ``` The package can be installed with: **Example** ```bash [root@basecm10 ~] module load cuda-dcgm [root@basecm10 ~] yum install cuda-dcgm-nvvs ``` After it is installed, the node on which the installation is done must be rebooted. Running the diagnostic after the reboot should display output similar to: **Example** ```bash [root@basecm10 ~] module load cuda-dcgm [root@basecm10 ~]# dcgmi diag -r 3 Successfully ran diagnostic for group. +---------------------------+------------------------------------------------+ | Diagnostic | Result | +===========================+================================================+ |----- Deployment --------+------------------------------------------------| | Blacklist | Pass | | NVML Library | Pass | | CUDA Main Library | Pass | | Permissions and OS Blocks | Pass | | Persistence Mode | Pass | | Environment Variables | Pass | | Page Retirement | Pass | | Graphics Processes | Pass | | Inforom | Pass | +----- Integration -------+------------------------------------------------+ | PCIe | Pass - All | +----- Hardware ----------+------------------------------------------------+ | GPU Memory | Pass - All | | Diagnostic | Pass - All | +----- Stress ------------+------------------------------------------------+ | SM Stress | Pass - All | | Targeted Stress | Pass - All | | Targeted Power | Pass - All | | Memory Bandwidth | Pass - All | +---------------------------+------------------------------------------------+ ``` ## 9.7 Further NVIDIA Configuration Via The Cluster Manager After the installation and verification has been carried out for CUDA drivers, further configuration as part of CMDaemon management can be carried out, as explained in [section 3.16.2 of the *Administrator Manual*](/admin/configuring#3162-configuring-gpu-settings).