> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # CUDA For GPUs The optional CUDA packages should be deployed in order to take advantage of the computational capabilities of NVIDIA GPUs. The packages may already be in place, and ready for deployment on the cluster, depending on the particular BCM software that was obtained. If the CUDA packages are not in place, then they can be picked up from the BCM repositories, or a local mirror. ## 9.1 Installing CUDA ### 9.1.1 CUDA Packages Available At the time of writing of this section (April 2026), the CUDA 13.1 package is the most recent version available in the BCM YUM, zypper, and APT repositories. The available versions are updated typically in the next point release version of NVIDIA Base Command Manager after the upstream changes are made available. An up-to-date list of packages available for a particular distribution and architecture can be viewed at [https://service.bcm.nvidia.com/packages-dashboard/](https://service.bcm.nvidia.com/packages-dashboard/). Thus, for example, at the time of writing, Rocky Linux 9.6 has many CUDA versions available, in the range from CUDA 11.7 to CUDA 13.1 for the `x86_64` architecture, while Ubuntu 24.04 just has CUDA 12.5, CUDA 12.6, CUDA 12.8, CUDA 12.9, CUDA-13.0, and CUDA 13.1 for `x86_64`. #### 9.1.1.1 Example: CUDA Packages Available For Ubuntu 24.04 At the time of writing of this section (April 2026), a cluster administrator can manage these packages for Ubuntu 24.04:
**Package** **Type** **Description**
`cuda12.5-toolkit` shared CUDA math libraries and utilities
`cuda12.6-toolkit` shared CUDA math libraries and utilities
`cuda12.8-toolkit` shared CUDA math libraries and utilities
`cuda12.9-toolkit` shared CUDA math libraries and utilities
`cuda13.0-toolkit` shared CUDA math libraries and utilities
`cuda13.1-toolkit` shared CUDA math libraries and utilities
`cuda12.5-visual-tools` shared CUDA visual toolkit
`cuda12.6-visual-tools` shared CUDA visual toolkit
`cuda12.8-visual-tools` shared CUDA visual toolkit
`cuda12.9-visual-tools` shared CUDA visual toolkit
`cuda13.0-visual-tools` shared CUDA visual toolkit
`cuda13.1-visual-tools` shared CUDA visual toolkit
`cuda12.5-sdk` shared CUDA software development kit
`cuda12.6-sdk` shared CUDA software development kit
`cuda12.8-sdk` shared CUDA software development kit
`cuda12.9-sdk` shared CUDA software development kit
`cuda13.0-sdk` shared CUDA software development kit
`cuda13.1-sdk` shared CUDA software development kit
`nvidia-driver` \**, † local CUDA GPU driver and libraries.
`cuda-dcgm` local CUDA Data Center GPU Manager (DCGM).
This includes the `dcgmi` CLI tool.
`nvidia-fabricmanager` *,* \* local Fabric Manager tools
`nvidia-fabricmanager-dev` *,* \* local Fabric Manager tools
\* optional, intended only for hardware containing an NVSwitch, such as a DGX system \*\* installed by default on software images, not on head nodes † the driver typically has a version number tag such as `-590`. \ * The packages of type `shared` in the preceding table should be installed on the head nodes of a cluster using CUDA-compatible GPUs. * The packages of type `local` should be installed to all nodes that access the GPUs. In most cases this means that the * `nvidia-driver` (NVIDIA driver metapackage) * `cuda-dcgm` (NVIDIA Data Center GPU Manager) * `nvidia-fabricmanager` (Fabric Manager for NVSwitch based systems) * `nvidia-fabricmanager-dev` (Fabric Manager API headers and associated library) packages should be installed in a software image ([section 2.1.2 of the *Administrator Manual*](/admin/clustermanagement#212-software-images)). By default, BCM software images already have these packages, which means that an explicit installation by the cluster administrator should not be required. If a head node also accesses GPUs, then the `nvidia-driver` and `cuda-dcgm` packages should be installed on it, too. For packages of type `shared`, the particular CUDA version that is run on the node can be selected via a modules environment command. For example, to select CUDA SDK version 13.0, the user can run: **Example** ```bash user@basecm11:~$ module add shared cuda13.0 ``` **Release notes and features list for the CUDA packages:** * Release notes for the NVIDIA toolkit can be found at:\ [https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) * The features list for the NVIDIA toolkit can be found at:\ [https://docs.nvidia.com/cuda/cuda-features-archive/index.html](https://docs.nvidia.com/cuda/cuda-features-archive/index.html) **CUDA packages that the cluster administrator normally does not manage:** As an aside, there are also the additional CUDA DCGM packages:
**Package** **Type** **Description** **Installation on GPU node**
`cuda-dcgm-libs` local NVIDIA DCGM libraries installed by default
`cuda-dcgm-devel` local NVIDIA DCGM development files not installed by default
`cuda-dcgm-nvvs` local NVIDIA DCGM validation suite installed by default, except for in RHEL8, and except for in a DGX node
At the time of writing of this paragraph (April 2026) the latest DCGM version supported by BCM was 4.5.2. Running a * `dnf info cuda-dcgm; dnf list cuda-dcgm` for RHEL-family distributions * `apt show -a cuda-dcgm` for Ubuntu distributions should reveal the currently installed and available versions for the cluster. * Release notes for the DCGM versions are available at [https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html](https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html). #### 9.1.1.2 CUDA Package That The Cluster Administrator May Wish To Install For CUDA Programming CUB is a CUDA programming library that developers may wish to access. It is available for some distributions for some older CUDA versions, and provided by the package `cm-cub-cuda`. #### 9.1.1.3 CUDA Package Installation Basic Sanity Check * The NVIDIA GPU hardware should be detectable by the kernel, otherwise the GPUs cannot be used by the drivers. Running the `lspci` command on the device with the GPU before the CUDA package driver installation is a quick check that should make it clear if the NVIDIA hardware is detected in the first place: **Example** *running `lspci` on `node001`. Here 4 GPUs are detected:* ```bash [root@node001]# { lspci | grep '3D controller: NVIDIA';} && echo "NVIDIA GPU hardware detected by kernel" 0008:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1) 0009:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1) 0018:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1) 0019:01:00.0 3D controller: NVIDIA Corporation Device 2941 (rev a1) NVIDIA GPU hardware detected by kernel [root@node001]# ``` If the hardware is not detected by the kernel already, then the administrator should reassess the situation. The `Device 2941` part of the string can be tallied against the identifiers in the `README.txt.gz` file of the `nvidia-driver-580` package series, at the node where it has been installed. **Example** ```bash root@basecm11:~# zgrep 2941 /cm/images/default-image/usr/share/doc/nvidia-driver-580/README.txt.gz NVIDIA GB200 2941 10DE 2046 K NVIDIA GB200 2941 10DE 20CA K NVIDIA GB200 2941 10DE 20D5 K NVIDIA GB200 2941 10DE 21C9 K NVIDIA GB200 2941 10DE 21CA K ``` The output of the `lspci` command can be more directly informative for some hardware: **Example** ```bash root@basecm11:~# ssh node001 "lspci | grep 3D controller: NVIDIA" 00:06.0 3D controller: NVIDIA Corporation GA100 [A100 PCIe 40GB] (rev a1) root@basecm11:~# zgrep A100 /cm/images/default-image/usr/share/doc/nvidia-driver-580/README.txt.gz \ | grep -i PCIE | grep 40GB NVIDIA A100-PCIE-40GB 20F1 10DE 145F J ``` * Only after CUDA package installation has taken place, and after rebooting the node with the GPU, and allowing CMDaemon some time (a few minutes) to detect the change, are GPU details visible using the `sysinfo` command: **Example** *running `sysinfo` on `node001`, which is where the GPUs are, via `cmsh` on the head node, while `cuda-dcgm` is not yet ready:* ```bash [root@basecm11 ~]# cmsh [basecm11]% device use node001 [basecm11->device[node001]]% sysinfo | grep GPU ``` **Example** *running `sysinfo` on `node001`, via `cmsh` on the head node, after `cuda-dcgm` is ready. 4 GPUs are detected* ```bash [basecm11->device[node001]]% sysinfo | grep GPU [maple->device[dgx-gb200-n01-c1]]% sysinfo | grep GPU Motherboard Name GB200 1CPU:2GPU Board PC Number of GPUs 4 GPU[0-3] name NVIDIA Graphics Device GPU[0-3] power limit 1.20KW GPU[0-3] BIOS 97.00.6C.00.03 GPU[0-3] driver version 570.82 GPU[0-3] nvlink 2 up ``` #### 9.1.1.4 CUDA Package Installation Guidelines For Compilation With Login Nodes * CUDA compilation should take place on a node that uses NVIDIA GPUs during compilation. * Using a workload manager to allocate this task to GPU nodes is recommended. * Cross compilation of CUDA software with a CPU is generally not a best practice due to resource consumption, which can even lead to crashes. * If, despite this, cross compilation with a CPU is done, then the `nvidia-driver` package should be installed on the node on which the compilation is done, and the GPU-related services on the node, such as: * `nvidia-persistenced.service` * `nvidia-fabricmanager.service` * `nvidia-dcgm.service` should be disabled. ### 9.1.2 CUDA Package And Modules The CUDA packages require access to the BCM repositories and to the distribution repositories. The `nvidia-driver` packages replace the older `cuda-driver` packages that were used in versions of BCM prior to version 11. The newer packages do not compile the driver modules source at boot time, but during package installation. The newer packages are also open source. **Example** For example, on a cluster where (some of) the nodes access GPUs, but the head node does not access a GPU, the following commands install the CUDA 13.0 packages on the head using APT: ```bash root@mycluster:~# apt install cuda13.0-toolkit cuda13.0-sdk cuda13.0-visual-tools ``` The `nvidia-driver-`, `cuda-dcgm`, `cuda-dcgm-nvvs`, and `nvidia-fabricmanager` are already installed by default on the default software image, which is `default-image` on a newly-installed default cluster. Carrying out an explicit installation or `imageupdate` for these is therefore normally not needed. In cases where it is required, then manual installation into a software image is described in section [9.4](/admin/softwareimagemanagement#94-managing-a-package-in-a-software-image-and-running-it-on-nodes) of the *Administrator Manual*, while running the `imageupdate` command ([section 5.6](/admin/provisioning#56-updating-running-nodes) of the *Administrator Manual*), for example on nodes 1 to 10, can be carried out with: **Example** ```bash [root@mycluster ~]# cmsh [mycluster]% device [mycluster->device]% imageupdate -n node0[01-10] -w ``` If the CUDA packages have been set up on the cluster, then the end user typically just switches between CUDA environments with a module change ([section 2.2 of the *Administrator Manual*](/admin/clustermanagement#22-modules-environment) and [section 2.3 of the *User Manual*](/user/clusterusage#23-environment-modules)). **Example** ```bash module purge && module load shared cuda13.0/toolkit ``` ## 9.2 Verifying CUDA The `cuda*-sdk` packages provide CUDA SDK sources that can be used to compile libraries and tools that are not part of the CUDA toolkit, but used by CUDA software developers, such as the `deviceQuery` binary (section [9.2](#92-verifying-cuda)). An extensive method to verify that CUDA is working is to run the `verify_cuda` \<*version*>`.sh` script, located in the CUDA SDK directory. The value of \<*version*> can be `13.0`, for example. This script first copies the CUDA SDK source to a local directory under `/tmp` or `/local`. It then builds CUDA test binaries. To build the makefiles, the compilation may need `cmake`, installed from the `cm-make` package into the image ([section 9.4 of the *Administrator Manual*](/admin/softwareimagemanagement#94-managing-a-package-in-a-software-image-and-running-it-on-nodes)). These binaries clutter up the disk and are not intended for use as regular tools, so the administrator is urged to remove them after they are built and run. A help text showing available script options is displayed when “`verify_cuda` \<*version*>`.sh -h`” is run. A typical run output is as follows (some output elided): **Example** ```bash root@node001:~# module load shared cuda13.0/toolkit root@node001:~# cd $CUDA_SDK root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2# ./verify_cuda13.0.sh Copy cuda13.0 sdk files to "/tmp/cuda13.0" directory. cmake make (may take a while) Run multiple tests non-interactive ? (y/N)? Executing: /tmp/cuda13.0/build/CMakeFiles/3.26.3/CMakeDetermineCompilerABI_CUDA.bin 80cmd: /tmp/cuda13.0/build/CMakeFiles/3.26.3/CMakeDetermineCompilerABI_CUDA.bin - rc : 0 Executing: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_C.bin INFO:OpenMP-date[201511] cmd: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_C.bin - rc : 0 Executing: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_CXX.bin INFO:OpenMP-date[201511] cmd: /tmp/cuda13.0/build/CMakeFiles/FindOpenMP/ompver_CXX.bin - rc : 0 Executing: /tmp/cuda13.0/build/Samples/0_Introduction/clock/clock ... Test passed cmd: /tmp/cuda13.0/build/Samples/6_Performance/cudaGraphsPerfScaling/cudaGraphsPerfScaling - rc : 0 All cuda13.0 compiled test programs can be found in the "/tmp/cuda13.0/build/" directory Alternatively the "run_tests.py" can be run in the "/tmp/cuda13.0" directory. For example: cd /tmp/cuda13.0; python3 run_tests.py --output ./test --dir ./build/Samples --config test_args.json The "/tmp/cuda13.0" directory may take up a lot of diskspace. Use "rm -rf /tmp/cuda13.0" to remove the data. ``` One of the binaries compiled is the `deviceQuery` command. This can be run on a node accessing one or more GPUs. The `deviceQuery` command lists all CUDA-capable GPUs that a device can access, along with several of their properties (some output elided). It can be run in CUDA 13 as follows: ```bash root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2# /tmp/cuda13.0/Samples/1_Utilities/deviceQuery/deviceQuery /tmp/cuda13.0/Samples/1_Utilities/deviceQuery/deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "NVIDIA A100-PCIE-40GB" CUDA Driver Version / Runtime Version 13.0 / 13.0 CUDA Capability Major/Minor version number: 8.0 Total amount of global memory: 40441 MBytes (42405855232 bytes) (108) Multiprocessors, (064) CUDA Cores/MP: 6912 CUDA Cores GPU Max Clock rate: 1410 MHz (1.41 GHz) Memory Clock rate: 1215 Mhz Memory Bus Width: 5120-bit L2 Cache Size: 41943040 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total shared memory per multiprocessor: 167936 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 3 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device supports Managed Memory: Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 6 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.0, CUDA Runtime Version = 13.0, NumDevs = 1 Result = PASS root@node001:/cm/shared/apps/cuda13.0/sdk/13.0.2# ``` The query can detect if there is more than 1 GPU, and display output for each GPU that it finds. The CUDA programming guide ([https://docs.nvidia.com/cuda/cuda-programming-guide/](https://docs.nvidia.com/cuda/cuda-programming-guide/)) has further information on how to run compute jobs using CUDA. #### 9.2.0.1 Further Information On CUDA Verification More on verification can be found in the *NVIDIA CUDA INSTALLATION GUIDE FOR LINUX* at [https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf](https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf). ## 9.3 Verifying OpenCL CUDA 12.8 contains an OpenCL compatible interface. To verify that OpenCL is working, or failing, the binaries can be compiled from the provided OpenCL source and run. For example: **Example** ```bash root@node001:~# module load shared cuda12.8/toolkit root@node001 ~# cd $CUDA_SDK root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1# cd opencl/OpenCL/ ; ls bin common doc Makefile releaseNotesData Samples.html src root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL# make a - obj/release/oclUtils.cpp.o make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/common' make[1]: Entering directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/shared' ar: `u' modifier ignored since `D' is the default (see `U') r - obj/x86_64/release/shrUtils.cpp.o r - obj/x86_64/release/rendercheckGL.cpp.o r - obj/x86_64/release/cmd_arg_reader.cpp.o make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/shared' make[1]: Entering directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/src/oclBandwidthTest' make[1]: Leaving directory '/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/src/oclBandwidthTest' ... ``` The binaries made using the `Makefile` can then be run from their directory: ```bash root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL# cd bin/linux/release/ root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/bin/linux/release# ls oclBandwidthTest oclBandwidthTest.txt oclBlackScholes oclConvolutionSeparable oclCopyComputeOverlap oclDCT8x8 oclDeviceQuery oclDotProduct oclDXTCompression oclFDTD3d oclHiddenMarkovModel oclHistogram oclInlinePTX oclMatrixMul oclMatVecMul oclMersenneTwister oclMultiThreads oclQuasirandomGenerator oclRadixSort oclReduction oclScan oclSimpleMultiGPU oclSortingNetworks oclTranspose oclTridiagonal oclVectorAdd root@node001:/cm/shared/apps/cuda12.8/sdk/12.8.1/opencl/OpenCL/bin/linux/release# for i in ocl*; do ./$i ; done [oclBandwidthTest] starting... ./oclBandwidthTest Starting... Running on... Tesla V100-SXM3-32GB Quick Mode Host to Device Bandwidth, 1 Device(s), Paged memory, direct access Transfer Size (Bytes) Bandwidth(MB/s) 33554432 5528.1 Device to Host Bandwidth, 1 Device(s), Paged memory, direct access Transfer Size (Bytes) Bandwidth(MB/s) 33554432 3915.9 Device to Device Bandwidth, 1 Device(s) Transfer Size (Bytes) Bandwidth(MB/s) 33554432 724307.1 [oclBandwidthTest] test results... PASSED > exiting in 3 seconds: 3...2...1...done! ``` ...*And so on. The other binaries listed should also run and pass.*... ## 9.4 NVIDIA Validation Suite (Package: `cuda-dcgm-nvvs`) The NVIDIA Validation Suite (NVVS) runs diagnostics to validate the NVIDIA software components. Running the `nvvs` binary directly is deprecated. The diagnostics for CUDA 12.8 are run instead as part of the `dcgmi diag` command. The package for NVVS is not installed by default. It is also not available for RHEL8 and derivatives. If the package is not installed, then an attempt to run the `dcgmi` utility fails: **Example** ```bash [root@basecm11 ~] module load cuda-dcgm [root@basecm11 ~]# dcgmi diag -r 1 Error: Diagnostic could not be run because the Tesla recommended driver is not being used. ``` The package can be installed with: **Example** ```bash [root@basecm11 ~] module load cuda-dcgm [root@basecm11 ~] yum install cuda-dcgm-nvvs ``` After it is installed, the node on which the installation is done must be rebooted. Running the diagnostic after the reboot should display output similar to: **Example** ```bash [root@basecm11 ~] module load cuda-dcgm [root@basecm11 ~]# dcgmi diag -r 3 Successfully ran diagnostic for group. +---------------------------+------------------------------------------------+ | Diagnostic | Result | +===========================+================================================+ |----- Metadata ----------+------------------------------------------------| | DCGM Version | 4.1.1 | | Driver Version Detected | 570.82 | | GPU Device IDs Detected | 2941, 2941, 2941, 2941 | | Number of CPUs Detected | 2 | |----- Deployment --------+------------------------------------------------| | software | Pass | | | GPU0: Pass | | | GPU1: Pass | | | GPU2: Pass | | | GPU3: Pass | +----- Hardware ----------+------------------------------------------------+ | memory | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | | diagnostic | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | | nvbandwidth | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | +----- Integration -------+------------------------------------------------+ | pcie | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | +----- Stress ------------+------------------------------------------------+ | memory_bandwidth | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | | targeted_stress | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | | targeted_power | Skip | | | GPU0: Skip | | | GPU1: Skip | | | GPU2: Skip | | | GPU3: Skip | +---------------------------+------------------------------------------------+ ``` ## 9.5 Further NVIDIA Configuration Via The Cluster Manager After the installation and verification has been carried out for CUDA drivers, further configuration as part of CMDaemon management can be carried out, as explained in [section 3.17.2 of the *Administrator Manual*](/admin/configuring#3172-configuring-gpu-settings).