> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Introduction And Machine Learning Packages Installation Download PDF ## 1.1 Introduction A number of machine learning and deep learning library and framework packages can be used with BCM. The packages provided make it faster and easier for organizations to install the latest state-of-the-art libraries, and gain insights from rich, complex data. ### 1.1.1 Associated Repository, And Support For Machine Learning Packages The packages are distributed via a separate BCM `cm-ml` repository. The repository is automatically available and enabled. The support team supports the integration of packages distributed in the dedicated `cm-ml` repository with BCM only if the cluster is licensed for the Data Science Add-on package. An administrator who is upgrading a cluster that has machine learning and deep learning packages installed on it should always make sure that the dedicated BCM `cm-ml` repository is accessible, if required by the new NVIDIA Base Command Manager version. ### 1.1.2 NVIDIA Base Command Manager Versions—Supported Distributions And Architecture For Machine Learning Packages At the time of writing of this section, October 2024, a number of different Linux distribution versions and architectures are supported, depending on the NVIDIA Base Command Manager version. For convenience, a support matrix for this is shown in table [1.1](#ml-archs-and-distros): *Table 1.1: Supported Linux distributions and architectures for BCM Machine Learning packages*
**BCM version** **Architectures** **Linux distributions**
8.0 x86_64 CentOS 7, RHEL 7
8.1 x86_64 CentOS 7, RHEL 7
8.2 x86_64 CentOS 7, RHEL 7, Ubuntu 18.04
9.0 x86_64 CentOS 7, Rocky 8, RHEL 7, RHEL 8, SLES 15, Ubuntu 18.04
9.1 x86_64 CentOS 7, RHEL 7, RHEL 8, Rocky 8, SLES 15, Ubuntu 18.04, Ubuntu 20.04
9.2 x86_64 CentOS 7, RHEL 7, RHEL 8, Rocky 8, RHEL 9, Rocky 9, SLES 15, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04
10.0 x86_64 RHEL 8, RHEL9, Rocky 8, Rocky 9, SLES 15, Ubuntu 20.04, Ubuntu 22.04
An updated list of the supported Linux distributions and architectures available for the various NVIDIA Base Command Manager versions can be found at [https://support.brightcomputing.com/feature-matrix/](https://support.brightcomputing.com/feature-matrix/), in the section of the `Feature` column dedicated to machine learning. ## 1.2 Available Packages An updated list of the machine learning packages available for the various NVIDIA Base Command Manager versions can be found at [https://service.bcm.nvidia.com/packages-dashboard/](https://service.bcm.nvidia.com/packages-dashboard/). Most of the machine learning packages are to be found within the `ml` group. However, some of them are found within the `cm` group for legacy reasons. At the time of writing, December 2021, the following packages were available: *Table 1.2: Machine Learning packages provided by the NVIDIA Base Command Manager repositories* | **Package name** | **Description** | | :------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cm-cub-cuda11.8` | A flexible library of cooperative threadblock primitives and other utilities for CUDA kernel programming. | | `cm-fastai2-py39-cuda11.8-gcc11` | A library that simplifies training fast and accurate neural nets using modern best practices. | | `cm-gpytorch-py39-cuda11.8-gcc11` | A Gaussian process library implemented using PyTorch. | | `cm-ml-distdeps-cuda11.8` | Meta-package containing distribution-specific dependencies for machine learning frameworks. | | `cm-ml-pythondeps-py39-cuda11.8-gcc11` | Meta-package containing Python dependencies for machine learning frameworks. It includes, for example, NumPy, PyCUDA, pandas, NetworkX, pydot and Matplotlib. | | `cm-nccl2-cuda11.8-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.0-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.1-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.2-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.3-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.4-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.5-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-nccl2-cuda12.6-gcc11` | Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. | | `cm-onnx-pytorch-py39-cuda11.8-gcc11` | An open format built to represent machine learning models. | | `cm-opencv4-py39-cuda11.8-gcc11` | An open-source BSD-licensed library that includes several hundreds of computer vision algorithms. | | `cm-protobuf3-gcc11` | Version 3 of Protocol Buffers, a language-neutral, platform-neutral extensible mechanism for serializing structured data. | | `cm-pytorch-extra-py39-cuda11.8-gcc11` | A collection of models, libraries, dataset and useful extra functionality for PyTorch. | | `cm-pytorch-py39-cuda11.8-gcc11` | An optimized tensor library for deep learning using GPUs and CPUs. It provides tensor computation (like NumPy) with strong GPU acceleration, and deep neural networks built on a tape-based autograd system. It now includes Caffe2. | | `cm-tensorflow2-py39-cuda11.8-gcc11` | An open source software library for numerical computation using data flow graphs originally developed by researchers and engineers working on the Google Brain team. | Legend: Packages are available for every distribution unless otherwise tagged. ### 1.2.1 Considerations There are some considerations that the cluster administrator should be aware of with the packages. * Some packages may be labelled in the table [1.2](#ml-packages-included) as deprecated. “Deprecated” in the software industry is not a well-defined term. Here it is used by BCM to mean a package that may no longer be offered in a future release, or for which a newer existing version is preferred. * Several different packages may be provided for the same machine learning library or framework. For example, TensorFlow may be provided by: * `cm-tensorflow2-py39-cuda11.2-gcc9` * `cm-tensorflow2-py39-cuda11.7-gcc9` As is the norm with other package management systems in the software industry, the name given to a BCM package includes the most relevant dependencies required to build and use it. The dependencies commonly highlighted in this manner are: * Python interpreter version used (e.g. `*-py36*`, `*-py37*` and `*-py39*`) * accelerator library used (e.g. `*-cuda10.1*`, `*-cuda10.2*`, `*-cuda11.2*`, `*-cuda11.7*`, and `*-mkl*`) * compiler used (e.g. `*-gcc*`, `*-gcc8*` and `*-gcc9*`) The availability of different variants of the same package makes it easier for administrators to set up a working environment that is suited to their needs. * Machine learning packages are designed to coexist, and can therefore all be installed at the same time. This also applies to different variants of the same library or framework. This means that administrators can install different versions of the same machine learning library or framework, simply by using different variants. For example: an older `*-py36*` version of TensorFlow, as well as a more recent `*-py37*` version of TensorFlow, can both be installed at the same time. * As is done with other packages provided by the BCM developers, the updates released for machine learning libraries and frameworks generally leave their major versions unchanged. Whenever a major version for a third party machine learning library or a framework is publicly released, a new package or a set of packages is typically placed in the repository. Such package(s) imply or contain a reference to the major version in the name. For example: * `cm-tensorflow-*` is the name used for TensorFlow major version 1 * `cm-tensorflow2-*` is the name for TensorFlow major version 2 As a result, administrators can safely upgrade cluster packages without breaking backward compatibility with users’ applications. * MPI modules and libraries should not be blindly added by the cluster administrator. During module loading, warnings are typically given to suggest an MPI library (Open MPI, or an MPICH or MVAPICH implementation of Open MPI) is required. However, the exact implementation of the MPI library that can be used depends upon the hardware (GPU, interface, architecture) used and requires judgment of suitability by the cluster administrator. BCM uses the `cm-openmpi4-cuda11.2-ofed51-gcc9` package in this manual as the reference MPI library implementation. This driver package corresponds with using Open MPI with Gigabit Ethernet networking, InfiniBand networking, and NVIDIA GPUs. ## 1.3 Requirements The following requirements must be met before installing the preceding machine learning packages. * RHEL users must have access to the YUM repositories and EPEL repository. * There must be enough free space for the packages that are installed on the head node and compute nodes. The actual amount depends on the packages installed. * 8 GB of RAM on the nodes is the minimum recommended amount. * In order to use packages built with the CUDA toolkit accelerator library version 10.2 and below (e.g. `*-cuda10.2*`), the NVIDIA GPUs must be Maxwell or more recent, with compute capability 3.5 or later. CUDA compute capability 6.0 or later is recommended. * In order to use packages built with the CUDA toolkit accelerator library version 11.0 and above (e.g. `*-cuda11.2*`), the NVIDIA GPUs must be Maxwell or more recent, with compute capability 5.2 or later. CUDA compute capability 6.0 or later is recommended. * In order to use packages built using CUDA as the accelerator (i.e. `*-cuda*`), the CPU must support the AVX/AVX2, FMA, and SSE4.2 instructions. This can be checked by inspecting the CPU flags: **Example** ```bash [root@node ~]# egrep -m1 -o '(avx|avx2|fma|sse4_2)' /proc/cpuinfo fma sse4_2 avx avx2 ``` * In order to use packages built using MKL as the accelerator (i.e. `*-mkl`), the CPU must support the AVX-512 Vector Neural Network Instructions (VNNI). Examples of such CPUs are Intel Xeon Scalable processors with Deep Learning Boost. ## 1.4 Machine Learning Packages Installation #### 1.4.0.1 Head Node Installation BCM machine learning packages are installed in the `/cm/shared` directory, which is by default exported over NFS. Packages installed on the head node are therefore also available to all the compute nodes by default. The `.rpm` and `.deb` files have proper dependencies defined. This means that the cluster administrator does not need to spend time figuring out what needs to be installed to set up a working environment. Whenever a package is installed or updated, the required dependencies will be also automatically fetched, if necessary. As a result, packages can be installed with the usual package manager that is provided by the Linux distribution in the usual way ([here](/admin/softwareimagemanagement#11213-updatinginstalling-packages-on-the-head-node)). For example, the administrator can install `cm-pytorch-py39-cuda11.8-gcc11` as follows: **Example** ```bash [root@basecm10 ~]# yum install cm-pytorch-py39-cuda11.8-gcc11 #on RHEL 8 and 9 ``` ```bash [root@basecm10 ~]# zypper install cm-pytorch-py39-cuda11.8-gcc11 #on SLES 15 ``` ```bash [root@basecm10 ~]# apt-get install cm-pytorch-py39-cuda11.8-gcc11 #on Ubuntu 22.04 ``` The package managers also automatically install the corresponding package dependencies, such as * `cm-cudnn8.6-cuda11.8` * `cm-ml-pythondeps-py39-cuda11.8-gcc11` * `cm-nccl2-cuda11.8-gcc11` * `cm-protobuf3-gcc11` * `cuda11.8-toolkit` Machine learning packages share several dependencies, usually providing useful Python or system libraries. For convenience, these dependencies are grouped in the `cm-ml-pythondeps-*` and `cm-ml-distdeps-*` meta-packages. * `cm-ml-pythondeps-*`: This meta-package provides the application libraries such as `numba`, `numpy`, `scikit-learn`, and `scipy`. * If RHEL8 is being run, and `cm-ml-pythondeps-py39-cuda11.8-gcc11` is being installed, then the `codeready-builder-for-rhel-8-x86_64-rpms` repository needs to be enabled. For example using:\ `subscription-manager repos --enable=codeready-builder-for-rhel-8-x86_64-rpms` * `cm-ml-distdeps-*`: This meta-package, on the other hand, provides development libraries such as `blas-devel`, `libjpeg-devel` and `libpng-devel`, and the utility library `gnuplot`. The appropriate meta-packages are automatically installed whenever a package installation requires it. Administrators only need to make sure that their clusters meet the preceding hardware requirements listed at the start of section [1.3](#13-requirements). If that is not done, then unexpected failures may occur during run time, such as segmentation faults. Examples of common mistakes are * using packages requiring CUDA (e.g. `cm-pytorch-py39-cuda11.8-gcc11`) on clusters without GPUs * using packages requiring VNNI (e.g. `cm-tensorflow2-py39-cuda11.8-gcc11`) on CPUs not supporting the instruction set #### 1.4.0.2 Compute Nodes Installation The `cm-ml-distdeps-*` meta-packages must be also installed onto all compute nodes that are to run machine learning applications. For example, if the name of the software image is `gpu-image`, then the administrator can install `cm-ml-distdeps-cuda11.8` on RHEL 9 as follows: **Example** ```bash [root@basecm10 ~]# yum install --installroot=/cm/images/gpu-image cm-ml-distdeps-cuda11.8 ``` The preceding command must be applied to all software images that are used to run machine learning applications. There are equivalents to the `--installroot` option of `yum` for the other distribution package managers. For SLES the installation command equivalent is: ```bash [root@basecm10 ~]# zypper --root /cm/images/gpu-image install cm-ml-distdeps-cuda11.8 ``` For Ubuntu the installation command equivalent is: ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/gpu-image [root@basecm10 ~]# apt install cm-ml-distdeps-cuda11.8 [root@basecm10 ~]# exit #get out of chroot ``` Details on using `zypper` and `apt` commands for installation to software images are given [here](/admin/softwareimagemanagement#11213-updatinginstalling-packages-on-the-head-node). The preceding command must be applied to all software images that are used to run machine learning applications. **No automatic install of `cuda-driver` and `cuda-dcgm` since `cm-ml-distdeps-cuda-*` v3.0.0:** The `cuda-driver` and `cuda-dcgm` packages used to be automatically installed during installation of earlier versions of the `cm-ml-distdeps-cuda-*` meta-package. This behavior has changed. Version 3.0.0 onward of the package requires a manual install of the `cuda-driver` and `cuda-dcgm` packages. Installation of the `cuda-driver` and `cuda-dcgm` packages is covered [in section 9.1 of the *Installation Manual*](/installation/nvidiagpus#91-installing-cuda). The version number of the available or installed `cm-ml-distdeps-cuda*` package can be found with the package manager. For example using `yum info`: **Example** ```bash [root@basecm10 ~]# yum info cm-ml-distdeps-cuda* ... Available Packages Name : cm-ml-distdeps-cuda11.7 Version : 3.0.1 ... ``` A distribution-agnostic way to find the version is with the `packages` command ([section 11.1.1 of the *Administrator Manual*](/admin/softwareimagemanagement#1111-the-packages-command)): **Example** ```bash [basecm10->device[basecm10]]% packages -f cm-ml-distdeps-cuda11.2 -n basecm10 Node Type Name Version Release Arch Size -------- -------- ------------------------ -------- ------------------------ -------- -------- basecm10 rpm cm-ml-distdeps-cuda11.2 3.0.1 100092_cm9.2_db74c57a10 x86_64 0B ``` The manual installation of the `cuda-driver` and `cuda-dcgm` packages allows a wider range of NVIDIA drivers and cluster configurations to be installed. Version 3.0.0 onward now allows the administrator to install custom NVIDIA drivers, for example for special hardware such as DGX machines. It also allows the administrator to install different versions of NVIDIA drivers for different groups of compute nodes. Just as for the `cm-ml-distdeps-*` meta-packages, the custom NVIDIA drivers must be installed onto all the compute nodes that are to run machine learning applications. ### 1.4.1 Module Loading BCM provides environment module definitions for all the machine learning packages. The environment module files are also compatible with the Lmod software introduced in NVIDIA Base Command Manager 7.3. They can be listed once the `shared` module is loaded, if it has not already been loaded: ```bash [root@basecm10 ~]# module purge; module available ------------------------------------- /cm/local/modulefiles -------------------------------------- boost/1.77.0 cm-setup/9.2 dot luajit openldap cluster-tools/9.2 cmake-gcc9/3.21.3 freeipmi/1.6.8 mariadb-libs python3 cm-bios-tools cmd gcc/11.2.0 module-git python39 cm-image/9.2 cmjob ipmitool/1.8.18 module-info shared cm-scale/cm-scale.module cmsh lua/5.4.4 null [root@basecm10 ~]# module load shared; module available ------------------------------------- /cm/local/modulefiles -------------------------------------- boost/1.77.0 cm-setup/9.2 dot luajit openldap cluster-tools/9.2 cmake-gcc9/3.21.3 freeipmi/1.6.8 mariadb-libs python3 cm-bios-tools cmd gcc/11.2.0 module-git python39 cm-image/9.2 cmjob ipmitool/1.8.18 module-info shared cm-scale/cm-scale.module cmsh lua/5.4.4 null ------------------------------------- /cm/shared/modulefiles ------------------------------------- blacs/openmpi/gcc/64/1.1patch03 intel-tbb-oss/ia32/2021.4.0 ucx/1.10.1 blas/gcc/64/3.10.0 intel-tbb-oss/intel64/2021.4.0 bonnie++/2.00a iozone/3_492 cm-pmix3/3.1.4 lapack/gcc/64/3.10.0 cm-pmix4/4.1.1 ml-pythondeps-py39-cuda11.2-gcc9/4.8.1 cuda11.2/blas/11.2.2 mpich/ge/gcc/64/3.4.2 cuda11.2/fft/11.2.2 mvapich2/gcc/64/2.3.7 cuda11.2/toolkit/11.2.2 nccl2-cuda11.2-gcc9/2.14.3 cudnn8.1-cuda11.2/8.1.1.33 netcdf/gcc/64/gcc/64/4.8.1 default-environment netperf/2.7.0 fftw3/openmpi/gcc/64/3.3.10 openblas/dynamic/(default) gcc9/9.5.0 openblas/dynamic/0.3.18 gdb/10.2 opencv4-py39-cuda11.2-gcc9/4.5.4 globalarrays/openmpi/gcc/64/5.8 openmpi/gcc/64/4.1.2 hdf5/1.12.1 openmpi4-cuda11.2-ofed51-gcc9/4.1.4 hdf5_18/1.8.21 openmpi4/gcc/4.1.2 hpcx/mlnx-ofed51/2.7.4 protobuf3-gcc9/3.9.2 hwloc/1.11.11 tensorflow2-extra-py39-cuda11.2-gcc9/2.7.0 hwloc2/2.7.1 tensorflow2-py39-cuda11.2-gcc9/2.7.0 ``` For example, after having installed the `cm-tensorflow2-py39-cuda11.8-gcc11` package, the associated TensorFlow module can be loaded with: ```bash [root@basecm10 ~]# module load tensorflow2-py39-cuda11.8-gcc11 Loading tensorflow2-py39-cuda11.8-gcc11/2.11.0 Loading requirement: openblas/dynamic/0.3.18 hdf5_18/1.8.21 gcc11/11.3.0 python39\ cuda11.8/toolkit/11.8.0 cudnn8.6-cuda11.8/8.6.0.163 ml-pythondeps-py39-cuda11.8-gcc11/4.12.0\ protobuf3-gcc11/3.9.2 nccl2-cuda11.8-gcc11/2.16.5 ``` The machine learning environment modules automatically load additional environment modules as dependencies, with the notable exception of Open MPI implementations for the reasons given in section [1.2.1](#121-considerations). The module dependencies are achieved via the module definition files: **Example** ```bash [root@basecm10 ~]# module show tensorflow2-py39-cuda11.8-gcc11/ ------------------------------------------------------------------- /cm/shared/modulefiles/tensorflow2-py39-cuda11.8-gcc11/2.11.0: module-whatis {adds TensorFlow2 to your environment variables} module load ml-pythondeps-py39-cuda11.8-gcc11 module load protobuf3-gcc11 module load cudnn8.1-cuda11.8 module load nccl2-cuda11.8-gcc11 prepend-path PYTHONPATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\ site-packages/ prepend-path PYTHONPATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\ site-packages/ prepend-path LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\ site-packages/tensorflow prepend-path LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\ site-packages/tensorflow_core prepend-path LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\ site-packages/tensorflow prepend-path LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\ site-packages/tensorflow_core prepend-path PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/bin ------------------------------------------------------------------- ``` ## 1.5 Machine Learning Packages Removal Machine learning packages can be removed in the usual way with the package manager commands used by the Linux distribution. For example, the administrator can remove the `cm-pytorch-py37-cuda10.2-gcc8` package with: **Example** ```bash [root@basecm10 ~]# yum remove cm-pytorch-py39-cuda11.8-gcc11 #on RHEL 8 and 9 ``` or ```bash [root@basecm10 ~]# zypper remove cm-pytorch-py39-cuda11.8-gcc11 #on SLES 15 ``` or ```bash [root@basecm10 ~]# apt-get remove cm-pytorch-py39-cuda11.8-gcc11 #on Ubuntu 22.04 ``` BCM machine learning packages are installed in the `/cm/shared` directory, which is by default exported over NFS. Packages removed from the head node are therefore also removed from all the compute nodes by default. The `cm-ml-distdeps-*` meta-packages must be also removed from all compute nodes that are to run machine learning applications. For example, if the name of the software image is `gpu-image`, then the images directory is `/cm/images/gpu-image`, and then the administrator can remove `cm-ml-distdeps-cuda11.8` from the image as follows in RHEL 9: **Example** ```bash [root@basecm10 ~]# yum remove --installroot=/cm/images/gpu-image cm-ml-distdeps-cuda11.8 ``` The preceding command must be applied to all software images that are used to run the machine learning applications. The equivalents to the `--installroot` option of `yum` for the other distribution package managers are described in section [1.4.0.2](#1402-compute-nodes-installation). --- [Download this manual as a PDF](https://docs.nvidia.com/base-command-manager/manuals/10/machine-learning-manual.pdf)