Skip to navigation

Introduction And Machine Learning Packages Installation

View as Markdown
Download PDF

1.1 Introduction

A number of machine learning and deep learning library and framework packages can be used with BCM. The packages provided make it faster and easier for organizations to install the latest state-of-the-art libraries, and gain insights from rich, complex data.

1.1.1 Associated Repository, And Support For Machine Learning Packages

The packages are distributed via a separate BCM cm-ml repository. The repository is automatically available and enabled.

The support team supports the integration of packages distributed in the dedicated cm-ml repository with BCM only if the cluster is licensed for the Data Science Add-on package.

An administrator who is upgrading a cluster that has machine learning and deep learning packages installed on it should always make sure that the dedicated BCM cm-ml repository is accessible, if required by the new NVIDIA Base Command Manager version.

1.1.2 NVIDIA Base Command Manager Versions—Supported Distributions And Architecture For Machine Learning Packages

At the time of writing of this section, October 2024, a number of different Linux distribution versions and architectures are supported, depending on the NVIDIA Base Command Manager version. For convenience, a support matrix for this is shown in table 1.1:

Table 1.1: Supported Linux distributions and architectures for BCM Machine Learning packages

BCM versionArchitecturesLinux distributions
8.0x86_64CentOS 7, RHEL 7
8.1x86_64CentOS 7, RHEL 7
8.2x86_64CentOS 7, RHEL 7, Ubuntu 18.04
9.0x86_64CentOS 7, Rocky 8, RHEL 7, RHEL 8, SLES 15, Ubuntu 18.04
9.1x86_64CentOS 7, RHEL 7, RHEL 8, Rocky 8, SLES 15, Ubuntu 18.04, Ubuntu 20.04
9.2x86_64CentOS 7, RHEL 7, RHEL 8, Rocky 8, RHEL 9, Rocky 9, SLES 15, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04
10.0x86_64RHEL 8, RHEL9, Rocky 8, Rocky 9, SLES 15, Ubuntu 20.04, Ubuntu 22.04

An updated list of the supported Linux distributions and architectures available for the various NVIDIA Base Command Manager versions can be found at https://support.brightcomputing.com/feature-matrix/, in the section of the Feature column dedicated to machine learning.

1.2 Available Packages

An updated list of the machine learning packages available for the various NVIDIA Base Command Manager versions can be found at https://service.bcm.nvidia.com/packages-dashboard/. Most of the machine learning packages are to be found within the ml group. However, some of them are found within the cm group for legacy reasons.

At the time of writing, December 2021, the following packages were available:

Table 1.2: Machine Learning packages provided by the NVIDIA Base Command Manager repositories

Package nameDescription
cm-cub-cuda11.8A flexible library of cooperative threadblock primitives and other utilities for CUDA kernel programming.
cm-fastai2-py39-cuda11.8-gcc11A library that simplifies training fast and accurate neural nets using modern best practices.
cm-gpytorch-py39-cuda11.8-gcc11A Gaussian process library implemented using PyTorch.
cm-ml-distdeps-cuda11.8Meta-package containing distribution-specific dependencies for machine learning frameworks.
cm-ml-pythondeps-py39-cuda11.8-gcc11Meta-package containing Python dependencies for machine learning frameworks. It includes, for example, NumPy, PyCUDA, pandas, NetworkX, pydot and Matplotlib.
cm-nccl2-cuda11.8-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.0-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.1-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.2-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.3-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.4-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.5-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-nccl2-cuda12.6-gcc11Version 2 of NCCL, an implementation of multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.
cm-onnx-pytorch-py39-cuda11.8-gcc11An open format built to represent machine learning models.
cm-opencv4-py39-cuda11.8-gcc11An open-source BSD-licensed library that includes several hundreds of computer vision algorithms.
cm-protobuf3-gcc11Version 3 of Protocol Buffers, a language-neutral, platform-neutral extensible mechanism for serializing structured data.
cm-pytorch-extra-py39-cuda11.8-gcc11A collection of models, libraries, dataset and useful extra functionality for PyTorch.
cm-pytorch-py39-cuda11.8-gcc11An optimized tensor library for deep learning using GPUs and CPUs. It provides tensor computation (like NumPy) with strong GPU acceleration, and deep neural networks built on a tape-based autograd system. It now includes Caffe2.
cm-tensorflow2-py39-cuda11.8-gcc11An open source software library for numerical computation using data flow graphs originally developed by researchers and engineers working on the Google Brain team.

Legend:

Packages are available for every distribution unless otherwise tagged.

1.2.1 Considerations

There are some considerations that the cluster administrator should be aware of with the packages.

  • Some packages may be labelled in the table 1.2 as deprecated. “Deprecated” in the software industry is not a well-defined term. Here it is used by BCM to mean a package that may no longer be offered in a future release, or for which a newer existing version is preferred.

  • Several different packages may be provided for the same machine learning library or framework. For example, TensorFlow may be provided by:

    • cm-tensorflow2-py39-cuda11.2-gcc9

    • cm-tensorflow2-py39-cuda11.7-gcc9

    As is the norm with other package management systems in the software industry, the name given to a BCM package includes the most relevant dependencies required to build and use it. The dependencies commonly highlighted in this manner are:

    • Python interpreter version used (e.g. *-py36*, *-py37* and *-py39*)

    • accelerator library used (e.g. *-cuda10.1*, *-cuda10.2*, *-cuda11.2*, *-cuda11.7*, and *-mkl*)

    • compiler used (e.g. *-gcc*, *-gcc8* and *-gcc9*)

    The availability of different variants of the same package makes it easier for administrators to set up a working environment that is suited to their needs.

  • Machine learning packages are designed to coexist, and can therefore all be installed at the same time. This also applies to different variants of the same library or framework.

    This means that administrators can install different versions of the same machine learning library or framework, simply by using different variants. For example: an older *-py36* version of TensorFlow, as well as a more recent *-py37* version of TensorFlow, can both be installed at the same time.

  • As is done with other packages provided by the BCM developers, the updates released for machine learning libraries and frameworks generally leave their major versions unchanged.

    Whenever a major version for a third party machine learning library or a framework is publicly released, a new package or a set of packages is typically placed in the repository.

    Such package(s) imply or contain a reference to the major version in the name. For example:

    • cm-tensorflow-* is the name used for TensorFlow major version 1

    • cm-tensorflow2-* is the name for TensorFlow major version 2

    As a result, administrators can safely upgrade cluster packages without breaking backward compatibility with users’ applications.

  • MPI modules and libraries should not be blindly added by the cluster administrator. During module loading, warnings are typically given to suggest an MPI library (Open MPI, or an MPICH or MVAPICH implementation of Open MPI) is required. However, the exact implementation of the MPI library that can be used depends upon the hardware (GPU, interface, architecture) used and requires judgment of suitability by the cluster administrator. BCM uses the cm-openmpi4-cuda11.2-ofed51-gcc9 package in this manual as the reference MPI library implementation. This driver package corresponds with using Open MPI with Gigabit Ethernet networking, InfiniBand networking, and NVIDIA GPUs.

1.3 Requirements

The following requirements must be met before installing the preceding machine learning packages.

  • RHEL users must have access to the YUM repositories and EPEL repository.

  • There must be enough free space for the packages that are installed on the head node and compute nodes. The actual amount depends on the packages installed.

  • 8 GB of RAM on the nodes is the minimum recommended amount.

  • In order to use packages built with the CUDA toolkit accelerator library version 10.2 and below (e.g. *-cuda10.2*), the NVIDIA GPUs must be Maxwell or more recent, with compute capability 3.5 or later. CUDA compute capability 6.0 or later is recommended.

  • In order to use packages built with the CUDA toolkit accelerator library version 11.0 and above (e.g. *-cuda11.2*), the NVIDIA GPUs must be Maxwell or more recent, with compute capability 5.2 or later. CUDA compute capability 6.0 or later is recommended.

  • In order to use packages built using CUDA as the accelerator (i.e. *-cuda*), the CPU must support the AVX/AVX2, FMA, and SSE4.2 instructions. This can be checked by inspecting the CPU flags:

    Example

        [root@node ~]# egrep -m1 -o '(avx|avx2|fma|sse4_2)' /proc/cpuinfo
        fma
        sse4_2
        avx
        avx2

  • In order to use packages built using MKL as the accelerator (i.e. *-mkl), the CPU must support the AVX-512 Vector Neural Network Instructions (VNNI). Examples of such CPUs are Intel Xeon Scalable processors with Deep Learning Boost.

1.4 Machine Learning Packages Installation

1.4.0.1 Head Node Installation

BCM machine learning packages are installed in the /cm/shared directory, which is by default exported over NFS. Packages installed on the head node are therefore also available to all the compute nodes by default.

The .rpm and .deb files have proper dependencies defined. This means that the cluster administrator does not need to spend time figuring out what needs to be installed to set up a working environment. Whenever a package is installed or updated, the required dependencies will be also automatically fetched, if necessary. As a result, packages can be installed with the usual package manager that is provided by the Linux distribution in the usual way (here).

For example, the administrator can install cm-pytorch-py39-cuda11.8-gcc11 as follows:

Example

[root@basecm10 ~]# yum install cm-pytorch-py39-cuda11.8-gcc11       #on RHEL 8 and 9
[root@basecm10 ~]# zypper install cm-pytorch-py39-cuda11.8-gcc11    #on SLES 15
[root@basecm10 ~]# apt-get install cm-pytorch-py39-cuda11.8-gcc11   #on Ubuntu 22.04

The package managers also automatically install the corresponding package dependencies, such as

  • cm-cudnn8.6-cuda11.8

  • cm-ml-pythondeps-py39-cuda11.8-gcc11

  • cm-nccl2-cuda11.8-gcc11

  • cm-protobuf3-gcc11

  • cuda11.8-toolkit

Machine learning packages share several dependencies, usually providing useful Python or system libraries. For convenience, these dependencies are grouped in the cm-ml-pythondeps-* and cm-ml-distdeps-* meta-packages.

  • cm-ml-pythondeps-*: This meta-package provides the application libraries such as numba, numpy, scikit-learn, and scipy.

    • If RHEL8 is being run, and cm-ml-pythondeps-py39-cuda11.8-gcc11 is being installed, then the codeready-builder-for-rhel-8-x86_64-rpms repository needs to be enabled. For example using:
      subscription-manager repos --enable=codeready-builder-for-rhel-8-x86_64-rpms

  • cm-ml-distdeps-*: This meta-package, on the other hand, provides development libraries such as blas-devel, libjpeg-devel and libpng-devel, and the utility library gnuplot.

The appropriate meta-packages are automatically installed whenever a package installation requires it.

Administrators only need to make sure that their clusters meet the preceding hardware requirements listed at the start of section 1.3. If that is not done, then unexpected failures may occur during run time, such as segmentation faults.

Examples of common mistakes are

  • using packages requiring CUDA (e.g. cm-pytorch-py39-cuda11.8-gcc11) on clusters without GPUs

  • using packages requiring VNNI (e.g. cm-tensorflow2-py39-cuda11.8-gcc11) on CPUs not supporting the instruction set

1.4.0.2 Compute Nodes Installation

The cm-ml-distdeps-* meta-packages must be also installed onto all compute nodes that are to run machine learning applications.

For example, if the name of the software image is gpu-image, then the administrator can install cm-ml-distdeps-cuda11.8 on RHEL 9 as follows:

Example

[root@basecm10 ~]# yum install --installroot=/cm/images/gpu-image cm-ml-distdeps-cuda11.8

The preceding command must be applied to all software images that are used to run machine learning applications.

There are equivalents to the --installroot option of yum for the other distribution package managers.

For SLES the installation command equivalent is:

[root@basecm10 ~]# zypper --root /cm/images/gpu-image install cm-ml-distdeps-cuda11.8

For Ubuntu the installation command equivalent is:

[root@basecm10 ~]# cm-chroot-sw-img /cm/images/gpu-image
[root@basecm10 ~]# apt install cm-ml-distdeps-cuda11.8
[root@basecm10 ~]# exit     #get out of chroot

Details on using zypper and apt commands for installation to software images are given here.

The preceding command must be applied to all software images that are used to run machine learning applications.

No automatic install of cuda-driver and cuda-dcgm since cm-ml-distdeps-cuda-* v3.0.0:

The cuda-driver and cuda-dcgm packages used to be automatically installed during installation of earlier versions of the cm-ml-distdeps-cuda-* meta-package. This behavior has changed.

Version 3.0.0 onward of the package requires a manual install of the cuda-driver and cuda-dcgm packages. Installation of the cuda-driver and cuda-dcgm packages is covered in section 9.1 of the Installation Manual.

The version number of the available or installed cm-ml-distdeps-cuda* package can be found with the package manager. For example using yum info:

Example

[root@basecm10 ~]# yum info cm-ml-distdeps-cuda*
...
Available Packages
Name         : cm-ml-distdeps-cuda11.7
Version      : 3.0.1
...

A distribution-agnostic way to find the version is with the packages command (section 11.1.1 of the Administrator Manual):

Example

[basecm10->device[basecm10]]% packages -f cm-ml-distdeps-cuda11.2 -n basecm10
Node     Type     Name                     Version  Release                  Arch     Size
-------- -------- ------------------------ -------- ------------------------ -------- --------
basecm10 rpm      cm-ml-distdeps-cuda11.2  3.0.1    100092_cm9.2_db74c57a10  x86_64   0B

The manual installation of the cuda-driver and cuda-dcgm packages allows a wider range of NVIDIA drivers and cluster configurations to be installed. Version 3.0.0 onward now allows the administrator to install custom NVIDIA drivers, for example for special hardware such as DGX machines. It also allows the administrator to install different versions of NVIDIA drivers for different groups of compute nodes.

Just as for the cm-ml-distdeps-* meta-packages, the custom NVIDIA drivers must be installed onto all the compute nodes that are to run machine learning applications.

1.4.1 Module Loading

BCM provides environment module definitions for all the machine learning packages. The environment module files are also compatible with the Lmod software introduced in NVIDIA Base Command Manager 7.3. They can be listed once the shared module is loaded, if it has not already been loaded:

[root@basecm10 ~]# module purge; module available
------------------------------------- /cm/local/modulefiles --------------------------------------
boost/1.77.0              cm-setup/9.2       dot              luajit        openldap
cluster-tools/9.2         cmake-gcc9/3.21.3  freeipmi/1.6.8   mariadb-libs  python3
cm-bios-tools             cmd                gcc/11.2.0       module-git    python39
cm-image/9.2              cmjob              ipmitool/1.8.18  module-info   shared
cm-scale/cm-scale.module  cmsh               lua/5.4.4        null
[root@basecm10 ~]# module load shared; module available
------------------------------------- /cm/local/modulefiles --------------------------------------
boost/1.77.0              cm-setup/9.2       dot              luajit        openldap
cluster-tools/9.2         cmake-gcc9/3.21.3  freeipmi/1.6.8   mariadb-libs  python3
cm-bios-tools             cmd                gcc/11.2.0       module-git    python39
cm-image/9.2              cmjob              ipmitool/1.8.18  module-info   shared
cm-scale/cm-scale.module  cmsh               lua/5.4.4        null
------------------------------------- /cm/shared/modulefiles -------------------------------------
blacs/openmpi/gcc/64/1.1patch03  intel-tbb-oss/ia32/2021.4.0                 ucx/1.10.1
blas/gcc/64/3.10.0               intel-tbb-oss/intel64/2021.4.0
bonnie++/2.00a                   iozone/3_492
cm-pmix3/3.1.4                   lapack/gcc/64/3.10.0
cm-pmix4/4.1.1                   ml-pythondeps-py39-cuda11.2-gcc9/4.8.1
cuda11.2/blas/11.2.2             mpich/ge/gcc/64/3.4.2
cuda11.2/fft/11.2.2              mvapich2/gcc/64/2.3.7
cuda11.2/toolkit/11.2.2          nccl2-cuda11.2-gcc9/2.14.3
cudnn8.1-cuda11.2/8.1.1.33       netcdf/gcc/64/gcc/64/4.8.1
default-environment              netperf/2.7.0
fftw3/openmpi/gcc/64/3.3.10      openblas/dynamic/(default)
gcc9/9.5.0                       openblas/dynamic/0.3.18
gdb/10.2                         opencv4-py39-cuda11.2-gcc9/4.5.4
globalarrays/openmpi/gcc/64/5.8  openmpi/gcc/64/4.1.2
hdf5/1.12.1                      openmpi4-cuda11.2-ofed51-gcc9/4.1.4
hdf5_18/1.8.21                   openmpi4/gcc/4.1.2
hpcx/mlnx-ofed51/2.7.4           protobuf3-gcc9/3.9.2
hwloc/1.11.11                    tensorflow2-extra-py39-cuda11.2-gcc9/2.7.0
hwloc2/2.7.1                     tensorflow2-py39-cuda11.2-gcc9/2.7.0

For example, after having installed the cm-tensorflow2-py39-cuda11.8-gcc11 package, the associated TensorFlow module can be loaded with:

[root@basecm10 ~]# module load tensorflow2-py39-cuda11.8-gcc11
Loading tensorflow2-py39-cuda11.8-gcc11/2.11.0
    Loading requirement: openblas/dynamic/0.3.18 hdf5_18/1.8.21 gcc11/11.3.0 python39\
    cuda11.8/toolkit/11.8.0 cudnn8.6-cuda11.8/8.6.0.163 ml-pythondeps-py39-cuda11.8-gcc11/4.12.0\
    protobuf3-gcc11/3.9.2 nccl2-cuda11.8-gcc11/2.16.5

The machine learning environment modules automatically load additional environment modules as dependencies, with the notable exception of Open MPI implementations for the reasons given in section 1.2.1.

The module dependencies are achieved via the module definition files:

Example

[root@basecm10 ~]#  module show tensorflow2-py39-cuda11.8-gcc11/
-------------------------------------------------------------------
/cm/shared/modulefiles/tensorflow2-py39-cuda11.8-gcc11/2.11.0:
module-whatis   {adds TensorFlow2 to your environment variables}
module          load ml-pythondeps-py39-cuda11.8-gcc11
module          load protobuf3-gcc11
module          load cudnn8.1-cuda11.8
module          load nccl2-cuda11.8-gcc11
prepend-path    PYTHONPATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\
    site-packages/
prepend-path    PYTHONPATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\
    site-packages/
prepend-path    LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\
    site-packages/tensorflow
prepend-path    LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib/python3.9/\
    site-packages/tensorflow_core
prepend-path    LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\
    site-packages/tensorflow
prepend-path    LD_LIBRARY_PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/lib64/python3.9/\
    site-packages/tensorflow_core
prepend-path    PATH /cm/shared/apps/tensorflow2-py39-cuda11.8-gcc11/2.11.0/bin
-------------------------------------------------------------------

1.5 Machine Learning Packages Removal

Machine learning packages can be removed in the usual way with the package manager commands used by the Linux distribution. For example, the administrator can remove the cm-pytorch-py37-cuda10.2-gcc8 package with:

Example

[root@basecm10 ~]# yum remove cm-pytorch-py39-cuda11.8-gcc11       #on RHEL 8 and 9

or

[root@basecm10 ~]# zypper remove cm-pytorch-py39-cuda11.8-gcc11    #on SLES 15

or

[root@basecm10 ~]# apt-get remove cm-pytorch-py39-cuda11.8-gcc11   #on Ubuntu 22.04

BCM machine learning packages are installed in the /cm/shared directory, which is by default exported over NFS. Packages removed from the head node are therefore also removed from all the compute nodes by default.

The cm-ml-distdeps-* meta-packages must be also removed from all compute nodes that are to run machine learning applications. For example, if the name of the software image is gpu-image, then the images directory is /cm/images/gpu-image, and then the administrator can remove cm-ml-distdeps-cuda11.8 from the image as follows in RHEL 9:

Example

[root@basecm10 ~]# yum remove --installroot=/cm/images/gpu-image cm-ml-distdeps-cuda11.8

The preceding command must be applied to all software images that are used to run the machine learning applications.

The equivalents to the --installroot option of yum for the other distribution package managers are described in section 1.4.0.2.

Download this manual as a PDF