Docker Engine
Docker integration with NVIDIA Base Command Manager 10 for Docker version 26.1.5 is available at the time of writing of this paragraph (October 2024) on the x86_64 architecture for all the BCM-supported Linux distributions. For a more up-to-date status, the feature matrix at https://service.bcm.nvidia.com/feature-matrix/ can be checked.
Docker Engine (or just Docker) is a tool for container management. Docker allows containers and their images to be created, controlled, and monitored on a host using Docker command line tools or the Docker API.
Swarm mode, which allows containers to spawn on several hosts, is not formally supported by NVIDIA Base Command Manager 10. This is because NVIDIA Base Command Manager 10 provides Kubernetes for this purpose instead.
Docker provides a utility called docker, and two daemons: one called containerd (the default provided by BCM), and the other called dockerd. Additional functionality includes pulling the container image from a specific image registry (Chapter 3), configuring the container network, setting systemd limits, and attaching volumes.
2.1 Docker Setup
BCM provides the cm-docker package. The package includes the following components:
-
Docker itself, that provides an API and delegates the container management to Containerd;
-
Containerd runtime, that manages OCI images and OCI containers (via runC);
-
runC, a CLI tool for spawning and running containers according to the OCI specification runtime;
-
docker-py, a Python library for the Docker API.
Typically, however, the administrator is expected to simply run the cm-docker-setup utility, which is provided by BCM’s cm-setup package. Running cm-docker-setup takes care of the installation of the cm-docker package and also takes care of Docker setup. If run without options then the utility starts up a TUI dialog (figure 2.1).
Figure 2.1: cm-docker-setup TUI startup
The cm-docker-setup utility asks several questions, such as which Docker registries are to be used, what nodes Docker is to be installed on, whether the NVIDIA container runtime should be installed, and so on. If cm-docker-setup is used with the -c option, and given a YAML configuration file <YAMLfile>, then a runtime configuration is loaded from that file. The YAML file is typically generated and saved from an earlier run.
When the questions in the TUI dialog have been answered and the deployment is carried out, the utility:
-
installs the
cm-dockerpackage, if it has not been installed yet -
then assigns the DockerHost role to the node categories or head nodes that were specified
-
adds health checks to the BCM monitoring configuration
-
performs the initial configuration of Docker.
The regular nodes on which Docker is to run, are restarted by the utility, if needed. The restart operation provisions the updated images from the image directory onto the nodes.
The cm-docker package also includes a modules environment file, which must be loaded in order to use the docker command. The modules environment and modules are introduced in section 2.2 of the Administrator Manual.
By default only the administrator can run the docker commands after setup (some output ellipsized):
Example
and the hello-world image can be run as usual with:
Example
Or, for example, importing and running Apache containers with Docker may result in the following output:
Example
Using Docker directly means being root on the host. It is rarely sensible to carry out regular user actions as the root user at all times.
So, to make Docker available to regular users, Kubernetes provides a user management layer and restrictions.
After Docker has been installed, Kubernetes can be set up to allow regular user access to the Docker containers as covered in Chapter 4. It is a best practice for regular users to use Kubernetes instead of Docker commands directly.
2.2 Integration With Workload Managers
BCM does not provide integration of Docker with workload managers. The administrator can however tune the workload managers in some cases to enable Docker support.
-
LSF – An open beta version of LSF with Docker support is available from the IBM web site. This LSF version allows jobs to run in Docker containers, and monitors the container resources per job.
-
PBS Professional – Altair provides a hook script that allows jobs to start in Docker containers. Altair should be contacted to obtain the script and instructions.
2.3 DockerHost Role
When cm-docker-setup is executed, the DockerHost role is assigned to nodes or categories. The DockerHost role is responsible for Docker service management and configuration.
From cmsh, the configuration parameters can be managed from the Docker::Host role:
Example
The Docker host parameters that CMDaemon can configure in the DockerHost role, along with a description, are shown in table 2.1:
* Boolean (takes yes or no as a value)
Table 2.1: Docker::Host role options
2.4 Iptables
By default iptables rules have been added to nodes that function as a Docker host, to let network traffic go from the containers to outside the pods network. If this conflicts with other software that uses iptables, then this option can be disabled. For example, if the docker::host role has already been assigned to the nodes via the default category, then the iptables rules that are set can be disabled by setting the iptables parameter in the Docker::Host role to no:
Example
2.5 Storage Backends
A core part of the Docker model is the efficient use of containers based on layered images. To implement this, Docker provides different storage back ends, also called storage drivers. These storage back ends rely heavily on various filesystem features in the kernel or volume manager. Some storage back ends perform better than others, depending on the circumstances.
The storage back end configured by cm-docker-setup is overlay2. It is the only storage back end that cm-docker-setup accepts, and the only one that the Docker Engine shipped with recent BCM 10 releases supports.
Tab-completion in the storagebackends submode shows the back end types that can be added:
Example
The aufs and devicemapper types are still offered by cmsh, but they must not be added. Docker Engine 24.0 removed the aufs storage driver, and Docker Engine 25.0 removed devicemapper. From BCM 10.24.09 onwards, BCM 10 ships Docker Engine 26 or later, so only overlay2 back ends are supported. Earlier BCM 10 releases shipped Docker Engine 24, which still supports devicemapper.
The docker info command, amongst many other items, shows the storage driver and related settings that are being used in Docker:
Example
Docker data volumes are not controlled by the storage driver. Reads and writes to data volumes bypass the storage driver. It is possible to mount any number of data volumes into a container. Multiple containers can also share one or more data volumes.
More information about Docker storage back ends is available at https://docs.docker.com/engine/userguide/storagedriver.
Settings for the overlay2 back end can be added as options if needed. In cmsh this is done by setting the options parameter in the storagebackends submode of the docker::host role.
2.6 Docker Monitoring
When cm-docker-setup runs, it configures and runs the following Docker health checks:
-
makes a test API call to the endpoint of the Docker daemon
-
checks containers to see that none is in a
deadstate
The Docker daemon can be monitored outside of BCM with the usual commands directly.
BCM ways to manage or check on Docker include the following:
In CMDaemon, the docker service can be checked:
Example
The docker0 interface statistics can be checked within the nodeoverview output:
Example
The health check measurable Docker checks if the docker service is healthy.
Example
2.7 Docker Setup For NVIDIA
NVIDIA GPU Cloud (NGC) is a cloud platform that runs on NVIDIA GPUs. NGC containers are lightweight containers that run applications on NVIDIA GPUs. The applications are typically HPC, machine learning, or deep learning applications.
An NGC can be set up to run NGC containers from the registry http://ngc.nvidia.com.
Docker can be configured as an NGC running NGC containers by using the NVIDIA Container Toolkit.
The BCM package provided for this is: cm-nvidia-container-toolkit.
One way to install and deploy this package is as part of the Docker installation, when running cm-docker-setup (section 2.1), where the cluster administrator selects yes as the answer to the request: “Do you want to install the NVIDIA Runtime for Docker”.
Alternatively, if Docker has already been installed via cm-docker-setup, and if the package has not been installed, then it can be installed via the package manager, yum or apt. The toolkit has to be running on the compute nodes that have GPUs, which means that the installation must go to the appropriate node image (section 11.4 of the Administrator Manual). For example, if the appropriate image is gpu-image, then the package manager command for RHEL-based distributions would be:
Example
The nodes that use that GPU image can then be rebooted to pick up the new package.
The GPU status can then be printed with the NVIDIA system management interface command. For example, if the image has been picked up by node001:
Example
The available CUDA Docker images can be found at https://hub.docker.com/r/nvidia/cuda.