> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Docker Engine Docker integration with NVIDIA Base Command Manager 11 for Docker version 28.0.4 is available at the time of writing of this paragraph (October 2024) on the `x86_64` architecture for all the BCM-supported Linux distributions. For a more up-to-date status, the feature matrix at [https://service.bcm.nvidia.com/feature-matrix/](https://service.bcm.nvidia.com/feature-matrix/) can be checked. Docker Engine (or just Docker) is a tool for container management. Docker allows containers and their images to be created, controlled, and monitored on a host using Docker command line tools or the Docker API. *Swarm mode*, which allows containers to spawn on several hosts, is not formally supported by NVIDIA Base Command Manager 11. This is because NVIDIA Base Command Manager 11 provides Kubernetes for this purpose instead. Docker provides a utility called `docker`, and two daemons: one called `containerd` (the default provided by BCM), and the other called `dockerd`. Additional functionality includes pulling the container image from a specific image registry (Chapter [3](/containerization/dockerregistries)), configuring the container network, setting `systemd` limits, and attaching volumes. ## 2.1 Docker Setup BCM provides the `cm-docker` package. The package includes the following components: * Docker itself, that provides an API and delegates the container management to Containerd; * Containerd runtime, that manages OCI (Open Container Initiative) images and OCI containers (using runC); * runC, a CLI tool for spawning and running containers according to the OCI runtime specification; * docker-py, a Python library for the Docker API. Typically, however, the administrator is expected to simply run the `cm-docker-setup` utility, which is provided by BCM’s `cm-setup` package. Running `cm-docker-setup` takes care of the installation of the `cm-docker` package and also takes care of Docker setup. If run without options, then the utility starts up a TUI dialog (figure [2.1](#fig-cm-docker-setup-ncurses-startup)). ![cm-docker-setup TUI startup](/base-command-manager/product-manuals/_fern-img/247a5b5ce826d3328ccd62d49e57537df721629bbc545af3ac9b27a793d07f77.webp) *Figure 2.1: cm-docker-setup TUI startup* The `cm-docker-setup` utility asks several questions, such as which Docker registries are to be used, what nodes Docker is to be installed on, whether the NVIDIA container runtime should be installed, and so on. If `cm-docker-setup` is used with the `-c` option, and given a YAML configuration file \<*YAMLfile*>, then a runtime configuration is loaded from that file. The YAML file is typically generated and saved from an earlier setup run. The setup run displays a summary as its final TUI screen. In the final summary screen, as with other scripts that come with the `cm-setup` package, the configuration setting can be one of the following kinds: * `minimal config`: For this minimal kind of configuration setting, the configuration consists only of the settings that differ from the default application configuration. The remaining default application configuration settings are not included in the configuration. * `config`: For this non-minimal kind of configuration setting, the configuration consists of the minimal configuration settings (the ones changed by BCM), as well as the remaining default configuration settings (the ones provided with the application). That is to say: the remaining default configuration application settings are included in the configuration. The default application configuration settings may change as the upstream packages change. When the questions in the TUI dialog have been answered and the deployment is carried out, the utility: * installs the `cm-docker` package, if it has not been installed yet * then assigns the DockerHost role to the node categories or head nodes that were specified * adds health checks to the BCM monitoring configuration * performs the initial configuration of Docker. The regular nodes on which Docker is to run, are restarted by the utility, if needed. The restart operation provisions the updated images from the image directory onto the nodes. The `cm-docker` package also includes a modules environment file, which must be loaded in order to use the `docker` command. The modules environment and modules are introduced in [section 2.2 of the *Administrator Manual*](/admin/clustermanagement#22-modules-environment). By default only the administrator can run the `docker` commands after setup (some output ellipsized): **Example** ```bash [root@basecm11 ~]# ssh node001 [root@node001 ~]# module load docker [root@node001 ~]# docker info ... Containers: 0 Images: 0 ... Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Registry Mirrors: https://harbor-proxy.brightcomputing.com/ Live Restore Enabled: false [root@node001 ~]# ``` and the `hello-world` image can be run as usual with: **Example** ```bash [root@node001 ~]# docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 2db29710123e: Pull complete Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. ... ``` Or, for example, importing and running Apache containers with Docker may result in the following output: **Example** ```bash [root@node001 ~]# module load docker [root@node001 ~]# docker run httpd & docker run httpd & ...runs a couple of Apache containers... [root@node001 ~]# docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES acdbe2f3667b httpd "httpd-foreground" 13 seconds ago Up 11 seconds 80/tcp quizzical_bhabha 64787a8524dd httpd "httpd-foreground" 13 seconds ago Up 11 seconds 80/tcp funny_hypatia ... [root@node001 ~]# ``` Using Docker directly means being root on the host. It is rarely sensible to carry out regular user actions as the root user at all times. So, to make Docker available to regular users, Kubernetes provides a user management layer and restrictions. After Docker has been installed, Kubernetes can be set up to allow regular user access to the Docker containers as covered in Chapter [4](/containerization/kubernetes). It is a best practice for regular users to use Kubernetes instead of Docker commands directly. ## 2.2 Integration With Workload Managers BCM does not provide integration of Docker with workload managers. The administrator can however tune the workload managers in some cases to enable Docker support. * LSF – An open beta version of LSF with Docker support is available from the IBM web site. This LSF version allows jobs to run in Docker containers, and monitors the container resources per job. ## 2.3 DockerHost Role When `cm-docker-setup` is executed, the DockerHost role is assigned to nodes or categories. The DockerHost role is responsible for Docker service management and configuration. From `cmsh`, the configuration parameters can be managed from the `Docker::Host` role: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% category use default [basecm11->category[default]]% roles [basecm11->category[default]->roles]% assign docker::host [basecm11->category*[default*]->roles*[Docker::Host*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name Docker::Host Revision Type DockerHostRole Add services yes Spool /var/lib/docker Tmp dir $spool/tmp Enable SELinux yes Default Ulimits Debug no Log Level info Bridge IP Bridge MTU 0 API Sockets unix:///var/run/docker.sock Iptables yes User Namespace Remap Insecure Registries Enable TLS no Verify TLS no TLS CA TLS Certificate TLS Key Certificates Path /etc/docker Storage Backends <0 in submode> Containerd Socket Runtime runc Options [basecm11->category*[default*]->roles*[Docker::Host*]]% ``` The Docker host parameters that CMDaemon can configure in the DockerHost role, along with a description, are shown in table [2.1](#dockerhost-role-table): | **Parameter** | **Description** | | :--------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Add services`\* | Add services to nodes belonging to this node. Care must be taken if setting this to `no`. (default: `yes`) | | `Spool` | Root of the Docker runtime (default: `/var/lib/docker`) | | `Tmp dir` | Location for temporary files. Default: `$` \<*spool*>`/tmp`, where `$` \<*spool*> is replaced by the path to the Docker runtime root directory | | `Enable SELinux`\* | Enable selinux support in Docker daemon (default: `yes`) | | `Default Ulimits` | Set the default ulimit options for all containers | | `Debug`\* | Enable debug mode (default: `no`) | | `Log Level` | Set the daemon logging level. In order of increasing verbosity: `fatal`, `error`, `warn`, `info`, `debug`. (default: `info`) | | `Bridge IP` | Network bridge IP (not defined by default) | | `Bridge` | Attach containers to a network bridge (not defined by default) | | `MTU` | Set the containers network MTU, in bytes (default: `0`, which does not set the MTU at all) | | `API Sockets` | Daemon socket(s) to connect to (default: `unix:///var/run/docker.sock`) | | `Iptables`\* | Enable iptables rules (default: `yes`) | | `User Namespace Remap` | User/Group setting for user namespaces (not defined by default). It can be set to any of ``, ``, \<*username*>, \<*username:groupname*>. If it is used, then `user_namespace.enable=1` must be set in the kernel options for the relevant nodes, and those nodes must be rebooted to pick up the new option. | | `Insecure Registries` | If registry access uses HTTPS but does not have proper certificates distributed, then the administrator can make Docker accept this situation by adding the registry to this list (empty by default) | | `Enable TLS`\* | Use TLS (default: `no`) | | `Verify TLS`\* | Use TLS and verify the remote (default: `no`) | | `TLS CA` | Trust only certificates that are signed by this CA (not defined by default) | | `TLS Certificate` | Path to TLS certificate file (not defined by default) | | `TLS Key` | Path to TLS key file (not defined by default) | | `Certificates Path` | Path to Docker certificates (default: `/etc/docker`) | | `Storage Backends` | [Docker storage back ends](/containerization/dockerengine#25-storage-backends). Storage types can be created and managed, in a submode under this mode. For BCM the only available type is `overlay2`. Each of these storage types has options that can be set from within the submode. | | `Containerd Socket` | Path to the `containerd` socket (default: not used) | | `Runtime` | Docker runtime | | `Options` | Additional parameters for `docker` daemon | \* Boolean (takes `yes` or `no` as a value) *Table 2.1: `Docker::Host` role options* ## 2.4 Iptables By default iptables rules have been added to nodes that function as a Docker host, to let network traffic go from the containers to outside the pods network. If this conflicts with other software that uses iptables, then this option can be disabled. For example, if the `docker::host` role has already been assigned to the nodes using the `default` category, then the iptables rules that are set can be disabled by setting the `iptables` parameter in the `Docker::Host` role to `no`: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% category use default [basecm11->category[default]]% roles [basecm11->category[default]->roles]% use docker::host [basecm11->category[default]->roles[Docker::Host]]% set iptables no [basecm11->category*[default*]->roles*[Docker::Host*]]% commit ``` ## 2.5 Storage Backends A core part of the Docker model is the efficient use of containers based on layered images. To implement this, Docker provides different storage back ends, also called storage drivers. These storage back ends rely heavily on various filesystem features in the kernel or volume manager. Some storage back ends perform better than others, depending on the circumstances. For BCM 11.0 the storage backend configured by `cm-docker-setup` is `overlay2`. The `docker info` command, amongst many other items, shows the storage driver and related settings that are being used in Docker: **Example** ```bash [root@basecm11 ~]# module load docker [root@basecm11 ~]# docker info Client: ... Context: default Debug Mode: false ... Server: Containers: 18 Running: 8 Paused: 6 Stopped: 4 Images: 1 Server Version: 26.1.5 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog ... ``` Docker data volumes are not controlled by the storage driver. Reads and writes to data volumes bypass the storage driver. It is possible to mount any number of data volumes into a container. Multiple containers can also share one or more data volumes. More information about Docker storage back ends is available at [https://docs.docker.com/engine/storage/drivers/](https://docs.docker.com/engine/storage/drivers/). For `overlay2` back end driver storage, settings can be added as options if needed. In `cmsh` this can be done by setting the `options` parameter in the `storagebackend` submode under the `docker::host` role. ## 2.6 Docker Monitoring When `cm-docker-setup` runs, it configures and runs the following Docker health checks: 1. makes a test API call to the endpoint of the Docker daemon 2. checks containers to see that none is in a `dead` state The Docker daemon can be monitored outside of BCM with the usual commands directly. BCM ways to manage or check on Docker include the following: In CMDaemon, the `docker` service can be checked: **Example** ```bash [basecm11->device[node001]->services]% list Service (key) Monitored Autostart ------------------------ ---------- ---------- docker yes yes nslcd yes yes [basecm11->device[node001]->services]% show docker Parameter Value -------------------------------- ------------------------------------------------ Revision Service docker Run if ALWAYS Monitored yes Autostart yes Timeout -1 Belongs to role yes Sickness check script Sickness check script timeout 10 Sickness check interval 60 ``` The `docker0` interface statistics can be checked within the `nodeoverview` output: **Example** ```bash [basecm11->device[node001]]% nodeoverview ... Interface Received Transmitted ------------ ------------ ------------ docker0 16.0 KiB 3.16 KiB ens3 492 MiB 72.5 MiB ens4 0 B 0 B ... ``` The health check measurable `Docker` checks if the `docker` service is healthy. **Example** ```bash [basecm11->device[node001]]% dumpmonitoringdata -1h now Docker ... Timestamp Value Info -------------------------- ---------- ---------- 2021/11/29 11:52:44.146 PASS 2021/11/30 18:28:44.146 PASS ``` ## 2.7 Docker Setup For NVIDIA NVIDIA GPU Cloud (NGC) is a cloud platform that runs on NVIDIA GPUs. NGC containers are lightweight containers that run applications on NVIDIA GPUs. The applications are typically HPC, machine learning, or deep learning applications. An NGC can be set up to run NGC containers from the registry [http://ngc.nvidia.com](http://ngc.nvidia.com). Docker can be configured as an NGC running NGC containers by using the NVIDIA Container Toolkit. The BCM package provided for this is: `cm-nvidia-container-toolkit`. One way to install and deploy this package is as part of the Docker installation, when running `cm-docker-setup` (section [2.1](#21-docker-setup)), where the cluster administrator selects `yes` as the answer to the request: “`Do you want to install the NVIDIA Runtime for Docker`”. Alternatively, if Docker has already been installed via `cm-docker-setup`, and if the package has not been installed, then it can be installed via the package manager, `yum` or `apt`. The toolkit has to be running on the compute nodes that have GPUs, which means that the installation must go to the appropriate node image ([section 9.4 of the *Administrator Manual*](/admin/softwareimagemanagement#94-managing-a-package-in-a-software-image-and-running-it-on-nodes)). For example, if the appropriate image is `gpu-image`, then the package manager command for RHEL-based distributions would be: **Example** ```bash # yum install --installroot=/cm/images/gpu-image cm-nvidia-container-toolkit ``` The nodes that use that GPU image can then be rebooted to pick up the new package. The GPU status can then be printed with the NVIDIA system management interface command. For example, if the image has been picked up by `node001`: **Example** ```bash [root@basecm11 ~]# ssh node001 Last login: Thu Dec 2 09:24:03 2021 from basecm11.cm.cluster [root@node001 ~]# module load docker [root@node001 ~]# docker run --runtime=nvidia --rm nvidia/cuda:11.4-base nvidia-smi Unable to find image 'nvidia/cuda:11.4.0-base' locally 11.4.0-base: Pulling from nvidia/cuda ... Digest: sha256:f0a5937399da5e4efb37fd7b75beb8c484b84dc381243c4b81fc5f9fcad42b66 Status: Downloaded newer image for nvidia/cuda:11.4.0-base Mon Mar 7 17:30:48 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K40c On | 00000000:00:06.0 Off | Off | | 23% 32C P8 22W / 235W | 0MiB / 12206MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ [root@node001 ~]# logout Connection to node001 closed. ``` The available CUDA Docker images can be found at [https://hub.docker.com/r/nvidia/cuda](https://hub.docker.com/r/nvidia/cuda).