> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Kubernetes Installation On An Air-gapped BCM Cluster An air-gapped cluster environment is one where the cluster is not directly connected to the internet. Typically this is done to protect the cluster from remote intrusions. Installation of Kubernetes in a non-air-gapped BCM cluster is covered in Chapter [4](/containerization/kubernetes). This appendix (Appendix B) describes how the following reference BCM air-gapped Kubernetes cluster can be created: * The cluster is air-gapped * The cluster uses Kubernetes version 1.34.7 * The cluster is running BCM version 11.33.0 or 11.33.1 * The BCM version is running on top of one of the following Linuxes: * Ubuntu 22.04 * Ubuntu 24.04 * Rocky Linux 9u3 * RHEL 9u3 Appendix [C](/containerization/kubernetesairgapupgrade) describes how an existing BCM air-gapped Kubernetes cluster can be upgraded. ## B.1 Initial Host Preparations For BCM Kubernetes Air-gapped Installation The initial steps must be performed on an initial host computer that has internet access. The initial host requires: * Internet access * An OS that matches that of the air-gapped cluster: Ubuntu 22.04, Ubuntu 24.04, Ubuntu 26.04, Rocky Linux 9, or RHEL 9. * An architecture that matches that of the air-gapped cluster * Currently, the tarball supports one architecture at a time. So if the cluster has a mixed architecture, for example with the head node being x86-64 (using `x86_64` RPM packages or `amd64` APT packages), but the Kubernetes control plane nodes being ARMv8 (using `arm64` packages), then that requires a more customized approach. For architecture configurations, the `--help` output of the download Bash scripts (format: `download_` \<*descriptive label*>`.sh`) indicates how the `--arch` option can be used. These scripts are located in the `airgap-scripts` directory, described shortly. * The BCM package repositories must be configured on the initial host. The relevant BCM repository list files can be copied over from a working cluster with the same OS. * For Ubuntu, these files are of the form `cm-*.list` and `ubuntu.sources`, and are located within the `/etc/apt/sources.list.d/` directory. There are some known architecture-related bugs in the file specifications for the Ubuntu repositories. These should be fixed for the BCM 11.34 point release. The issues and associated workarounds for the 11.33 release are in the following files: * `ubuntu.sources`: The default Ubuntu sources file needs an architecture specification to be added. The default file can be replaced with the following deb822-format (`man sources.list.5`) text: **Example** ```bash root@basecm11:/etc/apt/sources.list.d# cat ubuntu.sources Types: deb URIs: http://archive.ubuntu.com/ubuntu/ Suites: noble noble-updates noble-backports Components: main restricted universe multiverse Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg Architectures: amd64 Types: deb URIs: http://security.ubuntu.com/ubuntu/ Suites: noble-security Components: main restricted universe multiverse Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg Architectures: amd64 Types: deb URIs: http://ports.ubuntu.com/ubuntu-ports/ Suites: noble noble-updates noble-backports Components: main restricted universe multiverse Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg Architectures: arm64 Types: deb URIs: http://ports.ubuntu.com/ubuntu-ports/ Suites: noble-security Components: main restricted universe multiverse Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg Architectures: arm64 ``` * `cm-cuda-ubuntu2404-*.list`: The file for the default CUDA repositories list is modified as needed. Here, the “\*” is either `x86_64` or `arm64`. The change is to allow CUDA packages to be picked up from whichever architecture is needed: **Example** ```bash root@basecm11:/etc/apt/sources.list.d# cat cm-cuda-ubuntu2404-x86_64.list deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=amd64] https://\ developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ / deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=arm64] https://\ developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/ / ``` * `cm.list`. The BCM repository list file is modified so that the extra architectures that are needed are specified: **Example** ```bash root@basecm11:/etc/apt/sources.list.d# cat cm.list deb [arch=amd64] http://:@updates-us-east.brightcomputing.com/\ deb/cm/11.0/amd64/ubuntu/ noble main deb [arch=arm64] http://:@updates-us-east.brightcomputing.com/\ deb/cm/11.0/arm64/ubuntu/ noble main deb [arch=amd64] http://:@updates.brightcomputing.com/\ deb/cm/11.0/amd64/ubuntu/ noble main deb [arch=arm64] http://:@updates.brightcomputing.com/\ deb/cm/11.0/arm64/ubuntu/ noble main ``` In the preceding session: * The architecture key/value entries are specified for the repositories in the square brackets, using the traditional one-line format, but using a deb822 format is also possible. * The URLs specified in the preceding session take the \<*user*>`:<`*password*> format indicated. The values that are used for \<*user*>`:<`*password*> should already be configured in the original `cm.list` repository file, and can simply be taken over. * The server and path of the URL on the actual cluster may differ from the preceding example. * For RHEL variants these files should be `cm.repo` and `epel.repo`, located within `/etc/yum.repos.d/` * The BCM `airgap-scripts` directory from the matching BCM version must be on the initial host. How the installation of that directory is carried out depends on if `cm-setup` is on the initial host, as explained next. If Run:ai is to be deployed on this cluster, then additional prerequisites must be in place before the Kubernetes installation begins: * three reserved IP addresses from the internal network * DNS records for the Run:ai control plane FQDN * two wildcard subdomains * a TLS certificate and private key * an NGC API key None of these can be arranged once the cluster is air-gapped. Section [B.2.8.1](#b281-prerequisites-for-an-air-gapped-runai-deployment) has more details on these. #### B.1.0.1 Initial Host Has `cm-setup` Installed If the `cm-setup` package is already installed on the initial host, then for BCM version 11, the air-gap scripts are by default in the path defined by the environment variable `K8S_AG_SCRIPTS`. **Example** ```bash # echo $K8S_AG_SCRIPTS /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts/ ``` For BCM version 10, the Python directory path is `python3.9` instead. This environment variable path should be added to the existing PATH: ```bash # export PATH=$PATH:$K8S_AG_SCRIPTS ``` #### B.1.0.2 Initial Host Does Not Have `cm-setup` Installed If the `cm-setup` package is not already installed on the initial host, then the administrator should define the air-gap scripts path, `$K8S_AG_SCRIPTS`, and then add the air-gap scripts to the PATH environment. ```bash # export K8S_AG_SCRIPTS= # export PATH=$PATH:$K8S_AG_SCRIPTS ``` #### B.1.0.3 Using The Air-gap Scripts * `helm 3.x` (not 4.x at the time of writing, April 2026) must be added to `/usr/local/bin/helm` as per the instructions at [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/). Alternatively, the following helper script, that BCM ships with `cm-setup`, can be used: **Example** ```bash root@internet-host:~# download_helm_binary.sh Downloading helm v3.20.2 for linux-amd64... Helm binary installed at /usr/local/bin/helm root@internet-host:~# ``` Optionally, a Helm check can be carried out with: **Example** ```bash root@internet-host:~# helm version version.BuildInfo{Version:"v3.20.2", GitCommit:"8fb76d6ab555577e98e23b7500009537a471feee", GitTreeState:"clean", GoVersion:"go1.25.9"} ``` * Skopeo, an image manager, is installed. This installation is carried out on the internet-connected host only, and not on the air-gapped head node. Skopeo is needed here in order to pull the container images that make up the tarball: **Example** ```bash root@internet-host:~# apt install skopeo #or dnf install skopeo for RHEL variants ``` The air-gapped head node obtains its own copy of Skopeo from the tarball, and no attempt should be made to install Skopeo there from a distribution repository (section [B.2](#b2-head-node-preparation-steps)). * To work around a potential Docker Hub pull rate limit issue ([https://docs.docker.com/docker-hub/usage/pulls/](https://docs.docker.com/docker-hub/usage/pulls/)) the administrator can now sign up for an account at [https://app.docker.com/signup](https://app.docker.com/signup), and authenticate with `skopeo login docker.io`. This is to help with running the image installation scripts from the `airgapped` directory in the next bullet point. Any credentials are fine—the credentials will not end up in the air-gapped tarball or environment. They are only needed for authentication to conveniently pull images from Docker Hub. This is because unauthenticated pulls are limited to a rate of 100 per 6 hours. Authenticated pulls, on the other hand, are allowed at a rate of 200 per 6 hours, which should be sufficient for preparing the `airgapped` tarball. * A working directory, for example `/root/airgapped`, can then be created, and installation scripts within the `airgapped` directory can then be run: **Example** ```bash root@internet-host:~# pwd /root root@internet-host:~# mkdir airgapped; cd airgapped root@internet-host:~/airgapped# download_ubuntu_packages.sh --kube-version=1.34 # for Ubuntu Linux ... [root@internet-host airgapped]# download_r9_packages.sh --kube-version=1.34 # for RHEL variants ... root@internet-host:~/airgapped# download_container_images.sh --kube-version=1.34 ... root@internet-host:~/airgapped# download_helm_charts.sh ... root@internet-host:~/airgapped# download_crds.sh ... ``` The preceding scripted downloads are for Kubernetes version 1.34. To use a different version, such as v1.33 or v1.35, the `--kube-version` flag should be adjusted accordingly. If `--kube-version` is not set on `download_ubuntu_packages.sh`, `download_r9_packages.sh`, or `download_container_images.sh`, each script falls back to its own default Kubernetes version, which is not necessarily the version that is to be deployed. `download_helm_charts.sh` and `download_crds.sh` take no version-related option, and are unaffected by this. `download_ubuntu_packages.sh`, `download_r9_packages.sh`, and `download_container_images.sh` also accept a `-c`/`--config` option, giving the path to a `cm-kubernetes-setup.conf` configuration file to read the Kubernetes version from, instead of `--kube-version`. On `download_ubuntu_packages.sh` and `download_r9_packages.sh`, `--kube-version` and `-c` are mutually exclusive: only one may be given. On `download_container_images.sh` both may be given together, in which case `--kube-version` takes precedence over the value in the configuration file. Using `-c` this way requires a `cm-kubernetes-setup.conf` file to already exist. That file is normally generated later, on the air-gapped head node (section [B.2.1](#b21-generating-the-air-gapped-kubernetes-setup-configuration)). To guarantee that the version downloaded here matches the version that is deployed later, the configuration can instead be generated ahead of time, before this step, and the resulting `cm-kubernetes-setup.conf` copied to the internet-connected host for use with `-c`. The charts fetched by `download_helm_charts.sh` determine which NVIDIA GPU Operator versions can be deployed later on. The `cm-kubernetes-setup` wizard presents a list of GPU Operator versions, but in an air-gapped cluster only a version whose chart is among the downloaded ones can be installed. There is no route to fetch any of the others once the cluster is air-gapped. Certification and availability are separate properties: an asterisk next to a version in the wizard marks it as certified for NVIDIA AI Enterprise (section [6.7.1](/containerization/kubernetesoperators#671-installing-the-nvidia-gpu-operator-using-the-wizard)). A version that was downloaded is not necessarily a certified one, and a certified version is not necessarily among those that were downloaded. The downloaded charts can therefore be listed before `cm-kubernetes-setup` is run, so that the version picked in the wizard is known to be one that is available: ```bash root@internet-host:~/airgapped# ls -1 helm-charts/ ``` The same `helm-charts` directory is present inside the extracted `airgapped` directory on the head node, so the charts can also be checked there. The charts are fetched from chart repositories on the internet, and the set of repositories that is used changes between BCM releases. BCM 11.34 fetches from four repositories that BCM 11.33 did not: * `oci://cr.agentgateway.dev/charts/agentgateway`, for the AgentGateway operator * `oci://cr.agentgateway.dev/charts/agentgateway-crds`, for the AgentGateway CRDs operator * [https://argoproj.github.io/argo-helm](https://argoproj.github.io/argo-helm), for the Argo CD operator * [https://flannel-io.github.io/flannel](https://flannel-io.github.io/flannel), for the Flannel network plug-in A site that restricts outbound connections from the internet-connected host must permit these destinations before `download_helm_charts.sh` is run. A site that is moving from BCM 11.33 must add them to any existing list of permitted destinations. The repository that serves the Calico CRDs chart, [https://docs.tigera.io/calico/charts](https://docs.tigera.io/calico/charts), is not a new one. It already served the Tigera Operator chart in BCM 11.33. To ensure the commands all succeed, it is best to execute them in sequence and let them complete. The download scripts all write their output to the current working directory. The downloads take a while, and are best run inside a `screen` or `tmux` session so that a dropped connection does not interrupt them. The container image download accounts for most of the elapsed time—on a test system with fast networking it took around 15 times as long as the package, chart, and CRD downloads put together. Absolute times depend on the available bandwidth, so a slow download is not in itself a sign that something has gone wrong. While the packages are downloading, APT emits warnings similar to: ```bash W: Download is performed unsandboxed as root as file '/root/airgapped/packages/all/nginx-common_1.24.0-2ubuntu7.15_all.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) ``` One or more of these appears for each of the `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` directories. *These warnings are expected and can be ignored.* The `_apt` user (with the underscore) cannot read the working directory when it is under `/root`, so APT drops its download sandbox and continues unsandboxed. The downloads themselves succeed, and the scripts exit normally. This is the same permission restriction that later requires the tarball to be extracted somewhere that APT can write to (section [B.2](#b2-head-node-preparation-steps)). Without the Run:ai air-gapped package, the completed download is about 8.6 GB, of which the container images account for about 8.1 GB. The Run:ai package is a further 13–14 GB on top of that, so a working directory for a deployment that includes Run:ai needs upwards of 20 GB. Note that the `total` reported by `ls` in the listing that follows counts only the files in the directory itself, which is why it reflects the size of the Run:ai tarball rather than the size of the whole download. * Run:ai air-gapped setup If Run:ai is required in the air-gapped installation, then the Run:ai air-gapped package must also be downloaded to the working directory. The package is published as an NGC resource. The available versions are listed on the resource page at [https://catalog.ngc.nvidia.com/orgs/nvidia/teams/runai/resources/runai-airgapp-package/version](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/runai/resources/runai-airgapp-package/version), and an NGC API key is needed to download it ([https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#ngc-api-keys](https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#ngc-api-keys)): **Example** ```bash root@internet-host:~/airgapped# curl -LO --request GET \ "https://api.ngc.nvidia.com/v2/org/nvidia/team/runai/resources/\ runai-airgapp-package/versions//files/\ runai-airgapped-package-.tar.gz" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` The \<*`runai version`*> part is replaced with the required patch version. For example: `2.25.11`, which gives a tarball named: `runai-airgapped-package-2.25.11.tar.gz` Alternatively, NVIDIA Run:ai Support can be contacted to request the appropriate tarball. A minimum version of 2.23.60 is required, since that is when multi-arch support was added. The integrity of the downloaded file should be verified with `sha256sum` against the checksum provided by NVIDIA Run:ai Support. The `cm-setup-runai-bcm11` package must also be downloaded to the working directory. This package implements the `cm-runai-setup` wizard, and is not part of the Run:ai air-gapped tarball. Listing the contents of the tarball does not show it. The package is published separately, in the NVIDIA Mission Control package repository, which is publicly reachable and needs no credentials. The available versions are listed at: [https://edge.urm.nvidia.com/artifactory/sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/](https://edge.urm.nvidia.com/artifactory/sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/) The package is published only as a `.deb`, so an air-gapped Run:ai deployment on RHEL is not supported in this release. The package is architecture-specific, so the build that matches the head node architecture must be downloaded: **Example** ```bash root@internet-host:~/airgapped# curl -LO "https://edge.urm.nvidia.com/artifactory/\ sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/\ cm-setup-runai-bcm11__.deb" ``` In the repository each package has a corresponding `.sha256` file. The content of the `.sha256` file is the checksum. For the `sha256sum` utility to be able to check the package integrity, the `.sha256` file must have the file name of its corresponding package appended on the same line to the `.sha256` file checksum content: **Example** ```bash root@internet-host:~/airgapped# curl -sO "https://edge.urm.nvidia.com/artifactory/\ sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/\ cm-setup-runai-bcm11__.deb.sha256" root@internet-host:~/airgapped# grep -qF "cm-setup-runai-bcm11__.deb" \ cm-setup-runai-bcm11__.deb.sha256 || echo " cm-setup-runai-bcm11__.deb" \ >> cm-setup-runai-bcm11__.deb.sha256 root@internet-host:~/airgapped# sha256sum -c cm-setup-runai-bcm11__.deb.sha256 cm-setup-runai-bcm11__.deb: OK ``` The package must not be installed if the check reports anything other than `OK`. The directory contents should then appear similar to: ```bash root@internet-host:~/airgapped# ls -alrth total 14G drwx------ 10 root root 287 Jun 8 04:21 .. drwxr-xr-x 4 root root 30 Jun 8 04:21 packages drwxr-xr-x 4 root root 30 Jun 8 04:21 packages-head drwxr-xr-x 3 root root 19 Jun 8 04:21 packages-non-dgx drwxr-xr-x 3 root root 19 Jun 8 04:21 packages-cuda drwxr-xr-x 2 root root 88 Jun 8 04:27 k8s-images drwxr-xr-x 2 root root 4.0K Jun 8 04:29 helm-charts drwxr-xr-x 2 root root 40 Jun 8 04:29 crds drwxr-xr-x 9 root root 183 Jun 8 04:30 . -rw-r--r-- 1 root root 41K Jun 8 04:31 cm-setup-runai-bcm11_2.5.5_amd64.deb -rw-r--r-- 1 root root 14G Jun 8 04:31 runai-airgapped-package-2.25.11.tar.gz ``` * Confirming the downloaded content is complete **Verifying nothing is missing before tarballing:** The download scripts run in the preceding steps make this repository complete for the release being installed. No additional package, image, or chart list needs to be assembled by hand. Each directory in the preceding listing corresponds to one content category: * `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` hold the OS packages fetched by `download_ubuntu_packages.sh` or `download_r9_packages.sh` * `k8s-images` holds the container images fetched by `download_container_images.sh` * `helm-charts` holds the Helm charts fetched by `download_helm_charts.sh`, from every chart repository this release depends on (see the repository list earlier in this section) * `crds` holds the CRD manifests fetched by `download_crds.sh` The exact package names, image tags, and chart versions in each directory change between BCM and Kubernetes releases. Checking their contents against a fixed list is not useful. Instead, each category should be confirmed to have downloaded files before proceeding. A plain listing of the top-level directories is not enough: `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` organize their packages into subdirectories such as `all` and `amd64`, and those subdirectories can exist without containing any packages. Searching recursively for files avoids that gap: **Example** ```bash root@internet-host:~/airgapped# find packages packages-head packages-non-dgx \ packages-cuda k8s-images helm-charts crds -type f ``` A content category that produces no output has no downloaded files, even if its top-level directory (or a subdirectory within it) exists. Its download script did not complete, and should be re-run before the `airgapped` directory is tarballed. * The `airgapped` directory can now be tarballed. In the tarball, the packages that are to be used by the software images must include all the architectures that are to be used by the Kubernetes cluster. **Example** ```bash root@internet-host:~/airgapped# ls cm-setup-runai-bcm11_2.5.5_amd64.deb crds helm-charts k8s-images packages packages-cuda packages-head packages-non-dgx runai-airgapped-package-2.25.11.tar.gz root@internet-host:~/airgapped# cd .. root@internet-host:~# tar -cvzf airgapped.tgz airgapped ``` The `airgapped.tgz` file should be saved to some transfer media (for example: a USB stick), and should be copied over to the active head node of the air-gapped cluster. ## B.2 Head Node Preparation Steps Before starting this section, the cluster’s control-plane, worker, and etcd nodes must already be assigned, as covered under **Node assignment** (section [4.2.8](/containerization/kubernetes#428-kubernetes-setup-from-a-tui-session), [here](/containerization/kubernetes#kubernetes-setup-from-a-tui-session-node-assignment)). This assignment is done using the same `cm-kubernetes-setup` wizard screens used here, is not air-gap-specific, and cannot be arranged once the cluster is air-gapped. ### B.2.1 Generating The Air-gapped Kubernetes Setup Configuration A local container registry (section [3.2](/containerization/dockerregistries#32-container-registries-setup-and-configuration)) is needed on the air-gapped head node, so that Kubernetes operators can pull images. The registry itself is deployed in a later step, using the `cm-container-registry-setup` wizard (section [B.2.3](#b23-configuring-the-container-registry)). First, however, a `cm-kubernetes-setup.conf` configuration file that is aware of this registry must be generated. The configuration file is generated by running the `cm-kubernetes-setup` wizard TUI with special air-gap flags. These flags pre-configure the air-gap deployment mode and the registry address: **Example** `cm-kubernetes-setup --airgap-registry --airgap-registry-as-helm` The `` part is replaced with the value: `master.cm.cluster:5000` for the default Docker registry installation. When the wizard reaches the summary screen, the setup is not deployed—it merely has its configuration saved, so that there is a working `cm-kubernetes-setup.conf` file available for when the actual deployment is carried out later on. #### B.2.1.1 Selecting The Run:ai Operator Dependencies When the `cm-kubernetes-setup` wizard is run, with the intent of running Run:ai as well, the operator selection screen must include the dependencies that Run:ai requires. For Run:ai 2.25, these are: * NVIDIA GPU Operator * Grafana Operator * Kgateway * Knative Operator * Kubeflow Training Operator * Kubernetes Metrics Server * Kubernetes MPI Operator * Kubernetes State Metrics * LeaderWorkerSet Operator * MetalLB * Network Operator (technically optional, but recommended) * NIM Operator (optional) * Prometheus Adapter * Prometheus Operator Stack The Run:ai operator and its prerequisites are described in more detail in section [6.11](/containerization/kubernetesoperators#611-the-runai-operator). ### B.2.2 Unpacking The Files The `airgapped.tgz` file that was copied over to the head node can then be extracted. For Ubuntu, extracting to a file system that APT can write to, such as `/tmp`, is recommended. This is because APT runs as the `_apt` user (with the underscore) during a part of the installation process, and `_apt` does not have sufficient permissions under `/root` to process the extracted files as needed. For Rocky Linux and RHEL the archive can be extracted in place, and the move to `/tmp` or similar is not needed. **Example** ```bash root@basecm11:~# mv airgapped.tgz /tmp/ root@basecm11:~# cd /tmp root@basecm11:/tmp# tar xvf airgapped.tgz root@basecm11:/tmp# cd airgapped ``` The system packages can now be installed onto the head node and the required software images, using the untarballed airgap system files together with the configuration file saved from earlier on. The command must be run from within the extracted `airgapped` directory, because the installer looks for the `packages` subdirectories relative to the current working directory: **Example** ```bash root@basecm11:/tmp/airgapped# cm-kubernetes-setup --airgap-install-system-packages \ -c /root/cm-kubernetes-setup.conf ``` The packages that this step installs include Skopeo and its dependencies, which are carried in the `packages-head` directory of the tarball. Skopeo is required later on, when the container images are pushed into the local registry. It should not be installed on the head node with `apt` or `dnf`: an air-gapped head node has no distribution repository to install it from, and the step shown here has already provided it. Run:ai is deployed separately, after the Kubernetes cluster has been deployed, with the `cm-runai-setup` wizard (section [B.2.8](#b28-decoupled-runai-deployment)). If the air-gapped cluster has a secondary head node ([section 15.1 of the *Administrator Manual*](/admin/ha#151-ha-concepts)), then the system package installation does not have to be repeated on it. The `--airgap-install-system-packages` run on the active head node also updates the secondary head node. The tarball and the configuration file therefore do not have to be copied across for this step, and `cm-kubernetes-setup` should not be run on a passive head node. This applies to the system package installation. Other components can still have their own secondary head node steps. Earlier BCM releases did instruct that the tarball and the configuration file be copied to the secondary head node, and that the wizard be run there a second time with the `--skip-headnode-check` flag. That run reaches a stage which installs packages into the software image, and it is refused on a passive head node with the message `This node is not the active head node (PASSIVE)`. The documented answer was `Skip`, which leaves the software image untouched. The message also suggests `--force`, which must not be used here. Software images are local to each head node and are synchronized from the active head node. This means that a write made on a passive head node is reverted at the next synchronization, and that until then the passive head node serves a different image to any node provisioning from it compared to the active head node. Package installation for multiarch software images in the Kubernetes cluster is automatic. This is assuming that the software images have the packages for the required architectures in the tgz file. ### B.2.3 Configuring The Container Registry The container registry can now be configured with: **Example** `root@basecm11:~# cm-container-registry-setup --skip-packages` The `--skip-packages` option means that the wizard assumes that the packages needed are already installed. When configuring using the registry wizard, the active head node should be selected by the cluster administrator as the node to be used for the deployment. The TUI asks for hosts to be set for the SSL certificate (figure [B.1](#fig-kube-urwid-cm-container-registry-setup-san-spec-install)): ![SAN specification for SSL certificate](/base-command-manager/product-manuals/_fern-img/c2317b8efc653cb35aba890f37118f7d5ac868591c5b66ee28c926438439d32a.webp) *Figure B.1: SAN specification for SSL certificate* A custom domain name or additional subject name is set (`master.cm.cluster` in figure [B.1](#fig-kube-urwid-cm-container-registry-setup-san-spec-install)), and the configuration is saved and deployed with the `save and deploy` option in the summary screen of the wizard. Helm on the head node must be able to talk to this registry without TLS issues. The custom registry CA is therefore copied and trusted system-wide: ``` root@basecm11:~# cp /cm/local/apps/docker-registry/var/etc/$(hostname):5000-ca.crt \ /usr/local/share/ca-certificates/bcm-docker-registry.crt root@basecm11:~# update-ca-certificates ``` ### B.2.4 Configuring Docker For An Air-gapped Cluster Docker can now be configured. Docker is currently only needed for the BCM Run:ai installation, and is configured with: `root@basecm11:~# cm-docker-setup --skip-packages` The active head node is chosen for the node for Docker, and the configuration is saved and deployed. The successful deployment of Docker can be confirmed by running `docker ps` on the nodes on which it has been installed. The `docker` module must be loaded first, since the `docker` binary is not in the default path. The following output indicates readiness: `root@basecm11:~# module load docker root@basecm11:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES` Docker service initialization may take a few seconds. Authenticating with `docker login` is not needed in this case, since images from Docker Hub are not going to be pulled. ### B.2.5 Setting Up The Container Images And Helm Charts For An Air-gapped Cluster During Installation From the airgapped directory, all Docker images and Helm charts can be pushed to the registry with: ``` root@basecm11:~# cm-kubernetes-setup --airgap-push-container-images -c /root/cm-kubernetes-setup.conf root@basecm11:~# cm-kubernetes-setup --airgap-push-helm-charts -c /root/cm-kubernetes-setup.conf ``` ### B.2.6 Extracting The CRD Files For An Air-gapped Cluster During Installation The downloaded CRD files can then be extracted using: `root@basecm11:~# $K8S_AG_SCRIPTS/extract_crds.sh` ### B.2.7 Deploying The Air-gapped Kubernetes Cluster When the configuration is generated with the `--airgap-registry` and `--airgap-registry-as-helm` options (section [B.2.1](#b21-generating-the-air-gapped-kubernetes-setup-configuration)), the air-gap section of `/root/cm-kubernetes-setup.conf` is populated automatically. No manual editing of the file is normally needed. For reference, the relevant section then appears similar to: ```bash modules: kubernetes: airgap: helm: ca: '/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt' repo: 'oci://master.cm.cluster:5000/helm-charts' registry: 'master.cm.cluster:5000' registry_username: '' registry_password: '' ``` If Run:ai is to be deployed, then the operators that Run:ai depends on must have been selected when this configuration was generated (section [B.2.1.1](#b211-selecting-the-runai-operator-dependencies)). Kubernetes can then be deployed with: ```bash # cm-kubernetes-setup -c /root/cm-kubernetes-setup.conf ``` Run:ai itself is not deployed by `cm-kubernetes-setup`. It is deployed afterwards with the separate `cm-runai-setup` wizard (section [B.2.8.3](#b283-running-the-cm-runai-setup-wizard)). ### B.2.8 Decoupled Run:ai Deployment Run:ai has its own wizard, `cm-runai-setup`. The command is part of the `cm-setup` package, and is present on a head node by default. The plugin that implements it is delivered separately, by the `cm-setup-runai-bcm11` package. Running the wizard without that package fails with an error stating that the main component is not installed. The presence of the `cm-runai-setup` command therefore does not mean that Run:ai can be deployed. The `cm-kubernetes-setup` wizard is used to deploy Kubernetes itself and the operators that Run:ai depends on. Run:ai is a separate product, with its own release cycle. BCM and Run:ai are released on the same cadence, so a BCM release has a Run:ai release that corresponds to it: BCM 11.33 corresponds to Run:ai 2.25, and BCM 11.34 corresponds to Run:ai 2.26. That correspondence is a matter of timing and not a restriction. Any Run:ai version can be deployed, provided that the version is still supported and has not reached end of life. The air-gapped tarball must match the version being deployed. For Run:ai 2.25.11 the tarball is named `runai-airgapped-package-2.25.11.tar.gz`. #### B.2.8.1 Prerequisites For An Air-gapped Run:ai Deployment The following prerequisites must be in place before `cm-runai-setup` is run. All of them have to be prepared either on the internet-connected host, or during the earlier Kubernetes deployment. None of them can be arranged once the cluster is air-gapped. * The `cm-setup-runai-bcm11` package. If it is not already installed on the head nodes, then it must be downloaded on the internet-connected host and placed in the `airgapped` directory (section [B.2.8.2](#b282-obtaining-the-cm-setup-runai-bcm11-package)). The wizard cannot install it itself, because the `--skip-packages` flag that an air-gapped deployment requires also disables package installation. Checking whether the `cm-runai-setup` command exists is not a way to decide whether this prerequisite has been met, because that command is part of `cm-setup` and is present whether or not the plugin package has been installed. * Three reserved IP addresses, none of which may already be in use: one for the BCM gateway, one for the Run:ai control plane ingress, and one for Run:ai inference (Kourier). The address range that MetalLB is given during `cm-kubernetes-setup` must be large enough to cover all three. The wizard creates its own MetalLB address pools, so the addresses should be left free rather than placed in a pool beforehand. * DNS records resolving the Run:ai control plane FQDN, a wildcard for workspace subdomains, and a wildcard for inference endpoints. For example `runai.example.com` and `*.runai.example.com` pointing at the control plane address, and `*.runai-inference.example.com` pointing at the inference address. * A TLS certificate and private key issued for the control plane FQDN, with the FQDN present in the Subject Alternative Name. A local CA certificate is also needed if the certificate is not issued by a publicly-trusted authority. * An NGC API key. The wizard requires a value for this even for an air-gapped deployment. * A directory holding the certificate, key and NGC API key that is reachable from the head node. `/cm/shared/runai/` is the conventional location, and is used throughout the Run:ai documentation. The wizard also enforces a password policy on the Tenants Manager administrator account: at least 8 characters, including at least one digit, one uppercase letter, and one special character. #### B.2.8.2 Obtaining The `cm-setup-runai-bcm11` Package The `cm-setup-runai-bcm11` package implements the `cm-runai-setup` wizard. The package source depends on whether the cluster has internet access, and on the operating system: **Clusters with internet access:** No action is needed. The package is installed automatically when `cm-runai-setup` or `cm-mission-control-setup` is run. **Air-gapped clusters running Ubuntu:** The package is downloaded on the internet-connected host during the initial host preparations, and placed in the `airgapped` directory so that it is included in the tarball. An air-gapped head node has no way to retrieve it later. On the air-gapped head node, after the system packages have been installed (section [B.2](#b2-head-node-preparation-steps)), the wizard package can then be installed. **Example** ```bash root@basecm11:/tmp/airgapped# apt install -y ./cm-setup-runai-bcm11__.deb ``` If the air-gapped cluster has a secondary head node ([section 15.1 of the *Administrator Manual*](/admin/ha#151-ha-concepts)), then the package must be installed on that head node too. The `.deb` file is held in a location that is local to each head node rather than on shared storage, so it must first be copied across: **Example** ```bash root@basecm11:/tmp/airgapped# scp cm-setup-runai-bcm11__.deb \ :/root/ ``` It is then installed on the secondary head node in the same way as on the primary head node: **Example** ```bash root@:~# apt install -y \ ./cm-setup-runai-bcm11__.deb ``` The `cm-runai-setup` wizard itself is run on the active head node only. The package is installed on the secondary head node so that the wizard remains available if the cluster fails over to that head node. **Air-gapped clusters running RHEL:** An air-gapped Run:ai deployment is not supported in this release. Support is planned for a later release. #### B.2.8.3 Running The `cm-runai-setup` Wizard After Kubernetes has been deployed, together with the Run:ai operator dependencies, Run:ai itself is deployed with the `cm-runai-setup` wizard. For an air-gapped deployment, the wizard must be run with the `--skip-packages` flag. Without this flag, the automatic update feature of the wizard attempts to reach the internet, and fails. **Example** ```bash root@basecm11:~# cm-runai-setup --skip-packages ``` The wizard prompts for: * the deployment type (self-hosted) * the node category for the Run:ai control plane (the same category as the Kubernetes control-plane nodes is recommended) * the control plane FQDN, and the paths to the certificate and private key * an administrator username and password for the Run:ai control plane * the gateway IP address. The address that the wizard offers is not necessarily one that MetalLB has been given, so it should be checked against the reserved addresses (section [B.2.8.1](#b281-prerequisites-for-an-air-gapped-runai-deployment)) rather than accepted unchecked. * the LoadBalancer IP address used for Run:ai inference If the wizard detects that it is deploying onto an air-gapped Kubernetes cluster, then it also prompts for the air-gapped registry, the Helm repository, the Helm CA certificate, and the location of the Run:ai air-gapped tarball. The Run:ai version itself is not prompted for in this case: it is taken from the operator defaults, and must match the version supplied in the tarball. The wizard then checks the versions of the operators that Run:ai depends on, and reports each one against its requirement, before continuing to the deployment. At the summary screen, `Save & Exit` should be selected if BCM point release 11.33 is being run. This is because deploying in a single run is prevented by a bug in the version of the wizard that ships with that release. The configuration is written to `/root/cm-runai-setup.conf` in either case, so a deployment that is interrupted can always be resumed from the saved configuration. The deployment is then carried out with: **Example** ```bash root@basecm11:~# cm-runai-setup --skip-packages -c /root/cm-runai-setup.conf ``` The `--skip-packages` flag is only needed for air-gapped deployments. When the setup completes, Run:ai should be up and running. This can be checked with, for example: **Example** ```bash root@basecm11:~# curl -s https://runai.superpod-nvidia.local | grep title Run.ai ``` #### B.2.8.4 Uninstalling Kubernetes And Run:ai Uninstalling Kubernetes also removes Run:ai. The simplest method is to run the `cm-kubernetes-setup` wizard and select the uninstall option. Alternatively, the removal can be carried out from the command line: **Example** ```bash root@basecm11:~# cm-kubernetes-setup --cluster --remove --yes-i-really-mean-it ``` To remove only Run:ai, while leaving the Kubernetes cluster in place, the `cm-runai-setup` wizard provides an uninstall option. Equivalently, the removal can be carried out from the command line: **Example** ```bash root@basecm11:~# cm-runai-setup --cluster --remove --yes-i-really-mean-it ``` ### B.2.9 Post-installation Automatic Provisioning Section [B.2](#b2-head-node-preparation-steps) describes installing packages from the air-gapped tarball into software images. When deploying new software images for worker nodes, BCM cannot automatically provision them in air-gapped environments; this automatic provisioning functionality is available only in non-air-gapped clusters. For air-gapped clusters, administrators must manually execute the relevant installation commands, adjusting the configuration file to specify the appropriate target nodes and node categories.