Initial Host Preparations for BCM Kubernetes Air-gapped Installation#

These procedures are based on Appendix B.1 of the NVIDIA Base Command Manager 11 Containerization Manual.

Perform all steps on an internet-connected host. The resulting airgapped.tgz tarball is then transferred to the air-gapped BCM head node.

Note

A BCM 11.34 virtual machine is recommended, since it already provides a matching OS, configured BCM repositories, and the airgap-scripts directory.

Reference environment#

Reference cluster configuration:

  • Air-gapped (no internet) cluster

  • Kubernetes version 1.35 (adjust --kube-version as needed)

  • BCM version 11.34

Initial host requirements#

The preparation host must have:

  • Internet access

  • OS matching the air-gapped cluster: Ubuntu 24.04 or Ubuntu 26.04

  • CPU architecture matching the air-gapped cluster

    • The tarball supports one architecture at a time. Mixed-architecture clusters (for example, x86-64 BCM head node with ARMv8 control plane nodes) require a customized approach.

    • Use the --help output of the download_*.sh scripts and the --arch option for architecture-specific configurations.

Step 1: Prepare the internet-connected host#

Verify the preparation host meets these requirements:

  • OS matches the air-gapped cluster (Ubuntu 24.04 or Ubuntu 26.04)

  • CPU architecture matches the air-gapped cluster

  • BCM package repositories are configured (refer to Step 2)

  • airgap-scripts directory from BCM 11.33.1 or later is available

Step 2: Configure BCM package repositories#

Copy cm-*.list and ubuntu.sources from /etc/apt/sources.list.d/ on a BCM cluster running the same OS as your air-gapped cluster.

Note

The architecture workarounds in this step are only required if your cluster has mixed-architecture nodes — for example, when the BCM head node, Kubernetes nodes, or compute nodes do not all use the same CPU architecture. On Ubuntu 26.04, mixed-architecture clusters need no workarounds: archive.ubuntu.com provides both amd64 and arm64 packages. If all nodes use the same architecture, continue with Step 3: Install BCM air-gap scripts.

Known Architecture Workarounds (Ubuntu 24.04)#

The Ubuntu repository files included with BCM have no architecture qualifiers, so apt searches every repository for every configured architecture. On Ubuntu 24.04 (noble), ports.ubuntu.com provides arm64 packages, and archive.ubuntu.com provides amd64 packages. Without a qualifier, requests for arm64 packages from archive.ubuntu.com fail. This issue affects BCM 11.34.

Ubuntu 26.04 (resolute) provides both amd64 and arm64 packages from archive.ubuntu.com and does not require these workarounds. Do not restrict this repository to amd64 on Ubuntu 26.04; doing so prevents arm64 nodes from accessing their packages.

Apply the following workarounds only on Ubuntu 24.04.

In ubuntu.sources, specify the architecture for each repository. Replace the default file with the following configuration:

Ubuntu 24.04 Repository Configuration
Types: deb
URIs: http://archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: amd64

Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: amd64

Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports/
Suites: noble noble-updates noble-backports noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: arm64

In cm-cuda-ubuntu2404-*.list, specify the architecture for each CUDA repository. The * represents either x86_64 or sbsa:

deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=amd64] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /
deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=arm64] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ /

In cm.list, add entries for each required architecture:

deb [arch=amd64] http://<user>:<password>@updates-us-east.brightcomputing.com/deb/cm/11.0/amd64/ubuntu/ noble main
deb [arch=arm64] http://<user>:<password>@updates-us-east.brightcomputing.com/deb/cm/11.0/arm64/ubuntu/ noble main
deb [arch=amd64] http://<user>:<password>@updates.brightcomputing.com/deb/cm/11.0/amd64/ubuntu/ noble main
deb [arch=arm64] http://<user>:<password>@updates.brightcomputing.com/deb/cm/11.0/arm64/ubuntu/ noble main

  • The preceding examples use the traditional one-line format for architecture specifications in square brackets; deb822 format is also supported.

  • Use the <user>:<password> credentials from the original cm.list on the working cluster.

  • The server URL path may differ in your environment.

Step 3: Install BCM air-gap scripts#

The BCM airgap-scripts directory from the matching BCM version must be available on the preparation host.

If the cm-setup package is installed, the scripts are at:

/cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts

When the cm-setup module is loaded (enabled by default), this path is available via the K8S_AG_SCRIPTS environment variable.

If cm-setup is already installed#

For BCM version 11, the air-gap scripts are in the path defined by K8S_AG_SCRIPTS:

echo $K8S_AG_SCRIPTS
# /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts/

For BCM version 10, the Python directory path uses python3.9 instead.

Add the scripts to PATH:

export PATH=$PATH:$K8S_AG_SCRIPTS

If cm-setup is not installed#

Copy the airgap-scripts directory to the preparation host, define K8S_AG_SCRIPTS to point to it, then add it to PATH:

Replace <path-to-airgap-scripts> with the full path to the copied directory.

export K8S_AG_SCRIPTS="<path-to-airgap-scripts>"
export PATH="$PATH:$K8S_AG_SCRIPTS"

Step 4: Install Helm#

BCM 11.34 includes Helm 4.x. The BCM helper script installs the Helm version included with your BCM release, so the exact patch version varies.

Run the script to install Helm at /usr/local/bin/helm:

download_helm_binary.sh

Example output:

Downloading helm v4.2.4 for linux-amd64...
Helm binary installed at /usr/local/bin/helm

Verify the installation:

helm version

Step 5: Install Skopeo#

On the preparation host, install Skopeo (image manager):

apt update   # if necessary
apt install skopeo

Optional: Authenticate with Docker Hub#

To work around Docker Hub pull rate limits, sign up at https://app.docker.com/signup and authenticate before running the download scripts:

skopeo login docker.io

  • Unauthenticated pulls are limited to 100 per 6 hours.

  • Authenticated pulls are limited to 200 per 6 hours — sufficient for preparing the air-gapped tarball.

  • Credentials are only needed during download; they are not included in the air-gapped tarball or environment.

  • For further details, refer to the Docker Hub pull rate limit documentation.

Step 6: Download packages, images, charts, and CRDs#

Create a working directory and run the download scripts in sequence, completing each before starting the next. All scripts write output to the current working directory.

Download Packages and Artifacts
mkdir -p airgapped
cd airgapped

KUBE_VERSION="1.35"

# OS-specific package download (choose one):
download_ubuntu_packages.sh --kube-version=$KUBE_VERSION   # Ubuntu Linux
download_r9_packages.sh --kube-version=$KUBE_VERSION       # RHEL variants (Rocky, RHEL)

# Container images, Helm charts, and CRDs:
download_container_images.sh --kube-version=$KUBE_VERSION
download_helm_charts.sh
download_crds.sh

Kubernetes version#

The preceding example uses Kubernetes 1.35. Adjust the --kube-version flags to match your deployment target.

If --kube-version is omitted, the latest available version is used, which may not match your deployment target.

Multi-architecture clusters#

  • download_ubuntu_packages.sh, download_r9_packages.sh, and download_container_images.sh accept a single --arch option followed by one or more space-separated architectures. Use this option to override the download host’s architecture.

  • Helm charts and CRDs are not architecture-specific; those download scripts have no --arch option.

Example — download container images for more than one architecture:

download_container_images.sh \
    --kube-version=$KUBE_VERSION \
    --arch amd64 arm64

Note

If only one architecture is needed, run the command with --arch amd64 or --arch arm64 instead to avoid downloading images for the other architecture.

On Ubuntu, pulling packages for an architecture other than the host’s requires that architecture to be registered with dpkg; the scripts add it automatically where needed.

After all downloads, the directory should contain subdirectories such as crds, helm-charts, k8s-images, packages, packages-cuda, packages-head, and packages-non-dgx.

Step 7: Stage the Run:ai Air-Gapped Tarball#

The runai component fetched by nmc-airgap includes runai-airgapped-package-<version>.tar.gz for your NMC release. Copy this existing artifact into the airgapped working directory before creating the transfer archive.

From the airgapped directory, locate the package in the fetched bundle. Replace <bundle-dir> with the path to the bundle created by nmc-airgap fetch:

find <bundle-dir>/files -type f -name 'runai-airgapped-package-*.tar.gz'

Copy the package for your NMC release into the current directory. Replace <runai-package-path> with the matching path from the preceding command, and retain the original filename:

cp <runai-package-path> .

The Run:ai wizard uses this transferred package for installation. Its extracted runai-ag/charts/control-plane.tgz and runai-ag/charts/runai-cluster.tgz files are also used for the post-installation placement updates.

The remaining preparation steps in this topic are shared by both clusters. You do not repeat them for k8s-user.

Step 8: Create and transfer the tarball#

Create the tarball from the parent directory:

cd ..
tar -czf airgapped.tgz airgapped

Transfer airgapped.tgz to the air-gapped BCM head node using removable media (for example, a USB drive).