Initial Host Preparations for BCM Kubernetes Air-gapped Installation#

These procedures are based on Appendix C.1 of the NVIDIA Base Command Manager 11 Containerization Manual.

Perform all steps on an internet-connected host. The resulting airgapped.tgz tarball is then transferred to the air-gapped BCM head node.

Note

A BCM 11.33.1 virtual machine is recommended, since it already provides a matching OS, configured BCM repositories, and the airgap-scripts directory.

Reference environment#

Reference cluster configuration:

  • Air-gapped (no internet) cluster

  • Kubernetes version 1.34 (adjust --kube-version as needed)

  • BCM version 11.33.1

Initial host requirements#

The preparation host must have:

  • Internet access

  • OS matching the air-gapped cluster: Ubuntu 24.04

  • CPU architecture matching the air-gapped cluster

    • The tarball supports one architecture at a time. Mixed-architecture clusters (for example, x86-64 head node with ARMv8 control plane nodes) require a customized approach.

    • Use the --help output of the download_*.sh scripts and the --arch option for architecture-specific configurations.

Step 1: Prepare the internet-connected host#

Verify the preparation host meets these requirements:

  • OS matches the air-gapped cluster (Ubuntu 24.04)

  • CPU architecture matches the air-gapped cluster

  • BCM package repositories are configured (refer to Step 2)

  • airgap-scripts directory from BCM 11.33.1 or later is available

Step 2: Configure BCM package repositories#

Copy cm-*.list and ubuntu.sources from /etc/apt/sources.list.d/ on a BCM cluster running the same OS as your air-gapped cluster.

Note

The architecture workarounds in this step are only required if your cluster has mixed-architecture nodes — for example, when the head node, Kubernetes nodes, or compute nodes do not all use the same CPU architecture. If all nodes use the same architecture, continue with Step 3.

Known architecture workarounds (BCM 11.33.1)#

BCM 11.33.1 has known bugs in Ubuntu repository file specifications, expected to be fixed in BCM 11.34. Apply the following workarounds:

ubuntu.sources — Add an architecture specification. Replace the default file with:

Types: deb
URIs: http://archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: amd64

Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: amd64

Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports/
Suites: noble noble-updates noble-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: arm64

Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports/
Suites: noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: arm64

cm-cuda-ubuntu2404-*.list — Modify the CUDA repository list so packages can be picked up for the needed architecture (* is either x86_64 or arm64):

deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=amd64] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /
deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=arm64] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/ /

cm.list — Add the extra architectures needed:

deb [arch=amd64] http://<user>:<password>@updates-us-east.brightcomputing.com/deb/cm/11.0/amd64/ubuntu/ noble main
deb [arch=arm64] http://<user>:<password>@updates-us-east.brightcomputing.com/deb/cm/11.0/arm64/ubuntu/ noble main
deb [arch=amd64] http://<user>:<password>@updates.brightcomputing.com/deb/cm/11.0/amd64/ubuntu/ noble main
deb [arch=arm64] http://<user>:<password>@updates.brightcomputing.com/deb/cm/11.0/arm64/ubuntu/ noble main

  • The examples above use the traditional one-line format for architecture specifications in square brackets; deb822 format is also supported.

  • Use the <user>:<password> credentials from the original cm.list on the working cluster.

  • The server URL path may differ in your environment.

Step 3: Install BCM air-gap scripts#

The BCM airgap-scripts directory from the matching BCM version must be available on the preparation host.

If the cm-setup package is installed, the scripts are at:

/cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts

When the cm-setup module is loaded (enabled by default), this path is available via the K8S_AG_SCRIPTS environment variable.

If cm-setup is already installed#

For BCM version 11, the air-gap scripts are in the path defined by K8S_AG_SCRIPTS:

echo $K8S_AG_SCRIPTS
# /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts/

For BCM version 10, the Python directory path uses python3.9 instead.

Add the scripts to PATH:

export PATH=$PATH:$K8S_AG_SCRIPTS

If cm-setup is not installed#

Copy the airgap-scripts directory to the preparation host, define K8S_AG_SCRIPTS to point to it, then add it to PATH:

export PATH=$PATH:$K8S_AG_SCRIPTS

Step 4: Install Helm 3.x#

Install Helm 3.x (not 4.x) at /usr/local/bin/helm using the BCM helper script:

download_helm_binary.sh

Example output:

Downloading helm v3.20.2 for linux-amd64...
Helm binary installed at /usr/local/bin/helm

Verify the installation:

helm version

Step 5: Install Skopeo#

On a BCM head node, install Skopeo (image manager):

apt update   # if necessary
apt install skopeo

Optional: Authenticate with Docker Hub#

To work around Docker Hub pull rate limits, sign up at https://app.docker.com/signup and authenticate before running the download scripts:

skopeo login docker.io

  • Unauthenticated pulls are limited to 100 per 6 hours.

  • Authenticated pulls are limited to 200 per 6 hours — sufficient for preparing the air-gapped tarball.

  • Credentials are only needed during download; they are not included in the air-gapped tarball or environment.

  • For further details, refer to the Docker Hub pull rate limit documentation.

Step 6: Download packages, images, charts, and CRDs#

Create a working directory and run the download scripts in sequence, completing each before starting the next. All scripts write output to the current working directory.

mkdir -p airgapped
cd airgapped

KUBE_VERSION="1.34"

# OS-specific package download (choose one):
download_ubuntu_packages.sh --kube-version=$KUBE_VERSION   # Ubuntu Linux
download_r9_packages.sh --kube-version=$KUBE_VERSION       # RHEL variants (Rocky, RHEL)

# Container images, Helm charts, and CRDs:
download_container_images.sh --kube-version=$KUBE_VERSION
download_helm_charts.sh
download_crds.sh

Kubernetes version#

The example above uses Kubernetes 1.34. Adjust the --kube-version flags to match your deployment target.

If --kube-version is omitted, the latest available version is used, which may not match your deployment target.

Multi-architecture clusters#

  • download_ubuntu_packages.sh, download_r9_packages.sh, and download_container_images.sh accept --arch to override the architecture of the host performing the download.

  • Helm charts and CRDs are not architecture-specific; those download scripts have no --arch option.

Example — download container images for more than one architecture:

download_container_images.sh --kube-version=$KUBE_VERSION --arch amd64 --arch arm64

Note

If only one architecture is needed, run the command with --arch amd64 or --arch arm64 instead to avoid downloading images for the other architecture.

On Ubuntu, pulling packages for an architecture other than the host’s requires that architecture to be registered with dpkg; the scripts add it automatically where needed.

After all downloads, the directory should contain subdirectories such as crds, helm-charts, k8s-images, packages, packages-cuda, packages-head, and packages-non-dgx.

Step 7: Create and transfer the tarball#

Create the tarball from the parent directory:

cd ..
tar -czf airgapped.tgz airgapped

Transfer airgapped.tgz to the air-gapped BCM head node using removable media (for example, a USB drive).