> This page is for version 11 (default).
> For other versions, use one of these documentation indexes:
> - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt
> - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server.

# Kubernetes Upgrade On An Air-gapped BCM Cluster

Section [4.22](/containerization/kubernetes#422-kubernetes-upgrades) covers Kubernetes upgrades in the case of a BCM Kubernetes cluster connected directly to the internet. An air-gapped cluster environment is one where the cluster is not directly connected to the internet.

A BCM cluster that air-gapped and already runs Kubernetes can be upgraded with the procedure outlined in this current section (section C). It is based on the upgrade procedure in section [4.22](/containerization/kubernetes#422-kubernetes-upgrades). The main differences are that the BCM version requirements differ, and that a local registry must be built.

Appendix [B](/containerization/kubernetesairgapinstallation) describes how an air-gapped cluster can be created if it does not already exist.

## C.1  Air-gapped Kubernetes Upgrade Prerequisites

Before beginning the upgrade, the following general prerequisites and version checks must be carried out:

* The cluster version must be at a sufficiently high point release number ([section 9.1.2 of the *Administrator Manual*](/admin/softwareimagemanagement#912-bcm-package-point-release-versions-and-the-cm-package-release-info-command)):

  * for BCM version 10 the point release version number must be 10.31.0 or later

  * for BCM version 11 the point release version number must be 11.32.0 or later

* The upgrade path should take into account the Kubernetes version skew policy, as defined at [https://kubernetes.io/releases/version-skew-policy/](https://kubernetes.io/releases/version-skew-policy/). The principles to take into account are:

  * Only two adjacent minor releases may exist in the cluster at any time. So, v1.34 and v1.35 are acceptable; but v1.33, v1.34, and v1.35 are not acceptable.

  * The target version must not violate the skew values. So, if the cluster nodes are currently on two different adjacent minor releases, then the only valid upgrade path is to the newest of the two minor releases. For example if some nodes are running v1.34 and some are running v1.35, then the only valid upgrade path is to upgrade the v1.34 nodes to v1.35. This ensures that the cluster remains functional and compliant with the Kubernetes version skew policies during the rolling upgrade process.

* The upgrade path should also take into account the NVIDIA BCM Release Notes and Compatibility Matrix corresponding to the BCM version that is running. This must be consulted, to see if there is support for the desired target Kubernetes version (for example v1.35.x).

* An upgrade path preference, for minor or patch, must be decided on for the case where all the nodes are on the same minor version:

  * Minor release upgrade: the latest release of the next minor release (for example: v1.34 to v1.35)

  * Patch release upgrade: the latest patch release of the current minor release (for example: v1.35.3 to v1.35.4)

* The current Kubernetes cluster must already be operational, with healthy and ready nodes.

* A local registry must already be deployed, and its FQDN must be retrievable (for example: master.cm.cluster:5000)

* The upstream instructions on preparing for the upgrade ([https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin)) should be read.

* A node that is to be upgraded must be drained.

## C.2  Initial Host Preparations For BCM Kubernetes Air-gapped Upgrade

The initial steps must be performed on an initial host computer that has internet access. The initial host requires:

* Internet access

* An OS that matches that of the air-gapped cluster: Ubuntu 22.04, Ubuntu 24.04, Ubuntu 26.04, Rocky Linux 9, or RHEL 9.

* An architecture that matches that of the air-gapped cluster

  * Currently, the tarball supports one architecture at a time. So if the cluster has a mixed architecture, for example with the head node being x86-64 (using `x86_64` RPM packages or `amd64` APT packages), but the Kubernetes control plane nodes being ARMv8 (using `arm64` packages), then that requires a more customized approach.

    For architecture configurations, the `--help` output of the download Bash scripts (format: `download_` \<*descriptive label*>`.sh`) indicates how the `--arch` option can be used. These scripts are located in the `airgap-scripts` directory, described shortly.

* The BCM package repositories must be configured on the initial host. The relevant BCM repository list files can be copied over from a working cluster with the same OS.

  * For Ubuntu, these files are of the form `cm-*.list` and `ubuntu.sources`, and are located within the `/etc/apt/sources.list.d/` directory.

    There are some known architecture-related bugs in the file specifications for the Ubuntu repositories. These should be fixed for the BCM 11.34 point release. The issues and associated workarounds for the 11.33 release are in the following files:

    * `ubuntu.sources`: The default Ubuntu sources file needs an architecture specification to be added. The default file can be replaced with the following deb822-format (`man sources.list.5`) text:

      **Example**

      ```bash
      root@basecm11:/etc/apt/sources.list.d# cat ubuntu.sources
      Types: deb
      URIs: http://archive.ubuntu.com/ubuntu/
      Suites: noble noble-updates noble-backports
      Components: main restricted universe multiverse
      Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
      Architectures: amd64

      Types: deb
      URIs: http://security.ubuntu.com/ubuntu/
      Suites: noble-security
      Components: main restricted universe multiverse
      Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
      Architectures: amd64

      Types: deb
      URIs: http://ports.ubuntu.com/ubuntu-ports/
      Suites: noble noble-updates noble-backports
      Components: main restricted universe multiverse
      Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
      Architectures: arm64

      Types: deb
      URIs: http://ports.ubuntu.com/ubuntu-ports/
      Suites: noble-security
      Components: main restricted universe multiverse
      Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
      Architectures: arm64
      ```

    * `cm-cuda-ubuntu2404-*.list`: The file for the default CUDA repositories list is modified as needed. Here, the “\*” is either `x86_64` or `arm64`. The change is to allow CUDA packages to be picked up from whichever architecture is needed:

      **Example**

      ```bash
      root@basecm11:/etc/apt/sources.list.d# cat cm-cuda-ubuntu2404-x86_64.list
      deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=amd64] https://\
      developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /
      deb [signed-by=/usr/share/keyrings/cm-cuda-archive-keyring.gpg arch=arm64] https://\
      developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/ /
      ```

    * `cm.list`. The BCM repository list file is modified so that the extra architectures that are needed are specified:

      **Example**

      ```bash
      root@basecm11:/etc/apt/sources.list.d# cat cm.list
      deb [arch=amd64] http://<user>:<password>@updates-us-east.brightcomputing.com/\
      deb/cm/11.0/amd64/ubuntu/ noble main
      deb [arch=arm64] http://<user>:<password>@updates-us-east.brightcomputing.com/\
      deb/cm/11.0/arm64/ubuntu/ noble main
      deb [arch=amd64] http://<user>:<password>@updates.brightcomputing.com/\
      deb/cm/11.0/amd64/ubuntu/ noble main
      deb [arch=arm64] http://<user>:<password>@updates.brightcomputing.com/\
      deb/cm/11.0/arm64/ubuntu/ noble main
      ```

      In the preceding session:

      * The architecture key/value entries are specified for the repositories in the square brackets, using the traditional one-line format, but using a deb822 format is also possible.

      * The URLs specified in the preceding session take the \<*user*>`:<`*password*> format indicated. The values that are used for \<*user*>`:<`*password*> should already be configured in the original `cm.list` repository file, and can simply be taken over.

      * The server and path of the URL on the actual cluster may differ from the preceding example.

  * For RHEL variants these files should be `cm.repo` and `epel.repo`, located within `/etc/yum.repos.d/`

* The BCM `airgap-scripts` directory from the matching BCM version must be on the initial host. How the installation of that directory is carried out depends on if `cm-setup` is on the initial host, as explained next.

#### C.2.0.1  Initial Host Has `cm-setup` Installed

If the `cm-setup` package is already installed on the initial host, then for BCM version 11, the air-gap scripts are by default in the path defined by the environment variable `K8S_AG_SCRIPTS`.

**Example**

```bash
# echo $K8S_AG_SCRIPTS
/cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/airgap-scripts/
```

For BCM version 10, the Python directory path is `python3.9` instead.

This environment variable path should be added to the existing PATH:

```bash
# export PATH=$PATH:$K8S_AG_SCRIPTS
```

#### C.2.0.2  Initial Host Does Not Have `cm-setup` Installed

If the `cm-setup` package is not already installed on the initial host, then the administrator should define the air-gap scripts path, `$K8S_AG_SCRIPTS`, and then add the air-gap scripts to the PATH environment.

```bash
# export K8S_AG_SCRIPTS=<path-to-airgap-scripts>
# export PATH=$PATH:$K8S_AG_SCRIPTS
```

#### C.2.0.3  Using The Air-gap Scripts

* `helm 3.x` (not 4.x at the time of writing, April 2026) must be added to `/usr/local/bin/helm` as per the instructions at [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/).

  Alternatively, the following helper script, that BCM ships with `cm-setup`, can be used:

  **Example**

  ```bash
  root@internet-host:~# download_helm_binary.sh
  Downloading helm v3.20.2 for linux-amd64...
  Helm binary installed at /usr/local/bin/helm
  root@internet-host:~#
  ```

  Optionally, a Helm check can be carried out with:

  **Example**

  ```bash
  root@internet-host:~# helm version
  version.BuildInfo{Version:"v3.20.2", GitCommit:"8fb76d6ab555577e98e23b7500009537a471feee",
   GitTreeState:"clean", GoVersion:"go1.25.9"}
  ```

* Skopeo, an image manager, is installed. This installation is carried out on the internet-connected host only, and not on the air-gapped head node. Skopeo is needed here in order to pull the container images that make up the tarball:

  **Example**

  ```bash
  root@internet-host:~# apt install skopeo      #or dnf install skopeo for RHEL variants
  ```

  The air-gapped head node obtains its own copy of Skopeo from the tarball, and no attempt should be made to install Skopeo there from a distribution repository (section [C.3](#c3-head-node-preparation-steps)).

  * To work around a potential Docker Hub pull rate limit issue ([https://docs.docker.com/docker-hub/usage/pulls/](https://docs.docker.com/docker-hub/usage/pulls/)) the administrator can now sign up for an account at [https://app.docker.com/signup](https://app.docker.com/signup), and authenticate with `skopeo login docker.io`. This is to help with running the image installation scripts from the `airgapped` directory in the next bullet point.

    Any credentials are fine—the credentials will not end up in the air-gapped tarball or environment. They are only needed for authentication to conveniently pull images from Docker Hub. This is because unauthenticated pulls are limited to a rate of 100 per 6 hours. Authenticated pulls, on the other hand, are allowed at a rate of 200 per 6 hours, which should be sufficient for preparing the `airgapped` tarball.

* A working directory, for example `/root/airgapped`, can then be created, and installation scripts within the `airgapped` directory can then be run:

  **Example**

  ```bash
  root@internet-host:~# pwd
  /root
  root@internet-host:~# mkdir airgapped; cd airgapped
  root@internet-host:~/airgapped# download_ubuntu_packages.sh --kube-version=1.34  # for Ubuntu Linux
  ...
  [root@internet-host airgapped]# download_r9_packages.sh --kube-version=1.34    # for RHEL variants
  ...
  root@internet-host:~/airgapped# download_container_images.sh --kube-version=1.34
  ...
  root@internet-host:~/airgapped# download_helm_charts.sh
  ...
  root@internet-host:~/airgapped# download_crds.sh
  ...
  ```

  The preceding scripted downloads are for Kubernetes version 1.34. To use a different version, such as v1.33 or v1.35, the `--kube-version` flag should be adjusted accordingly. If `--kube-version` is not set on `download_ubuntu_packages.sh`, `download_r9_packages.sh`, or `download_container_images.sh`, each script falls back to its own default Kubernetes version, which is not necessarily the version that is to be deployed. `download_helm_charts.sh` and `download_crds.sh` take no version-related option, and are unaffected by this.

  `download_ubuntu_packages.sh`, `download_r9_packages.sh`, and `download_container_images.sh` also accept a `-c`/`--config` option, giving the path to a `cm-kubernetes-setup.conf` configuration file to read the Kubernetes version from, instead of `--kube-version`. On `download_ubuntu_packages.sh` and `download_r9_packages.sh`, `--kube-version` and `-c` are mutually exclusive: only one may be given. On `download_container_images.sh` both may be given together, in which case `--kube-version` takes precedence over the value in the configuration file.

  Using `-c` this way requires a `cm-kubernetes-setup.conf` file to already exist, generated by a previous run of the `cm-kubernetes-setup` wizard.

  The charts fetched by `download_helm_charts.sh` determine which NVIDIA GPU Operator versions can be deployed later on. The `cm-kubernetes-setup` wizard presents a list of GPU Operator versions, but in an air-gapped cluster only a version whose chart is among the downloaded ones can be installed. There is no route to fetch any of the others once the cluster is air-gapped.

  Certification and availability are separate properties: an asterisk next to a version in the wizard marks it as certified for NVIDIA AI Enterprise (section [6.7.1](/containerization/kubernetesoperators#671-installing-the-nvidia-gpu-operator-using-the-wizard)). A version that was downloaded is not necessarily a certified one, and a certified version is not necessarily among those that were downloaded.

  The downloaded charts can therefore be listed before `cm-kubernetes-setup` is run, so that the version picked in the wizard is known to be one that is available:

  ```bash
  root@internet-host:~/airgapped# ls -1 helm-charts/
  ```

  The same `helm-charts` directory is present inside the extracted `airgapped` directory on the head node, so the charts can also be checked there.

  The charts are fetched from chart repositories on the internet, and the set of repositories that is used changes between BCM releases. BCM 11.34 fetches from four repositories that BCM 11.33 did not:

  * `oci://cr.agentgateway.dev/charts/agentgateway`, for the AgentGateway operator

  * `oci://cr.agentgateway.dev/charts/agentgateway-crds`, for the AgentGateway CRDs operator

  * [https://argoproj.github.io/argo-helm](https://argoproj.github.io/argo-helm), for the Argo CD operator

  * [https://flannel-io.github.io/flannel](https://flannel-io.github.io/flannel), for the Flannel network plug-in

  A site that restricts outbound connections from the internet-connected host must permit these destinations before `download_helm_charts.sh` is run. A site that is moving from BCM 11.33 must add them to any existing list of permitted destinations.

  The repository that serves the Calico CRDs chart, [https://docs.tigera.io/calico/charts](https://docs.tigera.io/calico/charts), is not a new one. It already served the Tigera Operator chart in BCM 11.33.

  To ensure the commands all succeed, it is best to execute them in sequence and let them complete. The download scripts all write their output to the current working directory.

  The downloads take a while, and are best run inside a `screen` or `tmux` session so that a dropped connection does not interrupt them. The container image download accounts for most of the elapsed time—on a test system with fast networking it took around 15 times as long as the package, chart, and CRD downloads put together. Absolute times depend on the available bandwidth, so a slow download is not in itself a sign that something has gone wrong.

  While the packages are downloading, APT emits warnings similar to:

  ```bash
  W: Download is performed unsandboxed as root as file
  '/root/airgapped/packages/all/nginx-common_1.24.0-2ubuntu7.15_all.deb'
  couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
  ```

  One or more of these appears for each of the `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` directories. *These warnings are expected and can be ignored.* The `_apt` user (with the underscore) cannot read the working directory when it is under `/root`, so APT drops its download sandbox and continues unsandboxed. The downloads themselves succeed, and the scripts exit normally. This is the same permission restriction that later requires the tarball to be extracted somewhere that APT can write to (section [C.3](#c3-head-node-preparation-steps)).

  Without the Run:ai air-gapped package, the completed download is about 8.6 GB, of which the container images account for about 8.1 GB. The Run:ai package is a further 13–14 GB on top of that, so a working directory for a deployment that includes Run:ai needs upwards of 20 GB. Note that the `total` reported by `ls` in the listing that follows counts only the files in the directory itself, which is why it reflects the size of the Run:ai tarball rather than the size of the whole download.

* Run:ai air-gapped setup

  If Run:ai is required in the air-gapped installation, then the Run:ai air-gapped package must also be downloaded to the working directory. The package is published as an NGC resource. The available versions are listed on the resource page at [https://catalog.ngc.nvidia.com/orgs/nvidia/teams/runai/resources/runai-airgapp-package/version](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/runai/resources/runai-airgapp-package/version), and an NGC API key is needed to download it ([https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#ngc-api-keys](https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#ngc-api-keys)):

  **Example**

  ```bash
  root@internet-host:~/airgapped# curl -LO --request GET \
  "https://api.ngc.nvidia.com/v2/org/nvidia/team/runai/resources/\
  runai-airgapp-package/versions/<runai version>/files/\
  runai-airgapped-package-<runai version>.tar.gz" \
  -H "Authorization: Bearer <NGC API key>" \
  -H "Content-Type: application/json"
  ```

  The \<*`runai version`*> part is replaced with the required patch version. For example: `2.25.11`, which gives a tarball named:

  `runai-airgapped-package-2.25.11.tar.gz`

  Alternatively, NVIDIA Run:ai Support can be contacted to request the appropriate tarball.

  A minimum version of 2.23.60 is required, since that is when multi-arch support was added. The integrity of the downloaded file should be verified with `sha256sum` against the checksum provided by NVIDIA Run:ai Support.

  The `cm-setup-runai-bcm11` package must also be downloaded to the working directory. This package implements the `cm-runai-setup` wizard, and is not part of the Run:ai air-gapped tarball. Listing the contents of the tarball does not show it. The package is published separately, in the NVIDIA Mission Control package repository, which is publicly reachable and needs no credentials. The available versions are listed at:

  [https://edge.urm.nvidia.com/artifactory/sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/](https://edge.urm.nvidia.com/artifactory/sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/)

  The package is published only as a `.deb`, so an air-gapped Run:ai deployment on RHEL is not supported in this release. The package is architecture-specific, so the build that matches the head node architecture must be downloaded:

  **Example**

  ```bash
  root@internet-host:~/airgapped# curl -LO "https://edge.urm.nvidia.com/artifactory/\
  sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/\
  cm-setup-runai-bcm11_<version>_<arch>.deb"
  ```

  In the repository each package has a corresponding `.sha256` file. The content of the `.sha256` file is the checksum. For the `sha256sum` utility to be able to check the package integrity, the `.sha256` file must have the file name of its corresponding package appended on the same line to the `.sha256` file checksum content:

  **Example**

  ```bash
  root@internet-host:~/airgapped# curl -sO "https://edge.urm.nvidia.com/artifactory/\
  sw-dgxc-nmc-cm-setup-plugins-debian-local/pool/cm-setup-runai/bcm11/\
  cm-setup-runai-bcm11_<version>_<arch>.deb.sha256"
  root@internet-host:~/airgapped# grep -qF "cm-setup-runai-bcm11_<version>_<arch>.deb" \
  cm-setup-runai-bcm11_<version>_<arch>.deb.sha256 || echo "  cm-setup-runai-bcm11_<version>_<arch>.deb" \
  >> cm-setup-runai-bcm11_<version>_<arch>.deb.sha256
  root@internet-host:~/airgapped# sha256sum -c cm-setup-runai-bcm11_<version>_<arch>.deb.sha256
  cm-setup-runai-bcm11_<version>_<arch>.deb: OK
  ```

  The package must not be installed if the check reports anything other than `OK`.

  The directory contents should then appear similar to:

  ```bash
  root@internet-host:~/airgapped# ls -alrth
  total 14G
  drwx------ 10 root root  287 Jun  8 04:21 ..
  drwxr-xr-x  4 root root   30 Jun  8 04:21 packages
  drwxr-xr-x  4 root root   30 Jun  8 04:21 packages-head
  drwxr-xr-x  3 root root   19 Jun  8 04:21 packages-non-dgx
  drwxr-xr-x  3 root root   19 Jun  8 04:21 packages-cuda
  drwxr-xr-x  2 root root   88 Jun  8 04:27 k8s-images
  drwxr-xr-x  2 root root 4.0K Jun  8 04:29 helm-charts
  drwxr-xr-x  2 root root   40 Jun  8 04:29 crds
  drwxr-xr-x  9 root root  183 Jun  8 04:30 .
  -rw-r--r--  1 root root   41K Jun  8 04:31 cm-setup-runai-bcm11_2.5.5_amd64.deb
  -rw-r--r--  1 root root  14G Jun  8 04:31 runai-airgapped-package-2.25.11.tar.gz
  ```

* Confirming the downloaded content is complete

  **Verifying nothing is missing before tarballing:**

  The download scripts run in the preceding steps make this repository complete for the release being installed. No additional package, image, or chart list needs to be assembled by hand. Each directory in the preceding listing corresponds to one content category:

  * `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` hold the OS packages fetched by `download_ubuntu_packages.sh` or `download_r9_packages.sh`

  * `k8s-images` holds the container images fetched by `download_container_images.sh`

  * `helm-charts` holds the Helm charts fetched by `download_helm_charts.sh`, from every chart repository this release depends on (see the repository list earlier in this section)

  * `crds` holds the CRD manifests fetched by `download_crds.sh`

  The exact package names, image tags, and chart versions in each directory change between BCM and Kubernetes releases. Checking their contents against a fixed list is not useful. Instead, each category should be confirmed to have downloaded files before proceeding. A plain listing of the top-level directories is not enough: `packages`, `packages-head`, `packages-non-dgx`, and `packages-cuda` organize their packages into subdirectories such as `all` and `amd64`, and those subdirectories can exist without containing any packages. Searching recursively for files avoids that gap:

  **Example**

  ```bash
  root@internet-host:~/airgapped# find packages packages-head packages-non-dgx \
  packages-cuda k8s-images helm-charts crds -type f
  ```

  A content category that produces no output has no downloaded files, even if its top-level directory (or a subdirectory within it) exists. Its download script did not complete, and should be re-run before the `airgapped` directory is tarballed.

* The `airgapped` directory can now be tarballed.

  In the tarball, the packages that are to be used by the software images must include all the architectures that are to be used by the Kubernetes cluster.

  **Example**

  ```bash
  root@internet-host:~/airgapped# ls
  cm-setup-runai-bcm11_2.5.5_amd64.deb  crds  helm-charts  k8s-images
  packages  packages-cuda  packages-head  packages-non-dgx
  runai-airgapped-package-2.25.11.tar.gz
  root@internet-host:~/airgapped# cd ..
  root@internet-host:~# tar -cvzf airgapped.tgz airgapped
  ```

  The `airgapped.tgz` file should be saved to some transfer media (for example: a USB stick), and should be copied over to the active head node of the air-gapped cluster.

## C.3  Head Node Preparation Steps

Before starting this section, the cluster’s control-plane, worker, and etcd nodes must already be assigned, as covered under **Node assignment** (section [4.2.8](/containerization/kubernetes#428-kubernetes-setup-from-a-tui-session), [here](/containerization/kubernetes#kubernetes-setup-from-a-tui-session-node-assignment)). This assignment is done using the same `cm-kubernetes-setup` wizard screens used here, is not air-gap-specific, and cannot be arranged once the cluster is air-gapped.

#### C.3.0.1  Selecting The Run:ai Operator Dependencies

When the `cm-kubernetes-setup` wizard is run, with the intent of running Run:ai as well, the operator selection screen must include the dependencies that Run:ai requires. For Run:ai 2.25, these are:

* NVIDIA GPU Operator

* Grafana Operator

* Kgateway

* Knative Operator

* Kubeflow Training Operator

* Kubernetes Metrics Server

* Kubernetes MPI Operator

* Kubernetes State Metrics

* LeaderWorkerSet Operator

* MetalLB

* Network Operator (technically optional, but recommended)

* NIM Operator (optional)

* Prometheus Adapter

* Prometheus Operator Stack

The Run:ai operator and its prerequisites are described in more detail in section [6.11](/containerization/kubernetesoperators#611-the-runai-operator).

### C.3.1  Unpacking The Files

The `airgapped.tgz` file that was copied over to the head node can then be extracted. For Ubuntu, extracting to a file system that APT can write to, such as `/tmp`, is recommended. This is because APT runs as the `_apt` user (with the underscore) during a part of the installation process, and `_apt` does not have sufficient permissions under `/root` to process the extracted files as needed. For Rocky Linux and RHEL the archive can be extracted in place, and the move to `/tmp` or similar is not needed.

**Example**

```bash
root@basecm11:~# mv airgapped.tgz /tmp/
root@basecm11:~# cd /tmp
root@basecm11:/tmp# tar xvf airgapped.tgz
root@basecm11:/tmp# cd airgapped
```

The system packages can now be installed onto the head node and the required software images, using the untarballed airgap system files together with the configuration file saved from earlier on. The command must be run from within the extracted `airgapped` directory, because the installer looks for the `packages` subdirectories relative to the current working directory:

**Example**

```bash
root@basecm11:/tmp/airgapped# cm-kubernetes-setup --airgap-install-system-packages \
-c /root/cm-kubernetes-setup.conf
```

The packages that this step installs include Skopeo and its dependencies, which are carried in the `packages-head` directory of the tarball. Skopeo is required later on, when the container images are pushed into the local registry. It should not be installed on the head node with `apt` or `dnf`: an air-gapped head node has no distribution repository to install it from, and the step shown here has already provided it.

If the air-gapped cluster has a secondary head node ([section 15.1 of the *Administrator Manual*](/admin/ha#151-ha-concepts)), then the system package installation does not have to be repeated on it. The `--airgap-install-system-packages` run on the active head node also updates the secondary head node. The tarball and the configuration file therefore do not have to be copied across for this step, and `cm-kubernetes-setup` should not be run on a passive head node.

This applies to the system package installation. Other components can still have their own secondary head node steps.

Earlier BCM releases did instruct that the tarball and the configuration file be copied to the secondary head node, and that the wizard be run there a second time with the `--skip-headnode-check` flag. That run reaches a stage which installs packages into the software image, and it is refused on a passive head node with the message `This node is not the active head node (PASSIVE)`. The documented answer was `Skip`, which leaves the software image untouched. The message also suggests `--force`, which must not be used here. Software images are local to each head node and are synchronized from the active head node. This means that a write made on a passive head node is reverted at the next synchronization, and that until then the passive head node serves a different image to any node provisioning from it compared to the active head node.

Package installation for multiarch software images in the Kubernetes cluster is automatic. This is assuming that the software images have the packages for the required architectures in the tgz file.

### C.3.2  Setting Up The Container Images And Helm Charts For An Air-gapped Cluster During Upgrade

From the airgapped directory, all Docker images and Helm charts can be pushed to the registry with:

```
root@basecm11:~# cm-kubernetes-setup --airgap-push-container-images -c /root/cm-kubernetes-setup.conf
root@basecm11:~# cm-kubernetes-setup --airgap-push-helm-charts -c /root/cm-kubernetes-setup.conf
```

### C.3.3  Extracting The CRD Files For An Air-gapped Cluster During Upgrade

The downloaded CRD files can then be extracted using:

`root@basecm11:~# $K8S_AG_SCRIPTS/extract_crds.sh`

## C.4  Upgrading The Kubernetes Cluster

The upgrade steps start with the control plane nodes.

### C.4.1  Updating The First Control Plane Node

The node that is to be upgraded should first be updated to the latest binaries. This can be done with an `imageupdate` run from `cmsh` on the head node ([section 5.6.2 of the *Administrator Manual*](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)).

For example for a node with the arbitrary name of `controlplane01`:

**Example**

```bash
root@headnode:~# cmsh -c 'device; imageupdate -w controlplane01 --wait'
...
...[notice] headnode: Provisioning started: sending headnode:/cm/images/default-image
        to controlplane01:/, mode UPDATE, dry run = no
...
...[notice] headnode: Provisioning completed: sent headnode:/cm/images/default-image
        to controlplane01:/, mode UPDATE, dry run = no
```

If the `-w|--write` flag is not used, then a dry-run is carried out.

After the image update is done, the `kubeadm upgrade plan` command is run from the node itself:

**Example**

```bash
root@headnode:~# ssh controlplane01
...
root@controlplane01:~# kubeadm upgrade plan --kubeconfig=/root/.kube/config-default
```

Explicitly providing the kubeconfig for the correct Kubernetes cluster is a good practice. This is important if the BCM cluster has multiple Kubernetes clusters.

The following session shows output from an upgrade plan from 1.32.10 to 1.33.11. The plan shows that there is indeed a newer 1.32.11 available already:

**Example**

```bash
...
[upgrade/versions] Target version: v1.32.10
[upgrade/versions] Latest version in the v1.32 series: v1.32.11

Components that must be upgraded manually after you have upgraded the control plane
with 'kubeadm upgrade apply':

COMPONENT   NODE      CURRENT   TARGET

kubelet     node001   v1.32.10   v1.32.11
kubelet     node002   v1.32.10   v1.32.11
kubelet     node003   v1.32.10   v1.32.11
kubelet     node004   v1.32.10   v1.32.11
kubelet     node005   v1.32.10   v1.32.11
kubelet     node006   v1.32.10   v1.32.11
kubelet     node007   v1.32.10   v1.32.11
kubelet     node008   v1.32.10   v1.32.11

Upgrade to the latest version in the v1.32 series:

COMPONENT                 NODE      CURRENT    TARGET

kube-apiserver            node001   v1.32.10   v1.32.11
kube-apiserver            node002   v1.32.10   v1.32.11
kube-apiserver            node003   v1.32.10   v1.32.11
kube-controller-manager   node001   v1.32.10   v1.32.11
kube-controller-manager   node002   v1.32.10   v1.32.11
kube-controller-manager   node003   v1.32.10   v1.32.11
kube-scheduler            node001   v1.32.10   v1.32.11
kube-scheduler            node002   v1.32.10   v1.32.11
kube-scheduler            node003   v1.32.10   v1.32.11
kube-proxy                          1.32.10    v1.32.11
CoreDNS                             v1.11.3    v1.12.0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.32.11
```

The upgrade plan suggests that an upgrade to v1.32.11 must be done first.

However, the skew versions policies ([here](#c1-air-gapped-kubernetes-upgrade-prerequisites)) allow jumping to v1.33.6.

A check of the changelogs through the appropriate versions should be done. The changelogs are located under: [https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/) and are well-organized. After studying these, if the cluster administrator decides it makes sense to just jump ahead to v1.33.6, then that can be carried out as follows:

Instead using `v1.32.11` as suggested in the preceding session plan, the value `v1.33.6` is used. The `--kubeconfig` flag is also set as a matter of best practice:

**Example**

```bash
root@controlplane01:~# kubeadm upgrade apply v1.33.6 --kubeconfig=/root/.kube/config-default
```

The following should then show in the output:

**Example**

```bash
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.6".
```

#### C.4.1.1  Air-gapped Control Plane Validation

* checking that the first control plane node is in the `Ready` state, and running the target Kubernetes version

  **Example**

  ```bash
  root@controlplane01:~# kubectl get nodes
  NAME              STATUS   ROLES                         AGE    VERSION
  basecm11          Ready    control-plane          6h7m   v1.33.6
  controlplane01    Ready    control-plane,worker   6h5m   v1.33.6
  ...
  ```

* checking that all critical pods are running and in the `Ready` state, especially those in `kube-system` namespace:

  **Example**

  ```bash
  root@controlplane01:~# kubectl get pod -A
  NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE
  calico-system     calico-apiserver-5699d94dbc-hmj76          1/1     Running   0          6h11m
  calico-system     calico-apiserver-5699d94dbc-hmtmw          1/1     Running   0          6h11m
  calico-system     calico-kube-controllers-67c964bc98-5957z   1/1     Running   0          6h11m
  calico-system     calico-node-2jrxv                          1/1     Running   0          6h11m
  ...
  kube-system       coredns-66bc5c9577-lsvdt                   1/1     Running   0          6h13m
  kube-system       coredns-66bc5c9577-w7jhm                   1/1     Running   0          6h13m
  kube-system       kube-apiserver-basecm11                    1/1     Running   0          6h12m
  kube-system       kube-apiserver-controlplane01              1/1     Running   0          6h8m
  kube-system       kube-controller-manager-basecm11           1/1     Running   1          6
  ...
  ```

An extra check is to see if the appropriate kube-apiserver version image, v1.33.6, is being used by the upgraded `controlplane01` node, by executing:

**Example**

```bash
root@controlplane01:~# kubectl describe pod -n kube-system kube-apiserver-controlplane01 | grep Image:
   Image:         master.cm.cluster:5000/kube-apiserver:v1.33.6
```

#### C.4.1.2  Updating Subsequent Control Plane Nodes

The following commands can be repeated on the other control plane nodes.

From the head node:

**Example**

```bash
root@basecm11:~# cmsh -c 'device; imageupdate -w <node> --wait'
```

From the node itself:

**Example**

```bash
root@node001:~# kubeadm upgrade node --kubeconfig /root/.kube/config-default
```

In the preceding, an `upgrade apply` with version does not need to be provided.

The validation steps from before ([here](#c411-air-gapped-control-plane-validation)) should be repeated.

If all control-plane nodes are running without issues, then the worker nodes can also be updated.

#### C.4.1.3  Updating The Worker Nodes In The Air-gapped Update

Updating worker nodes is done in a similar way to updating control plane nodes, but usually with some additional care, since worker nodes can be running user workloads.

Section [4.22.6](/containerization/kubernetes#4226-updating-the-worker-nodes), about updating the worker nodes in a non-air-gapped cluster, has more on being careful during worker node updates. In particular, it explains how to drain nodes beforehand, so that end users do not get upset by lost jobs.

The upgrade step itself is carried out with:

**Example**

```bash
root@workernode001:~# kubeadm upgrade node --kubeconfig /root/.kube/config-default
```

The `--kubeconfig` flag should be provided in the preceding command.

To summarize, a node that needs to be drained requires the following commands:

#### C.4.1.4  Draining On The Head Node

**Example**

```bash
root@basecm11:~# kubectl drain $host --ignore-daemonsets           # check if drain succeeds

# update its software image if needed
root@basecm11:~# cmsh -c "device use $host; imageupdate -w --wait"
```

#### C.4.1.5  Draining On The Non-head Node

**Example**

```bash
root@<node>:~# kubeadm upgrade node --kubeconfig=/root/.kube/config-<Kubernetes instance>
root@<node>:~# systemctl daemon-reload
root@<node>:~# systemctl restart kubelet
root@<node>:~# kubectl uncordon $host                             # check if uncordon succeeds
```

Validation can be done, as before ([here](#c411-air-gapped-control-plane-validation)), by:

* checking that all critical pods are running and in the `Ready` state, especially those in `kube-system` namespace, by running: `kubectl get nodes`

* checking that all critical pods are running and in the Ready state (especially those in `kube-system` namespace, but not only those), by running: `kubectl get pod -A`