AHR TUI plugin in air-gapped mode#

After uploading all AHR dependencies, as described in Uploading AHR Artifacts to the Air-Gapped BCM Head Node, run the AHR TUI installer. The plugin automatically detects when it is running in an air-gapped environment and adjusts its behavior.

Note

Prerequisites. AHR shares the prerequisites documented in Autonomous Hardware Recovery Installation Guide. One of them affects the air-gap sequence: if Run:ai schedules any of the GPU nodes in your cluster, install Run:ai before AHR. Installing Run:ai first makes the Run:ai BCM node categories available when you select categories for the AHR agent. The k8s-user kubectl context is also available when you deploy the runbooks, so the air-gap installation completes in a single pass. Refer to Run:ai (Kubernetes) GPU node scheduling for details, including what to do if AHR is installed first.

Before running the installer, create a BCM user to deploy the AHR application. Replace <strong-password> with a secure value:

cmsh -c 'user; add mission-control-admin; set password <strong-password>; commit;'

Important

The AHR TUI plugin reads the imageRepository field from the kubeadm-config ConfigMap (kube-system namespace) to detect air-gapped mode. This field must be set to your local registry URL. If it still points to the default registry.k8s.io, the plugin will not recognize the environment as air-gapped. Registry mirroring alone is not sufficient — imageRepository must use the local registry for air-gap detection and image pulls to work correctly.

To verify that imageRepository is correctly set, run:

kubectl get configmap kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep imageRepository

The output should show your local registry URL (for example, imageRepository: master.cm.cluster:5000). If it still shows registry.k8s.io, refer to section 4.2 of the BCM containerization manual for instructions on configuring the Kubernetes cluster for air-gapped operation.

Kgateway container images

The AHR backend serves its endpoints through the shoreline gateway proxy, which is built on the third-party kgateway Helm chart. That chart pulls three container images from cr.kgateway.dev/kgateway-dev/:

  • kgateway — the xDS control plane.

  • envoy-wrapper — the Envoy-based data-plane proxy.

  • sds — the secret discovery service that provides TLS material to the proxy.

These images are not specific to AHR, so they are not part of the bundle downloaded by download_ahr_dependencies.sh. The BCM air-gap scripts normally mirror them into the local registry as part of the Kubernetes air-gapped installation. The following manual steps are only needed if the images are missing from the registry.

Important

When the kgateway images are missing, the kgateway controller and the shoreline gateway proxy pods stay in ImagePullBackOff, and the AHR WaitForBackendDiscoveryEndpoint stage times out later in the installation.

  1. Check whether the images are already present in the local registry:

    for image in kgateway envoy-wrapper sds; do
  curl -sS \
    --cacert /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt \
    https://master.cm.cluster:5000/v2/kgateway-dev/${image}/tags/list
done

    If every image returns a tag list that contains the version deployed in your cluster, the images are already available and you can skip the remaining steps.

  2. Mirror the images from a host that has egress to cr.kgateway.dev. Preserve the kgateway-dev/ image path so that registry mirroring continues to work:

    VERSION=v2.3.0
REGISTRY=master.cm.cluster:5000
CERT_DIR=/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000

for image in kgateway envoy-wrapper sds; do
  skopeo copy --format v2s2 --multi-arch all \
    --dest-cert-dir "${CERT_DIR}" \
    "docker://cr.kgateway.dev/kgateway-dev/${image}:${VERSION}" \
    "docker://${REGISTRY}/kgateway-dev/${image}:${VERSION}"
done

  3. Repeat the check in step 1 to confirm that each image now resolves in the local registry.

Running the AHR installation wizard#

After completing the preceding steps, follow Autonomous Hardware Recovery Installation Guide with the following modifications:

  1. The cm-mission-control-setup wizard must be invoked with the --skip-packages option (required in air-gapped environments):

    cm-mission-control-setup --skip-packages

  2. The NGC credential prompt is not displayed. The plugin automatically detects the air-gapped environment and pulls artifacts from the local registry instead. This requires registry mirroring to be configured on every node beforehand, as described in Configuring Registry Mirroring. Without it, AHR image pulls fail and the installation stalls.

Resolving dependency conflicts during installation

During the InstallAgentOnHeadNode or InstallAgentInSoftwareImages stages, the AHR TUI plugin installs packages from the air-gap bundle. If a dependency conflict occurs (for example, a version mismatch between a bundled package and a package already installed on the system), the installer pauses and waits for you to resolve the conflict manually.

Tip

The air-gap bundle contains the entire dependency tree for every required package. If you need to install or downgrade a conflicting dependency, look for the appropriate .deb file in /cm/local/apps/autonomous-hardware-recovery/var/packages/. You can install it manually with dpkg -i <package>.deb and then retry the stage.

For a mixed-architecture bundle (downloaded with download_ahr_dependencies.sh --arch=all), this directory holds the .deb files for both architectures side by side, distinguished by the _<arch>.deb filename suffix (for example, _amd64.deb versus _arm64.deb). Pick the file matching the architecture of the node or software image you are repairing (_all.deb packages apply to any architecture).

Self-signed TLS certificates

The AHR TUI plugin can generate self-signed TLS certificates for use in air-gapped environments where publicly trusted certificates are not available.

During installation, the TUI prompts whether to use self-signed certificates or to provide publicly trusted certificates:

  • If you select self-signed certificates, you are prompted for a base domain URL (default: autonomous-hardware-recovery.local).

  • The certificate and key files are written to /cm/local/apps/autonomous-hardware-recovery/etc/certs/.

  • AHR agents are automatically configured to trust the self-signed certificate when communicating with the backend.

Note

Note down the app URL and api URL displayed by the TUI installer during installation. You will need them to configure /etc/hosts on your local machine (step 3).

To access the AHR web UI from a local machine when using self-signed certificates:

  1. Retrieve the BCM head node IP address.

  2. Retrieve the certificate:

    cat /cm/local/apps/autonomous-hardware-recovery/etc/certs/cert.crt

  1. Add entries to /etc/hosts on your local machine pointing to the BCM head node IP. For example:

    10.0.0.1 mycluster.autonomous-hardware-recovery.local
10.0.0.1 mycluster-api.autonomous-hardware-recovery.local

  2. Save the certificate to a local file and configure your browser to trust it.

  3. Access the AHR web UI at https://<ahr_base_domain>/home/. For example:

    https://autonomous-hardware-recovery.local/home/

Uninstalling Autonomous Hardware Recovery#

To uninstall AHR, start the plugin via cm-mission-control-setup and select the Uninstall flow.

Important

The uninstall flow removes the var folder, which stores Helm charts, packages, and runbooks required by the air-gapped installation. Before reinstalling AHR, you must run the upload script again to restore these artifacts. Refer to Uploading AHR Artifacts to the Air-Gapped BCM Head Node for details.

Tip

Use the --skip-images flag to skip re-pushing images that are already present in the local registry.