Importing Artifacts into the Air-Gapped Environment

After you transfer the bundle into the BCM head node of your air-gapped environment, you must load container images into your private registry and make Helm charts and files available for installation.

If you transferred bundle.tar.gz, extract it into a bundle directory first. The archive holds the bundle contents directly, with no top-level directory, so extracting it without a target directory scatters those contents into the current directory:

mkdir -p ./bundle
tar -xzf ./bundle.tar.gz -C ./bundle

Verify the transferred bundle before importing it:

nmc-airgap verify bundle --bundle ./bundle

Trusting the BCM local registry certificate authority

The BCM local registry at master.cm.cluster:5000 uses a certificate signed by BCM’s own certificate authority. Trust that certificate on the BCM head node before you push charts or images. By default, BCM stores it at:

/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt

If your registry runs elsewhere, use its own certificate. Use one of the following methods.

Add the certificate to the system trust store (recommended)

This trusts the certificate for every tool on the BCM head node, so the push commands need no extra flags:

sudo cp "/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt" \
  /usr/local/share/ca-certificates/bcm-registry.crt
sudo update-ca-certificates

Pass the certificate on each command

Instead of trusting the certificate system-wide, add it to each push command:

• For skopeo, add --cert-dir to skopeo login and --dest-cert-dir to skopeo copy, pointing at the directory that holds ca.crt.

• For helm, add --ca-file with the path to ca.crt to helm push and helm registry login.

Pushing container images to your registry

Images in the bundle use the OCI directory layout. Tools such as skopeo (or crane) can be used to push them to your private registry while preserving the image path so that registry mirroring continues to work.

The manifest selects the container platforms stored in each layout. Keep --all in the following script so every stored platform is copied to one registry tag.

If your private registry requires authentication, authenticate first with its standard tools (for example, skopeo login my-registry.local or crane auth login my-registry.local). Establish CA trust as well, or every push in the loop fails.

Save the following as import-images.sh and run it with Bash from the directory that holds bundle. It reports every image that fails rather than stopping at the first one, and exits nonzero if any push failed:

#!/usr/bin/env bash
set -uo pipefail

TARGET_REGISTRY="my-registry.local"
pushed=0
failed=0

while read -r img_dir; do
  rel_path="${img_dir#bundle/images/}"
  image_with_tag="${rel_path#*/}"
  tag="${image_with_tag##*/}"
  image_path="${image_with_tag%/*}"
  echo "Pushing ${image_path}:${tag}"
  if skopeo copy --all \
    "oci:${img_dir}" \
    "docker://${TARGET_REGISTRY}/${image_path}:${tag}"; then
    pushed=$((pushed + 1))
  else
    echo "FAILED: ${image_path}:${tag}" >&2
    failed=$((failed + 1))
  fi
done < <(find bundle/images -name "index.json" -exec dirname {} \;)

if [ "$((pushed + failed))" -eq 0 ]; then
  echo "No image layouts found under bundle/images" >&2
  exit 1
fi

echo "Imported ${pushed} image(s), ${failed} failed."
[ "${failed}" -eq 0 ] || exit 1

Note

BCM supports a local registry and mirroring of the images from the internet. If you use the BCM registry, set TARGET_REGISTRY to master.cm.cluster:5000 and trust its certificate first (refer to Trusting the BCM local registry certificate authority). Refer to section 3.2 of the BCM containerization manual for more details.

Staging Helm charts

You can either push charts to an OCI-compatible registry or install from local files:

• Push to a Helm OCI registry:

  REGISTRY="oci://my-registry.local/helm-charts"
  
  for chart in bundle/helm/*.tgz; do
    echo "Pushing $chart"
    helm push "$chart" "$REGISTRY"
  done
  

Note

BCM supports a local registry for Helm charts from the internet. If you use the BCM registry, set REGISTRY to oci://master.cm.cluster:5000/helm-charts and trust its certificate first (refer to Trusting the BCM local registry certificate authority). Refer to section 3.2 of the BCM containerization manual for more details.

Uploading AHR Artifacts to the Air-Gapped BCM Head Node

After transferring the AHR bundle to the air-gapped BCM head node, run the upload_ahr_dependencies.sh script to push container images to the local registry, stage Helm charts and packages, and install supporting files.

Before running the upload script, load the Docker module and change into the bundle/ahr-airgap directory where the helper scripts were extracted:

module load docker
cd ./bundle/ahr-airgap

Prerequisites:

• Root access on the BCM head node.

• python3 on PATH (version 3.11 or higher) with the pyyaml package installed.

• skopeo — required unless --skip-images is passed. Used to copy each image’s OCI layout to the registry with --multi-arch all, preserving the multi-architecture manifest so each node pulls the variant matching its architecture.

• docker — required unless --skip-images is passed (load with module load docker). Used to authenticate to and query the local registry.

The script reads registry and authentication settings from airgap/upload_config.yaml:

registry_url: "master.cm.cluster:5000"
registry_ca_cert: "/path/to/registry/ca.crt"
registry_username: admin                   # optional
registry_password: ""                      # optional
skip_images: false
skip_charts: false
skip_packages: false
skip_files: false

Quick start:

cd ./bundle/ahr-airgap
sudo ./upload_ahr_dependencies.sh ./bundle

Usage:

cd ./bundle/ahr-airgap
sudo ./upload_ahr_dependencies.sh <bundle_dir> [options]

Argument	Description
<bundle_dir>	Path to the downloaded bundle directory (required).
--config <path>	Path to a custom configuration YAML (default: airgap/upload_config.yaml).
--skip-images	Skip pushing Docker images to the registry.
--skip-charts	Skip pushing Helm charts.
--skip-packages	Skip copying .deb packages.
--skip-files	Skip installing files (OpenTofu binary, runbooks).

Examples, all run from bundle/ahr-airgap:

# Upload everything
sudo ./upload_ahr_dependencies.sh ./bundle

# Upload only packages and files (skip images and charts)
sudo ./upload_ahr_dependencies.sh ./bundle --skip-images --skip-charts

The upload process runs five steps in order:

1. Images — Copies each OCI image layout from bundle/images/ to the local registry with skopeo (preserving the multi-architecture manifest list for bundles downloaded with download_ahr_dependencies.sh --arch=all) under the same image path so registry mirroring keeps working. Images that already exist remotely are skipped.

2. Charts — Copies each .tgz from bundle/charts/ to /cm/local/apps/autonomous-hardware-recovery/var/charts/.

3. Packages — Copies .deb files from bundle/packages/ to /cm/local/apps/autonomous-hardware-recovery/var/packages/.

   Note

   The upload script copies a package only if it is not already installed on the BCM head node or in any software image. This selective approach avoids overwriting existing packages. Because the download script captures the entire dependency tree, the bundle contains all transitive dependencies and should satisfy every package requirement during installation.

4. Files — Extracts the OpenTofu binary from bundle/terraform/bin/tofu_*.zip into /cm/local/apps/autonomous-hardware-recovery/bin/tofu and copies runbook archives to /cm/local/apps/autonomous-hardware-recovery/var/runbooks/downloaded/.

5. Plugin install — Installs the cm-setup-ahr-*.deb package from the packages directory using dpkg -i, deploying the AHR cm-setup plugin into the BCM site-packages.

A summary is printed when the upload completes. If any step fails, the script provides a retry command targeting only the failed categories. Logs are written to /var/log/nmc-autonomous-hardware-recovery-airgap-upload.log.

Expected plugin folder structure

After upload, the AHR plugin directory has the following layout:

/cm/local/apps/autonomous-hardware-recovery/
├── bin/
│   └── tofu
├── etc/
│   ├── autonomous-hardware-recovery.key
│   └── autonomous-hardware-recovery.pem
└── var/
    ├── charts/
    │   └── shoreline-onprem-backend-29.2.*.tgz
    ├── packages/
    │   ├── shoreline_29.2.*-enroot.deb
    │   ├── datacenter-gpu-manager-4-*.deb
    │   └── ...
    └── runbooks/
        └── downloaded/
            └── nmc-ahr-2.5.*.tgz

Setup PID dependencies

The following steps install PID dependencies on the BCM head node and into the compute/GPU node software image. Run these steps once — the installation targets both the BCM head node and the software image in a single pass.

Choose the architecture per artifact, not per cluster. nvdebug, nvssvt, and nvlmapper are installed on the BCM head node and follow the BCM head node’s architecture. partnerdiag and EUD are staged for the compute and GPU nodes and follow their architecture: x86_64 on B200, B300, and R; Arm (Grace) on GB200 and GB300; and Arm (Vera) on VR. The two architecture choices are independent, and mixed combinations are supported. Refer to downloading for mixed-architecture clusters.

The commands in this section refer to the artifact versions through shell variables. Run this block first, in the same shell you use for the remaining steps, and the commands that follow can be copied verbatim. When you download newer PID packages, update the versions here only — they are not repeated anywhere else in this section. The versions below match the Download PID dependencies table.

# PID artifact versions
export NVDEBUG_VERSION=2.1.0
export NVSSVT_VERSION=1.7.1
export NVLMAPPER_VERSION=14
export PARTNERDIAG_SWITCH_VERSION=42174
export PARTNERDIAG_COMPUTE_VERSION=60008-rev17
export EUD_VERSION=580.173.6

# Driver branch the EUD package targets, derived from EUD_VERSION (for example, 580).
export EUD_BRANCH=${EUD_VERSION%%.*}

# Ubuntu release the EUD .deb was built for (the "ubuntu2404" segment of its
# filename) — check the extracted EUD_${EUD_VERSION}/ directory and set this to
# match your GPU node image's OS release. B200/B300/GB200/GB300 currently ship
# ubuntu2404; confirm the release for R and VR against the extracted filenames,
# since they are not guaranteed to match.
export EUD_UBUNTU_RELEASE=ubuntu2404

1. nvdebug — install to /opt/nvidia/nvdebug/bin/:

   For an x86_64 BCM head node:

   unzip -o NVDebug_v${NVDEBUG_VERSION}.zip -d nvdebug_v${NVDEBUG_VERSION}_extract
   tar xzf nvdebug_v${NVDEBUG_VERSION}_extract/Release_${NVDEBUG_VERSION}/nvdebug-linux-amd64-${NVDEBUG_VERSION}.tar.gz -C nvdebug_v${NVDEBUG_VERSION}_extract/
   mkdir -p /opt/nvidia/nvdebug/bin
   cp -rf nvdebug_v${NVDEBUG_VERSION}_extract/nvdebug/* /opt/nvidia/nvdebug/bin/
   chmod +x /opt/nvidia/nvdebug/bin/nvdebug
   

   For an Arm (aarch64) BCM head node:

   unzip -o NVDebug_v${NVDEBUG_VERSION}.zip -d nvdebug_v${NVDEBUG_VERSION}_extract
   tar xzf nvdebug_v${NVDEBUG_VERSION}_extract/Release_${NVDEBUG_VERSION}/nvdebug-linux-arm64-${NVDEBUG_VERSION}.tar.gz -C nvdebug_v${NVDEBUG_VERSION}_extract/
   mkdir -p /opt/nvidia/nvdebug/bin
   cp -rf nvdebug_v${NVDEBUG_VERSION}_extract/nvdebug/* /opt/nvidia/nvdebug/bin/
   chmod +x /opt/nvidia/nvdebug/bin/nvdebug
   

2. nvssvt — install to /cm/shared/apps/nvssvt/:

   unzip -o nvssvt-v${NVSSVT_VERSION}.zip -d nvssvt_extract
   mkdir -p nvssvt_v${NVSSVT_VERSION}
   

   For an x86_64 BCM head node:

   tar xzf nvssvt_extract/v${NVSSVT_VERSION}/nvssvt-release-${NVSSVT_VERSION}-amd64.tar.gz -C nvssvt_v${NVSSVT_VERSION}/
   mkdir -p /cm/shared/apps/nvssvt/${NVSSVT_VERSION}
   cp -rf nvssvt_v${NVSSVT_VERSION}/* /cm/shared/apps/nvssvt/${NVSSVT_VERSION}/
   chmod +x /cm/shared/apps/nvssvt/${NVSSVT_VERSION}/nvssvt
   

   For an Arm (aarch64) BCM head node:

   tar xzf nvssvt_extract/v${NVSSVT_VERSION}/nvssvt-release-${NVSSVT_VERSION}-arm64.tar.gz -C nvssvt_v${NVSSVT_VERSION}/
   mkdir -p /cm/shared/apps/nvssvt/${NVSSVT_VERSION}
   cp -rf nvssvt_v${NVSSVT_VERSION}/* /cm/shared/apps/nvssvt/${NVSSVT_VERSION}/
   chmod +x /cm/shared/apps/nvssvt/${NVSSVT_VERSION}/nvssvt
   

3. nvlmapper — install to /cm/shared/nvlmapper/ and /tmp/nvlmapper/:

   unzip -o nvlmapper_archive_v${NVLMAPPER_VERSION}.zip -d nvlmapper_extract
   

   For an x86_64 BCM head node:

   tar xzf nvlmapper_extract/release_${NVLMAPPER_VERSION}.0_amd64.tgz -C nvlmapper_extract/
   mkdir -p /cm/shared/nvlmapper/${NVLMAPPER_VERSION}
   cp -rf nvlmapper_extract/dist_amd64/* /cm/shared/nvlmapper/${NVLMAPPER_VERSION}/
   chmod +x /cm/shared/nvlmapper/${NVLMAPPER_VERSION}/nvlmapper
   mkdir -p /tmp/nvlmapper/${NVLMAPPER_VERSION}
   cp -rf nvlmapper_extract/dist_amd64/* /tmp/nvlmapper/${NVLMAPPER_VERSION}/
   chmod +x /tmp/nvlmapper/${NVLMAPPER_VERSION}/nvlmapper
   

   For an Arm (aarch64) BCM head node:

   tar xzf nvlmapper_extract/release_${NVLMAPPER_VERSION}.0_arm64.tgz -C nvlmapper_extract/
   mkdir -p /cm/shared/nvlmapper/${NVLMAPPER_VERSION}
   cp -rf nvlmapper_extract/dist_arm64/* /cm/shared/nvlmapper/${NVLMAPPER_VERSION}/
   chmod +x /cm/shared/nvlmapper/${NVLMAPPER_VERSION}/nvlmapper
   mkdir -p /tmp/nvlmapper/${NVLMAPPER_VERSION}
   cp -rf nvlmapper_extract/dist_arm64/* /tmp/nvlmapper/${NVLMAPPER_VERSION}/
   chmod +x /tmp/nvlmapper/${NVLMAPPER_VERSION}/nvlmapper
   

4. partnerdiag mfg switch L10 — install to /cm/shared/partnerdiag_switch/ and /tmp/partnerdiag/:

   tar xzf 629-9K36F-00MV-FLD-${PARTNERDIAG_SWITCH_VERSION}.tgz
   
   mkdir -p /cm/shared/partnerdiag_switch/${PARTNERDIAG_SWITCH_VERSION}
   cp -rf 629-9K36F-00MV-FLD-${PARTNERDIAG_SWITCH_VERSION}/* /cm/shared/partnerdiag_switch/${PARTNERDIAG_SWITCH_VERSION}/
   chmod +x /cm/shared/partnerdiag_switch/${PARTNERDIAG_SWITCH_VERSION}/partnerdiag
   
   mkdir -p /tmp/partnerdiag/${PARTNERDIAG_SWITCH_VERSION}
   cp -rf 629-9K36F-00MV-FLD-${PARTNERDIAG_SWITCH_VERSION}/* /tmp/partnerdiag/${PARTNERDIAG_SWITCH_VERSION}/
   chmod +x /tmp/partnerdiag/${PARTNERDIAG_SWITCH_VERSION}/partnerdiag
   

5. partnerdiag mfg computer L10 — install to /cm/shared/partnerdiag/:

   tar xzf 629-24975-0000-FLD-${PARTNERDIAG_COMPUTE_VERSION}.tgz
   
   mkdir -p /cm/shared/partnerdiag/${PARTNERDIAG_COMPUTE_VERSION}
   cp -rf 629-24975-0000-FLD-${PARTNERDIAG_COMPUTE_VERSION}/* /cm/shared/partnerdiag/${PARTNERDIAG_COMPUTE_VERSION}/
   chmod +x /cm/shared/partnerdiag/${PARTNERDIAG_COMPUTE_VERSION}/partnerdiag
   

6. EUD — extract to /cm/shared/eud/, copy the .deb into the software image, and install:

   Note

   Replace <gpu-node-image> in the paths below with the actual BCM software image name for your GPU nodes (for example, dgx-gb200-slurm). Do not use the literal string default-image.

   Note

   The EUD package is not currently published for R and VR GPU nodes. You can use the commands in this step for those platforms when the package becomes available.

   mkdir -p /cm/shared/eud
   tar xzf EUD_${EUD_VERSION}.tar.gz -C /cm/shared/eud/
   
   mkdir -p /cm/images/<gpu-node-image>/pid/eud
   

   For x86_64 GPU nodes (B200, B300, R):

   cp -f /cm/shared/eud/EUD_${EUD_VERSION}/nvidia-diagnostic-local-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1_1.0-1_amd64.deb \
     /cm/images/<gpu-node-image>/pid/eud/
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> --chdir=/pid/eud \
     bash -c "dpkg -i /pid/eud/nvidia-diagnostic-local-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1_1.0-1_amd64.deb"
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> --chdir=/root \
     bash -c "dpkg -i /var/nvidia-diagnostic-local-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1/nvidia-diagnostic-${EUD_BRANCH}_${EUD_VERSION}-1_amd64.deb"
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> \
     bash -c "cp /var/nvidia-diagnostic-local-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1/nvidia-diagnostic-local-*-keyring.gpg /usr/share/keyrings/"
   

   For Arm GPU nodes (GB200 and GB300 on Grace; VR on Vera):

   cp -f /cm/shared/eud/EUD_${EUD_VERSION}/nvidia-diagnostic-local-tegra-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1_1.0-1_arm64.deb \
     /cm/images/<gpu-node-image>/pid/eud/
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> --chdir=/pid/eud \
     bash -c "dpkg -i /pid/eud/nvidia-diagnostic-local-tegra-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1_1.0-1_arm64.deb"
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> --chdir=/root \
     bash -c "dpkg -i /var/nvidia-diagnostic-local-tegra-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1/nvidia-diagnostic-${EUD_BRANCH}_${EUD_VERSION}-1_arm64.deb"
   
   systemd-nspawn --directory=/cm/images/<gpu-node-image> \
     bash -c "cp /var/nvidia-diagnostic-local-tegra-repo-${EUD_UBUNTU_RELEASE}-${EUD_VERSION}-mode1/nvidia-diagnostic-local-*-keyring.gpg /usr/share/keyrings/"
   

   After installing the packages, update the software image:

   cmsh
   [headnode]% device
   [headnode->device]% imageupdate -w -c <gpu-node-image>