Downloading NVIDIA Mission Control Artifacts with nmc-airgap and Helper Scripts

The steps on this page use the nmc-airgap CLI and the manifest. To obtain them, refer to the air-gap prerequisites.

nmc-airgap Tool

The nmc-airgap CLI ships as a container image and as a binary tarball. Both support x86 and arm64. The tarball expands into an nmc-airgap directory that holds one binary per platform, named nmc-airgap-darwin-{amd64|arm64} on macOS and nmc-airgap-linux-{amd64|arm64} on Linux. For example, on a macOS arm64 machine:

tar -xzf nmc-airgap.tar.gz
./nmc-airgap/nmc-airgap-darwin-arm64 --help

The rest of this guide refers to the tool as nmc-airgap. Add the binary for your platform to your PATH under that name so the commands work as written:

sudo install -m 0755 ./nmc-airgap/nmc-airgap-linux-amd64 /usr/local/bin/nmc-airgap
nmc-airgap --help

nmc-airgap CLI commands

• verify manifest — Verify manifest syntax, structure, and artifact resolution.

• list — List artifacts that would be downloaded without downloading.

• fetch — Download artifacts into a local bundle directory.

• verify bundle — Verify bundle integrity and requested downloads after transfer to the air-gapped environment.

• archive — Create a portable archive from a completed bundle.

Fetching the bundle

1. Set the environment variable for your NVIDIA NGC API key:

Variable	Description
NGC_NVIDIA_NV_MISSION_CONTROL_API_KEY	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).

export NGC_NVIDIA_NV_MISSION_CONTROL_API_KEY="your-ngc-api-key"

2. Extract the manifest resource for the DGX system you are installing. Run this from the directory that holds the downloaded archive:

   tar -xzf nmc-airgap-manifest-<dgx-system>.tar.gz --strip-components=1
   

   This yields blueprints/<dgx-system>/airgap-manifest.yaml. The following examples use <dgx-system> for that system.

3. Verify the manifest before downloading:

   nmc-airgap verify manifest --manifest blueprints/<dgx-system>/airgap-manifest.yaml
   

   This checks that every artifact the manifest names can be resolved.

4. Optional: Preview the resolved artifacts without downloading:

   nmc-airgap list --manifest blueprints/<dgx-system>/airgap-manifest.yaml
   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --dry-run
   

5. Fetch the bundle once for the DGX system:

   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle
   

   For a single-architecture cluster, add --arch amd64 or --arch arm64 to the fetch, list, and verify manifest commands. By default, the bundle includes all architectures listed in the manifest.

   The runai component includes runai-airgapped-package-<version>.tar.gz and the cm-setup-runai-bcm11 Debian package. The tarball contains the Run:ai installation charts and images; the Debian package provides cm-runai-setup. Retain the tarball when staging the artifacts for the air-gapped cluster. The post-installation placement procedure reuses the charts extracted from this tarball.

   The topograph component includes the Topograph Helm chart and its container images. Topograph discovers the physical network topology of the cluster and exposes it to schedulers for topology-aware scheduling.

   For NetQ, only the Debian packages download. The NetQ tarball must be downloaded separately. Refer to Install and Configure NetQ for instructions.

6. Optional: Instead of downloading every enabled component, verify and fetch a fixed component subset. Use the same component selection for both commands and for later resume operations:

   nmc-airgap verify manifest --manifest blueprints/<dgx-system>/airgap-manifest.yaml --components <component>,<component>
   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle --components <component>,<component>
   

7. Optional: Resume an interrupted download by skipping artifacts that are already present and verified:

   # Resume all enabled components with all manifest architectures.
   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle --skip-existing
   
   # Resume the fixed component subset from step 6.
   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle --components <component>,<component> --skip-existing
   
   # Resume a request originally limited to AMD64.
   nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle --arch amd64 --skip-existing
   

   One bundle directory represents one resolved request. Resume with the same manifest contents, environment values, component selection, and architecture selection. If the original request used both --components and --arch, include both when resuming.

   If the command reports a different resolved request, use the reported differences to restore the original manifest and selections before resuming. To rebuild with a changed request, use a new output directory or remove the existing bundle directory before fetching again.

8. Optional: Create a portable archive for transfer. Run this last, after you complete the helper script downloads in Helper scripts. Only content under bundle is packaged, and archive verifies checksums before it writes the tarball. Run it from the directory that holds bundle, because the helper steps leave you inside bundle/ahr-airgap:

   nmc-airgap archive --bundle ./bundle --output ./bundle.tar.gz
   

Common fetch flags:

• -m, --manifest — Path to manifest file (default manifest.yaml).

• -o, --output — Output directory (default ./bundle).

• -c, --components — Comma-separated list of components to download (default: all enabled).

• --arch — Target CPU architectures for files and container images (amd64, arm64, or amd64,arm64).

• -p, --parallel — Number of parallel downloads (default 10).

• --download-timeout — Per-artifact timeout (default 30m for Helm charts and files, 4h for containers).

DCGM Exporter Debian Package

Select the Ubuntu release of the DGX software image:

Ubuntu 24.04

The datacenter-gpu-manager-exporter package is already installed. No separate package download is required; enable the existing service when you configure DCGM metric scraping.

Ubuntu 26.04

On an internet-connected host, run the command for the target DGX node architecture, regardless of the download host architecture. Replace the example package version in the URL if your deployment requires a different version.

For x86_64 DGX nodes:

curl -fLO "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/datacenter-gpu-manager-exporter_4.8.4.3955-1_amd64.deb"

For aarch64 DGX nodes:

curl -fLO "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/sbsa/datacenter-gpu-manager-exporter_4.8.4.3955-1_arm64.deb"

Transfer the downloaded package alongside the bundle to the air-gapped environment. Keep this separately downloaded file outside bundle: nmc-airgap verify bundle checks only artifacts recorded in the bundle manifest.

On the BCM head node, copy the transferred package to /cm/images/<image name>/tmp/dcgm-exporter.deb. Then install it inside that image before configuring DCGM metric scraping:

cm-chroot /cm/images/<image name>
dpkg -i /tmp/dcgm-exporter.deb
exit

Verifying the bundle after transfer

After you copy the bundle to the air-gapped environment (for example, via physical media), verify integrity. If you transferred bundle.tar.gz, extract it into a bundle directory first, as described in Importing Artifacts into the Air-Gapped Environment:

nmc-airgap verify bundle --bundle ./bundle

The File architectures and Container platforms lines show the targets saved in the bundle, so you can check them against your cluster after transfer. The Container platform coverage section, when present, lists requested platforms for which individual images have not been published.

The tool recomputes SHA256 checksums and compares them to the values in bundle/manifest.json.

Helper scripts

The helper scripts will download the appropriate artifacts for the NVIDIA Mission Control components and place them in their respective tarballs. Extract those tarballs inside the bundle directory that nmc-airgap fetch created. Content outside bundle is not packaged when you archive it for transfer.

Autonomous Hardware Recovery (AHR)

The AHR airgap helper scripts automate the download and upload of all artifacts required by the AHR TUI plugin. They ship as a component in the manifest, so a full fetch already placed them in the bundle. To fetch them on their own:

nmc-airgap fetch --manifest blueprints/<dgx-system>/airgap-manifest.yaml --output ./bundle --components autonomousHardwareRecovery

The helper scripts are downloaded as a compressed archive at bundle/files/ahr-airgap.tar.gz. The archive has no top-level directory. It expands into download_ahr_dependencies.sh, upload_ahr_dependencies.sh, and the airgap directory. Extract it into an ahr-airgap directory inside the bundle so that the scripts and the downloaded artifacts stay together. Run the following commands from the directory that contains bundle:

mkdir -p ./bundle/ahr-airgap
tar -xzf ./bundle/files/ahr-airgap.tar.gz -C ./bundle/ahr-airgap

The archive contains the following items:

Item	Description
download_ahr_dependencies.sh	Runs on an internet-connected host. Checks that python3 and the pyyaml package are available, then runs airgap/download.py to collect the AHR artifacts into a bundle directory.
upload_ahr_dependencies.sh	Runs on the BCM head node of the air-gapped environment. Checks that python3 is available, then runs airgap/upload.py to push the bundled artifacts to the local registry and install them on the BCM head node.
airgap	Python implementation behind both scripts. Includes the manifests in airgap/manifests that pin the version and source of every image, chart, and package that AHR requires.

Both scripts locate the airgap directory relative to their own location, so keep the three items together if you move them. Change into bundle/ahr-airgap before you run either script:

cd ./bundle/ahr-airgap

Download dependencies script

The download_ahr_dependencies.sh script can be run on a Debian-based machine with internet access to collect all AHR air-gap dependencies into a self-contained bundle directory. After that, the bundle is transferred to the BCM head node of the air-gapped environment where, with the help of the upload_ahr_dependencies.sh script, the dependencies from the bundle are set up.

Important

For a single-architecture download (the --arch=amd64 or --arch=arm64 case), run the download script on a machine with the same CPU architecture as the target nodes (for example, run on an amd64 machine when targeting amd64 nodes). The installation will then be successful on all nodes sharing that architecture.

For a mixed-architecture cluster (a cluster with both amd64 and arm64 nodes), use --arch=all to produce a single bundle that installs on both architectures from one download host. The download host must have APT sources configured for both architectures — refer to Downloading for mixed-architecture clusters.

Important

It is recommended to run the download script on a machine with the same OS version as the BCM head node and the software images used for compute/GPU nodes. Because the download script resolves the entire APT dependency tree for each required package, OS version mismatches can cause dependency conflicts during installation.

Note

The download script fetches approximately 130 GB of data (PID files alone account for roughly 9 GB). Ensure that at least 130 GB of free disk space is available in the output directory before starting the download.

Both download_ahr_dependencies.sh and upload_ahr_dependencies.sh may take a significant amount of time to complete — expect over one hour each depending on network and disk speed.

Prerequisites:

• python3 on PATH (version 3.11 or higher) with the pyyaml package installed.

• skopeo — required unless --skip-images is passed.

• helm — required unless --skip-charts is passed.

• envsubst — used to expand environment variables in manifest files.

Set the following environment variables before running the script:

Variable	Description
NGC_API_KEY	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).

The script exits with an error if a required variable is missing for the artifacts being downloaded.

Quick start:

cd ./bundle/ahr-airgap
export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh ./bundle

Usage:

cd ./bundle/ahr-airgap
export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh <output_dir> [options]

Argument	Description
<output_dir>	Directory where the bundle is written (required). Created if it does not exist.
--arch <amd64|arm64|all>	Target architecture for images and packages. all downloads both architectures into one bundle for mixed clusters. The default is auto-detected from the download host: all when the host has working APT sources for both architectures, otherwise the host’s native architecture. Refer to Downloading for mixed-architecture clusters.
--set NAME=VER	Override an artifact version. Can be specified multiple times.
--retries <n>	Number of retry attempts for transient failures (default: 3).
--skip-images	Skip downloading Docker images.
--skip-charts	Skip downloading Helm charts.
--skip-packages	Skip downloading packages (NGC, URL, and APT).
--skip-files	Skip downloading files (OpenTofu, runbooks).
--skip-runbooks	Skip downloading runbook artifacts specifically.

Examples, all run from bundle/ahr-airgap:

# Full bundle for the download host's native architecture
./download_ahr_dependencies.sh ./bundle

# Full bundle for a mixed amd64 + arm64 cluster (one bundle, both arches)
./download_ahr_dependencies.sh ./bundle --arch=all

# Only packages and files (skip images and charts)
./download_ahr_dependencies.sh ./bundle --skip-images --skip-charts

Downloading for mixed-architecture clusters (--arch=all)

Use --arch=all when the target cluster mixes architectures — for example an amd64 BCM head node with arm64 (Grace) compute nodes, or vice versa. A single --arch=all bundle contains:

• Container images as multi-architecture OCI image indexes, so each node pulls the variant matching its own architecture.

• Per-architecture .deb packages that coexist in bundle/packages/ because Debian’s <name>_<version>_<arch>.deb naming convention prevents collisions (for example, libc6_2.39-0ubuntu8_amd64.deb and libc6_2.39-0ubuntu8_arm64.deb).

The script auto-detects a sensible default for --arch based on the download host: it uses all only when the host can already resolve APT packages for both architectures, and otherwise falls back to the host’s native architecture. Passing --arch=all explicitly exits with an error if the cross-architecture APT sources described in the following section are not configured.

Configure cross-architecture APT sources on the download host

--arch=all resolves the full APT dependency tree once per architecture, so the download host must be able to fetch base Ubuntu packages for the foreign architecture as well as its native one. On Ubuntu, the two architecture families are served by different mirrors:

• amd64 (and i386) — http://archive.ubuntu.com/ubuntu and http://security.ubuntu.com/ubuntu.

• arm64 (and other ports) — http://ports.ubuntu.com/ubuntu-ports.

A stock amd64 host only has the archive.ubuntu.com mirror, which does not carry arm64 packages, so APT must be told where to find the foreign architecture before the download. The NVIDIA CUDA and DOCA/Mellanox repositories are configured automatically by the script for each architecture and need no manual setup — only the base Ubuntu mirrors require the steps that follow.

Run one of the following blocks, matching the download host’s own architecture. Each block enables the foreign architecture, restricts the host’s existing Ubuntu sources to its native architecture (so apt-get update does not return 404 errors fetching the foreign architecture from the wrong mirror), adds the foreign architecture’s mirror, and refreshes the APT cache. The blocks auto-detect the Ubuntu release codename, are idempotent, and require sudo.

On an amd64 download host (adds the arm64 repositories):

set -euo pipefail
. /etc/os-release
CODENAME="${VERSION_CODENAME:?cannot detect Ubuntu codename}"

# 1. Enable arm64 as a foreign architecture.
sudo dpkg --add-architecture arm64

# 2. Restrict the existing (amd64) Ubuntu sources to amd64 so apt does not
#    try to fetch arm64 from archive.ubuntu.com (which does not carry it).
if [ -f /etc/apt/sources.list.d/ubuntu.sources ] && \
   ! grep -q '^Architectures:' /etc/apt/sources.list.d/ubuntu.sources; then
    sudo sed -i '/^Types:/a Architectures: amd64' \
        /etc/apt/sources.list.d/ubuntu.sources
fi

# 3. Add the arm64 "ports" repositories.
sudo tee /etc/apt/sources.list.d/ubuntu-arm64.sources >/dev/null <<EOF
Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports
Suites: ${CODENAME} ${CODENAME}-updates ${CODENAME}-backports ${CODENAME}-security
Components: main restricted universe multiverse
Architectures: arm64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF

# 4. Refresh the APT cache.
sudo apt-get update

On an arm64 download host (adds the amd64 repositories):

set -euo pipefail
. /etc/os-release
CODENAME="${VERSION_CODENAME:?cannot detect Ubuntu codename}"

# 1. Enable amd64 as a foreign architecture.
sudo dpkg --add-architecture amd64

# 2. Restrict the existing (arm64) Ubuntu sources to arm64 so apt does not
#    try to fetch amd64 from ports.ubuntu.com (which does not carry it).
if [ -f /etc/apt/sources.list.d/ubuntu.sources ] && \
   ! grep -q '^Architectures:' /etc/apt/sources.list.d/ubuntu.sources; then
    sudo sed -i '/^Types:/a Architectures: arm64' \
        /etc/apt/sources.list.d/ubuntu.sources
fi

# 3. Add the amd64 archive + security repositories.
sudo tee /etc/apt/sources.list.d/ubuntu-amd64.sources >/dev/null <<EOF
Types: deb
URIs: http://archive.ubuntu.com/ubuntu
Suites: ${CODENAME} ${CODENAME}-updates ${CODENAME}-backports
Components: main restricted universe multiverse
Architectures: amd64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: ${CODENAME}-security
Components: main restricted universe multiverse
Architectures: amd64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF

# 4. Refresh the APT cache.
sudo apt-get update

Tip

The preceding blocks target the deb822 source format used by Ubuntu 24.04 (Noble). If the host still uses the classic single-line /etc/apt/sources.list format, add the [arch=...] qualifier to each deb line instead — for example deb [arch=amd64] http://archive.ubuntu.com/ubuntu noble main ... and deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports noble main .... Without an explicit arch= qualifier, APT tries to fetch the foreign architecture from the native mirror and apt-get update fails with 404 errors.

After running the block for your host, verify that both architectures resolve. dpkg must list the foreign architecture and apt-cache madison must return rows for a known package in each architecture:

dpkg --print-foreign-architectures      # expect: arm64 (or amd64)
apt-cache madison curl:amd64             # expect: at least one row
apt-cache madison curl:arm64             # expect: at least one row

These are the same probes the download script uses to decide whether --arch=all is available; once all three succeed, --arch=all (and the auto-detected default) will resolve cleanly.

The download process runs four steps in order:

1. Images — Pulls container images from NGC and stores each as an OCI image layout (multi-architecture when --arch=all) under bundle/images/.

2. Charts — Pulls Helm charts from NGC and saves them as .tgz archives in bundle/charts/.

3. Packages — Downloads .deb packages from NGC, direct URLs, and APT repositories into bundle/packages/. APT packages include full recursive dependency resolution, meaning the entire dependency tree for every required package is downloaded. This ensures that the bundle is self-contained and all transitive dependencies are available for offline installation.

4. Files — Downloads miscellaneous files (OpenTofu binary, runbook archives) into subdirectories under bundle/terraform/.

Artifact lists are defined in YAML manifest files under airgap/manifests/:

• sources.yaml — Registry URLs and credentials.

• images.yaml — Container images.

• charts.yaml — Helm charts.

• packages.yaml — Packages and APT dependencies.

Output structure:

bundle/
├── images/          # Container images in OCI layout (per registry/repo/tag)
├── charts/          # Helm chart archives (.tgz)
├── packages/        # .deb packages (per-architecture when --arch=all)
└── terraform/
    ├── bin/         # OpenTofu zip
    └── runbooks/    # Runbook archives (.tgz)

A summary is printed when the download completes. If any step fails, the script logs errors to <output_dir>/download_errors.log and provides a retry command targeting only the failed categories.

Download PID dependencies

If you have the appropriate NVIDIA licenses, download the following dependencies separately from the NVIDIA Product Information Database (PID). These packages are required for AHR diagnostics and must be present on the BCM head node of the air-gapped environment.

Package	Version	PID link
nvdebug	v2.1.0	nvdebug
nvssvt	v1.7.1	nvssvt
nvlmapper	v14	nvlmapper
partnerdiag mfg switch L10	42174	partnerdiag mfg switch
partnerdiag mfg computer L10	60008-rev17	partnerdiag mfg computer
EUD	580.173.6	EUD

Copy these packages to a directory of your choice on the BCM head node. The Setup PID dependencies steps are run from that directory.

Manifest and bundle structure

The bundle output from nmc-airgap has this structure:

• bundle/helm/ — Helm chart tarballs (.tgz).

• bundle/images/ — Container images in OCI directory layout (by registry and image path). Compatible with skopeo copy and crane.

• bundle/files/ — Binary files (for example, Debian packages such as cm-setup-netq-*.deb and Grafana plugin zips such as yesoreyeram-infinity-datasource-3.7.4.zip).

• bundle/manifest.json — Bundle manifest with SHA256 checksums for verification and resume.

The helper scripts for BCM and AHR will download the appropriate artifacts for the NVIDIA Mission Control components and place them in their respective tarballs. Extract those tarballs inside the bundle directory so that nmc-airgap archive packages them with the rest of the bundle.