Downloading NVIDIA Mission Control Artifacts with nmc-airgap and Helper Scripts

The nmc-airgap tool is a CLI that reads a declarative manifest, downloads Helm charts, container images, and files in parallel, and outputs a portable bundle directory. The helper scripts perform similar functions but are specific to the NVIDIA Mission Control components they are intended for. Both nmc-airgap and the helper scripts are published on NGC in the NVIDIA Mission Control collection.

• nmc-airgap CLI (1.10.0) — container image or binary tarball

• nmc-airgap-manifests (1.12.0) — resources/nmc-airgap-manifests (one YAML per NVIDIA Mission Control release, e.g. nmc-2.3.0.yaml)

• AHR helper_scripts (2.3.31) — helper_scripts

nmc-airgap Tool

For both container image and binary, the nmc-airgap CLI tool supports x86 and arm64 architectures. For the binary, the tool is available as a tarball for both architectures and can be extracted to a directory. Once extracted, run the appropriate binary for the machine where the tool is used. For example, on macOS it would be nmc-airgap-darwin-{arm64|amd64} and on Linux it would be nmc-airgap-linux-{arm64|amd64}.

For example, for a mac arm64 machine, the binary would be:

tar -xzf nmc-airgap.tar.gz
./nmc-airgap-darwin-arm64

nmc-airgap CLI commands

• fetch — Download artifacts into a local bundle directory.

• validate — Validate manifest syntax and repository references.

• list — List artifacts that would be downloaded without downloading.

• verify — Verify bundle integrity with checksums after transfer to the air-gapped environment.

Fetching the bundle

1. Set environment variables for API keys and credentials:

Variable	Description
NGC_NVIDIA_NV_MISSION_CONTROL_API_KEY	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).
RUNAI_ARTIFACTORY_TOKEN	Token for the Run:ai air-gapped Artifactory (required for downloading Run:ai artifacts).

2. Run the fetch command. Use the manifest shipped with NVIDIA Mission Control (for example, manifest.yaml):

   nmc-airgap fetch --manifest manifest.yaml --arch=<amd64|arm64> --output ./bundle
   

   The preceding command will download all the components listed in the manifest.yaml file. This includes NVIDIA Mission Control Launchpad, NetQ (partially), K8s Security Policy, Grafana Dashboards, and the Grafana plugins (Infinity datasource, Loki Explore, and Metrics Drilldown).

   For NetQ, only the Debian packages will be downloaded. The NetQ tarball needs to be downloaded separately using the instructions provided in the NetQ installation guide. See NetQ installation guide for more details.

3. Optional: Download only specific components:

   nmc-airgap fetch --manifest manifest.yaml --output ./bundle --components launchpad,netq
   

4. Optional: Preview what will be downloaded (dry run):

   nmc-airgap fetch --manifest manifest.yaml --dry-run
   

5. Optional: Resume an interrupted download by skipping artifacts that are already present and verified:

   nmc-airgap fetch --manifest manifest.yaml --output ./bundle --skip-existing
   

6. Optional: Create a portable archive after download:

   nmc-airgap fetch --manifest manifest.yaml --output ./bundle --archive
   

Common fetch flags:

• -m, --manifest — Path to manifest file (default manifest.yaml).

• -o, --output — Output directory (default ./bundle).

• -c, --components — Comma-separated list of components to download (default: all enabled).

• -p, --parallel — Number of parallel downloads (default 10).

• --timeout — Per-artifact timeout (default 30m).

Validating the manifest

Before fetching, you can validate the manifest:

nmc-airgap validate --manifest manifest.yaml

This checks YAML syntax, required fields, repository references, and authentication configuration (and warns about unexpanded environment variables).

Listing artifacts

To list what the tool would download without downloading:

nmc-airgap list --manifest manifest.yaml

To list only certain components:

nmc-airgap list --manifest manifest.yaml --components launchpad

Verifying the bundle after transfer

After you copy the bundle to the air-gapped environment (for example, via physical media), verify integrity:

nmc-airgap verify --bundle ./bundle

The tool recomputes SHA256 checksums and compares them to the values in bundle/manifest.json.

Helper scripts

The helper scripts will download the appropriate artifacts for the NVIDIA Mission Control components and place them in their respective tarballs. We recommend extracting the tarballs to the directory where the bundle directory from the nmc-airgap fetch command is present to avoid any path issues as all files would be present in the same directory.

Autonomous Hardware Recovery (AHR)

The AHR airgap helper scripts automate the download and upload of all artifacts required by the AHR TUI plugin. The helper scripts were downloaded as part of the nmc-airgap fetch command.

nmc-airgap fetch --manifest manifest.yaml --output ./bundle --components ahr

The helper scripts are available in the bundle/ahr-airgap directory.

Download dependencies script

The download_ahr_dependencies.sh script can be run on a Debian-based machine with internet access to collect all AHR air-gap dependencies into a self-contained bundle directory. After that, the bundle is transferred to the head node of the air-gapped environment where, with the help of the upload_ahr_dependencies.sh script, the dependencies from the bundle are set up.

Important

For a single-architecture download (the --arch=amd64 or --arch=arm64 case), run the download script on a machine with the same CPU architecture as the target nodes (for example, run on an amd64 machine when targeting amd64 nodes). The installation will then be successful on all nodes sharing that architecture.

For a mixed-architecture cluster (a cluster with both amd64 and arm64 nodes), use --arch=all to produce a single bundle that installs on both architectures from one download host. The download host must have APT sources configured for both architectures — refer to Downloading for mixed-architecture clusters.

Important

It is recommended to run the download script on a machine with the same OS version as the head node and the software images used for compute/GPU nodes. Because the download script resolves the entire APT dependency tree for each required package, OS version mismatches can cause dependency conflicts during installation.

Note

The download script fetches approximately 130 GB of data (PID files alone account for roughly 9 GB). Ensure that at least 130 GB of free disk space is available in the output directory before starting the download.

Both download_ahr_dependencies.sh and upload_ahr_dependencies.sh may take a significant amount of time to complete — expect over one hour each depending on network and disk speed.

Prerequisites:

• python3 on PATH (version 3.11 or higher) with the pyyaml package installed.

• skopeo — required unless --skip-images is passed.

• helm — required unless --skip-charts is passed.

• envsubst — used to expand environment variables in manifest files.

Set the following environment variables before running the script:

Variable	Description
NGC_API_KEY	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).

The script exits with an error if a required variable is missing for the artifacts being downloaded.

Downloading for mixed-architecture clusters (–arch=all)

Use --arch=all when the target cluster mixes architectures — for example an amd64 head node with arm64 (Grace) compute nodes, or vice versa. A single --arch=all bundle contains:

• Container images as multi-architecture OCI image indexes, so each node pulls the variant matching its own architecture.

• Per-architecture .deb packages that coexist in bundle/packages/ because Debian’s <name>_<version>_<arch>.deb naming convention prevents collisions (for example, libc6_2.39-0ubuntu8_amd64.deb and libc6_2.39-0ubuntu8_arm64.deb).

The script auto-detects a sensible default for --arch based on the download host: it uses all only when the host can already resolve APT packages for both architectures, and otherwise falls back to the host’s native architecture. Passing --arch=all explicitly exits with an error if the cross-architecture APT sources described in the following section are not configured.

Configure cross-architecture APT sources on the download host

--arch=all resolves the full APT dependency tree once per architecture, so the download host must be able to fetch base Ubuntu packages for the foreign architecture as well as its native one. On Ubuntu, the two architecture families are served by different mirrors:

• amd64 (and i386) — http://archive.ubuntu.com/ubuntu and http://security.ubuntu.com/ubuntu.

• arm64 (and other ports) — http://ports.ubuntu.com/ubuntu-ports.

A stock amd64 host only has the archive.ubuntu.com mirror, which does not carry arm64 packages, so APT must be told where to find the foreign architecture before the download. The NVIDIA CUDA and DOCA/Mellanox repositories are configured automatically by the script for each architecture and need no manual setup — only the base Ubuntu mirrors require the steps that follow.

Run one of the following blocks, matching the download host’s own architecture. Each block enables the foreign architecture, restricts the host’s existing Ubuntu sources to its native architecture (so apt-get update does not return 404 errors fetching the foreign architecture from the wrong mirror), adds the foreign architecture’s mirror, and refreshes the APT cache. The blocks auto-detect the Ubuntu release codename, are idempotent, and require sudo.

On an amd64 download host (adds the arm64 repositories):

set -euo pipefail
. /etc/os-release
CODENAME="${VERSION_CODENAME:?cannot detect Ubuntu codename}"

# 1. Enable arm64 as a foreign architecture.
sudo dpkg --add-architecture arm64

# 2. Restrict the existing (amd64) Ubuntu sources to amd64 so apt does not
#    try to fetch arm64 from archive.ubuntu.com (which does not carry it).
if [ -f /etc/apt/sources.list.d/ubuntu.sources ] && \
   ! grep -q '^Architectures:' /etc/apt/sources.list.d/ubuntu.sources; then
    sudo sed -i '/^Types:/a Architectures: amd64' \
        /etc/apt/sources.list.d/ubuntu.sources
fi

# 3. Add the arm64 "ports" repositories.
sudo tee /etc/apt/sources.list.d/ubuntu-arm64.sources >/dev/null <<EOF
Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports
Suites: ${CODENAME} ${CODENAME}-updates ${CODENAME}-backports ${CODENAME}-security
Components: main restricted universe multiverse
Architectures: arm64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF

# 4. Refresh the APT cache.
sudo apt-get update

On an arm64 download host (adds the amd64 repositories):

set -euo pipefail
. /etc/os-release
CODENAME="${VERSION_CODENAME:?cannot detect Ubuntu codename}"

# 1. Enable amd64 as a foreign architecture.
sudo dpkg --add-architecture amd64

# 2. Restrict the existing (arm64) Ubuntu sources to arm64 so apt does not
#    try to fetch amd64 from ports.ubuntu.com (which does not carry it).
if [ -f /etc/apt/sources.list.d/ubuntu.sources ] && \
   ! grep -q '^Architectures:' /etc/apt/sources.list.d/ubuntu.sources; then
    sudo sed -i '/^Types:/a Architectures: arm64' \
        /etc/apt/sources.list.d/ubuntu.sources
fi

# 3. Add the amd64 archive + security repositories.
sudo tee /etc/apt/sources.list.d/ubuntu-amd64.sources >/dev/null <<EOF
Types: deb
URIs: http://archive.ubuntu.com/ubuntu
Suites: ${CODENAME} ${CODENAME}-updates ${CODENAME}-backports
Components: main restricted universe multiverse
Architectures: amd64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: ${CODENAME}-security
Components: main restricted universe multiverse
Architectures: amd64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF

# 4. Refresh the APT cache.
sudo apt-get update

Tip

The preceding blocks target the deb822 source format used by Ubuntu 24.04 (Noble). If the host still uses the classic single-line /etc/apt/sources.list format, add the [arch=...] qualifier to each deb line instead — for example deb [arch=amd64] http://archive.ubuntu.com/ubuntu noble main ... and deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports noble main .... Without an explicit arch= qualifier, APT tries to fetch the foreign architecture from the native mirror and apt-get update fails with 404 errors.

After running the block for your host, verify that both architectures resolve. dpkg must list the foreign architecture and apt-cache madison must return rows for a known package in each architecture:

dpkg --print-foreign-architectures      # expect: arm64 (or amd64)
apt-cache madison curl:amd64             # expect: at least one row
apt-cache madison curl:arm64             # expect: at least one row

These are the same probes the download script uses to decide whether --arch=all is available; once all three succeed, --arch=all (and the auto-detected default) will resolve cleanly.

Quick start:

export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh ./bundle

Usage:

export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh <output_dir> [options]

Argument	Description
<output_dir>	Directory where the bundle is written (required). Created if it does not exist.
--arch <amd64|arm64|all>	Target architecture for images and packages. all downloads both architectures into one bundle for mixed clusters. The default is auto-detected from the download host: all when the host has working APT sources for both architectures, otherwise the host’s native architecture. Refer to Downloading for mixed-architecture clusters.
--set NAME=VER	Override an artifact version. Can be specified multiple times.
--retries <n>	Number of retry attempts for transient failures (default: 3).
--skip-images	Skip downloading Docker images.
--skip-charts	Skip downloading Helm charts.
--skip-packages	Skip downloading packages (NGC, URL, and APT).
--skip-files	Skip downloading files (OpenTofu, runbooks).
--skip-runbooks	Skip downloading runbook artifacts specifically.

Examples:

# Full bundle for the download host's native architecture
./download_ahr_dependencies.sh ./bundle

# Full bundle for a mixed amd64 + arm64 cluster (one bundle, both arches)
./download_ahr_dependencies.sh ./bundle --arch=all

# Only packages and files (skip images and charts)
./download_ahr_dependencies.sh ./bundle --skip-images --skip-charts

The download process runs four steps in order:

1. Images — Pulls container images from NGC and stores each as an OCI image layout (multi-architecture when --arch=all) under bundle/images/.

2. Charts — Pulls Helm charts from NGC and saves them as .tgz archives in bundle/charts/.

3. Packages — Downloads .deb packages from NGC, direct URLs, and APT repositories into bundle/packages/. APT packages include full recursive dependency resolution, meaning the entire dependency tree for every required package is downloaded. This ensures that the bundle is self-contained and all transitive dependencies are available for offline installation.

4. Files — Downloads miscellaneous files (OpenTofu binary, runbook archives) into subdirectories under bundle/terraform/.

Artifact lists are defined in YAML manifest files under scripts/airgap/prod/:

• sources.yaml — Registry URLs and credentials.

• images.yaml — Container images.

• charts.yaml — Helm charts.

• packages.yaml — Packages and APT dependencies.

Output structure:

bundle/
├── images/          # Container images in OCI layout (per registry/repo/tag)
├── charts/          # Helm chart archives (.tgz)
├── packages/        # .deb packages (per-architecture when --arch=all)
└── terraform/
    ├── bin/         # OpenTofu zip
    └── runbooks/    # Runbook archives (.tgz)

A summary is printed when the download completes. If any step fails, the script logs errors to <output_dir>/download_errors.log and provides a retry command targeting only the failed categories.

Download PID dependencies

If you have the appropriate NVIDIA licenses, download the following dependencies separately from the NVIDIA Product Information Database (PID). These packages are required for AHR diagnostics and must be present on the head node of the air-gapped environment.

Package	Version	PID link
nvdebug	v2.1.0 (arm64), v2.0.1 (amd64)	nvdebug
nvssvt	v1.7.1	nvssvt
nvlmapper	v14	nvlmapper
partnerdiag mfg switch L10	42174	partnerdiag mfg switch
partnerdiag mfg computer L10	50896-rev13	partnerdiag mfg computer
EUD	580.159.5	EUD

Copy these packages to a directory of your choice on the head node. The Setup PID dependencies steps are run from that directory.

Manifest and bundle structure

The bundle output from nmc-airgap has this structure:

• bundle/helm/ — Helm chart tarballs (.tgz).

• bundle/images/ — Container images in OCI directory layout (by registry and image path). Compatible with skopeo copy and crane.

• bundle/files/ — Binary files (for example, Debian packages such as cm-setup-netq-*.deb and Grafana plugin zips such as yesoreyeram-infinity-datasource-3.7.4.zip).

• bundle/manifest.json — Bundle manifest with SHA256 checksums for verification and resume.

The helper scripts for BCM and AHR will download the appropriate artifacts for the NVIDIA Mission Control components and place them in their respective tarballs. We recommend extracting the tarballs to the directory where the bundle directory is present to avoid any path issues as all files would be present in the bundle directory.