Downloading NMC Artifacts with nmc-airgap and Helper Scripts#

The nmc-airgap tool is a CLI that reads a declarative manifest, downloads Helm charts, container images, and files in parallel, and outputs a portable bundle directory. The helper scripts perform similar functions but are specific to the NMC components they are intended for. Both nmc-airgap and the helper scripts are published on NGC in the NMC collection.

nmc-airgap CLI (1.6.1) — container image or binary tarball
nmc-airgap-manifests (1.6.5) — resources/nmc-airgap-manifests (one YAML per NMC release, e.g. nmc-2.3.0.yaml)
AHR helper_scripts (2.3.22) — helper_scripts

`nmc-airgap` Tool#

For both container image and binary, the nmc-airgap CLI tool supports x86 and arm64 architectures. In case of binary, the tool is available as a tarball for both architectures and can be extracted to a directory. Once extracted, run the appropriate binary depending on the machine where the tool is used. For e.g. for mac OS it would be nmc-airgap-darwin-{arm64|amd64} and for linux it would be nmc-airgap-linux-{arm64|amd64}.

For example, for a mac arm64 machine, the binary would be:

tar -xzf nmc-airgap.tar.gz
./nmc-airgap-darwin-arm64

`nmc-airgap` CLI commands#

fetch — Download artifacts into a local bundle directory.
validate — Validate manifest syntax and repository references.
list — List artifacts that would be downloaded without downloading.
verify — Verify bundle integrity with checksums after transfer to the air-gapped environment.

Fetching the bundle#

Set environment variables for API keys and credentials:

Variable	Description
`NGC_NVIDIA_NV_MISSION_CONTROL_API_KEY`	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).
`RUNAI_ARTIFACTORY_USER`	Username for RunAI Artifactory (required for downloading RunAI artifacts).
`RUNAI_ARTIFACTORY_PASSWORD`	Password for RunAI Artifactory (required for downloading RunAI artifacts).

Run the fetch command. Use the manifest shipped with NMC (for example, manifest.yaml):
```
nmc-airgap fetch --manifest manifest.yaml --arch=<amd64|arm64> --output ./bundle
```
The preceding command will download all the components listed in the manifest.yaml file. This includes NMC Launchpad, NetQ (partially), K8s Security Policy, and Grafana Dashboards.

For NetQ, only the debian packages will be downloaded. The NetQ tarball needs to be downloaded separately using the instructions provided in the NetQ installation guide. See NetQ installation guide for more details.

Optional: Download only specific components:

nmc-airgap fetch --manifest manifest.yaml --output ./bundle --components launchpad,netq

Optional: Preview what will be downloaded (dry run):

nmc-airgap fetch --manifest manifest.yaml --dry-run

Optional: Resume an interrupted download by skipping artifacts that are already present and verified:
```
nmc-airgap fetch --manifest manifest.yaml --output ./bundle --skip-existing
```

Optional: Create a portable archive after download:

nmc-airgap fetch --manifest manifest.yaml --output ./bundle --archive

Common fetch flags:

-m, --manifest — Path to manifest file (default manifest.yaml).
-o, --output — Output directory (default ./bundle).
-c, --components — Comma-separated list of components to download (default: all enabled).
-p, --parallel — Number of parallel downloads (default 10).
--timeout — Per-artifact timeout (default 30m).

Validating the manifest#

Before fetching, you can validate the manifest:

nmc-airgap validate --manifest manifest.yaml

This checks YAML syntax, required fields, repository references, and authentication configuration (and warns about unexpanded environment variables).

Listing artifacts#

To list what the tool would download without downloading:

nmc-airgap list --manifest manifest.yaml

To list only certain components:

nmc-airgap list --manifest manifest.yaml --components launchpad

Verifying the bundle after transfer#

After you copy the bundle to the air-gapped environment (for example, via physical media), verify integrity:

nmc-airgap verify --bundle ./bundle

The tool recomputes SHA256 checksums and compares them to the values in bundle/manifest.json.

Helper scripts#

The helper scripts will download the appropriate artifacts for the NMC components and place them in their respective tarballs. We recommend extracting the tarballs to the directory where the bundle directory from the nmc-airgap fetch command is present to avoid any path issues as all files would be present in the same directory.

Autonomous Hardware Recovery (AHR)#

The AHR airgap helper scripts automate the download and upload of all artifacts required by the AHR TUI plugin. The helper scripts were downloaded as part of the nmc-airgap fetch command.

nmc-airgap fetch --manifest manifest.yaml --output ./bundle --components ahr

The helper scripts are available in the bundle/ahr-airgap directory.

Download dependencies script

The download_ahr_dependencies.sh script can be run on a Debian-based machine with internet access to collect all AHR air-gap dependencies into a self-contained bundle directory. After that, the bundle is transferred to the head node of the air-gapped environment where, with the help of the upload_ahr_dependencies.sh script, the dependencies from the bundle are setup.

Important

The download script must be run on a machine with the same CPU architecture as the Kubernetes head node (for example, run on an amd64 machine when targeting amd64 nodes). The installation will then be successful on all nodes sharing the same architecture.

Important

It is recommended to run the download script on a machine with the same OS version as the head node and the software images used for compute/GPU nodes. Because the download script resolves the entire APT dependency tree for each required package, OS version mismatches can cause dependency conflicts during installation.

Note

The download script fetches approximately 130 GB of data (PID files alone account for roughly 9 GB). Ensure that at least 130 GB of free disk space is available in the output directory before starting the download.

Both download_ahr_dependencies.sh and upload_ahr_dependencies.sh may take a significant amount of time to complete — expect over one hour each depending on network and disk speed.

Prerequisites:

python3 on PATH (version 3.11 or higher) with the pyyaml package installed.
skopeo — required unless --skip-images is passed.
helm — required unless --skip-charts is passed.
envsubst — used to expand environment variables in manifest files.

Set the following environment variables before running the script:

Variable	Description
`NGC_API_KEY`	API key for NVIDIA NGC (required for downloading container images, Helm charts, artifacts).

The script exits with an error if a required variable is missing for the artifacts being downloaded.

Quick start:

export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh ./bundle

Usage:

export NGC_API_KEY=<your-ngc-api-key> && ./download_ahr_dependencies.sh <output_dir> [options]

Argument	Description
`<output_dir>`	Directory where the bundle is written (required). Created if it does not exist.
`--set NAME=VER`	Override an artifact version. Can be specified multiple times.
`--retries <n>`	Number of retry attempts for transient failures (default: 3).
`--skip-images`	Skip downloading Docker images.
`--skip-charts`	Skip downloading Helm charts.
`--skip-packages`	Skip downloading packages (NGC, URL, and APT).
`--skip-files`	Skip downloading files (OpenTofu, runbooks).
`--skip-runbooks`	Skip downloading runbook artifacts specifically.

Examples:

# Full bundle with a specific platform tag
./download_ahr_dependencies.sh ./bundle

# Only packages and files (skip images and charts)
./download_ahr_dependencies.sh ./bundle --skip-images --skip-charts

The download process runs four steps in order:

Images — Pulls Docker images from NGC and saves each as a .tar in bundle/images/.
Charts — Pulls Helm charts from NGC and saves them as .tgz archives in bundle/charts/.
Packages — Downloads .deb packages from NGC, direct URLs, and APT repositories into bundle/packages/. APT packages include full recursive dependency resolution, meaning the entire dependency tree for every required package is downloaded. This ensures that the bundle is self-contained and all transitive dependencies are available for offline installation.
Files — Downloads miscellaneous files (OpenTofu binary, runbook archives) into subdirectories under bundle/terraform/.

Artifact lists are defined in YAML manifest files under scripts/airgap/prod/:

sources.yaml — Registry URLs and credentials.
images.yaml — Container images.
charts.yaml — Helm charts.
packages.yaml — Packages and APT dependencies.

Output structure:

bundle/
├── images/          # Docker image tarballs (.tar)
├── charts/          # Helm chart archives (.tgz)
├── packages/        # .deb packages
└── terraform/
    ├── bin/         # OpenTofu zip
    └── runbooks/    # Runbook archives (.tgz)

A summary is printed when the download completes. If any step fails, the script logs errors to <output_dir>/download_errors.log and provides a retry command targeting only the failed categories.

Download PID dependencies

If you have the appropriate NVIDIA licenses, download the following dependencies separately from the NVIDIA Product Information Database (PID). These packages are required for AHR diagnostics and must be present on the head node of the air-gapped environment.

Package	Version	PID link
`nvdebug`	v2.0.1	nvdebug
`nvssvt`	v1.7.1	nvssvt
`nvlmapper`	v14	nvlmapper
`partnerdiag` mfg switch L10	42174	partnerdiag mfg switch
`partnerdiag` mfg computer L10	50896-rev13	partnerdiag mfg computer
`EUD`	580.126.12	EUD

Copy these packages to a directory of your choice on the head node. The Setup PID dependencies steps are run from that directory.

Manifest and bundle structure#

The bundle output from nmc-airgap has this structure:

bundle/helm/ — Helm chart tarballs (.tgz).
bundle/images/ — Container images in OCI directory layout (by registry and image path). Compatible with skopeo copy and crane.
bundle/files/ — Binary files (for example, Debian packages such as cm-setup-netq-*.deb).
bundle/manifest.json — Bundle manifest with SHA256 checksums for verification and resume.

The helper scripts for BCM and AHR will download the appropriate artifacts for the NMC components and place them in their respective tarballs. We recommend extracting the tarballs to the directory where the bundle directory is present to avoid any path issues as all files would be present in the bundle directory.