Air-Gapped Image Mirroring
Discover every container image and Helm chart a recipe needs, then mirror them into a private registry for air-gapped deployment. For the per-flag reference, see CLI reference: aicr mirror list. For the static image inventory across all registered components, see Container Image Inventory.
Overview
aicr mirror list renders each component’s Helm chart with recipe-resolved
values, scans referenced manifests, and produces a deduplicated list of
container images and chart references. Manifest reads honor the data source
resolved for the mirror list invocation itself: pass --data <dir> to
aicr mirror list and an overlay manifest shadowing an embedded path is used
in place of the embedded copy. The data directory is not remembered from
when the recipe was generated — a saved recipe carries no --data path, so
you must supply --data again on each mirror list run that needs the
overlay. The output is available in four formats — two general-purpose
(YAML, JSON) and two tool-specific (Hauler, Zarf).
Trust boundary: Discovery shells out to
helm template, which executes the full Go template engine (tpl,include,lookup). AICR recipes reference trusted, pinned charts from known repositories. Do not runmirror listagainst untrusted or unvetted recipe files — doing so executes arbitrary template code from those charts.
How this relates to chart vendoring: aicr bundle --vendor-charts embeds
Helm chart tarballs into a bundle so helm install needs no registry egress.
aicr mirror list discovers the container images those charts reference plus
the chart coordinates themselves. For a fully air-gapped deployment, use both:
mirror images and charts into your private registry, then deploy the vendored
bundle.
Enforcing signatures on the mirror: to admit only images you have signed, sign the mirrored digests with your own key and enforce the signature at admission; see Enforcing Your Signatures on Mirrored Images. With the Sigstore-bundle signatures cosign v3 writes by default, those policies need the registry to serve TLS, so the plain-HTTP examples below suit only a test registry. Those policies were measured on a cluster with outbound network access, not an air-gapped one.
Prerequisites
aicrCLI installed (see Installation)- A recipe file (
aicr recipe --output recipe.yaml) or query parameters helmv3+ on$PATH(required for chart rendering during discovery)- For the Hauler workflow:
haulerCLI - For the Zarf workflow:
zarfCLI - YAML/JSON output works without any additional tools
Output Formats
Quick Start
Discover Images
From an existing recipe file:
Or resolve a recipe from query parameters:
Save to File
Hauler Workflow
Hauler syncs container images and Helm charts into a local store, then copies them to a target registry.
1. Generate the Hauler manifest
The output is a multi-document YAML with an Images document (and optionally a
Charts document):
2. Sync into the Hauler store
To pull only a single platform (reduces download size):
3. Copy to your private registry
For registries using plain HTTP (e.g., local test registries):
Zarf Workflow
Zarf packages container images into a single distributable tarball that can be carried to an air-gapped environment and mirrored into a registry.
1. Generate the Zarf package config
The output is a ZarfPackageConfig:
Keys are sorted at every level: the renderer marshals through
serializer.MarshalYAMLDeterministic, so given the same discovered inventory it
produces byte-identical output. Discovery itself is best-effort and can omit a
component’s images without failing the command — see the completeness caveat
under Relationship to Container Image Inventory.
2. Create the Zarf package
Place zarf.yaml in its own directory and create the package:
This pulls every listed image and produces a zarf-package-*.tar.zst file.
3. Mirror to your private registry
Transfer the tarball to the air-gapped environment, then mirror:
For registries using plain HTTP:
Overriding Component Values
The --set flag overrides Helm values at discovery time, changing which images
appear in the output. This is useful when you know certain sub-components will
be disabled in your deployment and want to exclude their images from the mirror
list.
The GPU driver, GDRCopy, GDS, vGPU Manager and DOCA driver images are listed
with the tag the operator pulls on a node: <version>-<os><os-version> for the
GPU driver, GDRCopy (gdrdrv), GDS (nvidia-fs) and vGPU Manager
(vgpuManager, when enabled) (for example 570.86.16-ubuntu24.04 and
v2.6-ubuntu24.04) and <version>-<os><os-version>-<arch> for the DOCA
driver. mirror list reads the OS from the recipe’s OS.release.ID and
OS.release.VERSION_ID constraints (ubuntu and rhel are supported; for
rhel the GPU Operator tags use only the major version, for example
580.173.02-rhel9) and the DOCA architecture from the accelerator (gb200, gb300, and vr200 are
arm64; other accelerators are amd64). A driver whose tag cannot be resolved
for the recipe is not listed. This happens when the recipe does not pin one OS,
the OS has no known tag pattern, the accelerator is any (DOCA driver only),
or the driver is precompiled (its tags also name the node kernel). That driver is named in the
component’s warnings instead, so mirror the tag for your nodes yourself. The
Hauler and Zarf formats carry no warnings, so there the driver is simply absent
while the command still exits 0 (the warning is also logged to stderr). Run the
YAML or JSON format first to see which driver tags you must add. A
driver switched off with driver.enabled=false is not listed, and neither are
its GDRCopy and GDS containers. GDRCopy and GDS also follow their own enabled
flag.
The override key format is component:path.to.field=value, where the component
name matches either the component name in the recipe or its valueOverrideKeys
alias from the registry (e.g., gpuoperator for gpu-operator).
Relationship to Container Image Inventory
The Container Image Inventory is a static reference
generated from recipes/registry.yaml with default values. It lists every
image across all registered components regardless of recipe.
aicr mirror list is recipe-specific: it renders charts with the actual
resolved values for your target configuration (service, accelerator, intent,
OS). The resulting image list is typically a subset of the full inventory,
limited to the components and sub-components your recipe enables.
Use the inventory for security audit and compliance. Use mirror list to
generate the deployment-specific manifest for a specific deployment.
Completeness caveat: discovery is best-effort, not all-or-nothing. A per-component failure to load values, render the chart (
helm template), or extract images from a manifest, or a driver tag the recipe does not determine (see above), is non-fatal — the component’s images (for an undetermined driver tag, only that driver image) are omitted and the failure is recorded as a warning, but the command still exits 0. The YAML and JSON outputs carry these warnings in awarningsfield; the Hauler and Zarf outputs omit them entirely. Treat a tool-specific manifest as authoritative only after confirming the YAML/JSON run reported no warnings — otherwise the mirrored set may be silently incomplete. Invalid members in a recognized mapping-valuedimageorscalingPodImagedescriptor are the exception:mirror listexits nonzero instead of emitting a known-incomplete image set.
Runtime validation image:
slinky-slurm-healthlaunchesdocker.io/library/alpine:3.23.3dynamically throughsrunon GPU-backed NodeSets. This image is already covered byaicr mirror list: it is the same Alpine tag the slinky-slurm chart’s initconf/logfile sidecars pin, so chart rendering surfaces it for every Slurm recipe. Mirroring alone, however, does not redirect thesrunpull — setAICR_VALIDATOR_IMAGE_REGISTRYto your destination registry so the runtime pull resolves there; the validator preserves thelibrary/alpine:3.23.3repository and tag. Without a registry override, Slurm workers need Docker Hub egress for this check.
AICR agent and validator images
aicr mirror list covers what the recipe’s components deploy. It does not list
the images AICR itself runs in the cluster, so mirror these separately:
- The snapshot agent,
ghcr.io/nvidia/aicr, run byaicr snapshotand byaicr validatewhen no--snapshotis given. - The validator images under
ghcr.io/nvidia/aicr-validators/(deployment,performance,conformance, andaiperf-benchfor inference performance checks), run byaicr validate.
A CNCF submission run (aicr validate --cncf-submission) cannot complete
offline: its last step runs the upstream submission validator on the machine
aicr runs on, which fetches the CNCF landscape from
raw.githubusercontent.com and requests every vendor and documentation URL
the submission cites. Without that egress the run exits 6
(SERVICE_UNAVAILABLE); see
Exit status of a submission run.
Release builds use each image at the CLI’s own version tag. Copy them into your registry, then point AICR at the copies:
AICR_VALIDATOR_IMAGE_REGISTRY replaces the registry host and the first path
segment, so ghcr.io/nvidia/aicr-validators/conformance resolves to
${REGISTRY}/aicr-validators/conformance. If your mirroring tool keeps the
source path (${REGISTRY}/nvidia/aicr-validators/...), set the variable to
${REGISTRY}/nvidia instead. Add --image-pull-secret when the registry needs
credentials; the conformance checks that create a per-run namespace copy the
named Secrets into it for their probe pods, and the copies are deleted with the
namespace. Some benchmark checks pin workload images the variable does not
rewrite, such as the NCCL runtime image (override it with
AICR_NCCL_RUNTIME_IMAGE) and the vanilla docker.io/library/ubuntu:22.04
probe image of the driver-runtime-management and workload-sandboxing
conformance checks (override it with AICR_DRIVER_RUNTIME_IMAGE and
AICR_SANDBOX_PROBE_IMAGE, each a full reference to your copy); see
Validation. The opt-in CRE checks nccl-cre-all-reduce-bw
and cre-training-goodput pull their workload images by digest with no
override variable, so mirror them at the digests
recipes/components/nvcre/workload-images.yaml records; see the CRE section
of Validation. Their references keep the public.ecr.aws and
nvcr.io hosts, so a disconnected cluster also needs its nodes’ container
runtime to redirect both registries to your mirror, for example with a
containerd registry mirror per host. A digest reference resolves through such a
mirror unchanged.
The inference performance check also runs the Dynamo runtime images
nvcr.io/nvidia/ai-dynamo/vllm-runtime and
nvcr.io/nvidia/ai-dynamo/dynamo-frontend. The variable rewrites them by the
same rule, so nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version> resolves to
${REGISTRY}/ai-dynamo/vllm-runtime:<version>. <version> is the
dynamo-platform chart version your recipe deploys, which aicr mirror list
reports. The check tags both images with that version, so an overlay or
--data override of the chart moves them with it. Mirror both images to that
path at that tag. The check copies the Secrets named by --image-pull-secret
that exist in the validator namespace into its per-run namespace, so the Dynamo
pods and the model-cache Job pull with the same credentials. A named Secret
missing from the validator namespace is skipped with a warning in the check’s
log, and those pods then pull without it. If the check times out while one of
those pods is still failing to pull, the error names the pod and the image it
could not pull.
AICR_VALIDATOR_IMAGE_TAG does not retag the Dynamo images, because their tag
follows the dynamo-platform chart the recipe deploys, so a retag would run a
runtime that differs from the operator. Registry redirection does not make the
check fully offline, since its model-cache Job still downloads weights from
Hugging Face.