Configuring Registry Mirroring

For NMC to pull images from your air-gap registry without changing Helm charts, configure the container runtime on all cluster nodes to use your private registry as a mirror for the following upstream registries:

• nvcr.io

• docker.io

• ghcr.io

• quay.io

• reg.kyverno.io

• registry.k8s.io

The simplest approach is to configure a default mirror that redirects all image pulls to your local registry, as shown in the following Containerd and BCM examples.

RKE2 example

Use a registries.yaml configuration such as:

# registries.yaml
mirrors:
  nvcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  docker.io:
    endpoint:
      - https://airgap-registry.com:5443
  ghcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  quay.io:
    endpoint:
      - https://airgap-registry.com:5443
  reg.kyverno.io:
    endpoint:
      - https://airgap-registry.com:5443
  registry.k8s.io:
    endpoint:
      - https://airgap-registry.com:5443
configs:
  myhostname.com:5443:
    tls:
      ca_file: "/etc/ssl/certs/airgap-registry.pem"

Containerd example

Configure containerd to use the certs.d directory and add mirror host entries:

sudo mkdir -p /etc/containerd/certs.d
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
# Ensure containerd uses the certs.d config path
# (add or update config_path)
echo 'config_path = "/etc/containerd/certs.d"' | sudo tee -a /etc/containerd/config.toml
sudo systemctl restart containerd

# Mirror for NVIDIA registry
sudo mkdir -p /etc/containerd/certs.d/nvcr.io
sudo tee /etc/containerd/certs.d/nvcr.io/hosts.toml >/dev/null << 'EOF'
server = "https://nvcr.io"
[host."https://airgap-registry.com:5443"]
capabilities = ["pull", "resolve"]
EOF

# Default mirror for other registries (optional)
sudo mkdir -p /etc/containerd/certs.d/_default
sudo tee /etc/containerd/certs.d/_default/hosts.toml >/dev/null << 'EOF'
[host."https://my-default-mirror.example.com"]
capabilities = ["pull", "resolve"]
EOF
sudo systemctl restart containerd

Replace airgap-registry.com, ports, and paths with your registry hostname, port, and CA certificate path. If your registry uses a self-signed or enterprise CA, set ca_file to the path of that certificate.

BCM Example

# BCM head node
cat /cm/local/apps/containerd/var/etc/config.toml
version = 2
imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"] # this is the dir to add all other configurations

# The following steps must be run on BCM images (you must cm-chroot/systemd-nspawn into the image) for which containers are used.  This will likely be GPU node images, k8s node images, and also the headnode.

# Upstream NVIDIA registry
mkdir -p /cm/local/apps/containerd/var/etc/certs.d/nvcr.io
tee /cm/local/apps/containerd/var/etc/certs.d/nvcr.io/hosts.toml >/dev/null << 'EOF'
server = "https://nvcr.io"

[host."https://master.cm.cluster:5000"]
  capabilities = ["pull", "resolve"]
  ca = "/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt"
EOF

# For all registries
mkdir -p /cm/local/apps/containerd/var/etc/certs.d/_default

tee /cm/local/apps/containerd/var/etc/certs.d/_default/hosts.toml >/dev/null << 'EOF'
server = "https://registry-1.docker.io"

[host."https://master.cm.cluster:5000"]
  capabilities = ["pull", "resolve"]
  ca = "/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt"
EOF

# Trust the ca cert
cp /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt /usr/local/share/ca-certificates/master-cm-cluster-registry.crt

# Log out of the image
# Run imageupdate on the k8s node image
cmsh
device
imageupdate -w -c <k8s node image category>

# Run the following on headnode
systemctl restart containerd
update-ca-certificates

# Run the following to restart containerd and update certs on k8s nodes:
pdsh -g category=k8s-admin 'systemctl restart containerd'
pdsh -g category=k8s-admin 'update-ca-certificates'