Configuring Registry Mirroring#

Note

Disclaimer: Local registry mirroring is supported in NMC 2.3.0 but requires manual configuration. Automated registry mirroring with support for both x86 and Arm container images is planned for a future release. This page will be updated as availability is confirmed.

Important

The registry mirroring configuration described on this page has not been tested or validated by NVIDIA in NMC 2.3.0. Manual configuration is required, and only x86 container images are supported at this time. Use in production environments is not recommended until this feature has been fully validated. Automated mirroring with x86 and Arm support is planned for a future NMC release.

For NMC to pull images from your air-gap registry without changing Helm charts, configure the container runtime on all cluster nodes to use your private registry as a mirror for the following upstream registries:

nvcr.io
docker.io
ghcr.io
quay.io
reg.kyverno.io
registry.k8s.io

The simplest approach is to configure a default mirror that redirects all image pulls to your local registry, as shown in the following Containerd and BCM examples.

RKE2 example#

Use a registries.yaml configuration such as:

# registries.yaml
mirrors:
  nvcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  docker.io:
    endpoint:
      - https://airgap-registry.com:5443
  ghcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  quay.io:
    endpoint:
      - https://airgap-registry.com:5443
  reg.kyverno.io:
    endpoint:
      - https://airgap-registry.com:5443
  registry.k8s.io:
    endpoint:
      - https://airgap-registry.com:5443
configs:
  myhostname.com:5443:
    tls:
      ca_file: "/etc/ssl/certs/airgap-registry.pem"

Containerd example#

Configure containerd to use the certs.d directory and add mirror host entries:

sudo mkdir -p /etc/containerd/certs.d
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
# Ensure containerd uses the certs.d config path
# (add or update config_path)
echo 'config_path = "/etc/containerd/certs.d"' | sudo tee -a /etc/containerd/config.toml
sudo systemctl restart containerd

# Mirror for NVIDIA registry
sudo mkdir -p /etc/containerd/certs.d/nvcr.io
sudo tee /etc/containerd/certs.d/nvcr.io/hosts.toml >/dev/null << 'EOF'
server = "https://nvcr.io"
[host."https://airgap-registry.com:5443"]
capabilities = ["pull", "resolve"]
EOF

# Default mirror for other registries (optional)
sudo mkdir -p /etc/containerd/certs.d/_default
sudo tee /etc/containerd/certs.d/_default/hosts.toml >/dev/null << 'EOF'
[host."https://my-default-mirror.example.com"]
capabilities = ["pull", "resolve"]
EOF
sudo systemctl restart containerd

Replace airgap-registry.com, ports, and paths with your registry hostname, port, and CA certificate path. If your registry uses a self-signed or enterprise CA, set ca_file to the path of that certificate.

BCM Example#

# BCM head node
cat /cm/local/apps/containerd/var/etc/config.toml
version = 2
imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"] # this is the dir to add all other configurations

# The following steps must be run on BCM images (you must cm-chroot/systemd-nspawn into the image) for which containers are used.  This will likely be GPU node images, k8s node images, and also the headnode.

# Upstream NVIDIA registry
mkdir -p /cm/local/apps/containerd/var/etc/certs.d/nvcr.io
tee /cm/local/apps/containerd/var/etc/certs.d/nvcr.io/hosts.toml >/dev/null << 'EOF'
server = "https://nvcr.io"

[host."https://master.cm.cluster:5000"]
  capabilities = ["pull", "resolve"]
  ca = "/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt"
EOF

# For all registries
mkdir -p /cm/local/apps/containerd/var/etc/certs.d/_default

tee /cm/local/apps/containerd/var/etc/certs.d/_default/hosts.toml >/dev/null << 'EOF'
server = "https://registry-1.docker.io"

[host."https://master.cm.cluster:5000"]
  capabilities = ["pull", "resolve"]
  ca = "/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt"
EOF

# Trust the ca cert
cp /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt /usr/local/share/ca-certificates/master-cm-cluster-registry.crt

# Log out of the image
# Run imageupdate on the k8s node image
cmsh
device
imageupdate -w -c <k8s node image category>

# Run the following on headnode
systemctl restart containerd
update-ca-certificates

# Run the following to restart containerd and update certs on k8s nodes:
pdsh -g category=k8s-admin 'systemctl restart containerd'
pdsh -g category=k8s-admin 'update-ca-certificates'