Configuring Registry Mirroring#

Note

Disclaimer: Local registry mirroring is supported in NMC 2.3.0 but requires manual configuration. Automated registry mirroring with support for both x86 and Arm container images is planned for a future release. This page will be updated as availability is confirmed.

Configure the container runtime on every cluster node to use your local registry as a mirror for the following upstream registries. This configuration lets NMC pull images without changes to the Helm charts:

nvcr.io
docker.io
ghcr.io
quay.io
reg.kyverno.io
registry.k8s.io

The simplest approach is to configure a default mirror that redirects all image pulls to your local registry, as shown in the following BCM, RKE2, and containerd examples. Use the example that matches your environment.

BCM example (most deployments)#

Deploy the local container registry before you configure mirroring. The cm-container-registry-setup wizard deploys the registry and generates its CA certificate at /cm/local/apps/containerd/var/etc/certs.d/<registry-host>:<port>/ca.crt. If you have not yet deployed the registry, the steps that copy that certificate into the node trust store fail with cp: cannot stat.

Set the following variables to match your environment. The shell commands in the mirror and software-image procedures reference these variables:

REGISTRY_HOST="master.cm.cluster"        # Hostname of your local registry
REGISTRY_PORT="5000"                     # Port of your local registry
NODE_CATEGORY="k8s-admin"                # BCM category of the nodes to configure

CERTS_D="/cm/local/apps/containerd/var/etc/certs.d"
REGISTRY_CA="${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt"

Apply these instructions to the k8s-admin category. If the k8s-user cluster is already installed, repeat the image and node steps for the k8s-user category. If it is not installed, apply those steps after the cluster is deployed.

Before editing the software image, update the BCM configuration overlay that provides the generic::containerd role. BCM can regenerate containerd configuration during imageupdate. Set both environment values in the kube-k8s-admin-master overlay:

cmsh
configurationoverlay
use kube-k8s-admin-master
roles
use generic::containerd
environments
use registry_mirror
set value https://master.cm.cluster:5000
environments
use registry_mirror_ca_entry
set value "ca = '/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt'"
commit
quit

Replace master.cm.cluster:5000 with the host and port of your registry, and kube-k8s-admin-master with the overlay that applies to your category. Do not enclose the registry_mirror URL in quotation marks, because cmsh stores the quotation marks as part of the value.

You can also set both values with the following commands:

cmsh -c "configurationoverlay; use kube-k8s-admin-master; roles; use generic::containerd; environments; use registry_mirror; set value https://master.cm.cluster:5000; commit"
cmsh -c "configurationoverlay; use kube-k8s-admin-master; roles; use generic::containerd; environments; use registry_mirror_ca_entry; set value \"ca = '/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt'\"; commit"

On the BCM head node, confirm that containerd imports the additional configuration directory:

cat /cm/local/apps/containerd/var/etc/config.toml

The output is similar to the following. The version value depends on the containerd release. The line that matters is imports, which points to the directory that holds all other containerd configuration:

version = 4
imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"]

Add the mirror configuration#

Apply the mirror configuration in two places: directly on the head node and inside the software image for each category whose nodes run containers. The commands are identical in both places.

First, run the configuration commands directly on the head node:

# Upstream NVIDIA registry
mkdir -p "${CERTS_D}/nvcr.io"
tee "${CERTS_D}/nvcr.io/hosts.toml" >/dev/null << EOF
server = "https://nvcr.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# For all other registries
mkdir -p "${CERTS_D}/_default"
tee "${CERTS_D}/_default/hosts.toml" >/dev/null << EOF
[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# Override any existing Docker Hub mirror
mkdir -p "${CERTS_D}/docker.io"
tee "${CERTS_D}/docker.io/hosts.toml" >/dev/null << EOF
server = "https://registry-1.docker.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# Trust the CA certificate
cp "${REGISTRY_CA}" "/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
update-ca-certificates

Copy the registry CA and mirror files into the software image used by the selected node category, then update the CA bundle in that image:

set -e
NODE_IMAGE=$(cmsh -c "category; use ${NODE_CATEGORY}; get softwareimage")
echo "${NODE_IMAGE}"
IMAGE_ROOT="/cm/images/${NODE_IMAGE}"
mkdir -p "${IMAGE_ROOT}/usr/local/share/ca-certificates"
cp -fv "${REGISTRY_CA}" \
  "${IMAGE_ROOT}/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
mkdir -p "${IMAGE_ROOT}${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}" \
  "${IMAGE_ROOT}${CERTS_D}/nvcr.io" \
  "${IMAGE_ROOT}${CERTS_D}/_default" \
  "${IMAGE_ROOT}${CERTS_D}/docker.io"
cp -fv "${REGISTRY_CA}" "${IMAGE_ROOT}${REGISTRY_CA}"
cp -fv "${CERTS_D}/nvcr.io/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/nvcr.io/"
cp -fv "${CERTS_D}/_default/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/_default/"
cp -fv "${CERTS_D}/docker.io/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/docker.io/"
cm-chroot-sw-img "${IMAGE_ROOT}" << EOF
update-ca-certificates
exit
EOF

After you install the k8s-user cluster, repeat this software-image procedure and the following imageupdate, containerd restart, and validation steps with NODE_CATEGORY="k8s-user" and the configuration overlay that applies to that category.

Apply the changes#

After you exit the image, push the updated image to the nodes and restart containerd:

# Push the updated software image to the nodes in the category
cmsh -c "device; imageupdate -w -c ${NODE_CATEGORY} --wait"

# Run the following on the head node
systemctl restart containerd
update-ca-certificates

# Restart containerd and update certs on the nodes in the category
pdsh -g "category=${NODE_CATEGORY}" 'systemctl restart containerd'
pdsh -g "category=${NODE_CATEGORY}" 'update-ca-certificates'

After the image update and the containerd restart, confirm that a pull that uses an upstream image name reaches the local registry:

pdsh -g "category=${NODE_CATEGORY}" 'crictl pull registry.k8s.io/pause:3.10.1 2>/dev/null'

Warning

Containerd reads a registry-specific hosts.toml file in preference to _default/hosts.toml. Before you rely on the default mirror, inspect ${CERTS_D}/<registry>/hosts.toml on the head node and in every node software image. Update any existing registry-specific file to point to the local registry. Otherwise, pulls for that registry can continue to use an external mirror.

RKE2 example#

Use a registries.yaml configuration such as:

# /etc/rancher/rke2/registries.yaml
mirrors:
  nvcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  docker.io:
    endpoint:
      - https://airgap-registry.com:5443
  ghcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  quay.io:
    endpoint:
      - https://airgap-registry.com:5443
  reg.kyverno.io:
    endpoint:
      - https://airgap-registry.com:5443
  registry.k8s.io:
    endpoint:
      - https://airgap-registry.com:5443
configs:
  airgap-registry.com:5443:
    tls:
      ca_file: "/etc/ssl/certs/airgap-registry.pem"

Replace airgap-registry.com:5443 with the host and port of your registry, and replace /etc/ssl/certs/airgap-registry.pem with the CA certificate path on each node. The key under configs must match the endpoint host and port so that RKE2 applies the TLS configuration to the mirror.

Containerd example#

Set the following variables to match your environment:

REGISTRY_HOST="airgap-registry.com"   # Hostname of your local registry
REGISTRY_PORT="5443"                  # Port of your local registry
REGISTRY_CA="/etc/ssl/certs/airgap-registry.pem"
CERTS_D="/etc/containerd/certs.d"

Create the certs.d directory and back up the containerd configuration:

sudo mkdir -p "${CERTS_D}"
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak

Run containerd --version to identify the installed major version. In /etc/containerd/config.toml, add or update the registry section for that version.

For containerd 1.x, use:

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

For containerd 2.x, use:

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

Add the mirror host entries:

# Mirror for NVIDIA registry
sudo mkdir -p "${CERTS_D}/nvcr.io"
sudo tee "${CERTS_D}/nvcr.io/hosts.toml" >/dev/null << EOF
server = "https://nvcr.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# Default mirror for all other registries
sudo mkdir -p "${CERTS_D}/_default"
sudo tee "${CERTS_D}/_default/hosts.toml" >/dev/null << EOF
[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

sudo systemctl restart containerd

The example assumes that your registry uses a self-signed or enterprise CA certificate. If a CA that the node already trusts signed the registry certificate, omit the REGISTRY_CA assignment and the ca entries.