Configuring Registry Mirroring#

Note

Local registry mirroring is supported in NMC 2.3.1 but requires manual configuration. Automated registry mirroring with support for both x86 and ARM container images is planned for a future release. This page will be updated as availability is confirmed.

Registry mirroring is required for every air-gapped NMC deployment. The cluster has no route to the upstream registries, so you must configure the container runtime on every node to redirect image pulls to your local registry. Otherwise, image pulls fail and component installations stall. Complete the configuration on this page before you install any NMC component. If pods enter ImagePullBackOff during an air-gapped installation, registry mirroring is almost always the cause.

So that NMC can pull images without changes to the Helm charts, mirror the following upstream registries:

  • nvcr.io

  • docker.io

  • ghcr.io

  • quay.io

  • reg.kyverno.io

  • registry.k8s.io

The simplest approach is to configure a default mirror that redirects all image pulls to your local registry, as shown in the following BCM, RKE2, and containerd examples. Use the example that matches your environment.

BCM example (most deployments)#

Deploy the local container registry before you configure mirroring. The cm-container-registry-setup wizard, described in Step 4: Set up the container registry, deploys the registry and generates its CA certificate at /cm/local/apps/containerd/var/etc/certs.d/<registry-host>:<port>/ca.crt. If you have not yet deployed the registry, the steps that copy that certificate into the node trust store fail with cp: cannot stat.

Set the following variables to match your environment. The remaining commands reference these variables, so this is the only block you edit:

REGISTRY_HOST="master.cm.cluster"        # Hostname of your local registry
REGISTRY_PORT="5000"                     # Port of your local registry
NODE_CATEGORY="k8s-admin"                # BCM category of the nodes to configure

CERTS_D="/cm/local/apps/containerd/var/etc/certs.d"
REGISTRY_CA="${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt"

Apply these instructions to the k8s-admin category. If the k8s-user cluster is already installed, repeat the configuration-overlay, software-image, and node steps for the k8s-user category. If it is not installed, apply those steps after the cluster is deployed.

Before editing the software image, update the BCM configuration overlay that provides the generic::containerd role. BCM can regenerate containerd configuration during imageupdate. Set both environment values in the kube-k8s-admin-master overlay:

cmsh
configurationoverlay
use kube-k8s-admin-master
roles
use generic::containerd
environments
use registry_mirror
set value https://master.cm.cluster:5000
environments
use registry_mirror_ca_entry
set value "ca = '/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt'"
commit
quit

Replace master.cm.cluster:5000 with the host and port of your registry, and kube-k8s-admin-master with the overlay that applies to your category. Do not enclose the registry_mirror URL in quotation marks, because cmsh stores the quotation marks as part of the value.

You can also set both values with the following commands:

cmsh -c "configurationoverlay; use kube-k8s-admin-master; roles; use generic::containerd; environments; use registry_mirror; set value https://master.cm.cluster:5000; commit"
cmsh -c "configurationoverlay; use kube-k8s-admin-master; roles; use generic::containerd; environments; use registry_mirror_ca_entry; set value \"ca = '/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt'\"; commit"

On the BCM head node, confirm that containerd imports the additional configuration directory:

cat /cm/local/apps/containerd/var/etc/config.toml

The output is similar to the following. The version value depends on the containerd release. The line that matters is imports, which points to the directory that holds all other containerd configuration:

version = 4
imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"]

Add the mirror configuration#

Apply the mirror configuration in two places: directly on the head node and inside the software image for each category whose nodes run containers. Create the files on the head node, and then copy them into the software image.

First, run the configuration commands directly on the head node:

# Upstream NVIDIA registry
mkdir -p "${CERTS_D}/nvcr.io"
tee "${CERTS_D}/nvcr.io/hosts.toml" >/dev/null << EOF
server = "https://nvcr.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# For all other registries
mkdir -p "${CERTS_D}/_default"
tee "${CERTS_D}/_default/hosts.toml" >/dev/null << EOF
[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# Override any existing Docker Hub mirror
mkdir -p "${CERTS_D}/docker.io"
tee "${CERTS_D}/docker.io/hosts.toml" >/dev/null << EOF
server = "https://registry-1.docker.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF

# Trust the CA certificate
cp "${REGISTRY_CA}" "/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
update-ca-certificates

Copy the registry CA and mirror files into the software image used by the category that NODE_CATEGORY identifies, then update the CA bundle in that image:

set -e
SOFTWARE_IMAGE=$(cmsh -c "category; use ${NODE_CATEGORY}; get softwareimage")
echo "${SOFTWARE_IMAGE}"
IMAGE_ROOT="/cm/images/${SOFTWARE_IMAGE}"
mkdir -p "${IMAGE_ROOT}${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}" \
  "${IMAGE_ROOT}/usr/local/share/ca-certificates"
cp -fv "${REGISTRY_CA}" "${IMAGE_ROOT}${REGISTRY_CA}"
cp -fv "${REGISTRY_CA}" \
  "${IMAGE_ROOT}/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
mkdir -p "${IMAGE_ROOT}${CERTS_D}/nvcr.io" \
  "${IMAGE_ROOT}${CERTS_D}/_default" \
  "${IMAGE_ROOT}${CERTS_D}/docker.io"
cp -fv "${CERTS_D}/nvcr.io/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/nvcr.io/"
cp -fv "${CERTS_D}/_default/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/_default/"
cp -fv "${CERTS_D}/docker.io/hosts.toml" "${IMAGE_ROOT}${CERTS_D}/docker.io/"
cm-chroot-sw-img "${IMAGE_ROOT}" << EOF
update-ca-certificates
exit
EOF

Before you repeat the steps for the k8s-user category, set NODE_CATEGORY="k8s-user" and repeat the configuration-overlay procedure with the overlay that applies to that category. Then repeat this software-image procedure and the following imageupdate, containerd restart, and validation steps.

Apply the changes#

After you exit the image, push the updated image to the nodes and restart containerd:

# Push the updated software image to the nodes in the category
cmsh -c "device; imageupdate -w -c ${NODE_CATEGORY} --wait"

# Run the following on the head node
systemctl restart containerd
update-ca-certificates

# Restart containerd and update certs on the nodes in the category
pdsh -g "category=${NODE_CATEGORY}" 'systemctl restart containerd'
pdsh -g "category=${NODE_CATEGORY}" 'update-ca-certificates'

After the image update and the containerd restart, confirm that a pull that uses an upstream image name reaches the local registry:

pdsh -g "category=${NODE_CATEGORY}" 'crictl pull registry.k8s.io/pause:3.10.1 2>/dev/null'

Warning

Containerd reads a registry-specific hosts.toml file in preference to _default/hosts.toml. Before you rely on the default mirror, inspect <certs.d>/<registry>/hosts.toml on the head node and in every node image, and point any existing registry-specific file at the local registry. Otherwise, pulls for that registry can continue to use an external mirror.

RKE2 example#

On every server and agent node, create /etc/rancher/rke2/registries.yaml with a configuration such as the following.

RKE2 reads this file directly and does not expand shell variables, so replace the following values throughout the example:

Value in the example

Replace with

airgap-registry.com:5443

The hostname and port of your local registry. This value appears in every endpoint entry and again as the key under configs. The configs key must match the endpoint host and port exactly, or the TLS settings are not applied to that mirror.

/etc/ssl/certs/airgap-registry.pem

The path on each node to the CA certificate for your registry.

 
# /etc/rancher/rke2/registries.yaml
mirrors:
  nvcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  docker.io:
    endpoint:
      - https://airgap-registry.com:5443
  ghcr.io:
    endpoint:
      - https://airgap-registry.com:5443
  quay.io:
    endpoint:
      - https://airgap-registry.com:5443
  reg.kyverno.io:
    endpoint:
      - https://airgap-registry.com:5443
  registry.k8s.io:
    endpoint:
      - https://airgap-registry.com:5443
configs:
  airgap-registry.com:5443:
    tls:
      ca_file: "/etc/ssl/certs/airgap-registry.pem"

After you create the file, restart RKE2 on each node. Run systemctl restart rke2-server on server nodes and systemctl restart rke2-agent on agent nodes.

containerd example#

Use this example on nodes that BCM does not manage. On a BCM-managed node, /etc/containerd is a symbolic link to /cm/local/apps/containerd/var/etc and config_path is already set in conf.d. Use the BCM example on those nodes, and do not modify config.toml.

Set the following variables to match your environment. The remaining commands reference these variables, so this is the only block you edit:

REGISTRY_HOST="airgap-registry.com"   # Hostname of your local registry
REGISTRY_PORT="5443"                  # Port of your local registry

CERTS_D="/etc/containerd/certs.d"

Create the certs.d directory and back up the containerd configuration:

sudo mkdir -p "${CERTS_D}"
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak

Run containerd --version to identify the installed major version. In /etc/containerd/config.toml, add or update the registry section for that version.

For containerd 1.x, use:

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

For containerd 2.x, use:

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

Add the mirror host entries:

# Mirror for NVIDIA registry
sudo mkdir -p "${CERTS_D}/nvcr.io"
sudo tee "${CERTS_D}/nvcr.io/hosts.toml" >/dev/null << EOF
server = "https://nvcr.io"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
EOF

# Default mirror for all other registries
sudo mkdir -p "${CERTS_D}/_default"
sudo tee "${CERTS_D}/_default/hosts.toml" >/dev/null << EOF
[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
EOF

sudo systemctl restart containerd

Most air-gapped registries use a self-signed or enterprise CA certificate. If yours does, set a REGISTRY_CA variable alongside the others:

REGISTRY_CA="/etc/ssl/certs/airgap-registry.pem"

Add the ca entry to each [host] block:

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"

Omit ca only when a CA that the node already trusts signed the registry certificate.