Configuring Registry Mirroring#

Note

Local registry mirroring requires manual configuration. Automated registry mirroring with support for both x86 and ARM container images is planned for a future release. This page will be updated as availability is confirmed.

Registry mirroring is required for every air-gapped NMC deployment. The cluster has no route to the upstream registries, so you must configure the container runtime on every node to redirect image pulls to your local registry. Otherwise, image pulls fail and component installations stall. Complete the configuration on this page before you install any NMC component. If pods enter ImagePullBackOff during an air-gapped installation, registry mirroring is almost always the cause.

Mirror the following upstream registries so that NMC can pull images without changing chart image references:

nvcr.io
docker.io
ghcr.io
quay.io
reg.kyverno.io
registry.k8s.io

Configure a mirror for each upstream registry that your deployment uses. The following BCM and RKE2 Examples cover all six registries listed here. Use the example that matches your environment.

A containerd mirror changes the registry host but preserves the repository path and tag. For example, a pull for nvcr.io/<namespace>/<image>:<tag> asks the local mirror for <namespace>/<image>:<tag>. The image-import process must populate that same repository path and tag in the local registry. If it imports the image at the registry root instead, fix the import mapping; the mirror configuration cannot remove <namespace>/ from a request. Validate at least one image with a nested repository path through its original upstream name before installing a component. A successful pull of an image at the registry root does not verify that nested paths are available.

BCM Example (Most Deployments)#

Deploy the local container registry before you configure mirroring. The cm-container-registry-setup wizard, described in Step 4: Set up the container registry, deploys the registry and generates its CA certificate at /cm/local/apps/containerd/var/etc/certs.d/<registry-host>:<port>/ca.crt. If you have not yet deployed the registry, the steps that copy that certificate into the node trust store fail with cp: cannot stat.

The following example uses the BCM local registry at master.cm.cluster:5000. Change the address if your registry runs elsewhere. Run the complete script in one shell on the BCM head node. It reads cmsh JSON with jq to find the containerd overlays and their assigned categories. For overlays assigned directly to nodes, cmsh get nodes expands node ranges before the script looks up each node’s category. It then reads each category’s software image from cmsh.

On the BCM head node, confirm that containerd imports the additional configuration directory:

cat /cm/local/apps/containerd/var/etc/config.toml

The output is similar to the following. The version value depends on the containerd release. The line that matters is imports, which points to the directory that holds all other containerd configuration:

version = 4
imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"]

Add the Mirror Configuration#

The script checks that the registry CA and every category software image exist. It creates a hosts.toml file for each upstream registry on the BCM head node. It adds missing mirror entries to existing containerd overlays, updates existing entries, and prepares every category software image. It updates nodes in populated categories, but restarts containerd and checks image pulls only for categories with a containerd overlay. This prepares categories such as k8s-user and dgx-gb300-k8s before their clusters and overlays are installed.

Setting the local registry as server prevents fallback to an upstream registry when a local image is missing.

BCM can regenerate containerd configuration during imageupdate, so the script updates existing overlays before updating images.

The final loop waits for containerd and checks an image pull on each populated category with an existing containerd overlay. Import registry.k8s.io/pause:3.10.1 into the local registry before running the script, or change TEST_IMAGE to an image that you have imported. BCM can create the registry directory in a software image as an absolute symbolic link. The script writes the CA to the link’s target inside the image.

If both k8s-admin and k8s-user are already deployed, run the script once to configure both clusters. Rerun it after you deploy any additional Kubernetes cluster, and before you install components on that cluster.

On the BCM head node, create a script that you can run again after each cluster deployment. Change REGISTRY_HOST, REGISTRY_PORT, or TEST_IMAGE in the file if your deployment uses different values.

Registry Mirroring Script

cat > /root/configure-registry-mirroring.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
REGISTRY_HOST="master.cm.cluster"
REGISTRY_PORT="5000"
CERTS_D="/cm/local/apps/containerd/var/etc/certs.d"
REGISTRY_CA="${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}/ca.crt"
TEST_IMAGE="registry.k8s.io/pause:3.10.1"
UPSTREAM_REGISTRIES=(
  nvcr.io
  docker.io
  ghcr.io
  quay.io
  reg.kyverno.io
  registry.k8s.io
)

OVERLAY_JSON=$(cmsh --color=no -c \
  "configurationoverlay; list -f name,categories,roles -d {}")
DISCOVERY=$(jq -nr --argjson overlays "${OVERLAY_JSON}" '
    def names:
      (if type == "array" then .[] else (. // "" | split(",")[]) end)
      | gsub("^[[:space:]]+|[[:space:]]+$"; "")
      | select(length > 0);
    [$overlays[] | select((.roles | names) == "generic::containerd")]
    as $selected
    | ([$selected[] | .["name (key)"] // .name] | unique[] | "overlay\t\(.)"),
      ([$selected[] | .categories | names] | unique[] | "category\t\(.)")
  ')

CONTAINERD_OVERLAYS=()
NODE_CATEGORIES=()
while IFS=$'\t' read -r KIND NAME; do
  case "${KIND}" in
    overlay) CONTAINERD_OVERLAYS+=("${NAME}") ;;
    category) NODE_CATEGORIES+=("${NAME}") ;;
  esac
done <<< "${DISCOVERY}"
for OVERLAY_NAME in "${CONTAINERD_OVERLAYS[@]}"; do
  OVERLAY_NODES=$(cmsh --color=no -c \
    "configurationoverlay; use ${OVERLAY_NAME}; get nodes")
  while IFS= read -r NODE_NAME; do
    [[ -n "${NODE_NAME}" ]] || continue
    NODE_CATEGORY=$(cmsh --color=no -c \
      "device; use ${NODE_NAME}; get category")
    [[ -n "${NODE_CATEGORY}" ]] && NODE_CATEGORIES+=("${NODE_CATEGORY}")
  done <<< "${OVERLAY_NODES}"
done
declare -A CATEGORY_SET=()
for NODE_CATEGORY in "${NODE_CATEGORIES[@]}"; do
  CATEGORY_SET["${NODE_CATEGORY}"]=1
done
NODE_CATEGORIES=("${!CATEGORY_SET[@]}")
CATEGORY_ROWS=$(cmsh --color=no -c \
  "category; list -f name,softwareimage,nodes -d {}" | jq -r '
    .[] | select((.softwareimage // "") != "")
    | [(."name (key)" // .name), .softwareimage, (.nodes | tonumber? // 0)]
    | @tsv
  ')
ALL_CATEGORIES=()
CATEGORY_IMAGES=()
CATEGORY_NODE_COUNTS=()
while IFS=$'\t' read -r CATEGORY_NAME IMAGE_NAME NODE_COUNT; do
  [[ -n "${CATEGORY_NAME}" ]] || continue
  ALL_CATEGORIES+=("${CATEGORY_NAME}")
  CATEGORY_IMAGES+=("${IMAGE_NAME}")
  CATEGORY_NODE_COUNTS+=("${NODE_COUNT}")
done <<< "${CATEGORY_ROWS}"
if ((${#ALL_CATEGORIES[@]} == 0)); then
  echo "No category software images found in cmsh" >&2
  exit 1
fi
test -f "${REGISTRY_CA}" || { echo "Missing ${REGISTRY_CA}" >&2; exit 1; }
for INDEX in "${!ALL_CATEGORIES[@]}"; do
  test -d "/cm/images/${CATEGORY_IMAGES[INDEX]}" || {
    echo "Missing software image for ${ALL_CATEGORIES[INDEX]}: ${CATEGORY_IMAGES[INDEX]}" >&2
    exit 1
  }
  printf 'Category %s: %s (%s nodes)\n' \
    "${ALL_CATEGORIES[INDEX]}" "${CATEGORY_IMAGES[INDEX]}" \
    "${CATEGORY_NODE_COUNTS[INDEX]}"
done
printf 'Existing containerd overlays: %s\n' "${CONTAINERD_OVERLAYS[*]:-none}"

for UPSTREAM_REGISTRY in "${UPSTREAM_REGISTRIES[@]}"; do
  mkdir -p "${CERTS_D}/${UPSTREAM_REGISTRY}"
  tee "${CERTS_D}/${UPSTREAM_REGISTRY}/hosts.toml" >/dev/null << EOF
server = "https://${REGISTRY_HOST}:${REGISTRY_PORT}"

[host."https://${REGISTRY_HOST}:${REGISTRY_PORT}"]
  capabilities = ["pull", "resolve"]
  ca = "${REGISTRY_CA}"
EOF
done

mkdir -p /usr/local/share/ca-certificates
cp "${REGISTRY_CA}" "/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
update-ca-certificates
for OVERLAY_NAME in "${CONTAINERD_OVERLAYS[@]}"; do
  ENV_JSON=$(cmsh --color=no -c "configurationoverlay; use ${OVERLAY_NAME}; roles; use generic::containerd; environments; list -f name -d {}")
  if jq -e --arg name registry_mirror 'any(.[]; (."name (key)" // .name // "") == $name)' <<< "${ENV_JSON}" >/dev/null; then
    MIRROR_ACTION=use
  else
    MIRROR_ACTION=add
  fi
  if jq -e --arg name registry_mirror_ca_entry 'any(.[]; (."name (key)" // .name // "") == $name)' <<< "${ENV_JSON}" >/dev/null; then
    CA_ACTION=use
  else
    CA_ACTION=add
  fi
  cmsh --color=no -c "configurationoverlay; use ${OVERLAY_NAME}; roles; use generic::containerd; environments; ${MIRROR_ACTION} registry_mirror; set value https://${REGISTRY_HOST}:${REGISTRY_PORT}; commit"
  cmsh --color=no -c "configurationoverlay; use ${OVERLAY_NAME}; roles; use generic::containerd; environments; ${CA_ACTION} registry_mirror_ca_entry; set value \"ca = '${REGISTRY_CA}'\"; commit"
  ACTUAL_MIRROR=$(cmsh --color=no -c "configurationoverlay; use ${OVERLAY_NAME}; roles; use generic::containerd; environments; use registry_mirror; get value")
  ACTUAL_CA=$(cmsh --color=no -c "configurationoverlay; use ${OVERLAY_NAME}; roles; use generic::containerd; environments; use registry_mirror_ca_entry; get value")
  if [[ "${ACTUAL_MIRROR}" != "https://${REGISTRY_HOST}:${REGISTRY_PORT}" ||
        "${ACTUAL_CA}" != "ca = '${REGISTRY_CA}'" ]]; then
    echo "Containerd mirror values were not saved in ${OVERLAY_NAME}" >&2
    exit 1
  fi
done

declare -A PREPARED_IMAGES=()
declare -A NODE_COUNT_BY_CATEGORY=()
for INDEX in "${!ALL_CATEGORIES[@]}"; do
  NODE_COUNT_BY_CATEGORY["${ALL_CATEGORIES[INDEX]}"]="${CATEGORY_NODE_COUNTS[INDEX]}"
done
for INDEX in "${!ALL_CATEGORIES[@]}"; do
  NODE_CATEGORY="${ALL_CATEGORIES[INDEX]}"
  NODE_IMAGE="${CATEGORY_IMAGES[INDEX]}"
  IMAGE_ROOT="/cm/images/${NODE_IMAGE}"
  if [[ -z "${PREPARED_IMAGES[$NODE_IMAGE]:-}" ]]; then
    mkdir -p "${IMAGE_ROOT}/usr/local/share/ca-certificates"
    cp -fv "${REGISTRY_CA}" \
      "${IMAGE_ROOT}/usr/local/share/ca-certificates/${REGISTRY_HOST}-registry.crt"
    IMAGE_REGISTRY_DIR="${IMAGE_ROOT}${CERTS_D}/${REGISTRY_HOST}:${REGISTRY_PORT}"
    if [[ -L "${IMAGE_REGISTRY_DIR}" ]]; then
      LINK_TARGET=$(readlink "${IMAGE_REGISTRY_DIR}")
      if [[ "${LINK_TARGET}" == /* ]]; then
        IMAGE_CA="${IMAGE_ROOT}${LINK_TARGET}/ca.crt"
      else
        IMAGE_CA="$(realpath -m "$(dirname "${IMAGE_REGISTRY_DIR}")/${LINK_TARGET}")/ca.crt"
      fi
    else
      IMAGE_CA="${IMAGE_REGISTRY_DIR}/ca.crt"
    fi
    mkdir -p "$(dirname "${IMAGE_CA}")"
    cp -fv "${REGISTRY_CA}" "${IMAGE_CA}"
    for UPSTREAM_REGISTRY in "${UPSTREAM_REGISTRIES[@]}"; do
      mkdir -p "${IMAGE_ROOT}${CERTS_D}/${UPSTREAM_REGISTRY}"
      cp -fv "${CERTS_D}/${UPSTREAM_REGISTRY}/hosts.toml" \
        "${IMAGE_ROOT}${CERTS_D}/${UPSTREAM_REGISTRY}/"
    done
    cm-chroot-sw-img "${IMAGE_ROOT}" << EOF
update-ca-certificates
exit
EOF
    PREPARED_IMAGES["${NODE_IMAGE}"]=1
  fi

  if (( CATEGORY_NODE_COUNTS[INDEX] > 0 )); then
    cmsh --color=no -c "device; imageupdate -w -c ${NODE_CATEGORY} --wait"
    pdsh -g "category=${NODE_CATEGORY}" 'update-ca-certificates'
    if [[ -n "${CATEGORY_SET[$NODE_CATEGORY]:-}" ]]; then
      pdsh -g "category=${NODE_CATEGORY}" 'systemctl restart containerd'
    fi
  fi
done

# Apply the head-node changes after all node images are updated.
systemctl restart containerd
for NODE_CATEGORY in "${NODE_CATEGORIES[@]}"; do
  [[ "${NODE_COUNT_BY_CATEGORY[$NODE_CATEGORY]:-0}" != 0 ]] || continue
  printf '\nTest image pull on category: %s\n' "${NODE_CATEGORY}"
  pdsh -g "category=${NODE_CATEGORY}" \
    'attempt=0; until crictl --runtime-endpoint unix:///run/containerd/containerd.sock info >/dev/null 2>&1; do attempt=$((attempt + 1)); [ "$attempt" -lt 30 ] || exit 1; sleep 2; done'
  pdsh -g "category=${NODE_CATEGORY}" \
    "crictl --runtime-endpoint unix:///run/containerd/containerd.sock pull ${TEST_IMAGE}" | dshbak -c
done
BASH

Run the script on the BCM head node:

chmod 700 /root/configure-registry-mirroring.sh
/root/configure-registry-mirroring.sh

Note

Required rerun after cluster deployment. You must rerun the complete script after deploying each additional Kubernetes cluster and before installing NMC components on it. The first run prepares its software image, but its containerd overlays do not exist until cluster deployment. The rerun configures those overlays, restarts containerd on the new cluster’s nodes, and validates the image pull.

/root/configure-registry-mirroring.sh

If your deployment pulls images from another upstream registry, add its host name to UPSTREAM_REGISTRIES and import its images into the local registry before rerunning the script.

Containerd reads a registry-specific hosts.toml file in preference to _default/hosts.toml. For any additional registry, inspect existing registry-specific files on the BCM head node and in the node images before relying on the default mirror.

RKE2 Example#

On every server and agent node, create /etc/rancher/rke2/registries.yaml with a configuration such as the following.

RKE2 reads this file directly and does not expand shell variables, so replace the following values throughout the example:

Value in the example	Replace with
`airgap-registry.com:5443`	The hostname and port of your local registry. This value appears in every `endpoint` entry and again as the key under `configs`. The `configs` key must match the endpoint host and port exactly, or the TLS settings are not applied to that mirror.
`/etc/ssl/certs/airgap-registry.pem`	The path on each node to the CA certificate for your registry.

After you create the file, restart RKE2 on each node. Run systemctl restart rke2-server on server nodes and systemctl restart rke2-agent on agent nodes.

containerd Example#

Use this example on nodes that BCM does not manage. On a BCM-managed node, /etc/containerd is a symbolic link to /cm/local/apps/containerd/var/etc and config_path is already set in conf.d. Use the BCM example on those nodes, and do not modify config.toml.

Set the following variables to match your environment. The remaining commands reference these variables, so this is the only block you edit:

REGISTRY_HOST="airgap-registry.com"   # Hostname of your local registry
REGISTRY_PORT="5443"                  # Port of your local registry

CERTS_D="/etc/containerd/certs.d"

Configure containerd to use the certs.d directory and add mirror host entries:

Most air-gapped registries use a self-signed or enterprise CA certificate. If yours does, set a REGISTRY_CA variable alongside the others and add a ca entry to each [host] block, as the BCM example does. Omit ca only when a CA that the node already trusts signed the registry certificate.