Image Mirroring

All required self-hosted NVCF artifacts (see self-hosted-artifact-manifest) must be available to be pulled by pods in your Kubernetes cluster. The deployment bundles are nvcf-self-managed-stack for the control plane, nvcf-compute-plane-stack for the compute plane, and nvcf-observability-stack for standalone shared observability. This page provides examples on how to pull artifacts from NGC and push them to your desired registry.

Mirroring Workflow

Use this page to mirror NVCF artifacts into the registry your cluster can access. The examples below show the commands for pulling from NGC and pushing to a private registry such as ECR, Harbor, or another OCI-compatible registry.

Prerequisites

NVCF artifacts are available from the public nvidia/nvcf namespace in NGC. You can mirror these artifacts using a Personal API key from any NGC organization. Keep nvidia/nvcf as the source namespace in the pull commands, regardless of the organization used to create your key.

1. Navigate to NGC API Keys and select your organization.

2. Create a Personal API key with the required scopes to pull entities.

3. Set the NGC API key as an environment variable for use in subsequent commands:

   export NGC_API_KEY="nvapi-xxxxxxxxxxxxx"  # Replace with your NGC API key

LLS-Specific Artifacts

If you plan to deploy Low Latency Streaming (LLS), you must mirror the following additional artifacts beyond the core NVCF control plane:

Container Images:

• streaming-proxy - Streaming Proxy container for streaming

Helm Charts:

• gdn-streaming - GDN Streaming Proxy Helm chart

Optional (for streaming workloads):

• Streaming application images (e.g., usd-composer)

See self-hosted-lls-installation for LLS deployment instructions.

Pulling Artifacts from NGC

Pulling Images

# Pull for amd64 clusters (most common)
docker pull --platform linux/amd64 <image>
# Pull for arm64 clusters
docker pull --platform linux/arm64 <image>

1. Login using the Personal API key you have generated in the previous step:

   docker login nvcr.io -u '$oauthtoken' -p <NGC_API_KEY>

2. Pull the image (specify platform matching your target cluster):

   # For amd64 clusters (most EKS, GKE, AKS clusters)
   docker pull --platform linux/amd64 nvcr.io/nvidia/nvcf/nats-box:0.19.7-nonroot
   # For arm64 clusters (Graviton-based EKS, etc.)
   docker pull --platform linux/arm64 nvcr.io/nvidia/nvcf/nats-box:0.19.7-nonroot

Pulling Helm Charts

The example below uses the selected NVCA Operator chart. For another chart, set its name, version and pull reference from the artifact manifest. The commands save one archive for the registry-specific push steps below. Repeat for each chart that you need to mirror.

CHART_NAME="helm-nvca-operator"
CHART_VERSION="1.37.1"
CHART_REFERENCE="nvcf/helm-nvca-operator"
CHART_ARCHIVE="${CHART_NAME}-${CHART_VERSION}.tgz"
(
set +x
set -euo pipefail
umask 077
HELM_MIRROR_TMP="$(mktemp -d)"
trap 'rm -rf -- "${HELM_MIRROR_TMP}"' EXIT
export HELM_REPOSITORY_CONFIG="${HELM_MIRROR_TMP}/repositories.yaml"
export HELM_REPOSITORY_CACHE="${HELM_MIRROR_TMP}/cache"
: "${NGC_API_KEY:?Set a scoped NGC_API_KEY}"
printf '%s\n' "${NGC_API_KEY}" |
  helm repo add nvcf https://helm.ngc.nvidia.com/nvidia/nvcf \
    --username '$oauthtoken' --password-stdin
helm repo update nvcf
helm pull "${CHART_REFERENCE}" --version "${CHART_VERSION}"
)

The temporary Helm configuration keeps these credentials separate from your existing repository configuration. The downloaded archive remains in the current directory.

Other Repository-based Helm Charts (Non-OCI)

The GPU Operator and the Omniverse OVDC, OVCC, storage-service, and discovery-service charts are also available from traditional Helm repositories rather than OCI registries. Use the chart versions and repositories selected for your release. The public repository below applies to published artifacts; for prerelease artifacts, use the repository and credentials supplied with that release.

OVDC uses chart ovderivedcache; OVCC uses chart ovcontentcache. See the OVDC deployment guide and OVCC deployment guide. Use an NGC_API_KEY with access to the nvidia/omniverse chart repository. The examples use OVDC chart/image 6.0.1 and OVCC chart 4.0.4 with image usd-content-cache:3.0.2. Chart and image versions are independent. Retain image.tag: "3.0.2" in the OVCC Helm values; chart 4.0.4 defaults to image 3.0.1.

Use a scoped, short-lived key. These Bash examples keep Helm credentials in temporary configuration and remove it when the subshell exits, including on failure. Existing Helm repository entries and registry credentials are preserved.

(
set +x
set -euo pipefail
umask 077
HELM_MIRROR_TMP="$(mktemp -d)"
trap 'rm -rf -- "${HELM_MIRROR_TMP}"' EXIT
export HELM_REPOSITORY_CONFIG="${HELM_MIRROR_TMP}/repositories.yaml"
export HELM_REPOSITORY_CACHE="${HELM_MIRROR_TMP}/cache"
export HELM_REGISTRY_CONFIG="${HELM_MIRROR_TMP}/registry.json"
: "${NGC_API_KEY:?Set a scoped NGC_API_KEY}"
# Add the NVIDIA Helm repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
printf '%s\n' "${NGC_API_KEY}" |
  helm repo add omniverse https://helm.ngc.nvidia.com/nvidia/omniverse \
    --username '$oauthtoken' --password-stdin
# Update repositories
helm repo update
# Pull charts (downloads as .tgz files)
OVDC_CHART_VERSION="${OVDC_CHART_VERSION:-6.0.1}"
helm pull nvidia/gpu-operator --version 25.3.1
helm pull omniverse/ovderivedcache --version "${OVDC_CHART_VERSION}"
helm pull omniverse/ovcontentcache --version 4.0.4
helm pull omniverse/storage-service --version 1.0.2
helm pull omniverse/discovery-service --version 2.3.8
)

Converting Non-OCI Charts for ECR

To push repository-based Helm charts to Amazon ECR (which requires OCI format), you must convert them:

(
set +x
set -euo pipefail
umask 077
HELM_MIRROR_TMP="$(mktemp -d)"
trap 'rm -rf -- "${HELM_MIRROR_TMP}"' EXIT
export HELM_REPOSITORY_CONFIG="${HELM_MIRROR_TMP}/repositories.yaml"
export HELM_REPOSITORY_CACHE="${HELM_MIRROR_TMP}/cache"
export HELM_REGISTRY_CONFIG="${HELM_MIRROR_TMP}/registry.json"
: "${NGC_API_KEY:?Set a scoped NGC_API_KEY}"
# Pull the chart from the traditional repository
# Select the version and repository for your release, as described above.
OVDC_CHART_VERSION="${OVDC_CHART_VERSION:-6.0.1}"
printf '%s\n' "${NGC_API_KEY}" |
  helm repo add omniverse https://helm.ngc.nvidia.com/nvidia/omniverse \
    --username '$oauthtoken' --password-stdin
helm repo update
helm pull omniverse/ovderivedcache --version "${OVDC_CHART_VERSION}"
# Login to ECR
aws ecr get-login-password --region us-east-1 | \
  helm registry login --username AWS --password-stdin <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com
# Create ECR repository for the chart (include your repository prefix)
aws ecr create-repository --repository-name nvcf-self-hosted/charts/ovderivedcache --region us-east-1
# Push the .tgz file as an OCI artifact (include repository prefix)
helm push "ovderivedcache-${OVDC_CHART_VERSION}.tgz" oci://<aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/nvcf-self-hosted/charts
)

Obtain stack packages

Use Consume an NVCF release for the selected stack packages or an umbrella archive. That guide owns the download commands. Continue here when charts or images need a mirror.

Run that guide in a separate terminal. It enables fail-fast mode and changes the working directory. Push charts from this terminal so CHART_NAME, CHART_ARCHIVE and the downloaded archive remain available.

Downloading nvcf-cli

The nvcf-cli is a command-line interface for managing NVIDIA Cloud Functions in self-hosted deployments. Install and configure the NGC CLI with your Personal API key before downloading it.

Download and extract:

# Set the version
export VERSION="1.16.8"
# Set your platform (linux-amd64, linux-arm64, darwin-amd64, darwin-arm64, windows-amd64)
export PLATFORM="linux-amd64"
ngc registry resource download-version "nvidia/nvcf/nvcf-cli:${VERSION}"
tar -xzf nvcf-cli_v${VERSION}/${PLATFORM}/nvcf-cli-${PLATFORM}-${VERSION}.tar.gz
mv nvcf-cli-${PLATFORM}-${VERSION} nvcf-cli
chmod +x nvcf-cli/nvcf-cli

The extracted directory contains:

• nvcf-cli - The CLI binary
• .nvcf-cli.yaml.template - Configuration template
• examples/ - Sample configuration files for different environments
• USAGE-GUIDE.md - Detailed usage documentation

See self-hosted-cli for detailed configuration instructions

Pushing to Your Registry

Example: Pushing to Amazon ECR

This example assumes you’re configured and authenticated using the AWS CLI.

aws sts get-caller-identity --query Account --output text

global:
  image:
    registry: <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com
    repository: nvcf-self-hosted

Initial Setup

# Set your repository prefix (must match global.image.repository in your environment config)
REPO_PREFIX="nvcf-self-hosted"
# Login to AWS ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com

Push an Image to ECR

# Create ECR repository with the full path (including prefix)
aws ecr create-repository --repository-name ${REPO_PREFIX}/nats-box --region us-east-1
# Tag the image for ECR (include repository prefix in path)
docker tag nvcr.io/nvidia/nvcf/nats-box:0.19.7-nonroot \
  <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/${REPO_PREFIX}/nats-box:0.19.7-nonroot
# Push to ECR
docker push <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/${REPO_PREFIX}/nats-box:0.19.7-nonroot

Push a Helm Chart to ECR

Use the archive and CHART_NAME from Pulling Helm Charts.

# Login to AWS ECR with Helm
aws ecr get-login-password --region us-east-1 | \
  helm registry login --username AWS --password-stdin <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com
# Create the repository if it does not already exist
aws ecr create-repository --repository-name "${REPO_PREFIX}/${CHART_NAME}" --region us-east-1
# Push the downloaded archive as an OCI artifact
helm push "${CHART_ARCHIVE}" oci://<aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/${REPO_PREFIX}

Example: Pushing to Volcano Engine Container Registry

This example shows how to push images and Helm charts to Volcano Engine Container Registry (CR) using the web console, Docker commands and Helm commands.

global:
  image:
    registry: cr-example-cn-beijing.cr.volces.com
    repository: nvcf-self-hosted

Docker Authentication

# Set your Volcano Engine CR endpoint
CR_ENDPOINT="cr-example-cn-beijing.cr.volces.com"
CR_USERNAME="your-username"
CR_PASSWORD="your-password"
# Login to Volcano Engine CR
echo "${CR_PASSWORD}" | docker login "${CR_ENDPOINT}" \
  --username "${CR_USERNAME}" --password-stdin

Push an Image to Volcano Engine CR

# Set your registry endpoint and namespace
CR_ENDPOINT="cr-example-cn-beijing.cr.volces.com"
NAMESPACE="nvcf-self-hosted"
# Tag the image for Volcano Engine CR
docker tag nvcr.io/nvidia/nvcf/nats-box:0.19.7-nonroot \
  ${CR_ENDPOINT}/${NAMESPACE}/nats-box:0.19.7-nonroot
# Push to Volcano Engine CR
docker push ${CR_ENDPOINT}/${NAMESPACE}/nats-box:0.19.7-nonroot

Push a Helm Chart to Volcano Engine CR

Use the archive from Pulling Helm Charts.

# Set your registry endpoint and namespace
CR_ENDPOINT="cr-example-cn-beijing.cr.volces.com"
NAMESPACE="nvcf-self-hosted"
CR_USERNAME="your-username"
CR_PASSWORD="your-password"
# Login to Volcano Engine CR with Helm
printf '%s\n' "${CR_PASSWORD}" |
  helm registry login "${CR_ENDPOINT}" \
    --username "${CR_USERNAME}" --password-stdin
# Push the downloaded archive as an OCI artifact
helm push "${CHART_ARCHIVE}" "oci://${CR_ENDPOINT}/${NAMESPACE}"

Troubleshooting

exec format error

Symptom: Pods fail to start with Init:CrashLoopBackOff or CrashLoopBackOff status. Checking the logs shows:

exec /bin/sh: exec format error

or

exec /usr/local/bin/docker-entrypoint.sh: exec format error

Cause: This error occurs when container images were pulled/pushed with an architecture that doesn’t match your cluster’s node architecture. This commonly happens when:

• Mirroring from an Apple Silicon Mac (arm64) to an amd64 EKS/GKE cluster
• Mirroring from an Intel/AMD machine (amd64) to an arm64 cluster (e.g., AWS Graviton)

Solution:

1. Delete the incorrectly mirrored images from your registry (e.g., ECR):

   # Delete all repositories with your prefix
   aws ecr describe-repositories --region us-west-2 \
     --query "repositories[?starts_with(repositoryName, 'nvcf-self-hosted')].repositoryName" \
     --output text | tr '\t' '\n' | while read repo; do
     aws ecr delete-repository --repository-name "$repo" --region us-west-2 --force
   done

2. Clean local Docker cache to ensure fresh pulls:

   # Remove all NGC and ECR images from local cache
   docker images --format "{{.Repository}}:{{.Tag}}" | \
     grep -E "(nvcr.io|\.ecr\.)" | \
     xargs -r docker rmi -f
   # Prune dangling images
   docker image prune -f

3. Re-mirror images with the correct platform:

   When pulling images, explicitly specify the target platform:

   # For amd64 clusters (most common)
   docker pull --platform linux/amd64 <image>
   # For arm64 clusters
   docker pull --platform linux/arm64 <image>

   Then re-tag and push to your registry.

4. Force Kubernetes to re-pull images by either:

   • Setting imagePullPolicy: Always temporarily in your Helm values
   • Deleting and redeploying the affected StatefulSets/Deployments

   # Delete StatefulSets to force recreation
   kubectl -n cassandra-system delete statefulset cassandra
   kubectl -n nats-system delete statefulset nats
   # Redeploy using helmfile
   HELMFILE_ENV=<environment-name> helmfile sync