Upgrade from NVCF 1.0.1 to NVCF 1.1.0

This guide describes an upgrade of a running deployment from NVCF 1.0.1 to NVCF 1.1.0. NVCF 1.1.0 selects Self-Managed 1.1.5 for the control plane, Compute Plane 1.1.0 for each GPU cluster, and Observability 1.1.0. Read the whole guide, including What changes, before planning an upgrade.

The stack releases this guide covers are:

Cross-stack upgrade order

These declarations apply to the supported starting releases in this guide:

After the prerequisite NVCA patch below, upgrade the control plane first, then each GPU cluster to Compute Plane 1.1.0.

Starting from 0.6.x? Complete the 0.6.2 to 1.0.1 upgrade first. Do not skip its schema or backfill stops.

What changes

Control plane: charts

The chart versions the 1.1.5 stack pins, compared with 1.0.1. The cert-manager release (0.1.0) is unchanged.

The LLM, vanity gateway and UI releases matter only if you enable those add-ons.

Control plane: data and security components

The data and security components below are the ones that matter for an in-place upgrade. Versions are the ones each release deploys, taken from the published stack inventories.

The target’s main OpenBao migrations image is 0.23.1; 0.19.10 is used only by the optional UI add-on.

Control plane: behavior that changes

Observability

Observability 1.1.0 is optional. Self-Managed 1.1.5 also works with the Observability 1.0.0 stack you already run, so you can upgrade Self-Managed on its own and keep Observability at 1.0.0. The 1.1.0 release changes only two charts and no images. In the default monitors it sets the job label on the worker and DCGM series and adds pod target labels (function-id, function-version-id and nca-id) to the worker series. The chart notes say federation clients such as the BYOO collector select series by that job name, and the autoscaler queries worker metrics by function identity. Upgrade Observability if you rely on either. Function scaling and federation were not exercised with Observability 1.0.0. The control-plane apply reads an Observability package when the observability profile is enabled, so either 1.0.0 or 1.1.0 must be in place before the render (see Get the 1.1.5 packages and check artifacts).

Compute plane

The values that differ between Compute Plane 1.0.0 and 1.1.0:

The byooOtelCollector setting first appears in 1.0.2 and sharedStorage in 1.0.3, so if you start from an earlier release those keys are new to your environment file.

New optional settings: global.nvcaOperator.agent.workerCallbackCaPem (only for TLS worker callbacks), byoo.* sidecar resources, utils.resources, byooFluentBit and eventLedgerServiceURL. Dynamo, Grove and KAI Scheduler pins are unchanged. Running function pods keep the worker images they started with; the new defaults apply to deployments created or replaced afterwards.

Known issues

• Historical packages can retain the train version 1.1.0 instead of the selected patch 1.1.5. The package and receipt checks above reject this mismatch. Obtain a correctly stamped, qualified package before maintenance.
• On editions 1.0.0 through 1.0.3 (Self-Managed 1.0.1), tasks fail before the upgrade when images come from nvcr.io/nvidia/nvcf. Self-Managed 1.0.1 selects nvcf_worker_init:2.107.0, nvcf_worker_task:2.107.0 and ess-agent:1.1.0, which that catalog does not publish, so task pods stop in ImagePullBackOff. Self-Managed 1.1.5 selects published task worker images; complete the task worker image checks before maintenance. Those checks cover the two worker images only; tasks with secrets also pull the ESS agent image set in ess-agent-container (ess-agent:1.4.5 unless you override it), so confirm each compute node pool can pull that image from your registry too. Tasks created after the upgrade use those images.
• A single-member OpenBao installation is not covered; see Replace the OpenBao pods.
• Whether Compute Plane 1.0.0 keeps working against Self-Managed 1.1.5 during the upgrade window was not checked live.

Before starting

1. Schedule a maintenance window. You will pause new function invocations and task submissions, and a single-member Cassandra restarts once.
2. You need an operator workstation with kubectl, helm 3.12 or later, helmfile 1.1.x (not 1.2.x), the helm-diff plugin 3.11 or later, jq, curl, Mike Farah’s yq v4, and ngc if you download packages from NGC. Helmfile 1.2.0 removed sequential execution, which these stacks need. See required tools.
3. Keep your existing control-plane and compute-plane directories, your environments/<environment-name>.yaml, your secrets/<environment-name>-secrets.yaml, and the registration file for each GPU cluster (<cluster-name>-register-values.yaml). The upgrade reuses each cluster’s registration. Do not run register-cluster again.
4. Choose canary workloads: one serving function, one function that reads a secret, and one running task. You will test them before and after.
5. If you mirror images and charts to a private registry, mirror the 1.1.5 artifacts first. Get the 1.1.5 packages and check artifacts checks availability.
6. Before NATS restarts in Apply the dependency stage, patch every compute cluster running NVCA 3.10.0 through 3.10.2 to a compatible 3.10.3 or later 3.10 patch. Preserve registration; see NATS recovery. The Compute Plane 1.1.0 upgrade still waits until Upgrade each compute cluster.

1. Set deployment values

Set up the session

Download this read-only script into one directory. Check the starting state and Render and review the changes use it to stop an upgrade that would move an image to an older version:

Use a dedicated shell. Replace every value in angle brackets. Run this block first, and again after any shell restart.

bash
set -euo pipefail
umask 077
export WORKDIR="$HOME/nvcf-upgrade-1.1.5"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export OLD_CONTROL_STACK="/absolute/path/to/existing/nvcf-self-managed-stack"
export OLD_COMPUTE_STACK="/absolute/path/to/existing/nvcf-compute-plane-stack"
export HELMFILE_ENV="<environment-name>"
export CONTROL_CONTEXT="<control-plane-kube-context>"
export BACKUP_DIR="$WORKDIR/backup"
mkdir -p "$WORKDIR" "$BACKUP_DIR"
# One kubeconfig per cluster, so a stray context switch cannot redirect a command.
export CONTROL_KUBECONFIG="$WORKDIR/control.kubeconfig"
kubectl config view --minify --flatten --raw --context "$CONTROL_CONTEXT" > "$CONTROL_KUBECONFIG"
kc() { kubectl --kubeconfig "$CONTROL_KUBECONFIG" "$@"; }
control_helmfile() {
  (cd "$CONTROL_STACK" && HELMFILE_ENV="$HELMFILE_ENV" \
    helmfile --environment default --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
wait_workloads() {
  local kubeconfig="$1" namespace="$2" workload strategy
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get deployments,statefulsets -o name |
    while IFS= read -r workload; do
      strategy=$(kubectl --kubeconfig "$kubeconfig" -n "$namespace" \
        get "$workload" -o jsonpath='{.spec.updateStrategy.type}')
      if [ "$strategy" = "OnDelete" ]; then
        printf '%s uses OnDelete; replace its pods with the procedure in this guide\n' "$workload"
        continue
      fi
      kubectl --kubeconfig "$kubeconfig" -n "$namespace" rollout status "$workload" --timeout=15m
    done
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get pods
}
declare -f kc control_helmfile wait_workloads > "$WORKDIR/helpers.sh"
declare -p WORKDIR UPGRADE_SCRIPTS OLD_CONTROL_STACK OLD_COMPUTE_STACK HELMFILE_ENV \
  CONTROL_CONTEXT BACKUP_DIR CONTROL_KUBECONFIG > "$WORKDIR/session.sh"
kc config current-context
kc get nodes

All nodes must be Ready. To resume later, start a new bash, run set -euo pipefail, set WORKDIR to the directory you used above (export WORKDIR="$HOME/nvcf-upgrade-1.1.5"), then source "$WORKDIR/session.sh" and source "$WORKDIR/helpers.sh". The saved kubeconfig holds credentials, so keep $WORKDIR private.

Get the 1.1.5 packages and check artifacts

Both exact target packages must be published and available to your customer account before maintenance. Stop if either download or package-stamp check fails. The current 1.0.4 manifest does not select or qualify these targets.

Download the three target packages next to each other, in a new directory, so the relative paths in the Helmfiles resolve the same way they did for your current release.

export STACK_VERSION="1.1.5"
export COMPUTE_STACK_VERSION="1.1.0"
# Observability is optional: export OBSERVABILITY_STACK_VERSION=1.0.0 before
# this block to stay on the version you run now. An existing value is kept.
export OBSERVABILITY_STACK_VERSION="${OBSERVABILITY_STACK_VERSION:-1.1.0}"
cd "$WORKDIR"
ngc registry resource download-version "nvidia/nvcf/nvcf-self-managed-stack:${STACK_VERSION}"
mkdir -p nvcf-self-managed-stack
tar -xzf "nvcf-self-managed-stack_v${STACK_VERSION}/nvcf-self-managed-stack-${STACK_VERSION}.tar.gz" \
  -C nvcf-self-managed-stack
rm -rf "nvcf-self-managed-stack_v${STACK_VERSION}"
ngc registry resource download-version "nvidia/nvcf/nvcf-compute-plane-stack:${COMPUTE_STACK_VERSION}"
mkdir -p nvcf-compute-plane-stack
tar -xzf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}/nvcf-compute-plane-stack-${COMPUTE_STACK_VERSION}.tar.gz" \
  -C nvcf-compute-plane-stack
rm -rf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}"
ngc registry resource download-version "nvidia/nvcf/nvcf-observability-stack:${OBSERVABILITY_STACK_VERSION}"
mkdir -p observability
tar -xzf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}/nvcf-observability-stack-${OBSERVABILITY_STACK_VERSION}.tar.gz" \
  -C observability
rm -rf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}"
export CONTROL_STACK="$WORKDIR/nvcf-self-managed-stack"
export COMPUTE_STACK="$WORKDIR/nvcf-compute-plane-stack"
export OBSERVABILITY_STACK="$WORKDIR/observability"
declare -p STACK_VERSION COMPUTE_STACK_VERSION OBSERVABILITY_STACK_VERSION CONTROL_STACK COMPUTE_STACK OBSERVABILITY_STACK >> "$WORKDIR/session.sh"
ls "$CONTROL_STACK" "$COMPUTE_STACK" "$OBSERVABILITY_STACK"

If you obtain the packages another way, extract them to these three directories. If the observability profile is enabled (observability.profile is not disabled), the control-plane Helmfile reads ../../observability/helmfile.d/01-observability.yaml.gotmpl, which resolves to $WORKDIR/observability. That directory must hold an Observability package, either 1.1.0 or your existing 1.0.0. To keep 1.0.0, export OBSERVABILITY_STACK_VERSION=1.0.0 before running the download block, or copy your existing 1.0.0 stack to that directory and skip its download. Leaving the stack only in its original location makes the render fail. With the profile disabled the directory is not read.

Check both new package stamps against the selected versions before maintenance:

check_selected_stack_versions() {
  local actual
  actual=$(cat "$CONTROL_STACK/VERSION") || return 1
  if [ "$actual" != "$STACK_VERSION" ]; then
    printf 'Self-Managed package must report %s; found %s\n' "$STACK_VERSION" "$actual" >&2
    return 1
  fi
  actual=$(cat "$COMPUTE_STACK/VERSION") || return 1
  if [ "$actual" != "$COMPUTE_STACK_VERSION" ]; then
    printf 'Compute Plane package must report %s; found %s\n' "$COMPUTE_STACK_VERSION" "$actual" >&2
    return 1
  fi
}
check_selected_stack_versions
declare -f check_selected_stack_versions >> "$WORKDIR/helpers.sh"

Stop on a missing or mismatched stamp. Obtain a correctly stamped, qualified package through the release process; do not edit VERSION, replace immutable artifacts or substitute another stack version.

Confirm the charts and images are reachable from your registry. Set the repositories to the ones in your environment file (global.helm.sources and global.image), or to your mirror:

If your environment file uses an HTTPS Helm repository (global.helm.sources.url) instead of an OCI registry, add it under an alias first and use the alias as HELM_REPO below. The password is read from standard input so it stays out of the process list:

export HELM_REPOSITORY_CONFIG="$WORKDIR/helm-repositories.yaml"
read -r -s -p "Repository password: " HELM_PASSWORD; echo
printf '%s' "$HELM_PASSWORD" | helm repo add nvcf-check "<https-repository-url>" \
  --username "<username>" --password-stdin
unset HELM_PASSWORD
helm repo update nvcf-check
export HELM_REPO="nvcf-check"

For an OCI registry, set HELM_REPO to its address instead:

export HELM_REPO="oci://<registry>/<repository>"
export IMAGE_REPO="<registry>/<repository>"
artifact_missing=0
while IFS=: read -r chart version; do
  if helm show chart "$HELM_REPO/$chart" --version "$version" > /dev/null 2>&1; then
    echo "ok       $chart:$version"
  else
    echo "MISSING  $chart:$version"
    artifact_missing=1
  fi
done <<'EOF'
helm-nvcf-cassandra:0.26.0
helm-nvcf-nats:0.9.0
helm-nvcf-openbao-server:0.36.1
helm-nvcf-pki:0.2.0
nvcf-gateway-routes:1.18.3
helm-nvcf-api:1.35.0
helm-nvcf-api-keys:1.9.4
helm-admin-token-issuer-proxy:1.7.0
helm-nvcf-ess-api:1.11.1
helm-nvcf-sis:2.6.3
helm-nvcf-nvct-api:1.11.0
helm-nvcf-notary-service:1.9.1
helm-nvcf-invocation-service:1.9.0
helm-nvcf-grpc-proxy:1.14.0
helm-nvcf-rate-limiter:1.4.1
helm-reval:1.6.0
helm-nvcf-nats-auth-callout-service:1.4.2
helm-nvcf-event-ledger:0.10.0
helm-nvcf-state-metrics:1.0.7
helm-nvcf-function-autoscaler:0.8.0
EOF
while IFS= read -r image; do
  if docker manifest inspect "$IMAGE_REPO/$image" > /dev/null 2>&1; then
    echo "ok       $image"
  else
    echo "MISSING  $image"
    artifact_missing=1
  fi
done <<'EOF'
nvcf-service-oss:1.28.4
nvcf-api-keys-service:1.10.10
nvct-service-oss:1.70.4
nvcf-ess:0.5.7
icms-service-oss:0.8.5
nvcf-notary:1.14.8
nvcf-invocation-service:0.14.2
nvcf-grpc-proxy:1.35.2
nvcf-event-ledger:0.21.0
admin-token-issuer-proxy:1.1.3
nvcf-ratelimiter:1.18.3
reval-server:0.20.4
nvcf-nats-auth-callout-service:0.8.8
nvcf-state-metrics-service:1.24.1
nvcf-function-autoscaler:1.23.1
nvcf-openbao:2.6.3-nv-1.3.7
nvcf-openbao-migrations:0.23.1
nvcf-cassandra-migrations:0.19.4
cassandra:5.0.9-nv-2.0.7
EOF
test "$artifact_missing" -eq 0

The list covers the images that change plus the data-service images. The artifact manifest has the complete list, including optional add-ons. Fix every MISSING line before continuing; a missing image shows up later as ImagePullBackOff in the middle of a stage.

Decide the HA mode and prepare configuration

Choose the HA mode

If this is a single-node cluster, or you are not ready to add capacity, choose none. You can move to HA later. If your Cassandra runs a single member, first follow Expand a single-member Cassandra first. The full behavior, including zone spread, is in the high availability guide.

Check what your cluster offers:

kc get nodes -L topology.kubernetes.io/zone
kc get storageclass
kc get nodes --no-headers | wc -l

For enforced, the StorageClass your Cassandra, NATS and OpenBao volumes use must show WaitForFirstConsumer as its volume binding mode. Replica-safe services roll with maxUnavailable: 0 and maxSurge: 1, so leave room for one extra pod per service.

Carry your settings into the new packages

The new base.yaml files contain the 1.1.5 defaults and chart settings. Copy only your own changes into new environment files. Do not copy the whole old file, because that would overwrite new defaults with old ones.

# Your customizations: the lines in your environment file that differ from the old base.
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_CONTROL_STACK/environments/base.yaml" \
  "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/control-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/control-overrides.diff"
cp "$CONTROL_STACK/environments/base.yaml" "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml"

Open $WORKDIR/control-overrides.diff and re-apply each + line in $CONTROL_STACK/environments/$HELMFILE_ENV.yaml. Typical lines are your registry and chart source (global.helm.sources, global.image), Gateway settings, storage class and size, node selectors and imagePullSecrets. Do not carry over image tag or chart version overrides unless you deliberately pin them; the 1.1.5 defaults are the qualified set. The 1.1.5 base.yaml no longer pins cassandra.migrations.image.tag, so the chart default (0.19.4) applies. If you pin that tag, remove the pin or set 0.19.4 or later: an older pin skips the schema changes in Apply the dependency stage.

The 1.1.5 defaults pull the NATS server, nats-box and config reloader images from public registries (docker.io/synadia/nats-server, docker.io/natsio/nats-box and docker.io/natsio/nats-server-config-reloader). Your release pulled the server and nats-box from the NVCF catalog. If your cluster cannot pull from public registries, mirror the three images and set them in the new environment file, as in this example (the reloader uses nats.reloader.image.*):

nats:
  container:
    image:
      registry: <your-registry>
      repository: <your-repository>/nats-server
  natsBox:
    container:
      image:
        registry: <your-registry>
        repository: <your-repository>/nats-box
  reloader:
    image:
      registry: <your-registry>
      repository: <your-repository>/nats-server-config-reloader

If you set none of these settings and your cluster can pull from docker.io, there is nothing to do. The 1.1.5 NATS pods run as a non-root user, so confirm in Apply the dependency stage that each NATS pod becomes Ready.

Then set the HA mode and, if you want to turn Event Ledger off, its switch. In the environment file:

highAvailability:
  mode: preferred   # none | preferred | enforced
addons:
  eventLedger:
    enabled: true   # set false to skip the new Event Ledger release

Merge these keys into the existing addons: section; do not duplicate it.

Check the secrets file for replica overrides

Under preferred and enforced, the render fails if the secrets file sets a value that conflicts with the HA sizing, for example replicaCount: 1. Look for them now:

grep -nE 'replica(Count|s):' "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  || echo "no replica overrides"

Remove any replica override the HA sizing should control, or choose highAvailability.mode: none. Record the mode you chose:

yq '.highAvailability' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"

2. Check the starting state and take backups

Check the starting state

Do not continue unless every check passes.

# Releases deployed
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json > "$WORKDIR/control-before.json"
jq -r '.[] | "\(.namespace)\t\(.name)\t\(.chart)\t\(.status)"' "$WORKDIR/control-before.json" | sort
test -z "$(jq -r '.[] | select(.status != "deployed") | .name' "$WORKDIR/control-before.json")" \
  && echo "all releases deployed"
# Cassandra: every member UN
for pod in $(kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra \
  -o jsonpath='{.items[*].metadata.name}'); do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool status
  break
done
# Workloads ready everywhere; sis only when the ICMS release is installed
namespaces="cassandra-system nats-system vault-system nvcf api-keys ess"
if jq -e '.[] | select(.name == "sis")' "$WORKDIR/control-before.json" > /dev/null; then
  namespaces="$namespaces sis"
fi
for namespace in $namespaces; do
  kc -n "$namespace" get pods -o json | jq -e '
    (.items | length > 0) and
    any(.items[]; any(.status.conditions[]?; .type == "Ready" and .status == "True")) and
    all(.items[]; .metadata.deletionTimestamp == null and
      (.status.phase == "Succeeded" or
        any(.status.conditions[]?; .type == "Ready" and .status == "True")))'
done
# Record replica counts and images before the upgrade
kc get deploy,sts -A -o wide > "$WORKDIR/workloads-before.txt"
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" before \
  "$CONTROL_KUBECONFIG" "$WORKDIR/images-before.tsv"
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml > "$WORKDIR/receipt-before.yaml" 2>/dev/null || true

Cassandra must report every member UN. The OpenBao check function used in this guide, and again in Apply the dependency stage, is:

check_openbao() {
  local pod pods
  pods=$(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}') || return 1
  [ -n "$pods" ] || { echo "No OpenBao server pods found" >&2; return 1; }
  for pod in $pods; do
    echo "== $pod =="
    kc -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json' || return 1
  done
}
declare -f check_openbao >> "$WORKDIR/helpers.sh"
check_openbao

Every OpenBao member must be initialized: true and sealed: false, with the same cluster_id. Exactly one reports is_self: true; standbys report is_self: false or omit the field. Record the cluster ID.

Run your canary workloads now and record their results. With the configured nvcf-cli:

export NVCF_CLI="/absolute/path/to/nvcf-cli"
export NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
export FUNCTION_ID="<existing-function-id>"
export VERSION_ID="<existing-version-id>"
export CANARY_REQUEST='{"input":"your-known-canary-input"}'
declare -p NVCF_CLI NVCF_CLI_CONFIG FUNCTION_ID VERSION_ID CANARY_REQUEST >> "$WORKDIR/session.sh"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
  --request-body "$CANARY_REQUEST" --timeout 120
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Back up persistent state

Create restorable backups before you change anything. Follow your storage provider’s procedure for volume snapshots as well; the commands below create the application-level backups.

CASSANDRA_REPLICAS=$(kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool snapshot -t pre-1-1-5
done

The Cassandra snapshot stays on each persistent volume. It is not an off-cluster backup, so also copy or snapshot the volumes.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from a standby fails with incomplete snapshot.

OPENBAO_ACTIVE=""
for pod in $(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  if kc -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
    OPENBAO_ACTIVE="$pod"
  fi
done
echo "active member: $OPENBAO_ACTIVE"
kc -n vault-system get secret openbao-server-root-token -o jsonpath='{.data.root_token}' | base64 -d |
  kc -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
  'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-1-5.snap'
kc -n vault-system cp -c openbao "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-1-5.snap" \
  "$BACKUP_DIR/openbao-pre-1-1-5.snap"
kc -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-1-5.snap
# The snapshot restores only with the unseal key and root token.
kc -n vault-system get secret openbao-server-unseal -o yaml > "$BACKUP_DIR/openbao-server-unseal.yaml"
kc -n vault-system get secret openbao-server-root-token -o yaml > "$BACKUP_DIR/openbao-server-root-token.yaml"
# Helm values of every release, and your configuration files.
mkdir -p "$BACKUP_DIR/helm-values"
jq -r '.[] | "\(.namespace) \(.name)"' "$WORKDIR/control-before.json" |
  while read -r namespace release; do
    helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" -n "$namespace" -o yaml \
      > "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
  done
cp "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" "$BACKUP_DIR/old-environment.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" "$BACKUP_DIR/old-secrets.yaml"
ls -l "$BACKUP_DIR"

The unseal and root-token files, the secrets copy and the exported values contain credentials. Protect them like the snapshot. NVCF NATS streams use memory storage, so a NATS backup holds stream metadata only. Do not continue until the Cassandra and OpenBao backups exist.

Render and review the changes

Render into a new directory, then review the diff the first stage would apply.

make -C "$CONTROL_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" OUTPUT_DIR="$WORKDIR/control-render"
# Images the 1.1.5 render pulls; make sure each one is in your registry or mirror.
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/control-render" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u

The render can include secrets, so keep it private. Check that the namespaces match your cluster and that no image lacks the registry you expect.

Compare the render with the images that run now. The check fails if any image would move to an older version, which happens when an old pin survives in your environment file:

python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" compare \
  "$WORKDIR/images-before.tsv" "$WORKDIR/control-render"

Every BACKWARD line names an image and both versions. Stop on any of them. For a pin you did not set on purpose, remove it as described in Decide the HA mode and prepare configuration, render again and re-run the check.

Before maintenance, verify the receipt image from the rendered Job. Its default is docker.io/alpine/k8s:1.33.1; it does not follow global.image. For a mirror, set upgradeReceipt.image.registry and upgradeReceipt.image.repository, then render again. Use the resulting image reference for the check.

Run a temporary pull-test Pod in the Job’s namespace on each eligible node pool, using that image, its pull credentials and scheduling settings. Set imagePullPolicy: Always and run true instead of the receipt script. Confirm the image pulls and the Pod succeeds, then remove the test Pod. A host-side docker manifest inspect alone does not verify node access.

If you run tasks, complete the task worker image checks using this target render and the Self-Managed 1.1.5 inventory before proceeding.

3. Refresh the hostname policy

This is a required one-time step for release 1.0.1, because it runs gateway routes chart 1.18.2.

The 1.18.2 chart installed a hostname policy that denies the admin issuer proxy route patch, and that release applies before ingress. The target checks that the pinned chart (1.18.3) carries the corrected policy and then applies only the ingress release.

make -C "$CONTROL_STACK" apply-hostname-policy HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
helm --kubeconfig "$CONTROL_KUBECONFIG" -n envoy-gateway-system status ingress
kc get httproutes -A

The ingress release must be deployed, and your existing routes and canaries must still work. Stop on any error; do not disable admission to get past it. If the target reports that the pinned chart “still reserves hostnames”, the packages do not match the selected release; recheck Get the 1.1.5 packages and check artifacts.

4. Drain pending work

Stop new production requests and task submissions at their callers. NATS request queues use memory storage, and the dependency stage can restart NATS. Wait for queued work to finish; do not cancel it to get an empty queue.

For every affected function and version, use a key with queue_details scope (NVCF_API_KEY). The task command below uses a separate key with list_tasks scope (NVCF_NVCT_API_KEY) and your Tasks endpoint (base_nvct_url and nvct_host in the CLI configuration). Verify both with the CLI authentication settings before you start:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Queue sizes must reach zero. Record the identities and results of running tasks. Then check the control plane’s own single-replica memory streams on every NATS server, through the monitoring port 8222:

NATS_MONITOR_PORT=18223
NATS_PODS=$(kc -n nats-system get pods -o json | jq -r '.items[] |
  select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
NATS_DRAIN_DIR="$WORKDIR/nats-drain-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir "$NATS_DRAIN_DIR"
for pod in $NATS_PODS; do
  (
    kc -n nats-system port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
      > "$NATS_DRAIN_DIR/$pod.log" 2>&1 &
    forward_pid=$!
    trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
    for attempt in {1..30}; do
      kill -0 "$forward_pid"
      grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log" && break
      sleep 1
    done
    grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log"
    curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
      "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
      > "$NATS_DRAIN_DIR/$pod.json"
    jq -e '.account_details | type == "array"' "$NATS_DRAIN_DIR/$pod.json"
    jq -e 'all(.account_details[];
      ((.stream_detail // []) | type == "array") and
      all((.stream_detail // [])[];
        (.config.storage | type == "string") and
        (.state.messages | type == "number")))' "$NATS_DRAIN_DIR/$pod.json"
  )
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
  select(.config.storage == "memory" and (.config.num_replicas // 1) == 1) |
  {name: .name, messages: .state.messages}] |
  if length > 0 and all(.[]; .messages == 0)
  then . else error("Memory streams missing or not drained; do not roll NATS") end' \
  "$NATS_DRAIN_DIR/"*.json

Every single-replica memory stream must show zero messages. A failed or slow read, a missing stream or a nonzero count blocks the upgrade. A server whose accounts hold no streams reports no stream details, which counts as empty; the final check still requires at least one single-replica memory stream. Each attempt writes to a new directory, so files from an earlier attempt cannot satisfy the check. Wait for the work to drain and repeat; do not discard messages. If your NATS monitoring is configured differently, use its endpoint and keep the same per-server check. For retained creation messages or stuck deployments, follow the recovery steps before proceeding.

5. Apply control-plane stages in order

Expand a single-member Cassandra first

Do this only if CASSANDRA_REPLICAS from Back up persistent state is 1 and you chose preferred or enforced. Skip it if Cassandra already runs 3 members, or if you chose none. Run it in the maintenance window, after you drain pending work: while system_auth is being raised and repaired, new logins can fail.

Do not let the upgrade expand Cassandra from one member to three. The chart raises the system_auth replication factor to the new member count in a hook that runs after the StatefulSet scales, and it does not repair the data. The new members then hold none of the existing login data, so administrator and service logins can fail while every member shows UN, and the dependency stage times out. Expand and repair now, on the release you run today, so the upgrade starts from a normal three-member cluster.

Place the new members in different zones. A member’s volume stays in the zone where the member first starts, and the placement rules that the upgrade applies later cannot move it. Under enforced, two members in one zone leave a pod Pending during the upgrade. If every schedulable node has a topology.kubernetes.io/zone label, add a zone spread rule to the StatefulSet before you scale. The rule is hard under enforced and soft under preferred, so a zone without capacity never leaves Cassandra Pending under preferred. This restarts the single member once, and the upgrade replaces the rule with the chart’s own. If any node has no zone label, skip this rule: a node without the label cannot take a pod that has it.

kc get nodes -L topology.kubernetes.io/zone
HA_MODE=$(yq -r '.highAvailability.mode // "preferred"' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml")
case "$HA_MODE" in
  enforced) SPREAD=DoNotSchedule ;;
  preferred) SPREAD=ScheduleAnyway ;;
  *) echo "Unsupported highAvailability.mode: $HA_MODE" >&2; exit 1 ;;
esac
kc -n cassandra-system patch statefulset cassandra --type merge -p "{\"spec\":{\"template\":{\"spec\":{\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"$SPREAD\",\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/name\":\"cassandra\"}}}]}}}}"
kc -n cassandra-system rollout status statefulset/cassandra --timeout=10m

Scale the StatefulSet directly. Do not set cassandra.replicaCount and apply the old stack, because that runs the chart hook with the same problem. The block reads the member count that the backup step saved, so it also works after a shell restart, and it shows the live count. Scaling to 3 is safe to repeat. Wait for all three members to be UN:

CASSANDRA_REPLICAS=$(cat "$BACKUP_DIR/cassandra-replicas")
test "$CASSANDRA_REPLICAS" -eq 1
kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}{" members now\n"}'
kc -n cassandra-system scale statefulset cassandra --replicas=3
kc -n cassandra-system rollout status statefulset/cassandra --timeout=20m
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

Every member must report UN. If a pod stays Pending, check the node and volume capacity before you continue. Confirm that each member runs in a different zone:

kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra -o wide
kc get nodes -L topology.kubernetes.io/zone

Raise the system_auth replication factor to 3, repair it on every member, then confirm that the administrator can log in on every member. The password is that of the cassandra administrator role, not the service role password in your secrets file. The commands read it from standard input into a temporary protected file, so do not enable shell tracing.

CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
cql_on cassandra-0 "ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full system_auth
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  cql_on "$pod" "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
  echo "administrator login ok on $pod"
done

All three members must print administrator login ok. If any login fails, stop. Do not continue to the upgrade. Restore from the backups in Back up persistent state with your reviewed recovery procedure, and contact NVIDIA support with the output.

Then raise the application keyspaces to 3 and repair them on every member. The loop alters only the keyspaces your installation has:

EXISTING_KEYSPACES=$(cql_on cassandra-0 "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql_on cassandra-0 "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD

Repeat your canary invocation from Check the starting state and confirm that the services report no authentication errors:

AUTH_LOGS=$(kc -n nvcf logs deploy/nvcf-api --since=10m)
printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true

The count must be 0. Raise replication for HA later in this guide repeats the application keyspace step, which is safe to run again.

Apply the dependency stage

This stage updates Cassandra, NATS, cert-manager, OpenBao and the PKI. Review the diff first, and stop on unexpected deletions or version overrides:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets

Apply it:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
  kc -n "$namespace" get jobs
done

What you will see:

• Cassandra restarts one pod at a time because of the config change and the new server image (2.0.7), and grows to 3 members under preferred or enforced. A single-member cluster is unavailable while its one pod restarts.
• Migrations run as Helm hook jobs. They must complete successfully. The Cassandra migrations image moves to 0.19.4 from 0.17.6. The new migrations only add schema: nullable columns on sis_api.requests, sis_api.instances and the event_ledger tables, and indexes on the new sis_api creation_bucket column. They drop nothing. Existing rows keep null in the new columns, and nothing backfills them. Building the indexes reads the existing rows, so a large sis_api table can make that job run longer.
• The OpenBao hooks (image 0.23.1) refresh the service, admin issuer and Event Ledger configuration, and add a signing role for gRPC proxy events. They only add or merge configuration and keep existing roles.
• OpenBao shows the new template, but its pods still run the old image until Replace the OpenBao pods. wait_workloads reports it as OnDelete.

Check that the data is intact, not only that pods are Ready:

kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
kc -n cassandra-system get jobs
kc -n vault-system get jobs

All Cassandra members must be UN and every migration job Complete. Investigate any failed job before continuing; do not disable hooks.

Replace the OpenBao pods

The OpenBao StatefulSet uses OnDelete: Helm updated its template, but the running pods keep their old image. Replace them one at a time, standbys first. Never delete a persistent volume claim, reinitialize OpenBao, or replace more than one member at once. This procedure is the 2.6.2 to 2.6.3 patch procedure from the OpenBao patch upgrade runbook. Use that runbook if a step fails or if you customized plugin registration, policies or PKI.

This procedure is written for the supported OpenBao deployment, a Raft cluster of several members. Replace a standby first so that leadership does not move twice. Check the member count:

kc -n vault-system get statefulset openbao-server -o jsonpath='{.spec.replicas}{"\n"}'

If it prints 1, stop. A single-member OpenBao is not a supported deployment and this guide has no procedure for it: there is no standby to replace first and no leadership handover, and OpenBao is unavailable while its one pod restarts and unseals. Do not continue on your own. Confirm the Raft snapshot from the backup step and use the OpenBao patch upgrade runbook to plan the change with your NVCF contact.

Confirm the new template and the update strategy:

kc -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
check_openbao

The server image must be nvcf-openbao:2.6.3-nv-1.3.7 and the strategy OnDelete. If either is different, stop. Identify the standbys (the members that do not report is_self: true) and replace one:

POD="<standby-pod-name>"
kc -n vault-system delete pod "$POD" --wait=true --timeout=3m
kc -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=5m
check_openbao

If the new pod has not appeared yet, wait for it before repeating the wait. The replacement must report version: "2.6.3", sealed: false and the original cluster_id. If it is Ready but still sealed, let auto-unseal finish and run check_openbao again. Repeat for the second standby only after all three members are healthy.

Then replace the former active member. Transfer leadership first:

ACTIVE_POD="<current-active-pod-name>"
set -o pipefail
kc -n vault-system get secret openbao-server-root-token -o json |
  jq -er '.data.root_token | @base64d' |
  kc -n vault-system exec -i "$ACTIVE_POD" -c openbao -- sh -ec '
    IFS= read -r BAO_TOKEN
    test -n "$BAO_TOKEN"
    export BAO_TOKEN
    export BAO_ADDR=http://127.0.0.1:8200
    bao operator step-down
  '
check_openbao

Wait until a 2.6.3 member reports is_self: true and the old active reports standby. A successful request alone does not confirm the transfer. Then replace the former active with the same delete, wait, check_openbao sequence. Do not enable shell tracing or print the token.

Verify all three members and their images:

check_openbao
kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'

All three must be unsealed on 2.6.3, with one active member and the original cluster ID. Updating the injector affects only new pods. The services the core stage rolls in the next step pick up the new agent; roll any other injected workload through its normal deployment procedure, one service at a time, and check its authentication and rendered secrets. A single-replica service can be interrupted during that roll, so never restart all consumers together.

Apply the core services

control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets
control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
done
kc get httproutes -A

Core releases must be deployed, their workloads ready and their routes Accepted. The new event-ledger release appears in the nvcf namespace unless you turned it off. Under preferred or enforced, services scale to 2 replicas and roll with maxUnavailable: 0, so a rollout needs room for one extra pod per service. Check any add-on namespaces you enable (LLM, vanity gateway, UI) the same way.

Repeat your canary invocation now. It must return the result you recorded.

Apply state metrics and autoscaling

control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl status
wait_workloads "$CONTROL_KUBECONFIG" nvcf

Keep your existing observability profile. Enabled state-metrics and autoscaler releases must be deployed and healthy. Do not turn off state metrics to get past a validation failure; the control and all profiles require it. Also reapply the observability infrastructure stage. It applies the Observability version in $WORKDIR/observability. With 1.0.0, changed configuration can still update releases. With 1.1.0, the new charts replace the 1.0.0 ones. Review the diff first:

control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl diff --suppress-secrets

Stop on unexpected changes. Then apply the stage:

control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl apply

Raise replication for HA

Skip this step if you chose highAvailability.mode: none.

After the dependency stage, Cassandra has 3 members, but your application keyspaces keep the replication factor they were created with, so losing one Cassandra node can still make data unavailable. Wait until all three Cassandra pods are Ready, then raise each keyspace to RF=3 and repair, one member at a time. The loop below alters only the keyspaces your installation has and skips the rest. It validates the deployed data-center name against the live database before any ALTER. Credentials travel over stdin into a temporary protected cqlsh credentials file. Do not enable shell tracing.

CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i cassandra-0 -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$1"
)
LIVE_DC=$(cql "SELECT JSON data_center FROM system.local;" |
  sed -n '/^[[:space:]]*{/{s/^[[:space:]]*//;p;}' |
  jq -ers 'if length == 1 then .[0].data_center else error("Missing data-center row") end')
test "$LIVE_DC" = "$CASSANDRA_DC"
cql "SELECT keyspace_name, replication FROM system_schema.keyspaces;"
EXISTING_KEYSPACES=$(cql "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD

Until the repairs finish, reads at LOCAL_ONE can miss rows that exist only on the original replica. The application keyspaces must now report RF=3 for the validated data center.

JetStream streams are created with the replication factor current at creation time. The icms streams raise themselves to the configured factor when it starts. Streams created earlier by nvcf-api and invocation-service stay at 1 replica until you edit them. List them and check:

kc -n nats-system exec deploy/nats-box -- nats stream report

The Replicas column lists each member that holds a copy of the stream. A highly available stream shows three (nats-0, nats-1, nats-2); a stream created before HA shows one. Raise each stream that shows one:

STREAM="<stream-name>"
kc -n nats-system exec deploy/nats-box -- nats stream edit "$STREAM" --replicas=3 --force
kc -n nats-system exec deploy/nats-box -- nats stream info "$STREAM"

Replace <stream-name> with the name from the report, run both commands for each such stream, and confirm nats stream info shows Replicas: 3. Use 3, not 2: a 2-replica stream loses quorum when either member is down. The NATS commands run in the nats-box deployment because the nats server container does not include the CLI.

Write the upgrade receipt

Run this only after every earlier stage succeeded.

Complete the receipt image pull check before reaching this stage. Recheck it if the image or pull credentials changed.

check_selected_stack_versions
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kc -n nvcf get job upgrade-receipt --ignore-not-found
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml
test "$(kc -n nvcf get configmap nvcf-upgrade-receipt \
  -o jsonpath='{.data.installed_stack_version}')" = "$STACK_VERSION"
echo "receipt matches the selected Self-Managed release"

The receipt must match the selected Self-Managed 1.1.5 release. It does not record the outcome of workload checks. The hook job is removed after ten minutes; the ConfigMap is the lasting record.

6. Verify the control plane

Check releases, versions and images:

helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kc -n nvcf get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n api-keys get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n sis get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n ess get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image

Key images you should see:

If you chose HA, confirm the replicas and spread (these are the same checks the high availability guide uses):

kc get nodes -L topology.kubernetes.io/zone
kc -n cassandra-system get pods -o wide
kc -n nats-system get pods -o wide
kc -n vault-system get pods -o wide
kc -n nvcf get deploy nvcf-api -o wide
kc -n nvcf get deploy invocation-service grpc-proxy-deployment -o wide
kc -n ess get deploy ess-api-deployment -o wide
kc get pdb -A

Replica-safe services show 2 replicas, and Cassandra, NATS and OpenBao show 3. invocation-service and grpc-proxy stay at 1 replica if you set a fixed worker callback URL. Do not resume traffic until the checks pass. Repeat the canary from Check the starting state, create and invoke a new function with a secret, and confirm queues make progress. Existing workers keep the images they started with.

7. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Use the cluster’s existing registration; the cluster, group and account identity must not change.

Set the variables

export COMPUTE_CONTEXT="<gpu-cluster-kube-context>"
export CLUSTER_NAME="<gpu-cluster-name>"
export NCA_ID="<existing-account-id>"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export COMPUTE_KUBECONFIG="$WORKDIR/compute-$CLUSTER_NAME.kubeconfig"
kubectl config view --minify --flatten --raw --context "$COMPUTE_CONTEXT" > "$COMPUTE_KUBECONFIG"
kg() { kubectl --kubeconfig "$COMPUTE_KUBECONFIG" "$@"; }
record_identity() {
  kg -n nvca-operator get configmap nvcfbackend-self-managed \
    -o jsonpath='{.data.cluster-dto\.yaml}' |
    yq -o=json '{"clusterId": .clusterId, "clusterGroupId": .clusterGroupId,
      "clusterName": .clusterName, "ncaID": .ncaID}' |
    jq -eS 'if all(.[]; type == "string" and length > 0)
      then . else error("Missing cluster identity") end'
}
kg get nodes
record_identity

If the cluster name or account ID was unknown, set CLUSTER_NAME and NCA_ID from this output before continuing. The registration file must exist and match that identity; every node and the NVCA backend must be healthy:

test -s "$REGISTRATION_VALUES_DIR/$CLUSTER_NAME-register-values.yaml"
declare -p COMPUTE_CONTEXT CLUSTER_NAME NCA_ID REGISTRATION_VALUES_DIR COMPUTE_KUBECONFIG >> "$WORKDIR/session.sh"
declare -f kg record_identity >> "$WORKDIR/helpers.sh"
record_identity | tee "$WORKDIR/identity-before-$CLUSTER_NAME.json"
kg -n nvca-operator get nvcfbackends
kg -n nvca-operator get pods
kg -n nvca-system get pods

Carry your settings into the compute package

# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_COMPUTE_STACK/environments/base.yaml" \
  "$OLD_COMPUTE_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/compute-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/compute-overrides.diff"
cp "$COMPUTE_STACK/environments/base.yaml" "$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml"

Re-apply each + line from compute-overrides.diff in $COMPUTE_STACK/environments/$HELMFILE_ENV.yaml (registry, image settings, observability profile and similar). Do not copy old nvcaVersion, imageTag or chart pins over the 1.1.0 defaults. Leave workerCallbackCaPem empty unless you serve worker callbacks over TLS with a private CA. If your clusters share one environment file, do this once.

Check images, render and review

make -C "$COMPUTE_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/compute-render-$CLUSTER_NAME" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u

Mirror every image the list shows (NVCA 3.18.6, shared storage 1.0.9, image credential helper 0.11.4 and the BYOO collector 0.160.0-nv-0.4.7 are new defaults) before you apply. Then review the diff:

(cd "$COMPUTE_STACK" && HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME="$CLUSTER_NAME" \
  NCA_ID="$NCA_ID" OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME" \
  helmfile --environment default --kubeconfig "$COMPUTE_KUBECONFIG" \
  --kube-context "$COMPUTE_CONTEXT" diff --suppress-secrets)

Expect the nvca-operator chart to move from 1.28.0 to 1.37.1 and the NVCA agent to 3.18.6. Stop if the diff shows the registration or cluster identity changing, or unexpected deletions.

Apply

make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
wait_workloads "$COMPUTE_KUBECONFIG" nvca-operator
kg -n nvca-system get pods
kg -n nvca-operator get nvcfbackends
record_identity > "$WORKDIR/identity-after-$CLUSTER_NAME.json"
diff -u "$WORKDIR/identity-before-$CLUSTER_NAME.json" "$WORKDIR/identity-after-$CLUSTER_NAME.json" \
  && echo "cluster identity unchanged"

The operator and agent must be ready, the backend must report healthy, and the identity files must match. If they differ, stop and inspect them before doing anything else. Do not install a second copy of shared observability resources on separate compute clusters; keep your existing configuration.

Confirm from the control plane that the cluster is registered and healthy:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" cluster list-registered

Existing function and task canaries must still work with their old worker images. To move a function to the new worker defaults, replace its deployment through the deployment API or CLI; updating the stack defaults alone does not change running workers or explicit per-function image overrides. Plan for the interruption while a replacement rolls.

Repeat this step for each remaining GPU cluster, with its own CLUSTER_NAME, NCA_ID, registration directory and kube context.

8. Resume workloads

Reapply the control plane and confirm it is stable:

make -C "$CONTROL_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
control_helmfile status
set +e
control_helmfile diff --suppress-secrets --detailed-exitcode
DIFF_EXIT=$?
set -e
echo "diff exit code: $DIFF_EXIT"
if [ "$DIFF_EXIT" -ne 0 ]; then
  echo "STOP: the final diff exited $DIFF_EXIT. Inspect it before you resume traffic." >&2
  exit "$DIFF_EXIT"
fi

The final diff must report no changes and exit 0. A nonzero exit means there are pending changes or a command error. Resolve them and rerun this gate before continuing:

1. Repeat the canary invocations and compare IDs, results and persistent data with the values from Check the starting state.
2. Create and invoke a new function and a new task with the inputs you normally use, including one function that reads a secret.
3. Resume production submissions and watch queue progress and telemetry.

Keep $WORKDIR, the diffs, the render directories and your notes with the installed configuration as the upgrade record.

Troubleshooting

Rollback boundary

If an upgrade fails, stop and use your deployment’s reviewed recovery procedure. Do not apply your old packages unchanged as a blanket rollback. Helm rollback does not reverse stored data, does not validate a database or OpenBao downgrade, and does not revert a Cassandra replication factor you raised. The Cassandra migrations in 1.1.5 add columns and indexes, and a rollback does not remove them. Older services have not been tested against that schema. Restore data only through the reviewed procedure for that datastore, using the backups from Back up persistent state. After any recovery, verify the original cluster identities, old-image canaries, durable data and availability. Changing which package you use does not roll back a running deployment.

Related

• Compatibility matrix
• Artifact manifest
• High availability guide
• OpenBao patch upgrade runbook
• Control-plane operations
• 0.6.2 to 1.0.1 upgrade