Skip to navigation

Upgrade from NVCF 1.0.4 to NVCF 1.1.1

View as MarkdownOpen in Claude

This upgrade procedure and its validation are still in progress. Follow the 1.1.1 qualification issue for the remaining work. Do not treat the publication of NVCF 1.1.1 as a qualified upgrade path until that issue records it.

This guide describes an upgrade of a running deployment from NVCF 1.0.4 to NVCF 1.1.1. NVCF 1.1.1 selects Self-Managed 1.1.9 for the control plane, Compute Plane 1.1.1 for each GPU cluster, and Observability 1.1.0. Read the whole guide, including What changes, before planning an upgrade.

This guide is for deployments on NVCF 1.0.4.

Choose your upgrade path

NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach NVCF 1.1.1:

You runDo this
NVCF 1.0.1, 1.0.2 or 1.0.3Upgrade to NVCF 1.0.4 first with the 1.0.2 to 1.0.4 upgrade, then use the 1.0.4 to 1.1.1 upgrade.
NVCF 1.0.4Use the 1.0.4 to 1.1.1 upgrade.
NVCF 1.1.0You have already passed the 1.0.4 stop. Use the 1.1.0 to 1.1.1 upgrade.

Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.

NVCF editionSelf-ManagedCompute PlaneObservability
1.0.41.0.101.0.51.0.0
1.1.1 (target)1.1.91.1.11.1.0

The stack releases this guide covers are:

StackSourceTargetTarget tag
Self-Managed (control plane)1.0.101.1.9deploy/stacks/self-managed/v1.1.9
Compute Plane1.0.51.1.1deploy/stacks/nvcf-compute-plane/v1.1.1
Observability1.0.01.1.0deploy/stacks/observability/v1.1.0

Cross-stack upgrade order

These declarations apply to the supported starting release in this guide:

TransitionRequired peer
Compute Plane 1.1.1Self-Managed 1.1.0 or later, subject to the compatibility and maintenance policy
Self-Managed 1.1.9 during the upgrade windowExisting Compute Plane 1.0.5 can remain until its upgrade

Upgrade the control plane first, then each GPU cluster to Compute Plane 1.1.1.

Do not run a whole-stack make apply or helmfile sync first. The control plane has stops that must happen in order: dependencies and migrations, the OpenBao check, core services, then Cassandra replication for high availability. Choose the high availability (HA) mode in Prepare the HA mode and configuration before you apply anything. The default, preferred, needs little preparation; enforced needs the full preparation in that section.

What changes

Control plane: charts

The chart versions the 1.1.9 stack pins, compared with 1.0.10. The cert-manager (0.2.0), ingress (1.18.3) and nvcf-ui (1.1.4) releases are unchanged.

Release1.0.101.1.9
admin-issuer-proxy1.5.31.7.0
api1.27.51.35.0
api-keys1.8.21.9.4
cassandra0.21.40.26.0
ess-api1.8.31.11.1
event-ledgernot installed0.10.2 (new)
function-autoscaler0.5.20.8.1
grpc-proxy1.7.41.14.0
invocation-service1.6.21.10.0
llm-api-gateway1.4.31.12.0
llm-request-router1.14.31.18.2
nats0.8.60.9.1
nats-auth-callout-service1.2.11.4.2
notary-service1.6.11.9.1
nvcf-pki0.1.00.2.0
nvct-api1.6.11.11.0
openbao-server0.32.90.37.1
ratelimiter1.2.11.4.1
reval1.4.11.6.0
sis2.4.22.6.3
state-metrics1.0.21.0.7
vanity-gateway0.5.00.11.0
worker-callback-proxynot installed0.1.0 (new, off by default)

The LLM and vanity gateway releases matter only if you enable those add-ons.

Control plane: data and security components

The data and security components below are the ones that matter for an in-place upgrade. Versions are the ones each release deploys, taken from the published stack inventories.

Component1.0.10Target 1.1.9
Cassandra chart0.21.40.26.0
Cassandra image5.0.9-nv-2.0.75.0.9-nv-2.0.7
Cassandra migrations image0.17.80.19.4
OpenBao chart0.32.90.37.1
OpenBao image2.6.3-nv-1.3.92.6.3-nv-1.3.9
OpenBao migrations image0.19.110.23.4
NATS chart0.8.60.9.1
NATS server image2.14.9-alpine3.222.14.9-alpine3.22
NATS config reloader0.25.00.25.0
cert-manager imagesv1.20.5v1.20.5
API service image1.18.21.28.4
Invocation service image0.12.30.15.1
ICMS0.7.70.8.6

The NATS, cert-manager and OpenBao server images are the same as in 1.0.4, and 1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle. This upgrade does not repeat that plugin update.

Control plane: behavior that changes

AreaWhat happens on upgradeWhat you do
High availabilityhighAvailability.mode is new and defaults to preferred. An environment file that does not set it activates HA: replica-safe services scale to 2, Cassandra, NATS and OpenBao run 3 members, and PodDisruptionBudgets and placement rules apply.Choose none, preferred or enforced in Prepare the HA mode and configuration.
CassandraThe chart adds a config setting (range_request_timeout: 5000ms). The pod template changes, so the StatefulSet restarts its pods one at a time. The migrations image moves to nvcf-cassandra-migrations:0.19.4, whose new migrations only add schema (Apply the dependency stage).Plan a short Cassandra interruption if you run a single member.
OpenBaoThe chart moves to 0.37.1 and the migrations image to 0.23.4. The server image and the JWT plugin lifecycle are unchanged. The StatefulSet uses OnDelete, so Helm success does not replace the running pods.Check whether the pods need replacing in Check the OpenBao pods.
Event LedgerNew release event-ledger, on by default (addons.eventLedger.enabled: true). It uses the existing event_ledger keyspace, and its OpenBao setup runs in the OpenBao migrations job.Keep it on, or set addons.eventLedger.enabled: false before you apply.
NATS worker authenticationNew nats.workerAuth setting. The default, nkey, keeps today’s behavior.Nothing, unless you switch to token. See “Worker NATS Authentication” in control-plane operations.
Admin issuer proxyNow optional (adminIssuerProxy.enabled), still on by default.Nothing, unless you want to disable it.
NVCT remote configNew nvctApi.remoteConfig, empty by default.Nothing.
Upgrade receiptThe final stage records the installed stack version in nvcf-upgrade-receipt.Validate package stamps first, then compare the receipt with the selected release.
Worker sidecarsAPI and NVCT remote config pin the ESS agent sidecar image to ess-agent:1.4.5 and the patched NICLLS worker to nvcf_worker_niclls:2.118.5. Running functions keep the images they started with.If you mirror sidecar images, mirror both.
LLM add-on image tagsWith addons.llm.enabled, the request router and Pylon tag must match, and addons.llm.requestRouter.image.tag is required. base.yaml sets 0.29.3.Keep the default unless you mirror or pin it.
LLM routesingress.gatewayApi.routes.llmApiGateway.enabled and ingress.gatewayApi.routes.llmInvocation.enabled turn off an LLM route while the add-on is on.Nothing, unless callers reach the gateway in-cluster.
Account bootstrapapi.accountBootstrap.enabled lets an install whose account is provisioned another way turn off the bootstrap hook Job. Unset, nothing changes.Nothing, unless you provision the account another way.
Pod annotationsOperator pod annotations per release, for example api.podAnnotations. Unset, nothing is rendered.Nothing.
Worker callback proxyNew, off by default. See Optional: enable the worker callback proxy.Nothing, unless you need it.

cert-manager images come from upstream

NVCF 1.0.4 already pulls the cert-manager controller, webhook, cainjector, acmesolver and startupapicheck images from the cert-manager chart’s upstream repositories (quay.io/jetstack), and these images do not inherit global.image. NVCF 1.1.0 had briefly defaulted them to the NVCF registry, and 1.1.1 returns to upstream. Coming from 1.0.4, the images stay the same (v1.20.5).

Keep the image.repository settings you already use for a private mirror or an air-gapped registry. Verify them before you apply the dependencies stage; otherwise the cert-manager pods try to pull from the upstream registry and fail. The settings are certManager.image.repository for the controller and certManager.webhook.image.repository, certManager.cainjector.image.repository, certManager.acmesolver.image.repository and certManager.startupapicheck.image.repository. See upstream cert-manager images.

Observability

Observability 1.1.0 is optional. Self-Managed 1.1.9 also works with the Observability 1.0.0 stack you already run, so you can upgrade Self-Managed on its own and keep Observability at 1.0.0. The 1.1.0 release changes only two charts and no images. In the default monitors it sets the job label on the worker and DCGM series and adds pod target labels (function-id, function-version-id and nca-id) to the worker series. The chart notes say federation clients such as the BYOO collector select series by that job name, and the autoscaler queries worker metrics by function identity. Upgrade Observability if you rely on either. Function scaling and federation were not exercised with Observability 1.0.0. The control-plane apply reads an Observability package when the observability profile is enabled, so either 1.0.0 or 1.1.0 must be in place before the render (see Get the 1.1.9 packages and check artifacts).

Chart1.0.01.1.0
nvcf-default-monitors0.2.00.3.0
nvcf-otel-collector0.2.00.3.0
BehaviorWhat changesWhat to do
PodMonitor job labelsThe worker and DCGM PodMonitors now set the job label to nvcf-worker and nvidia-dcgm-exporter.If your dashboards or alerts select the old job names for these series, update them.
Worker pod labelsThe worker monitor also carries the function-id, function-version-id and nca-id pod labels onto its series.Nothing, unless you limit label cardinality downstream.

Compute plane

The values that differ between Compute Plane 1.0.5 and 1.1.1:

Component1.0.51.1.1
nvca-operator chart1.28.01.37.1
NVCA agent and operator (nvcaVersion)3.10.33.18.10
Image credential helper (imageCredHelper.imageTag)0.11.30.11.4
BYOO collector sidecar (byooOtelCollector.imageTag)0.160.0-nv-0.2.90.160.0-nv-0.4.9
NVCF collector0.160.0-nv-0.2.90.160.0-nv-0.4.9

New optional settings: global.nvcaOperator.agent.workerCallbackCaPem (only for TLS worker callbacks), byoo.* sidecar resources, utils.resources, byooFluentBit and eventLedgerServiceURL. global.nvcaOperator.byooFluentBit is off by default and its image defaults to docker.io/fluent/fluent-bit:3.2. If you turn on bring-your-own observability in a disconnected install, mirror that image and set image.registry, image.repository and image.tag. Dynamo, Grove and KAI Scheduler pins are unchanged. Running function pods keep the worker images they started with; the new defaults apply to deployments created or replaced afterwards.

Known issues

  • A single-member OpenBao installation is not covered; see Check the OpenBao pods.
  • Whether Compute Plane 1.0.5 keeps working against Self-Managed 1.1.9 during the upgrade window was not checked live.

Before starting

  1. Schedule a maintenance window. You will pause new function invocations and task submissions, and a single-member Cassandra restarts once.
  2. You need an operator workstation with kubectl, helm 3.12 or later, helmfile 1.1.x (not 1.2.x), the helm-diff plugin 3.11 or later, jq, curl, Mike Farah’s yq v4, and ngc if you download packages from NGC. Helmfile 1.2.0 removed sequential execution, which these stacks need. See required tools.
  3. Keep your existing control-plane and compute-plane directories, your environments/<environment-name>.yaml, your secrets/<environment-name>-secrets.yaml, and the registration file for each GPU cluster (<cluster-name>-register-values.yaml). The upgrade reuses each cluster’s registration. Do not run register-cluster again.
  4. Choose canary workloads: one serving function, one function that reads a secret, and one running task. You will test them before and after.
  5. If you mirror images and charts to a private registry, mirror the 1.1.9 artifacts first. Get the 1.1.9 packages and check artifacts checks availability.
  6. Check the cert-manager image settings in cert-manager images come from upstream if the cluster uses a mirror.

1. Set deployment values

Set up the session

Download this read-only script into one directory. Check the starting state and Render and review the changes use it to stop an upgrade that would move an image to an older version:

Image move check

Use a dedicated shell. Replace every value in angle brackets. Run this block first, and again after any shell restart.

bash
set -euo pipefail
umask 077
export WORKDIR="$HOME/nvcf-upgrade-1.1.9"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export OLD_CONTROL_STACK="/absolute/path/to/existing/nvcf-self-managed-stack"
export OLD_COMPUTE_STACK="/absolute/path/to/existing/nvcf-compute-plane-stack"
export HELMFILE_ENV="<environment-name>"
export CONTROL_CONTEXT="<control-plane-kube-context>"
export BACKUP_DIR="$WORKDIR/backup"
mkdir -p "$WORKDIR" "$BACKUP_DIR"
# One kubeconfig per cluster, so a stray context switch cannot redirect a command.
export CONTROL_KUBECONFIG="$WORKDIR/control.kubeconfig"
kubectl config view --minify --flatten --raw --context "$CONTROL_CONTEXT" > "$CONTROL_KUBECONFIG"
kc() { kubectl --kubeconfig "$CONTROL_KUBECONFIG" "$@"; }
control_helmfile() {
(cd "$CONTROL_STACK" && HELMFILE_ENV="$HELMFILE_ENV" \
helmfile --environment default --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
redact_diff() {
awk '
/^[-+ ]+- name: / { sensitive = (toupper($0) ~ /PASSWORD|SECRET|TOKEN|KEY|CREDENTIAL/); print; next }
sensitive && /^[-+ ]+value: / { sub(/value: .*/, "value: [REDACTED]"); print; next }
{ sensitive = 0; print }'
}
wait_workloads() {
local kubeconfig="$1" namespace="$2" workload strategy
kubectl --kubeconfig "$kubeconfig" -n "$namespace" get deployments,statefulsets -o name |
while IFS= read -r workload; do
strategy=$(kubectl --kubeconfig "$kubeconfig" -n "$namespace" \
get "$workload" -o jsonpath='{.spec.updateStrategy.type}')
if [ "$strategy" = "OnDelete" ]; then
printf '%s uses OnDelete; replace its pods with the procedure in this guide\n' "$workload"
continue
fi
kubectl --kubeconfig "$kubeconfig" -n "$namespace" rollout status "$workload" --timeout=15m
done
kubectl --kubeconfig "$kubeconfig" -n "$namespace" get pods
}
declare -f kc control_helmfile redact_diff wait_workloads > "$WORKDIR/helpers.sh"
declare -p WORKDIR UPGRADE_SCRIPTS OLD_CONTROL_STACK OLD_COMPUTE_STACK HELMFILE_ENV \
CONTROL_CONTEXT BACKUP_DIR CONTROL_KUBECONFIG > "$WORKDIR/session.sh"
kc config current-context
kc get nodes

--suppress-secrets hides Kubernetes Secret data only. The OpenBao migration Job carries the Cassandra password and the registry credential as plain environment values, so a diff that includes it prints them. redact_diff replaces the value of any environment entry whose name contains PASSWORD, SECRET, TOKEN, KEY or CREDENTIAL. Pipe the dependency diff through it, and keep that output private.

All nodes must be Ready. To resume later, start a new bash, run set -euo pipefail, set WORKDIR to the directory you used above (export WORKDIR="$HOME/nvcf-upgrade-1.1.9"), then source "$WORKDIR/session.sh" and source "$WORKDIR/helpers.sh". The saved kubeconfig holds credentials, so keep $WORKDIR private.

Get the 1.1.9 packages and check artifacts

Both exact target packages must be published and available to your customer account before maintenance. Stop if either download or package-stamp check fails. The NVCF 1.1.1 manifest selects these three targets together; do not mix them with other stack versions.

Download the three target packages next to each other, in a new directory, so the relative paths in the Helmfiles resolve the same way they did for your current release.

export STACK_VERSION="1.1.9"
export COMPUTE_STACK_VERSION="1.1.1"
# Observability is optional: export OBSERVABILITY_STACK_VERSION=1.0.0 before
# this block to stay on the version you run now. An existing value is kept.
export OBSERVABILITY_STACK_VERSION="${OBSERVABILITY_STACK_VERSION:-1.1.0}"
cd "$WORKDIR"
ngc registry resource download-version "nvidia/nvcf/nvcf-self-managed-stack:${STACK_VERSION}"
mkdir -p nvcf-self-managed-stack
tar -xzf "nvcf-self-managed-stack_v${STACK_VERSION}/nvcf-self-managed-stack-${STACK_VERSION}.tar.gz" \
-C nvcf-self-managed-stack
rm -rf "nvcf-self-managed-stack_v${STACK_VERSION}"
ngc registry resource download-version "nvidia/nvcf/nvcf-compute-plane-stack:${COMPUTE_STACK_VERSION}"
mkdir -p nvcf-compute-plane-stack
tar -xzf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}/nvcf-compute-plane-stack-${COMPUTE_STACK_VERSION}.tar.gz" \
-C nvcf-compute-plane-stack
rm -rf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}"
ngc registry resource download-version "nvidia/nvcf/nvcf-observability-stack:${OBSERVABILITY_STACK_VERSION}"
mkdir -p observability
tar -xzf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}/nvcf-observability-stack-${OBSERVABILITY_STACK_VERSION}.tar.gz" \
-C observability
rm -rf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}"
export CONTROL_STACK="$WORKDIR/nvcf-self-managed-stack"
export COMPUTE_STACK="$WORKDIR/nvcf-compute-plane-stack"
export OBSERVABILITY_STACK="$WORKDIR/observability"
declare -p STACK_VERSION COMPUTE_STACK_VERSION OBSERVABILITY_STACK_VERSION CONTROL_STACK COMPUTE_STACK OBSERVABILITY_STACK >> "$WORKDIR/session.sh"
ls "$CONTROL_STACK" "$COMPUTE_STACK" "$OBSERVABILITY_STACK"

If you obtain the packages another way, extract them to these three directories. If the observability profile is enabled (observability.profile is not disabled), the control-plane Helmfile reads ../../observability/helmfile.d/01-observability.yaml.gotmpl, which resolves to $WORKDIR/observability. That directory must hold an Observability package, either 1.1.0 or your existing 1.0.0. To keep 1.0.0, export OBSERVABILITY_STACK_VERSION=1.0.0 before running the download block, or copy your existing 1.0.0 stack to that directory and skip its download. Leaving the stack only in its original location makes the render fail. With the profile disabled the directory is not read.

Check both new package stamps against the selected versions before maintenance:

check_selected_stack_versions() {
local actual
actual=$(cat "$CONTROL_STACK/VERSION") || return 1
if [ "$actual" != "$STACK_VERSION" ]; then
printf 'Self-Managed package must report %s; found %s\n' "$STACK_VERSION" "$actual" >&2
return 1
fi
actual=$(cat "$COMPUTE_STACK/VERSION") || return 1
if [ "$actual" != "$COMPUTE_STACK_VERSION" ]; then
printf 'Compute Plane package must report %s; found %s\n' "$COMPUTE_STACK_VERSION" "$actual" >&2
return 1
fi
}
check_selected_stack_versions
declare -f check_selected_stack_versions >> "$WORKDIR/helpers.sh"

Stop on a missing or mismatched stamp. Obtain a correctly stamped, qualified package through the release process; do not edit VERSION, replace immutable artifacts or substitute another stack version.

Confirm the charts and images are reachable from your registry. Set the repositories to the ones in your environment file (global.helm.sources and global.image), or to your mirror:

If your environment file uses an HTTPS Helm repository (global.helm.sources.url) instead of an OCI registry, add it under an alias first and use the alias as HELM_REPO below. The password is read from standard input so it stays out of the process list:

export HELM_REPOSITORY_CONFIG="$WORKDIR/helm-repositories.yaml"
read -r -s -p "Repository password: " HELM_PASSWORD; echo
printf '%s' "$HELM_PASSWORD" | helm repo add nvcf-check "<https-repository-url>" \
--username "<username>" --password-stdin
unset HELM_PASSWORD
helm repo update nvcf-check
export HELM_REPO="nvcf-check"

For an OCI registry, set HELM_REPO to its address instead:

export HELM_REPO="oci://<registry>/<repository>"
export IMAGE_REPO="<registry>/<repository>"
artifact_missing=0
while IFS=: read -r chart version; do
if helm show chart "$HELM_REPO/$chart" --version "$version" > /dev/null 2>&1; then
echo "ok $chart:$version"
else
echo "MISSING $chart:$version"
artifact_missing=1
fi
done <<'EOF'
helm-nvcf-cassandra:0.26.0
helm-nvcf-nats:0.9.1
helm-nvcf-openbao-server:0.37.1
helm-nvcf-pki:0.2.0
nvcf-gateway-routes:1.18.3
helm-nvcf-api:1.35.0
helm-nvcf-api-keys:1.9.4
helm-admin-token-issuer-proxy:1.7.0
helm-nvcf-ess-api:1.11.1
helm-nvcf-sis:2.6.3
helm-nvcf-nvct-api:1.11.0
helm-nvcf-notary-service:1.9.1
helm-nvcf-invocation-service:1.10.0
helm-nvcf-grpc-proxy:1.14.0
helm-nvcf-rate-limiter:1.4.1
helm-reval:1.6.0
helm-nvcf-nats-auth-callout-service:1.4.2
helm-nvcf-event-ledger:0.10.2
helm-nvcf-state-metrics:1.0.7
helm-nvcf-function-autoscaler:0.8.1
EOF
while IFS= read -r image; do
if docker manifest inspect "$IMAGE_REPO/$image" > /dev/null 2>&1; then
echo "ok $image"
else
echo "MISSING $image"
artifact_missing=1
fi
done <<'EOF'
nvcf-service-oss:1.28.4
nvcf-api-keys-service:1.10.10
nvct-service-oss:1.70.4
nvcf-ess:0.5.7
icms-service-oss:0.8.6
nvcf-notary:1.14.8
nvcf-invocation-service:0.15.1
nvcf-grpc-proxy:1.35.2
nvcf-event-ledger:0.21.2
admin-token-issuer-proxy:1.1.3
nvcf-ratelimiter:1.18.3
reval-server:0.20.4
nvcf-nats-auth-callout-service:0.8.8
nvcf-state-metrics-service:1.24.1
nvcf-function-autoscaler:1.23.2
nvcf-openbao:2.6.3-nv-1.3.9
nvcf-openbao-migrations:0.23.4
nvcf-cassandra-migrations:0.19.4
cassandra:5.0.9-nv-2.0.7
EOF
test "$artifact_missing" -eq 0

The list covers the images that change plus the data-service images. The artifact manifest has the complete list, including optional add-ons. Fix every MISSING line before continuing; a missing image shows up later as ImagePullBackOff in the middle of a stage.

Prepare the HA mode and configuration

Choose the HA mode

ModeResultNeeds
noneKeeps your current footprint: no extra replicas, no new PodDisruptionBudgets.Nothing.
preferred (default)Replica-safe services run 2 replicas. Cassandra, NATS and OpenBao run 3 members with soft anti-affinity and zone spread. Pods still schedule when capacity is short.At least 3 schedulable nodes.
enforcedSame sizing with hard placement. Pods stay Pending rather than co-locate.At least 3 nodes, 3 zones with capacity, topology.kubernetes.io/zone labels, and a WaitForFirstConsumer StorageClass.

preferred is the default and needs only enough schedulable nodes, so the checks below are enough for it. enforced refuses to place a pod when a requirement is missing, so complete every step under Prepare for enforced mode before you apply anything. If this is a single-node cluster, or you are not ready to add capacity, choose none. You can move to HA later. If your Cassandra runs a single member, first follow Expand a single-member Cassandra first. The full behavior, including zone spread, is in the high availability guide.

Check what your cluster offers:

kc get nodes -L topology.kubernetes.io/zone
kc get storageclass
kc get nodes --no-headers | wc -l

Replica-safe services roll with maxUnavailable: 0 and maxSurge: 1, so leave room for one extra pod per service.

Prepare for enforced mode

Skip this section for none and preferred. With enforced, pods stay Pending instead of co-locating, and an unmet requirement blocks the stage that creates them. Complete every step before the first apply, and keep the results in $WORKDIR:

  1. Three nodes with capacity. Each Cassandra, NATS and OpenBao member needs its own node. Check that three nodes are Ready, schedulable and not tainted against these workloads (kc get nodes), and that they have the CPU, memory and volume capacity for the extra replicas. Replica-safe services also run 2 replicas, so keep room for one extra pod per service during a rollout: free CPU and memory on at least one node must equal the largest pod request.

  2. Three zones. The nodes must be spread over three zones, each with capacity, and every node must carry the topology.kubernetes.io/zone label. List the label with the first command above. A node without the label cannot take a zone-spread pod.

  3. A WaitForFirstConsumer StorageClass. The StorageClass your Cassandra, NATS and OpenBao volumes use must show WaitForFirstConsumer as its volume binding mode (kc get storageclass). With Immediate binding, a volume can be created in a zone that has no node for its pod, and the pod stays Pending. Existing volumes keep their zone, so check where each existing Cassandra, NATS and OpenBao volume lives, and that the zone has a schedulable node:

    kc get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone
  4. Single-member Cassandra. If Cassandra runs one member, complete Expand a single-member Cassandra first before the dependency stage. The new members land in other zones, so check steps 1 to 3 for them too.

  5. Replica overrides. Remove replica overrides from the secrets file, as in Check the secrets file for replica overrides. An override conflicts with the enforced sizing and fails the render.

  6. Render first. After you set the mode, render and review the diff in Render and review the changes. Stop if a workload would not fit.

If a pod stays Pending after the apply, run kc describe pod on it and read the scheduling events before you change the mode. Moving back to preferred later is an environment change and a re-apply of the affected stage.

Carry your settings into the new packages

The new base.yaml files contain the 1.1.9 defaults and chart settings. Copy only your own changes into new environment files. Do not copy the whole old file, because that would overwrite new defaults with old ones.

# Your customizations: the lines in your environment file that differ from the old base.
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_CONTROL_STACK/environments/base.yaml" \
"$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" \
> "$WORKDIR/control-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/control-overrides.diff"
cp "$CONTROL_STACK/environments/base.yaml" "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
"$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml"

Open $WORKDIR/control-overrides.diff and re-apply each + line in $CONTROL_STACK/environments/$HELMFILE_ENV.yaml. Typical lines are your registry and chart source (global.helm.sources, global.image), Gateway settings, storage class and size, node selectors and imagePullSecrets. Do not carry over image tag or chart version overrides unless you deliberately pin them; the 1.1.9 defaults are the qualified set. The 1.1.9 base.yaml no longer pins cassandra.migrations.image.tag, so the chart default (0.19.4) applies. If you pin that tag, remove the pin or set 0.19.4 or later: an older pin skips the schema changes in Apply the dependency stage.

NVCF 1.0.4 and the 1.1.9 defaults pull the same NATS server, nats-box and config reloader images from public registries (docker.io/synadia/nats-server, docker.io/natsio/nats-box and docker.io/natsio/nats-server-config-reloader). If you mirror them, keep the nats.container.image, nats.natsBox.container.image and nats.reloader.image settings from your environment file. If you set none of these and your cluster can pull from docker.io, there is nothing to do. The NATS pods run as a non-root user, so confirm in Apply the dependency stage that each NATS pod becomes Ready.

Then set the HA mode and, if you want to turn Event Ledger off, its switch. In the environment file. This sample keeps the default, preferred; set enforced only after you complete Prepare for enforced mode:

highAvailability:
mode: preferred # none | preferred | enforced
addons:
eventLedger:
enabled: true # set false to skip the new Event Ledger release

Merge these keys into the existing addons: section; do not duplicate it.

Check the secrets file for replica overrides

Under preferred and enforced, the render fails if the secrets file sets a value that conflicts with the HA sizing, for example replicaCount: 1. Look for them now:

grep -nE 'replica(Count|s):' "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
|| echo "no replica overrides"

Remove any replica override the HA sizing should control, or choose highAvailability.mode: none. Record the mode you chose:

yq '.highAvailability' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"

2. Check the starting state and take backups

Check the starting state

Do not continue unless every check passes.

# Releases deployed
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json > "$WORKDIR/control-before.json"
jq -r '.[] | "\(.namespace)\t\(.name)\t\(.chart)\t\(.status)"' "$WORKDIR/control-before.json" | sort
test -z "$(jq -r '.[] | select(.status != "deployed") | .name' "$WORKDIR/control-before.json")" \
&& echo "all releases deployed"
# Cassandra: every member UN
for pod in $(kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra \
-o jsonpath='{.items[*].metadata.name}'); do
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool status
break
done
# Workloads ready everywhere; sis only when the ICMS release is installed
namespaces="cassandra-system nats-system vault-system nvcf api-keys ess"
if jq -e '.[] | select(.name == "sis")' "$WORKDIR/control-before.json" > /dev/null; then
namespaces="$namespaces sis"
fi
for namespace in $namespaces; do
kc -n "$namespace" get pods -o json | jq -e '
(.items | length > 0) and
any(.items[]; any(.status.conditions[]?; .type == "Ready" and .status == "True")) and
all(.items[]; .metadata.deletionTimestamp == null and
(.status.phase == "Succeeded" or
any(.status.conditions[]?; .type == "Ready" and .status == "True")))'
done
# Record replica counts and images before the upgrade
kc get deploy,sts -A -o wide > "$WORKDIR/workloads-before.txt"
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" before \
"$CONTROL_KUBECONFIG" "$WORKDIR/images-before.tsv"
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml > "$WORKDIR/receipt-before.yaml" 2>/dev/null || true

Cassandra must report every member UN. The OpenBao check function used in this guide, and again in Apply the dependency stage, is:

check_openbao() {
local pod pods
pods=$(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
-o jsonpath='{.items[*].metadata.name}') || return 1
[ -n "$pods" ] || { echo "No OpenBao server pods found" >&2; return 1; }
for pod in $pods; do
echo "== $pod =="
kc -n vault-system exec "$pod" -c openbao -- sh -c \
'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json' || return 1
done
}
declare -f check_openbao >> "$WORKDIR/helpers.sh"
check_openbao

Every OpenBao member must be initialized: true and sealed: false, with the same cluster_id. Exactly one reports is_self: true; standbys report is_self: false or omit the field. Record the cluster ID.

Run your canary workloads now and record their results. With the configured nvcf-cli:

export NVCF_CLI="/absolute/path/to/nvcf-cli"
export NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
export FUNCTION_ID="<existing-function-id>"
export VERSION_ID="<existing-version-id>"
export CANARY_REQUEST='{"input":"your-known-canary-input"}'
declare -p NVCF_CLI NVCF_CLI_CONFIG FUNCTION_ID VERSION_ID CANARY_REQUEST >> "$WORKDIR/session.sh"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
--function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
--request-body "$CANARY_REQUEST" --timeout 120
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Back up persistent state

Create restorable backups before you change anything. Follow your storage provider’s procedure for volume snapshots as well; the commands below create the application-level backups.

CASSANDRA_REPLICAS=$(kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
pod="cassandra-${ordinal}"
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool snapshot -t pre-1-1-9
done

The Cassandra snapshot stays on each persistent volume. It is not an off-cluster backup, so also copy or snapshot the volumes.

Record the migration ledger too. Each application keyspace keeps its applied migration version and dirty flag in a table of the schema_migrations keyspace. The migration jobs read it to apply only the pending migrations, so a member that cannot see a row makes a job replay a migration that already ran. The block saves the ledger from the original member now, while that member holds all of it:

read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
set +x
printf '%s\n' "$CASSANDRA_PASSWORD" |
kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
umask 077
auth_dir=$(mktemp -d)
trap '\''rm -rf "$auth_dir"'\'' EXIT
IFS= read -r password
printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
' sh "$2"
)
LEDGER_TABLES=$(cql_on cassandra-0 "SELECT table_name FROM system_schema.tables WHERE keyspace_name = 'schema_migrations';" |
tr -d ' \r' | grep -E '^[a-z][a-z0-9_]*$' | grep -vx table_name)
test -n "$LEDGER_TABLES"
for table in $LEDGER_TABLES; do
printf '%s: ' "$table"
cql_on cassandra-0 "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|'
done > "$BACKUP_DIR/schema-migrations-ledger.txt"
cat "$BACKUP_DIR/schema-migrations-ledger.txt"
unset CASSANDRA_PASSWORD

Every line must show a version and a dirty flag of False. Resolve a dirty ledger before you continue, then run this block again so the saved file records the corrected state. Do not run it again after the expansion: it would overwrite the saved ledger with what the new members can see.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from a standby fails with incomplete snapshot.

OPENBAO_ACTIVE=""
for pod in $(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
-o jsonpath='{.items[*].metadata.name}'); do
if kc -n vault-system exec "$pod" -c openbao -- sh -c \
'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
OPENBAO_ACTIVE="$pod"
fi
done
echo "active member: $OPENBAO_ACTIVE"
kc -n vault-system get secret openbao-server-root-token -o jsonpath='{.data.root_token}' | base64 -d |
kc -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-1-9.snap'
kc -n vault-system cp -c openbao "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-1-9.snap" \
"$BACKUP_DIR/openbao-pre-1-1-9.snap"
kc -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-1-9.snap
# The snapshot restores only with the unseal key and root token.
kc -n vault-system get secret openbao-server-unseal -o yaml > "$BACKUP_DIR/openbao-server-unseal.yaml"
kc -n vault-system get secret openbao-server-root-token -o yaml > "$BACKUP_DIR/openbao-server-root-token.yaml"
# Helm values of every release, and your configuration files.
mkdir -p "$BACKUP_DIR/helm-values"
jq -r '.[] | "\(.namespace) \(.name)"' "$WORKDIR/control-before.json" |
while read -r namespace release; do
helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" -n "$namespace" -o yaml \
> "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
done
cp "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" "$BACKUP_DIR/old-environment.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" "$BACKUP_DIR/old-secrets.yaml"
ls -l "$BACKUP_DIR"

The unseal and root-token files, the secrets copy and the exported values contain credentials. Protect them like the snapshot. NVCF NATS streams use memory storage, so a NATS backup holds stream metadata only. Do not continue until the Cassandra and OpenBao backups exist.

Render and review the changes

Render into a new directory, then review the diff the first stage would apply.

make -C "$CONTROL_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
KUBECONFIG_FILE="$CONTROL_KUBECONFIG" OUTPUT_DIR="$WORKDIR/control-render"
# Images the 1.1.9 render pulls; make sure each one is in your registry or mirror.
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/control-render" \
| sed -E 's/image:[[:space:]]*"?//' | sort -u

The render can include secrets, so keep it private. Check that the namespaces match your cluster and that no image lacks the registry you expect.

Compare the render with the images that run now. The check fails if any image would move to an older version, which happens when an old pin survives in your environment file:

python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" compare \
"$WORKDIR/images-before.tsv" "$WORKDIR/control-render"

Every BACKWARD line names an image and both versions. Stop on any of them. For a pin you did not set on purpose, remove it as described in Prepare the HA mode and configuration, render again and re-run the check.

Before maintenance, verify the receipt image from the rendered Job. Its default is docker.io/alpine/k8s:1.37.0; it does not follow global.image. For a mirror, set upgradeReceipt.image.registry and upgradeReceipt.image.repository, then render again. Use the resulting image reference for the check.

Run a temporary pull-test Pod in the Job’s namespace on each eligible node pool, using that image, its pull credentials and scheduling settings. Set imagePullPolicy: Always and run true instead of the receipt script. Confirm the image pulls and the Pod succeeds, then remove the test Pod. A host-side docker manifest inspect alone does not verify node access.

If you run tasks, complete the task worker image checks using this target render and the Self-Managed 1.1.9 inventory before proceeding.

3. Drain pending work

Stop new production requests and task submissions at their callers. NATS request queues use memory storage, and the dependency stage can restart NATS. Wait for queued work to finish; do not cancel it to get an empty queue.

For every affected function and version, use a key with queue_details scope (NVCF_API_KEY). The task command below uses a separate key with list_tasks scope (NVCF_NVCT_API_KEY) and your Tasks endpoint (base_nvct_url and nvct_host in the CLI configuration). Verify both with the CLI authentication settings before you start:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Queue sizes must reach zero. Record the identities and results of running tasks. Then check the control plane’s own memory streams on every NATS server, through the monitoring port 8222:

NATS_MONITOR_PORT=18223
NATS_PODS=$(kc -n nats-system get pods -o json | jq -r '.items[] |
select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
NATS_DRAIN_DIR="$WORKDIR/nats-drain-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir "$NATS_DRAIN_DIR"
for pod in $NATS_PODS; do
(
# Background kubectl itself, not the kc function: $! must be the port-forward
# process, or the trap leaves it running and the next pod cannot bind the port.
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
> "$NATS_DRAIN_DIR/$pod.log" 2>&1 &
forward_pid=$!
trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
for attempt in {1..30}; do
kill -0 "$forward_pid"
grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log" && break
sleep 1
done
grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log"
curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
"http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
> "$NATS_DRAIN_DIR/$pod.json"
jq -e '.account_details | type == "array"' "$NATS_DRAIN_DIR/$pod.json"
jq -e 'all(.account_details[];
((.stream_detail // []) | type == "array") and
all((.stream_detail // [])[];
(.config.storage | type == "string") and
(.state.messages | type == "number")))' "$NATS_DRAIN_DIR/$pod.json"
)
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
select(.config.storage == "memory") |
{name: .name, replicas: (.config.num_replicas // 1), messages: .state.messages}] as $s |
if ($s | length) == 0 then error("No memory streams found; check the NATS monitoring endpoint")
elif any($s[]; .messages > 0) then error("Memory streams not drained; do not roll NATS")
else {memory_streams: ($s | length), single_replica: ([$s[] | select(.replicas == 1)] | length)} end' \
"$NATS_DRAIN_DIR/"*.json

Every memory stream must show zero messages. A replicated stream (under preferred or enforced HA) survives a single restart, but draining it keeps the result unambiguous. A failed or slow read, a missing stream or a nonzero count blocks the upgrade. A server whose accounts hold no streams reports no stream details, which counts as empty; the final check still requires at least one memory stream across the servers. Each attempt writes to a new directory, so files from an earlier attempt cannot satisfy the check. Wait for the work to drain and repeat; do not discard messages. If your NATS monitoring is configured differently, use its endpoint and keep the same per-server check. For retained creation messages or stuck deployments, follow the recovery steps before proceeding.

4. Apply control-plane stages in order

Expand a single-member Cassandra first

Do this only if CASSANDRA_REPLICAS from Back up persistent state is 1 and you chose preferred or enforced. Skip it if Cassandra already runs 3 members, or if you chose none. Run it in the maintenance window, after you drain pending work: while system_auth is being raised and repaired, new logins can fail.

Do not let the upgrade expand Cassandra from one member to three. The chart raises the system_auth replication factor to the new member count in a hook that runs after the StatefulSet scales, and it does not repair the data. The new members then hold none of the existing login data, so administrator and service logins can fail while every member shows UN, and the dependency stage times out. Expand and repair now, on the release you run today, so the upgrade starts from a normal three-member cluster.

First, make cassandra-0 the only seed. Cassandra does not stream the existing data to a member that lists itself as a seed, so a new member that appears in the old seed list joins without the data. Set the seed list to the original member only, for the expansion, so that the new members bootstrap. The upgrade later restores the chart’s default two-seed list. The member turns Ready a few seconds before it accepts CQL connections, so the block also waits for the CQL port. It stops unless Cassandra is a single member:

test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system get statefulset cassandra \
-o jsonpath='{.spec.replicas}')" = "1" || {
echo "Expected a single-member Cassandra deployment; stop here."
exit 1
}
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system patch statefulset cassandra \
--type strategic -p '{
"spec": {
"template": {
"spec": {
"containers": [{
"name": "cassandra",
"env": [{
"name": "CASSANDRA_SEEDS",
"value": "cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local"
}]
}]
}
}
}
}'
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system rollout status \
statefulset/cassandra --timeout=10m
for attempt in $(seq 1 60); do
if [ "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
nodetool statusbinary)" = "running" ]; then break; fi
sleep 5
done
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
nodetool statusbinary)" = "running"
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
printenv CASSANDRA_SEEDS
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
nodetool status

This restarts the original single member. Before you continue, check four things: the seed list contains only cassandra-0, the member is UN, the administrator and service logins still work, and the migration ledger is unchanged from the copy that Back up persistent state saved. The block asks for the password and defines the helper itself, so it also runs in a new shell:

SEED_LIST=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
printenv CASSANDRA_SEEDS)
test "$SEED_LIST" = "cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local"
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
nodetool status | grep -c '^UN')" = "1"
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
set +x
printf '%s\n' "$CASSANDRA_PASSWORD" |
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
umask 077
auth_dir=$(mktemp -d)
trap '\''rm -rf "$auth_dir"'\'' EXIT
IFS= read -r password
printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
' sh "$2"
)
cql_on cassandra-0 "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
echo "administrator login ok on cassandra-0"
for table in $(cut -d: -f1 "$BACKUP_DIR/schema-migrations-ledger.txt"); do
printf '%s: ' "$table"
cql_on cassandra-0 "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|' || echo "no row"
done > "$BACKUP_DIR/schema-migrations-ledger-after-seed.txt"
diff "$BACKUP_DIR/schema-migrations-ledger-after-seed.txt" "$BACKUP_DIR/schema-migrations-ledger.txt"
echo "migration ledger unchanged after the seed change"
unset CASSANDRA_PASSWORD

Repeat your canary invocation from Check the starting state, which checks the service logins, and confirm that the services report no authentication errors:

AUTH_LOGS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf logs deploy/nvcf-api --since=10m)
test "$(printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true)" = "0"

Stop if any check fails, and restore from the backups in Back up persistent state with your reviewed recovery procedure. Recovering an expansion that has already failed was not validated.

Place the new members in different zones. A member’s volume stays in the zone where the member first starts, and the placement rules that the upgrade applies later cannot move it. Under enforced, two members in one zone leave a pod Pending during the upgrade. If every schedulable node has a topology.kubernetes.io/zone label, add a zone spread rule to the StatefulSet before you scale. The rule is hard under enforced and soft under preferred, so a zone without capacity never leaves Cassandra Pending under preferred. This restarts the single member once, and the upgrade replaces the rule with the chart’s own. If any node has no zone label, skip this rule: a node without the label cannot take a pod that has it.

kc get nodes -L topology.kubernetes.io/zone
HA_MODE=$(yq -r '.highAvailability.mode // "preferred"' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml")
case "$HA_MODE" in
enforced) SPREAD=DoNotSchedule ;;
preferred) SPREAD=ScheduleAnyway ;;
*) echo "Unsupported highAvailability.mode: $HA_MODE" >&2; exit 1 ;;
esac
kc -n cassandra-system patch statefulset cassandra --type merge -p "{\"spec\":{\"template\":{\"spec\":{\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"$SPREAD\",\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/name\":\"cassandra\"}}}]}}}}"
kc -n cassandra-system rollout status statefulset/cassandra --timeout=10m

Scale the StatefulSet directly. Do not set cassandra.replicaCount and apply the old stack, because that runs the chart hook with the same problem. The block reads the member count that the backup step saved, so it also works after a shell restart, and it shows the live count. Scaling to 3 is safe to repeat. Wait for all three members to be UN:

CASSANDRA_REPLICAS=$(cat "$BACKUP_DIR/cassandra-replicas")
test "$CASSANDRA_REPLICAS" -eq 1
kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}{" members now\n"}'
kc -n cassandra-system scale statefulset cassandra --replicas=3
kc -n cassandra-system rollout status statefulset/cassandra --timeout=20m
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

Every member must report UN. If a pod stays Pending, check the node and volume capacity before you continue. Confirm that each member runs in a different zone:

kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra -o wide
kc get nodes -L topology.kubernetes.io/zone

Each ALTER KEYSPACE below replaces the whole replication map with the one data center that this guide validates against the running cluster. If your keyspaces also replicate to another data center, add that entry to every ALTER here and in the application keyspace loop that follows, so the expansion does not remove it.

Raise the system_auth replication factor to 3, repair it on every member, then confirm that the administrator can log in on every member. The password is that of the cassandra administrator role, not the service role password in your secrets file. The commands read it from standard input into a temporary protected file, so do not enable shell tracing.

CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
.env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
set +x
printf '%s\n' "$CASSANDRA_PASSWORD" |
kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
umask 077
auth_dir=$(mktemp -d)
trap '\''rm -rf "$auth_dir"'\'' EXIT
IFS= read -r password
printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
' sh "$2"
)
cql_on cassandra-0 "ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
for pod in cassandra-0 cassandra-1 cassandra-2; do
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full system_auth
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
cql_on "$pod" "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
echo "administrator login ok on $pod"
done

All three members must print administrator login ok. If any login fails, stop. Do not continue to the upgrade. Restore from the backups in Back up persistent state with your reviewed recovery procedure, and contact NVIDIA support with the output.

Then raise the application keyspaces to 3 and repair them on every member. The loop alters only the keyspaces your installation has:

EXISTING_KEYSPACES=$(cql_on cassandra-0 "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in schema_migrations nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
cql_on cassandra-0 "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
else
echo "skipping $ks: not present in this installation"
fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done

Repeat your canary invocation from Check the starting state and confirm that the services report no authentication errors:

AUTH_LOGS=$(kc -n nvcf logs deploy/nvcf-api --since=10m)
printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true

The count must be 0. Raise replication for HA later in this guide repeats the application keyspace step, which is safe to run again.

The loop above also raises schema_migrations. Its replication factor is 1 on a single-member cluster, so after the expansion the migration ledger can sit on a different member from the one the migration job reads, and the job replays migrations that already ran. Compare the ledger on every member with the copy you saved in Back up persistent state. The block asks for the password and defines the helper itself, so it also runs in a new shell:

read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
set +x
printf '%s\n' "$CASSANDRA_PASSWORD" |
kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
umask 077
auth_dir=$(mktemp -d)
trap '\''rm -rf "$auth_dir"'\'' EXIT
IFS= read -r password
printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
' sh "$2"
)
for pod in cassandra-0 cassandra-1 cassandra-2; do
for table in $(cut -d: -f1 "$BACKUP_DIR/schema-migrations-ledger.txt"); do
printf '%s: ' "$table"
cql_on "$pod" "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|' || echo "no row"
done > "$BACKUP_DIR/schema-migrations-ledger-$pod.txt"
if diff "$BACKUP_DIR/schema-migrations-ledger-$pod.txt" "$BACKUP_DIR/schema-migrations-ledger.txt"; then
echo "migration ledger matches on $pod"
else
echo "migration ledger differs on $pod" >&2
exit 1
fi
done
unset CASSANDRA_PASSWORD

All three members must print migration ledger matches. If any differs, stop. Do not continue to the upgrade; repair schema_migrations again and run this block again, or restore from the backups.

Apply the dependency stage

This stage updates Cassandra, NATS, cert-manager, OpenBao and the PKI. Review the diff first, and stop on unexpected deletions or version overrides:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets | redact_diff

Apply it:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
kc -n "$namespace" get jobs
done

What you will see:

  • Cassandra restarts one pod at a time because of the config change (the server image stays at 2.0.7), and grows to 3 members under preferred or enforced. A single-member cluster is unavailable while its one pod restarts.
  • Migrations run as Helm hook jobs. They must complete successfully. The Cassandra migrations image moves to 0.19.4 from 0.17.8. The new migrations only add schema: nullable columns on sis_api.requests, sis_api.instances and the event_ledger tables, and indexes on the new sis_api creation_bucket column. They drop nothing. Existing rows keep null in the new columns, and nothing backfills them. Building the indexes reads the existing rows, so a large sis_api table can make that job run longer.
  • The OpenBao hooks (image 0.23.4) refresh the service, admin issuer and Event Ledger configuration, and add a signing role for gRPC proxy events. They only add or merge configuration and keep existing roles.
  • OpenBao shows the new template, but its pods keep running the previous template until you check them in Check the OpenBao pods. wait_workloads reports it as OnDelete.

Check that the data is intact, not only that pods are Ready:

kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
kc -n cassandra-system get jobs
kc -n vault-system get jobs

All Cassandra members must be UN and every migration job Complete. Investigate any failed job before continuing; do not disable hooks.

Check the OpenBao pods

The server image is the same as in 1.0.4 (nvcf-openbao:2.6.3-nv-1.3.9), and 1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle, so this upgrade does not repeat the plugin update or the bridge step. The chart does change, from 0.32.9 to 0.37.1, and the StatefulSet uses OnDelete: Helm updates the template, but the running pods keep the template they started with. Check whether any member still needs replacing:

kc -n vault-system get statefulset openbao-server \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}{"updated: "}{.status.updatedReplicas}{"/"}{.spec.replicas}{"\n"}'
check_openbao

The server image must be nvcf-openbao:2.6.3-nv-1.3.9 and the strategy OnDelete. If either is different, stop. With OnDelete, the StatefulSet can keep reporting the old currentRevision after every member is replaced, so read updated: instead. When it shows 3/3, every member already runs the new template; do not replace anything and continue with the core services. When it shows fewer, replace the members that are not updated, one at a time, standbys first, as below. Never delete a persistent volume claim, reinitialize OpenBao, or replace more than one member at once. This is the member replacement procedure from the OpenBao patch upgrade runbook. Use that runbook if a step fails or if you customized plugin registration, policies or PKI.

This procedure is written for the supported OpenBao deployment, a Raft cluster of several members. Replace a standby first so that leadership does not move twice. Check the member count:

kc -n vault-system get statefulset openbao-server -o jsonpath='{.spec.replicas}{"\n"}'

If it prints 1, stop. A single-member OpenBao is not a supported deployment and this guide has no procedure for it: there is no standby to replace first and no leadership handover, and OpenBao is unavailable while its one pod restarts and unseals. Do not continue on your own. Confirm the Raft snapshot from the backup step and use the OpenBao patch upgrade runbook to plan the change with your NVCF contact.

Identify the standbys (the members that do not report is_self: true) and replace one:

POD="<standby-pod-name>"
kc -n vault-system delete pod "$POD" --wait=true --timeout=3m
kc -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=5m
check_openbao

If the new pod has not appeared yet, wait for it before repeating the wait. The replacement must report version: "2.6.3", sealed: false and the original cluster_id. If it is Ready but still sealed, let auto-unseal finish and run check_openbao again. Repeat for the second standby only after all three members are healthy.

Then replace the former active member. Transfer leadership first:

ACTIVE_POD="<current-active-pod-name>"
set -o pipefail
kc -n vault-system get secret openbao-server-root-token -o json |
jq -er '.data.root_token | @base64d' |
kc -n vault-system exec -i "$ACTIVE_POD" -c openbao -- sh -ec '
IFS= read -r BAO_TOKEN
test -n "$BAO_TOKEN"
export BAO_TOKEN
export BAO_ADDR=http://127.0.0.1:8200
bao operator step-down
'
check_openbao

Wait until a 2.6.3 member reports is_self: true and the old active reports standby. A successful request alone does not confirm the transfer. Then replace the former active with the same delete, wait, check_openbao sequence. Do not enable shell tracing or print the token.

Verify all three members and their images:

check_openbao
kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{" "}{.name}{": "}{.image}{"\n"}{end}{end}'

All three must be unsealed on 2.6.3, with one active member and the original cluster ID, and updated: must show 3/3. Updating the injector affects only new pods. The services the core stage rolls in the next step pick up the new agent; roll any other injected workload through its normal deployment procedure, one service at a time, and check its authentication and rendered secrets. A single-replica service can be interrupted during that roll, so never restart all consumers together.

Apply the core services

control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets
control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
done
kc get httproutes -A

Core releases must be deployed, their workloads ready and their routes Accepted. The new event-ledger release appears in the nvcf namespace unless you turned it off. Under preferred or enforced, services scale to 2 replicas and roll with maxUnavailable: 0, so a rollout needs room for one extra pod per service. Check any add-on namespaces you enable (LLM, vanity gateway, UI) the same way.

If a release fails or a rollout stalls, see A release stays failed after a stalled rollout.

Repeat your canary invocation now. It must return the result you recorded.

Apply state metrics and autoscaling

Review the diff first, and stop on unexpected changes:

control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl diff --suppress-secrets

Then apply it:

control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl status
wait_workloads "$CONTROL_KUBECONFIG" nvcf

Keep your existing observability profile. Enabled state-metrics and autoscaler releases must be deployed and healthy. Do not turn off state metrics to get past a validation failure; the control and all profiles require it. Also reapply the observability infrastructure stage. It applies the Observability version in $WORKDIR/observability. With 1.0.0, changed configuration can still update releases. With 1.1.0, the new charts replace the 1.0.0 ones. Review the diff first:

control_helmfile --allow-no-matching-release \
-f helmfile.d/00-observability-infrastructure.yaml.gotmpl diff --suppress-secrets

Stop on unexpected changes. Then apply the stage:

control_helmfile --allow-no-matching-release \
-f helmfile.d/00-observability-infrastructure.yaml.gotmpl apply

Raise replication for HA

Skip this step if you chose highAvailability.mode: none.

After the dependency stage, Cassandra has 3 members, but your application keyspaces keep the replication factor they were created with, so losing one Cassandra node can still make data unavailable. Wait until all three Cassandra pods are Ready, then raise each keyspace to RF=3 and repair, one member at a time. The loop below alters only the keyspaces your installation has and skips the rest. It validates the deployed data-center name against the live database before any ALTER. Credentials travel over stdin into a temporary protected cqlsh credentials file. Do not enable shell tracing.

Each ALTER KEYSPACE in this loop replaces the whole replication map with the data center it validates. If your keyspaces also replicate to another data center, add that entry to every ALTER, as described in Expand a single-member Cassandra first.

CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
.env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql() (
set +x
printf '%s\n' "$CASSANDRA_PASSWORD" |
kc -n cassandra-system exec -i cassandra-0 -c cassandra -- sh -ec '
umask 077
auth_dir=$(mktemp -d)
trap '\''rm -rf "$auth_dir"'\'' EXIT
IFS= read -r password
printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
' sh "$1"
)
LIVE_DC=$(cql "SELECT JSON data_center FROM system.local;" |
sed -n '/^[[:space:]]*{/{s/^[[:space:]]*//;p;}' |
jq -ers 'if length == 1 then .[0].data_center else error("Missing data-center row") end')
test "$LIVE_DC" = "$CASSANDRA_DC"
cql "SELECT keyspace_name, replication FROM system_schema.keyspaces;"
EXISTING_KEYSPACES=$(cql "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in schema_migrations nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
cql "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
else
echo "skipping $ks: not present in this installation"
fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD

Until the repairs finish, reads at LOCAL_ONE can miss rows that exist only on the original replica. The application keyspaces must now report RF=3 for the validated data center.

JetStream streams are created with the replication factor current at creation time. The icms streams raise themselves to the configured factor when it starts. Streams created earlier by nvcf-api and invocation-service stay at 1 replica until you edit them. List them and check:

kc -n nats-system exec deploy/nats-box -- nats stream report

The Replicas column lists each member that holds a copy of the stream. A highly available stream shows three (nats-0, nats-1, nats-2); a stream created before HA shows one. Raise each stream that shows one:

STREAM="<stream-name>"
kc -n nats-system exec deploy/nats-box -- nats stream edit "$STREAM" --replicas=3 --force
kc -n nats-system exec deploy/nats-box -- nats stream info "$STREAM"

Replace <stream-name> with the name from the report, run both commands for each such stream, and confirm nats stream info shows Replicas: 3. Use 3, not 2: a 2-replica stream loses quorum when either member is down. The NATS commands run in the nats-box deployment because the nats server container does not include the CLI.

Write the upgrade receipt

Run this only after every earlier stage succeeded.

Complete the receipt image pull check before reaching this stage. Recheck it if the image or pull credentials changed.

check_selected_stack_versions
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kc -n nvcf get job upgrade-receipt --ignore-not-found
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml
test "$(kc -n nvcf get configmap nvcf-upgrade-receipt \
-o jsonpath='{.data.installed_stack_version}')" = "$STACK_VERSION"
echo "receipt matches the selected Self-Managed release"

The receipt must match the selected Self-Managed 1.1.9 release. It does not record the outcome of workload checks. The hook job is removed after ten minutes; the ConfigMap is the lasting record.

5. Verify the control plane

Check releases, versions and images:

helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kc -n nvcf get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n api-keys get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n sis get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n ess get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image

Key images you should see:

ServiceImage
nvcf-apinvcf-service-oss:1.28.4
api-keysnvcf-api-keys-service:1.10.10
nvct-apinvct-service-oss:1.70.4
ess-apinvcf-ess:0.5.7
sis (ICMS)icms-service-oss:0.8.6
notary-servicenvcf-notary:1.14.8
invocation-servicenvcf-invocation-service:0.15.1
grpc-proxynvcf-grpc-proxy:1.35.2
event-ledgernvcf-event-ledger:0.21.2
admin-issuer-proxyadmin-token-issuer-proxy:1.1.3
OpenBao servernvcf-openbao:2.6.3-nv-1.3.9
Cassandracassandra:5.0.9-nv-2.0.7

If you chose HA, confirm the replicas and spread (these are the same checks the high availability guide uses):

kc get nodes -L topology.kubernetes.io/zone
kc -n cassandra-system get pods -o wide
kc -n nats-system get pods -o wide
kc -n vault-system get pods -o wide
kc -n nvcf get deploy nvcf-api -o wide
kc -n nvcf get deploy invocation-service grpc-proxy-deployment -o wide
kc -n ess get deploy ess-api-deployment -o wide
kc get pdb -A

Replica-safe services show 2 replicas, and Cassandra, NATS and OpenBao show 3. invocation-service and grpc-proxy stay at 1 replica if you set a fixed worker callback URL. Do not resume traffic until the checks pass. Repeat the canary from Check the starting state, create and invoke a new function with a secret, and confirm queues make progress. Existing workers keep the images they started with.

6. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Use the cluster’s existing registration; the cluster, group and account identity must not change.

Set the variables

export COMPUTE_CONTEXT="<gpu-cluster-kube-context>"
export CLUSTER_NAME="<gpu-cluster-name>"
export NCA_ID="<existing-account-id>"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export COMPUTE_KUBECONFIG="$WORKDIR/compute-$CLUSTER_NAME.kubeconfig"
kubectl config view --minify --flatten --raw --context "$COMPUTE_CONTEXT" > "$COMPUTE_KUBECONFIG"
kg() { kubectl --kubeconfig "$COMPUTE_KUBECONFIG" "$@"; }
record_identity() {
kg -n nvca-operator get configmap nvcfbackend-self-managed \
-o jsonpath='{.data.cluster-dto\.yaml}' |
yq -o=json '{"clusterId": .clusterId, "clusterGroupId": .clusterGroupId,
"clusterName": .clusterName, "ncaID": .ncaID}' |
jq -eS 'if all(.[]; type == "string" and length > 0)
then . else error("Missing cluster identity") end'
}
kg get nodes
record_identity

If the cluster name or account ID was unknown, set CLUSTER_NAME and NCA_ID from this output before continuing. The registration file must exist and match that identity; every node and the NVCA backend must be healthy:

test -s "$REGISTRATION_VALUES_DIR/$CLUSTER_NAME-register-values.yaml"
declare -p COMPUTE_CONTEXT CLUSTER_NAME NCA_ID REGISTRATION_VALUES_DIR COMPUTE_KUBECONFIG >> "$WORKDIR/session.sh"
declare -f kg record_identity >> "$WORKDIR/helpers.sh"
record_identity | tee "$WORKDIR/identity-before-$CLUSTER_NAME.json"
kg -n nvca-operator get nvcfbackends
kg -n nvca-operator get pods
kg -n nvca-system get pods

Carry your settings into the compute package

# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_COMPUTE_STACK/environments/base.yaml" \
"$OLD_COMPUTE_STACK/environments/$HELMFILE_ENV.yaml" \
> "$WORKDIR/compute-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/compute-overrides.diff"
cp "$COMPUTE_STACK/environments/base.yaml" "$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml"

Re-apply each + line from compute-overrides.diff in $COMPUTE_STACK/environments/$HELMFILE_ENV.yaml (registry, image settings, observability profile and similar). Do not copy old nvcaVersion, imageTag or chart pins over the 1.1.1 defaults. Leave workerCallbackCaPem empty unless you serve worker callbacks over TLS with a private CA. If your clusters share one environment file, do this once.

Check images, render and review

make -C "$COMPUTE_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/compute-render-$CLUSTER_NAME" \
| sed -E 's/image:[[:space:]]*"?//' | sort -u

Mirror every image the list shows (NVCA 3.18.10, image credential helper 0.11.4 and the collectors 0.160.0-nv-0.4.9 are new defaults) before you apply. Then review the diff:

(cd "$COMPUTE_STACK" && HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME="$CLUSTER_NAME" \
NCA_ID="$NCA_ID" OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME" \
helmfile --environment default --kubeconfig "$COMPUTE_KUBECONFIG" \
--kube-context "$COMPUTE_CONTEXT" diff --suppress-secrets)

Expect the nvca-operator chart to move from 1.28.0 to 1.37.1 and the NVCA agent to 3.18.10. Stop if the diff shows the registration or cluster identity changing, or unexpected deletions.

Apply

make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
wait_workloads "$COMPUTE_KUBECONFIG" nvca-operator
kg -n nvca-system get pods
kg -n nvca-operator get nvcfbackends
record_identity > "$WORKDIR/identity-after-$CLUSTER_NAME.json"
diff -u "$WORKDIR/identity-before-$CLUSTER_NAME.json" "$WORKDIR/identity-after-$CLUSTER_NAME.json" \
&& echo "cluster identity unchanged"

The operator and agent must be ready, the backend must report healthy, and the identity files must match. If they differ, stop and inspect them before doing anything else. Do not install a second copy of shared observability resources on separate compute clusters; keep your existing configuration.

Confirm from the control plane that the cluster is registered and healthy:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" cluster list-registered

Existing function and task canaries must still work with their old worker images. To move a function to the new worker defaults, replace its deployment through the deployment API or CLI; updating the stack defaults alone does not change running workers or explicit per-function image overrides. Plan for the interruption while a replacement rolls.

Repeat this step for each remaining GPU cluster, with its own CLUSTER_NAME, NCA_ID, registration directory and kube context.

7. Resume workloads

Reapply the control plane and confirm it is stable:

make -C "$CONTROL_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
control_helmfile status
set +e
control_helmfile diff --suppress-secrets --detailed-exitcode
DIFF_EXIT=$?
set -e
echo "diff exit code: $DIFF_EXIT"
if [ "$DIFF_EXIT" -ne 0 ]; then
echo "STOP: the final diff exited $DIFF_EXIT. Inspect it before you resume traffic." >&2
exit "$DIFF_EXIT"
fi

The final diff must report no changes and exit 0. A nonzero exit means there are pending changes or a command error. Resolve them and rerun this gate before continuing:

  1. Repeat the canary invocations and compare IDs, results and persistent data with the values from Check the starting state.
  2. Create and invoke a new function and a new task with the inputs you normally use, including one function that reads a secret.
  3. Resume production submissions and watch queue progress and telemetry.

Keep $WORKDIR, the diffs, the render directories and your notes with the installed configuration as the upgrade record.

Troubleshooting

SymptomLikely causeWhat to do
Render fails with overrides the highAvailability sizingThe secrets file sets a replica count that conflicts with the HA sizing.Remove the override, or set highAvailability.mode: none. See Prepare the HA mode and configuration.
ImagePullBackOff in a stageAn image is not in your registry or mirror.Re-run the Get the 1.1.9 packages and check artifacts checks, mirror the missing image, and retry the stage.
Pods Pending under enforcedFewer than 3 nodes or zones with capacity, or a StorageClass without WaitForFirstConsumer.Add capacity or fix the StorageClass. preferred schedules them but does not give zone protection.
A migration job failedA dependency was not ready, or a hook could not reach its service.Read the job logs (kc -n <namespace> logs job/<name>), fix the cause, and rerun the same stage. Do not disable hooks.
OpenBao member sealed after replacementAuto-unseal has not finished.Wait, then rerun check_openbao. If it stays sealed after five minutes, follow the runbook’s recovery guidance.
Cassandra member not UNThe pod is restarting or joining.Wait for it to become Ready and rerun nodetool status; do not continue while any member is down.
The image check prints BACKWARDAn old image pin survived in your environment file.Stop. Remove pins you did not set on purpose (Prepare the HA mode and configuration) and render again.
A NATS pod is not Ready after Apply the dependency stageThe NATS pods run as a non-root user (uid 1000), or an image could not be pulled.Read the pod events and logs. Confirm the images pull from your registry or mirror (Prepare the HA mode and configuration) and that the volume is writable by group 1000.
A release fails with context deadline exceeded or stays failedA rolling update could not start its extra pod, usually for lack of CPU or memory on one node.Free capacity and rerun the stage. See A release stays failed after a stalled rollout.
Compute plane identity changedThe registration file or NCA_ID was wrong.Stop. Restore the original registration file and rerun the diff before applying.

A release stays failed after a stalled rollout

A release can fail with context deadline exceeded when a rolling update cannot start its extra pod, usually for lack of CPU or memory on one node. Stop and free capacity, then rerun the same stage apply. If the release then stays failed in helm list because Helmfile sees no diff, record it as deployed before you continue, with the name of the failed release:

control_helmfile -f helmfile.d/02-core.yaml.gotmpl --selector name=<release> sync

Use the stage file that owns the release. Then confirm that every release is deployed and every workload is ready before the next stage.

Optional: enable the worker callback proxy

The proxy is off by default, and this upgrade does not turn it on. Enable it after the upgrade passes if you run the control plane and the compute plane in different clusters and want more than one replica of the invocation service or gRPC proxy. Set workerCallbackProxy.enabled: true and workerCallbackProxy.workerDomain in your environment file. The proxy needs its own DNS name, ports and certificate; follow Worker callback proxy, including Expose the proxy and its DNS name. The stack refuses to render if the proxy is on while a fixed callback URL such as global.workerEndpoints.invocationServiceURL is set. For the replica rules, see Multi-cluster installs.

Rollback boundary

If an upgrade fails, stop and use your deployment’s reviewed recovery procedure. Do not apply your old packages unchanged as a blanket rollback. Helm rollback does not reverse stored data, does not validate a database or OpenBao downgrade, and does not revert a Cassandra replication factor you raised. Keep the OpenBao image and both JWT plugins that 1.0.4 installed; do not return to an older OpenBao image. The Cassandra migrations in 1.1.9 add columns and indexes, and a rollback does not remove them. Older services have not been tested against that schema. Restore data only through the reviewed procedure for that datastore, using the backups from Back up persistent state. After any recovery, verify the original cluster identities, old-image canaries, durable data and availability. Changing which package you use does not roll back a running deployment.