Upgrade from NVCF Self-Managed 0.6.2 to 1.0.1

Complete the 0.6.1 to 0.6.2 patch upgrade and its health checks first. The remaining steps start from healthy 0.6.2. Use the stable bundle and inventory, not a release candidate.

Before scheduling maintenance, check the task worker image and NATS recovery notes for the historical target’s known limitations.

The 1.0.1 control-plane stack is a patch release on top of 1.0.0. The compute-plane and observability targets remain at 1.0.0. The ordered migration stops in this procedure still apply.

This procedure uses staged schema changes and Kubernetes rolling updates. The maintenance window must be rehearsed for the deployment topology. The availability rehearsal below did not pass. Block control-plane mutations during the final validation and contract steps. Existing function workloads can continue running in compute clusters throughout the upgrade.

Rehearsal coverage

A 2026-10-01 diagnostic rehearsal applied the 0.6.2 source commit, later published as stable, then exercised this migration on two ARM64 k3d nodes on one host. It used one Cassandra member, three OpenBao and NATS members, and fake GPUs. Bounded migrations, service backfills, contract steps, and repeat syncs completed; the original function, deployment, worker, and storage identities were retained.

Availability did not pass: probes recorded 77 failures during Cassandra restart and recovery, 21 HTTP 401 failures during the OpenBao transition, and one invocation connection failure during full sync. Testing continued only after health and clean migration ledgers were restored. This is not evidence for an uninterrupted or production-qualified upgrade.

The rehearsal did not cover real GPUs, independent failure domains, observability, optional add-ons, successful task execution, shared storage, stale-catalog recovery, or backup restoration. It used a receipt-helper image override and an explicit control-plane profile. Verify required artifact availability and the target CLI’s profile export before the maintenance window. The worker-image, receipt-helper, profile-export distribution, and instance-timestamp follow-ups record the remaining findings from that rehearsal.

Upgrade stops

The source Cassandra migration image is 0.10.3. A healthy 0.6.2 database has already completed the NVCF GPU specification and model specification migrations and records NVCF schema version 7. The source NVCF API maps only the post-migration data model. Do not deploy an older NVCF API runtime to repeat those migrations.

The remaining Cassandra and service changes require the following ordered gates:

The migration image contains the standard migrate CLI. It supports goto V and up N for one migration path and migration table. The image’s default /app/execute_sqls.sh entrypoint always runs up for every keyspace, and Cassandra migrations 0.17.1 through 0.17.6 package NVCT migrations 4, 5, and 6 together. The released chart does not expose a command override or a per-keyspace target. Use a bounded Job that invokes the existing image with exact targets, or a chart update that exposes the same control. A new migration binary is not required.

Each bounded Job must use the preprocessed SQL path for one keyspace and the matching table in the schema_migrations keyspace. Before invoking goto, it must require the expected clean source version. Afterward, it must require the exact clean target version. Never use goto to move a ledger backward. For example, the NVCT expand Job uses the nvct_api migration path and ledger, requires version 3, invokes goto 4, and then requires version 4.

The service rollout and the data task are separate gates. NVCT 1.64.3 provides the required dual-read compatibility and the health backfill. ICMS 0.7.5 provides the instance lookup backfill. Both tasks expose explicit MIGRATE, VALIDATE, and DISABLED modes and default to DISABLED. Run migration mode to completion, switch to validation mode, require zero mismatches, and disable the task before rolling past the bridge. A completion log without a validation pass is not a safe checkpoint.

The two upgrade-only images are:

Use the immutable published versions listed above.

Helmfile upgrades one release at a time with --selector name=<release>. Helm updates the Deployment and Kubernetes performs the rolling update. Use at least two replicas for each service stop and require maxUnavailable: 0 and maxSurge: 1. The cluster must have capacity for the surge pod. A three-member Cassandra cluster is required to keep the database available while its StatefulSet rolls. A one-member Cassandra deployment has unavoidable database downtime during the server restart.

The full path is:

1. Roll the Cassandra members with hooks disabled, then apply only the bounded expand migrations.
2. Upgrade OpenBao and complete its migrations. The target ICMS and NVCT charts require the target OpenBao roles and policies.
3. Roll ICMS 0.7.5 and NVCT 1.64.3. Run and validate both data tasks, then set both modes back to DISABLED.
4. Roll ICMS 0.7.2 and NVCT 1.66.0. Apply the NVCT contract migrations while NVCT remains online.
5. Sync the remaining target control-plane and observability releases. Run the normal Cassandra release and require its migration hook to be a no-op.
6. Re-register each existing compute cluster without deleting its identity, then sync compute-plane 1.0.0 and NVCA 3.10.0.

The source and target service versions remain:

The version-number decreases for ICMS and ESS reflect component lineage changes. They are the runtimes shipped with the target stack set.

Prepare the deployment host

Before maintenance, download a packaged nvcf-cli from NGC. Before running the linked commands, replace their VERSION="1.16.2" assignment with VERSION="1.16.8" or a later release. If a qualifying binary is unavailable for your host, postpone the upgrade.

Download and extract these target bundles into new directories. The download commands are on the image mirroring page.

• nvcf-self-managed-stack 1.0.1
• nvcf-compute-plane-stack 1.0.0
• nvcf-observability-stack 1.0.0

The default observability.profile is control. With the control, compute, or all profile, the control-plane Helmfile loads ../../observability/helmfile.d/01-observability.yaml.gotmpl, so the observability bundle is required. When the profile is explicitly set to disabled, Helmfile does not load that file, and the observability components and Function Autoscaler are not installed.

For the default control profile, extract the observability bundle into a directory named observability next to the control-plane bundle directory:

<target-directory>/
|-- nvcf-self-managed-stack/
`-- observability/

The observability tarball has no top-level directory, so create the directory first:

mkdir -p <target-directory>/observability
tar -xzf <observability-tarball> -C <target-directory>/observability

Without this directory, make template and make install fail with no matches for path: ../../observability/helmfile.d/01-observability.yaml.gotmpl.

Do not extract a target bundle over the 0.6.2 directories. Keep the source configuration available for recovery.

Set variables for the deployment. Use a kubeconfig that selects only the control-plane cluster, or verify the selected context before every command.

export HELMFILE_ENV=<environment-name>
export SOURCE_CONTROL=<absolute-path-to-0.6.2-control-plane-stack>
export TARGET_CONTROL=<absolute-path-to-1.0.1-control-plane-stack>
export TARGET_COMPUTE=<absolute-path-to-1.0.0-compute-plane-stack>
export CONTROL_KUBECONFIG=<absolute-path-to-control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export NVCF_CLI=<absolute-path-to-nvcf-cli>
export NVCF_CLI_CONFIG=<absolute-path-to-nvcf-cli-config>
kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}
kubeconfig_context=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
  config current-context)
test "$kubeconfig_context" = "$CONTROL_CONTEXT"
kcp cluster-info

Use Helm 3, Helmfile 1.1.x, and kubectl 1.31 or later. The OpenBao rotation uses kubectl wait --for=create, which older kubectl versions do not support. Later steps reuse the shell variables and helper functions defined in earlier steps, so run the whole procedure in one shell, or put the variables and functions in a file and source it in every terminal you use. Authenticate Helm and the target clusters to all registries used by the target bundles. Mirror every required target artifact before the maintenance window. Use the inventory JSON attached to each target stack’s GitHub release as the artifact list. See the image mirroring page for the mirroring procedure.

Verify the selected binary with "$NVCF_CLI" version before maintenance. Record the selected CLI separately from the immutable stack inventory. The target registration flow refreshes the existing cluster metadata while preserving its identity.

The task commands in the baseline and validation steps require base_nvct_url and nvct_host in the CLI configuration file. CLI configurations created for 0.6.x often lack them. See the CLI page for the values. The function invoke and task commands also need API keys. The administrator token from nvcf-cli init has no invocation scope and no NVCT key, so function invoke fails with 403 missing requested authorities and task get fails with 401. Generate the keys once per CLI binary:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" init
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" api-key generate --expires-in 48h

The keys are stored in the CLI state, so repeat the command after switching to a different nvcf-cli binary.

Prepare the target configuration

Start from the target bundle templates. Reconcile the site-specific settings from 0.6.2 into these target files:

$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
$TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml

Do not copy the 0.6.2 files wholesale. Preserve the existing registry, storage, endpoint, credential, ingress, certificate, and scheduling values, then add settings required by the target templates.

If the deployment uses a mirror, point api.accountBootstrap.image and nats.reloader.image at that mirror. The commented examples in environments/base.yaml show the keys.

Observability moves to the separate nvcf-observability-stack in this upgrade. The 0.6.2 environment file carries the global.observability tracing and metrics settings and the optional State Metrics service, but it has no observability.profile, and the target defaults to control. With that profile, the full sync installs the bundled metrics pipeline: Prometheus Operator CRDs, the OpenTelemetry Operator and collector, VictoriaMetrics, the default control-plane monitors, and the Function Autoscaler. Carry the existing global.observability values forward, then choose a layout and set its keys in $TARGET_CONTROL/environments/$HELMFILE_ENV.yaml before rendering:

The defaults and remaining keys are in the observability bundle’s environments/base.yaml. See Observability for the profiles, component modes, and layouts.

In $TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml, replace the template placeholders. The 1.0.0 template ships nvcr.io and YOUR_ORG/YOUR_TEAM:

Registration writes the ICMS, reval, and NATS endpoints and host overrides. Non-empty icmsServiceURL, icmsServiceHostHeaderOverride, revalServiceURL, revalServiceHostHeaderOverride, natsURL, and natsHostOverride values in the environment file take precedence.

Express Dynamo support with the DynamoOperatorSupport feature gate. The compute plane also adds these feature gates by default: DynamicGPUDiscovery, InfraResourceOverhead, EnforceHelmFunctionResourceLimits, EnforceContainerFunctionResourceLimits, EnforceHelmTaskResourceLimits, and EnforceContainerTaskResourceLimits. To turn off a default gate, add -<gate> to featureGateValues.

The KAI Scheduler, Grove, and Dynamo operators are compute-plane add-ons in 1.0.0, under addons.kaiScheduler, addons.groveOperator, and addons.dynamoOperator. They are disabled by default. A disabled add-on leaves an existing operator installation unchanged. Enabling one makes the stack manage that release at the version pinned in the compute-plane bundle.

The Cassandra service-role password in the target secrets file must match the password already stored for the service-specific Cassandra roles. The same value must be supplied to the OpenBao migration:

cassandra:
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>
openbao:
  migrations:
    env:
      - name: DEFAULT_CASSANDRA_PASSWORD
        value: *cassandra_service_role_password
      - name: NVCF_API_SIDECARS_IMAGE_PULL_SECRET
        value: <base64-docker-credential>
      - name: ADMIN_CLIENT_ID
        value: ncp

To confirm the value, log in as one of the existing service roles, for example nvcf_api_app_v0:

read -rs -p "Cassandra service-role password: " SERVICE_ROLE_PASSWORD
echo
kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
  cqlsh -u nvcf_api_app_v0 -p "$SERVICE_ROLE_PASSWORD" -e 'SELECT now() FROM system.local;'
unset SERVICE_ROLE_PASSWORD

A wrong value fails with AuthenticationFailed.

If the Cassandra administrator password was changed from the chart default, also carry the existing cassandra.dbUser values into the target secrets file. Do not rotate Cassandra credentials as part of this upgrade.

Keep the live Cassandra storage class, size, and replica count unchanged. Record them before rendering the target release:

kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}'
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='replicas: {.spec.replicas}{"\n"}'

Render the target configuration before the maintenance window:

make -C "$TARGET_CONTROL" template \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"

Review the rendered storage settings, images, registries, domains, and namespaces. Do not continue if the target render changes a Cassandra volume claim template.

The rendered manifests are written to $TARGET_CONTROL/out. To review the Cassandra volume claim template, run:

grep -n -A12 'volumeClaimTemplates' \
  "$TARGET_CONTROL"/out/*-cassandra/*/templates/statefulset.yaml

Also compare the data volume mount, which is part of the pod template and not of the volume claim template. Installations that adopted Bitnami volumes during the 0.6.0 to 0.6.1 upgrade mount the volume with subPath: data:

kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="cassandra")].volumeMounts[?(@.name=="data")]}{.mountPath}{" subPath="}{.subPath}{"\n"}{end}'
grep -n -A2 '^            - name: data$' \
  "$TARGET_CONTROL"/out/*-cassandra/*/templates/statefulset.yaml

The second command prints the rendered data mount and the two lines after it. If the live StatefulSet shows subPath=data and the rendered mount has no subPath, stop. The 1.0.1 stack does not forward cassandra.persistence.subPath from the environment file to the Cassandra chart, although the chart still supports it. A member rolled without the subPath mounts the volume root, does not find its data, and starts empty.

To carry the setting through every Cassandra sync in this guide, including the full make install, add it to the target secrets file, which the stack passes to the Cassandra release unchanged:

cassandra:
  persistence:
    subPath: "data"
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>

Render again and confirm that the data mount shows subPath: data before continuing. Keep the setting for as long as the volumes retain the Bitnami layout.

To remove the dependency on subPath permanently, the relocation script in the Cassandra chart source moves the contents of the data/ directory to the volume root, which is the layout a fresh installation uses. The script runs with Cassandra stopped and each volume mounted in a helper pod, so it has its own downtime. Run it in a separate maintenance window, either before this upgrade or after it, and keep the subPath setting until the volumes have been relocated.

Verify the 0.6.2 starting state

Confirm that every source Helm release is deployed and every active controller is ready:

helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A

Confirm that Cassandra has no pending repair or schema disagreement and that all OpenBao members are initialized, unsealed, and ready. Identify the active OpenBao member. Standby members report HA Mode standby; one member reports HA Mode active.

kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done

Record the identity and version of every registered compute cluster. Run these commands with that compute cluster’s kubeconfig and context:

kubectl --kubeconfig <compute-kubeconfig> --context <compute-context> \
  get nvcfbackend <cluster-name> -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'

The source NVCA version must be 3.0.3 and agent must be healthy.

Verify the GPU inventory precondition

Resolve the Cassandra administrator password from the original installation configuration. The 0.6.2 chart does not always create a Kubernetes Secret for this password, so do not assume a Secret name. Do not enable shell tracing while the password variable is set.

read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
echo
export CASSANDRA_PASSWORD
kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
  cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
  -e 'SELECT cluster_id, cluster_status, gpus_v4, gpus_v5 FROM sis_api.cluster_by_cluster_id;'

Every active cluster row must have a nonempty gpus_v5 value. Stop if this check fails.

Record application baselines

Record at least one active function, function version, deployment, completed task, cluster ID, and cluster-group ID. Invoke the function and read the task before the upgrade. Use the same records for post-upgrade validation. Run nvcf-cli api-key generate first, as described in the deployment host section, or the invoke and task commands fail with 403 and 401.

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function get \
  --function-id <function-id> --version-id <version-id> --json
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function deploy get \
  --function-id <function-id> --version-id <version-id> --json
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id <function-id> --version-id <version-id> \
  --request-body '<sample-request>' --timeout 120 --json
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task get <task-id> --json

Back up persistent state

Create restorable backups of Cassandra, OpenBao Raft storage, NATS JetStream, the Helm values, and all target configuration files. Follow the storage provider’s backup procedure and verify the restore path before continuing.

Create a backup directory readable only by the operator, record the Cassandra replica count, then flush Cassandra and create a named snapshot on every Cassandra member before the storage-level backup:

BACKUP_DIR=<backup-directory>
mkdir -p "$BACKUP_DIR" && chmod 700 "$BACKUP_DIR"
CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
  kcp -n cassandra-system exec "$pod" -c cassandra -- \
    nodetool snapshot -t pre-1-0-1
done

The Cassandra snapshot remains on each persistent volume. It is not an off-cluster backup by itself. Copy or snapshot the volumes with your storage provider.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from a standby member fails with incomplete snapshot:

OPENBAO_ACTIVE=""
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  if kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
    OPENBAO_ACTIVE="$pod"
  fi
done
kcp -n vault-system get secret openbao-server-root-token \
  -o jsonpath='{.data.root_token}' | base64 -d | \
  kcp -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
  'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-0-1.snap'
kcp -n vault-system cp -c openbao \
  "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-0-1.snap" "$BACKUP_DIR/openbao-pre-1-0-1.snap"
kcp -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-0-1.snap

The Raft snapshot can only be restored with the unseal key. Back up the openbao-server-unseal and openbao-server-root-token Secrets in vault-system with the same protection as the snapshot.

The NVCF NATS streams use memory storage. A JetStream backup contains stream metadata only, not in-flight messages.

Export the values of every Helm release on both clusters:

mkdir -p "$BACKUP_DIR/helm-values"
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json | \
  jq -r '.[] | "\(.namespace) \(.name)"' | \
  while read -r namespace release; do
    helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" \
      -n "$namespace" -o yaml > "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
  done

Repeat the export with each compute cluster’s kubeconfig. The exported values contain credentials. Protect them like the other backups.

Do not continue until the Cassandra and OpenBao backups are complete.

Prepare the staged rollout

Do not scale the application Deployments to zero. Block configuration changes that can invalidate the upgrade baseline, but keep normal read and task traffic running for the availability test. Record the current images, replicas, and rolling strategies:

kcp -n nvcf get deployment nvcf-api nvct-api -o wide
kcp -n sis get deployment spot-instance-service -o wide
kcp -n ess get deployment -o wide
kcp -n cassandra-system get statefulset cassandra -o wide

Use at least two application replicas to limit rollout interruptions. The rollout must have an effective maxUnavailable of 0 and maxSurge of at least one pod. Kubernetes calculates those values as 0 and 1 from the default 25 percent settings when a Deployment has two replicas. A zero-downtime Cassandra server upgrade requires at least three healthy members.

Start continuous external probes before the first rollout. Exercise the same API and task reads and HTTP/gRPC invocations used for the retained baseline. Keep timestamped successes, failures, and step start/end times through final validation.

Agree on a maintenance cutoff and a clean probe observation interval during the topology rehearsal. This procedure has no guaranteed interruption duration and does not qualify as a zero-downtime upgrade.

During these transitions, a failed probe pauses advancement to the next stop. Let the running sync and required rotations finish while their health checks pass. Do not interrupt a sync or roll back solely for a probe failure. Resume only after the gate passes and every probe succeeds throughout the agreed observation interval. Keep the interruption in the availability report.

Pause further mutations and investigate if a probe fails outside these transitions, a required completion check fails, or recovery exceeds the agreed maintenance cutoff. Follow Stop and recovery criteria.

Stop 1: roll Cassandra and apply expand migrations

Upgrade only the Cassandra release with Helm hooks disabled. Preserve the existing replica count and storage settings. If the live StatefulSet mounts the data volume with subPath: data, confirm that the target secrets file carries cassandra.persistence.subPath as described in the target configuration section before running this sync. With three members, rotate one member at a time and require it to rejoin before continuing. A one-member development deployment has unavoidable database downtime during this step.

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=cassandra \
  sync --args "--no-hooks"
kcp -n cassandra-system rollout status statefulset/cassandra --timeout=30m
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

The Cassandra StatefulSet uses the RollingUpdate strategy. Kubernetes replaces the members one at a time, from the highest ordinal to the lowest, and waits for each member to become ready before it replaces the next one. Every member must report UN in nodetool status, with the same host IDs as before the roll. The rollout status command is the gate for this step. Helm waits at most 15 minutes by default. If it reports a timeout on a large database, the StatefulSet keeps rolling: wait for rollout status to finish, then run the same sync again so Helm records the release as deployed.

Check the source ledgers before the first migration Job. Use the Cassandra administrator password from the GPU inventory check:

for table in api_keys_api ess_api event_ledger nvcf_api nvcf_autoscaler nvct_api sis_api; do
  echo "== $table"
  kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
    cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
    -e "SELECT version, dirty FROM schema_migrations.$table;"
done

Replace cassandra with the administrator user if it was changed. The event_ledger table does not exist before Stop 1, so its query fails with table event_ledger does not exist.

Run migration image 0.17.6 as bounded Jobs. Require the expected clean source version before each goto call and require the exact clean target afterward. Stop at these versions:

Do not use the image’s default /app/execute_sqls.sh. It runs every available migration, including the destructive NVCT contract migrations. Each bounded Job must select one preprocessed keyspace path and its matching migration table, fail on an unexpected or dirty starting ledger, and stop immediately on the first error.

Run the bounded migration Jobs

The reference Job below uses the migration image without its default command. It reproduces the preprocessing and connection settings of /app/execute_sqls.sh for one keyspace. See the Cassandra migrations README for the image layout, environment variables, and connection parameters. In the image, the SQL files for each keyspace are in /app/keyspaces/<keyspace>/, and each keyspace records its ledger in the schema_migrations.<keyspace> table. The KS value passed to the Job selects both. The script:

1. Substitutes SERVICE_ROLE_PASSWORD and REPLICA_COUNT into the SQL files of one keyspace.
2. Reads the current ledger version and exits if it differs from the expected source. A dirty ledger reports <version> (dirty) and fails this check.
3. Runs migrate goto to the target version.
4. Exits with an error unless the ledger reports exactly the target version.

Store the Cassandra credentials in a temporary Secret. The values are read from the terminal and are not written to disk:

read -r -p "Cassandra administrator user: " CASSANDRA_USER
read -rs -p "Cassandra service-role password: " SERVICE_ROLE_PASSWORD
echo
kcp -n cassandra-system create secret generic upgrade-migration-credentials \
  --from-env-file=<(printf 'CASSANDRA_USER=%s\nCASSANDRA_PASSWORD=%s\nSERVICE_ROLE_PASSWORD=%s\n' \
    "$CASSANDRA_USER" "$CASSANDRA_PASSWORD" "$SERVICE_ROLE_PASSWORD")
unset SERVICE_ROLE_PASSWORD

Create the script ConfigMap:

kcp -n cassandra-system create configmap upgrade-bounded-migration \
  --from-file=bounded-migration.sh=/dev/stdin <<'EOF'
set -eu
mkdir -p "/tmp/keyspaces/$KS"
for f in /app/keyspaces/"$KS"/*.sql; do
  envsubst '$SERVICE_ROLE_PASSWORD $REPLICA_COUNT' < "$f" \
    > "/tmp/keyspaces/$KS/$(basename "$f")"
done
enc() {
  printf '%s' "$1" | \
    python3 -c 'import sys; from urllib.parse import quote; print(quote(sys.stdin.read(), safe=""))'
}
DB="cassandra://${CASSANDRA_HOSTS}:9042/schema_migrations?x-multi-statement=true&x-migrations-table=${KS}&username=$(enc "$CASSANDRA_USER")&password=$(enc "$CASSANDRA_PASSWORD")&consistency=LOCAL_QUORUM&protocol=4&timeout=2m&connect-timeout=30s&disable-host-lookup=true"
cur=$(migrate -path "/tmp/keyspaces/$KS" -database "$DB" version 2>&1 || true)
echo "[$KS] start version: $cur (expected $SRC)"
if [ "$SRC" = "absent" ]; then
  case "$cur" in *"no migration"*) ;; *) echo "[$KS] unexpected start version"; exit 1 ;; esac
else
  [ "$cur" = "$SRC" ] || { echo "[$KS] unexpected start version"; exit 1; }
fi
if [ "$SRC" != "$TGT" ]; then
  migrate -path "/tmp/keyspaces/$KS" -database "$DB" goto "$TGT"
fi
after=$(migrate -path "/tmp/keyspaces/$KS" -database "$DB" version 2>&1)
echo "[$KS] end version: $after (expected $TGT)"
[ "$after" = "$TGT" ] || { echo "[$KS] unexpected end version"; exit 1; }
echo "[$KS] OK"
EOF

Define a function that runs one bounded Job and waits for it. Set MIGRATIONS_IMAGE to the mirrored migration image and MIGRATIONS_PULL_SECRET to the pull secret that the Cassandra chart uses in cassandra-system. If the registry needs no authentication, remove the imagePullSecrets block from the Job.

export MIGRATIONS_IMAGE=<registry>/<repository>/nvcf-cassandra-migrations:0.17.6
export MIGRATIONS_PULL_SECRET=<pull-secret-name>
run_bounded_migration() {
  ks="$1"; src="$2"; tgt="$3"
  job="upgrade-migrate-$(printf '%s' "$ks" | tr '_' '-')-$src-$tgt"
  kcp -n cassandra-system create -f - <<EOF || return 1
apiVersion: batch/v1
kind: Job
metadata:
  name: $job
  namespace: cassandra-system
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      imagePullSecrets:
        - name: $MIGRATIONS_PULL_SECRET
      securityContext:
        fsGroup: 999
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: migrate
          image: $MIGRATIONS_IMAGE
          command: ["/bin/sh", "/scripts/bounded-migration.sh"]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            runAsUser: 999
            runAsGroup: 999
          envFrom:
            - secretRef:
                name: upgrade-migration-credentials
          env:
            - name: KS
              value: "$ks"
            - name: SRC
              value: "$src"
            - name: TGT
              value: "$tgt"
            - name: REPLICA_COUNT
              value: "$CASSANDRA_REPLICAS"
            - name: CASSANDRA_HOSTS
              value: cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local
          volumeMounts:
            - name: script
              mountPath: /scripts
              readOnly: true
      volumes:
        - name: script
          configMap:
            name: upgrade-bounded-migration
EOF
  while true; do
    result=$(kcp -n cassandra-system get "job/$job" \
      -o jsonpath='{.status.succeeded}/{.status.failed}')
    case "$result" in
      1/*) kcp -n cassandra-system logs "job/$job"; return 0 ;;
      */1) kcp -n cassandra-system logs "job/$job"; return 1 ;;
    esac
    sleep 5
  done
}

REPLICA_COUNT must equal the Cassandra replica count recorded in the backup step. The event_ledger migration creates a keyspace with this replication factor; an empty value fails the Job with ConfigurationException and leaves the event_ledger ledger dirty. The block below runs the expand Jobs in this order only when the live replica count matches the recorded value, and the && chain stops at the first failed Job:

CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')
if [ -z "$CASSANDRA_REPLICAS" ] ||
   [ "$CASSANDRA_REPLICAS" != "$(cat "$BACKUP_DIR/cassandra-replicas")" ]; then
  echo "Stop: replica count '$CASSANDRA_REPLICAS' differs from the recorded value" >&2
else
  run_bounded_migration api_keys_api 3 3 &&
  run_bounded_migration ess_api 8 9 &&
  run_bounded_migration event_ledger absent 3 &&
  run_bounded_migration nvcf_api 7 8 &&
  run_bounded_migration nvcf_autoscaler 3 4 &&
  run_bounded_migration nvct_api 3 4 &&
  run_bounded_migration sis_api 6 10
fi

Each Job log ends with [<keyspace>] OK. If a Job fails, read its log and do not continue. Do not delete or edit a dirty ledger row.

Confirm Cassandra is healthy after the server roll and expand Jobs:

kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

Do not roll either bridge until every expanded ledger has the exact version in the table and dirty = False. Rerun the source ledger query loop to check.

Stop 2: migrate OpenBao

The OpenBao StatefulSet uses OnDelete. The target release also changes the JWT plugin binary and catalog checksum. The safe sequence requires two manual pod rotations:

1. Update the StatefulSet without hooks and rotate onto the target image.
2. Start the normal Helm sync so the catalog-refresh hook runs.
3. When the target migration Job appears, rotate the pods again so each server reloads the refreshed plugin catalog.
4. Let the migration Job and Helm sync finish.

Never delete the StatefulSet, its persistent volume claims, or more than one OpenBao server pod at a time.

Update the StatefulSet without hooks

Run this command from the target control-plane bundle:

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=openbao-server \
  sync --args "--no-hooks"

Confirm the StatefulSet template uses OnDelete and the target image. Existing pods can still show the source image.

kcp -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'

Perform the first OpenBao rotation

Identify the active and standby members with bao status. Rotate every standby member first. For each standby pod, run:

wait_openbao_unsealed() {
  pod="$1"
  attempt=0
  while true; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status' 2>/dev/null || true)
    if printf '%s\n' "$status" | grep -Eq '^Initialized[[:space:]]+true$' && \
       printf '%s\n' "$status" | grep -Eq '^Sealed[[:space:]]+false$'; then
      printf '%s\n' "$status"
      return 0
    fi
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 120 ]; then
      echo "OpenBao member did not become initialized and unsealed: $pod" >&2
      return 1
    fi
    sleep 2
  done
}
export POD=<standby-pod-name>
kcp -n vault-system delete pod "$POD" --wait=true &&
  kcp -n vault-system wait "pod/$POD" --for=create --timeout=5m &&
  kcp -n vault-system wait "pod/$POD" \
    --for=condition=Ready --timeout=10m &&
  wait_openbao_unsealed "$POD"

Kubernetes readiness can become true before auto-unseal completes. The polling check must report Initialized true, Sealed false, and a healthy standby HA mode. Recheck every member, identify the current active member, and rotate it last with the same commands.

Confirm all members are ready and use the target image before continuing:

kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.containerStatuses[*].ready}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'

Refresh the catalog and run migrations

Use two terminals. Start the normal target OpenBao sync in terminal 1:

make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server

The command waits while the target catalog-refresh and migration hooks run.

In terminal 2, use a 15-minute polling deadline for the migration Job to use OpenBao migrations 0.19.5. It appears after the catalog-refresh hook succeeds. If this wait fails, inspect terminal 1 and the Job logs before continuing:

wait_openbao_migration() {
  local deadline=$((SECONDS + 900)) job image failed
  while [ "$SECONDS" -lt "$deadline" ]; do
    job=$(kcp -n vault-system get job openbao-server-migrations \
      --ignore-not-found --request-timeout=10s \
      -o jsonpath='{.spec.template.spec.containers[0].image}{"|"}{.status.conditions[?(@.type=="Failed")].status}') || return 1
    IFS='|' read -r image failed <<< "$job"
    if printf '%s\n' "$image" | grep -Eq ':0[.]19[.]5$'; then
      if [ "$failed" = True ]; then
        echo "OpenBao migration Job failed; inspect its logs before continuing" >&2
        return 1
      fi
      return 0
    fi
    sleep 2
  done
  echo "Timed out waiting for the target OpenBao migration Job" >&2
  return 1
}
wait_openbao_migration

As soon as the target migration Job appears, repeat the standby-first, active-last rotation from the preceding section. This second rotation reloads the refreshed plugin catalog. The migration Job uses OnFailure retries while the rotation is in progress.

Wait for the sync in terminal 1 to finish. Then verify the migration Job and all OpenBao members:

kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  kcp -n vault-system wait "pod/$pod" \
    --for=condition=Ready --timeout=10m
  wait_openbao_unsealed "$pod"
done

All members must be initialized, unsealed, ready, and running the same target image. Do not continue if the migration Job failed or any member is unhealthy. Verify fresh administrator login while preserving the probe API keys:

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" refresh >/dev/null

Before Stop 3, require the original invocation credentials and all retained probes to pass for the agreed observation interval. Continued authentication failures after both rotations are a recovery condition.

Stop 3: run the service backfills and contract migrations

The target service charts require the OpenBao roles and policies created at Stop 2. Use the target sis and nvct-api charts for both bridge rollouts. Set at least two replicas and keep the external availability probe running.

Set the published bridge image locations:

export BRIDGE_REGISTRY=nvcr.io
export BRIDGE_REPOSITORY=nvidia/nvcf
export ICMS_BRIDGE_TAG=0.7.5
export NVCT_BRIDGE_TAG=1.64.3
export BRIDGE_PULL_POLICY=IfNotPresent

If the deployment pulls from a mirror, set BRIDGE_REGISTRY and BRIDGE_REPOSITORY to the mirror location of the bridge images.

The backfill records are log lines written by the bridge pods. Define a helper that waits for a record and prints it:

wait_backfill_record() {
  namespace="$1"; selector="$2"; record="$3"
  attempt=0
  until kcp -n "$namespace" logs -l "$selector" --tail=-1 --prefix | \
      grep -F "$record"; do
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 60 ]; then
      echo "Backfill record not found: $record" >&2
      return 1
    fi
    sleep 10
  done
}

Backfill ICMS instance lookup columns

Define a helper that keeps every bridge override in each Helm invocation:

sync_icms_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=sis sync \
    --set sis.replicaCount=2 \
    --set sis.image.registry="$BRIDGE_REGISTRY" \
    --set sis.image.repository="$BRIDGE_REPOSITORY/icms-service-oss" \
    --set sis.image.tag="$ICMS_BRIDGE_TAG" \
    --set sis.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set sis.env.ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE="$mode"
}

Roll the bridge with the task disabled, then run the migration and validation as separate rollouts:

cd "$TARGET_CONTROL"
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
ICMS_SELECTOR=app.kubernetes.io/instance=sis,app.kubernetes.io/name=helm-nvcf-sis
sync_icms_bridge MIGRATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
wait_backfill_record sis "$ICMS_SELECTOR" \
  'Completed ICMS instance lookup backfill: mode=MIGRATE'
sync_icms_bridge VALIDATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
wait_backfill_record sis "$ICMS_SELECTOR" \
  'Completed ICMS instance lookup backfill: mode=VALIDATE'
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

Require a completed MIGRATE record with missingRequest=0, mismatched=0, and failed=0. Then require a completed VALIDATE record with migrated=0, missingRequest=0, mismatched=0, and failed=0. The validation record must report the migrated active rows as current. Keep the task DISABLED after the validation pass.

A completed record has this form. The record appears on only one ICMS replica:

Completed ICMS instance lookup backfill: mode=MIGRATE, scanned=<n>, migrated=<n>, current=<n>, skippedTerminated=<n>, missingRequest=0, mismatched=0, failed=0

Roll the stack-pinned ICMS image by removing the bridge image overrides:

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=sis sync \
  --set sis.replicaCount=2
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

Confirm that the final Deployment uses ICMS 0.7.2 and that no bridge pod remains.

Backfill NVCT task health

Define the corresponding NVCT helper:

sync_nvct_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=nvct-api sync \
    --set nvctApi.replicaCount=2 \
    --set nvctApi.image.registry="$BRIDGE_REGISTRY" \
    --set nvctApi.image.repository="$BRIDGE_REPOSITORY/nvct-service-oss" \
    --set nvctApi.image.tag="$NVCT_BRIDGE_TAG" \
    --set nvctApi.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set nvctApi.env.NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE="$mode"
}

Roll the bridge and move through the same explicit task modes:

sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
NVCT_SELECTOR=app.kubernetes.io/instance=nvct-api,app.kubernetes.io/name=helm-nvcf-nvct-api
sync_nvct_bridge MIGRATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
wait_backfill_record nvcf "$NVCT_SELECTOR" \
  'Completed NVCT health backfill: mode=MIGRATE'
sync_nvct_bridge VALIDATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
wait_backfill_record nvcf "$NVCT_SELECTOR" \
  'Completed NVCT health backfill: mode=VALIDATE'
sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

Require a completed MIGRATE record with missing=0, mismatched=0, and failed=0. Then require a completed VALIDATE record with migrated=0, missing=0, mismatched=0, failed=0, and current + withoutLegacyHealth = scanned. Re-read the retained task and record its new health value before continuing.

withoutLegacyHealth counts tasks that have no legacy health_info value, such as completed or canceled tasks, so it is usually nonzero. It is not a failure.

The routine runs on a schedule, about once a minute by default, so each mode writes more than one record and more than one NVCT replica can write one. Only the first MIGRATE record carries a nonzero migrated count; later MIGRATE records report the same rows as current. Every record must meet these conditions:

Completed NVCT health backfill: mode=MIGRATE, scanned=<n>, migrated=<n>, current=<n>, withoutLegacyHealth=<n>, missing=0, mismatched=0, failed=0

Roll the stack-pinned NVCT 1.66.0 image by removing the bridge overrides:

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=nvct-api sync \
  --set nvctApi.replicaCount=2
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

After every bridge pod is gone, run the bounded NVCT migration Job from clean version 4 to version 6. This drops health_info and health_udt. NVCT 1.66.0 remains online because it uses only the replacement health column.

Confirm that only NVCT 1.66.0 pods remain, then run the contract Job with the Stop 1 Secret, ConfigMap, and run_bounded_migration function:

kcp -n nvcf get pods -l "$NVCT_SELECTOR" \
  -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.containers[*].image}{"\n"}{end}'
run_bounded_migration nvct_api 4 6

Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal target Cassandra release. Its migration hook must be a no-op because all ledgers are already at their target versions.

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=api sync
kcp -n nvcf rollout status deployment/nvcf-api --timeout=10m
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=ess-api sync
kcp -n ess rollout status deployment/ess-api-deployment --timeout=10m
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=cassandra sync

The ESS chart renames the Deployment from ess-api-helm-nvcf-ess-api-deployment to ess-api-deployment. Helm deletes the old Deployment and creates the new one instead of rolling it.

The Cassandra migration Job log must report no change once for each of the seven keyspaces. The block below deletes the temporary migration resources only when the count is seven:

no_change=$(kcp -n cassandra-system logs job/cassandra-migrations | grep -ci 'no change')
if [ "$no_change" != 7 ]; then
  echo "Stop: expected 7 keyspaces with no change, found $no_change" >&2
else
  for job in $(kcp -n cassandra-system get jobs -o name | grep '/upgrade-migrate-'); do
    kcp -n cassandra-system delete "$job"
  done
  kcp -n cassandra-system delete secret upgrade-migration-credentials
  kcp -n cassandra-system delete configmap upgrade-bounded-migration
fi

After all steps succeed, confirm that the Cassandra migration Jobs completed and that nodetool status reports a healthy topology:

kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

Use the Cassandra administrator credentials from your deployment to query the migration ledgers. Each row must have dirty = False and the following version:

Run these CQL statements through the site’s approved Cassandra administration path:

SELECT version, dirty FROM schema_migrations.api_keys_api;
SELECT version, dirty FROM schema_migrations.ess_api;
SELECT version, dirty FROM schema_migrations.event_ledger;
SELECT version, dirty FROM schema_migrations.nvcf_api;
SELECT version, dirty FROM schema_migrations.nvcf_autoscaler;
SELECT version, dirty FROM schema_migrations.nvct_api;
SELECT version, dirty FROM schema_migrations.sis_api;

Confirm that the NVCT task schema contains only the replacement health column:

SELECT column_name, type
FROM system_schema.columns
WHERE keyspace_name = 'nvct_api'
  AND table_name = 'tasks_v2'
  AND column_name IN ('health', 'health_info');

The result must contain only health. Re-read the retained task and require the same serialized health value recorded before the contract migration. Do not continue if a ledger is dirty, a version differs, or the schema or retained data check fails.

Sync the target control plane and observability stack

The selector of the reval Deployment changed after 0.6.2. Kubernetes does not permit that field to change in place. Delete only the Deployment before the full sync. Do not uninstall the Helm release or delete its namespace.

kcp -n nvcf delete deployment reval

Sync the full target control-plane bundle. Its default control profile also installs the target observability components and Function Autoscaler. Invocation timeouts can occur while NATS and services roll. Keep probes running and let the sync finish; each Helm release has a 15-minute timeout by default.

make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"

This sync reruns the OpenBao hooks. They must finish successfully without another pod rotation. Confirm that every Helm release is deployed and every active Deployment or StatefulSet is ready:

helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A

The environment file has no replica setting for sis or nvct-api. The full sync returns spot-instance-service and nvct-api to the chart default of one replica, replacing the two replicas set during Stop 3.

Confirm the required global NATS streams still exist:

NATS_BOX_POD=$(kcp -n nats-system get pod \
  -l app.kubernetes.io/component=nats-box \
  -o jsonpath='{.items[0].metadata.name}')
for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do
  kcp -n nats-system exec "$NATS_BOX_POD" -- \
    nats --context default stream info "$stream"
done

If observability is managed by a separate stack instance, keep the self-managed ownership settings consistent with the existing deployment and sync nvcf-observability-stack 1.0.0 as the only owner. Do not install two owners for the same observability resources.

Before upgrading compute clusters, require all retained reads and invocations to pass for the agreed observation interval. Continued failures after the sync and readiness checks pass require investigation before further mutations.

Upgrade each compute cluster

Refresh authentication without clearing saved API keys, then export a fresh control-plane profile from the target bundle:

set -euo pipefail
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" = "$CONTROL_CONTEXT"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" refresh >/dev/null
KUBECONFIG="$CONTROL_KUBECONFIG" \
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" self-hosted \
  --control-plane-stack "$TARGET_CONTROL" \
  --env "$HELMFILE_ENV" \
  control-plane profile export
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" self-hosted \
  control-plane profile validate \
  --file "$TARGET_CONTROL/out/control-plane-profile.yaml" \
  --require compute-reachable

Stop if export or validation fails. Inspect the profile’s endpoints before registration; they must identify this installation and be reachable from the compute cluster. Validation checks the file, not network connectivity.

For each compute cluster, set its values and verify the selected context:

export CLUSTER_NAME=<existing-cluster-name>
export CLUSTER_REGION=<cluster-region>
export NCA_ID=<existing-nca-id>
export COMPUTE_KUBECONFIG=<absolute-path-to-compute-kubeconfig>
export COMPUTE_CONTEXT=<compute-context>
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" cluster-info

Set CLUSTER_REGION to the region of the existing registration, not a new value:

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='{.status.clusterConfig.region}{"\n"}'

Confirm that nvcr-pull-secret, or the pull secret named in the compute-plane environment, exists in nvca-operator. The operator mirrors it to nvca-system for NVCA sidecars and managed workloads. If the compute-plane registry does not require authentication and global.imagePullSecrets is [], no pull secret is needed.

Re-register the existing cluster with the target control-plane profile:

make -C "$TARGET_COMPUTE" register-cluster \
  CLUSTER_NAME="$CLUSTER_NAME" \
  CLUSTER_REGION="$CLUSTER_REGION" \
  CONTROL_PLANE_PROFILE="$TARGET_CONTROL/out/control-plane-profile.yaml" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  NVCF_CLI="$NVCF_CLI" \
  NVCF_CLI_CONFIG="$NVCF_CLI_CONFIG"

Compare the generated registration values with the IDs recorded before the upgrade. Stop if either ID changed.

Install compute-plane 1.0.0:

make -C "$TARGET_COMPUTE" install \
  CLUSTER_NAME="$CLUSTER_NAME" \
  HELMFILE_ENV="$HELMFILE_ENV" \
  NCA_ID="$NCA_ID" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT"

Wait for the operator and backend, then verify the preserved identity and target NVCA version:

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  -n nvca-operator rollout status deployment/nvca-operator --timeout=10m
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  wait nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'

The IDs must match the source values, NVCA must be 3.10.0, and agent must be healthy. Repeat this section for every compute cluster. In a single-cluster deployment, the control-plane and compute-plane kubeconfig values can point to the same cluster. In a multi-cluster deployment, always pass the compute cluster’s kubeconfig and context explicitly.

Validate the upgraded deployment

Repeat the baseline checks with the same IDs:

1. Get the source function and version. Their status must remain ACTIVE.
2. Get the source deployment. Its ID and cluster placement must be unchanged.
3. Invoke the source function and verify its response.
4. Get the source task. Its ID, name, status, and completion percentage must be unchanged.
5. Create a new task on a target compute cluster and wait for it to complete.
6. Recheck all seven Cassandra ledgers and OpenBao health.
7. Confirm every Helm release is deployed and every active controller is ready.

Use the same CLI commands from the baseline section. If the target CLI is a different binary from the one used for the baseline, run nvcf-cli api-key generate again before the invoke and task commands. function deploy get requires an API key with the deploy_function scope. A key without it fails with 403 Access Denied. A successful invocation validates the API, invocation service, gRPC proxy, NATS, NVCA, retained function deployment, and compute data path. A completed new task validates the target NVCT, ICMS, ESS, NVCA, and task-worker path.

After all checks pass, reopen control-plane mutations and end the maintenance window. Clear the shell copy of the Cassandra password:

unset CASSANDRA_PASSWORD

Stop and recovery criteria

Pause further mutations if any of these conditions occurs. Capture command output, Job logs, resource health, and probe timestamps before choosing recovery:

• A probe fails outside the planned transitions or fails their recovery gate.
• A required completion check fails, or the maintenance cutoff is reached.
• A Cassandra migration ledger is dirty or has an unexpected version.
• The rendered Cassandra StatefulSet drops a subPath that the live StatefulSet uses on the data volume mount.
• tasks_v2 still has health_info or lacks health.
• The OpenBao migration Job fails.
• An OpenBao member is sealed, uninitialized, or not ready.
• A compute cluster receives a different cluster ID or cluster-group ID.
• The required NATS streams are missing.
• A retained function, deployment, or task cannot be read after the upgrade.

Do not run a normal 0.6.2 Helm sync after the target Cassandra or OpenBao migrations have completed. The target services can write data that the source schemas and runtimes do not understand. Recovery after a completed data stop requires restoring the Cassandra, OpenBao, and NATS backups as a consistent set, restoring the 0.6.2 configuration, and then reinstalling the source stack.

Do not delete compute-cluster registrations during recovery. Restore the control plane first, then reapply the source compute-plane bundle with the preserved cluster identity.

Related

• Helmfile installation
• Self-managed cluster registration
• Control-plane operations