Upgrade from NVCF Self-Managed 0.6.1 to 0.6.2

The 0.6.2 patch refreshes OpenBao’s JWT plugin catalog before the 0.6.2 to 1.0.1 upgrade. It changes the OpenBao chart from 0.30.25 to 0.32.6 while retaining runtime 2.5.5-nv-1.3.1 and migrations 0.16.2.

An installation upgraded from 0.6.0 can still run 2.5.4-nv-1.3.0 in every OpenBao server pod. OnDelete updates the StatefulSet template without replacing pods. An unchanged template in the 0.6.2 diff does not mean those pods are current. Do not restart them at 0.6.1: refresh the catalog with 0.6.2 first, then replace the stale members using this procedure.

This procedure supports three healthy voting members, a Shamir seal, the standard /openbao/data volume layout, and one CPU architecture across every cluster node. The refresh hook has no architecture selector, so a cluster with mixed architectures is unsupported even if all OpenBao servers share one. It preserves the StatefulSet, PVCs, credentials and Raft node IDs. Use a separately tested site procedure for other layouts. Allow a maintenance window for member restarts and leadership transfer. Clients can see transient authentication failures while members restart. Never reload the plugin while binaries differ, change to RollingUpdate, force-delete a pod, or delete multiple members together. Keep other operators and automation from changing this release or node architecture during the procedure. Data PVCs must be retained, have no deletion timestamp and have no garbage-collection owner references.

1. Record and check the source deployment

Every source Helm release must be deployed, every OpenBao member initialized and unsealed, and every Cassandra migration ledger clean. Complete the 0.6.0 to 0.6.1 checks if upgrading from 0.6.0. The checks below are required; Kubernetes Ready alone accepts sealed or uninitialized responses with these charts.

Use Bash, Python 3.9 or later, jq, curl, kubectl, Helm 3 and Helmfile 1.1.x. Download the read-only validator. It reads the captured files and never contacts or changes a cluster. Run every block in the same Bash session. Supply the control-plane API base URL without /v2/nvcf, the function’s NVIDIA Cloud Account (NCA) ID, function ID and version ID, and its normal synchronous invocation URL and JSON request file. The commands construct the administrative read URL with the account in its path: /v2/nvcf/accounts/{ncaId}/functions/{functionId}/versions/{functionVersionId}. This route accepts the admin JWT without an extra account header. The ordinary /v2/nvcf/functions/... read route requires user scopes and rejects this JWT. Provide an existing API key authorized to invoke that function in a private header file containing one line: Authorization: Bearer <api-key>. Set INVOCATION_AUTH_HEADER to that file’s absolute path. Keep the key out of command arguments, shell history and logs. These probes must succeed before maintenance; use a deterministic test function whose response you can verify. Protect their outputs as application data.

set -euo pipefail
set +x
export SOURCE_CONTROL='<absolute-path-to-0.6.1-bundle>'
export TARGET_CONTROL='<new-absolute-path-for-0.6.2-bundle>'
export CONTROL_KUBECONFIG='<control-plane-kubeconfig>'
export CONTROL_CONTEXT='<intended-control-plane-context>'
export HELMFILE_ENV='<existing-environment-name>'
export BACKUP_DIR='<new-private-backup-directory>'
export OPENBAO_CHECK='<absolute-path-to-openbao-upgrade-check.py>'
export NVCF_API_URL='<control-plane-api-base-url>'
export NCA_ID='<existing-function-account-id>'
export FUNCTION_ID='<existing-function-id>'
export FUNCTION_VERSION_ID='<existing-function-version-id>'
export FUNCTION_READ_URL="${NVCF_API_URL%/}/v2/nvcf/accounts/${NCA_ID}/functions/${FUNCTION_ID}/versions/${FUNCTION_VERSION_ID}"
export FUNCTION_INVOKE_URL='<existing-function-synchronous-invocation-url>'
export INVOCATION_REQUEST='<absolute-path-to-json-request>'
export INVOCATION_AUTH_HEADER='<absolute-path-to-private-api-key-header-file>'
umask 077
mkdir "$BACKUP_DIR"
test -s "$OPENBAO_CHECK"
test -s "$INVOCATION_REQUEST"
test -s "$INVOCATION_AUTH_HEADER"
chmod 600 "$INVOCATION_AUTH_HEADER"
export UPGRADE_KUBECONFIG="$BACKUP_DIR/control.kubeconfig"
kubectl --kubeconfig "$CONTROL_KUBECONFIG" --context "$CONTROL_CONTEXT" \
  config view --minify --flatten --raw > "$UPGRADE_KUBECONFIG"
kcp() {
  kubectl --kubeconfig "$UPGRADE_KUBECONFIG" --context "$CONTROL_CONTEXT" "$@"
}
hcp() {
  helm --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" "$@"
}
CONTROL_CLUSTER_UID=$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}')
test -n "$CONTROL_CLUSTER_UID"
readonly UPGRADE_KUBECONFIG CONTROL_CONTEXT CONTROL_CLUSTER_UID
check_context() {
  test "$(kubectl --kubeconfig "$UPGRADE_KUBECONFIG" config current-context)" \
    = "$CONTROL_CONTEXT" || return 1
  test "$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}')" \
    = "$CONTROL_CLUSTER_UID" || return 1
}
check_context
hcp list --all-namespaces --all -o json > "$BACKUP_DIR/helm-releases.json"
jq -e 'length > 0 and all(.[]; .status == "deployed")' \
  "$BACKUP_DIR/helm-releases.json" >/dev/null

The private kubeconfig contains only the selected context. The bundle’s Makefile accepts a kubeconfig but no context flag, so all syncs below use this copy. Record the existing function/deployment IDs, cluster ID, worker pod UID, container IDs and restart counts for comparison after the patch.

Capture health and test fresh authentication

These helpers use the chart-managed root token over stdin for OpenBao administrative reads. Each call to fresh_auth runs two separate probes:

Create a fresh service-account JWT, log in with the existing bootstrap role, sign a new NVCF JWT, and use it for the administrative function read. Signing exercises the JWT secrets plugin; a successful read proves the API accepts the newly signed JWT. A cached token cannot establish fresh signing works.
Invoke the same function through its normal synchronous invocation URL using the existing invocation-scoped API key from INVOCATION_AUTH_HEADER. This checks invocation continuity; it does not prove fresh JWT signing works.

The nvcf-api-admin signing role grants admin:list_functions_details but not invoke_function, so its JWT cannot authorize normal invocation. Keep the existing role and scopes unchanged. Both probes must return HTTP 200 before fresh-auth-passed.txt is written. The example uses the standard bootstrap service account, role and audience. Stop if your deployment uses different identities or cannot pass either probe.

bao_admin() {
  local pod=$1
  shift
  kcp -n vault-system get secret openbao-server-root-token -o json |
    jq -er '.data.root_token | @base64d | select(length > 0)' |
    kcp -n vault-system exec -i "$pod" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      exec bao "$@"
    ' sh "$@"
}
bao_status() {
  kcp -n vault-system exec "$1" -c openbao -- sh -ec \
    'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json'
}
fresh_auth() {
  local pod=$1 directory=$2 code
  kcp -n nvcf create token nvcf-api-account-bootstrap \
    --audience=http://openbao-server.vault-system.svc.cluster.local:8200 \
    --duration=10m |
    kcp -n vault-system exec -i "$pod" -c openbao -- sh -ec '
      IFS= read -r service_jwt || test -n "$service_jwt"
      export BAO_ADDR=http://127.0.0.1:8200
      response=$(printf "%s" "$service_jwt" |
        bao write -format=json auth/jwt/login role=nvcf-api-account-bootstrap jwt=-)
      BAO_TOKEN=$(printf "%s" "$response" | jq -er ".auth.client_token | select(length > 0)")
      export BAO_TOKEN
      bao write -format=json -f services/nvcf-api/jwt/sign/nvcf-api-admin |
        jq -er "\"Authorization: Bearer \" + (.data.token | select(length > 0))"
    ' > "$directory/auth-header" || return 1
  code=$(curl --fail --silent --show-error --max-time 60 --write-out '%{http_code}' \
    --header @"$directory/auth-header" --output "$directory/function.json" \
    "$FUNCTION_READ_URL") || return 1
  test "$code" = 200 || return 1
  code=$(curl --fail --silent --show-error --max-time 120 --write-out '%{http_code}' \
    --header @"$INVOCATION_AUTH_HEADER" --header 'Content-Type: application/json' \
    --output "$directory/invocation.json" --data-binary @"$INVOCATION_REQUEST" \
    "$FUNCTION_INVOKE_URL") || return 1
  test "$code" = 200 || return 1
  date -u +%Y-%m-%dT%H:%M:%SZ > "$directory/fresh-auth-passed.txt"
}
capture_openbao() {
  local directory=$1 pod active=""
  check_context || return 1
  kcp -n vault-system get statefulset openbao-server -o json \
    > "$directory/statefulset.json" || return 1
  kcp -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
    -o json > "$directory/pods.json" || return 1
  kcp -n vault-system get pvc -o json > "$directory/pvcs.json" || return 1
  kcp get nodes -o json > "$directory/nodes.json" || return 1
  for pod in openbao-server-0 openbao-server-1 openbao-server-2; do
    bao_status "$pod" > "$directory/$pod-status.json" || return 1
    kcp -n vault-system exec "$pod" -c openbao -- cat /openbao/data/node-id \
      > "$directory/$pod-node-id.txt" || return 1
    kcp -n vault-system exec "$pod" -c openbao -- \
      sha256sum /openbao/plugins/vault-plugin-secrets-jwt \
      > "$directory/$pod-plugin.txt" || return 1
    if jq -e '.is_self == true' "$directory/$pod-status.json" >/dev/null; then
      test -z "$active" || return 1
      active=$pod
    fi
  done
  test -n "$active" || return 1
  bao_admin "$active" operator raft list-peers -format=json \
    > "$directory/peers.json" || return 1
  bao_admin "$active" read -format=json sys/storage/raft/autopilot/state \
    > "$directory/autopilot.json" || return 1
  bao_admin "$active" read -format=json sys/plugins/catalog/secret/vault-plugin-secrets-jwt \
    > "$directory/catalog.json" || return 1
}
BASELINE="$BACKUP_DIR/source"
mkdir "$BASELINE"
capture_openbao "$BASELINE"
python3 "$OPENBAO_CHECK" state "$BASELINE" > "$BASELINE/decision.json"
active=$(jq -er '.active' "$BASELINE/decision.json")
fresh_auth "$active" "$BASELINE"
bao_admin "$active" read -format=json sys/mounts > "$BASELINE/mounts.json"
jq '.members | to_entries[] | {pod:.key, images:.value.images,
  imageIDs:.value.image_ids, revision:.value.revision}' "$BASELINE/decision.json"

The validator requires exactly the expected owned pods and container statuses, three matching healthy Raft voters, a single active member, Bound data PVCs, and matching source catalog/binary checksums. Both healthy sources are supported: all live servers already on 2.5.5, or all still on 2.5.4 despite a 2.5.5 template. After the patch, only pods whose live images, image IDs, plugin checksum or revision differ from the reviewed target need replacement.

Keep this original baseline when resuming a partially completed rotation. Do not overwrite it with mixed-version state or treat a post-refresh catalog mismatch on retained old pods as a new healthy source baseline.

2. Back up state

Save the source environment values and secrets securely, then take an OpenBao Raft snapshot. Run these commands in Bash on the operator host, using the kcp helper and $BACKUP_DIR from section 1. You need jq installed locally and Kubernetes permissions to read the two Secrets, list the server pods, and execute commands in the OpenBao containers. The bao CLI runs inside the selected pod, so it does not need to be installed on your host.

The snapshot request must reach the active member because, with the bundled OpenBao 2.5.5, a request through a standby’s localhost address fails with incomplete snapshot. This example authenticates with the existing chart-managed root token; a token granting read on sys/storage/raft/snapshot is also sufficient for snapshot download. The token is passed over stdin to keep it out of command arguments and shell history. Do not enable shell tracing or print the credential files.

The standard chart uses a Shamir seal with one key share and a threshold of one, so recovery requires the single unseal key that matches the snapshot. Back up both chart-managed Secrets alongside it: the unseal key unlocks the restored state, while the root token authenticates the chart’s hooks and migrations. The root token cannot replace the unseal key. These exports contain usable credentials, so keep them in protected recovery storage and out of shared logs.

The following example runs after the required source health checks. It saves the credentials and finds the active member, but does not establish whether any expected members are missing. Complete the health checks in section 1 before upgrading. Each backup attempt uses a new private directory under $BACKUP_DIR and a unique temporary file in the active pod, so retries do not overwrite earlier backups.

(
  set +x
  set -euo pipefail
  umask 077
  # Keep each attempt separate and save the credentials needed for recovery.
  backup=$(mktemp -d "$BACKUP_DIR/openbao.XXXXXX")
  kcp -n vault-system get secret openbao-server-unseal openbao-server-root-token \
    -o json > "$backup/openbao-secrets.json"
  chmod 600 "$backup/openbao-secrets.json"
  jq -e '.items[] | select(.metadata.name == "openbao-server-unseal") |
    .data.unseal_key | strings | length > 0' \
    "$backup/openbao-secrets.json" > /dev/null || {
      echo 'Stop: the saved OpenBao unseal key is missing or empty' >&2
      exit 1
    }
  # Check the listed servers and find one active member for the snapshot.
  pods=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}')
  active=""
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi
  # Authenticate over stdin and save the snapshot in a unique pod-side file.
  snapshot=$(kcp -n vault-system exec "$active" -c openbao -- \
    mktemp /tmp/openbao-backup.XXXXXX)
  jq -er '.items[] | select(.metadata.name == "openbao-server-root-token") |
    .data.root_token | strings | @base64d | select(length > 0)' \
    "$backup/openbao-secrets.json" |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      bao operator raft snapshot save "$1"
    ' sh "$snapshot"
  # Verify the host copy before removing the pod-side file.
  kcp -n vault-system cp -c openbao "$active:$snapshot" "$backup/openbao.snap"
  test -s "$backup/openbao.snap"
  chmod 600 "$backup/openbao.snap"
  kcp -n vault-system exec "$active" -c openbao -- rm -f "$snapshot"
  printf 'OpenBao backup saved to %s\n' "$backup"
)

Continue only after the block succeeds and the printed directory contains openbao.snap and openbao-secrets.json. The commands check that the host copy is nonempty before removing the pod copy. If snapshot creation or transfer fails, stop and investigate; any pod-side temporary file is left in place.

The checks detect missing backup files and an absent or empty unseal key, but do not prove that the key matches the snapshot. Verify recovery in an isolated environment using your site’s restore procedure, and keep the backup in protected recovery storage. File permissions restrict access but do not encrypt these files. If your deployment uses a different seal or credential arrangement, follow its tested recovery procedure instead of assuming these two Secrets are sufficient.

Take a Cassandra snapshot on every member using your site’s backup procedure. Save the migration ledgers, database topology, and persistent-volume identities. Keep the source bundle and mirrored images with the recovery baseline so a restore does not depend on a later registry lookup. Verify your restore procedure before continuing. Do not put tokens, unseal material, or rendered secrets into shared logs.

3. Prepare the patch bundle

Obtain the nvcf-self-managed-stack 0.6.2 Helmfile bundle through your NVCF artifact distribution channel. Confirm access to that exact version before scheduling the upgrade. Compare its contents and source commit with the stable 0.6.2 release and its artifact inventory. The recorded source commit is 55803ad7cb647de45cf29415cfd3766225534617. Verify the bundle checksum supplied by your distribution channel, then extract it into $TARGET_CONTROL; keep $SOURCE_CONTROL intact for recovery.

Use the Image Mirroring process with the versions in that inventory. The current documentation manifest describes the 1.0.1 stack; do not substitute its artifact versions for this legacy patch.

Start with the target templates and carry your settings into these files:

$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml

Preserve registry and pull-secret settings, endpoints, certificates, storage classes and sizes, replica counts, scheduling constraints, Cassandra passwords, OpenBao configuration, and injector webhook scope. Keep enabled optional components and existing secret and cluster identities.

Keep the target global.yaml.gotmpl. It contains the runtime pinning fix, so copying the source file over it would remove part of the patch. Mirror required artifacts at the exact versions listed in the 0.6.2 inventory.

4. Review the rendered change

Render the OpenBao release into the private backup directory and inspect its diff before applying it:

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server template > "$BACKUP_DIR/openbao-target.yaml"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server diff --suppress-secrets

Check these pins against the recorded source overrides:

Component	Target version
OpenBao chart	0.32.6
Server, injected agent, auto-unseal sidecar	2.5.5-nv-1.3.1
OpenBao migrations	0.16.2
Injector	1.7.4

The StatefulSet must keep OnDelete. The catalog refresh Job must run before migrations. Review changes to RBAC, hooks, webhook configuration, probes, and pod templates. Stop if the diff changes a volume, identity, runtime image, or credential that you did not intend to change.

Record the reviewed target image

Convert the rendered resources to JSON without creating them. Extract the one server StatefulSet and verify the migration image. Do not use chart 0.32.6’s newer default runtime or migration versions.

TARGET_STATE="$BACKUP_DIR/target"
mkdir "$TARGET_STATE"
kcp create --dry-run=client --validate=false -f "$BACKUP_DIR/openbao-target.yaml" \
  -o json | jq -s '.' > "$TARGET_STATE/rendered.json"
jq -e '[.. | objects | select(.kind? == "StatefulSet" and .metadata.name? == "openbao-server")] |
  if length == 1 then .[0] else error("expected one OpenBao StatefulSet") end' \
  "$TARGET_STATE/rendered.json" > "$TARGET_STATE/rendered-statefulset.json"
jq -e '[.. | objects | select(.kind? == "Job" and .metadata.name? == "openbao-server-migrations") |
  .spec.template.spec.containers[].image] |
  length == 1 and all(.[]; endswith(":0.16.2"))' "$TARGET_STATE/rendered.json" >/dev/null
TARGET_IMAGE=$(jq -er '.spec.template.spec.containers[] |
  select(.name == "openbao") | .image | select(endswith(":2.5.5-nv-1.3.1"))' \
  "$TARGET_STATE/rendered-statefulset.json")
architecture=$(jq -er '.architecture' "$BASELINE/decision.json")

Run a short image-check Job on the same architecture. It has no credentials or Raft volumes and only hashes the plugin. Its resolved image ID and plugin hash are the expected values for server replacements; this also checks image-pull access before changing the release. The pod inherits the reviewed image pull Secrets. Use unique generated names and retain the result with the backup.

jq --arg image "$TARGET_IMAGE" --arg arch "$architecture" '{
  apiVersion:"batch/v1", kind:"Job",
  metadata:{generateName:"openbao-upgrade-image-",namespace:"vault-system"},
  spec:{backoffLimit:0,activeDeadlineSeconds:300,template:{spec:{
    restartPolicy:"Never",automountServiceAccountToken:false,
    nodeSelector:{"kubernetes.io/arch":$arch},
    imagePullSecrets:(.spec.template.spec.imagePullSecrets // []),
    containers:[{name:"probe",image:$image,imagePullPolicy:"Always",
      command:["sha256sum","/openbao/plugins/vault-plugin-secrets-jwt"]}]
  }}}}' "$TARGET_STATE/rendered-statefulset.json" > "$TARGET_STATE/image-job.json"
check_context
image_job=$(kcp create -f "$TARGET_STATE/image-job.json" -o jsonpath='{.metadata.name}')
kcp -n vault-system wait "job/$image_job" --for=condition=Complete --timeout=5m
kcp -n vault-system get pods -l "job-name=$image_job" -o json > "$TARGET_STATE/image-pods.json"
kcp -n vault-system logs "job/$image_job" > "$TARGET_STATE/plugin.txt"
plugin_sha=$(awk 'NR==1 {print $1}' "$TARGET_STATE/plugin.txt")
image_id=$(jq -er '.items | select(length == 1) | .[0].status.containerStatuses |
  select(length == 1) | .[0] | select(.state.terminated.exitCode == 0) | .imageID' \
  "$TARGET_STATE/image-pods.json")
jq -n --arg image "$TARGET_IMAGE" --arg image_id "$image_id" \
  --arg plugin_sha256 "$plugin_sha" \
  '{image:$image,image_id:$image_id,plugin_sha256:$plugin_sha256}' > "$TARGET_STATE/image.json"

5. Refresh the catalog, then replace stale members

Apply the selected release and prove the hooks ran

The refresh hook uses the target image and registers its checksum without reloading existing plugins. Migrations run afterward. The old active process can keep serving during this transition, but an old plugin process that restarts can fail against the new catalog. Do not leave the upgrade here or start another maintenance operation.

Start a bounded Job watch before sync. It records completion even when Helm deletes the successful refresh Job. The validator requires a fresh completed refresh and a different, completed migrations Job created after it. Missing watch evidence stops rotation; a successful Helm status alone is insufficient.

sync_openbao() {
  local old_uid started watcher
  SYNC_DIR=$(mktemp -d "$BACKUP_DIR/sync.XXXXXX") || return 1
  kcp get nodes -o json > "$SYNC_DIR/nodes.json" || return 1
  jq -e --arg arch "$(jq -er '.architecture' "$BASELINE/decision.json")" \
    '.items | length > 0 and all(.[]; .metadata.labels["kubernetes.io/arch"] == $arch)' \
    "$SYNC_DIR/nodes.json" >/dev/null || return 1
  old_uid=$(kcp -n vault-system get job openbao-server-migrations \
    -o jsonpath='{.metadata.uid}' --ignore-not-found) || return 1
  started=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  kcp -n vault-system get jobs --watch-only --output-watch-events -o json \
    --request-timeout=20m > "$SYNC_DIR/hook-events.json" 2> "$SYNC_DIR/hook-watch.log" &
  watcher=$!
  sleep 2
  kill -0 "$watcher" || return 1
  if ! check_context || ! make -C "$TARGET_CONTROL" install HELMFILE_ENV="$HELMFILE_ENV" \
    KUBECONFIG_FILE="$UPGRADE_KUBECONFIG" HELMFILE_SELECTOR=name=openbao-server; then
    kill "$watcher" 2>/dev/null || true
    wait "$watcher" || true
    return 1
  fi
  sleep 2
  kill "$watcher" 2>/dev/null || true
  wait "$watcher" || true
  python3 "$OPENBAO_CHECK" hooks "$SYNC_DIR/hook-events.json" \
    --old-uid "$old_uid" --started "$started" > "$SYNC_DIR/hooks-passed.json" || return 1
  hcp list -n vault-system --all --filter '^openbao-server$' -o json \
    > "$SYNC_DIR/helm.json" || return 1
  jq -e 'length == 1 and .[0].name == "openbao-server" and
    .[0].namespace == "vault-system" and .[0].status == "deployed" and
    .[0].chart == "helm-nvcf-openbao-server-0.32.6"' \
    "$SYNC_DIR/helm.json" >/dev/null || return 1
  kcp -n vault-system logs job/openbao-server-migrations > "$SYNC_DIR/migrations.log" || return 1
}
sync_openbao
kcp -n vault-system get statefulset openbao-server -o json > "$TARGET_STATE/statefulset.json"

If sync fails, stop before deletion. The function stops its local watcher and retains evidence in $SYNC_DIR. Do not skip hooks or repeatedly resync while replacing servers.

Guard each replacement

The gate below checks the original PVC/node identities, every container and member, the frozen target specification, and the target catalog. An unchanged source pod is allowed temporarily; every replacement must have the target image ID, revision, version and plugin hash. A changed but incorrect pod is rejected. Each successful gate also reads the function with a newly signed JWT and invokes it separately with the existing API key.

openbao_gate() {
  local stage=$1
  shift
  CHECK_DIR=$(mktemp -d "$BACKUP_DIR/check.XXXXXX") || return 1
  capture_openbao "$CHECK_DIR" || return 1
  python3 "$OPENBAO_CHECK" state "$CHECK_DIR" --stage "$stage" \
    --baseline "$BASELINE" --target "$TARGET_STATE" "$@" \
    > "$CHECK_DIR/decision.json" || return 1
  ACTIVE=$(jq -er '.active' "$CHECK_DIR/decision.json") || return 1
  fresh_auth "$ACTIVE" "$CHECK_DIR" || return 1
}
openbao_gate transition
if jq -e '.stale | length > 0' "$CHECK_DIR/decision.json" >/dev/null; then
  original_active=$(jq -er '.active' "$BASELINE/decision.json")
  original_active_uid=$(jq -er --arg pod "$original_active" '.members[$pod].uid' \
    "$BASELINE/decision.json")
else
  original_active=$ACTIVE
  original_active_uid=$(jq -er --arg pod "$original_active" '.members[$pod].uid' \
    "$CHECK_DIR/decision.json")
fi
openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
jq '{active,stale}' "$CHECK_DIR/decision.json"

If stale is empty, skip replacement and run the final checks. This handles already-current sources and a rerun after convergence. Otherwise, discover stale standbys and replace one at a time. The commands recheck the original active UID and leadership before every deletion. A replacement failure or unexpected leader change stops the loop before another deletion.

while :; do
  openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
  pod=$(jq -r '.active as $active | [.stale[] | select(. != $active)] | first // empty' \
    "$CHECK_DIR/decision.json")
  [ -n "$pod" ] || break
  old_uid=$(jq -er --arg pod "$pod" '.members[$pod].uid' "$CHECK_DIR/decision.json")
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" = "$old_uid"
  bao_status "$pod" | jq -e '.initialized == true and .sealed == false and .ha_enabled == true and
    (.is_self == false or (has("is_self") | not))' >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$BACKUP_DIR/$old_uid-before.log"
  check_context
  kcp -n vault-system delete pod "$pod" --grace-period=60 --wait=true --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=create --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=condition=Ready --timeout=10m
  unsealed=false
  for attempt in $(seq 1 120); do
    if status=$(bao_status "$pod") &&
      jq -e '.initialized and (.sealed == false)' <<< "$status" >/dev/null; then
      unsealed=true
      break
    fi
    sleep 5
  done
  test "$unsealed" = true
  openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
  jq -e --arg pod "$pod" --arg old "$old_uid" \
    '.members[$pod].uid != $old and (.stale | index($pod) == null)' \
    "$CHECK_DIR/decision.json" >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$CHECK_DIR/replacement.log"
done

Inspect each replacement’s startup and previous pod’s saved logs. Require no plugin checksum errors, failed secret mounts, or forced termination. The 60-second deletion grace allows orderly shutdown; do not shorten it to force progress. Keep the two upgraded standbys healthy before the last replacement.

Transfer leadership by replacing the old active

When only the old active remains stale, both standbys have passed the target checks. Gracefully terminate that old active and let an upgraded member take over. A standalone bao operator step-down leaves the old binary eligible to regain leadership, so it is not used here. Never force a new-version leader to fail back to an old version.

openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
remaining=$(jq '.stale | length' "$CHECK_DIR/decision.json")
if [ "$remaining" -ne 0 ]; then
  jq -e --arg active "$original_active" '.stale == [$active]' "$CHECK_DIR/decision.json" >/dev/null
  pod=$original_active
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" = "$original_active_uid"
  bao_status "$pod" | jq -e '.initialized and (.sealed == false) and .is_self' >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$BACKUP_DIR/old-active.log"
  check_context
  kcp -n vault-system delete pod "$pod" --grace-period=60 --wait=true --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=create --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=condition=Ready --timeout=10m
  unsealed=false
  for attempt in $(seq 1 120); do
    if status=$(bao_status "$pod") &&
      jq -e '.initialized and (.sealed == false)' <<< "$status" >/dev/null; then
      unsealed=true
      break
    fi
    sleep 5
  done
  test "$unsealed" = true
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" != "$original_active_uid"
fi
openbao_gate complete
cp "$CHECK_DIR/decision.json" "$BACKUP_DIR/converged.json"

The final gate requires three target members, one active member, the original PVCs and node IDs, matching catalog/binary hashes, and fresh authentication and invocation. Every pod’s controller-revision-hash must equal updateRevision, the controller must have observed the generation, and updatedReplicas must be three. currentRevision can remain historical with OnDelete; do not rotate again just to make the two StatefulSet status revision fields equal.

6. Validate injection and repeat the upgrade checks

Compare the function/deployment records and invocation result with the baseline, including original worker pod UID, container IDs and restart counts. Record transient probe failures even if final checks pass. At every gate, the function read validates a freshly signed JWT, and the separate API-key request checks invocation. Retain the timestamped result files privately.

Admit a new workload through the injector

This short test uses the existing nvcf-api identity in namespace nvcf and its signing policy. Its application container checks that the injected init container wrote a nonempty signed token, without printing it. It creates only a uniquely named ConfigMap and Pod. Confirm that this namespace is within the existing webhook scope; do not broaden webhook selectors for the test.

injection_dir=$(mktemp -d "$BACKUP_DIR/injection.XXXXXX")
cat > "$injection_dir/config-init.hcl" <<'HCL'
exit_after_auth = true
vault { address = "http://openbao-server.vault-system.svc.cluster.local:8200" }
auto_auth {
  method "jwt" {
    mount_path = "auth/jwt"
    config = {
      role = "nvcf-api"
      path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      remove_jwt_after_reading = false
    }
  }
  sink "file" { config = { path = "/home/vault/.vault-token" } }
}
template {
  destination = "/vault/secrets/probe.jwt"
  contents = "{{ with secret \"services/nvcf-api/jwt/sign/nvcf-api\" }}{{ .Data.token }}{{ end }}"
}
HCL
injection_name="openbao-upgrade-$(date -u +%Y%m%d%H%M%S)"
check_context
kcp -n nvcf create configmap "$injection_name" \
  --from-file=config-init.hcl="$injection_dir/config-init.hcl" \
  --from-file=config.hcl="$injection_dir/config-init.hcl"
jq -n --arg name "$injection_name" --arg image "$TARGET_IMAGE" '{
  apiVersion:"v1",kind:"Pod",metadata:{name:$name,namespace:"nvcf",annotations:{
    "vault.hashicorp.com/agent-inject":"true",
    "vault.hashicorp.com/agent-pre-populate-only":"true",
    "vault.hashicorp.com/agent-configmap":$name,
    "vault.hashicorp.com/copy-volume-mounts":"probe"}},
  spec:{restartPolicy:"Never",serviceAccountName:"nvcf-api",
    volumes:[{name:"token",projected:{sources:[{serviceAccountToken:{path:"token",
      expirationSeconds:600,audience:"http://openbao-server.vault-system.svc.cluster.local:8200"}}]}}],
    containers:[{name:"probe",image:$image,command:["sh","-ec","test -s /vault/secrets/probe.jwt"],
      volumeMounts:[{name:"token",mountPath:"/var/run/secrets/kubernetes.io/serviceaccount",readOnly:true}]}]}}
' > "$injection_dir/pod.json"
# Preserve the same pull Secrets as the reviewed server image.
jq --slurpfile target "$TARGET_STATE/rendered-statefulset.json" \
  '.spec.imagePullSecrets = ($target[0].spec.template.spec.imagePullSecrets // [])' \
  "$injection_dir/pod.json" > "$injection_dir/pod-with-pull-secrets.json"
kcp create -f "$injection_dir/pod-with-pull-secrets.json"
kcp -n nvcf wait "pod/$injection_name" --for=jsonpath='{.status.phase}'=Succeeded --timeout=5m
kcp -n nvcf get pod "$injection_name" -o json > "$injection_dir/result.json"
jq -e 'any(.status.initContainerStatuses[]?;
  .name == "vault-agent-init" and .state.terminated.exitCode == 0) and
  any(.status.containerStatuses[]?; .name == "probe" and .state.terminated.exitCode == 0)' \
  "$injection_dir/result.json" >/dev/null

The matching image pull Secrets must already exist in nvcf. A completed Pod without the injected init container does not pass. Preserve its result, then remove only these generated test resources when finished:

kcp -n nvcf delete pod "$injection_name"
kcp -n nvcf delete configmap "$injection_name"
kcp -n vault-system delete job "$image_job"

Repeat sync and prove no additional replacement is needed

Repeat the selected sync. The same function creates a new evidence directory, captures the previous migrations UID, and requires fresh completed hooks. The complete gate must select no stale members and preserve every current UID.

sync_openbao
openbao_gate complete
jq -S '.members | map_values(.uid)' "$BACKUP_DIR/converged.json" > "$BACKUP_DIR/uids-converged.json"
jq -S '.members | map_values(.uid)' "$CHECK_DIR/decision.json" > "$CHECK_DIR/uids-selected-repeat.json"
diff -u "$BACKUP_DIR/uids-converged.json" "$CHECK_DIR/uids-selected-repeat.json"
jq -e '.stale | length == 0' "$CHECK_DIR/decision.json" >/dev/null

Review the full target diff, then apply the remaining changes and repeat the full sync. Explicitly clear the selector. Each pass must create a new completed migration Job, retain the same server UIDs, and pass fresh authentication.

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" diff --suppress-secrets
for pass in 1 2; do
  old_uid=$(kcp -n vault-system get job openbao-server-migrations -o jsonpath='{.metadata.uid}')
  check_context
  kcp get nodes -o json | jq -e --arg arch "$architecture" \
    '.items | length > 0 and all(.[]; .metadata.labels["kubernetes.io/arch"] == $arch)' >/dev/null
  make -C "$TARGET_CONTROL" install HELMFILE_ENV="$HELMFILE_ENV" \
    KUBECONFIG_FILE="$UPGRADE_KUBECONFIG" HELMFILE_SELECTOR=
  kcp -n vault-system wait job/openbao-server-migrations --for=condition=Complete --timeout=15m
  new_uid=$(kcp -n vault-system get job openbao-server-migrations -o jsonpath='{.metadata.uid}')
  test "$new_uid" != "$old_uid"
  openbao_gate complete
  jq -S '.members | map_values(.uid)' "$CHECK_DIR/decision.json" > "$BACKUP_DIR/uids-full-$pass.json"
  diff -u "$BACKUP_DIR/uids-converged.json" "$BACKUP_DIR/uids-full-$pass.json"
  jq -e '.stale | length == 0' "$CHECK_DIR/decision.json" >/dev/null
done

Repeat the new-injection block with its new unique resource names. Verify Cassandra ledgers remain clean, credentials and records are retained, and the original function still serves before continuing to 1.0.1. A rerun of the replacement decision must select no pods; do not rotate already-current pods.

Record the topology, released image IDs, member replacement and leadership transfer observations, and probe results from your rehearsal before production use.

Stop and recover

Stop on a failed hook or health/authentication gate, unexpected leader or UID change, missing member/PVC, changed target, dirty migration ledger, or failed replacement. Preserve evidence, credentials and all persistent volumes. Do not retry pod deletion to work around an unexplained failure.

Retain the original baseline and frozen target when resuming after a successful standby replacement. Rerun the transition gate with the saved original leader and UID; it selects only remaining stale members. After an unexpected leader change, establish the current version and recovery state before proceeding. Never force leadership back to an old binary.

Once refresh changes the catalog, a Helm rollback does not restore that state or the OnDelete pods. After a new-version member becomes active, downgrading its binary against the current data store is not a supported rollback. Prefer repairing forward; otherwise use your site’s tested coordinated snapshot, matching credential and source-runtime restore procedure. This guide does not provide destructive restore automation or claim that backup transfer proves recoverability. Keep snapshot recovery and other topologies separately qualified.