Upgrade from NVCF Self-Managed 0.6.1 to 0.6.2

The 0.6.2 Helmfile bundle patch refreshes OpenBao’s JWT plugin catalog before the larger 1.0.1 upgrade. The patch changes the OpenBao chart from 0.30.25 to 0.32.6 and keeps the 0.6.1 runtime versions. Complete these steps and validate the deployment before starting the 0.6.2 to 1.0.1 upgrade.

1. Record a healthy source deployment

Start with a healthy 0.6.1 deployment. Every source Helm release must be deployed, every OpenBao member initialized and unsealed, and every Cassandra migration ledger clean. Resolve degraded health before changing charts.

Use Helm 3 and Helmfile 1.1.x. Set the bundle paths, environment, and intended Kubernetes context. Use a new private directory for backups:

set -euo pipefail
export SOURCE_CONTROL=<absolute-path-to-0.6.1-bundle>
export TARGET_CONTROL=<new-absolute-path-for-0.6.2-bundle>
export CONTROL_KUBECONFIG=<control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export HELMFILE_ENV=<existing-environment-name>
export BACKUP_DIR=<new-private-backup-directory>
umask 077
mkdir -p "$BACKUP_DIR"
kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" \
  = "$CONTROL_CONTEXT"
kcp cluster-info
helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  list --all-namespaces > "$BACKUP_DIR/helm-releases.txt"
kcp -n vault-system get statefulset openbao-server -o yaml \
  > "$BACKUP_DIR/openbao-statefulset.yaml"
kcp -n vault-system get pods -o json > "$BACKUP_DIR/openbao-pods.json"

Record the chart versions, image digests, replica counts, storage settings, plugin catalog digest, JWT mount configuration, and injector webhook settings. Keep any intentional server or migration image override in this baseline.

Record an existing function and deployment, their cluster ID, worker pod UID, container IDs, and restart counts. Invoke that function through the existing authentication flow, then keep a timestamped probe running during the patch. Use the same function afterward to check that the upgrade retained the original deployment.

Cluster health is required before upgrading. The optional checks below can supplement your normal health checks by comparing the server pods with the StatefulSet and Raft membership.

(
  set +x
  set -euo pipefail
  umask 077
  checks=$(mktemp -d "$BACKUP_DIR/openbao-preflight.XXXXXX")
  # Require every expected pod to belong to this StatefulSet and be ready.
  statefulset=$(kcp -n vault-system get statefulset openbao-server -o json)
  replicas=$(jq -er '.spec.replicas | select(. > 0)' <<< "$statefulset")
  owner_uid=$(jq -er '.metadata.uid' <<< "$statefulset")
  pod_list=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server -o json)
  jq -e --arg uid "$owner_uid" --argjson replicas "$replicas" '
    (.items | length) == $replicas and
    all(.items[];
      any(.metadata.ownerReferences[]?; .uid == $uid and .controller == true) and
      .metadata.deletionTimestamp == null and .status.phase == "Running" and
      any(.status.conditions[]?; .type == "Ready" and .status == "True"))
  ' <<< "$pod_list" > /dev/null || {
    echo 'Stop: server pod count, ownership, or readiness does not match the StatefulSet' >&2
    exit 1
  }
  pods=$(jq -r '.items[].metadata.name' <<< "$pod_list")
  # Check OpenBao itself and collect the node IDs for comparison with Raft.
  active=""
  : > "$checks/raft-node-ids.txt"
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    node_id=$(kcp -n vault-system exec "$pod" -c openbao -- cat /openbao/data/node-id)
    test -n "$node_id"
    printf '%s\n' "$node_id" >> "$checks/raft-node-ids.txt"
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
      active_id="$node_id"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi
  # Read Raft membership with the token passed over stdin, not in arguments.
  kcp -n vault-system get secret openbao-server-root-token -o json |
    jq -er '.data.root_token | strings | @base64d | select(length > 0)' |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      exec bao operator raft list-peers -format=json
    ' > "$checks/raft-peers.json"
  # Require the same voting members and leader, not just the same count.
  node_ids=$(jq -Rsc 'split("\n") | map(select(length > 0)) | sort' \
    "$checks/raft-node-ids.txt")
  jq -e --argjson expected "$node_ids" --arg active "$active_id" '
    .data.config.servers |
    (map(.node_id) | sort) == $expected and
    ($expected | unique | length) == ($expected | length) and
    all(.[]; .voter == true) and
    ([.[] | select(.leader == true) | .node_id] == [$active])
  ' "$checks/raft-peers.json" > /dev/null || {
    echo 'Stop: Raft voters or leader do not match the healthy server pods' >&2
    exit 1
  }
  printf 'OpenBao preflight passed; details saved to %s\n' "$checks"
)

2. Back up state

Save the source environment values and secrets securely, then take an OpenBao Raft snapshot. Run these commands in Bash on the operator host, using the kcp helper and $BACKUP_DIR from section 1. You need jq installed locally and Kubernetes permissions to read the two Secrets, list the server pods, and execute commands in the OpenBao containers. The bao CLI runs inside the selected pod, so it does not need to be installed on your host.

The snapshot request must reach the active member because, with the bundled OpenBao 2.5.5, a request through a standby’s localhost address fails with incomplete snapshot. This example authenticates with the existing chart-managed root token; a token granting read on sys/storage/raft/snapshot is also sufficient for snapshot download. The token is passed over stdin to keep it out of command arguments and shell history. Do not enable shell tracing or print the credential files.

The standard chart uses a Shamir seal with one key share and a threshold of one, so recovery requires the single unseal key that matches the snapshot. Back up both chart-managed Secrets alongside it: the unseal key unlocks the restored state, while the root token authenticates the chart’s hooks and migrations. The root token cannot replace the unseal key. These exports contain usable credentials, so keep them in protected recovery storage and out of shared logs.

The following example runs independently of the optional preflight. It saves the credentials and finds the active member, but does not establish whether any expected members are missing. Complete the health checks in section 1 before upgrading. Each backup attempt uses a new private directory under $BACKUP_DIR and a unique temporary file in the active pod, so retries do not overwrite earlier backups.

(
  set +x
  set -euo pipefail
  umask 077
  # Keep each attempt separate and save the credentials needed for recovery.
  backup=$(mktemp -d "$BACKUP_DIR/openbao.XXXXXX")
  kcp -n vault-system get secret openbao-server-unseal openbao-server-root-token \
    -o json > "$backup/openbao-secrets.json"
  chmod 600 "$backup/openbao-secrets.json"
  jq -e '.items[] | select(.metadata.name == "openbao-server-unseal") |
    .data.unseal_key | strings | length > 0' \
    "$backup/openbao-secrets.json" > /dev/null || {
      echo 'Stop: the saved OpenBao unseal key is missing or empty' >&2
      exit 1
    }
  # Check the listed servers and find one active member for the snapshot.
  pods=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}')
  active=""
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi
  # Authenticate over stdin and save the snapshot in a unique pod-side file.
  snapshot=$(kcp -n vault-system exec "$active" -c openbao -- \
    mktemp /tmp/openbao-backup.XXXXXX)
  jq -er '.items[] | select(.metadata.name == "openbao-server-root-token") |
    .data.root_token | strings | @base64d | select(length > 0)' \
    "$backup/openbao-secrets.json" |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      bao operator raft snapshot save "$1"
    ' sh "$snapshot"
  # Verify the host copy before removing the pod-side file.
  kcp -n vault-system cp -c openbao "$active:$snapshot" "$backup/openbao.snap"
  test -s "$backup/openbao.snap"
  chmod 600 "$backup/openbao.snap"
  kcp -n vault-system exec "$active" -c openbao -- rm -f "$snapshot"
  printf 'OpenBao backup saved to %s\n' "$backup"
)

Continue only after the block succeeds and the printed directory contains openbao.snap and openbao-secrets.json. The commands check that the host copy is nonempty before removing the pod copy. If snapshot creation or transfer fails, stop and investigate; any pod-side temporary file is left in place.

The checks detect missing backup files and an absent or empty unseal key, but do not prove that the key matches the snapshot. Verify recovery in an isolated environment using your site’s restore procedure, and keep the backup in protected recovery storage. File permissions restrict access but do not encrypt these files. If your deployment uses a different seal or credential arrangement, follow its tested recovery procedure instead of assuming these two Secrets are sufficient.

Take a Cassandra snapshot on every member using your site’s backup procedure. Save the migration ledgers, database topology, and persistent-volume identities. Keep the source bundle and mirrored images with the recovery baseline so a restore does not depend on a later registry lookup. Verify your restore procedure before continuing. Do not put tokens, unseal material, or rendered secrets into shared logs.

3. Prepare the patch bundle

Obtain the nvcf-self-managed-stack 0.6.2 Helmfile bundle through your NVCF artifact distribution channel. Confirm access to that exact version before scheduling the upgrade. Compare its contents and source commit with the stable 0.6.2 release and its artifact inventory. The recorded source commit is 55803ad7cb647de45cf29415cfd3766225534617. Verify the bundle checksum supplied by your distribution channel, then extract it into $TARGET_CONTROL; keep $SOURCE_CONTROL intact for recovery.

Use the Image Mirroring process with the versions in that inventory. The current documentation manifest describes the 1.0.1 stack; do not substitute its artifact versions for this legacy patch.

Start with the target templates and carry your settings into these files:

$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml

Preserve registry and pull-secret settings, endpoints, certificates, storage classes and sizes, replica counts, scheduling constraints, Cassandra passwords, OpenBao configuration, and injector webhook scope. Keep enabled optional components and existing secret and cluster identities.

Keep the target global.yaml.gotmpl. It contains the runtime pinning fix, so copying the source file over it would remove part of the patch. Mirror required artifacts at the exact versions listed in the 0.6.2 inventory.

4. Review the rendered change

Render the OpenBao release into the private backup directory and inspect its diff before applying it:

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server template > "$BACKUP_DIR/openbao-target.yaml"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server diff --suppress-secrets

Check these pins against the recorded source overrides:

The StatefulSet must keep OnDelete. The catalog refresh Job must run before migrations. Review changes to RBAC, hooks, webhook configuration, probes, and pod templates. Stop if the diff changes a volume, identity, runtime image, or credential that you did not intend to change.

5. Upgrade OpenBao

Both releases use migration image 0.16.2. The image tag does not show whether Helm ran a new migration Job, so record the old Job UID:

old_migration_uid=$(kcp -n vault-system get job openbao-server-migrations \
  -o jsonpath='{.metadata.uid}' --ignore-not-found)

Open a second terminal with the same environment variables and kcp helper. Watch the hook Jobs and collect their logs while the sync runs. Helm can delete a successful refresh Job, so capture its completion when it happens.

Run the bundle’s selected sync in the first terminal:

make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server

Check the hooks in this order:

1. openbao-server-refresh-jwt-plugin-catalog registers the digest from the retained server image and completes successfully.
2. A new openbao-server-migrations Job starts after refresh completes. Its UID must differ from $old_migration_uid.
3. The new migration Job and Helm sync complete successfully.

Catalog registration does not reload plugins in running servers. If the source catalog digest already matches the retained server image and the server pod template is unchanged, wait for migrations without rotating servers. This was the path exercised in the healthy-source rehearsal.

A stale catalog or changed server image or pod template needs a separately validated rotation and recovery sequence. Stop rather than assuming the healthy-source result covers that case. If that sequence calls for rotation, handle one server at a time, standbys first and the active member last. A single-member deployment has downtime during rotation.

For each server selected for rotation:

export POD=<one-standby-pod>
old_pod_uid=$(kcp -n vault-system get pod "$POD" -o jsonpath='{.metadata.uid}')
kcp -n vault-system delete pod "$POD" --wait=true &&
  kcp -n vault-system wait "pod/$POD" --for=create --timeout=5m &&
  kcp -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m &&
  kcp -n vault-system exec "$POD" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' || {
      echo 'Stop: replacement OpenBao pod did not pass health checks' >&2
      exit 1
    }
new_pod_uid=$(kcp -n vault-system get pod "$POD" -o jsonpath='{.metadata.uid}')
test -n "$new_pod_uid"
test "$new_pod_uid" != "$old_pod_uid"

Require a new pod UID, Initialized true, Sealed false, the expected image, and healthy Raft membership before rotating another member. Readiness alone does not prove that auto-unseal has finished. Recheck which member is active before the last rotation. Never delete the StatefulSet, persistent-volume claims, or multiple members together.

Confirm the migration Job and release status:

kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m
helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  status openbao-server -n vault-system

6. Validate the deployment

Check that every OpenBao member is healthy and unsealed, the catalog digest matches the expected plugin, and the new migration Job succeeded. Then:

1. Exercise the existing JWT-backed authentication flow and invoke the original function again.
2. Compare the original function, deployment, cluster, database records, and worker identity with the baseline.
3. Admit a new test workload within the injector’s configured webhook scope and verify that it receives its secrets.
4. Review a full stack diff. After the OpenBao checks pass, apply the remaining expected changes with the bundle’s normal make install command.
5. Repeat the selected OpenBao sync and the full stack sync to check idempotency. Repeat the health, JWT, injector, and invocation checks.

Record failed probes even if the final checks pass. Keep the 0.6.2 values and backups as the source baseline for the 1.0.1 upgrade.

Rehearsal results

A rehearsal upgraded released 0.6.1 to commit 55803ad7cb647de45cf29415cfd3766225534617, later published as stable 0.6.2, on isolated ARM64 k3d. It used three OpenBao members, one Cassandra member, and fake GPUs on one host. Optional add-ons and observability were disabled, and local registry and resource settings were adapted for the test.

Refresh completed before the new migration Job started. The server templates and runtime images were unchanged, and all three members kept their pod identities without rotation. Fresh JWT issuance, new secret injection, full syncs, and repeat syncs passed. Existing credentials, PVCs, function and deployment records, clean Cassandra ledgers, and worker identities were retained. Each of the function-read, deployment-read, and invocation probes recorded 218 successes with no failures.

This evidence covers the healthy-source path at that commit. It does not cover stale-catalog recovery, backup restoration, optional-component upgrades, or production failure domains.

Stop and recover

Stop on a failed hook, sealed member, dirty migration ledger, JWT or injector failure, missing record, or unexpected identity change. Save Helm history and hook logs before retrying. Do not skip hooks to force a successful release status.

The original deployment and bundle are the recovery baseline before state changes. Once a hook changes OpenBao state, a chart rollback alone is not a complete recovery plan. Validate the catalog and runtime combination before retrying, or restore the coordinated backup with your site’s tested procedure. Preserve all persistent volumes and original credentials.