> This page is for version 1.1.0.
> For other versions, use one of these documentation indexes:
> - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt
> - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt
> - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.

# Upgrade from NVCF Self-Managed 0.6.1 to 0.6.2

The 0.6.2 patch refreshes OpenBao's JWT plugin catalog before the
[0.6.2 to 1.0.1 upgrade](/nvcf/1.1.0/release-notes/0-6-1-to-1-0-1-upgrade). It changes the OpenBao
chart from 0.30.25 to 0.32.6 while retaining runtime 2.5.5-nv-1.3.1 and
migrations 0.16.2.

An installation upgraded from 0.6.0 can still run 2.5.4-nv-1.3.0 in every
OpenBao server pod. `OnDelete` updates the StatefulSet template without
replacing pods. An unchanged template in the 0.6.2 diff does not mean those
pods are current. Do not restart them at 0.6.1: refresh the catalog with 0.6.2
first, then replace the stale members using this procedure.

This procedure supports three healthy voting members, a Shamir seal, the
standard `/openbao/data` volume layout, and one CPU architecture across every
cluster node. The refresh hook has no architecture selector, so a cluster with
mixed architectures is unsupported even if all OpenBao servers share one.
It preserves the StatefulSet, PVCs, credentials and Raft node IDs.
Use a separately tested site procedure for other layouts. Allow a maintenance
window for member restarts and leadership transfer. Clients can see transient
authentication failures while members restart. Never reload the plugin while
binaries differ, change to `RollingUpdate`, force-delete a pod, or delete
multiple members together.
Keep other operators and automation from changing this release or node
architecture during the procedure. Data PVCs must be retained, have no deletion
timestamp and have no garbage-collection owner references.

## 1. Record and check the source deployment

Every source Helm release must be deployed, every OpenBao member initialized
and unsealed, and every Cassandra migration ledger clean. Complete the
[0.6.0 to 0.6.1 checks](/nvcf/1.1.0/release-notes/legacy-0-6-0-to-0-6-1-upgrade) if upgrading from 0.6.0.
The checks below are required; Kubernetes Ready alone accepts sealed or
uninitialized responses with these charts.

Use Bash, Python 3.9 or later, jq, curl, kubectl, Helm 3 and Helmfile 1.1.x.
Download the <Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/4e6f37a63a6e0c39be2e60b0e85d868065adb87dff4954f1ee48a87a7515a08b/versions/1.1.0/fern/assets/upgrade-checks/openbao-upgrade-check.py" filename="openbao-upgrade-check.py">read-only validator</Download>.
It reads the captured files and never contacts or changes a cluster.
Run every block in the same Bash session. Supply the control-plane API base URL
without `/v2/nvcf`, the function's NVIDIA Cloud Account (NCA) ID, function ID and
version ID, and its normal synchronous invocation URL and JSON request file.
The commands construct the administrative read URL with the account in its
path: `/v2/nvcf/accounts/{ncaId}/functions/{functionId}/versions/{functionVersionId}`.
This route accepts the admin JWT without an extra account header. The ordinary
`/v2/nvcf/functions/...` read route requires user scopes and rejects this JWT.
Provide an existing API key authorized to invoke that function in a private
header file containing one line: `Authorization: Bearer <api-key>`.
Set `INVOCATION_AUTH_HEADER` to that file's absolute path. Keep the key out of
command arguments, shell history and logs. These probes must succeed before
maintenance; use a deterministic test function whose response you can verify.
Protect their outputs as application data.

```bash
set -euo pipefail
set +x
export SOURCE_CONTROL='<absolute-path-to-0.6.1-bundle>'
export TARGET_CONTROL='<new-absolute-path-for-0.6.2-bundle>'
export CONTROL_KUBECONFIG='<control-plane-kubeconfig>'
export CONTROL_CONTEXT='<intended-control-plane-context>'
export HELMFILE_ENV='<existing-environment-name>'
export BACKUP_DIR='<new-private-backup-directory>'
export OPENBAO_CHECK='<absolute-path-to-openbao-upgrade-check.py>'
export NVCF_API_URL='<control-plane-api-base-url>'
export NCA_ID='<existing-function-account-id>'
export FUNCTION_ID='<existing-function-id>'
export FUNCTION_VERSION_ID='<existing-function-version-id>'
export FUNCTION_READ_URL="${NVCF_API_URL%/}/v2/nvcf/accounts/${NCA_ID}/functions/${FUNCTION_ID}/versions/${FUNCTION_VERSION_ID}"
export FUNCTION_INVOKE_URL='<existing-function-synchronous-invocation-url>'
export INVOCATION_REQUEST='<absolute-path-to-json-request>'
export INVOCATION_AUTH_HEADER='<absolute-path-to-private-api-key-header-file>'

umask 077
mkdir "$BACKUP_DIR"
test -s "$OPENBAO_CHECK"
test -s "$INVOCATION_REQUEST"
test -s "$INVOCATION_AUTH_HEADER"
chmod 600 "$INVOCATION_AUTH_HEADER"
export UPGRADE_KUBECONFIG="$BACKUP_DIR/control.kubeconfig"
kubectl --kubeconfig "$CONTROL_KUBECONFIG" --context "$CONTROL_CONTEXT" \
  config view --minify --flatten --raw > "$UPGRADE_KUBECONFIG"
kcp() {
  kubectl --kubeconfig "$UPGRADE_KUBECONFIG" --context "$CONTROL_CONTEXT" "$@"
}
hcp() {
  helm --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" "$@"
}
CONTROL_CLUSTER_UID=$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}')
test -n "$CONTROL_CLUSTER_UID"
readonly UPGRADE_KUBECONFIG CONTROL_CONTEXT CONTROL_CLUSTER_UID
check_context() {
  test "$(kubectl --kubeconfig "$UPGRADE_KUBECONFIG" config current-context)" \
    = "$CONTROL_CONTEXT" || return 1
  test "$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}')" \
    = "$CONTROL_CLUSTER_UID" || return 1
}
check_context
hcp list --all-namespaces --all -o json > "$BACKUP_DIR/helm-releases.json"
jq -e 'length > 0 and all(.[]; .status == "deployed")' \
  "$BACKUP_DIR/helm-releases.json" >/dev/null
```

The private kubeconfig contains only the selected context. The bundle's Makefile
accepts a kubeconfig but no context flag, so all syncs below use this copy.
Record the existing function/deployment IDs, cluster ID, worker pod UID,
container IDs and restart counts for comparison after the patch.

### Capture health and test fresh authentication

These helpers use the chart-managed root token over stdin for OpenBao
administrative reads. Each call to `fresh_auth` runs two separate probes:

- Create a fresh service-account JWT, log in with the existing bootstrap role,
  sign a new NVCF JWT, and use it for the administrative function read. Signing
  exercises the JWT secrets plugin; a successful read proves the API accepts
  the newly signed JWT. A cached token cannot establish fresh signing works.
- Invoke the same function through its normal synchronous invocation URL using
  the existing invocation-scoped API key from `INVOCATION_AUTH_HEADER`. This
  checks invocation continuity; it does not prove fresh JWT signing works.

The `nvcf-api-admin` signing role grants `admin:list_functions_details` but not
`invoke_function`, so its JWT cannot authorize normal invocation. Keep the
existing role and scopes unchanged. Both probes must return HTTP 200 before
`fresh-auth-passed.txt` is written. The example uses the standard bootstrap
service account, role and audience. Stop if your deployment uses different
identities or cannot pass either probe.

```bash
bao_admin() {
  local pod=$1
  shift
  kcp -n vault-system get secret openbao-server-root-token -o json |
    jq -er '.data.root_token | @base64d | select(length > 0)' |
    kcp -n vault-system exec -i "$pod" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      exec bao "$@"
    ' sh "$@"
}
bao_status() {
  kcp -n vault-system exec "$1" -c openbao -- sh -ec \
    'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json'
}
fresh_auth() {
  local pod=$1 directory=$2 code
  kcp -n nvcf create token nvcf-api-account-bootstrap \
    --audience=http://openbao-server.vault-system.svc.cluster.local:8200 \
    --duration=10m |
    kcp -n vault-system exec -i "$pod" -c openbao -- sh -ec '
      IFS= read -r service_jwt || test -n "$service_jwt"
      export BAO_ADDR=http://127.0.0.1:8200
      response=$(printf "%s" "$service_jwt" |
        bao write -format=json auth/jwt/login role=nvcf-api-account-bootstrap jwt=-)
      BAO_TOKEN=$(printf "%s" "$response" | jq -er ".auth.client_token | select(length > 0)")
      export BAO_TOKEN
      bao write -format=json -f services/nvcf-api/jwt/sign/nvcf-api-admin |
        jq -er "\"Authorization: Bearer \" + (.data.token | select(length > 0))"
    ' > "$directory/auth-header" || return 1
  code=$(curl --fail --silent --show-error --max-time 60 --write-out '%{http_code}' \
    --header @"$directory/auth-header" --output "$directory/function.json" \
    "$FUNCTION_READ_URL") || return 1
  test "$code" = 200 || return 1
  code=$(curl --fail --silent --show-error --max-time 120 --write-out '%{http_code}' \
    --header @"$INVOCATION_AUTH_HEADER" --header 'Content-Type: application/json' \
    --output "$directory/invocation.json" --data-binary @"$INVOCATION_REQUEST" \
    "$FUNCTION_INVOKE_URL") || return 1
  test "$code" = 200 || return 1
  date -u +%Y-%m-%dT%H:%M:%SZ > "$directory/fresh-auth-passed.txt"
}
capture_openbao() {
  local directory=$1 pod active=""
  check_context || return 1
  kcp -n vault-system get statefulset openbao-server -o json \
    > "$directory/statefulset.json" || return 1
  kcp -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
    -o json > "$directory/pods.json" || return 1
  kcp -n vault-system get pvc -o json > "$directory/pvcs.json" || return 1
  kcp get nodes -o json > "$directory/nodes.json" || return 1
  for pod in openbao-server-0 openbao-server-1 openbao-server-2; do
    bao_status "$pod" > "$directory/$pod-status.json" || return 1
    kcp -n vault-system exec "$pod" -c openbao -- cat /openbao/data/node-id \
      > "$directory/$pod-node-id.txt" || return 1
    kcp -n vault-system exec "$pod" -c openbao -- \
      sha256sum /openbao/plugins/vault-plugin-secrets-jwt \
      > "$directory/$pod-plugin.txt" || return 1
    if jq -e '.is_self == true' "$directory/$pod-status.json" >/dev/null; then
      test -z "$active" || return 1
      active=$pod
    fi
  done
  test -n "$active" || return 1
  bao_admin "$active" operator raft list-peers -format=json \
    > "$directory/peers.json" || return 1
  bao_admin "$active" read -format=json sys/storage/raft/autopilot/state \
    > "$directory/autopilot.json" || return 1
  bao_admin "$active" read -format=json sys/plugins/catalog/secret/vault-plugin-secrets-jwt \
    > "$directory/catalog.json" || return 1
}
BASELINE="$BACKUP_DIR/source"
mkdir "$BASELINE"
capture_openbao "$BASELINE"
python3 "$OPENBAO_CHECK" state "$BASELINE" > "$BASELINE/decision.json"
active=$(jq -er '.active' "$BASELINE/decision.json")
fresh_auth "$active" "$BASELINE"
bao_admin "$active" read -format=json sys/mounts > "$BASELINE/mounts.json"
jq '.members | to_entries[] | {pod:.key, images:.value.images,
  imageIDs:.value.image_ids, revision:.value.revision}' "$BASELINE/decision.json"
```

The validator requires exactly the expected owned pods and container statuses,
three matching healthy Raft voters, a single active member, Bound data PVCs,
and matching source catalog/binary checksums. Both healthy sources are supported:
all live servers already on 2.5.5, or all still on 2.5.4 despite a 2.5.5 template.
After the patch, only pods whose live images, image IDs, plugin checksum or
revision differ from the reviewed target need replacement.

Keep this original baseline when resuming a partially completed rotation.
Do not overwrite it with mixed-version state or treat a post-refresh catalog
mismatch on retained old pods as a new healthy source baseline.

## 2. Back up state

Save the source environment values and secrets securely, then take an OpenBao
Raft snapshot. Run these commands in Bash on the operator host, using the `kcp`
helper and `$BACKUP_DIR` from section 1. You need `jq` installed locally and
Kubernetes permissions to read the two Secrets, list the server pods, and
execute commands in the OpenBao containers. The `bao` CLI runs inside the
selected pod, so it does not need to be installed on your host.

The snapshot request must reach the active member because, with the bundled
OpenBao 2.5.5, a request through a standby's localhost address fails with
`incomplete snapshot`. This example authenticates with the existing
chart-managed root token; a token granting `read` on
`sys/storage/raft/snapshot` is also sufficient for snapshot download. The token
is passed over stdin to keep it out of command arguments and shell history.
Do not enable shell tracing or print the credential files.

The standard chart uses a Shamir seal with one key share and a threshold of
one, so recovery requires the single unseal key that matches the snapshot.
Back up both chart-managed Secrets alongside it: the unseal key unlocks the
restored state, while the root token authenticates the chart's hooks and
migrations. The root token cannot replace the unseal key. These exports contain
usable credentials, so keep them in protected recovery storage and out of
shared logs.

The following example runs after the required source health checks. It saves
the credentials and finds the active member, but does not establish whether
any expected members are missing. Complete the health checks in section 1
before upgrading. Each backup attempt uses a new private directory under
`$BACKUP_DIR` and a unique temporary file in the active pod, so retries do not
overwrite earlier backups.

```bash
(
  set +x
  set -euo pipefail
  umask 077

  # Keep each attempt separate and save the credentials needed for recovery.
  backup=$(mktemp -d "$BACKUP_DIR/openbao.XXXXXX")

  kcp -n vault-system get secret openbao-server-unseal openbao-server-root-token \
    -o json > "$backup/openbao-secrets.json"
  chmod 600 "$backup/openbao-secrets.json"
  jq -e '.items[] | select(.metadata.name == "openbao-server-unseal") |
    .data.unseal_key | strings | length > 0' \
    "$backup/openbao-secrets.json" > /dev/null || {
      echo 'Stop: the saved OpenBao unseal key is missing or empty' >&2
      exit 1
    }

  # Check the listed servers and find one active member for the snapshot.
  pods=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}')
  active=""
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi

  # Authenticate over stdin and save the snapshot in a unique pod-side file.
  snapshot=$(kcp -n vault-system exec "$active" -c openbao -- \
    mktemp /tmp/openbao-backup.XXXXXX)
  jq -er '.items[] | select(.metadata.name == "openbao-server-root-token") |
    .data.root_token | strings | @base64d | select(length > 0)' \
    "$backup/openbao-secrets.json" |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      bao operator raft snapshot save "$1"
    ' sh "$snapshot"

  # Verify the host copy before removing the pod-side file.
  kcp -n vault-system cp -c openbao "$active:$snapshot" "$backup/openbao.snap"
  test -s "$backup/openbao.snap"
  chmod 600 "$backup/openbao.snap"
  kcp -n vault-system exec "$active" -c openbao -- rm -f "$snapshot"
  printf 'OpenBao backup saved to %s\n' "$backup"
)
```

Continue only after the block succeeds and the printed directory contains
`openbao.snap` and `openbao-secrets.json`. The commands check that the host copy
is nonempty before removing the pod copy. If snapshot creation or transfer
fails, stop and investigate; any pod-side temporary file is left in place.

The checks detect missing backup files and an absent or empty unseal key, but
do not prove that the key matches the snapshot. Verify recovery in an isolated
environment using your site's restore procedure, and keep the backup in
protected recovery storage. File permissions restrict access but do not
encrypt these files. If your deployment uses a different seal or credential
arrangement, follow its tested recovery procedure instead of assuming these
two Secrets are sufficient.

Take a Cassandra snapshot on every member using your site's backup procedure.
Save the migration ledgers, database topology, and persistent-volume identities.
Keep the source bundle and mirrored images with the recovery baseline so a
restore does not depend on a later registry lookup. Verify your restore
procedure before continuing. Do not put tokens, unseal material, or rendered
secrets into shared logs.

## 3. Prepare the patch bundle

Obtain the `nvcf-self-managed-stack` 0.6.2 Helmfile bundle through your
NVCF artifact distribution channel. Confirm access to that exact version before
scheduling the upgrade. Compare its contents and source commit with the
[stable 0.6.2 release](https://github.com/NVIDIA/nvcf/releases/tag/deploy/stacks/self-managed/v0.6.2)
and its
[artifact inventory](https://github.com/NVIDIA/nvcf/releases/download/deploy/stacks/self-managed/v0.6.2/nvcf-self-managed-stack-inventory.json).
The recorded source commit is `55803ad7cb647de45cf29415cfd3766225534617`.
Verify the bundle checksum supplied by your distribution channel, then extract
it into `$TARGET_CONTROL`; keep `$SOURCE_CONTROL` intact for recovery.

Use the [Image Mirroring](/nvcf/1.1.0/manifest/image-mirroring) process with the versions in
that inventory. The current documentation manifest describes the 1.0.1 stack;
do not substitute its artifact versions for this legacy patch.

Start with the target templates and carry your settings into these files:

```text
$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
```

Preserve registry and pull-secret settings, endpoints, certificates, storage
classes and sizes, replica counts, scheduling constraints, Cassandra passwords,
OpenBao configuration, and injector webhook scope. Keep enabled optional
components and existing secret and cluster identities.

Keep the target `global.yaml.gotmpl`. It contains the runtime pinning fix, so
copying the source file over it would remove part of the patch. Mirror required
artifacts at the exact versions listed in the 0.6.2 inventory.

## 4. Review the rendered change

Render the OpenBao release into the private backup directory and inspect its
diff before applying it:

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server template > "$BACKUP_DIR/openbao-target.yaml"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server diff --suppress-secrets
```

Check these pins against the recorded source overrides:

| Component | Target version |
| --- | --- |
| OpenBao chart | 0.32.6 |
| Server, injected agent, auto-unseal sidecar | 2.5.5-nv-1.3.1 |
| OpenBao migrations | 0.16.2 |
| Injector | 1.7.4 |

The StatefulSet must keep `OnDelete`. The catalog refresh Job must run before
migrations. Review changes to RBAC, hooks, webhook configuration, probes, and
pod templates. Stop if the diff changes a volume, identity, runtime image, or
credential that you did not intend to change.

### Record the reviewed target image

Convert the rendered resources to JSON without creating them. Extract the one
server StatefulSet and verify the migration image. Do not use chart 0.32.6's
newer default runtime or migration versions.

```bash
TARGET_STATE="$BACKUP_DIR/target"
mkdir "$TARGET_STATE"
kcp create --dry-run=client --validate=false -f "$BACKUP_DIR/openbao-target.yaml" \
  -o json | jq -s '.' > "$TARGET_STATE/rendered.json"
jq -e '[.. | objects | select(.kind? == "StatefulSet" and .metadata.name? == "openbao-server")] |
  if length == 1 then .[0] else error("expected one OpenBao StatefulSet") end' \
  "$TARGET_STATE/rendered.json" > "$TARGET_STATE/rendered-statefulset.json"
jq -e '[.. | objects | select(.kind? == "Job" and .metadata.name? == "openbao-server-migrations") |
  .spec.template.spec.containers[].image] |
  length == 1 and all(.[]; endswith(":0.16.2"))' "$TARGET_STATE/rendered.json" >/dev/null
TARGET_IMAGE=$(jq -er '.spec.template.spec.containers[] |
  select(.name == "openbao") | .image | select(endswith(":2.5.5-nv-1.3.1"))' \
  "$TARGET_STATE/rendered-statefulset.json")
architecture=$(jq -er '.architecture' "$BASELINE/decision.json")
```

Run a short image-check Job on the same architecture. It has no credentials or
Raft volumes and only hashes the plugin. Its resolved image ID and plugin hash
are the expected values for server replacements; this also checks image-pull
access before changing the release. The pod inherits the reviewed image pull
Secrets. Use unique generated names and retain the result with the backup.

```bash
jq --arg image "$TARGET_IMAGE" --arg arch "$architecture" '{
  apiVersion:"batch/v1", kind:"Job",
  metadata:{generateName:"openbao-upgrade-image-",namespace:"vault-system"},
  spec:{backoffLimit:0,activeDeadlineSeconds:300,template:{spec:{
    restartPolicy:"Never",automountServiceAccountToken:false,
    nodeSelector:{"kubernetes.io/arch":$arch},
    imagePullSecrets:(.spec.template.spec.imagePullSecrets // []),
    containers:[{name:"probe",image:$image,imagePullPolicy:"Always",
      command:["sha256sum","/openbao/plugins/vault-plugin-secrets-jwt"]}]
  }}}}' "$TARGET_STATE/rendered-statefulset.json" > "$TARGET_STATE/image-job.json"
check_context
image_job=$(kcp create -f "$TARGET_STATE/image-job.json" -o jsonpath='{.metadata.name}')
kcp -n vault-system wait "job/$image_job" --for=condition=Complete --timeout=5m
kcp -n vault-system get pods -l "job-name=$image_job" -o json > "$TARGET_STATE/image-pods.json"
kcp -n vault-system logs "job/$image_job" > "$TARGET_STATE/plugin.txt"
plugin_sha=$(awk 'NR==1 {print $1}' "$TARGET_STATE/plugin.txt")
image_id=$(jq -er '.items | select(length == 1) | .[0].status.containerStatuses |
  select(length == 1) | .[0] | select(.state.terminated.exitCode == 0) | .imageID' \
  "$TARGET_STATE/image-pods.json")
jq -n --arg image "$TARGET_IMAGE" --arg image_id "$image_id" \
  --arg plugin_sha256 "$plugin_sha" \
  '{image:$image,image_id:$image_id,plugin_sha256:$plugin_sha256}' > "$TARGET_STATE/image.json"
```

## 5. Refresh the catalog, then replace stale members

### Apply the selected release and prove the hooks ran

The refresh hook uses the target image and registers its checksum without
reloading existing plugins. Migrations run afterward. The old active process
can keep serving during this transition, but an old plugin process that
restarts can fail against the new catalog. Do not leave the upgrade here or
start another maintenance operation.

Start a bounded Job watch before sync. It records completion even when Helm
deletes the successful refresh Job. The validator requires a fresh completed
refresh and a different, completed migrations Job created after it. Missing
watch evidence stops rotation; a successful Helm status alone is insufficient.

```bash
sync_openbao() {
  local old_uid started watcher
  SYNC_DIR=$(mktemp -d "$BACKUP_DIR/sync.XXXXXX") || return 1
  kcp get nodes -o json > "$SYNC_DIR/nodes.json" || return 1
  jq -e --arg arch "$(jq -er '.architecture' "$BASELINE/decision.json")" \
    '.items | length > 0 and all(.[]; .metadata.labels["kubernetes.io/arch"] == $arch)' \
    "$SYNC_DIR/nodes.json" >/dev/null || return 1
  old_uid=$(kcp -n vault-system get job openbao-server-migrations \
    -o jsonpath='{.metadata.uid}' --ignore-not-found) || return 1
  started=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  kcp -n vault-system get jobs --watch-only --output-watch-events -o json \
    --request-timeout=20m > "$SYNC_DIR/hook-events.json" 2> "$SYNC_DIR/hook-watch.log" &
  watcher=$!
  sleep 2
  kill -0 "$watcher" || return 1
  if ! check_context || ! make -C "$TARGET_CONTROL" install HELMFILE_ENV="$HELMFILE_ENV" \
    KUBECONFIG_FILE="$UPGRADE_KUBECONFIG" HELMFILE_SELECTOR=name=openbao-server; then
    kill "$watcher" 2>/dev/null || true
    wait "$watcher" || true
    return 1
  fi
  sleep 2
  kill "$watcher" 2>/dev/null || true
  wait "$watcher" || true
  python3 "$OPENBAO_CHECK" hooks "$SYNC_DIR/hook-events.json" \
    --old-uid "$old_uid" --started "$started" > "$SYNC_DIR/hooks-passed.json" || return 1
  hcp list -n vault-system --all --filter '^openbao-server$' -o json \
    > "$SYNC_DIR/helm.json" || return 1
  jq -e 'length == 1 and .[0].name == "openbao-server" and
    .[0].namespace == "vault-system" and .[0].status == "deployed" and
    .[0].chart == "helm-nvcf-openbao-server-0.32.6"' \
    "$SYNC_DIR/helm.json" >/dev/null || return 1
  kcp -n vault-system logs job/openbao-server-migrations > "$SYNC_DIR/migrations.log" || return 1
}
sync_openbao
kcp -n vault-system get statefulset openbao-server -o json > "$TARGET_STATE/statefulset.json"
```

If sync fails, stop before deletion. The function stops its local watcher and
retains evidence in `$SYNC_DIR`. Do not skip hooks or repeatedly resync while
replacing servers.

### Guard each replacement

The gate below checks the original PVC/node identities, every container and
member, the frozen target specification, and the target catalog. An unchanged
source pod is allowed temporarily; every replacement must have the target
image ID, revision, version and plugin hash. A changed but incorrect pod is
rejected. Each successful gate also reads the function with a newly signed JWT
and invokes it separately with the existing API key.

```bash
openbao_gate() {
  local stage=$1
  shift
  CHECK_DIR=$(mktemp -d "$BACKUP_DIR/check.XXXXXX") || return 1
  capture_openbao "$CHECK_DIR" || return 1
  python3 "$OPENBAO_CHECK" state "$CHECK_DIR" --stage "$stage" \
    --baseline "$BASELINE" --target "$TARGET_STATE" "$@" \
    > "$CHECK_DIR/decision.json" || return 1
  ACTIVE=$(jq -er '.active' "$CHECK_DIR/decision.json") || return 1
  fresh_auth "$ACTIVE" "$CHECK_DIR" || return 1
}
openbao_gate transition
if jq -e '.stale | length > 0' "$CHECK_DIR/decision.json" >/dev/null; then
  original_active=$(jq -er '.active' "$BASELINE/decision.json")
  original_active_uid=$(jq -er --arg pod "$original_active" '.members[$pod].uid' \
    "$BASELINE/decision.json")
else
  original_active=$ACTIVE
  original_active_uid=$(jq -er --arg pod "$original_active" '.members[$pod].uid' \
    "$CHECK_DIR/decision.json")
fi
openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
jq '{active,stale}' "$CHECK_DIR/decision.json"
```

If `stale` is empty, skip replacement and run the final checks. This handles
already-current sources and a rerun after convergence. Otherwise, discover
stale standbys and replace one at a time. The commands recheck the original
active UID and leadership before every deletion. A replacement failure or
unexpected leader change stops the loop before another deletion.

```bash
while :; do
  openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
  pod=$(jq -r '.active as $active | [.stale[] | select(. != $active)] | first // empty' \
    "$CHECK_DIR/decision.json")
  [ -n "$pod" ] || break
  old_uid=$(jq -er --arg pod "$pod" '.members[$pod].uid' "$CHECK_DIR/decision.json")
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" = "$old_uid"
  bao_status "$pod" | jq -e '.initialized == true and .sealed == false and .ha_enabled == true and
    (.is_self == false or (has("is_self") | not))' >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$BACKUP_DIR/$old_uid-before.log"
  check_context
  kcp -n vault-system delete pod "$pod" --grace-period=60 --wait=true --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=create --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=condition=Ready --timeout=10m
  unsealed=false
  for attempt in $(seq 1 120); do
    if status=$(bao_status "$pod") &&
      jq -e '.initialized and (.sealed == false)' <<< "$status" >/dev/null; then
      unsealed=true
      break
    fi
    sleep 5
  done
  test "$unsealed" = true
  openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
  jq -e --arg pod "$pod" --arg old "$old_uid" \
    '.members[$pod].uid != $old and (.stale | index($pod) == null)' \
    "$CHECK_DIR/decision.json" >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$CHECK_DIR/replacement.log"
done
```

Inspect each replacement's startup and previous pod's saved logs. Require no
plugin checksum errors, failed secret mounts, or forced termination. The
60-second deletion grace allows orderly shutdown; do not shorten it to force
progress. Keep the two upgraded standbys healthy before the last replacement.

### Transfer leadership by replacing the old active

When only the old active remains stale, both standbys have passed the target
checks. Gracefully terminate that old active and let an upgraded member take
over. A standalone `bao operator step-down` leaves the old binary eligible to
regain leadership, so it is not used here. Never force a new-version leader to
fail back to an old version.

```bash
openbao_gate transition --leader "$original_active" --leader-uid "$original_active_uid"
remaining=$(jq '.stale | length' "$CHECK_DIR/decision.json")
if [ "$remaining" -ne 0 ]; then
  jq -e --arg active "$original_active" '.stale == [$active]' "$CHECK_DIR/decision.json" >/dev/null
  pod=$original_active
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" = "$original_active_uid"
  bao_status "$pod" | jq -e '.initialized and (.sealed == false) and .is_self' >/dev/null
  kcp -n vault-system logs "$pod" -c openbao > "$BACKUP_DIR/old-active.log"
  check_context
  kcp -n vault-system delete pod "$pod" --grace-period=60 --wait=true --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=create --timeout=5m
  kcp -n vault-system wait "pod/$pod" --for=condition=Ready --timeout=10m
  unsealed=false
  for attempt in $(seq 1 120); do
    if status=$(bao_status "$pod") &&
      jq -e '.initialized and (.sealed == false)' <<< "$status" >/dev/null; then
      unsealed=true
      break
    fi
    sleep 5
  done
  test "$unsealed" = true
  test "$(kcp -n vault-system get pod "$pod" -o jsonpath='{.metadata.uid}')" != "$original_active_uid"
fi
openbao_gate complete
cp "$CHECK_DIR/decision.json" "$BACKUP_DIR/converged.json"
```

The final gate requires three target members, one active member, the original
PVCs and node IDs, matching catalog/binary hashes, and fresh authentication and
invocation. Every pod's `controller-revision-hash` must equal `updateRevision`,
the controller must have observed the generation, and `updatedReplicas` must
be three. `currentRevision` can remain historical with `OnDelete`; do not rotate
again just to make the two StatefulSet status revision fields equal.

## 6. Validate injection and repeat the upgrade checks

Compare the function/deployment records and invocation result with the baseline,
including original worker pod UID, container IDs and restart counts. Record
transient probe failures even if final checks pass. At every gate, the function
read validates a freshly signed JWT, and the separate API-key request checks
invocation. Retain the timestamped result files privately.

### Admit a new workload through the injector

This short test uses the existing `nvcf-api` identity in namespace `nvcf` and
its signing policy. Its application container checks that the injected init
container wrote a nonempty signed token, without printing it. It creates only
a uniquely named ConfigMap and Pod. Confirm that this namespace is within the
existing webhook scope; do not broaden webhook selectors for the test.

```bash
injection_dir=$(mktemp -d "$BACKUP_DIR/injection.XXXXXX")
cat > "$injection_dir/config-init.hcl" <<'HCL'
exit_after_auth = true
vault { address = "http://openbao-server.vault-system.svc.cluster.local:8200" }
auto_auth {
  method "jwt" {
    mount_path = "auth/jwt"
    config = {
      role = "nvcf-api"
      path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      remove_jwt_after_reading = false
    }
  }
  sink "file" { config = { path = "/home/vault/.vault-token" } }
}
template {
  destination = "/vault/secrets/probe.jwt"
  contents = "{{ with secret \"services/nvcf-api/jwt/sign/nvcf-api\" }}{{ .Data.token }}{{ end }}"
}
HCL
injection_name="openbao-upgrade-$(date -u +%Y%m%d%H%M%S)"
check_context
kcp -n nvcf create configmap "$injection_name" \
  --from-file=config-init.hcl="$injection_dir/config-init.hcl" \
  --from-file=config.hcl="$injection_dir/config-init.hcl"
jq -n --arg name "$injection_name" --arg image "$TARGET_IMAGE" '{
  apiVersion:"v1",kind:"Pod",metadata:{name:$name,namespace:"nvcf",annotations:{
    "vault.hashicorp.com/agent-inject":"true",
    "vault.hashicorp.com/agent-pre-populate-only":"true",
    "vault.hashicorp.com/agent-configmap":$name,
    "vault.hashicorp.com/copy-volume-mounts":"probe"}},
  spec:{restartPolicy:"Never",serviceAccountName:"nvcf-api",
    volumes:[{name:"token",projected:{sources:[{serviceAccountToken:{path:"token",
      expirationSeconds:600,audience:"http://openbao-server.vault-system.svc.cluster.local:8200"}}]}}],
    containers:[{name:"probe",image:$image,command:["sh","-ec","test -s /vault/secrets/probe.jwt"],
      volumeMounts:[{name:"token",mountPath:"/var/run/secrets/kubernetes.io/serviceaccount",readOnly:true}]}]}}
' > "$injection_dir/pod.json"
# Preserve the same pull Secrets as the reviewed server image.
jq --slurpfile target "$TARGET_STATE/rendered-statefulset.json" \
  '.spec.imagePullSecrets = ($target[0].spec.template.spec.imagePullSecrets // [])' \
  "$injection_dir/pod.json" > "$injection_dir/pod-with-pull-secrets.json"
kcp create -f "$injection_dir/pod-with-pull-secrets.json"
kcp -n nvcf wait "pod/$injection_name" --for=jsonpath='{.status.phase}'=Succeeded --timeout=5m
kcp -n nvcf get pod "$injection_name" -o json > "$injection_dir/result.json"
jq -e 'any(.status.initContainerStatuses[]?;
  .name == "vault-agent-init" and .state.terminated.exitCode == 0) and
  any(.status.containerStatuses[]?; .name == "probe" and .state.terminated.exitCode == 0)' \
  "$injection_dir/result.json" >/dev/null
```

The matching image pull Secrets must already exist in `nvcf`. A completed Pod
without the injected init container does not pass. Preserve its result, then
remove only these generated test resources when finished:

```bash
kcp -n nvcf delete pod "$injection_name"
kcp -n nvcf delete configmap "$injection_name"
kcp -n vault-system delete job "$image_job"
```

### Repeat sync and prove no additional replacement is needed

Repeat the selected sync. The same function creates a new evidence directory,
captures the previous migrations UID, and requires fresh completed hooks.
The complete gate must select no stale members and preserve every current UID.

```bash
sync_openbao
openbao_gate complete
jq -S '.members | map_values(.uid)' "$BACKUP_DIR/converged.json" > "$BACKUP_DIR/uids-converged.json"
jq -S '.members | map_values(.uid)' "$CHECK_DIR/decision.json" > "$CHECK_DIR/uids-selected-repeat.json"
diff -u "$BACKUP_DIR/uids-converged.json" "$CHECK_DIR/uids-selected-repeat.json"
jq -e '.stale | length == 0' "$CHECK_DIR/decision.json" >/dev/null
```

Review the full target diff, then apply the remaining changes and repeat the
full sync. Explicitly clear the selector. Each pass must create a new completed
migration Job, retain the same server UIDs, and pass fresh authentication.

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$UPGRADE_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" diff --suppress-secrets
for pass in 1 2; do
  old_uid=$(kcp -n vault-system get job openbao-server-migrations -o jsonpath='{.metadata.uid}')
  check_context
  kcp get nodes -o json | jq -e --arg arch "$architecture" \
    '.items | length > 0 and all(.[]; .metadata.labels["kubernetes.io/arch"] == $arch)' >/dev/null
  make -C "$TARGET_CONTROL" install HELMFILE_ENV="$HELMFILE_ENV" \
    KUBECONFIG_FILE="$UPGRADE_KUBECONFIG" HELMFILE_SELECTOR=
  kcp -n vault-system wait job/openbao-server-migrations --for=condition=Complete --timeout=15m
  new_uid=$(kcp -n vault-system get job openbao-server-migrations -o jsonpath='{.metadata.uid}')
  test "$new_uid" != "$old_uid"
  openbao_gate complete
  jq -S '.members | map_values(.uid)' "$CHECK_DIR/decision.json" > "$BACKUP_DIR/uids-full-$pass.json"
  diff -u "$BACKUP_DIR/uids-converged.json" "$BACKUP_DIR/uids-full-$pass.json"
  jq -e '.stale | length == 0' "$CHECK_DIR/decision.json" >/dev/null
done
```

Repeat the new-injection block with its new unique resource names. Verify
Cassandra ledgers remain clean, credentials and records are retained, and the
original function still serves before continuing to 1.0.1. A rerun of the
replacement decision must select no pods; do not rotate already-current pods.

Record the topology, released image IDs, member replacement and leadership
transfer observations, and probe results from your rehearsal before production
use.

## Stop and recover

Stop on a failed hook or health/authentication gate, unexpected leader or UID
change, missing member/PVC, changed target, dirty migration ledger, or failed
replacement. Preserve evidence, credentials and all persistent volumes. Do not
retry pod deletion to work around an unexplained failure.

Retain the original baseline and frozen target when resuming after a successful
standby replacement. Rerun the transition gate with the saved original leader
and UID; it selects only remaining stale members. After an unexpected leader
change, establish the current version and recovery state before proceeding.
Never force leadership back to an old binary.

Once refresh changes the catalog, a Helm rollback does not restore that state
or the OnDelete pods. After a new-version member becomes active, downgrading
its binary against the current data store is not a supported rollback. Prefer
repairing forward; otherwise use your site's tested coordinated snapshot,
matching credential and source-runtime restore procedure. This guide does not
provide destructive restore automation or claim that backup transfer proves
recoverability. Keep snapshot recovery and other topologies separately qualified.