> This page is for version Development.
> For other versions, use one of these documentation indexes:
> - 1.0.3 (default): https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server.

# Upgrade from NVCF Self-Managed 0.6.1 to 0.6.2

The 0.6.2 Helmfile bundle patch refreshes OpenBao's JWT plugin catalog before the
larger 1.0.1 upgrade. The patch changes the OpenBao chart from 0.30.25 to
0.32.6 and keeps the 0.6.1 runtime versions. Complete these steps and validate
the deployment before starting the [0.6.2 to 1.0.1 upgrade](/nvcf/dev/release-notes/0-6-1-to-1-0-1-upgrade).

## 1. Record a healthy source deployment

Start with a healthy 0.6.1 deployment. Every source Helm release must be
`deployed`, every OpenBao member initialized and unsealed, and every Cassandra
migration ledger clean. Resolve degraded health before changing charts.

Use Helm 3 and Helmfile 1.1.x. Set the bundle paths, environment, and intended
Kubernetes context. Use a new private directory for backups:

```bash
set -euo pipefail
export SOURCE_CONTROL=<absolute-path-to-0.6.1-bundle>
export TARGET_CONTROL=<new-absolute-path-for-0.6.2-bundle>
export CONTROL_KUBECONFIG=<control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export HELMFILE_ENV=<existing-environment-name>
export BACKUP_DIR=<new-private-backup-directory>

umask 077
mkdir -p "$BACKUP_DIR"
kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" \
  = "$CONTROL_CONTEXT"
kcp cluster-info
helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  list --all-namespaces > "$BACKUP_DIR/helm-releases.txt"
kcp -n vault-system get statefulset openbao-server -o yaml \
  > "$BACKUP_DIR/openbao-statefulset.yaml"
kcp -n vault-system get pods -o json > "$BACKUP_DIR/openbao-pods.json"
```

Record the chart versions, image digests, replica counts, storage settings,
plugin catalog digest, JWT mount configuration, and injector webhook settings.
Keep any intentional server or migration image override in this baseline.

Record an existing function and deployment, their cluster ID, worker pod UID,
container IDs, and restart counts. Invoke that function through the existing
authentication flow, then keep a timestamped probe running during the patch.
Use the same function afterward to check that the upgrade retained the
original deployment.

Cluster health is required before upgrading. The optional checks below can
supplement your normal health checks by comparing the server pods with the
StatefulSet and Raft membership.

<Accordion title="Optional preflight checks">

Run this block in Bash using `kcp` and `$BACKUP_DIR` from above. It requires
`jq`, Kubernetes access to the StatefulSet, server pods, and root-token Secret,
and permission to execute commands in the server containers. It uses the
chart-managed root token to read the Raft configuration.

These checks assume the standard chart layout, with one voting member per
server pod and node IDs stored at `/openbao/data/node-id`. Adapt them if you
use a different data path or intentional non-voting members. They check the
current state, not whether a later upgrade or restore will succeed. If a check
fails, resolve the discrepancy before upgrading.

```bash
(
  set +x
  set -euo pipefail
  umask 077
  checks=$(mktemp -d "$BACKUP_DIR/openbao-preflight.XXXXXX")

  # Require every expected pod to belong to this StatefulSet and be ready.
  statefulset=$(kcp -n vault-system get statefulset openbao-server -o json)
  replicas=$(jq -er '.spec.replicas | select(. > 0)' <<< "$statefulset")
  owner_uid=$(jq -er '.metadata.uid' <<< "$statefulset")
  pod_list=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server -o json)
  jq -e --arg uid "$owner_uid" --argjson replicas "$replicas" '
    (.items | length) == $replicas and
    all(.items[];
      any(.metadata.ownerReferences[]?; .uid == $uid and .controller == true) and
      .metadata.deletionTimestamp == null and .status.phase == "Running" and
      any(.status.conditions[]?; .type == "Ready" and .status == "True"))
  ' <<< "$pod_list" > /dev/null || {
    echo 'Stop: server pod count, ownership, or readiness does not match the StatefulSet' >&2
    exit 1
  }
  pods=$(jq -r '.items[].metadata.name' <<< "$pod_list")

  # Check OpenBao itself and collect the node IDs for comparison with Raft.
  active=""
  : > "$checks/raft-node-ids.txt"
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    node_id=$(kcp -n vault-system exec "$pod" -c openbao -- cat /openbao/data/node-id)
    test -n "$node_id"
    printf '%s\n' "$node_id" >> "$checks/raft-node-ids.txt"
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
      active_id="$node_id"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi

  # Read Raft membership with the token passed over stdin, not in arguments.
  kcp -n vault-system get secret openbao-server-root-token -o json |
    jq -er '.data.root_token | strings | @base64d | select(length > 0)' |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      exec bao operator raft list-peers -format=json
    ' > "$checks/raft-peers.json"

  # Require the same voting members and leader, not just the same count.
  node_ids=$(jq -Rsc 'split("\n") | map(select(length > 0)) | sort' \
    "$checks/raft-node-ids.txt")
  jq -e --argjson expected "$node_ids" --arg active "$active_id" '
    .data.config.servers |
    (map(.node_id) | sort) == $expected and
    ($expected | unique | length) == ($expected | length) and
    all(.[]; .voter == true) and
    ([.[] | select(.leader == true) | .node_id] == [$active])
  ' "$checks/raft-peers.json" > /dev/null || {
    echo 'Stop: Raft voters or leader do not match the healthy server pods' >&2
    exit 1
  }
  printf 'OpenBao preflight passed; details saved to %s\n' "$checks"
)
```

</Accordion>

## 2. Back up state

Save the source environment values and secrets securely, then take an OpenBao
Raft snapshot. Run these commands in Bash on the operator host, using the `kcp`
helper and `$BACKUP_DIR` from section 1. You need `jq` installed locally and
Kubernetes permissions to read the two Secrets, list the server pods, and
execute commands in the OpenBao containers. The `bao` CLI runs inside the
selected pod, so it does not need to be installed on your host.

The snapshot request must reach the active member because, with the bundled
OpenBao 2.5.5, a request through a standby's localhost address fails with
`incomplete snapshot`. This example authenticates with the existing
chart-managed root token; a token granting `read` on
`sys/storage/raft/snapshot` is also sufficient for snapshot download. The token
is passed over stdin to keep it out of command arguments and shell history.
Do not enable shell tracing or print the credential files.

The standard chart uses a Shamir seal with one key share and a threshold of
one, so recovery requires the single unseal key that matches the snapshot.
Back up both chart-managed Secrets alongside it: the unseal key unlocks the
restored state, while the root token authenticates the chart's hooks and
migrations. The root token cannot replace the unseal key. These exports contain
usable credentials, so keep them in protected recovery storage and out of
shared logs.

The following example runs independently of the optional preflight. It saves
the credentials and finds the active member, but does not establish whether
any expected members are missing. Complete the health checks in section 1
before upgrading. Each backup attempt uses a new private directory under
`$BACKUP_DIR` and a unique temporary file in the active pod, so retries do not
overwrite earlier backups.

```bash
(
  set +x
  set -euo pipefail
  umask 077

  # Keep each attempt separate and save the credentials needed for recovery.
  backup=$(mktemp -d "$BACKUP_DIR/openbao.XXXXXX")

  kcp -n vault-system get secret openbao-server-unseal openbao-server-root-token \
    -o json > "$backup/openbao-secrets.json"
  chmod 600 "$backup/openbao-secrets.json"
  jq -e '.items[] | select(.metadata.name == "openbao-server-unseal") |
    .data.unseal_key | strings | length > 0' \
    "$backup/openbao-secrets.json" > /dev/null || {
      echo 'Stop: the saved OpenBao unseal key is missing or empty' >&2
      exit 1
    }

  # Check the listed servers and find one active member for the snapshot.
  pods=$(kcp -n vault-system get pods \
    -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}')
  active=""
  for pod in $pods; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json')
    jq -e '.initialized == true and .sealed == false' <<< "$status" > /dev/null
    if jq -e '.is_self == true' <<< "$status" > /dev/null; then
      if [ -n "$active" ]; then
        echo 'Stop: multiple active OpenBao members reported; retry after leadership stabilizes' >&2
        exit 1
      fi
      active="$pod"
    fi
  done
  if [ -z "$active" ]; then
    echo 'Stop: no active OpenBao member found' >&2
    exit 1
  fi

  # Authenticate over stdin and save the snapshot in a unique pod-side file.
  snapshot=$(kcp -n vault-system exec "$active" -c openbao -- \
    mktemp /tmp/openbao-backup.XXXXXX)
  jq -er '.items[] | select(.metadata.name == "openbao-server-root-token") |
    .data.root_token | strings | @base64d | select(length > 0)' \
    "$backup/openbao-secrets.json" |
    kcp -n vault-system exec -i "$active" -c openbao -- sh -ec '
      IFS= read -r BAO_TOKEN
      test -n "$BAO_TOKEN"
      export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200
      bao operator raft snapshot save "$1"
    ' sh "$snapshot"

  # Verify the host copy before removing the pod-side file.
  kcp -n vault-system cp -c openbao "$active:$snapshot" "$backup/openbao.snap"
  test -s "$backup/openbao.snap"
  chmod 600 "$backup/openbao.snap"
  kcp -n vault-system exec "$active" -c openbao -- rm -f "$snapshot"
  printf 'OpenBao backup saved to %s\n' "$backup"
)
```

Continue only after the block succeeds and the printed directory contains
`openbao.snap` and `openbao-secrets.json`. The commands check that the host copy
is nonempty before removing the pod copy. If snapshot creation or transfer
fails, stop and investigate; any pod-side temporary file is left in place.

The checks detect missing backup files and an absent or empty unseal key, but
do not prove that the key matches the snapshot. Verify recovery in an isolated
environment using your site's restore procedure, and keep the backup in
protected recovery storage. File permissions restrict access but do not
encrypt these files. If your deployment uses a different seal or credential
arrangement, follow its tested recovery procedure instead of assuming these
two Secrets are sufficient.

Take a Cassandra snapshot on every member using your site's backup procedure.
Save the migration ledgers, database topology, and persistent-volume identities.
Keep the source bundle and mirrored images with the recovery baseline so a
restore does not depend on a later registry lookup. Verify your restore
procedure before continuing. Do not put tokens, unseal material, or rendered
secrets into shared logs.

## 3. Prepare the patch bundle

Obtain the `nvcf-self-managed-stack` 0.6.2 Helmfile bundle through your
NVCF artifact distribution channel. Confirm access to that exact version before
scheduling the upgrade. Compare its contents and source commit with the
[stable 0.6.2 release](https://github.com/NVIDIA/nvcf/releases/tag/deploy/stacks/self-managed/v0.6.2)
and its
[artifact inventory](https://github.com/NVIDIA/nvcf/releases/download/deploy/stacks/self-managed/v0.6.2/nvcf-self-managed-stack-inventory.json).
The recorded source commit is `55803ad7cb647de45cf29415cfd3766225534617`.
Verify the bundle checksum supplied by your distribution channel, then extract
it into `$TARGET_CONTROL`; keep `$SOURCE_CONTROL` intact for recovery.

Use the [Image Mirroring](/nvcf/dev/manifest/image-mirroring) process with the versions in
that inventory. The current documentation manifest describes the 1.0.1 stack;
do not substitute its artifact versions for this legacy patch.

Start with the target templates and carry your settings into these files:

```text
$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
```

Preserve registry and pull-secret settings, endpoints, certificates, storage
classes and sizes, replica counts, scheduling constraints, Cassandra passwords,
OpenBao configuration, and injector webhook scope. Keep enabled optional
components and existing secret and cluster identities.

Keep the target `global.yaml.gotmpl`. It contains the runtime pinning fix, so
copying the source file over it would remove part of the patch. Mirror required
artifacts at the exact versions listed in the 0.6.2 inventory.

## 4. Review the rendered change

Render the OpenBao release into the private backup directory and inspect its
diff before applying it:

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server template > "$BACKUP_DIR/openbao-target.yaml"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  --selector name=openbao-server diff --suppress-secrets
```

Check these pins against the recorded source overrides:

| Component | Target version |
| --- | --- |
| OpenBao chart | 0.32.6 |
| Server, injected agent, auto-unseal sidecar | 2.5.5-nv-1.3.1 |
| OpenBao migrations | 0.16.2 |
| Injector | 1.7.4 |

The StatefulSet must keep `OnDelete`. The catalog refresh Job must run before
migrations. Review changes to RBAC, hooks, webhook configuration, probes, and
pod templates. Stop if the diff changes a volume, identity, runtime image, or
credential that you did not intend to change.

## 5. Upgrade OpenBao

Both releases use migration image 0.16.2. The image tag does not show whether
Helm ran a new migration Job, so record the old Job UID:

```bash
old_migration_uid=$(kcp -n vault-system get job openbao-server-migrations \
  -o jsonpath='{.metadata.uid}' --ignore-not-found)
```

Open a second terminal with the same environment variables and `kcp` helper.
Watch the hook Jobs and collect their logs while the sync runs. Helm can delete
a successful refresh Job, so capture its completion when it happens.

Run the bundle's selected sync in the first terminal:

```bash
make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server
```

Check the hooks in this order:

1. `openbao-server-refresh-jwt-plugin-catalog` registers the digest from the
   retained server image and completes successfully.
1. A new `openbao-server-migrations` Job starts after refresh completes. Its
   UID must differ from `$old_migration_uid`.
1. The new migration Job and Helm sync complete successfully.

Catalog registration does not reload plugins in running servers. If the source
catalog digest already matches the retained server image and the server pod
template is unchanged, wait for migrations without rotating servers. This was
the path exercised in the healthy-source rehearsal.

A stale catalog or changed server image or pod template needs a separately
validated rotation and recovery sequence. Stop rather than assuming the
healthy-source result covers that case. If that sequence calls for rotation,
handle one server at a time, standbys first and the active member last. A
single-member deployment has downtime during rotation.

For each server selected for rotation:

```bash
export POD=<one-standby-pod>
old_pod_uid=$(kcp -n vault-system get pod "$POD" -o jsonpath='{.metadata.uid}')
kcp -n vault-system delete pod "$POD" --wait=true &&
  kcp -n vault-system wait "pod/$POD" --for=create --timeout=5m &&
  kcp -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m &&
  kcp -n vault-system exec "$POD" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' || {
      echo 'Stop: replacement OpenBao pod did not pass health checks' >&2
      exit 1
    }
new_pod_uid=$(kcp -n vault-system get pod "$POD" -o jsonpath='{.metadata.uid}')
test -n "$new_pod_uid"
test "$new_pod_uid" != "$old_pod_uid"
```

Require a new pod UID, `Initialized true`, `Sealed false`, the expected image,
and healthy Raft membership before rotating another member. Readiness alone
does not prove that auto-unseal has finished. Recheck which member is active
before the last rotation. Never delete the StatefulSet, persistent-volume
claims, or multiple members together.

Confirm the migration Job and release status:

```bash
kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m
helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \
  status openbao-server -n vault-system
```

## 6. Validate the deployment

Check that every OpenBao member is healthy and unsealed, the catalog digest
matches the expected plugin, and the new migration Job succeeded. Then:

1. Exercise the existing JWT-backed authentication flow and invoke the original
   function again.
1. Compare the original function, deployment, cluster, database records, and
   worker identity with the baseline.
1. Admit a new test workload within the injector's configured webhook scope
   and verify that it receives its secrets.
1. Review a full stack diff. After the OpenBao checks pass, apply the remaining
   expected changes with the bundle's normal `make install` command.
1. Repeat the selected OpenBao sync and the full stack sync to check
   idempotency. Repeat the health, JWT, injector, and invocation checks.

Record failed probes even if the final checks pass. Keep the 0.6.2 values and
backups as the source baseline for the [1.0.1 upgrade](/nvcf/dev/release-notes/0-6-1-to-1-0-1-upgrade).

## Rehearsal results

A rehearsal upgraded released 0.6.1 to commit
`55803ad7cb647de45cf29415cfd3766225534617`, later published as stable 0.6.2,
on isolated ARM64 k3d. It used
three OpenBao members, one Cassandra member, and fake GPUs on one host.
Optional add-ons and observability were disabled, and local registry and
resource settings were adapted for the test.

Refresh completed before the new migration Job started. The server templates
and runtime images were unchanged, and all three members kept their pod
identities without rotation. Fresh JWT issuance, new secret injection, full
syncs, and repeat syncs passed. Existing credentials, PVCs, function and
deployment records, clean Cassandra ledgers, and worker identities were retained.
Each of the function-read, deployment-read, and invocation probes recorded
218 successes with no failures.

This evidence covers the healthy-source path at that commit. It does not cover
stale-catalog recovery, backup restoration, optional-component upgrades, or
production failure domains.

## Stop and recover

Stop on a failed hook, sealed member, dirty migration ledger, JWT or injector
failure, missing record, or unexpected identity change. Save Helm history and
hook logs before retrying. Do not skip hooks to force a successful release
status.

The original deployment and bundle are the recovery baseline before state
changes. Once a hook changes OpenBao state, a chart rollback alone is not a
complete recovery plan. Validate the catalog and runtime combination before
retrying, or restore the coordinated backup with your site's tested procedure.
Preserve all persistent volumes and original credentials.