Release Notes

NVCF Self-Hosted 0.6.1 Release Notes

View as Markdown

NVCF Self-Hosted 0.6.1 is a maintenance release. It updates OpenBao to address security issues in the runtime and migration images. It also replaces the archived Bitnami Cassandra runtime with the official Apache Cassandra 5.0.8 image and an in-house Cassandra Helm chart.

Highlights

ChangeStatusSummary
OpenBao security updateGAOpenBao uses the remediated runtime and migration images. Existing installations require a controlled server pod rotation after the Helm upgrade.
Cassandra off BitnamiGACassandra runs on the official Apache Cassandra 5.0.8 image, deployed by an in-house Helm chart, replacing the Bitnami image and subchart.

Cassandra runtime and chart

The Cassandra runtime image is now built on the official Apache Cassandra 5.0.8 image instead of the archived Bitnami image, and the Bitnami Cassandra Helm subchart is replaced by an in-house StatefulSet chart. This removes the CVE-affected Bitnami base and its bundled components while keeping the same operational surface: PasswordAuthenticator and CassandraAuthorizer, stable-DNS seed discovery, the Prometheus metrics exporter, keyspace initialization, and schema migrations.

The chart runs the nodes with storage_compatibility_mode: NONE to match the existing fleet’s on-disk and streaming format.

Upgrade Notes

Fresh installs need no special handling; Cassandra comes up on the official image with the in-house chart.

Existing clusters that already hold data need a migration, because the runtime image, the Helm chart shape, and the on-disk data layout all change. A normal in-place Helm upgrade does not carry an existing Bitnami-based cluster onto the new stack. Follow the 0.6.0 to 0.6.1 Upgrade procedure, which describes the supported migration methods (datacenter expansion, in-place volume adoption, backup and restore) and their trade-offs.

OpenBao security update

The OpenBao runtime and migration images include security fixes. OpenBao stores control-plane secrets and uses persistent Raft storage, so upgrade it before applying the rest of the patch stack.

The OpenBao StatefulSet uses OnDelete. The Helm upgrade updates the StatefulSet template and runs the post-upgrade migration Job, but Kubernetes does not restart existing OpenBao server pods. After the OpenBao release is synced from the 0.6.1 stack, rotate the OpenBao server pods manually.

The pod rotation steps recreate pods from the existing StatefulSet. They do not delete or recreate the OpenBao persistent volume claims (PVCs).

Troubleshooting OpenBao itself is outside the scope of this upgrade guide. If OpenBao is sealed, uninitialized, unhealthy, or failing migration before or during the procedure, stop and resolve OpenBao before continuing.

These steps assume 0.6.0 is already deployed and healthy, that you have downloaded the 0.6.1 stack, and that you run make from the extracted nvcf-self-managed-stack/ directory with your environment set.

$export HELMFILE_ENV=<your-environment>

Prepare the 0.6.1 configuration

Return to the extracted 0.6.1 stack before running any 0.6.1 sync command. Reconcile the site-specific settings from the 0.6.0 deployment into the 0.6.1 configuration files:

  • environments/$HELMFILE_ENV.yaml
  • secrets/$HELMFILE_ENV-secrets.yaml

Start from the files or templates included with 0.6.1 and carry forward the required values from 0.6.0. Do not replace the 0.6.1 files wholesale because available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them.

Confirm that both files exist before continuing:

$for file in \
>  "environments/${HELMFILE_ENV}.yaml" \
>  "secrets/${HELMFILE_ENV}-secrets.yaml"; do
$  [ -f "$file" ] || { echo "missing required file: $file"; exit 1; }
$done

Check OpenBao before the patch

Confirm that the current OpenBao release is healthy before starting the patch upgrade:

$kubectl -n vault-system get pods,jobs
$kubectl -n vault-system get statefulset openbao-server \
>  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'

The OpenBao server pods should be Running, the OpenBao Jobs should be Complete, and the update strategy should be OnDelete. Record the full image references printed from the StatefulSet template. These are OpenBao image references, not NVCF stack version numbers. After step 1, the StatefulSet template should show the updated OpenBao image reference. Existing pods can still show the previous OpenBao image reference until step 2 is complete.

Inspect each OpenBao server pod and record which pod is active:

$for pod in $(kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server \
>  -o jsonpath='{.items[*].metadata.name}'); do
$  echo "== $pod =="
$  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
>    'BAO_ADDR=http://127.0.0.1:8200 bao status'
$done

In the command output, the active pod reports HA Mode active. Standby pods report HA Mode standby and include the active node address.

Do not continue if any pod is sealed, uninitialized, or not ready.

Step 1: sync OpenBao from the 0.6.1 stack

Sync only the OpenBao release first. This updates the OpenBao release and StatefulSet template, rolls the injector deployment, and runs the OpenBao post-upgrade migration Job.

$make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=openbao-server

The command blocks until Helm finishes. If the command fails, stop and resolve the OpenBao release before continuing. Do not rotate OpenBao server pods until the post-upgrade migration Job has completed.

Confirm the OpenBao migration Job succeeded:

$kubectl -n vault-system get jobs
$# the openbao-server-migrations Job should report COMPLETIONS 1/1

If the migration Job fails, stop and resolve the OpenBao release before continuing. Do not rotate OpenBao server pods while the migration Job is failed or incomplete.

Confirm that the StatefulSet template now points at the updated OpenBao image reference and still uses OnDelete:

$kubectl -n vault-system get statefulset openbao-server \
>  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'

Confirm that the injector now injects the updated OpenBao image reference:

$kubectl -n vault-system get deployment openbao-server-agent-injector \
>  -o jsonpath='{range .spec.template.spec.containers[*].env[?(@.name=="AGENT_INJECT_VAULT_IMAGE")]}{.value}{"\n"}{end}'

At this point the OpenBao Helm release and StatefulSet template are updated, but existing OpenBao server pods can still be running the previous OpenBao image. This is expected because the StatefulSet uses OnDelete.

Step 2: rotate OpenBao server pods

Rotate OpenBao server pods manually so each pod is recreated from the updated StatefulSet template. Rotate standby pods first. Rotate the active pod last.

Do not delete all OpenBao server pods at once. Delete one pod, wait for it to become ready, and verify that it is initialized and unsealed before deleting the next pod. Do not delete the OpenBao StatefulSet or its PVCs.

If your installation has only one OpenBao server pod, there are no standby pods. Rotate the single pod after the migration Job completes, then wait for it to become ready and verify that it is initialized and unsealed.

List the pods and identify the standby pods:

$for pod in $(kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server \
>  -o jsonpath='{.items[*].metadata.name}'); do
$  echo "== $pod =="
$  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
>    'BAO_ADDR=http://127.0.0.1:8200 bao status'
$done

In the command output, rotate pods that report HA Mode standby first. Do not rotate the pod that reports HA Mode active until all standby pods have been recreated and verified.

For each standby pod, delete the pod and wait for it to return. Repeat this block for every standby pod:

$export POD=<standby-pod-name>
$
$# Waits for the old pod to terminate, not for the replacement pod to be ready.
$kubectl -n vault-system delete pod "$POD" --wait=true
$kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m
$kubectl -n vault-system exec "$POD" -c openbao -- sh -c \
>  'BAO_ADDR=http://127.0.0.1:8200 bao status'

The recreated pod must report Initialized true, Sealed false, and a healthy HA mode before you rotate the next pod.

After all standby pods are recreated and healthy, identify the active pod again:

$for pod in $(kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server \
>  -o jsonpath='{.items[*].metadata.name}'); do
$  echo "== $pod =="
$  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
>    'BAO_ADDR=http://127.0.0.1:8200 bao status'
$done

The active pod is the pod that reports HA Mode active.

Rotate the active pod last:

$export POD=<active-pod-name>
$
$# Waits for the old pod to terminate, not for the replacement pod to be ready.
$kubectl -n vault-system delete pod "$POD" --wait=true
$kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m
$kubectl -n vault-system exec "$POD" -c openbao -- sh -c \
>  'BAO_ADDR=http://127.0.0.1:8200 bao status'

If a recreated pod does not become ready before the timeout, or if it reports Sealed true or Initialized false, stop the procedure and resolve that pod before rotating any other OpenBao server pod.

Verify OpenBao after rotation

Confirm that all OpenBao server pods are ready and use the updated OpenBao image reference from the StatefulSet template:

$kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server
$
$kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server \
>  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'

Confirm that the StatefulSet reports all replicas updated and ready:

$kubectl -n vault-system get statefulset openbao-server \
>  -o jsonpath='replicas: {.status.replicas}{"\n"}readyReplicas: {.status.readyReplicas}{"\n"}updatedReplicas: {.status.updatedReplicas}{"\n"}'

The replicas, readyReplicas, and updatedReplicas values should match. If updatedReplicas briefly lags after the final pod is recreated, wait a few seconds and rerun the command. The per-pod image reference check above and bao status check below are the primary verification points.

Confirm that each OpenBao server pod is initialized, unsealed, and participating in HA:

$for pod in $(kubectl -n vault-system get pods \
>  -l app.kubernetes.io/name=openbao,component=server \
>  -o jsonpath='{.items[*].metadata.name}'); do
$  echo "== $pod =="
$  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
>    'BAO_ADDR=http://127.0.0.1:8200 bao status'
$done

Do not continue if any pod is sealed, uninitialized, not ready, or still running the previous OpenBao image.

Sync the remaining 0.6.1 stack

After OpenBao is healthy on the updated OpenBao image, sync the rest of the 0.6.1 stack:

$make install HELMFILE_ENV="$HELMFILE_ENV"

Confirm that the control plane is healthy:

$kubectl get pods -A

This command should print nothing. Any output is a pod that is not yet Running or Completed.