> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server.

# NVCF Self-Hosted 0.6.1 Release Notes

NVCF Self-Hosted 0.6.1 is a maintenance release. It updates OpenBao to address
security issues in the runtime and migration images. It also replaces the
archived Bitnami Cassandra runtime with the official Apache Cassandra 5.0.8
image and an in-house Cassandra Helm chart.

## Highlights

| Change | Status | Summary |
| --- | --- | --- |
| OpenBao security update | GA | OpenBao uses the remediated runtime and migration images. Existing installations require a controlled server pod rotation after the Helm upgrade. |
| Cassandra off Bitnami | GA | Cassandra runs on the official Apache Cassandra 5.0.8 image, deployed by an in-house Helm chart, replacing the Bitnami image and subchart. |

## Cassandra runtime and chart

The Cassandra runtime image is now built on the official Apache Cassandra 5.0.8
image instead of the archived Bitnami image, and the Bitnami Cassandra Helm
subchart is replaced by an in-house StatefulSet chart. This removes the
CVE-affected Bitnami base and its bundled components while keeping the same
operational surface: PasswordAuthenticator and CassandraAuthorizer, stable-DNS
seed discovery, the Prometheus metrics exporter, keyspace initialization, and
schema migrations.

The chart runs the nodes with `storage_compatibility_mode: NONE` to match the
existing fleet's on-disk and streaming format.

## Upgrade Notes

Fresh installs need no special handling; Cassandra comes up on the official
image with the in-house chart.

Existing clusters that already hold data need a migration, because the runtime
image, the Helm chart shape, and the on-disk data layout all change. A normal
in-place Helm upgrade does not carry an existing Bitnami-based cluster onto the
new stack. Follow the
[0.6.0 to 0.6.1 Upgrade](/nvcf/v0.6.1-rc/0-6-0-to-0-6-1-upgrade) procedure, which describes
the supported migration methods (datacenter expansion, in-place volume
adoption, backup and restore) and their trade-offs.

## OpenBao security update

The OpenBao runtime and migration images include security fixes. OpenBao stores
control-plane secrets and uses persistent Raft storage, so upgrade it before
applying the rest of the patch stack.

The OpenBao StatefulSet uses `OnDelete`. The Helm upgrade updates the
StatefulSet template and runs the post-upgrade migration Job, but Kubernetes
does not restart existing OpenBao server pods. After the OpenBao release is
synced from the 0.6.1 stack, rotate the OpenBao server pods manually.

The pod rotation steps recreate pods from the existing StatefulSet. They do not
delete or recreate the OpenBao persistent volume claims (PVCs).

Troubleshooting OpenBao itself is outside the scope of this upgrade guide. If
OpenBao is sealed, uninitialized, unhealthy, or failing migration before or
during the procedure, stop and resolve OpenBao before continuing.

<Info>
These steps assume 0.6.0 is already deployed and healthy, that you have
downloaded the 0.6.1 stack, and that you run `make` from the extracted
`nvcf-self-managed-stack/` directory with your environment set.
</Info>

```bash
export HELMFILE_ENV=<your-environment>
```

### Prepare the 0.6.1 configuration

Return to the extracted 0.6.1 stack before running any 0.6.1 sync command.
Reconcile the site-specific settings from the 0.6.0 deployment into the 0.6.1
configuration files:

- `environments/$HELMFILE_ENV.yaml`
- `secrets/$HELMFILE_ENV-secrets.yaml`

Start from the files or templates included with 0.6.1 and carry forward the
required values from 0.6.0. Do not replace the 0.6.1 files wholesale because
available settings can change between releases. Preserve the existing registry,
storage, endpoint, credential, and secret values unless this procedure
explicitly instructs you to change them.

Confirm that both files exist before continuing:

```bash
for file in \
  "environments/${HELMFILE_ENV}.yaml" \
  "secrets/${HELMFILE_ENV}-secrets.yaml"; do
  [ -f "$file" ] || { echo "missing required file: $file"; exit 1; }
done
```

### Check OpenBao before the patch

Confirm that the current OpenBao release is healthy before starting the patch
upgrade:

```bash
kubectl -n vault-system get pods,jobs
kubectl -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
```

The OpenBao server pods should be `Running`, the OpenBao Jobs should be
`Complete`, and the update strategy should be `OnDelete`. Record the full image
references printed from the StatefulSet template. These are OpenBao image
references, not NVCF stack version numbers. After step 1, the StatefulSet
template should show the updated OpenBao image reference. Existing pods can
still show the previous OpenBao image reference until step 2 is complete.

Inspect each OpenBao server pod and record which pod is active:

```bash
for pod in $(kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

In the command output, the active pod reports `HA Mode active`. Standby pods
report `HA Mode standby` and include the active node address.

Do not continue if any pod is sealed, uninitialized, or not ready.

### Step 1: sync OpenBao from the 0.6.1 stack

Sync only the OpenBao release first. This updates the OpenBao release and
StatefulSet template, rolls the injector deployment, and runs the OpenBao
post-upgrade migration Job.

```bash
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=openbao-server
```

The command blocks until Helm finishes. If the command fails, stop and resolve
the OpenBao release before continuing. Do not rotate OpenBao server pods until
the post-upgrade migration Job has completed.

Confirm the OpenBao migration Job succeeded:

```bash
kubectl -n vault-system get jobs
# the openbao-server-migrations Job should report COMPLETIONS 1/1
```

If the migration Job fails, stop and resolve the OpenBao release before
continuing. Do not rotate OpenBao server pods while the migration Job is failed
or incomplete.

Confirm that the StatefulSet template now points at the updated OpenBao image
reference and still uses `OnDelete`:

```bash
kubectl -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
```

Confirm that the injector now injects the updated OpenBao image reference:

```bash
kubectl -n vault-system get deployment openbao-server-agent-injector \
  -o jsonpath='{range .spec.template.spec.containers[*].env[?(@.name=="AGENT_INJECT_VAULT_IMAGE")]}{.value}{"\n"}{end}'
```

<Info>
At this point the OpenBao Helm release and StatefulSet template are updated, but
existing OpenBao server pods can still be running the previous OpenBao image.
This is expected because the StatefulSet uses `OnDelete`.
</Info>

### Step 2: rotate OpenBao server pods

Rotate OpenBao server pods manually so each pod is recreated from the updated
StatefulSet template. Rotate standby pods first. Rotate the active pod last.

<Warning>
Do not delete all OpenBao server pods at once. Delete one pod, wait for it to
become ready, and verify that it is initialized and unsealed before deleting the
next pod. Do not delete the OpenBao StatefulSet or its PVCs.
</Warning>

<Info>
If your installation has only one OpenBao server pod, there are no standby pods.
Rotate the single pod after the migration Job completes, then wait for it to
become ready and verify that it is initialized and unsealed.
</Info>

List the pods and identify the standby pods:

```bash
for pod in $(kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

In the command output, rotate pods that report `HA Mode standby` first. Do not
rotate the pod that reports `HA Mode active` until all standby pods have been
recreated and verified.

For each standby pod, delete the pod and wait for it to return. Repeat this
block for every standby pod:

```bash
export POD=<standby-pod-name>

# Waits for the old pod to terminate, not for the replacement pod to be ready.
kubectl -n vault-system delete pod "$POD" --wait=true
kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m
kubectl -n vault-system exec "$POD" -c openbao -- sh -c \
  'BAO_ADDR=http://127.0.0.1:8200 bao status'
```

The recreated pod must report `Initialized true`, `Sealed false`, and a healthy
HA mode before you rotate the next pod.

After all standby pods are recreated and healthy, identify the active pod again:

```bash
for pod in $(kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

The active pod is the pod that reports `HA Mode active`.

Rotate the active pod last:

```bash
export POD=<active-pod-name>

# Waits for the old pod to terminate, not for the replacement pod to be ready.
kubectl -n vault-system delete pod "$POD" --wait=true
kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m
kubectl -n vault-system exec "$POD" -c openbao -- sh -c \
  'BAO_ADDR=http://127.0.0.1:8200 bao status'
```

If a recreated pod does not become ready before the timeout, or if it reports
`Sealed true` or `Initialized false`, stop the procedure and resolve that pod
before rotating any other OpenBao server pod.

### Verify OpenBao after rotation

Confirm that all OpenBao server pods are ready and use the updated OpenBao image
reference from the StatefulSet template:

```bash
kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server

kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'
```

Confirm that the StatefulSet reports all replicas updated and ready:

```bash
kubectl -n vault-system get statefulset openbao-server \
  -o jsonpath='replicas: {.status.replicas}{"\n"}readyReplicas: {.status.readyReplicas}{"\n"}updatedReplicas: {.status.updatedReplicas}{"\n"}'
```

The `replicas`, `readyReplicas`, and `updatedReplicas` values should match.
If `updatedReplicas` briefly lags after the final pod is recreated, wait a few
seconds and rerun the command. The per-pod image reference check above and
`bao status` check below are the primary verification points.

Confirm that each OpenBao server pod is initialized, unsealed, and participating
in HA:

```bash
for pod in $(kubectl -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kubectl -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

Do not continue if any pod is sealed, uninitialized, not ready, or still running
the previous OpenBao image.

### Sync the remaining 0.6.1 stack

After OpenBao is healthy on the updated OpenBao image, sync the rest of the
0.6.1 stack:

```bash
make install HELMFILE_ENV="$HELMFILE_ENV"
```

Confirm that the control plane is healthy:

```bash
kubectl get pods -A
```

This command should print nothing. Any output is a pod that is not yet
`Running` or `Completed`.