> This page is for version 1.1.0. > For other versions, use one of these documentation indexes: > - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt > - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt > - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. # 0.6.0 to 0.6.1 Upgrade The 0.6.1 release replaces the archived Bitnami Cassandra runtime with the official Apache Cassandra 5.0.8 image and an in-house Cassandra Helm chart. This removes the CVE-affected Bitnami runtime and its bundled components. Fresh installs need no special handling. Existing clusters that already hold data need a migration, because three things change at once: the runtime image, the Helm chart shape, and the on-disk data layout. A normal in-place Helm upgrade does not carry an existing Bitnami-based cluster onto the new stack: - The new Cassandra chart intentionally removes the `app.kubernetes.io/name` and `app.kubernetes.io/instance` labels from `volumeClaimTemplates.metadata.labels`. This is the only change to an immutable StatefulSet field. Kubernetes treats the entire volume claim template as immutable, so the StatefulSet has to be recreated. - The Bitnami image stored data nested one directory deeper than the official image expects (`/bitnami/cassandra/data/...` versus `/var/lib/cassandra/...`), so the new image does not find the old data by default. - The old and new node configuration must agree on the settings the existing data was written with. Because of this, an operator with an existing data-bearing cluster chooses a migration method. This guide describes the methods and their trade-offs; none is mandated over the others. The commands below use Bash, Helm 3, Helmfile 1.1.x, kubectl, jq, Python 3, and the NGC CLI. The data-bearing example covers the standard three-member Cassandra cluster and its Bitnami data layout. Other replica counts, separate commitlog volumes, custom data paths, or custom authentication require a site-tested migration procedure. For an existing data-bearing cluster, do not run an unscoped 0.6.1 stack install before completing the Cassandra steps for your chosen migration method. This includes `make install`, `helmfile sync`, and equivalent sync commands. An unscoped sync attempts to patch the existing Cassandra StatefulSet, fails on its immutable fields, leaves the Cassandra Helm release in a failed state, and stops the later Helmfile stages. If an unscoped sync already failed, do not uninstall Cassandra or delete its PersistentVolumeClaims. Stop and reconcile the failed release against your saved 0.6.0 configuration and data baseline before using a site-reviewed recovery procedure. The healthy-source checks below deliberately reject that failed state. ## Select the bundles and cluster Download each version into a separate directory. Keep your deployed 0.6.0 configuration and credentials; a download does not contain your site settings. Use a new directory outside the cluster for the evidence and backups. It must have capacity for all node snapshots and use your approved backup storage. ```bash set -euo pipefail set +x umask 077 helm version --short | grep -Eq '^v3\.' helmfile version --short | grep -Eq '^v?1\.1\.[0-9]+($|[+-])' export HELMFILE_ENV="your-existing-environment" export CONTROL_KUBECONFIG="/absolute/path/to/control-plane-kubeconfig" export CONTROL_CONTEXT="your-reviewed-control-plane-context" export UPGRADE_DIR="/absolute/path/to/new-upgrade-directory" if [ -e "$UPGRADE_DIR" ]; then echo "Directory already exists. Use its saved session for diagnostics; do not overwrite the baseline." >&2 exit 1 fi mkdir "$UPGRADE_DIR" cd "$UPGRADE_DIR" for version in 0.6.0 0.6.1; do ngc registry resource download-version \ "nvidia/nvcf/nvcf-self-managed-stack:${version}" mkdir "stack-${version}" tar -xzf \ "nvcf-self-managed-stack_v${version}/nvcf-self-managed-stack-${version}.tar.gz" \ -C "stack-${version}" done export SOURCE_CONTROL="$UPGRADE_DIR/stack-0.6.0" export TARGET_CONTROL="$UPGRADE_DIR/stack-0.6.1" export BACKUP_DIR="$UPGRADE_DIR/backup" mkdir "$BACKUP_DIR" kcp() { kubectl --kubeconfig "$CONTROL_KUBECONFIG" \ --context "$CONTROL_CONTEXT" "$@" } test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" \ = "$CONTROL_CONTEXT" kcp cluster-info CONTROL_CLUSTER_UID=$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}') test -n "$CONTROL_CLUSTER_UID" assert_context() { test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" \ = "$CONTROL_CONTEXT" && test "$(kcp get namespace kube-system -o jsonpath='{.metadata.uid}')" \ = "$CONTROL_CLUSTER_UID" } save_session() { local name session session=$(mktemp "$UPGRADE_DIR/.session.XXXXXX") { printf '%s\n' 'set -euo pipefail' 'set +x' 'umask 077' 'unset CQL_USER CQL_PASSWORD' for name in HELMFILE_ENV CONTROL_KUBECONFIG CONTROL_CONTEXT CONTROL_CLUSTER_UID \ UPGRADE_DIR SOURCE_CONTROL TARGET_CONTROL BACKUP_DIR CHECKER CQL_POD \ SNAPSHOT_TAG TARGET_CASSANDRA_IMAGE; do if declare -p "$name" >/dev/null 2>&1; then declare -p "$name"; fi done for name in kcp assert_context save_session capture_members load_cql_credentials \ cql cql_json capture_data; do if declare -F "$name" >/dev/null; then declare -f "$name"; fi done } > "$session" mv "$session" "$UPGRADE_DIR/session.sh" } save_session ``` Keep this shell open for the remaining commands. The bundle Makefile accepts a kubeconfig, but no context flag. `assert_context` must succeed immediately before each `make install`. Do not run concurrent maintenance or change this kubeconfig during the procedure. Stop on any command failure; do not skip a failed check or resume at a later mutation. ### Reopen a session for diagnostics The commands save the pinned cluster identity and available helpers in `$UPGRADE_DIR/session.sh`, adding the snapshot tag and target image when they are known. This private file excludes credentials. If a check closes the shell after setup, open a new Bash shell and source the saved session at its prompt: ```bash set -euo pipefail set +x export UPGRADE_DIR="/absolute/path/to/the-existing-upgrade-directory" source "$UPGRADE_DIR/session.sh" assert_context kcp -n cassandra-system get pods ``` Use this session only for read-only diagnostics until the failed check is resolved and the recovery procedure is reviewed. Do not rerun setup, recapture the original baseline, or repeat mutations to bypass the failure. If CQL setup was reached, run `load_cql_credentials` before CQL diagnostics; it reloads the credentials from the protected baseline files, even after the old Secret is removed. A changed context or cluster UID must be investigated, not repinned. ## Prepare the 0.6.1 configuration Run `cd "$TARGET_CONTROL"` before preparing the configuration or running a 0.6.1 sync command. Reconcile the site-specific settings from the 0.6.0 deployment into the 0.6.1 configuration files: - `environments/$HELMFILE_ENV.yaml` - `secrets/$HELMFILE_ENV-secrets.yaml` Start from the files or templates included with 0.6.1 and carry forward the required values from 0.6.0. Do not replace the 0.6.1 files wholesale because available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them. Do not change the Cassandra superuser password or application role passwords during this migration. Preserve the Cassandra and OpenBao credential values from the 0.6.0 deployment. The 0.6.1 Cassandra chart does not create the old `cassandra-system/cassandra` Secret, so do not rely on that Secret after the upgrade. Rotate Cassandra credentials in a separate maintenance procedure. Confirm that both files exist before continuing: ```bash cd "$TARGET_CONTROL" for file in \ "environments/${HELMFILE_ENV}.yaml" \ "secrets/${HELMFILE_ENV}-secrets.yaml"; do [ -s "$file" ] || { echo "missing or empty required file: $file"; exit 1; } done ``` ## Fresh installs A new 0.6.1 install has no existing Cassandra data and needs no migration. Install the stack normally; Cassandra comes up on the official image with the in-house chart: ```bash assert_context make install HELMFILE_ENV="$HELMFILE_ENV" \ KUBECONFIG_FILE="$CONTROL_KUBECONFIG" HELMFILE_SELECTOR= ``` The rest of this guide applies only when migrating an existing data-bearing cluster. ## Check Cassandra before migrating Download the read-only checker to `$UPGRADE_DIR/cassandra-upgrade-check.py`. It uses only the Python standard library and validates captured evidence; it does not contact or change a cluster. The checker rejects unsupported layouts and incomplete results. ```bash export CHECKER="$UPGRADE_DIR/cassandra-upgrade-check.py" save_session test -s "$CHECKER" python3 "$CHECKER" bundle "$SOURCE_CONTROL" 0.6.0 python3 "$CHECKER" bundle "$TARGET_CONTROL" 0.6.1 capture_members() { local directory=$1 pod mkdir "$directory" assert_context kcp -n cassandra-system get statefulset cassandra -o json > "$directory/statefulset.json" kcp -n cassandra-system get pods -o json > "$directory/pods.json" kcp -n cassandra-system get pvc -o json > "$directory/pvcs.json" for pod in cassandra-0 cassandra-1 cassandra-2; do kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool status \ > "$directory/$pod-ring.txt" done } save_session capture_members "$BACKUP_DIR/before" python3 "$CHECKER" topology "$BACKUP_DIR/before" > "$BACKUP_DIR/topology-before.json" helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \ -n cassandra-system list --all --filter '^cassandra$' --output json > "$BACKUP_DIR/cassandra-release.json" jq -e 'length == 1 and .[0].name == "cassandra" and .[0].namespace == "cassandra-system" and .[0].status == "deployed" and .[0].chart == "helm-nvcf-cassandra-0.15.5"' \ "$BACKUP_DIR/cassandra-release.json" >/dev/null helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \ -n cassandra-system get values cassandra --all -o json > "$BACKUP_DIR/cassandra-values.json" kcp -n cassandra-system get secret cassandra -o json > "$BACKUP_DIR/cassandra-secret.json" kcp -n cassandra-system get jobs -o json > "$BACKUP_DIR/jobs-before.json" ``` Success requires exactly the three expected pods, owned by the current StatefulSet, Running with every container Ready. Each node must report the same three `UN` (Up/Normal) ring members, matching the pod IPs. Each original data PVC must be Bound, without a deletion owner or Delete retention policy. The saved PVC/PV and Cassandra host identities are compared after adoption. The standard 0.6.0 superuser name is in `cassandra.dbUser.user` in the Helm values; its live password is in Secret `cassandra`, key `cassandra-password`. If the installation uses `dbUser.existingSecret`, a mapped key, password files, or a customized authentication/TLS setup, stop and adapt the site procedure before continuing. Do not substitute a default password. ```bash load_cql_credentials() { CQL_USER=$(jq -er '.cassandra.dbUser.user | strings | select(length > 0)' \ "$BACKUP_DIR/cassandra-values.json") CQL_PASSWORD=$(jq -er '.data["cassandra-password"] | @base64d | select(length > 0)' \ "$BACKUP_DIR/cassandra-secret.json") } CQL_POD=cassandra-0 cql() { printf '%s\n%s\n' "$CQL_USER" "$CQL_PASSWORD" | kcp -n cassandra-system exec -i "$CQL_POD" -c cassandra -- \ sh -c 'IFS= read -r u; IFS= read -r p; cqlsh -u "$u" -p "$p" "$@"' \ -- --no-color -e "$1" } cql_json() { local output=$1 query=$2 cql "PAGING OFF; CONSISTENCY LOCAL_QUORUM; $query" > "${output}.txt" python3 "$CHECKER" cql-json "${output}.txt" > "$output" } save_session load_cql_credentials cql 'DESCRIBE SCHEMA;' > "$BACKUP_DIR/schema-before.cql" cql_json "$BACKUP_DIR/keyspaces-before.json" \ 'SELECT JSON keyspace_name, replication FROM system_schema.keyspaces;' jq -e 'length > 0' "$BACKUP_DIR/keyspaces-before.json" >/dev/null ``` Keep the saved credentials private and this shell open. They remain available after the old Secret is removed. The password travels over stdin to the pod, not in the kubectl command arguments. Do not enable shell tracing. ### Preserve the target credentials In the existing target `secrets/$HELMFILE_ENV-secrets.yaml`, set `cassandra.dbUser.user` and `cassandra.dbUser.password` to the recorded superuser values. Set `cassandra.serviceRolePassword` to the source Helm value with that name. Preserve the existing `openbao.migrations.env` entry named `DEFAULT_CASSANDRA_PASSWORD` and its other array entries. This entry must equal the service-role password, which can differ from the superuser password. ```yaml cassandra: dbUser: user: "existing-superuser-name" password: "existing-superuser-password" serviceRolePassword: "existing-service-role-password" ``` Merge these keys into the existing secrets file; do not create a second `cassandra` key or replace the file. In 0.6.1, the environment file forwards `persistence.subPath`, but does not forward `dbUser`, `serviceRolePassword`, `hooks`, `cluster`, or `config`. Those chart settings belong in this secrets values file. A selected target render is checked before deleting the StatefulSet. ### Record a nonempty data and migration baseline Pause changes to the account records during this maintenance window. The example uses the normally small `nvcf_api.accounts` table; avoid unbounded counts on large workload tables. A missing or empty table cannot serve as a survival baseline. Record an existing function and its successful invocation through your normal client as an additional application-level baseline. ```bash capture_data() { local directory=$1 mkdir -p "$directory" cql_json "$directory/accounts-count.json" 'SELECT JSON count(*) FROM nvcf_api.accounts;' cql_json "$directory/accounts.json" 'SELECT JSON nca_id, name FROM nvcf_api.accounts;' jq -e --slurpfile rows "$directory/accounts.json" \ 'length == 1 and .[0].count > 0 and .[0].count == ($rows[0] | length)' \ "$directory/accounts-count.json" >/dev/null cql_json "$directory/migration-tables.json" \ "SELECT JSON table_name FROM system_schema.tables WHERE keyspace_name = 'schema_migrations';" jq -e 'length > 0 and all(.[]; .table_name | test("^[a-z][a-z0-9_]*$"))' \ "$directory/migration-tables.json" >/dev/null local tables table tables=$(jq -r '.[].table_name' "$directory/migration-tables.json") for table in $tables; do cql_json "$directory/migration-$table.json" \ "SELECT JSON version, dirty FROM schema_migrations.$table;" done } save_session capture_data "$BACKUP_DIR/data-before" python3 "$CHECKER" ledger "$BACKUP_DIR/data-before" > "$BACKUP_DIR/ledger-before.json" ``` Every discovered migration table must contain exactly one clean version record. The checker rejects CQL errors, partial output, empty inventories, missing or dirty records, and later missing tables or regressed versions. Target migrations can advance versions or add tables. Do not clear dirty flags to bypass a failure. ## Back up every Cassandra member Keep application writes paused until the baseline and snapshots are complete. Snapshots are node-local; copying one node is not a cluster backup. These commands save every tagged snapshot, schema, credentials, configuration, and identity evidence outside the cluster. Confirm that the destination satisfies your site's backup retention and access controls. The capacity check conservatively requires one current data footprint free on each node before flushing, and space for all nodes plus 10 percent at the backup destination. Resolve insufficient capacity before taking snapshots. ```bash for pod in cassandra-0 cassandra-1 cassandra-2; do kcp -n cassandra-system exec "$pod" -c cassandra -- sh -ec ' test -d /bitnami/cassandra/data/data command -v tar >/dev/null du -sk /bitnami/cassandra/data/data df -Pk /bitnami/cassandra/data/data ' > "$BACKUP_DIR/$pod-space.txt" kcp -n cassandra-system exec "$pod" -c cassandra -- \ cat /opt/bitnami/cassandra/conf/cassandra.yaml > "$BACKUP_DIR/$pod-cassandra.yaml" done python3 "$CHECKER" space "$BACKUP_DIR" SNAPSHOT_TAG="pre-0.6.1-$(date -u +%Y%m%dT%H%M%SZ)" save_session printf '%s\n' "$SNAPSHOT_TAG" > "$BACKUP_DIR/snapshot-tag.txt" for pod in cassandra-0 cassandra-1 cassandra-2; do capture_members "$BACKUP_DIR/pre-snapshot-$pod" python3 "$CHECKER" topology "$BACKUP_DIR/pre-snapshot-$pod" \ --baseline "$BACKUP_DIR/topology-before.json" --unchanged >/dev/null assert_context kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool flush kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool snapshot -t "$SNAPSHOT_TAG" kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool listsnapshots \ > "$BACKUP_DIR/$pod-snapshots.txt" kcp -n cassandra-system exec "$pod" -c cassandra -- sh -ec ' cd /bitnami/cassandra/data paths=$(find data -type d -path "*/snapshots/$1") test -n "$paths" printf "%s\n" "$paths" | tar -czf - -T - ' -- "$SNAPSHOT_TAG" > "$BACKUP_DIR/$pod-$SNAPSHOT_TAG.tar.gz" python3 "$CHECKER" archive "$BACKUP_DIR/$pod-$SNAPSHOT_TAG.tar.gz" "$SNAPSHOT_TAG" \ > "$BACKUP_DIR/$pod-backup.json" done ``` Each archive must be readable to its end, contain the requested tag, include application, authentication, and migration-state snapshots, and contain the SSTables listed in its manifests, including their data, metadata, compression or checksum information, and primary index files. A table-of-contents file is required for SSTables with SAI components. It is optional only for SSTables without SAI components, as seen in released system-table snapshots. An SAI column index can omit data components only when its completion marker identifies an empty index. The checker validates the released marker format and rejects missing primary data or partial nonempty indexes. See the [Cassandra SAI index format](https://github.com/apache/cassandra/blob/cassandra-5.0.5/src/java/org/apache/cassandra/index/sai/disk/v1/ColumnCompletionMarkerUtil.java). The checker records a SHA-256 for each archive. A checksum and readable archive establish transfer integrity, not a successful restore. Keep a separately rehearsed recovery procedure. If your backup destination is temporary, copy the whole evidence directory to durable protected storage and compare the archive checksums there before proceeding. Never remove the only backup, clear snapshots, or delete PVCs as an automatic retry. ## Choose a migration method Choose the method that fits your deployment and downtime requirements. The commands above and the in-place adoption checks cover the standard three-member source cluster. Datacenter expansion and backup/restore remain outlines that need a separate site-tested execution plan; they are not qualified by the in-place checks. | Method | Downtime | Data handling | Requires | |---|---|---|---| | Datacenter expansion | Effectively none | New nodes stream the data; old volumes untouched | Capacity to run both datacenters at once, datacenter-aware clients | | In-place volume adoption | Brief restart per node | The new pods reuse the existing volumes | A manual StatefulSet recreate; a subPath or a one-time layout relocation | | Backup and restore | Maintenance window | Snapshot the old cluster, load into a fresh one | Capacity for a parallel cluster during the copy | Datacenter expansion keeps the cluster online by adding the new stack as a second Cassandra datacenter and streaming data to it, then cutting clients over. It has the most moving parts and needs enough capacity to run both datacenters during the migration. In-place volume adoption reuses the existing data volumes. It is the smallest change in footprint, but it requires deleting and recreating the Cassandra StatefulSet (its volumes are retained) and pointing the new pods at the existing Bitnami on-disk layout. Backup and restore is the most conservative for data integrity: the existing cluster is left untouched until a fresh 0.6.1 cluster is verified. It needs a maintenance window and capacity to run both clusters during the copy. ## Method: datacenter expansion Add the 0.6.1 stack as a second Cassandra datacenter in the same cluster, replicate the data to it, move clients over, then remove the old datacenter. Because the new nodes bootstrap empty and receive their data by streaming, this method avoids the volume-layout and configuration-compatibility concerns of an in-place adoption. Validate this procedure in a non-production environment before running it against production. Perform the keyspace replication and decommission steps deliberately; an incorrect replication change or an early decommission can lose data. Two settings on the new datacenter's nodes decide whether the join and rebuild succeed: - `storage_compatibility_mode` must match the existing cluster. The existing Cassandra fleet runs `NONE` (full Cassandra 5.0 native format), and the 0.6.1 chart defaults the new nodes to `NONE` for this reason. If the new nodes run a different mode (the Apache Cassandra 5.0 default is `CASSANDRA_4`), gossip and schema exchange still succeed and the datacenters appear healthy, but the `nodetool rebuild` in step 4 fails: the streaming connection is rejected as version-incompatible. - Token allocation. When the existing cluster already holds a large schema, a node that joins with schema-aware token allocation (`allocate_tokens_for_local_replication_factor`) must reach schema readiness within a fixed window during bootstrap, or it aborts with a "Could not achieve schema readiness" error and restarts. If the new nodes fail to finish joining with that error, have them allocate tokens without the schema-aware option (random tokens are acceptable for a new, empty datacenter) and retry. 1. Stand up the new datacenter joined to the existing cluster. Deploy the 0.6.1 Cassandra nodes with the same cluster name as the existing cluster, a distinct datacenter name, seeds that reach the existing datacenter, and `auto_bootstrap: false` so the new nodes join without streaming during bootstrap. Do not run the schema-initialization or migration jobs yet. The schema already exists in the cluster, and the new nodes receive it through gossip. Disable both jobs in the new datacenter's values with `cassandra.hooks.initializeCluster.enabled: false` and `cassandra.hooks.migrations.enabled: false`. Keep them disabled until the new datacenter is included in the application and migration-state keyspaces in step 3. 2. Confirm both datacenters are visible and healthy from a node in either datacenter: ```bash kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status ``` Every node should report `UN`. The new datacenter nodes appear with no or minimal load until the rebuild in step 4. 3. Extend replication for every application keyspace, `schema_migrations`, `system_auth`, `system_distributed`, and `system_traces` to include the new datacenter. Replicating `schema_migrations` preserves the version and dirty state that the 0.6.1 migration job uses to apply only pending migrations. For each keyspace: ```sql ALTER KEYSPACE WITH replication = { 'class': 'NetworkTopologyStrategy', '': , '': }; ``` 4. On each node in the new datacenter, stream the existing data across: ```bash kcp -n cassandra-system exec "" -c cassandra -- \ nodetool rebuild -- "" ``` 5. Run the pending 0.6.1 schema migrations before moving clients. Keep cluster initialization disabled and enable only the migration hook: ```yaml cassandra: hooks: initializeCluster: enabled: false migrations: enabled: true ``` Apply the Cassandra release and wait for its migration job: ```bash set -euo pipefail assert_context make install HELMFILE_ENV="$HELMFILE_ENV" \ KUBECONFIG_FILE="$CONTROL_KUBECONFIG" HELMFILE_SELECTOR=name=cassandra kcp -n cassandra-system wait \ --for=condition=complete job/cassandra-migrations --timeout=10m kcp -n cassandra-system logs job/cassandra-migrations ``` The logs must end with `All Cassandra migrations completed successfully`. Do not continue if the job failed or timed out. 6. Inventory application keyspace replication after the migration job: ```sql SELECT keyspace_name, replication FROM system_schema.keyspaces; ``` For each application keyspace that does not list both datacenters, update its replication: ```sql ALTER KEYSPACE WITH replication = { 'class': 'NetworkTopologyStrategy', '': , '': }; ``` Run the `system_schema.keyspaces` query again. Confirm that every application keyspace lists both datacenters before continuing. If a newly created keyspace already contains data, rebuild that keyspace on each new-datacenter node before continuing: ```bash kcp -n cassandra-system exec "" -c cassandra -- \ nodetool rebuild -ks "" -- "" ``` 7. Confirm the migration state is clean: ```sql SELECT table_name FROM system_schema.tables WHERE keyspace_name = 'schema_migrations'; SELECT version, dirty FROM schema_migrations.; ``` Run the second query for each table returned by the first query. Each query must return one record with `dirty` set to `false`. This confirms that the cluster can receive later schema migrations. Do not move clients or remove the old datacenter while a migration is missing or dirty. 8. Move clients to the new datacenter. Point application traffic at the new datacenter using `LOCAL_QUORUM` against the new datacenter name, and confirm reads and writes succeed there. 9. Remove the old datacenter from replication, then decommission it. Alter every keyspace again to drop the old datacenter from the replication map, then decommission the old-datacenter nodes and remove the old stack. ## Method: in-place volume adoption Reuse the existing data volumes with the new stack. The StatefulSet must be recreated because its shape is immutable, and the new pods must be pointed at the Bitnami on-disk layout. The StatefulSet recreate step deletes the StatefulSet object but keeps its pods and PersistentVolumeClaims. Do not delete the PersistentVolumeClaims. Confirm the StatefulSet's `persistentVolumeClaimRetentionPolicy.whenDeleted` is not `Delete` before deleting it. Use `persistence.subPath: data` to expose the existing Bitnami layout at the paths the official image expects. This procedure does not relocate data files. Review the saved Cassandra configuration before continuing. Custom cluster names, datacenters, TLS, storage paths, or Cassandra settings need matching chart values and a site-tested procedure; do not assume defaults are compatible. 1. In the target `environments/$HELMFILE_ENV.yaml`, merge this value into the existing `cassandra` section: ```yaml cassandra: persistence: subPath: "data" ``` 2. Render the site's target Cassandra and OpenBao charts before stopping any old pod. Keep this output private because it contains credentials. The checker verifies the replica count, subPath, cluster/datacenter names, released runtime overrides, and preserved superuser/service credentials. The OpenBao render verifies the shared service credential; it does not apply or restart OpenBao. ```bash cd "$TARGET_CONTROL" assert_context python3 "$CHECKER" bundle "$TARGET_CONTROL" 0.6.1 HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \ --kubeconfig "$CONTROL_KUBECONFIG" \ --selector name=cassandra --selector name=openbao-server template \ > "$BACKUP_DIR/target-selected.yaml" kcp create --dry-run=client --validate=false -f "$BACKUP_DIR/target-selected.yaml" \ -o json > "$BACKUP_DIR/target-selected.json" python3 "$CHECKER" target "$BACKUP_DIR/target-selected.json" \ "$BACKUP_DIR/cassandra-values.json" "$BACKUP_DIR/cassandra-secret.json" \ "$BACKUP_DIR/before/statefulset.json" "$BACKUP_DIR" > "$BACKUP_DIR/target-checked.json" TARGET_CASSANDRA_IMAGE=$(jq -er '.image' "$BACKUP_DIR/target-checked.json") save_session ``` 3. Recheck the original members and every off-node archive immediately before adoption. Require a fresh, clean ledger and the same account baseline. ```bash capture_members "$BACKUP_DIR/pre-adoption" python3 "$CHECKER" topology "$BACKUP_DIR/pre-adoption" \ --baseline "$BACKUP_DIR/topology-before.json" --unchanged >/dev/null capture_data "$BACKUP_DIR/data-pre-adoption" python3 "$CHECKER" ledger "$BACKUP_DIR/data-pre-adoption" \ --baseline "$BACKUP_DIR/ledger-before.json" >/dev/null diff -u "$BACKUP_DIR/data-before/accounts.json" "$BACKUP_DIR/data-pre-adoption/accounts.json" diff -u "$BACKUP_DIR/data-before/accounts-count.json" "$BACKUP_DIR/data-pre-adoption/accounts-count.json" for pod in cassandra-0 cassandra-1 cassandra-2; do python3 "$CHECKER" archive "$BACKUP_DIR/$pod-$SNAPSHOT_TAG.tar.gz" "$SNAPSHOT_TAG" \ > "$BACKUP_DIR/$pod-backup-recheck.json" cmp "$BACKUP_DIR/$pod-backup.json" "$BACKUP_DIR/$pod-backup-recheck.json" done assert_context test "$(kcp -n cassandra-system get statefulset cassandra -o jsonpath='{.metadata.uid}')" \ = "$(jq -r '.statefulset_uid' "$BACKUP_DIR/topology-before.json")" kcp -n cassandra-system delete statefulset cassandra --cascade=orphan assert_context test "$(kcp -n cassandra-system get pod cassandra-0 -o jsonpath='{.metadata.uid}')" \ = "$(jq -r '.members["cassandra-0"].uid' "$BACKUP_DIR/topology-before.json")" kcp -n cassandra-system delete pod cassandra-0 assert_context make install HELMFILE_ENV="$HELMFILE_ENV" \ KUBECONFIG_FILE="$CONTROL_KUBECONFIG" HELMFILE_SELECTOR=name=cassandra kcp -n cassandra-system rollout status statefulset/cassandra --timeout=30m kcp -n cassandra-system wait --for=condition=complete \ job/cassandra-initialize-cluster job/cassandra-migrations --timeout=10m kcp -n cassandra-system get jobs -o json > "$BACKUP_DIR/jobs-after-cassandra.json" python3 "$CHECKER" jobs "$BACKUP_DIR/jobs-after-cassandra.json" \ "$BACKUP_DIR/jobs-before.json" kcp -n cassandra-system logs job/cassandra-migrations \ > "$BACKUP_DIR/cassandra-migrations.log" ``` The StatefulSet controller recreates `cassandra-0` and replaces the remaining pods in reverse ordinal order. Do not delete those pods manually. A failed install or timeout stops the procedure even if a pod later becomes Ready. Keep the PVCs and backups and inspect the ring, events, and hook logs. 4. Continue to [Verify Cassandra](#verify-cassandra-before-continuing). A new Ready pod or a successful Helm command alone does not establish adoption. ## Method: backup and restore Leave the existing cluster in place and load its data into a fresh 0.6.1 cluster. 1. Snapshot the existing cluster: on every node, run `nodetool flush` then `nodetool snapshot`, and copy the snapshots off-node. 2. Deploy a fresh 0.6.1 Cassandra cluster on new volumes and let the init and migration jobs create the schema. 3. Restore the data into the new cluster with `sstableloader`, or place the snapshot sstables and run `nodetool refresh` per table. 4. Verify row counts and application connectivity against the new cluster, then decommission the old cluster. ## Verify Cassandra before continuing For in-place adoption, require all three new pods on the target image and StatefulSet revision, the same original PVC UIDs/PVs and Cassandra host IDs, and the same healthy ring. All original pods must have been replaced. For a separately tested datacenter or restore migration, use that procedure's identity mapping instead; the in-place identity comparison does not apply to new volumes. ```bash capture_members "$BACKUP_DIR/after-cassandra" python3 "$CHECKER" topology "$BACKUP_DIR/after-cassandra" \ --baseline "$BACKUP_DIR/topology-before.json" --image "$TARGET_CASSANDRA_IMAGE" \ > "$BACKUP_DIR/topology-after-cassandra.json" capture_data "$BACKUP_DIR/data-after-cassandra" python3 "$CHECKER" ledger "$BACKUP_DIR/data-after-cassandra" \ --baseline "$BACKUP_DIR/ledger-before.json" > "$BACKUP_DIR/ledger-after-cassandra.json" diff -u "$BACKUP_DIR/data-before/accounts.json" "$BACKUP_DIR/data-after-cassandra/accounts.json" diff -u "$BACKUP_DIR/data-before/accounts-count.json" "$BACKUP_DIR/data-after-cassandra/accounts-count.json" ``` Do not proceed if these commands fail. Count equality is a bounded baseline check, not proof that every table survived. Confirm that the original function and its deployment are still retrievable through your existing client. ## Apply the remaining 0.6.1 stack The Cassandra selector leaves other releases at their old versions. After Cassandra verification, render the full target stack with the same site files and review the changes. The template contains secrets; keep it in the private backup directory. Clear an inherited selector explicitly on the full install. ```bash cd "$TARGET_CONTROL" assert_context python3 "$CHECKER" bundle "$TARGET_CONTROL" 0.6.1 HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \ --kubeconfig "$CONTROL_KUBECONFIG" template > "$BACKUP_DIR/target-full.yaml" kcp -n cassandra-system get jobs -o json > "$BACKUP_DIR/jobs-before-full.json" ``` Check the render against your source configuration before running the next block. Preserve the Cassandra `subPath` and credentials and OpenBao `OnDelete` strategy. Do not restart OpenBao at this stage. Save the existing OpenBao pod UIDs and live images for the required continuation below. ```bash kcp -n vault-system get pods -o json > "$BACKUP_DIR/openbao-before-full.json" assert_context make install HELMFILE_ENV="$HELMFILE_ENV" \ KUBECONFIG_FILE="$CONTROL_KUBECONFIG" HELMFILE_SELECTOR= HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \ --kubeconfig "$CONTROL_KUBECONFIG" list --skip-charts --output json \ > "$BACKUP_DIR/expected-releases.json" helm --kubeconfig "$CONTROL_KUBECONFIG" --kube-context "$CONTROL_CONTEXT" \ list --all-namespaces --all --max 10000 --output json > "$BACKUP_DIR/installed-releases.json" python3 "$CHECKER" releases "$BACKUP_DIR/expected-releases.json" \ "$BACKUP_DIR/installed-releases.json" kcp -n cassandra-system get jobs -o json > "$BACKUP_DIR/jobs-after-full.json" python3 "$CHECKER" jobs "$BACKUP_DIR/jobs-after-full.json" "$BACKUP_DIR/jobs-before-full.json" capture_members "$BACKUP_DIR/after-full" python3 "$CHECKER" topology "$BACKUP_DIR/after-full" \ --baseline "$BACKUP_DIR/topology-before.json" --image "$TARGET_CASSANDRA_IMAGE" >/dev/null capture_data "$BACKUP_DIR/data-after-full" python3 "$CHECKER" ledger "$BACKUP_DIR/data-after-full" \ --baseline "$BACKUP_DIR/ledger-after-cassandra.json" >/dev/null diff -u "$BACKUP_DIR/data-before/accounts.json" "$BACKUP_DIR/data-after-full/accounts.json" diff -u "$BACKUP_DIR/data-before/accounts-count.json" "$BACKUP_DIR/data-after-full/accounts-count.json" ``` The release comparison requires every enabled target release to be `deployed` with its expected chart version. It includes optional releases when enabled; it does not compare unrelated Helm releases. Recheck application readiness, the original function, and fresh authentication. This command updates the control-plane bundle, not the separate compute-plane or observability bundles. ## Continue through the OpenBao 0.6.2 patch The full 0.6.1 apply updates the OpenBao StatefulSet template to 2.5.5-nv-1.3.1. Its `OnDelete` strategy leaves existing server pods running 2.5.4-nv-1.3.0 until they are replaced. A deployed Helm release does not mean those running servers have upgraded. Compare the template and the actual server and auto-unseal container images: ```bash kcp -n vault-system get statefulset openbao-server \ -o jsonpath='{.spec.updateStrategy.type}{"\n"}{range .spec.template.spec.containers[*]}{.name}{" "}{.image}{"\n"}{end}' kcp -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \ -o json | jq -r '.items[] | .metadata.name as $pod | .metadata.labels["controller-revision-hash"] as $revision | .spec.containers[] | [$pod,.name,.image,$revision] | @tsv' ``` Do not restart or delete OpenBao server pods, reload the JWT plugin, or change the update strategy at 0.6.1. The old plugin catalog must be refreshed before new binaries can start safely. Keep the healthy existing members running and continue directly to the [0.6.1 to 0.6.2 procedure](/nvcf/1.1.0/release-notes/0-6-1-to-0-6-2-upgrade), which detects inherited old pods even when the target template is unchanged, refreshes the catalog, and replaces standbys before gracefully replacing the active member. If a member has already restarted, is sealed, or cannot issue fresh JWTs, stop and assess that state before continuing. Do not delete another member to recover. The overall upgrade is complete only after the 0.6.2 runtime, per-pod revision, Raft, authentication and invocation checks pass.