> This page is for version 1.1.0.
> For other versions, use one of these documentation indexes:
> - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt
> - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt
> - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.

# Upgrade from NVCF Self-Managed 0.6.2 to 1.0.1

Complete the [0.6.1 to 0.6.2 patch upgrade](/nvcf/1.1.0/release-notes/0-6-1-to-0-6-2-upgrade)
and its health checks first. The remaining steps start from healthy 0.6.2.
Use the stable bundle and inventory, not a release candidate.

Before scheduling maintenance, check the
[task worker image and NATS recovery notes](/nvcf/1.1.0/self-managed/upgrade-recovery)
for the historical target's known limitations.

| Plane | Source | Target |
| --- | --- | --- |
| Control plane | `nvcf-self-managed-stack` 0.6.2 | `nvcf-self-managed-stack` 1.0.1 |
| Compute plane | N/A | `nvcf-compute-plane-stack` 1.0.0 |
| Observability | Included with the 0.6.2 control plane | `nvcf-observability-stack` 1.0.0 |
| NVCA | 3.0.3 | 3.10.0 |

The 1.0.1 control-plane stack is a patch release on top of 1.0.0. The
compute-plane and observability targets remain at 1.0.0. The ordered
migration stops in this procedure still apply.

This procedure uses staged schema changes and Kubernetes rolling updates. The
maintenance window must be rehearsed for the deployment topology. The
availability rehearsal below did not pass. Block control-plane mutations during
the final validation and contract steps. Existing function workloads can
continue running in compute clusters throughout the upgrade.

<Warning>
This procedure supports only a healthy 0.6.2 starting state. Do not use it to
skip an earlier documented upgrade. Do not continue after a failed or dirty
database migration, a sealed OpenBao member, a changed cluster identity, or a
missing GPU inventory record.
</Warning>

## Rehearsal coverage

A 2026-10-01 diagnostic rehearsal applied the 0.6.2 source commit, later
published as stable, then exercised this migration on two ARM64 k3d nodes on
one host. It used one Cassandra member, three OpenBao and NATS members, and
fake GPUs. Bounded migrations, service backfills, contract steps, and repeat
syncs completed; the original function, deployment, worker, and storage
identities were retained.

Availability did not pass: probes recorded 77 failures during Cassandra
restart and recovery, 21 HTTP 401 failures during the OpenBao transition, and
one invocation connection failure during full sync. Testing continued only
after health and clean migration ledgers were restored. This is not evidence
for an uninterrupted or production-qualified upgrade.

The rehearsal did not cover real GPUs, independent failure domains,
observability, optional add-ons, successful task execution, shared storage,
stale-catalog recovery, or backup restoration. It used a receipt-helper image
override and an explicit control-plane profile. Verify required artifact
availability and the target CLI's profile export before the maintenance window.
The [worker-image](https://github.com/dsx-ai-factory/nvcf/issues/2196),
[receipt-helper](https://github.com/dsx-ai-factory/nvcf/issues/2197),
[profile-export distribution](https://github.com/dsx-ai-factory/nvcf/issues/2785), and
[instance-timestamp](https://github.com/dsx-ai-factory/nvcf/issues/2199) follow-ups
record the remaining findings from that rehearsal.

## Upgrade stops

The source Cassandra migration image is 0.10.3. A healthy 0.6.2 database has
already completed the NVCF GPU specification and model specification
migrations and records NVCF schema version 7. The source NVCF API maps only
the post-migration data model. Do not deploy an older NVCF API runtime to
repeat those migrations.

The remaining Cassandra and service changes require the following ordered
gates:

| Gate | Schema transition | Service action | Completion check |
| --- | --- | --- | --- |
| Cassandra server | 5.0.8 to 5.0.9 | Upgrade the StatefulSet with migration hooks disabled. Rotate one member at a time. | Every member rejoins and `nodetool status` reports the expected healthy topology. |
| Expand | `nvcf_api` 7 to 8 | None. The change adds nullable `llm_config`. | Ledger is 8 and not dirty. |
| Expand | `sis_api` 6 to 10 | Add the instance lookup columns and indexes, the reservation flag, and cluster lookup indexes. | Ledger is 10 and not dirty. |
| OpenBao | No Cassandra change | Upgrade OpenBao and run migrations 0.16.2 to 0.19.5 before using the target service charts. | Every member is initialized, unsealed, ready, and uses the target image. The migration Job completes. |
| Roll ICMS bridge | `sis_api` remains at 10 | Roll ICMS 0.7.5 with the backfill mode initially `DISABLED`. New writes populate deployment and GPU specification IDs. | The Deployment rollout completes and probes pass the agreed observation interval. |
| Backfill ICMS | `sis_api` remains at 10 | Set `ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE` to `MIGRATE`, then `VALIDATE`, then `DISABLED`. | Validation reports no unresolved active instance rows and no mismatched IDs. |
| Roll ICMS target | `sis_api` remains at 10 | Roll the stack-pinned ICMS 0.7.2 after disabling the bridge task. | Every bridge pod is gone and probes pass the agreed observation interval. |
| Expand | `nvct_api` 3 to 4 | Add `tasks_v2.health` while retaining `health_info` and `health_udt`. | Ledger is 4 and both columns exist. |
| Roll NVCT bridge | `nvct_api` remains at 4 | Roll NVCT 1.64.3 with the backfill mode initially `DISABLED`. It writes `health` and reads `health` before falling back to `health_info`. | Every old pod is gone, the rollout completes, and probes pass the agreed observation interval. |
| Backfill NVCT | `nvct_api` remains at 4 | Set `NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE` to `MIGRATE`, then `VALIDATE`, then `DISABLED`. | Validation reports no row with legacy health and an empty or mismatched replacement value. |
| Roll NVCT target | `nvct_api` remains at 4 | Roll NVCT 1.66.0 after the backfill passes. This runtime no longer reads the legacy column. | Every bridge pod is gone and probes pass the agreed observation interval. |
| Contract | `nvct_api` 4 to 6 | Drop `health_info`, then drop `health_udt` while NVCT 1.66.0 remains online. | Ledger is 6, only `health` exists, and retained task health is unchanged. |
| Finish | Remaining ledgers reach the target versions | Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal Cassandra Helm release once all bounded migrations are complete. | All ledger, service, and data checks pass. |

The migration image contains the standard `migrate` CLI. It supports
`goto V` and `up N` for one migration path and migration table. The image's
default `/app/execute_sqls.sh` entrypoint always runs `up` for every keyspace,
and Cassandra migrations 0.17.1 through 0.17.6 package NVCT migrations 4, 5,
and 6 together. The released chart does not expose a command override or a
per-keyspace target. Use a bounded Job that invokes the existing image with
exact targets, or a chart update that exposes the same control. A new migration
binary is not required.

Each bounded Job must use the preprocessed SQL path for one keyspace and the
matching table in the `schema_migrations` keyspace. Before invoking `goto`, it
must require the expected clean source version. Afterward, it must require the
exact clean target version. Never use `goto` to move a ledger backward. For
example, the NVCT expand Job uses the `nvct_api` migration path and ledger,
requires version 3, invokes `goto 4`, and then requires version 4.

The service rollout and the data task are separate gates. NVCT 1.64.3 provides
the required dual-read compatibility and the health backfill. ICMS 0.7.5
provides the instance lookup backfill. Both tasks expose explicit `MIGRATE`,
`VALIDATE`, and `DISABLED` modes and default to `DISABLED`.
Run migration mode to completion, switch to validation mode, require zero
mismatches, and disable the task before rolling past the bridge. A completion
log without a validation pass is not a safe checkpoint.

The two upgrade-only images are:

| Service | Upgrade-only version | Final stack version | Task environment variable |
| --- | --- | --- | --- |
| ICMS | 0.7.5 | 0.7.2 | `ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE` |
| NVCT | 1.64.3 | 1.66.0 | `NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE` |

Use the immutable published versions listed above.

Helmfile upgrades one release at a time with `--selector name=<release>`.
Helm updates the Deployment and Kubernetes performs the rolling update. Use at
least two replicas for each service stop and require `maxUnavailable: 0` and
`maxSurge: 1`. The cluster must have capacity for the surge pod. A three-member
Cassandra cluster is required to keep the database available while its
StatefulSet rolls. A one-member Cassandra deployment has unavoidable database
downtime during the server restart.

The full path is:

1. Roll the Cassandra members with hooks disabled, then apply only the bounded
   expand migrations.
2. Upgrade OpenBao and complete its migrations. The target ICMS and NVCT
   charts require the target OpenBao roles and policies.
3. Roll ICMS 0.7.5 and NVCT 1.64.3. Run and validate both data tasks, then set
   both modes back to `DISABLED`.
4. Roll ICMS 0.7.2 and NVCT 1.66.0. Apply the NVCT contract migrations while
   NVCT remains online.
5. Sync the remaining target control-plane and observability releases. Run the
   normal Cassandra release and require its migration hook to be a no-op.
6. Re-register each existing compute cluster without deleting its identity,
   then sync compute-plane 1.0.0 and NVCA 3.10.0.

The source and target service versions remain:

| Service | Source runtime | Target runtime |
| --- | --- | --- |
| NVCF API | 1.9.0-hotfix.1 | 1.18.0 |
| NVCT API | 1.5.9-hotfix.1 | 1.66.0 |
| SIS / ICMS | 1.563.1-hotfix.1 | 0.7.2 |
| ESS API | 0.57.26 | 0.5.0 |
| NVCA | 3.0.3 | 3.10.0 |

The version-number decreases for ICMS and ESS reflect component lineage
changes. They are the runtimes shipped with the target stack set.

## Prepare the deployment host

Before maintenance, [download a packaged `nvcf-cli`](/nvcf/1.1.0/manifest/image-mirroring#downloading-nvcf-cli)
from NGC. Before running the linked commands, replace their `VERSION="1.16.2"`
assignment with `VERSION="1.16.8"` or a later release. If a qualifying binary
is unavailable for your host, postpone the upgrade.

Download and extract these target bundles into new directories. The download
commands are on the [image mirroring](/nvcf/1.1.0/manifest/image-mirroring) page.

- `nvcf-self-managed-stack` 1.0.1
- `nvcf-compute-plane-stack` 1.0.0
- `nvcf-observability-stack` 1.0.0

The default `observability.profile` is `control`. With the `control`, `compute`,
or `all` profile, the control-plane Helmfile loads
`../../observability/helmfile.d/01-observability.yaml.gotmpl`, so the
observability bundle is required. When the profile is explicitly set to
`disabled`, Helmfile does not load that file, and the observability components
and Function Autoscaler are not installed.

For the default `control` profile, extract the observability bundle into a
directory named `observability` next to the control-plane bundle directory:

```text
<target-directory>/
|-- nvcf-self-managed-stack/
`-- observability/
```

The observability tarball has no top-level directory, so create the
directory first:

```bash
mkdir -p <target-directory>/observability
tar -xzf <observability-tarball> -C <target-directory>/observability
```

Without this directory, `make template` and `make install` fail with
`no matches for path: ../../observability/helmfile.d/01-observability.yaml.gotmpl`.

Do not extract a target bundle over the 0.6.2 directories. Keep the source
configuration available for recovery.

Set variables for the deployment. Use a kubeconfig that selects only the
control-plane cluster, or verify the selected context before every command.

```bash
export HELMFILE_ENV=<environment-name>
export SOURCE_CONTROL=<absolute-path-to-0.6.2-control-plane-stack>
export TARGET_CONTROL=<absolute-path-to-1.0.1-control-plane-stack>
export TARGET_COMPUTE=<absolute-path-to-1.0.0-compute-plane-stack>
export CONTROL_KUBECONFIG=<absolute-path-to-control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export NVCF_CLI=<absolute-path-to-nvcf-cli>
export NVCF_CLI_CONFIG=<absolute-path-to-nvcf-cli-config>

kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}

kubeconfig_context=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
  config current-context)
test "$kubeconfig_context" = "$CONTROL_CONTEXT"
kcp cluster-info
```

Use Helm 3, Helmfile 1.1.x, and kubectl 1.31 or later. The OpenBao rotation
uses `kubectl wait --for=create`, which older kubectl versions do not
support. Later steps reuse the shell variables and helper functions defined in
earlier steps, so run the whole procedure in one shell, or put the variables
and functions in a file and source it in every terminal you use. Authenticate
Helm and the target clusters to all registries used by the target bundles.
Mirror every required target artifact before the maintenance window. Use the
inventory JSON attached to each target stack's GitHub release as the artifact
list. See the
[image mirroring](/nvcf/1.1.0/manifest/image-mirroring) page for the mirroring procedure.

| Stack | GitHub release | Inventory file |
| --- | --- | --- |
| Control plane | `deploy/stacks/self-managed/v1.0.1` | `nvcf-self-managed-stack-inventory.json` |
| Compute plane | `deploy/stacks/nvcf-compute-plane/v1.0.0` | `nvcf-compute-plane-stack-inventory.json` |
| Observability | `deploy/stacks/observability/v1.0.0` | `nvcf-observability-stack-inventory.json` |

Verify the selected binary with `"$NVCF_CLI" version` before maintenance.
Record the selected CLI separately from the immutable stack inventory.
The target registration flow refreshes the existing cluster metadata while
preserving its identity.

The task commands in the baseline and validation steps require `base_nvct_url`
and `nvct_host` in the CLI configuration file. CLI configurations created for
0.6.x often lack them. See the [CLI](/nvcf/1.1.0/overview/cli) page for the values. The
`function invoke` and `task` commands also need API keys. The administrator
token from `nvcf-cli init` has no invocation scope and no NVCT key, so
`function invoke` fails with `403 missing requested authorities` and `task get`
fails with `401`. Generate the keys once per CLI binary:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" init
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" api-key generate --expires-in 48h
```

The keys are stored in the CLI state, so repeat the command after switching
to a different `nvcf-cli` binary.

## Prepare the target configuration

Start from the target bundle templates. Reconcile the site-specific settings
from 0.6.2 into these target files:

```text
$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
$TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml
```

Do not copy the 0.6.2 files wholesale. Preserve the existing registry,
storage, endpoint, credential, ingress, certificate, and scheduling values,
then add settings required by the target templates.

If the deployment uses a mirror, point `api.accountBootstrap.image` and
`nats.reloader.image` at that mirror. The commented examples in
`environments/base.yaml` show the keys.

Observability moves to the separate `nvcf-observability-stack` in this
upgrade. The 0.6.2 environment file carries the `global.observability`
tracing and metrics settings and the optional State Metrics service, but it
has no `observability.profile`, and the target defaults to `control`. With that
profile, the full sync installs the bundled metrics pipeline: Prometheus
Operator CRDs, the OpenTelemetry Operator and collector, VictoriaMetrics, the
default control-plane monitors, and the Function Autoscaler. Carry the
existing `global.observability` values forward, then choose a layout and set
its keys in `$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml` before rendering:

| Layout | Keys |
| --- | --- |
| Bundled backend with public chart and image sources | `victoriaMetrics.server.persistentVolume.storageClass`, unless the cluster has a default storage class. The volume is 16Gi by default. |
| Bundled backend from a mirror | The `chartRepository` blocks under `prometheusOperatorCrds`, `opentelemetryOperator`, and `victoriaMetrics`, plus `opentelemetryOperator.manager.collectorImage` and `collector.image`. The defaults use `prometheus-community.github.io`, `open-telemetry.github.io`, `victoriametrics.github.io`, and `ghcr.io`. |
| Existing Prometheus Operator CRDs, OpenTelemetry Operator, or metrics backend | `observability.components.prometheusOperatorCrds.mode`, `observability.components.otelOperator.mode`, or `metricsBackend.mode` set to `existing`, with the backend endpoints for `metricsBackend`. |

The defaults and remaining keys are in the observability bundle's
`environments/base.yaml`. See [Observability](/nvcf/1.1.0/observability/observability)
for the profiles, component modes, and layouts.

In `$TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml`, replace the template
placeholders. The 1.0.0 template ships `nvcr.io` and `YOUR_ORG/YOUR_TEAM`:

| Key | Value |
| --- | --- |
| `global.helm.sources.registry`, `global.helm.sources.repository` | The chart registry used by the existing compute plane. |
| `global.image.registry`, `global.image.repository` | The image registry used by the existing compute plane. |
| `global.imagePullSecrets` | The existing pull secrets, or `[]` if the registry needs none. |
| `global.nvcaOperator.selfManaged.featureGateValues` | Feature gates enabled on the existing cluster, such as `KAIScheduler` or `DynamoOperatorSupport`. |

Registration writes the ICMS, reval, and NATS endpoints and host overrides.
Non-empty `icmsServiceURL`, `icmsServiceHostHeaderOverride`,
`revalServiceURL`, `revalServiceHostHeaderOverride`, `natsURL`, and
`natsHostOverride` values in the environment file take precedence.

Express Dynamo support with the `DynamoOperatorSupport` feature gate. The
compute plane also adds these feature gates by default: `DynamicGPUDiscovery`,
`InfraResourceOverhead`, `EnforceHelmFunctionResourceLimits`,
`EnforceContainerFunctionResourceLimits`, `EnforceHelmTaskResourceLimits`, and
`EnforceContainerTaskResourceLimits`. To turn off a default gate, add
`-<gate>` to `featureGateValues`.

The KAI Scheduler, Grove, and Dynamo operators are compute-plane add-ons in
1.0.0, under `addons.kaiScheduler`, `addons.groveOperator`, and
`addons.dynamoOperator`. They are disabled by default. A disabled add-on
leaves an existing operator installation unchanged. Enabling one makes the
stack manage that release at the version pinned in the compute-plane bundle.

The Cassandra service-role password in the target secrets file must match the
password already stored for the service-specific Cassandra roles. The same
value must be supplied to the OpenBao migration:

```yaml
cassandra:
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>

openbao:
  migrations:
    env:
      - name: DEFAULT_CASSANDRA_PASSWORD
        value: *cassandra_service_role_password
      - name: NVCF_API_SIDECARS_IMAGE_PULL_SECRET
        value: <base64-docker-credential>
      - name: ADMIN_CLIENT_ID
        value: ncp
```

To confirm the value, log in as one of the existing service roles, for example
`nvcf_api_app_v0`:

```bash
read -rs -p "Cassandra service-role password: " SERVICE_ROLE_PASSWORD
echo
kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
  cqlsh -u nvcf_api_app_v0 -p "$SERVICE_ROLE_PASSWORD" -e 'SELECT now() FROM system.local;'
unset SERVICE_ROLE_PASSWORD
```

A wrong value fails with `AuthenticationFailed`.

If the Cassandra administrator password was changed from the chart default,
also carry the existing `cassandra.dbUser` values into the target secrets file.
Do not rotate Cassandra credentials as part of this upgrade.

Keep the live Cassandra storage class, size, and replica count unchanged.
Record them before rendering the target release:

```bash
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}'
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='replicas: {.spec.replicas}{"\n"}'
```

Render the target configuration before the maintenance window:

```bash
make -C "$TARGET_CONTROL" template \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
```

Review the rendered storage settings, images, registries, domains, and
namespaces. Do not continue if the target render changes a Cassandra volume
claim template.

The rendered manifests are written to `$TARGET_CONTROL/out`. To review the
Cassandra volume claim template, run:

```bash
grep -n -A12 'volumeClaimTemplates' \
  "$TARGET_CONTROL"/out/*-cassandra/*/templates/statefulset.yaml
```

Also compare the `data` volume mount, which is part of the pod template and
not of the volume claim template. Installations that adopted Bitnami volumes
during the 0.6.0 to 0.6.1 upgrade mount the volume with `subPath: data`:

```bash
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="cassandra")].volumeMounts[?(@.name=="data")]}{.mountPath}{" subPath="}{.subPath}{"\n"}{end}'
grep -n -A2 '^            - name: data$' \
  "$TARGET_CONTROL"/out/*-cassandra/*/templates/statefulset.yaml
```

The second command prints the rendered `data` mount and the two lines after
it. If the live StatefulSet shows `subPath=data` and the rendered mount has
no `subPath`, stop. The 1.0.1 stack does not forward
`cassandra.persistence.subPath` from the environment file to the Cassandra
chart, although the chart still supports it. A member rolled without the
`subPath` mounts the volume root, does not find its data, and starts empty.

To carry the setting through every Cassandra sync in this guide, including
the full `make install`, add it to the target secrets file, which the stack
passes to the Cassandra release unchanged:

```yaml
cassandra:
  persistence:
    subPath: "data"
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>
```

Render again and confirm that the `data` mount shows `subPath: data` before
continuing. Keep the setting for as long as the volumes retain the Bitnami
layout.

To remove the dependency on `subPath` permanently, the
[relocation script](https://github.com/NVIDIA/nvcf/blob/deploy/helm/cassandra/v0.21.3/deploy/helm/cassandra/upgrade/relocate-bitnami-layout.sh)
in the Cassandra chart source moves the contents of the `data/` directory to
the volume root, which is the layout a fresh installation uses. The script
runs with Cassandra stopped and each volume mounted in a helper pod, so it
has its own downtime. Run it in a separate maintenance window, either before
this upgrade or after it, and keep the `subPath` setting until the volumes
have been relocated.

<a id="verify-the-061-starting-state"></a>

## Verify the 0.6.2 starting state

Confirm that every source Helm release is deployed and every active controller
is ready:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A
```

Confirm that Cassandra has no pending repair or schema disagreement and that
all OpenBao members are initialized, unsealed, and ready. Identify the active
OpenBao member. Standby members report `HA Mode standby`; one member reports
`HA Mode active`.

```bash
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status

for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

Record the identity and version of every registered compute cluster. Run these
commands with that compute cluster's kubeconfig and context:

```bash
kubectl --kubeconfig <compute-kubeconfig> --context <compute-context> \
  get nvcfbackend <cluster-name> -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'
```

The source NVCA version must be 3.0.3 and `agent` must be `healthy`.

### Verify the GPU inventory precondition

Resolve the Cassandra administrator password from the original installation
configuration. The 0.6.2 chart does not always create a
Kubernetes Secret for this password, so do not assume a Secret name. Do not
enable shell tracing while the password variable is set.

```bash
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
echo
export CASSANDRA_PASSWORD

kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
  cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
  -e 'SELECT cluster_id, cluster_status, gpus_v4, gpus_v5 FROM sis_api.cluster_by_cluster_id;'
```

Every active cluster row must have a nonempty `gpus_v5` value. Stop if this
check fails.

### Record application baselines

Record at least one active function, function version, deployment, completed
task, cluster ID, and cluster-group ID. Invoke the function and read the task
before the upgrade. Use the same records for post-upgrade validation. Run
`nvcf-cli api-key generate` first, as described in the deployment host
section, or the invoke and task commands fail with 403 and 401.

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function get \
  --function-id <function-id> --version-id <version-id> --json

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function deploy get \
  --function-id <function-id> --version-id <version-id> --json

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id <function-id> --version-id <version-id> \
  --request-body '<sample-request>' --timeout 120 --json

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task get <task-id> --json
```

## Back up persistent state

Create restorable backups of Cassandra, OpenBao Raft storage, NATS JetStream,
the Helm values, and all target configuration files. Follow the storage
provider's backup procedure and verify the restore path before continuing.

Create a backup directory readable only by the operator, record the Cassandra
replica count, then flush Cassandra and create a named snapshot on every
Cassandra member before the storage-level backup:

```bash
BACKUP_DIR=<backup-directory>
mkdir -p "$BACKUP_DIR" && chmod 700 "$BACKUP_DIR"

CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"

for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kcp -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
  kcp -n cassandra-system exec "$pod" -c cassandra -- \
    nodetool snapshot -t pre-1-0-1
done
```

The Cassandra snapshot remains on each persistent volume. It is not an
off-cluster backup by itself. Copy or snapshot the volumes with your storage
provider.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from a
standby member fails with `incomplete snapshot`:

```bash
OPENBAO_ACTIVE=""
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  if kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
    OPENBAO_ACTIVE="$pod"
  fi
done

kcp -n vault-system get secret openbao-server-root-token \
  -o jsonpath='{.data.root_token}' | base64 -d | \
  kcp -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
  'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-0-1.snap'
kcp -n vault-system cp -c openbao \
  "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-0-1.snap" "$BACKUP_DIR/openbao-pre-1-0-1.snap"
kcp -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-0-1.snap
```

The Raft snapshot can only be restored with the unseal key. Back up the
`openbao-server-unseal` and `openbao-server-root-token` Secrets in
`vault-system` with the same protection as the snapshot.

The NVCF NATS streams use memory storage. A JetStream backup contains stream
metadata only, not in-flight messages.

Export the values of every Helm release on both clusters:

```bash
mkdir -p "$BACKUP_DIR/helm-values"
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json | \
  jq -r '.[] | "\(.namespace) \(.name)"' | \
  while read -r namespace release; do
    helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" \
      -n "$namespace" -o yaml > "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
  done
```

Repeat the export with each compute cluster's kubeconfig. The exported values
contain credentials. Protect them like the other backups.

Do not continue until the Cassandra and OpenBao backups are complete.

## Prepare the staged rollout

Do not scale the application Deployments to zero. Block configuration changes
that can invalidate the upgrade baseline, but keep normal read and task traffic
running for the availability test. Record the current images, replicas, and
rolling strategies:

```bash
kcp -n nvcf get deployment nvcf-api nvct-api -o wide
kcp -n sis get deployment spot-instance-service -o wide
kcp -n ess get deployment -o wide
kcp -n cassandra-system get statefulset cassandra -o wide
```

Use at least two application replicas to limit rollout interruptions. The rollout
must have an effective `maxUnavailable` of 0 and `maxSurge` of at least one pod.
Kubernetes calculates those values as 0 and 1 from the default 25 percent
settings when a Deployment has two replicas. A zero-downtime Cassandra server
upgrade requires at least three healthy members.

Start continuous external probes before the first rollout. Exercise the same
API and task reads and HTTP/gRPC invocations used for the retained baseline.
Keep timestamped successes, failures, and step start/end times through final
validation.

Agree on a maintenance cutoff and a clean probe observation interval during
the topology rehearsal. This procedure has no guaranteed interruption duration
and does not qualify as a zero-downtime upgrade.

| Transition | Possible interruption | Gate before the next stop |
| --- | --- | --- |
| Stop 1, only for a one-member development Cassandra deployment | Database and dependent API downtime during the restart. | The member rejoins with its original host ID; ledgers are clean; retained reads and invocations recover. |
| Stop 2, from the first OpenBao rotation through migration completion | HTTP 401 or gRPC `Unauthenticated`, including fresh administrator login failures. | Both rotations and the migration complete; every member is healthy; fresh administrator login and the original invocation credentials work. |
| Full target control-plane sync, while NATS and services roll | Invocation timeouts or HTTP 504. | The sync succeeds; all controllers are ready; required NATS streams exist; retained reads and invocations recover. |

During these transitions, a failed probe pauses advancement to the next stop.
Let the running sync and required rotations finish while their health checks
pass. Do not interrupt a sync or roll back solely for a probe failure.
Resume only after the gate passes and every probe succeeds throughout the
agreed observation interval. Keep the interruption in the availability report.

Pause further mutations and investigate if a probe fails outside these
transitions, a required completion check fails, or recovery exceeds the agreed
maintenance cutoff. Follow [Stop and recovery criteria](#stop-and-recovery-criteria).

## Stop 1: roll Cassandra and apply expand migrations

Upgrade only the Cassandra release with Helm hooks disabled. Preserve the
existing replica count and storage settings. If the live StatefulSet mounts
the data volume with `subPath: data`, confirm that the target secrets file
carries `cassandra.persistence.subPath` as described in the target
configuration section before running this sync. With three members, rotate one
member at a time and require it to rejoin before continuing. A one-member
development deployment has unavoidable database downtime during this step.

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=cassandra \
  sync --args "--no-hooks"

kcp -n cassandra-system rollout status statefulset/cassandra --timeout=30m
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
```

The Cassandra StatefulSet uses the `RollingUpdate` strategy. Kubernetes
replaces the members one at a time, from the highest ordinal to the lowest,
and waits for each member to become ready before it replaces the next one.
Every member must report `UN` in `nodetool status`, with the same host IDs as
before the roll. The `rollout status` command is the gate for this step. Helm
waits at most 15 minutes by default. If it reports a timeout on a large
database, the StatefulSet keeps rolling: wait for `rollout status` to finish,
then run the same sync again so Helm records the release as deployed.

Check the source ledgers before the first migration Job. Use the Cassandra
administrator password from the GPU inventory check:

```bash
for table in api_keys_api ess_api event_ledger nvcf_api nvcf_autoscaler nvct_api sis_api; do
  echo "== $table"
  kcp -n cassandra-system exec cassandra-0 -c cassandra -- \
    cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
    -e "SELECT version, dirty FROM schema_migrations.$table;"
done
```

Replace `cassandra` with the administrator user if it was changed. The
`event_ledger` table does not exist before Stop 1, so its query fails with
`table event_ledger does not exist`.

Run migration image 0.17.6 as bounded Jobs. Require the expected clean source
version before each `goto` call and require the exact clean target afterward.
Stop at these versions:

| Migration table | Source | Expand target |
| --- | ---: | ---: |
| `api_keys_api` | 3 | 3 |
| `ess_api` | 8 | 9 |
| `event_ledger` | absent | 3 |
| `nvcf_api` | 7 | 8 |
| `nvcf_autoscaler` | 3 | 4 |
| `nvct_api` | 3 | 4 |
| `sis_api` | 6 | 10 |

Do not use the image's default `/app/execute_sqls.sh`. It runs every available
migration, including the destructive NVCT contract migrations. Each bounded
Job must select one preprocessed keyspace path and its matching migration
table, fail on an unexpected or dirty starting ledger, and stop immediately
on the first error.

### Run the bounded migration Jobs

The reference Job below uses the migration image without its default command.
It reproduces the preprocessing and connection settings of
`/app/execute_sqls.sh` for one keyspace. See the
[Cassandra migrations README](https://github.com/NVIDIA/nvcf/blob/migrations/cassandra/v0.17.6/migrations/cassandra/README.md) for the
image layout, environment variables, and connection parameters. In the
image, the SQL files for each keyspace are in `/app/keyspaces/<keyspace>/`,
and each keyspace records its ledger in the `schema_migrations.<keyspace>`
table. The `KS` value passed to the Job selects both. The script:

1. Substitutes `SERVICE_ROLE_PASSWORD` and `REPLICA_COUNT` into the SQL files
   of one keyspace.
2. Reads the current ledger version and exits if it differs from the expected
   source. A dirty ledger reports `<version> (dirty)` and fails this check.
3. Runs `migrate goto` to the target version.
4. Exits with an error unless the ledger reports exactly the target version.

Store the Cassandra credentials in a temporary Secret. The values are read
from the terminal and are not written to disk:

```bash
read -r -p "Cassandra administrator user: " CASSANDRA_USER
read -rs -p "Cassandra service-role password: " SERVICE_ROLE_PASSWORD
echo

kcp -n cassandra-system create secret generic upgrade-migration-credentials \
  --from-env-file=<(printf 'CASSANDRA_USER=%s\nCASSANDRA_PASSWORD=%s\nSERVICE_ROLE_PASSWORD=%s\n' \
    "$CASSANDRA_USER" "$CASSANDRA_PASSWORD" "$SERVICE_ROLE_PASSWORD")
unset SERVICE_ROLE_PASSWORD
```

Create the script ConfigMap:

```bash
kcp -n cassandra-system create configmap upgrade-bounded-migration \
  --from-file=bounded-migration.sh=/dev/stdin <<'EOF'
set -eu
mkdir -p "/tmp/keyspaces/$KS"
for f in /app/keyspaces/"$KS"/*.sql; do
  envsubst '$SERVICE_ROLE_PASSWORD $REPLICA_COUNT' < "$f" \
    > "/tmp/keyspaces/$KS/$(basename "$f")"
done
enc() {
  printf '%s' "$1" | \
    python3 -c 'import sys; from urllib.parse import quote; print(quote(sys.stdin.read(), safe=""))'
}
DB="cassandra://${CASSANDRA_HOSTS}:9042/schema_migrations?x-multi-statement=true&x-migrations-table=${KS}&username=$(enc "$CASSANDRA_USER")&password=$(enc "$CASSANDRA_PASSWORD")&consistency=LOCAL_QUORUM&protocol=4&timeout=2m&connect-timeout=30s&disable-host-lookup=true"

cur=$(migrate -path "/tmp/keyspaces/$KS" -database "$DB" version 2>&1 || true)
echo "[$KS] start version: $cur (expected $SRC)"
if [ "$SRC" = "absent" ]; then
  case "$cur" in *"no migration"*) ;; *) echo "[$KS] unexpected start version"; exit 1 ;; esac
else
  [ "$cur" = "$SRC" ] || { echo "[$KS] unexpected start version"; exit 1; }
fi
if [ "$SRC" != "$TGT" ]; then
  migrate -path "/tmp/keyspaces/$KS" -database "$DB" goto "$TGT"
fi
after=$(migrate -path "/tmp/keyspaces/$KS" -database "$DB" version 2>&1)
echo "[$KS] end version: $after (expected $TGT)"
[ "$after" = "$TGT" ] || { echo "[$KS] unexpected end version"; exit 1; }
echo "[$KS] OK"
EOF
```

Define a function that runs one bounded Job and waits for it. Set
`MIGRATIONS_IMAGE` to the mirrored migration image and `MIGRATIONS_PULL_SECRET`
to the pull secret that the Cassandra chart uses in `cassandra-system`. If the
registry needs no authentication, remove the `imagePullSecrets` block from the
Job.

```bash
export MIGRATIONS_IMAGE=<registry>/<repository>/nvcf-cassandra-migrations:0.17.6
export MIGRATIONS_PULL_SECRET=<pull-secret-name>

run_bounded_migration() {
  ks="$1"; src="$2"; tgt="$3"
  job="upgrade-migrate-$(printf '%s' "$ks" | tr '_' '-')-$src-$tgt"
  kcp -n cassandra-system create -f - <<EOF || return 1
apiVersion: batch/v1
kind: Job
metadata:
  name: $job
  namespace: cassandra-system
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      imagePullSecrets:
        - name: $MIGRATIONS_PULL_SECRET
      securityContext:
        fsGroup: 999
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: migrate
          image: $MIGRATIONS_IMAGE
          command: ["/bin/sh", "/scripts/bounded-migration.sh"]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            runAsUser: 999
            runAsGroup: 999
          envFrom:
            - secretRef:
                name: upgrade-migration-credentials
          env:
            - name: KS
              value: "$ks"
            - name: SRC
              value: "$src"
            - name: TGT
              value: "$tgt"
            - name: REPLICA_COUNT
              value: "$CASSANDRA_REPLICAS"
            - name: CASSANDRA_HOSTS
              value: cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local
          volumeMounts:
            - name: script
              mountPath: /scripts
              readOnly: true
      volumes:
        - name: script
          configMap:
            name: upgrade-bounded-migration
EOF
  while true; do
    result=$(kcp -n cassandra-system get "job/$job" \
      -o jsonpath='{.status.succeeded}/{.status.failed}')
    case "$result" in
      1/*) kcp -n cassandra-system logs "job/$job"; return 0 ;;
      */1) kcp -n cassandra-system logs "job/$job"; return 1 ;;
    esac
    sleep 5
  done
}
```

`REPLICA_COUNT` must equal the Cassandra replica count recorded in the backup
step. The `event_ledger` migration creates a keyspace with this replication
factor; an empty value fails the Job with `ConfigurationException` and leaves
the `event_ledger` ledger dirty. The block below runs the expand Jobs in this
order only when the live replica count matches the recorded value, and the
`&&` chain stops at the first failed Job:

```bash
CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')
if [ -z "$CASSANDRA_REPLICAS" ] ||
   [ "$CASSANDRA_REPLICAS" != "$(cat "$BACKUP_DIR/cassandra-replicas")" ]; then
  echo "Stop: replica count '$CASSANDRA_REPLICAS' differs from the recorded value" >&2
else
  run_bounded_migration api_keys_api 3 3 &&
  run_bounded_migration ess_api 8 9 &&
  run_bounded_migration event_ledger absent 3 &&
  run_bounded_migration nvcf_api 7 8 &&
  run_bounded_migration nvcf_autoscaler 3 4 &&
  run_bounded_migration nvct_api 3 4 &&
  run_bounded_migration sis_api 6 10
fi
```

Each Job log ends with `[<keyspace>] OK`. If a Job fails, read its log and do
not continue. Do not delete or edit a dirty ledger row.

Confirm Cassandra is healthy after the server roll and expand Jobs:

```bash
kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
```

Do not roll either bridge until every expanded ledger has the exact version in
the table and `dirty = False`. Rerun the source ledger query loop to check.

## Stop 2: migrate OpenBao

The OpenBao StatefulSet uses `OnDelete`. The target release also changes the
JWT plugin binary and catalog checksum. The safe sequence requires two manual
pod rotations:

1. Update the StatefulSet without hooks and rotate onto the target image.
2. Start the normal Helm sync so the catalog-refresh hook runs.
3. When the target migration Job appears, rotate the pods again so each server
   reloads the refreshed plugin catalog.
4. Let the migration Job and Helm sync finish.

Never delete the StatefulSet, its persistent volume claims, or more than one
OpenBao server pod at a time.

### Update the StatefulSet without hooks

Run this command from the target control-plane bundle:

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=openbao-server \
  sync --args "--no-hooks"
```

Confirm the StatefulSet template uses `OnDelete` and the target image. Existing
pods can still show the source image.

```bash
kcp -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
```

### Perform the first OpenBao rotation

Identify the active and standby members with `bao status`. Rotate every standby
member first. For each standby pod, run:

```bash
wait_openbao_unsealed() {
  pod="$1"
  attempt=0
  while true; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status' 2>/dev/null || true)
    if printf '%s\n' "$status" | grep -Eq '^Initialized[[:space:]]+true$' && \
       printf '%s\n' "$status" | grep -Eq '^Sealed[[:space:]]+false$'; then
      printf '%s\n' "$status"
      return 0
    fi
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 120 ]; then
      echo "OpenBao member did not become initialized and unsealed: $pod" >&2
      return 1
    fi
    sleep 2
  done
}

export POD=<standby-pod-name>
kcp -n vault-system delete pod "$POD" --wait=true &&
  kcp -n vault-system wait "pod/$POD" --for=create --timeout=5m &&
  kcp -n vault-system wait "pod/$POD" \
    --for=condition=Ready --timeout=10m &&
  wait_openbao_unsealed "$POD"
```

Kubernetes readiness can become true before auto-unseal completes. The polling
check must report `Initialized true`, `Sealed false`, and a healthy standby HA
mode. Recheck every member, identify the current active member, and rotate it
last with the same commands.

Confirm all members are ready and use the target image before continuing:

```bash
kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.containerStatuses[*].ready}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'
```

### Refresh the catalog and run migrations

Use two terminals. Start the normal target OpenBao sync in terminal 1:

```bash
make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server
```

The command waits while the target catalog-refresh and migration hooks run.

In terminal 2, use a 15-minute polling deadline for the migration Job to use
OpenBao migrations 0.19.5. It appears after the catalog-refresh hook succeeds.
If this wait fails, inspect terminal 1 and the Job logs before continuing:

```bash
wait_openbao_migration() {
  local deadline=$((SECONDS + 900)) job image failed
  while [ "$SECONDS" -lt "$deadline" ]; do
    job=$(kcp -n vault-system get job openbao-server-migrations \
      --ignore-not-found --request-timeout=10s \
      -o jsonpath='{.spec.template.spec.containers[0].image}{"|"}{.status.conditions[?(@.type=="Failed")].status}') || return 1
    IFS='|' read -r image failed <<< "$job"
    if printf '%s\n' "$image" | grep -Eq ':0[.]19[.]5$'; then
      if [ "$failed" = True ]; then
        echo "OpenBao migration Job failed; inspect its logs before continuing" >&2
        return 1
      fi
      return 0
    fi
    sleep 2
  done
  echo "Timed out waiting for the target OpenBao migration Job" >&2
  return 1
}
wait_openbao_migration
```

As soon as the target migration Job appears, repeat the standby-first,
active-last rotation from the preceding section. This second rotation reloads
the refreshed plugin catalog. The migration Job uses `OnFailure` retries while
the rotation is in progress.

Wait for the sync in terminal 1 to finish. Then verify the migration Job and
all OpenBao members:

```bash
kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m

for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  kcp -n vault-system wait "pod/$pod" \
    --for=condition=Ready --timeout=10m
  wait_openbao_unsealed "$pod"
done
```

All members must be initialized, unsealed, ready, and running the same target
image. Do not continue if the migration Job failed or any member is unhealthy.
Verify fresh administrator login while preserving the probe API keys:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" refresh >/dev/null
```

Before Stop 3, require the original invocation credentials and all retained
probes to pass for the agreed observation interval. Continued authentication
failures after both rotations are a recovery condition.

## Stop 3: run the service backfills and contract migrations

The target service charts require the OpenBao roles and policies created at
Stop 2. Use the target `sis` and `nvct-api` charts for both bridge rollouts.
Set at least two replicas and keep the external availability probe running.

Set the published bridge image locations:

```bash
export BRIDGE_REGISTRY=nvcr.io
export BRIDGE_REPOSITORY=nvidia/nvcf
export ICMS_BRIDGE_TAG=0.7.5
export NVCT_BRIDGE_TAG=1.64.3
export BRIDGE_PULL_POLICY=IfNotPresent
```

If the deployment pulls from a mirror, set `BRIDGE_REGISTRY` and
`BRIDGE_REPOSITORY` to the mirror location of the bridge images.

The backfill records are log lines written by the bridge pods. Define a helper
that waits for a record and prints it:

```bash
wait_backfill_record() {
  namespace="$1"; selector="$2"; record="$3"
  attempt=0
  until kcp -n "$namespace" logs -l "$selector" --tail=-1 --prefix | \
      grep -F "$record"; do
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 60 ]; then
      echo "Backfill record not found: $record" >&2
      return 1
    fi
    sleep 10
  done
}
```

### Backfill ICMS instance lookup columns

Define a helper that keeps every bridge override in each Helm invocation:

```bash
sync_icms_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=sis sync \
    --set sis.replicaCount=2 \
    --set sis.image.registry="$BRIDGE_REGISTRY" \
    --set sis.image.repository="$BRIDGE_REPOSITORY/icms-service-oss" \
    --set sis.image.tag="$ICMS_BRIDGE_TAG" \
    --set sis.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set sis.env.ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE="$mode"
}
```

Roll the bridge with the task disabled, then run the migration and validation
as separate rollouts:

```bash
cd "$TARGET_CONTROL"
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

ICMS_SELECTOR=app.kubernetes.io/instance=sis,app.kubernetes.io/name=helm-nvcf-sis

sync_icms_bridge MIGRATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
wait_backfill_record sis "$ICMS_SELECTOR" \
  'Completed ICMS instance lookup backfill: mode=MIGRATE'

sync_icms_bridge VALIDATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
wait_backfill_record sis "$ICMS_SELECTOR" \
  'Completed ICMS instance lookup backfill: mode=VALIDATE'

sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
```

Require a completed `MIGRATE` record with `missingRequest=0`, `mismatched=0`,
and `failed=0`. Then require a completed `VALIDATE` record with `migrated=0`,
`missingRequest=0`, `mismatched=0`, and `failed=0`. The validation record must
report the migrated active rows as current. Keep the task `DISABLED` after the
validation pass.

A completed record has this form. The record appears on only one ICMS
replica:

```text
Completed ICMS instance lookup backfill: mode=MIGRATE, scanned=<n>, migrated=<n>, current=<n>, skippedTerminated=<n>, missingRequest=0, mismatched=0, failed=0
```

Roll the stack-pinned ICMS image by removing the bridge image overrides:

```bash
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=sis sync \
  --set sis.replicaCount=2

kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
```

Confirm that the final Deployment uses ICMS 0.7.2 and that no bridge pod
remains.

### Backfill NVCT task health

Define the corresponding NVCT helper:

```bash
sync_nvct_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=nvct-api sync \
    --set nvctApi.replicaCount=2 \
    --set nvctApi.image.registry="$BRIDGE_REGISTRY" \
    --set nvctApi.image.repository="$BRIDGE_REPOSITORY/nvct-service-oss" \
    --set nvctApi.image.tag="$NVCT_BRIDGE_TAG" \
    --set nvctApi.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set nvctApi.env.NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE="$mode"
}
```

Roll the bridge and move through the same explicit task modes:

```bash
sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

NVCT_SELECTOR=app.kubernetes.io/instance=nvct-api,app.kubernetes.io/name=helm-nvcf-nvct-api

sync_nvct_bridge MIGRATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
wait_backfill_record nvcf "$NVCT_SELECTOR" \
  'Completed NVCT health backfill: mode=MIGRATE'

sync_nvct_bridge VALIDATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
wait_backfill_record nvcf "$NVCT_SELECTOR" \
  'Completed NVCT health backfill: mode=VALIDATE'

sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
```

Require a completed `MIGRATE` record with `missing=0`, `mismatched=0`, and
`failed=0`. Then require a completed `VALIDATE` record with `migrated=0`,
`missing=0`, `mismatched=0`, `failed=0`, and
`current + withoutLegacyHealth = scanned`. Re-read the retained task and
record its new `health` value before continuing.

`withoutLegacyHealth` counts tasks that have no legacy `health_info` value,
such as completed or canceled tasks, so it is usually nonzero. It is not a
failure.

The routine runs on a schedule, about once a minute by default, so each mode
writes more than one record and more than one NVCT replica can write one.
Only the first `MIGRATE` record carries a nonzero `migrated` count; later
`MIGRATE` records report the same rows as `current`. Every record must meet
these conditions:

```text
Completed NVCT health backfill: mode=MIGRATE, scanned=<n>, migrated=<n>, current=<n>, withoutLegacyHealth=<n>, missing=0, mismatched=0, failed=0
```

Roll the stack-pinned NVCT 1.66.0 image by removing the bridge overrides:

```bash
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=nvct-api sync \
  --set nvctApi.replicaCount=2

kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
```

After every bridge pod is gone, run the bounded NVCT migration Job from clean
version 4 to version 6. This drops `health_info` and `health_udt`. NVCT 1.66.0
remains online because it uses only the replacement `health` column.

Confirm that only NVCT 1.66.0 pods remain, then run the contract Job with the
Stop 1 Secret, ConfigMap, and `run_bounded_migration` function:

```bash
kcp -n nvcf get pods -l "$NVCT_SELECTOR" \
  -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.containers[*].image}{"\n"}{end}'

run_bounded_migration nvct_api 4 6
```

Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal target Cassandra
release. Its migration hook must be a no-op because all ledgers are already at
their target versions.

```bash
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=api sync
kcp -n nvcf rollout status deployment/nvcf-api --timeout=10m

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=ess-api sync
kcp -n ess rollout status deployment/ess-api-deployment --timeout=10m

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=cassandra sync
```

The ESS chart renames the Deployment from
`ess-api-helm-nvcf-ess-api-deployment` to `ess-api-deployment`. Helm deletes
the old Deployment and creates the new one instead of rolling it.

The Cassandra migration Job log must report `no change` once for each of the
seven keyspaces. The block below deletes the temporary migration resources
only when the count is seven:

```bash
no_change=$(kcp -n cassandra-system logs job/cassandra-migrations | grep -ci 'no change')
if [ "$no_change" != 7 ]; then
  echo "Stop: expected 7 keyspaces with no change, found $no_change" >&2
else
  for job in $(kcp -n cassandra-system get jobs -o name | grep '/upgrade-migrate-'); do
    kcp -n cassandra-system delete "$job"
  done
  kcp -n cassandra-system delete secret upgrade-migration-credentials
  kcp -n cassandra-system delete configmap upgrade-bounded-migration
fi
```

After all steps succeed, confirm that the Cassandra migration Jobs completed
and that `nodetool status` reports a healthy topology:

```bash
kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
```

Use the Cassandra administrator credentials from your deployment to query the
migration ledgers. Each row must have `dirty = False` and the following
version:

| Migration table | Version |
| --- | ---: |
| `api_keys_api` | 3 |
| `ess_api` | 9 |
| `event_ledger` | 3 |
| `nvcf_api` | 8 |
| `nvcf_autoscaler` | 4 |
| `nvct_api` | 6 |
| `sis_api` | 10 |

Run these CQL statements through the site's approved Cassandra administration
path:

```sql
SELECT version, dirty FROM schema_migrations.api_keys_api;
SELECT version, dirty FROM schema_migrations.ess_api;
SELECT version, dirty FROM schema_migrations.event_ledger;
SELECT version, dirty FROM schema_migrations.nvcf_api;
SELECT version, dirty FROM schema_migrations.nvcf_autoscaler;
SELECT version, dirty FROM schema_migrations.nvct_api;
SELECT version, dirty FROM schema_migrations.sis_api;
```

Confirm that the NVCT task schema contains only the replacement health column:

```sql
SELECT column_name, type
FROM system_schema.columns
WHERE keyspace_name = 'nvct_api'
  AND table_name = 'tasks_v2'
  AND column_name IN ('health', 'health_info');
```

The result must contain only `health`. Re-read the retained task and require
the same serialized health value recorded before the contract migration. Do
not continue if a ledger is dirty, a version differs, or the schema or retained
data check fails.

## Sync the target control plane and observability stack

The selector of the `reval` Deployment changed after 0.6.2. Kubernetes does
not permit that field to change in place. Delete only the Deployment before
the full sync. Do not uninstall the Helm release or delete its namespace.

```bash
kcp -n nvcf delete deployment reval
```

Sync the full target control-plane bundle. Its default control profile also
installs the target observability components and Function Autoscaler.
Invocation timeouts can occur while NATS and services roll. Keep probes running
and let the sync finish; each Helm release has a 15-minute timeout by default.

```bash
make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
```

This sync reruns the OpenBao hooks. They must finish successfully without
another pod rotation. Confirm that every Helm release is deployed and every
active Deployment or StatefulSet is ready:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A
```

The environment file has no replica setting for `sis` or `nvct-api`. The full
sync returns `spot-instance-service` and `nvct-api` to the chart default of one
replica, replacing the two replicas set during Stop 3.

Confirm the required global NATS streams still exist:

```bash
NATS_BOX_POD=$(kcp -n nats-system get pod \
  -l app.kubernetes.io/component=nats-box \
  -o jsonpath='{.items[0].metadata.name}')

for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do
  kcp -n nats-system exec "$NATS_BOX_POD" -- \
    nats --context default stream info "$stream"
done
```

If observability is managed by a separate stack instance, keep the
self-managed ownership settings consistent with the existing deployment and
sync `nvcf-observability-stack` 1.0.0 as the only owner. Do not install two
owners for the same observability resources.

Before upgrading compute clusters, require all retained reads and invocations
to pass for the agreed observation interval. Continued failures after the sync
and readiness checks pass require investigation before further mutations.

## Upgrade each compute cluster

Refresh authentication without clearing saved API keys, then export a fresh
control-plane profile from the target bundle:

```bash
set -euo pipefail
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" config current-context)" = "$CONTROL_CONTEXT"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" refresh >/dev/null

KUBECONFIG="$CONTROL_KUBECONFIG" \
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" self-hosted \
  --control-plane-stack "$TARGET_CONTROL" \
  --env "$HELMFILE_ENV" \
  control-plane profile export

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" self-hosted \
  control-plane profile validate \
  --file "$TARGET_CONTROL/out/control-plane-profile.yaml" \
  --require compute-reachable
```

Stop if export or validation fails. Inspect the profile's endpoints before
registration; they must identify this installation and be reachable from the
compute cluster. Validation checks the file, not network connectivity.

For each compute cluster, set its values and verify the selected context:

```bash
export CLUSTER_NAME=<existing-cluster-name>
export CLUSTER_REGION=<cluster-region>
export NCA_ID=<existing-nca-id>
export COMPUTE_KUBECONFIG=<absolute-path-to-compute-kubeconfig>
export COMPUTE_CONTEXT=<compute-context>

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" cluster-info
```

Set `CLUSTER_REGION` to the region of the existing registration, not a new
value:

```bash
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='{.status.clusterConfig.region}{"\n"}'
```

Confirm that `nvcr-pull-secret`, or the pull secret named in the compute-plane
environment, exists in `nvca-operator`. The operator mirrors it to
`nvca-system` for NVCA sidecars and managed workloads. If the compute-plane
registry does not require authentication and `global.imagePullSecrets` is
`[]`, no pull secret is needed.

Re-register the existing cluster with the target control-plane profile:

```bash
make -C "$TARGET_COMPUTE" register-cluster \
  CLUSTER_NAME="$CLUSTER_NAME" \
  CLUSTER_REGION="$CLUSTER_REGION" \
  CONTROL_PLANE_PROFILE="$TARGET_CONTROL/out/control-plane-profile.yaml" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  NVCF_CLI="$NVCF_CLI" \
  NVCF_CLI_CONFIG="$NVCF_CLI_CONFIG"
```

<Warning>
Do not delete the existing cluster registration. Re-registration must preserve
the source `clusterID` and `clusterGroupID`. A new identity can orphan existing
function deployments and tasks.
</Warning>

Compare the generated registration values with the IDs recorded before the
upgrade. Stop if either ID changed.

Install compute-plane 1.0.0:

```bash
make -C "$TARGET_COMPUTE" install \
  CLUSTER_NAME="$CLUSTER_NAME" \
  HELMFILE_ENV="$HELMFILE_ENV" \
  NCA_ID="$NCA_ID" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT"
```

Wait for the operator and backend, then verify the preserved identity and
target NVCA version:

```bash
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  -n nvca-operator rollout status deployment/nvca-operator --timeout=10m

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  wait nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'
```

The IDs must match the source values, `NVCA` must be 3.10.0, and `agent` must
be `healthy`. Repeat this section for every compute cluster. In a single-cluster
deployment, the control-plane and compute-plane kubeconfig values can point to
the same cluster. In a multi-cluster deployment, always pass the compute
cluster's kubeconfig and context explicitly.

## Validate the upgraded deployment

Repeat the baseline checks with the same IDs:

1. Get the source function and version. Their status must remain `ACTIVE`.
2. Get the source deployment. Its ID and cluster placement must be unchanged.
3. Invoke the source function and verify its response.
4. Get the source task. Its ID, name, status, and completion percentage must be
   unchanged.
5. Create a new task on a target compute cluster and wait for it to complete.
6. Recheck all seven Cassandra ledgers and OpenBao health.
7. Confirm every Helm release is deployed and every active controller is
   ready.

Use the same CLI commands from the baseline section. If the target CLI is a
different binary from the one used for the baseline, run
`nvcf-cli api-key generate` again before the invoke and task commands.
`function deploy get` requires an API key with the `deploy_function` scope. A
key without it fails with `403 Access Denied`. A successful invocation
validates the API, invocation service, gRPC proxy, NATS, NVCA, retained
function deployment, and compute data path. A completed new task validates the
target NVCT, ICMS, ESS, NVCA, and task-worker path.

After all checks pass, reopen control-plane mutations and end the maintenance
window. Clear the shell copy of the Cassandra password:

```bash
unset CASSANDRA_PASSWORD
```

## Stop and recovery criteria

Pause further mutations if any of these conditions occurs. Capture command
output, Job logs, resource health, and probe timestamps before choosing recovery:

- A probe fails outside the planned transitions or fails their recovery gate.
- A required completion check fails, or the maintenance cutoff is reached.
- A Cassandra migration ledger is dirty or has an unexpected version.
- The rendered Cassandra StatefulSet drops a `subPath` that the live
  StatefulSet uses on the data volume mount.
- `tasks_v2` still has `health_info` or lacks `health`.
- The OpenBao migration Job fails.
- An OpenBao member is sealed, uninitialized, or not ready.
- A compute cluster receives a different cluster ID or cluster-group ID.
- The required NATS streams are missing.
- A retained function, deployment, or task cannot be read after the upgrade.

Do not run a normal 0.6.2 Helm sync after the target Cassandra or OpenBao
migrations have completed. The target services can write data that the source
schemas and runtimes do not understand. Recovery after a completed data stop
requires restoring the Cassandra, OpenBao, and NATS backups as a consistent
set, restoring the 0.6.2 configuration, and then reinstalling the source stack.

Do not delete compute-cluster registrations during recovery. Restore the
control plane first, then reapply the source compute-plane bundle with the
preserved cluster identity.

## Related

- [Helmfile installation](/nvcf/1.1.0/self-managed/helmfile-installation)
- [Self-managed cluster registration](/nvcf/1.1.0/compute-plane/self-managed-clusters)
- [Control-plane operations](/nvcf/1.1.0/self-managed/control-plane-operations)