> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server.

# Upgrade from NVCF Self-Managed 0.6.1 to 1.0.0

| Plane | Source | Target |
| --- | --- | --- |
| Control plane | `nvcf-self-managed-stack` 0.6.1 | `nvcf-self-managed-stack` 1.0.0 |
| Compute plane | N/A | `nvcf-compute-plane-stack` 1.0.0 |
| Observability | Included with the 0.6.1 control plane | `nvcf-observability-stack` 1.0.0 |
| NVCA | 3.0.3 | 3.10.0 |

The 1.0.0 stack tags are release aliases of control-plane 0.20.6,
compute-plane 0.4.4, and observability 0.2.2. The component versions and
migration stops in this procedure are unchanged.

This procedure uses staged schema changes and Kubernetes rolling updates. The
control-plane API and task service can remain available while compatible
schemas and service versions are rolled. Block control-plane mutations during
the final validation and contract steps. Existing function workloads can
continue running in compute clusters throughout the upgrade.

<Warning>
This procedure supports only a healthy 0.6.1 starting state. Do not use it to
skip an earlier documented upgrade. Do not continue after a failed or dirty
database migration, a sealed OpenBao member, a changed cluster identity, or a
missing GPU inventory record.
</Warning>

## Upgrade stops

The source Cassandra migration image is 0.10.3. A healthy 0.6.1 database has
already completed the NVCF GPU specification and model specification
migrations and records NVCF schema version 7. The source NVCF API maps only
the post-migration data model. Do not deploy an older NVCF API runtime to
repeat those migrations.

The remaining Cassandra and service changes require the following ordered
gates:

| Gate | Schema transition | Service action | Completion check |
| --- | --- | --- | --- |
| Cassandra server | 5.0.8 to 5.0.9 | Upgrade the StatefulSet with migration hooks disabled. Rotate one member at a time. | Every member rejoins and `nodetool status` reports the expected healthy topology. |
| Expand | `nvcf_api` 7 to 8 | None. The change adds nullable `llm_config`. | Ledger is 8 and not dirty. |
| Expand | `sis_api` 6 to 10 | Add the instance lookup columns and indexes, the reservation flag, and cluster lookup indexes. | Ledger is 10 and not dirty. |
| OpenBao | No Cassandra change | Upgrade OpenBao and run migrations 0.16.2 to 0.19.5 before using the target service charts. | Every member is initialized, unsealed, ready, and uses the target image. The migration Job completes. |
| Roll ICMS bridge | `sis_api` remains at 10 | Roll ICMS 0.7.5 with the backfill mode initially `DISABLED`. New writes populate deployment and GPU specification IDs. | The Deployment rollout completes and an availability probe has no failed requests. |
| Backfill ICMS | `sis_api` remains at 10 | Set `ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE` to `MIGRATE`, then `VALIDATE`, then `DISABLED`. | Validation reports no unresolved active instance rows and no mismatched IDs. |
| Roll ICMS target | `sis_api` remains at 10 | Roll the stack-pinned ICMS 0.7.2 after disabling the bridge task. | Every bridge pod is gone and the availability probe has no failed requests. |
| Expand | `nvct_api` 3 to 4 | Add `tasks_v2.health` while retaining `health_info` and `health_udt`. | Ledger is 4 and both columns exist. |
| Roll NVCT bridge | `nvct_api` remains at 4 | Roll NVCT 1.64.3 with the backfill mode initially `DISABLED`. It writes `health` and reads `health` before falling back to `health_info`. | Every old pod is gone, the rollout completes, and an availability probe has no failed requests. |
| Backfill NVCT | `nvct_api` remains at 4 | Set `NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE` to `MIGRATE`, then `VALIDATE`, then `DISABLED`. | Validation reports no row with legacy health and an empty or mismatched replacement value. |
| Roll NVCT target | `nvct_api` remains at 4 | Roll NVCT 1.66.0 after the backfill passes. This runtime no longer reads the legacy column. | Every bridge pod is gone and the availability probe has no failed requests. |
| Contract | `nvct_api` 4 to 6 | Drop `health_info`, then drop `health_udt` while NVCT 1.66.0 remains online. | Ledger is 6, only `health` exists, and retained task health is unchanged. |
| Finish | Remaining ledgers reach the target versions | Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal Cassandra Helm release once all bounded migrations are complete. | All ledger, service, and data checks pass. |

The migration image contains the standard `migrate` CLI. It supports
`goto V` and `up N` for one migration path and migration table. The image's
default `/app/execute_sqls.sh` entrypoint always runs `up` for every keyspace,
and Cassandra migrations 0.17.1 through 0.17.6 package NVCT migrations 4, 5,
and 6 together. The released chart does not expose a command override or a
per-keyspace target. Use a bounded Job that invokes the existing image with
exact targets, or a chart update that exposes the same control. A new migration
binary is not required.

Each bounded Job must use the preprocessed SQL path for one keyspace and the
matching table in the `schema_migrations` keyspace. Before invoking `goto`, it
must require the expected clean source version. Afterward, it must require the
exact clean target version. Never use `goto` to move a ledger backward. For
example, the NVCT expand Job uses the `nvct_api` migration path and ledger,
requires version 3, invokes `goto 4`, and then requires version 4.

The service rollout and the data task are separate gates. NVCT 1.64.3 provides
the required dual-read compatibility and the health backfill. ICMS 0.7.5
provides the instance lookup backfill. Both tasks expose explicit `MIGRATE`,
`VALIDATE`, and `DISABLED` modes and default to `DISABLED`.
Run migration mode to completion, switch to validation mode, require zero
mismatches, and disable the task before rolling past the bridge. A completion
log without a validation pass is not a safe checkpoint.

The two upgrade-only images are:

| Service | Upgrade-only version | Final stack version | Task environment variable |
| --- | --- | --- | --- |
| ICMS | 0.7.5 | 0.7.2 | `ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE` |
| NVCT | 1.64.3 | 1.66.0 | `NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE` |

Use the immutable published versions listed above.

Helmfile upgrades one release at a time with `--selector name=<release>`.
Helm updates the Deployment and Kubernetes performs the rolling update. Use at
least two replicas for each service stop and require `maxUnavailable: 0` and
`maxSurge: 1`. The cluster must have capacity for the surge pod. A three-member
Cassandra cluster is required to keep the database available while its
StatefulSet rolls. A one-member Cassandra deployment has unavoidable database
downtime during the server restart.

The full path is:

1. Roll the Cassandra members with hooks disabled, then apply only the bounded
   expand migrations.
2. Upgrade OpenBao and complete its migrations. The target ICMS and NVCT
   charts require the target OpenBao roles and policies.
3. Roll ICMS 0.7.5 and NVCT 1.64.3. Run and validate both data tasks, then set
   both modes back to `DISABLED`.
4. Roll ICMS 0.7.2 and NVCT 1.66.0. Apply the NVCT contract migrations while
   NVCT remains online.
5. Sync the remaining target control-plane and observability releases. Run the
   normal Cassandra release and require its migration hook to be a no-op.
6. Re-register each existing compute cluster without deleting its identity,
   then sync compute-plane 1.0.0 and NVCA 3.10.0.

The source and target service versions remain:

| Service | Source runtime | Target runtime |
| --- | --- | --- |
| NVCF API | 1.9.0-hotfix.1 | 1.18.0 |
| NVCT API | 1.5.9-hotfix.1 | 1.66.0 |
| SIS / ICMS | 1.563.1-hotfix.1 | 0.7.2 |
| ESS API | 0.57.26 | 0.5.0 |
| NVCA | 3.0.3 | 3.10.0 |

The version-number decreases for ICMS and ESS reflect component lineage
changes. They are the runtimes shipped with the target stack set.

## Prepare the deployment host

Download and extract these target bundles into new directories:

- `nvcf-self-managed-stack` 1.0.0
- `nvcf-compute-plane-stack` 1.0.0
- `nvcf-observability-stack` 1.0.0, if observability is managed separately

Do not extract a target bundle over the 0.6.1 directories. Keep the source
configuration available for recovery.

Set variables for the deployment. Use a kubeconfig that selects only the
control-plane cluster, or verify the selected context before every command.

```bash
export HELMFILE_ENV=<environment-name>
export SOURCE_CONTROL=<absolute-path-to-0.6.1-control-plane-stack>
export TARGET_CONTROL=<absolute-path-to-1.0.0-control-plane-stack>
export TARGET_COMPUTE=<absolute-path-to-1.0.0-compute-plane-stack>
export CONTROL_KUBECONFIG=<absolute-path-to-control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export NVCF_CLI=<absolute-path-to-nvcf-cli>
export NVCF_CLI_CONFIG=<absolute-path-to-nvcf-cli-config>

kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}

kubeconfig_context=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
  config current-context)
test "$kubeconfig_context" = "$CONTROL_CONTEXT"
kcp cluster-info
```

Use Helm 3 and Helmfile 1.1.x. Authenticate Helm and the target clusters to all
registries used by the target bundles. Mirror every required target artifact
before the maintenance window. Use the inventory JSON attached to each 1.0.0
stack release as the exact artifact list. See the
[image mirroring](/nvcf/overview/image-mirroring) page for the mirroring procedure.

Use `nvcf-cli` 1.16.2 from the target release inventory. The target
registration flow refreshes the existing cluster metadata while preserving its
identity.

## Prepare the target configuration

Start from the target bundle templates. Reconcile the site-specific settings
from 0.6.1 into these target files:

```text
$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
$TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml
```

Do not copy the 0.6.1 files wholesale. Preserve the existing registry,
storage, endpoint, credential, ingress, certificate, and scheduling values,
then add settings required by the target templates.

The Cassandra service-role password in the target secrets file must match the
password already stored for the service-specific Cassandra roles. The same
value must be supplied to the OpenBao migration:

```yaml
cassandra:
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>

openbao:
  migrations:
    env:
      - name: DEFAULT_CASSANDRA_PASSWORD
        value: *cassandra_service_role_password
      - name: NVCF_API_SIDECARS_IMAGE_PULL_SECRET
        value: <base64-docker-credential>
      - name: ADMIN_CLIENT_ID
        value: ncp
```

If the Cassandra administrator password was changed from the chart default,
also carry the existing `cassandra.dbUser` values into the target secrets file.
Do not rotate Cassandra credentials as part of this upgrade.

Keep the live Cassandra storage class, size, and replica count unchanged.
Record them before rendering the target release:

```bash
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}'
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='replicas: {.spec.replicas}{"\n"}'
```

Render the target configuration before the maintenance window:

```bash
make -C "$TARGET_CONTROL" template \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
```

Review the rendered storage settings, images, registries, domains, and
namespaces. Do not continue if the target render changes a Cassandra volume
claim template.

## Verify the 0.6.1 starting state

Confirm that every source Helm release is deployed and every active controller
is ready:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A
```

Confirm that Cassandra has no pending repair or schema disagreement and that
all OpenBao members are initialized, unsealed, and ready. Identify the active
OpenBao member. Standby members report `HA Mode standby`; one member reports
`HA Mode active`.

```bash
kcp -n cassandra-system exec cassandra-0 -- nodetool status

for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done
```

Record the identity and version of every registered compute cluster. Run these
commands with that compute cluster's kubeconfig and context:

```bash
kubectl --kubeconfig <compute-kubeconfig> --context <compute-context> \
  get nvcfbackend <cluster-name> -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'
```

The source NVCA version must be 3.0.3 and `agent` must be `healthy`.

### Verify the GPU inventory precondition

Resolve the Cassandra administrator password from the original 0.6.1
installation configuration. The 0.6.1 chart does not always create a
Kubernetes Secret for this password, so do not assume a Secret name. Do not
enable shell tracing while the password variable is set.

```bash
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
echo
export CASSANDRA_PASSWORD

kcp -n cassandra-system exec cassandra-0 -- \
  cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
  -e 'SELECT cluster_id, cluster_status, gpus_v4, gpus_v5 FROM sis_api.cluster_by_cluster_id;'
```

Every active cluster row must have a nonempty `gpus_v5` value. Stop if this
check fails.

### Record application baselines

Record at least one active function, function version, deployment, completed
task, cluster ID, and cluster-group ID. Invoke the function and read the task
before the upgrade. Use the same records for post-upgrade validation.

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function get \
  --function-id <function-id> --version-id <version-id> --json

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id <function-id> --version-id <version-id> \
  --request-body '<sample-request>' --timeout 120 --json

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task get <task-id> --json
```

## Back up persistent state

Create restorable backups of Cassandra, OpenBao Raft storage, NATS JetStream,
the Helm values, and all target configuration files. Follow the storage
provider's backup procedure and verify the restore path before continuing.

Flush Cassandra and create a named snapshot on every Cassandra member before
the storage-level backup:

```bash
CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')

for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kcp -n cassandra-system exec "$pod" -- nodetool flush
  kcp -n cassandra-system exec "$pod" -- \
    nodetool snapshot -t pre-0-20-6
done
```

The Cassandra snapshot remains on each persistent volume. It is not an
off-cluster backup by itself. Copy or snapshot the volumes with your storage
provider.

Do not continue until the Cassandra and OpenBao backups are complete.

## Prepare the staged rollout

Do not scale the application Deployments to zero. Block configuration changes
that can invalidate the upgrade baseline, but keep normal read and task traffic
running for the availability test. Record the current images, replicas, and
rolling strategies:

```bash
kcp -n nvcf get deployment nvcf-api nvct-api -o wide
kcp -n sis get deployment spot-instance-service -o wide
kcp -n ess get deployment -o wide
kcp -n cassandra-system get statefulset cassandra -o wide
```

Use at least two application replicas for a zero-downtime upgrade. The rollout
must have an effective `maxUnavailable` of 0 and `maxSurge` of at least one pod.
Kubernetes calculates those values as 0 and 1 from the default 25 percent
settings when a Deployment has two replicas. A zero-downtime Cassandra server
upgrade requires at least three healthy members.

Start a continuous external probe before the first rollout. It must exercise
the same API and task reads used for the retained baseline. Record every
non-success response and stop the upgrade if any request fails. Keep the probe
running through the final schema and service validation.

## Stop 1: roll Cassandra and apply expand migrations

Upgrade only the Cassandra release with Helm hooks disabled. Preserve the
existing replica count and storage settings. With three members, rotate one
member at a time and require it to rejoin before continuing. A one-member
development deployment has unavoidable database downtime during this step.

Run migration image 0.17.6 as bounded Jobs. Require the expected clean source
version before each `goto` call and require the exact clean target afterward.
Stop at these versions:

| Migration table | Source | Expand target |
| --- | ---: | ---: |
| `api_keys_api` | 3 | 3 |
| `ess_api` | 8 | 9 |
| `event_ledger` | absent | 3 |
| `nvcf_api` | 7 | 8 |
| `nvcf_autoscaler` | 3 | 4 |
| `nvct_api` | 3 | 4 |
| `sis_api` | 6 | 10 |

Do not use the image's default `/app/execute_sqls.sh`. It runs every available
migration, including the destructive NVCT contract migrations. Each bounded
Job must select one preprocessed keyspace path and its matching migration
table, fail on an unexpected or dirty starting ledger, and stop immediately
on the first error.

Confirm Cassandra is healthy after the server roll and expand Jobs:

```bash
kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -- nodetool status
```

Do not roll either bridge until every expanded ledger has the exact version in
the table and `dirty = False`.

## Stop 2: migrate OpenBao

The OpenBao StatefulSet uses `OnDelete`. The target release also changes the
JWT plugin binary and catalog checksum. The safe sequence requires two manual
pod rotations:

1. Update the StatefulSet without hooks and rotate onto the target image.
2. Start the normal Helm sync so the catalog-refresh hook runs.
3. When the target migration Job appears, rotate the pods again so each server
   reloads the refreshed plugin catalog.
4. Let the migration Job and Helm sync finish.

Never delete the StatefulSet, its persistent volume claims, or more than one
OpenBao server pod at a time.

### Update the StatefulSet without hooks

Run this command from the target control-plane bundle:

```bash
cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=openbao-server \
  sync --args "--no-hooks"
```

Confirm the StatefulSet template uses `OnDelete` and the target image. Existing
pods can still show the source image.

```bash
kcp -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
```

### Perform the first OpenBao rotation

Identify the active and standby members with `bao status`. Rotate every standby
member first. For each standby pod, run:

```bash
wait_openbao_unsealed() {
  pod="$1"
  attempt=0
  while true; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status' 2>/dev/null || true)
    if printf '%s\n' "$status" | grep -Eq '^Initialized[[:space:]]+true$' && \
       printf '%s\n' "$status" | grep -Eq '^Sealed[[:space:]]+false$'; then
      printf '%s\n' "$status"
      return 0
    fi
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 120 ]; then
      echo "OpenBao member did not become initialized and unsealed: $pod" >&2
      return 1
    fi
    sleep 2
  done
}

export POD=<standby-pod-name>
kcp -n vault-system delete pod "$POD" --wait=true
kcp -n vault-system wait "pod/$POD" \
  --for=condition=Ready --timeout=10m
wait_openbao_unsealed "$POD"
```

Kubernetes readiness can become true before auto-unseal completes. The polling
check must report `Initialized true`, `Sealed false`, and a healthy standby HA
mode. Recheck every member, identify the current active member, and rotate it
last with the same commands.

Confirm all members are ready and use the target image before continuing:

```bash
kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.containerStatuses[*].ready}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'
```

### Refresh the catalog and run migrations

Use two terminals. Start the normal target OpenBao sync in terminal 1:

```bash
make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server
```

The command waits while the target catalog-refresh and migration hooks run.

In terminal 2, wait until the migration Job uses OpenBao migrations 0.19.5.
The migration Job is created only after the catalog-refresh hook succeeds:

```bash
until kcp -n vault-system get job openbao-server-migrations \
  -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null | \
  grep -Eq ':0[.]19[.]5$'; do
  sleep 2
done
```

As soon as the target migration Job appears, repeat the standby-first,
active-last rotation from the preceding section. This second rotation reloads
the refreshed plugin catalog. The migration Job uses `OnFailure` retries while
the rotation is in progress.

Wait for the sync in terminal 1 to finish. Then verify the migration Job and
all OpenBao members:

```bash
kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m

for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  kcp -n vault-system wait "pod/$pod" \
    --for=condition=Ready --timeout=10m
  wait_openbao_unsealed "$pod"
done
```

All members must be initialized, unsealed, ready, and running the same target
image. Do not continue if the migration Job failed or any member is unhealthy.

## Stop 3: run the service backfills and contract migrations

The target service charts require the OpenBao roles and policies created at
Stop 2. Use the target `sis` and `nvct-api` charts for both bridge rollouts.
Set at least two replicas and keep the external availability probe running.

Set the published bridge image locations:

```bash
export BRIDGE_REGISTRY=nvcr.io
export BRIDGE_REPOSITORY=nvidia/nvcf
export ICMS_BRIDGE_TAG=0.7.5
export NVCT_BRIDGE_TAG=1.64.3
export BRIDGE_PULL_POLICY=IfNotPresent
```

### Backfill ICMS instance lookup columns

Define a helper that keeps every bridge override in each Helm invocation:

```bash
sync_icms_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=sis sync \
    --set sis.replicaCount=2 \
    --set sis.image.registry="$BRIDGE_REGISTRY" \
    --set sis.image.repository="$BRIDGE_REPOSITORY/icms-service-oss" \
    --set sis.image.tag="$ICMS_BRIDGE_TAG" \
    --set sis.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set sis.env.ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE="$mode"
}
```

Roll the bridge with the task disabled, then run the migration and validation
as separate rollouts:

```bash
cd "$TARGET_CONTROL"
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

sync_icms_bridge MIGRATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

sync_icms_bridge VALIDATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
```

Require a completed `MIGRATE` record with `missingRequest=0`, `mismatched=0`,
and `failed=0`. Then require a completed `VALIDATE` record with `migrated=0`,
`missingRequest=0`, `mismatched=0`, and `failed=0`. The validation record must
report the migrated active rows as current. Keep the task `DISABLED` after the
validation pass.

Roll the stack-pinned ICMS image by removing the bridge image overrides:

```bash
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=sis sync \
  --set sis.replicaCount=2

kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
```

Confirm that the final Deployment uses ICMS 0.7.2 and that no bridge pod
remains.

### Backfill NVCT task health

Define the corresponding NVCT helper:

```bash
sync_nvct_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=nvct-api sync \
    --set nvctApi.replicaCount=2 \
    --set nvctApi.image.registry="$BRIDGE_REGISTRY" \
    --set nvctApi.image.repository="$BRIDGE_REPOSITORY/nvct-service-oss" \
    --set nvctApi.image.tag="$NVCT_BRIDGE_TAG" \
    --set nvctApi.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set nvctApi.env.NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE="$mode"
}
```

Roll the bridge and move through the same explicit task modes:

```bash
sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

sync_nvct_bridge MIGRATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

sync_nvct_bridge VALIDATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
```

Require a completed `MIGRATE` record with `withoutLegacyHealth=0`, `missing=0`,
`mismatched=0`, and `failed=0`. Then require a completed `VALIDATE` record with
`migrated=0`, `withoutLegacyHealth=0`, `missing=0`, `mismatched=0`, and
`failed=0`. Re-read the retained task and record its new `health` value before
continuing.

Roll the stack-pinned NVCT 1.66.0 image by removing the bridge overrides:

```bash
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=nvct-api sync \
  --set nvctApi.replicaCount=2

kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
```

After every bridge pod is gone, run the bounded NVCT migration Job from clean
version 4 to version 6. This drops `health_info` and `health_udt`. NVCT 1.66.0
remains online because it uses only the replacement `health` column.

Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal target Cassandra
release. Its migration hook must be a no-op because all ledgers are already at
their target versions.

After all steps succeed, confirm that the Cassandra migration Jobs completed
and that `nodetool status` reports a healthy topology:

```bash
kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -- nodetool status
```

Use the Cassandra administrator credentials from your deployment to query the
migration ledgers. Each row must have `dirty = False` and the following
version:

| Migration table | Version |
| --- | ---: |
| `api_keys_api` | 3 |
| `ess_api` | 9 |
| `event_ledger` | 3 |
| `nvcf_api` | 8 |
| `nvcf_autoscaler` | 4 |
| `nvct_api` | 6 |
| `sis_api` | 10 |

Run these CQL statements through the site's approved Cassandra administration
path:

```sql
SELECT version, dirty FROM schema_migrations.api_keys_api;
SELECT version, dirty FROM schema_migrations.ess_api;
SELECT version, dirty FROM schema_migrations.event_ledger;
SELECT version, dirty FROM schema_migrations.nvcf_api;
SELECT version, dirty FROM schema_migrations.nvcf_autoscaler;
SELECT version, dirty FROM schema_migrations.nvct_api;
SELECT version, dirty FROM schema_migrations.sis_api;
```

Confirm that the NVCT task schema contains only the replacement health column:

```sql
SELECT column_name, type
FROM system_schema.columns
WHERE keyspace_name = 'nvct_api'
  AND table_name = 'tasks_v2'
  AND column_name IN ('health', 'health_info');
```

The result must contain only `health`. Re-read the retained task and require
the same serialized health value recorded before the contract migration. Do
not continue if a ledger is dirty, a version differs, or the schema or retained
data check fails.

## Sync the target control plane and observability stack

The selector of the `reval` Deployment changed after 0.6.1. Kubernetes does
not permit that field to change in place. Delete only the Deployment before
the full sync. Do not uninstall the Helm release or delete its namespace.

```bash
kcp -n nvcf delete deployment reval
```

Sync the full target control-plane bundle. Its default control profile also
installs the target observability components and function autoscaler.

```bash
make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
```

This sync reruns the OpenBao hooks. They must finish successfully without
another pod rotation. Confirm that every Helm release is deployed and every
active Deployment or StatefulSet is ready:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A
```

Confirm the required global NATS streams still exist:

```bash
NATS_BOX_POD=$(kcp -n nats-system get pod \
  -l app.kubernetes.io/component=nats-box \
  -o jsonpath='{.items[0].metadata.name}')

for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do
  kcp -n nats-system exec "$NATS_BOX_POD" -- \
    nats --context default stream info "$stream"
done
```

If observability is managed by a separate stack instance, keep the
self-managed ownership settings consistent with the existing deployment and
sync `nvcf-observability-stack` 1.0.0 as the only owner. Do not install two
owners for the same observability resources.

## Upgrade each compute cluster

Export a fresh control-plane profile from the target bundle:

```bash
"$NVCF_CLI" self-hosted \
  --control-plane-stack "$TARGET_CONTROL" \
  --env "$HELMFILE_ENV" \
  control-plane profile export

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" init
```

For each compute cluster, set its values and verify the selected context:

```bash
export CLUSTER_NAME=<existing-cluster-name>
export CLUSTER_REGION=<cluster-region>
export NCA_ID=<existing-nca-id>
export COMPUTE_KUBECONFIG=<absolute-path-to-compute-kubeconfig>
export COMPUTE_CONTEXT=<compute-context>

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" cluster-info
```

Confirm that `nvcr-pull-secret`, or the pull secret named in the compute-plane
environment, exists in `nvca-operator`. The operator mirrors it to
`nvca-system` for NVCA sidecars and managed workloads.

Re-register the existing cluster with the target control-plane profile:

```bash
make -C "$TARGET_COMPUTE" register-cluster \
  CLUSTER_NAME="$CLUSTER_NAME" \
  CLUSTER_REGION="$CLUSTER_REGION" \
  CONTROL_PLANE_PROFILE="$TARGET_CONTROL/out/control-plane-profile.yaml" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  NVCF_CLI="$NVCF_CLI" \
  NVCF_CLI_CONFIG="$NVCF_CLI_CONFIG"
```

<Warning>
Do not delete the existing cluster registration. Re-registration must preserve
the source `clusterID` and `clusterGroupID`. A new identity can orphan existing
function deployments and tasks.
</Warning>

Compare the generated registration values with the IDs recorded before the
upgrade. Stop if either ID changed.

Install compute-plane 1.0.0:

```bash
make -C "$TARGET_COMPUTE" install \
  CLUSTER_NAME="$CLUSTER_NAME" \
  HELMFILE_ENV="$HELMFILE_ENV" \
  NCA_ID="$NCA_ID" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT"
```

Wait for the operator and backend, then verify the preserved identity and
target NVCA version:

```bash
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  -n nvca-operator rollout status deployment/nvca-operator --timeout=10m

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  wait nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'
```

The IDs must match the source values, `NVCA` must be 3.10.0, and `agent` must
be `healthy`. Repeat this section for every compute cluster. In a single-cluster
deployment, the control-plane and compute-plane kubeconfig values can point to
the same cluster. In a multi-cluster deployment, always pass the compute
cluster's kubeconfig and context explicitly.

## Validate the upgraded deployment

Repeat the baseline checks with the same IDs:

1. Get the source function and version. Their status must remain `ACTIVE`.
2. Get the source deployment. Its ID and cluster placement must be unchanged.
3. Invoke the source function and verify its response.
4. Get the source task. Its ID, name, status, and completion percentage must be
   unchanged.
5. Create a new task on a target compute cluster and wait for it to complete.
6. Recheck all seven Cassandra ledgers and OpenBao health.
7. Confirm every Helm release is deployed and every active controller is
   ready.

Use the same CLI commands from the baseline section. A successful invocation
validates the API, invocation service, gRPC proxy, NATS, NVCA, retained
function deployment, and compute data path. A completed new task validates the
target NVCT, ICMS, ESS, NVCA, and task-worker path.

After all checks pass, reopen control-plane mutations and end the maintenance
window. Clear the shell copy of the Cassandra password:

```bash
unset CASSANDRA_PASSWORD
```

## Stop and recovery criteria

Stop the upgrade if any of these conditions occurs:

- A Cassandra migration ledger is dirty or has an unexpected version.
- `tasks_v2` still has `health_info` or lacks `health`.
- The OpenBao migration Job fails.
- An OpenBao member is sealed, uninitialized, or not ready.
- A compute cluster receives a different cluster ID or cluster-group ID.
- The required NATS streams are missing.
- A retained function, deployment, or task cannot be read after the upgrade.

Do not run a normal 0.6.1 Helm sync after the target Cassandra or OpenBao
migrations have completed. The target services can write data that the source
schemas and runtimes do not understand. Recovery after a completed data stop
requires restoring the Cassandra, OpenBao, and NATS backups as a consistent
set, restoring the 0.6.1 configuration, and then reinstalling the source stack.

Do not delete compute-cluster registrations during recovery. Restore the
control plane first, then reapply the source compute-plane bundle with the
preserved cluster identity.

## Related

- [Helmfile installation](/nvcf/self-managed/helmfile-installation)
- [Self-managed cluster registration](/nvcf/compute-plane/self-managed-clusters)
- [Control-plane operations](/nvcf/self-managed/control-plane-operations)