Upgrade from NVCF Self-Managed 0.6.1 to 1.0.0

The 1.0.0 stack tags are release aliases of control-plane 0.20.6, compute-plane 0.4.4, and observability 0.2.2. The component versions and migration stops in this procedure are unchanged.

This procedure uses staged schema changes and Kubernetes rolling updates. The control-plane API and task service can remain available while compatible schemas and service versions are rolled. Block control-plane mutations during the final validation and contract steps. Existing function workloads can continue running in compute clusters throughout the upgrade.

Upgrade stops

The source Cassandra migration image is 0.10.3. A healthy 0.6.1 database has already completed the NVCF GPU specification and model specification migrations and records NVCF schema version 7. The source NVCF API maps only the post-migration data model. Do not deploy an older NVCF API runtime to repeat those migrations.

The remaining Cassandra and service changes require the following ordered gates:

The migration image contains the standard migrate CLI. It supports goto V and up N for one migration path and migration table. The image’s default /app/execute_sqls.sh entrypoint always runs up for every keyspace, and Cassandra migrations 0.17.1 through 0.17.6 package NVCT migrations 4, 5, and 6 together. The released chart does not expose a command override or a per-keyspace target. Use a bounded Job that invokes the existing image with exact targets, or a chart update that exposes the same control. A new migration binary is not required.

Each bounded Job must use the preprocessed SQL path for one keyspace and the matching table in the schema_migrations keyspace. Before invoking goto, it must require the expected clean source version. Afterward, it must require the exact clean target version. Never use goto to move a ledger backward. For example, the NVCT expand Job uses the nvct_api migration path and ledger, requires version 3, invokes goto 4, and then requires version 4.

The service rollout and the data task are separate gates. NVCT 1.64.3 provides the required dual-read compatibility and the health backfill. ICMS 0.7.5 provides the instance lookup backfill. Both tasks expose explicit MIGRATE, VALIDATE, and DISABLED modes and default to DISABLED. Run migration mode to completion, switch to validation mode, require zero mismatches, and disable the task before rolling past the bridge. A completion log without a validation pass is not a safe checkpoint.

The two upgrade-only images are:

Use the immutable published versions listed above.

Helmfile upgrades one release at a time with --selector name=<release>. Helm updates the Deployment and Kubernetes performs the rolling update. Use at least two replicas for each service stop and require maxUnavailable: 0 and maxSurge: 1. The cluster must have capacity for the surge pod. A three-member Cassandra cluster is required to keep the database available while its StatefulSet rolls. A one-member Cassandra deployment has unavoidable database downtime during the server restart.

The full path is:

1. Roll the Cassandra members with hooks disabled, then apply only the bounded expand migrations.
2. Upgrade OpenBao and complete its migrations. The target ICMS and NVCT charts require the target OpenBao roles and policies.
3. Roll ICMS 0.7.5 and NVCT 1.64.3. Run and validate both data tasks, then set both modes back to DISABLED.
4. Roll ICMS 0.7.2 and NVCT 1.66.0. Apply the NVCT contract migrations while NVCT remains online.
5. Sync the remaining target control-plane and observability releases. Run the normal Cassandra release and require its migration hook to be a no-op.
6. Re-register each existing compute cluster without deleting its identity, then sync compute-plane 1.0.0 and NVCA 3.10.0.

The source and target service versions remain:

The version-number decreases for ICMS and ESS reflect component lineage changes. They are the runtimes shipped with the target stack set.

Prepare the deployment host

Download and extract these target bundles into new directories:

• nvcf-self-managed-stack 1.0.0
• nvcf-compute-plane-stack 1.0.0
• nvcf-observability-stack 1.0.0, if observability is managed separately

Do not extract a target bundle over the 0.6.1 directories. Keep the source configuration available for recovery.

Set variables for the deployment. Use a kubeconfig that selects only the control-plane cluster, or verify the selected context before every command.

export HELMFILE_ENV=<environment-name>
export SOURCE_CONTROL=<absolute-path-to-0.6.1-control-plane-stack>
export TARGET_CONTROL=<absolute-path-to-1.0.0-control-plane-stack>
export TARGET_COMPUTE=<absolute-path-to-1.0.0-compute-plane-stack>
export CONTROL_KUBECONFIG=<absolute-path-to-control-plane-kubeconfig>
export CONTROL_CONTEXT=<control-plane-context>
export NVCF_CLI=<absolute-path-to-nvcf-cli>
export NVCF_CLI_CONFIG=<absolute-path-to-nvcf-cli-config>
kcp() {
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
    --context "$CONTROL_CONTEXT" "$@"
}
kubeconfig_context=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" \
  config current-context)
test "$kubeconfig_context" = "$CONTROL_CONTEXT"
kcp cluster-info

Use Helm 3 and Helmfile 1.1.x. Authenticate Helm and the target clusters to all registries used by the target bundles. Mirror every required target artifact before the maintenance window. Use the inventory JSON attached to each 1.0.0 stack release as the exact artifact list. See the image mirroring page for the mirroring procedure.

Use nvcf-cli 1.16.2 from the target release inventory. The target registration flow refreshes the existing cluster metadata while preserving its identity.

Prepare the target configuration

Start from the target bundle templates. Reconcile the site-specific settings from 0.6.1 into these target files:

$TARGET_CONTROL/environments/$HELMFILE_ENV.yaml
$TARGET_CONTROL/secrets/$HELMFILE_ENV-secrets.yaml
$TARGET_COMPUTE/environments/$HELMFILE_ENV.yaml

Do not copy the 0.6.1 files wholesale. Preserve the existing registry, storage, endpoint, credential, ingress, certificate, and scheduling values, then add settings required by the target templates.

The Cassandra service-role password in the target secrets file must match the password already stored for the service-specific Cassandra roles. The same value must be supplied to the OpenBao migration:

cassandra:
  serviceRolePassword: &cassandra_service_role_password <existing-service-role-password>
openbao:
  migrations:
    env:
      - name: DEFAULT_CASSANDRA_PASSWORD
        value: *cassandra_service_role_password
      - name: NVCF_API_SIDECARS_IMAGE_PULL_SECRET
        value: <base64-docker-credential>
      - name: ADMIN_CLIENT_ID
        value: ncp

If the Cassandra administrator password was changed from the chart default, also carry the existing cassandra.dbUser values into the target secrets file. Do not rotate Cassandra credentials as part of this upgrade.

Keep the live Cassandra storage class, size, and replica count unchanged. Record them before rendering the target release:

kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}'
kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='replicas: {.spec.replicas}{"\n"}'

Render the target configuration before the maintenance window:

make -C "$TARGET_CONTROL" template \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"

Review the rendered storage settings, images, registries, domains, and namespaces. Do not continue if the target render changes a Cassandra volume claim template.

Verify the 0.6.1 starting state

Confirm that every source Helm release is deployed and every active controller is ready:

helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A

Confirm that Cassandra has no pending repair or schema disagreement and that all OpenBao members are initialized, unsealed, and ready. Identify the active OpenBao member. Standby members report HA Mode standby; one member reports HA Mode active.

kcp -n cassandra-system exec cassandra-0 -- nodetool status
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $pod =="
  kcp -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status'
done

Record the identity and version of every registered compute cluster. Run these commands with that compute cluster’s kubeconfig and context:

kubectl --kubeconfig <compute-kubeconfig> --context <compute-context> \
  get nvcfbackend <cluster-name> -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'

The source NVCA version must be 3.0.3 and agent must be healthy.

Verify the GPU inventory precondition

Resolve the Cassandra administrator password from the original 0.6.1 installation configuration. The 0.6.1 chart does not always create a Kubernetes Secret for this password, so do not assume a Secret name. Do not enable shell tracing while the password variable is set.

read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
echo
export CASSANDRA_PASSWORD
kcp -n cassandra-system exec cassandra-0 -- \
  cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
  -e 'SELECT cluster_id, cluster_status, gpus_v4, gpus_v5 FROM sis_api.cluster_by_cluster_id;'

Every active cluster row must have a nonempty gpus_v5 value. Stop if this check fails.

Record application baselines

Record at least one active function, function version, deployment, completed task, cluster ID, and cluster-group ID. Invoke the function and read the task before the upgrade. Use the same records for post-upgrade validation.

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function get \
  --function-id <function-id> --version-id <version-id> --json
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id <function-id> --version-id <version-id> \
  --request-body '<sample-request>' --timeout 120 --json
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task get <task-id> --json

Back up persistent state

Create restorable backups of Cassandra, OpenBao Raft storage, NATS JetStream, the Helm values, and all target configuration files. Follow the storage provider’s backup procedure and verify the restore path before continuing.

Flush Cassandra and create a named snapshot on every Cassandra member before the storage-level backup:

CASSANDRA_REPLICAS=$(kcp -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kcp -n cassandra-system exec "$pod" -- nodetool flush
  kcp -n cassandra-system exec "$pod" -- \
    nodetool snapshot -t pre-0-20-6
done

The Cassandra snapshot remains on each persistent volume. It is not an off-cluster backup by itself. Copy or snapshot the volumes with your storage provider.

Do not continue until the Cassandra and OpenBao backups are complete.

Prepare the staged rollout

Do not scale the application Deployments to zero. Block configuration changes that can invalidate the upgrade baseline, but keep normal read and task traffic running for the availability test. Record the current images, replicas, and rolling strategies:

kcp -n nvcf get deployment nvcf-api nvct-api -o wide
kcp -n sis get deployment spot-instance-service -o wide
kcp -n ess get deployment -o wide
kcp -n cassandra-system get statefulset cassandra -o wide

Use at least two application replicas for a zero-downtime upgrade. The rollout must have an effective maxUnavailable of 0 and maxSurge of at least one pod. Kubernetes calculates those values as 0 and 1 from the default 25 percent settings when a Deployment has two replicas. A zero-downtime Cassandra server upgrade requires at least three healthy members.

Start a continuous external probe before the first rollout. It must exercise the same API and task reads used for the retained baseline. Record every non-success response and stop the upgrade if any request fails. Keep the probe running through the final schema and service validation.

Stop 1: roll Cassandra and apply expand migrations

Upgrade only the Cassandra release with Helm hooks disabled. Preserve the existing replica count and storage settings. With three members, rotate one member at a time and require it to rejoin before continuing. A one-member development deployment has unavoidable database downtime during this step.

Run migration image 0.17.6 as bounded Jobs. Require the expected clean source version before each goto call and require the exact clean target afterward. Stop at these versions:

Do not use the image’s default /app/execute_sqls.sh. It runs every available migration, including the destructive NVCT contract migrations. Each bounded Job must select one preprocessed keyspace path and its matching migration table, fail on an unexpected or dirty starting ledger, and stop immediately on the first error.

Confirm Cassandra is healthy after the server roll and expand Jobs:

kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -- nodetool status

Do not roll either bridge until every expanded ledger has the exact version in the table and dirty = False.

Stop 2: migrate OpenBao

The OpenBao StatefulSet uses OnDelete. The target release also changes the JWT plugin binary and catalog checksum. The safe sequence requires two manual pod rotations:

1. Update the StatefulSet without hooks and rotate onto the target image.
2. Start the normal Helm sync so the catalog-refresh hook runs.
3. When the target migration Job appears, rotate the pods again so each server reloads the refreshed plugin catalog.
4. Let the migration Job and Helm sync finish.

Never delete the StatefulSet, its persistent volume claims, or more than one OpenBao server pod at a time.

Update the StatefulSet without hooks

Run this command from the target control-plane bundle:

cd "$TARGET_CONTROL"
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=openbao-server \
  sync --args "--no-hooks"

Confirm the StatefulSet template uses OnDelete and the target image. Existing pods can still show the source image.

kcp -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'

Perform the first OpenBao rotation

Identify the active and standby members with bao status. Rotate every standby member first. For each standby pod, run:

wait_openbao_unsealed() {
  pod="$1"
  attempt=0
  while true; do
    status=$(kcp -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status' 2>/dev/null || true)
    if printf '%s\n' "$status" | grep -Eq '^Initialized[[:space:]]+true$' && \
       printf '%s\n' "$status" | grep -Eq '^Sealed[[:space:]]+false$'; then
      printf '%s\n' "$status"
      return 0
    fi
    attempt=$((attempt + 1))
    if [ "$attempt" -ge 120 ]; then
      echo "OpenBao member did not become initialized and unsealed: $pod" >&2
      return 1
    fi
    sleep 2
  done
}
export POD=<standby-pod-name>
kcp -n vault-system delete pod "$POD" --wait=true
kcp -n vault-system wait "pod/$POD" \
  --for=condition=Ready --timeout=10m
wait_openbao_unsealed "$POD"

Kubernetes readiness can become true before auto-unseal completes. The polling check must report Initialized true, Sealed false, and a healthy standby HA mode. Recheck every member, identify the current active member, and rotate it last with the same commands.

Confirm all members are ready and use the target image before continuing:

kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.containerStatuses[*].ready}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'

Refresh the catalog and run migrations

Use two terminals. Start the normal target OpenBao sync in terminal 1:

make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" \
  HELMFILE_SELECTOR=name=openbao-server

The command waits while the target catalog-refresh and migration hooks run.

In terminal 2, wait until the migration Job uses OpenBao migrations 0.19.5. The migration Job is created only after the catalog-refresh hook succeeds:

until kcp -n vault-system get job openbao-server-migrations \
  -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null | \
  grep -Eq ':0[.]19[.]5$'; do
  sleep 2
done

As soon as the target migration Job appears, repeat the standby-first, active-last rotation from the preceding section. This second rotation reloads the refreshed plugin catalog. The migration Job uses OnFailure retries while the rotation is in progress.

Wait for the sync in terminal 1 to finish. Then verify the migration Job and all OpenBao members:

kcp -n vault-system wait job/openbao-server-migrations \
  --for=condition=Complete --timeout=15m
for pod in $(kcp -n vault-system get pods \
  -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  kcp -n vault-system wait "pod/$pod" \
    --for=condition=Ready --timeout=10m
  wait_openbao_unsealed "$pod"
done

All members must be initialized, unsealed, ready, and running the same target image. Do not continue if the migration Job failed or any member is unhealthy.

Stop 3: run the service backfills and contract migrations

The target service charts require the OpenBao roles and policies created at Stop 2. Use the target sis and nvct-api charts for both bridge rollouts. Set at least two replicas and keep the external availability probe running.

Set the published bridge image locations:

export BRIDGE_REGISTRY=nvcr.io
export BRIDGE_REPOSITORY=nvidia/nvcf
export ICMS_BRIDGE_TAG=0.7.5
export NVCT_BRIDGE_TAG=1.64.3
export BRIDGE_PULL_POLICY=IfNotPresent

Backfill ICMS instance lookup columns

Define a helper that keeps every bridge override in each Helm invocation:

sync_icms_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=sis sync \
    --set sis.replicaCount=2 \
    --set sis.image.registry="$BRIDGE_REGISTRY" \
    --set sis.image.repository="$BRIDGE_REPOSITORY/icms-service-oss" \
    --set sis.image.tag="$ICMS_BRIDGE_TAG" \
    --set sis.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set sis.env.ICMS_INSTANCE_LOOKUP_BACKFILL_TASK_MODE="$mode"
}

Roll the bridge with the task disabled, then run the migration and validation as separate rollouts:

cd "$TARGET_CONTROL"
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
sync_icms_bridge MIGRATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
sync_icms_bridge VALIDATE
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m
sync_icms_bridge DISABLED
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

Require a completed MIGRATE record with missingRequest=0, mismatched=0, and failed=0. Then require a completed VALIDATE record with migrated=0, missingRequest=0, mismatched=0, and failed=0. The validation record must report the migrated active rows as current. Keep the task DISABLED after the validation pass.

Roll the stack-pinned ICMS image by removing the bridge image overrides:

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=sis sync \
  --set sis.replicaCount=2
kcp -n sis rollout status deployment/spot-instance-service --timeout=10m

Confirm that the final Deployment uses ICMS 0.7.2 and that no bridge pod remains.

Backfill NVCT task health

Define the corresponding NVCT helper:

sync_nvct_bridge() {
  mode="$1"
  HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
    --kubeconfig "$CONTROL_KUBECONFIG" \
    --selector name=nvct-api sync \
    --set nvctApi.replicaCount=2 \
    --set nvctApi.image.registry="$BRIDGE_REGISTRY" \
    --set nvctApi.image.repository="$BRIDGE_REPOSITORY/nvct-service-oss" \
    --set nvctApi.image.tag="$NVCT_BRIDGE_TAG" \
    --set nvctApi.image.pullPolicy="$BRIDGE_PULL_POLICY" \
    --set nvctApi.env.NVCT_SCHEDULED_ROUTINES_HEALTH_BACKFILL_ROUTINE_MODE="$mode"
}

Roll the bridge and move through the same explicit task modes:

sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
sync_nvct_bridge MIGRATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
sync_nvct_bridge VALIDATE
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m
sync_nvct_bridge DISABLED
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

Require a completed MIGRATE record with withoutLegacyHealth=0, missing=0, mismatched=0, and failed=0. Then require a completed VALIDATE record with migrated=0, withoutLegacyHealth=0, missing=0, mismatched=0, and failed=0. Re-read the retained task and record its new health value before continuing.

Roll the stack-pinned NVCT 1.66.0 image by removing the bridge overrides:

HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
  --kubeconfig "$CONTROL_KUBECONFIG" \
  --selector name=nvct-api sync \
  --set nvctApi.replicaCount=2
kcp -n nvcf rollout status deployment/nvct-api --timeout=10m

After every bridge pod is gone, run the bounded NVCT migration Job from clean version 4 to version 6. This drops health_info and health_udt. NVCT 1.66.0 remains online because it uses only the replacement health column.

Roll NVCF API 1.18.0 and ESS 0.5.0, then run the normal target Cassandra release. Its migration hook must be a no-op because all ledgers are already at their target versions.

After all steps succeed, confirm that the Cassandra migration Jobs completed and that nodetool status reports a healthy topology:

kcp -n cassandra-system get jobs
kcp -n cassandra-system exec cassandra-0 -- nodetool status

Use the Cassandra administrator credentials from your deployment to query the migration ledgers. Each row must have dirty = False and the following version:

Run these CQL statements through the site’s approved Cassandra administration path:

SELECT version, dirty FROM schema_migrations.api_keys_api;
SELECT version, dirty FROM schema_migrations.ess_api;
SELECT version, dirty FROM schema_migrations.event_ledger;
SELECT version, dirty FROM schema_migrations.nvcf_api;
SELECT version, dirty FROM schema_migrations.nvcf_autoscaler;
SELECT version, dirty FROM schema_migrations.nvct_api;
SELECT version, dirty FROM schema_migrations.sis_api;

Confirm that the NVCT task schema contains only the replacement health column:

SELECT column_name, type
FROM system_schema.columns
WHERE keyspace_name = 'nvct_api'
  AND table_name = 'tasks_v2'
  AND column_name IN ('health', 'health_info');

The result must contain only health. Re-read the retained task and require the same serialized health value recorded before the contract migration. Do not continue if a ledger is dirty, a version differs, or the schema or retained data check fails.

Sync the target control plane and observability stack

The selector of the reval Deployment changed after 0.6.1. Kubernetes does not permit that field to change in place. Delete only the Deployment before the full sync. Do not uninstall the Helm release or delete its namespace.

kcp -n nvcf delete deployment reval

Sync the full target control-plane bundle. Its default control profile also installs the target observability components and function autoscaler.

make -C "$TARGET_CONTROL" install \
  HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"

This sync reruns the OpenBao hooks. They must finish successfully without another pod rotation. Confirm that every Helm release is deployed and every active Deployment or StatefulSet is ready:

helm --kubeconfig "$CONTROL_KUBECONFIG" \
  --kube-context "$CONTROL_CONTEXT" list -A
kcp get deployments,statefulsets -A
kcp get jobs -A

Confirm the required global NATS streams still exist:

NATS_BOX_POD=$(kcp -n nats-system get pod \
  -l app.kubernetes.io/component=nats-box \
  -o jsonpath='{.items[0].metadata.name}')
for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do
  kcp -n nats-system exec "$NATS_BOX_POD" -- \
    nats --context default stream info "$stream"
done

If observability is managed by a separate stack instance, keep the self-managed ownership settings consistent with the existing deployment and sync nvcf-observability-stack 1.0.0 as the only owner. Do not install two owners for the same observability resources.

Upgrade each compute cluster

Export a fresh control-plane profile from the target bundle:

"$NVCF_CLI" self-hosted \
  --control-plane-stack "$TARGET_CONTROL" \
  --env "$HELMFILE_ENV" \
  control-plane profile export
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" init

For each compute cluster, set its values and verify the selected context:

export CLUSTER_NAME=<existing-cluster-name>
export CLUSTER_REGION=<cluster-region>
export NCA_ID=<existing-nca-id>
export COMPUTE_KUBECONFIG=<absolute-path-to-compute-kubeconfig>
export COMPUTE_CONTEXT=<compute-context>
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" cluster-info

Confirm that nvcr-pull-secret, or the pull secret named in the compute-plane environment, exists in nvca-operator. The operator mirrors it to nvca-system for NVCA sidecars and managed workloads.

Re-register the existing cluster with the target control-plane profile:

make -C "$TARGET_COMPUTE" register-cluster \
  CLUSTER_NAME="$CLUSTER_NAME" \
  CLUSTER_REGION="$CLUSTER_REGION" \
  CONTROL_PLANE_PROFILE="$TARGET_CONTROL/out/control-plane-profile.yaml" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  NVCF_CLI="$NVCF_CLI" \
  NVCF_CLI_CONFIG="$NVCF_CLI_CONFIG"

Compare the generated registration values with the IDs recorded before the upgrade. Stop if either ID changed.

Install compute-plane 1.0.0:

make -C "$TARGET_COMPUTE" install \
  CLUSTER_NAME="$CLUSTER_NAME" \
  HELMFILE_ENV="$HELMFILE_ENV" \
  NCA_ID="$NCA_ID" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" \
  COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT"

Wait for the operator and backend, then verify the preserved identity and target NVCA version:

kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  -n nvca-operator rollout status deployment/nvca-operator --timeout=10m
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  wait nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" \
  --context "$COMPUTE_CONTEXT" \
  get nvcfbackend "$CLUSTER_NAME" -n nvca-operator \
  -o jsonpath='ncaID: {.status.accountConfig.ncaID}{"\n"}clusterID: {.status.clusterConfig.clusterID}{"\n"}clusterGroupID: {.status.clusterConfig.clusterGroupID}{"\n"}NVCA: {.status.version}{"\n"}agent: {.status.agentStatus}{"\n"}'

The IDs must match the source values, NVCA must be 3.10.0, and agent must be healthy. Repeat this section for every compute cluster. In a single-cluster deployment, the control-plane and compute-plane kubeconfig values can point to the same cluster. In a multi-cluster deployment, always pass the compute cluster’s kubeconfig and context explicitly.

Validate the upgraded deployment

Repeat the baseline checks with the same IDs:

1. Get the source function and version. Their status must remain ACTIVE.
2. Get the source deployment. Its ID and cluster placement must be unchanged.
3. Invoke the source function and verify its response.
4. Get the source task. Its ID, name, status, and completion percentage must be unchanged.
5. Create a new task on a target compute cluster and wait for it to complete.
6. Recheck all seven Cassandra ledgers and OpenBao health.
7. Confirm every Helm release is deployed and every active controller is ready.

Use the same CLI commands from the baseline section. A successful invocation validates the API, invocation service, gRPC proxy, NATS, NVCA, retained function deployment, and compute data path. A completed new task validates the target NVCT, ICMS, ESS, NVCA, and task-worker path.

After all checks pass, reopen control-plane mutations and end the maintenance window. Clear the shell copy of the Cassandra password:

unset CASSANDRA_PASSWORD

Stop and recovery criteria

Stop the upgrade if any of these conditions occurs:

• A Cassandra migration ledger is dirty or has an unexpected version.
• tasks_v2 still has health_info or lacks health.
• The OpenBao migration Job fails.
• An OpenBao member is sealed, uninitialized, or not ready.
• A compute cluster receives a different cluster ID or cluster-group ID.
• The required NATS streams are missing.
• A retained function, deployment, or task cannot be read after the upgrade.

Do not run a normal 0.6.1 Helm sync after the target Cassandra or OpenBao migrations have completed. The target services can write data that the source schemas and runtimes do not understand. Recovery after a completed data stop requires restoring the Cassandra, OpenBao, and NATS backups as a consistent set, restoring the 0.6.1 configuration, and then reinstalling the source stack.

Do not delete compute-cluster registrations during recovery. Restore the control plane first, then reapply the source compute-plane bundle with the preserved cluster identity.

Related

• Helmfile installation
• Self-managed cluster registration
• Control-plane operations