0.5.0 to 0.6.0 Upgrade

The upgrade from 0.5.0 to 0.6.0 crosses two scheduled data migrations and one required Spot Instance Service cleanup boundary. Run Stops 1 and 2, sync the full 0.6.0 stack while the Spot Instance Service migration runtime can restore the global NATS streams, and then run Stop 3.

$
export HELMFILE_ENV=<your-environment>   # for example eks-cds-qa

Prerequisite: upgrade to 0.5.1 or later first

Moving from 0.5.0 to 0.5.1 is a normal patch sync, not a migration stop. Download the 0.5.1 stack and sync it with your existing environment:

$
# Create the pull secret in every NVCF namespace (private registry only).
$
# The 0.5.1 charts reference this secret by name; without it there is nothing to propagate.
$
kubectl create namespace cert-manager \
>
  --dry-run=client -o yaml | kubectl apply -f -
$
$
for ns in cassandra-system nats-system nvcf api-keys ess sis vault-system nvca-operator nvca-system nvcf-backend cert-manager; do
$
  kubectl create secret docker-registry nvcr-pull-secret \
>
    --docker-server=<registry> --docker-username='<user>' --docker-password='<token>' \
>
    -n "$ns" --dry-run=client -o yaml | kubectl apply -f -
$
done
$
$
# Confirm the env references the secret so every chart inherits it.
$
# environments/<env>.yaml should contain:
$
#   global:
$
#     imagePullSecrets:
$
#       - name: nvcr-pull-secret
$
$
# Sync the full 0.5.1 stack so every chart re-renders with imagePullSecrets set.
$
make install HELMFILE_ENV="$HELMFILE_ENV"
$
$
# Confirm the control plane is healthy before starting the 0.6.0 upgrade.
$
# This should print nothing; any output is a pod that is not yet Running/Completed.
$
kubectl get pods -A | grep -vE 'Running|Completed|READY'

Prerequisite: install nvcf-cli 1.8.3 or later

Use nvcf-cli 1.8.3 or later when you re-register GPU clusters after the control-plane upgrade. Earlier versions return the existing cluster identity without refreshing its OIDC issuer and JSON Web Key Set (JWKS). A cluster upgraded from 0.5.x can therefore retain an empty JWKS and the NVCA agent cannot authenticate.

Confirm the CLI version before starting the upgrade:

$
nvcf-cli version

Prepare the 0.6.0 configuration

Return to the extracted 0.6.0 stack before running any 0.6.0 sync command. Reconcile the site-specific settings from the 0.5.1 deployment into the 0.6.0 configuration files:

• environments/$HELMFILE_ENV.yaml
• secrets/$HELMFILE_ENV-secrets.yaml

Start from the files or templates included with 0.6.0 and carry forward the required values from 0.5.1. Do not replace the 0.6.0 files wholesale because the available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them.

Confirm that both files exist before continuing:

$
for file in \
>
  "environments/${HELMFILE_ENV}.yaml" \
>
  "secrets/${HELMFILE_ENV}-secrets.yaml"; do
$
  [ -f "$file" ] || { echo "missing required file: $file"; exit 1; }
$
done

Enable OpenBao issuer discovery for CSP clusters

Cloud service provider (CSP) clusters such as Amazon EKS use an external OpenID Connect (OIDC) issuer for Kubernetes service account tokens. Enable issuer discovery in environments/$HELMFILE_ENV.yaml so the OpenBao migration configures the external issuer’s JWKS URL:

1
openbao:
2
  migrations:
3
    issuerDiscovery:
4
      enabled: true

Enable NATS Gateway routing

NVCA connects to NATS through the external Gateway on TCP port 4222. A Gateway created with an earlier version of the quickstart might not have the required listener. Set the Gateway name and namespace used by your environment, label the route-owning namespace, and add the listener if it is missing:

$
export NATS_GATEWAY_NAME=nvcf-gateway
$
export NATS_GATEWAY_NAMESPACE=envoy-gateway
$
$
kubectl label namespace "$NATS_GATEWAY_NAMESPACE" nvcf/platform=true --overwrite
$
$
if [ "$(kubectl -n "$NATS_GATEWAY_NAMESPACE" get gateway "$NATS_GATEWAY_NAME" \
>
  -o jsonpath='{.spec.listeners[?(@.name=="nats")].name}')" != "nats" ]; then
$
  kubectl -n "$NATS_GATEWAY_NAMESPACE" patch gateway "$NATS_GATEWAY_NAME" \
>
    --type=json \
>
    -p='[{"op":"add","path":"/spec/listeners/-","value":{"name":"nats","protocol":"TCP","port":4222,"allowedRoutes":{"namespaces":{"from":"Selector","selector":{"matchLabels":{"nvcf/platform":"true"}}}}}}]'
$
fi

Enable the NATS route in environments/$HELMFILE_ENV.yaml. Use the same Gateway name and namespace as the preceding command:

1
ingress:
2
  gatewayApi:
3
    routes:
4
      nats:
5
        enabled: true
6
    gateways:
7
      nats:
8
        name: nvcf-gateway
9
        namespace: envoy-gateway

Sync the ingress release so it creates the NATS TCPRoute:

$
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=ingress

Confirm that the listener and route exist and that the Envoy service exposes port 4222 before continuing:

$
kubectl -n "$NATS_GATEWAY_NAMESPACE" get gateway "$NATS_GATEWAY_NAME" \
>
  -o jsonpath='{.spec.listeners[?(@.name=="nats")].port}{"\n"}'
$
kubectl -n "$NATS_GATEWAY_NAMESPACE" get tcproute nats
$
kubectl get service -n envoy-gateway-system \
>
  -l "gateway.envoyproxy.io/owning-gateway-name=$NATS_GATEWAY_NAME" \
>
  -o jsonpath='{.items[0].spec.ports[?(@.port==4222)].port}{"\n"}'

Both port checks must print 4222, and the nats TCPRoute must exist. Do not continue to the NVCA refresh if any check fails.

Stop 1: Cassandra schema migration

Before entering the maintenance window, record the storage class and size from the live Cassandra StatefulSet:

$
kubectl -n cassandra-system get statefulset cassandra \
>
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}'

Confirm that the effective global.storageClass and global.storageSize settings for HELMFILE_ENV match the live values. If they differ, stop and restore the settings from the original installation before continuing.

Quiesce the control plane so no service writes to the database while the migration runs:

$
for ns in nvcf api-keys ess sis; do
$
  kubectl -n "$ns" scale deployment --all --replicas=0
$
done

Sync only the Cassandra release from the 0.6.0 stack. This runs the schema migration:

$
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=cassandra

The migration runs as a Helm post-upgrade hook, so this command blocks until the migration Job completes and fails if it does not. A non-zero exit means the migration did not finish; resolve it before continuing. Confirm the migration Job succeeded:

$
kubectl get jobs -n cassandra-system
$
# the *-migrations Job should report COMPLETIONS 1/1

Stop 2: Spot Instance Service auth-client-id migration

Stop 2 must reach Spot Instance Service runtime 1.561.1. This is the released version of the patched Spot Instance Service runtime that enables the auth-client-id migration task for the self-hosted Spring profile. The task copies each existing cluster row’s ssa_client_id into auth_client_id. This runtime also enables the global NATS stream validation task for the self-hosted Spring profile.

Temporarily override the Spot Instance Service runtime image in environments/$HELMFILE_ENV.yaml:

1
sis:
2
  image:
3
    tag: 1.561.1

Sync only the Spot Instance Service release from the 0.6.0 stack. The make target loads the selected site environment and its matching secrets file:

$
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=sis

Confirm that the Spot Instance Service deployment uses runtime 1.561.1:

$
kubectl -n sis get deployment spot-instance-service \
>
  -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

Scale Spot Instance Service up and wait for it to become ready. Keep the other control-plane deployments stopped while the migration runs:

$
kubectl -n sis scale deployment/spot-instance-service --replicas=1
$
kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m

Enable the auth-client-id migration task on Spot Instance Service:

$
kubectl set env deployment/spot-instance-service -n sis \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_ENABLED=true \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_LOCK_TTL_IN_SECONDS=60 \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_INITIAL_DELAY_DURATION=PT30S \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_SCHEDULE_DURATION=PT1M
$
kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m

Wait at least two minutes for the scheduled task to run:

$
sleep 120

Inspect the cluster rows:

$
CASSANDRA_PASSWORD=$(kubectl get secret -n cassandra-system cassandra \
>
  -o jsonpath='{.data.cassandra-password}' | base64 --decode)
$
kubectl exec -n cassandra-system cassandra-0 -- \
>
  cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \
>
  -e "SELECT cluster_id, ssa_client_id, auth_client_id FROM sis_api.cluster_by_cluster_id;"

For every row with a non-empty ssa_client_id, auth_client_id must contain the same value. Do not continue if any eligible row has an empty or mismatched auth_client_id.

Enable Spot Instance Service NATS reconnect

The Spot Instance Service chart included with 0.6.0 does not enable NATS reconnect. Enable it on the live ConfigMap before the full-stack sync rolls NATS, then restart Spot Instance Service so the pod reads the new value:

$
kubectl patch configmap spot-instance-service-env -n sis --type merge \
>
  -p '{"data":{"NATS_RECONNECT_ALLOWED":"true"}}'
$
kubectl rollout restart deployment/spot-instance-service -n sis
$
kubectl rollout status deployment/spot-instance-service -n sis --timeout=10m

Confirm that reconnect is enabled:

$
test "$(kubectl -n sis get configmap spot-instance-service-env \
>
  -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true"

Do not continue if the command returns a non-zero exit code. A later Spot Instance Service chart release manages this setting directly, but the 0.6.0 upgrade requires this one-time patch.

Sync the full 0.6.0 stack

Keep the sis.image.tag: 1.561.1 override in place and sync the full 0.6.0 stack. This upgrades NATS while Spot Instance Service can recreate the global JetStream streams if the NATS upgrade removes them:

$
make install HELMFILE_ENV="$HELMFILE_ENV"

Confirm that Spot Instance Service remains ready after the full sync:

$
kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m

Confirm that the full-stack sync preserved the reconnect setting:

$
test "$(kubectl -n sis get configmap spot-instance-service-env \
>
  -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true" || {
>
  echo "Spot Instance Service NATS reconnect is not enabled; reapply the preceding patch" >&2
>
  exit 1
>
}

The Spot Instance Service validation task runs every three minutes. During the NATS roll, Spot Instance Service may log A JetStream context can't be established during close while its NATS client reconnects. This error is transient during reconnection. Wait up to ten minutes for both global streams to exist:

$
NATS_BOX_POD="$(kubectl -n nats-system get pod \
>
  -l app.kubernetes.io/component=nats-box \
>
  -o jsonpath='{.items[0].metadata.name}')"
$
$
for attempt in 0 1 2 3 4 5 6 7 8 9 10; do
$
  if kubectl -n nats-system exec "$NATS_BOX_POD" -- \
>
       nats --context default stream info TerminateNvcaStream >/dev/null && \
>
     kubectl -n nats-system exec "$NATS_BOX_POD" -- \
>
       nats --context default stream info CreateNvcaFunctionTaskStream >/dev/null; then
$
    break
$
  fi
$
$
  if [ "$attempt" -eq 10 ]; then
$
    echo "required global NATS streams were not recreated" >&2
$
    exit 1
$
  fi
$
  sleep 60
$
done
$
$
for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do
$
  kubectl -n nats-system exec "$NATS_BOX_POD" -- \
>
    nats --context default stream info "$stream"
$
done

Do not continue unless both stream information commands succeed.

If the streams are still absent after ten minutes, restart Spot Instance Service to reestablish its JetStream context, wait for the rollout, and then rerun the preceding stream check:

$
kubectl rollout restart deployment/spot-instance-service -n sis
$
kubectl rollout status deployment/spot-instance-service -n sis --timeout=10m

Stop 3: upgrade Spot Instance Service to the cleanup runtime

Remove the temporary sis.image.tag override from environments/$HELMFILE_ENV.yaml. Then sync the Spot Instance Service release again. The 0.6.0 Spot Instance Service chart deploys runtime 1.563.1. Helm preserves the migration-task environment variables because Stop 2 added them directly to the live Deployment. Remove them explicitly after the sync.

$
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=sis
$
kubectl set env deployment/spot-instance-service -n sis \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_ENABLED- \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_LOCK_TTL_IN_SECONDS- \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_INITIAL_DELAY_DURATION- \
>
  NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_SCHEDULE_DURATION-
$
kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m

Confirm that the cleanup-runtime sync preserved the reconnect setting:

$
test "$(kubectl -n sis get configmap spot-instance-service-env \
>
  -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true" || {
>
  echo "Spot Instance Service NATS reconnect is not enabled; reapply the preceding patch" >&2
>
  exit 1
>
}

Confirm that Spot Instance Service now uses runtime 1.563.1:

$
kubectl -n sis get deployment spot-instance-service \
>
  -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

Confirm that the temporary migration-task variables are no longer present. The output must not include variables whose names start with NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_:

$
kubectl set env deployment/spot-instance-service -n sis --list

Refresh the GPU cluster registration and NVCA operator

After the control plane is on 0.6.0, use the compute-plane stack bundle shipped for the 0.6.0 release to re-register each GPU cluster and reinstall the NVCA operator so the worker layer reconnects to the upgraded control plane with the migrated nvca auth-client identity. nvcf-default is the default cluster name; substitute your own where it applies. See Self-Managed Clusters for the full registration reference.

Download and extract the nvcf-compute-plane-stack bundle listed in the 0.6.0 artifact manifest. Run the remaining commands in this section from the extracted compute-plane stack directory.

Re-register the cluster with the upgraded control plane (idempotent):

$
make register-cluster HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default

1
icms_url: "http://<gateway-address>"
2
icms_host: "sis.<stack-domain>"

$
make register-cluster HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default \
>
  CLUSTER_REGION=<region> ICMS_URL=http://<gateway-address> \
>
  NVCF_CLI=<path-to-nvcf-cli> NVCF_CLI_CONFIG=<path-to-cli-config>

Install the compute-plane stack from the refreshed registration values, then wait for the NVCA operator to roll out:

$
make install HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default
$
kubectl -n nvca-operator rollout status deployment/nvca-operator --timeout=10m

Confirm the backend reconnects and reports healthy:

$
kubectl wait nvcfbackend nvcf-default -n nvca-operator \
>
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m

Recover an NVCA agent that returns HTTP 401

If an earlier registration attempt left the cluster JWKS empty, the NVCA agent can repeatedly fail to register with ICMS and return HTTP 401. Rotate the JWKS for the existing cluster instead of deleting its registration:

$
nvcf-cli --config <path-to-cli-config> cluster rotate \
>
  --cluster-id <cluster-id> \
>
  --kubeconfig <path-to-gpu-cluster-kubeconfig> \
>
  --icms-url http://<gateway-address> \
>
  --force

The command discovers the current OIDC issuer and JWKS from the GPU cluster and updates the existing ICMS row. It preserves the cluster identity. Confirm that the backend becomes healthy after the agent retries:

$
kubectl wait nvcfbackend nvcf-default -n nvca-operator \
>
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m