> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server. # 0.5.0 to 0.6.0 Upgrade The upgrade from 0.5.0 to 0.6.0 crosses two scheduled data migrations and one required Spot Instance Service cleanup boundary. Run Stops 1 and 2, sync the full 0.6.0 stack while the Spot Instance Service migration runtime can restore the global NATS streams, and then run Stop 3. These steps assume 0.5.0 is already deployed and healthy, that you have downloaded the 0.6.0 stack, and that you run `make` from the extracted `nvcf-self-managed-stack/` directory with your environment set. ```bash export HELMFILE_ENV= # for example eks-cds-qa ``` ## Prerequisite: upgrade to 0.5.1 or later first If you pull NVCF images from a private registry (for example NGC `nvcr.io`) and rely on `global.imagePullSecrets` to deliver the pull secret, move to 0.5.1 or later before you start this procedure. Release 0.5.0 does not propagate `global.imagePullSecrets` to the chart pods, so private migration and sidecar images (for example `nvcf-cassandra-migrations` and `nats-box`) fail to pull with `ImagePullBackOff` / `403`, and a clean 0.5.0 install never becomes healthy. 0.5.1 adds that propagation. If you already run 0.5.1 or later, skip this section. Moving from 0.5.0 to 0.5.1 is a normal patch sync, not a migration stop. Download the 0.5.1 stack and sync it with your existing environment: ```bash # Create the pull secret in every NVCF namespace (private registry only). # The 0.5.1 charts reference this secret by name; without it there is nothing to propagate. kubectl create namespace cert-manager \ --dry-run=client -o yaml | kubectl apply -f - for ns in cassandra-system nats-system nvcf api-keys ess sis vault-system nvca-operator nvca-system nvcf-backend cert-manager; do kubectl create secret docker-registry nvcr-pull-secret \ --docker-server= --docker-username='' --docker-password='' \ -n "$ns" --dry-run=client -o yaml | kubectl apply -f - done # Confirm the env references the secret so every chart inherits it. # environments/.yaml should contain: # global: # imagePullSecrets: # - name: nvcr-pull-secret # Sync the full 0.5.1 stack so every chart re-renders with imagePullSecrets set. make install HELMFILE_ENV="$HELMFILE_ENV" # Confirm the control plane is healthy before starting the 0.6.0 upgrade. # This should print nothing; any output is a pod that is not yet Running/Completed. kubectl get pods -A | grep -vE 'Running|Completed|READY' ``` ## Prerequisite: install nvcf-cli 1.8.3 or later Use `nvcf-cli` 1.8.3 or later when you re-register GPU clusters after the control-plane upgrade. Earlier versions return the existing cluster identity without refreshing its OIDC issuer and JSON Web Key Set (JWKS). A cluster upgraded from 0.5.x can therefore retain an empty JWKS and the NVCA agent cannot authenticate. Confirm the CLI version before starting the upgrade: ```bash nvcf-cli version ``` ## Prepare the 0.6.0 configuration Return to the extracted 0.6.0 stack before running any 0.6.0 sync command. Reconcile the site-specific settings from the 0.5.1 deployment into the 0.6.0 configuration files: - `environments/$HELMFILE_ENV.yaml` - `secrets/$HELMFILE_ENV-secrets.yaml` Start from the files or templates included with 0.6.0 and carry forward the required values from 0.5.1. Do not replace the 0.6.0 files wholesale because the available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them. Confirm that both files exist before continuing: ```bash for file in \ "environments/${HELMFILE_ENV}.yaml" \ "secrets/${HELMFILE_ENV}-secrets.yaml"; do [ -f "$file" ] || { echo "missing required file: $file"; exit 1; } done ``` ### Enable OpenBao issuer discovery for CSP clusters Cloud service provider (CSP) clusters such as Amazon EKS use an external OpenID Connect (OIDC) issuer for Kubernetes service account tokens. Enable issuer discovery in `environments/$HELMFILE_ENV.yaml` so the OpenBao migration configures the external issuer's JWKS URL: ```yaml openbao: migrations: issuerDiscovery: enabled: true ``` Do not continue with the 0.6.0 sync on a CSP cluster while issuer discovery is disabled. The OpenBao migration otherwise falls back to a Kubernetes API server public key that cannot validate tokens signed by an external CSP issuer. New vault-agent logins then fail and prevent service pods from starting. ### Enable NATS Gateway routing NVCA connects to NATS through the external Gateway on TCP port `4222`. A Gateway created with an earlier version of the quickstart might not have the required listener. Set the Gateway name and namespace used by your environment, label the route-owning namespace, and add the listener if it is missing: ```bash export NATS_GATEWAY_NAME=nvcf-gateway export NATS_GATEWAY_NAMESPACE=envoy-gateway kubectl label namespace "$NATS_GATEWAY_NAMESPACE" nvcf/platform=true --overwrite if [ "$(kubectl -n "$NATS_GATEWAY_NAMESPACE" get gateway "$NATS_GATEWAY_NAME" \ -o jsonpath='{.spec.listeners[?(@.name=="nats")].name}')" != "nats" ]; then kubectl -n "$NATS_GATEWAY_NAMESPACE" patch gateway "$NATS_GATEWAY_NAME" \ --type=json \ -p='[{"op":"add","path":"/spec/listeners/-","value":{"name":"nats","protocol":"TCP","port":4222,"allowedRoutes":{"namespaces":{"from":"Selector","selector":{"matchLabels":{"nvcf/platform":"true"}}}}}}]' fi ``` Enable the NATS route in `environments/$HELMFILE_ENV.yaml`. Use the same Gateway name and namespace as the preceding command: ```yaml ingress: gatewayApi: routes: nats: enabled: true gateways: nats: name: nvcf-gateway namespace: envoy-gateway ``` Sync the ingress release so it creates the NATS `TCPRoute`: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=ingress ``` Confirm that the listener and route exist and that the Envoy service exposes port `4222` before continuing: ```bash kubectl -n "$NATS_GATEWAY_NAMESPACE" get gateway "$NATS_GATEWAY_NAME" \ -o jsonpath='{.spec.listeners[?(@.name=="nats")].port}{"\n"}' kubectl -n "$NATS_GATEWAY_NAMESPACE" get tcproute nats kubectl get service -n envoy-gateway-system \ -l "gateway.envoyproxy.io/owning-gateway-name=$NATS_GATEWAY_NAME" \ -o jsonpath='{.items[0].spec.ports[?(@.port==4222)].port}{"\n"}' ``` Both port checks must print `4222`, and the `nats` `TCPRoute` must exist. Do not continue to the NVCA refresh if any check fails. ## Stop 1: Cassandra schema migration Keep the Cassandra storage settings unchanged during this upgrade. `global.storageClass` and `global.storageSize` in the upgrade environment must produce the same values as the original installation. These settings render into the Cassandra StatefulSet `volumeClaimTemplates`, which Kubernetes treats as immutable. Changing either value, including introducing a value that was previously unset, causes the Cassandra Helm upgrade to fail. Do not use this upgrade procedure to resize Cassandra storage or change its storage class. Before entering the maintenance window, record the storage class and size from the live Cassandra StatefulSet: ```bash kubectl -n cassandra-system get statefulset cassandra \ -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{": storageClass="}{.spec.storageClassName}{", size="}{.spec.resources.requests.storage}{"\n"}{end}' ``` Confirm that the effective `global.storageClass` and `global.storageSize` settings for `HELMFILE_ENV` match the live values. If they differ, stop and restore the settings from the original installation before continuing. Quiesce the control plane so no service writes to the database while the migration runs: ```bash for ns in nvcf api-keys ess sis; do kubectl -n "$ns" scale deployment --all --replicas=0 done ``` Sync only the Cassandra release from the 0.6.0 stack. This runs the schema migration: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=cassandra ``` The migration runs as a Helm `post-upgrade` hook, so this command blocks until the migration Job completes and fails if it does not. A non-zero exit means the migration did not finish; resolve it before continuing. Confirm the migration Job succeeded: ```bash kubectl get jobs -n cassandra-system # the *-migrations Job should report COMPLETIONS 1/1 ``` ## Stop 2: Spot Instance Service auth-client-id migration Stop 2 must reach Spot Instance Service runtime `1.561.1`. This is the released version of the patched Spot Instance Service runtime that enables the auth-client-id migration task for the self-hosted Spring profile. The task copies each existing cluster row's `ssa_client_id` into `auth_client_id`. This runtime also enables the global NATS stream validation task for the self-hosted Spring profile. Do not deploy Spot Instance Service runtime `1.562.2` or later until this migration is complete. Those versions remove the migration task and expect `auth_client_id` to already be populated. Temporarily override the Spot Instance Service runtime image in `environments/$HELMFILE_ENV.yaml`: ```yaml sis: image: tag: 1.561.1 ``` Sync only the Spot Instance Service release from the 0.6.0 stack. The `make` target loads the selected site environment and its matching secrets file: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=sis ``` Confirm that the Spot Instance Service deployment uses runtime `1.561.1`: ```bash kubectl -n sis get deployment spot-instance-service \ -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' ``` Scale Spot Instance Service up and wait for it to become ready. Keep the other control-plane deployments stopped while the migration runs: ```bash kubectl -n sis scale deployment/spot-instance-service --replicas=1 kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m ``` Enable the auth-client-id migration task on Spot Instance Service: Spot Instance Service already runs with the self-hosted Spring profile set by its chart. Do not change `SPRING_PROFILES_ACTIVE`. The command below only adds the migration-task variables and leaves the existing profile untouched. ```bash kubectl set env deployment/spot-instance-service -n sis \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_ENABLED=true \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_LOCK_TTL_IN_SECONDS=60 \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_INITIAL_DELAY_DURATION=PT30S \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_SCHEDULE_DURATION=PT1M kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m ``` Wait at least two minutes for the scheduled task to run: ```bash sleep 120 ``` Inspect the cluster rows: ```bash CASSANDRA_PASSWORD=$(kubectl get secret -n cassandra-system cassandra \ -o jsonpath='{.data.cassandra-password}' | base64 --decode) kubectl exec -n cassandra-system cassandra-0 -- \ cqlsh -u cassandra -p "$CASSANDRA_PASSWORD" \ -e "SELECT cluster_id, ssa_client_id, auth_client_id FROM sis_api.cluster_by_cluster_id;" ``` For every row with a non-empty `ssa_client_id`, `auth_client_id` must contain the same value. Do not continue if any eligible row has an empty or mismatched `auth_client_id`. ### Enable Spot Instance Service NATS reconnect The Spot Instance Service chart included with 0.6.0 does not enable NATS reconnect. Enable it on the live ConfigMap before the full-stack sync rolls NATS, then restart Spot Instance Service so the pod reads the new value: ```bash kubectl patch configmap spot-instance-service-env -n sis --type merge \ -p '{"data":{"NATS_RECONNECT_ALLOWED":"true"}}' kubectl rollout restart deployment/spot-instance-service -n sis kubectl rollout status deployment/spot-instance-service -n sis --timeout=10m ``` Confirm that reconnect is enabled: ```bash test "$(kubectl -n sis get configmap spot-instance-service-env \ -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true" ``` Do not continue if the command returns a non-zero exit code. A later Spot Instance Service chart release manages this setting directly, but the 0.6.0 upgrade requires this one-time patch. ## Sync the full 0.6.0 stack Keep the `sis.image.tag: 1.561.1` override in place and sync the full 0.6.0 stack. This upgrades NATS while Spot Instance Service can recreate the global JetStream streams if the NATS upgrade removes them: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" ``` Confirm that Spot Instance Service remains ready after the full sync: ```bash kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m ``` Confirm that the full-stack sync preserved the reconnect setting: ```bash test "$(kubectl -n sis get configmap spot-instance-service-env \ -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true" || { echo "Spot Instance Service NATS reconnect is not enabled; reapply the preceding patch" >&2 exit 1 } ``` The Spot Instance Service validation task runs every three minutes. During the NATS roll, Spot Instance Service may log `A JetStream context can't be established during close` while its NATS client reconnects. This error is transient during reconnection. Wait up to ten minutes for both global streams to exist: ```bash NATS_BOX_POD="$(kubectl -n nats-system get pod \ -l app.kubernetes.io/component=nats-box \ -o jsonpath='{.items[0].metadata.name}')" for attempt in 0 1 2 3 4 5 6 7 8 9 10; do if kubectl -n nats-system exec "$NATS_BOX_POD" -- \ nats --context default stream info TerminateNvcaStream >/dev/null && \ kubectl -n nats-system exec "$NATS_BOX_POD" -- \ nats --context default stream info CreateNvcaFunctionTaskStream >/dev/null; then break fi if [ "$attempt" -eq 10 ]; then echo "required global NATS streams were not recreated" >&2 exit 1 fi sleep 60 done for stream in TerminateNvcaStream CreateNvcaFunctionTaskStream; do kubectl -n nats-system exec "$NATS_BOX_POD" -- \ nats --context default stream info "$stream" done ``` Do not continue unless both stream information commands succeed. If the streams are still absent after ten minutes, restart Spot Instance Service to reestablish its JetStream context, wait for the rollout, and then rerun the preceding stream check: ```bash kubectl rollout restart deployment/spot-instance-service -n sis kubectl rollout status deployment/spot-instance-service -n sis --timeout=10m ``` ## Stop 3: upgrade Spot Instance Service to the cleanup runtime Remove the temporary `sis.image.tag` override from `environments/$HELMFILE_ENV.yaml`. Then sync the Spot Instance Service release again. The 0.6.0 Spot Instance Service chart deploys runtime `1.563.1`. Helm preserves the migration-task environment variables because Stop 2 added them directly to the live Deployment. Remove them explicitly after the sync. ```bash make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=sis kubectl set env deployment/spot-instance-service -n sis \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_ENABLED- \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_LOCK_TTL_IN_SECONDS- \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_INITIAL_DELAY_DURATION- \ NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_SCHEDULE_DURATION- kubectl -n sis rollout status deployment/spot-instance-service --timeout=10m ``` Confirm that the cleanup-runtime sync preserved the reconnect setting: ```bash test "$(kubectl -n sis get configmap spot-instance-service-env \ -o jsonpath='{.data.NATS_RECONNECT_ALLOWED}')" = "true" || { echo "Spot Instance Service NATS reconnect is not enabled; reapply the preceding patch" >&2 exit 1 } ``` Confirm that Spot Instance Service now uses runtime `1.563.1`: ```bash kubectl -n sis get deployment spot-instance-service \ -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' ``` Confirm that the temporary migration-task variables are no longer present. The output must not include variables whose names start with `NVCA_AUTH_CLIENT_ID_MIGRATION_TASK_`: ```bash kubectl set env deployment/spot-instance-service -n sis --list ``` Spot Instance Service runtime `1.563.1` does not run periodic global NATS stream validation under the self-hosted Spring profile. If either global stream is lost after Stop 3, temporarily restore the `1.561.1` image override, sync Spot Instance Service, repeat the stream checks above, remove the override, and sync Spot Instance Service back to `1.563.1`. Do not refresh the NVCA operator while either stream is missing. ## Refresh the GPU cluster registration and NVCA operator After the control plane is on 0.6.0, use the compute-plane stack bundle shipped for the 0.6.0 release to re-register each GPU cluster and reinstall the NVCA operator so the worker layer reconnects to the upgraded control plane with the migrated `nvca` auth-client identity. `nvcf-default` is the default cluster name; substitute your own where it applies. See [Self-Managed Clusters](/nvcf/self-managed-clusters) for the full registration reference. Download and extract the `nvcf-compute-plane-stack` bundle listed in the 0.6.0 artifact manifest. Run the remaining commands in this section from the extracted compute-plane stack directory. Do not delete the existing cluster registration. `nvcf-cli` 1.8.3 or later refreshes the OIDC issuer and JWKS while preserving the existing `clusterID` and `clusterGroupID`. Deleting the registration creates a new cluster identity and can orphan existing function deployments. Re-register the cluster with the upgraded control plane (idempotent): ```bash make register-cluster HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default ``` `make register-cluster` defaults to local development values (`CLUSTER_REGION=us-west-1`, `ICMS_URL=http://sis.localhost:8080`) and a sibling CLI checkout. On EKS or any other CSP, pass your real region, the control-plane ingress address, and your CLI build and config so registration reaches the right ICMS and writes correct values: Set the ICMS URL and Host header in the CLI configuration. Replace both placeholders with resolved values. Static YAML does not expand shell variables such as `$GATEWAY_ADDR`. ```yaml icms_url: "http://" icms_host: "sis." ``` ```bash make register-cluster HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default \ CLUSTER_REGION= ICMS_URL=http:// \ NVCF_CLI= NVCF_CLI_CONFIG= ``` Install the compute-plane stack from the refreshed registration values, then wait for the NVCA operator to roll out: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME=nvcf-default kubectl -n nvca-operator rollout status deployment/nvca-operator --timeout=10m ``` Confirm the backend reconnects and reports healthy: ```bash kubectl wait nvcfbackend nvcf-default -n nvca-operator \ --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m ``` ### Recover an NVCA agent that returns HTTP 401 If an earlier registration attempt left the cluster JWKS empty, the NVCA agent can repeatedly fail to register with ICMS and return HTTP 401. Rotate the JWKS for the existing cluster instead of deleting its registration: ```bash nvcf-cli --config cluster rotate \ --cluster-id \ --kubeconfig \ --icms-url http:// \ --force ``` The command discovers the current OIDC issuer and JWKS from the GPU cluster and updates the existing ICMS row. It preserves the cluster identity. Confirm that the backend becomes healthy after the agent retries: ```bash kubectl wait nvcfbackend nvcf-default -n nvca-operator \ --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m ```