> This page is for version Development. > For other versions, use one of these documentation indexes: > - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt > - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt > - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. # Upgrade from NVCF 1.0.2 or 1.0.3 to NVCF 1.0.4 Use this procedure for NVCF 1.0.2 or the docs-only 1.0.3 edition. | | NVCF 1.0.2 or 1.0.3 | Target: NVCF 1.0.4 | | ------------- | ------------------- | ------------------ | | Self-Managed | 1.0.1 | 1.0.10 | | Compute Plane | 1.0.0 | 1.0.5 | | Observability | 1.0.0 | 1.0.0 | Single-replica Cassandra is not recommended. With one replica, the control-plane API is unavailable for ~1-2 minute while Cassandra restarts during the dependencies stage. A 10-15 minute user maintenance window is recommended for this upgrade procedure, due to required NATS, gRPC proxy and Invocation Service rollovers, which can impact in-flight invocations. HA-enabled service defaults are planned in the 1.1.0 and 1.2.0 NVCF releases which will improve upgrades. ## Prerequisites - Python 3, Helm 3.21.4, Helmfile 1.1.9, `kubectl`, `jq`, `curl`, Mike Farah's `yq` v4, and the [helm-diff plugin](/nvcf/dev/self-managed/helmfile-installation#required-tools-and-software). - The Python `cryptography` package, to verify JWT canary signatures against the mount's JWKS. - `nvcf-cli` 1.16.8 or later from the [NGC download instructions](/nvcf/dev/manifest/image-mirroring#downloading-nvcf-cli). - Download the NVCF 1.0.4 packaged release and manifest. Mirror the required artifacts to your configured registry; see [image mirroring](/nvcf/dev/manifest/image-mirroring). Include both task worker images if you use tasks and the compute-plane Samba image. The [upstream supporting images](/nvcf/dev/manifest/artifacts#supporting-images-that-ship-from-docker-hub) (NATS server/reloader/box and account-bootstrap/receipt helpers) need their own mirror settings; they do not inherit `global.image`. - cert-manager images default to `quay.io/jetstack` and also do not inherit `global.image`. For a private mirror, set each component's `image.repository`; see [upstream cert-manager images](/nvcf/dev/manifest/artifacts#use-upstream-cert-manager-images). - Extract the stacks under `deploy/stacks/self-managed`, `deploy/stacks/nvcf-compute-plane` and `deploy/stacks/observability` in a new bundle directory. ## Known issues in 1.0.2 - NVCA 3.10.0 can skip new deployments after a NATS restart. Fixed in Compute Plane 1.0.5. - Tasks select unpublished worker images. Fixed in Self-Managed 1.0.10. ## Before starting 1. Back up configuration, databases and persistent volumes. 2. Prepare the target [environment and secrets files](/nvcf/dev/self-managed/helmfile-installation). Transfer deployment settings and secrets from `environments/.yaml` and `secrets/-secrets.yaml`; keep the target chart/image versions. 3. Retain the compute-plane registration file and account ID. Do not run `register-cluster` again. 4. Locate the kubeconfig for each control-plane and compute cluster. Step 1 exports them as `CONTROL_KUBECONFIG` and `COMPUTE_KUBECONFIG`, and every later step uses those variables. In a new shell, export the step 1 values again before continuing. 5. Schedule a maintenance window based on a rehearsal of this procedure. For older source stacks, complete the [staged upgrade to Self-Managed 1.0.1](/nvcf/dev/release-notes/0-6-1-to-1-0-1-upgrade) first, including its schema and data-backfill stops. ## 1. Set deployment values Download these read-only scripts into one directory: Workload readiness check Receipt check JWT mount preservation check Use Bash with `set -euo pipefail` so a failed command stops the procedure. Fill in your existing deployment values below. Each kubeconfig must already select its intended context. For separate clusters, replace the two `COMPUTE_*` values; for a single cluster, keep them equal to the control plane. ```bash umask 077 export TARGET_BUNDLE="/absolute/path/to/stack-files" export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts" export HELMFILE_ENV="production" export CLUSTER_NAME="existing-gpu-cluster" export NCA_ID="existing-account-id" export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration" export CONTROL_KUBECONFIG="/secure/path/to/control-kubeconfig" export CONTROL_CONTEXT="existing-control-context" export COMPUTE_KUBECONFIG="$CONTROL_KUBECONFIG" export COMPUTE_CONTEXT="$CONTROL_CONTEXT" ``` `REGISTRATION_VALUES_DIR` contains `-register-values.yaml`. Set the stack paths and define the helper used below: ```bash export CONTROL_STACK="$TARGET_BUNDLE/deploy/stacks/self-managed" export COMPUTE_STACK="$TARGET_BUNDLE/deploy/stacks/nvcf-compute-plane" export KUBECONFIG="$CONTROL_KUBECONFIG" ``` ```bash control_helmfile() { (cd "$CONTROL_STACK" && HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \ --kubeconfig "$CONTROL_KUBECONFIG" "$@") } ``` ## 2. Refresh the hostname policy Apply this before the core stage changes the admin issuer route. Stop if the policy check fails; keep admission enabled. ```bash make -C "$CONTROL_STACK" apply-hostname-policy HELMFILE_ENV="$HELMFILE_ENV" \ KUBECONFIG_FILE="$CONTROL_KUBECONFIG" helm --kubeconfig "$CONTROL_KUBECONFIG" -n envoy-gateway-system status ingress kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A ``` Require deployed ingress and accepted routes before continuing. Routes for disabled optional add-ons are expected to report `ResolvedRefs=False` with reason `BackendNotFound`. For example, `llm-api-gateway` reports this when the LLM add-on is disabled. This does not block the upgrade if the route still reports `Accepted=True`. ## 3. Upgrade the existing compute plane Apply Compute Plane 1.0.5 on each GPU cluster with its retained registration and account ID. Use a separate output directory from the registration file: ```bash make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \ CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \ REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \ KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \ OUTPUT_DIR="$COMPUTE_STACK/out-upgrade" python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$COMPUTE_KUBECONFIG" "$COMPUTE_CONTEXT" nvca-operator kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-system get pods kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-operator get nvcfbackends ``` Wait for ready operator/agent pods and a healthy backend. Keep the existing Observability 1.0.0 configuration on separate compute clusters. ## 4. Drain pending work Pause new production requests and task submissions. Check every affected function/version before rolling NATS or dependent services. The checks in this step take under one minute, even for large clusters. The total drain time depends on your longest in-flight request or running task. Plan the maintenance window around that duration. Configure [CLI authentication](/nvcf/dev/overview/cli): `NVCF_API_KEY` needs `queue_details`; `NVCF_NVCT_API_KEY` needs `list_tasks`. Point `base_nvct_url` and `nvct_host` at this control plane's Tasks endpoint. ```bash NVCF_CLI="/absolute/path/to/nvcf-cli" NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml" FUNCTION_ID="existing-function-id" VERSION_ID="existing-version-id" "$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID" "$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING ``` Require queue size zero for every affected queue; a missing response blocks progress. Let running tasks finish; do not discard work to clear queues. Check single-replica memory streams on every NATS server too. Choose an unused local port for monitoring port 8222 and keep submissions paused. The temporary directory holds working files for this check and the OpenBao transition: ```bash umask 077 export UPGRADE_RECORD="$(mktemp -d)" NATS_MONITOR_PORT=18223 NATS_PODS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \ get pods -o json | jq -r '.items[] | select(any(.spec.containers[]; .name == "nats")) | .metadata.name') test -n "$NATS_PODS" mkdir "$UPGRADE_RECORD/nats-drain" for pod in $NATS_PODS; do ( kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \ port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \ > "$UPGRADE_RECORD/nats-drain/$pod.log" 2>&1 & forward_pid=$! trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT for attempt in {1..30}; do kill -0 "$forward_pid" if grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"; then break fi sleep 1 done grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log" curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \ "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \ > "$UPGRADE_RECORD/nats-drain/$pod.json" jq -e '.account_details | type == "array"' "$UPGRADE_RECORD/nats-drain/$pod.json" jq -e 'all(.account_details[]; (.stream_detail | type == "array") and all(.stream_detail[]; (.config.storage | type == "string") and (.state.messages | type == "number")))' "$UPGRADE_RECORD/nats-drain/$pod.json" ) done jq -se '[.[] | .account_details[]? | .stream_detail[]? | select(.config.storage == "memory" and (.config.num_replicas // 1) == 1) | {name: .name, messages: .state.messages}] | if length > 0 and all(.[]; .messages == 0) then . else error("Memory streams missing or not drained; do not roll NATS") end' \ "$UPGRADE_RECORD/nats-drain/"*.json ``` Require all single-replica memory streams to be present and empty. Failed reads, missing streams or nonzero counts block the rollout. If responses are paginated, retrieve every account/stream page before accepting the result. Use your configured monitoring endpoint if it differs. Keep NATS running while work drains. For stuck deployments or retained creation messages, follow the [read-only recovery steps](/nvcf/dev/self-managed/upgrade-recovery#inspect-retained-creation-messages). For a retry, use a new `nats-drain` directory; keep the OpenBao baseline. ## 5. Apply control-plane stages in order Keep submissions paused. Review each Helmfile diff before confirming the apply. Wait for each stage to succeed before continuing. ### OpenBao bridge before dependency migrations The target uses chart `0.32.9`, server/agent `2.6.3-nv-1.3.9` and migrations `0.19.11`. The bridge retains JWT plugin `v1.3.7` alongside `v1.4.1`. Existing mounts keep their plugin selection; new mounts use `v1.4.1`. Existing mounts receive its fixes through a separate [plugin migration](/nvcf/dev/self-managed/open-bao-jwt-plugin-upgrade#pin-rotate-then-upgrade). Before changing OpenBao: 1. Complete the [snapshot and health checks](/nvcf/dev/self-managed/open-bao-patch-upgrade#before-you-start) and define `check_openbao`. Require three healthy, unsealed members on the same architecture, the original Raft cluster ID and retained unseal material. The active member is the pod whose `check_openbao` output reports `is_self: true`. 2. Follow the [JWT preflight](/nvcf/dev/self-managed/open-bao-jwt-plugin-upgrade#preflight) to define `bao_admin`. Set `BAO_POD` to the active member from the previous item. Use the control-plane `KUBECONFIG` and verify a working JWT canary. For the preflight `BRIDGE_IMAGE`, use the target server image from your configured registry, for example `nvcr.io/nvidia/nvcf/nvcf-openbao:2.6.3-nv-1.3.9`. 3. Pause external mount creation and tuning. Capture the baseline once: ```bash (set -o noclobber; bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-before.json") python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" "$UPGRADE_RECORD/jwt-before.json" ``` Stop on missing metadata or an incomplete plugin transition. The bridge hook must match the retained binary to the stored catalog SHA. On mismatch, obtain a qualified bridge with the matching binary; do not rewrite the SHA or bypass hooks. Set bridge mode and apply only OpenBao, leaving NATS running: ```bash export CONTROL_VALUES="$CONTROL_STACK/environments/$HELMFILE_ENV.yaml" yq -i '.openbao.jwtPluginLifecycle.mode = "bridge"' "$CONTROL_VALUES" control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \ --selector name=openbao-server diff --suppress-secrets --skip-needs control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \ --selector name=openbao-server apply --skip-needs ``` Bridge mode pauses migrations and catalog refresh. `OnDelete` leaves server pods on the old image until you replace them. Follow the [standby replacement and leadership transfer](/nvcf/dev/self-managed/open-bao-patch-upgrade#replace-standbys-one-at-a-time) for every member, one at a time. Before the next replacement, verify unsealed health, the original cluster ID, the selected bridge digest and JWT issuance. Update `BAO_POD` after leadership changes. With `OnDelete`, the StatefulSet can keep reporting the old `currentRevision` after every member is replaced. Confirm the rollout with `updatedReplicas` equal to `replicas` instead: ```bash kubectl -n vault-system get statefulset openbao-server \ -o jsonpath='{.status.updatedReplicas}/{.spec.replicas}{"\n"}' ``` Once all members run the bridge, check retained mount versions/SHAs and enable migrations. New mounts must use the target catalog entry: ```bash check_upgrade_jwt_mounts() { check_openbao bao_admin jwt-plugin-lifecycle preflight bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-current.json" bao_admin sh -ec '. /openbao/jwt-plugin-versions.env bao plugin info -format=json -version="$JWT_PLUGIN_TARGET_VERSION" secret vault-plugin-secrets-jwt' \ > "$UPGRADE_RECORD/jwt-target-catalog.json" python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" \ "$UPGRADE_RECORD/jwt-before.json" "$UPGRADE_RECORD/jwt-current.json" \ "$UPGRADE_RECORD/jwt-target-catalog.json" } check_upgrade_jwt_mounts yq -i '.openbao.jwtPluginLifecycle.mode = "versioned"' "$CONTROL_VALUES" control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \ --selector name=openbao-server diff --suppress-secrets --skip-needs control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \ --selector name=openbao-server apply --skip-needs check_upgrade_jwt_mounts ``` Keep `versioned` mode for subsequent applies. It enables migrations and new mounts without retuning existing ones. Verify existing and new secret consumers, including JWT issuance and signature checks after normal failover. Retain both binaries and catalog entries. After a shell failure, redefine `check_openbao`, `bao_admin` and `check_upgrade_jwt_mounts`, then reselect `BAO_POD`. Keep the original baseline. ### Dependencies and migrations Proceed only after the OpenBao checks pass. Use this staged apply instead of the patch runbook's whole-stack `make apply`: ```bash control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status for namespace in nats-system cert-manager vault-system cassandra-system; do python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace" kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n "$namespace" get jobs done check_upgrade_jwt_mounts ``` Require successful dependency releases and migration jobs before continuing. ### Core services ```bash control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply control_helmfile -f helmfile.d/02-core.yaml.gotmpl status for namespace in api-keys sis nvcf ess nats-system; do python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace" done kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A ``` Require deployed releases, ready workloads and accepted routes, including any additional enabled addon namespaces. ### Upgrade receipt After all stages pass, write and check the receipt: ```bash export EXPECTED_CONTROL_STACK_VERSION="1.0.10" control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \ get job upgrade-receipt --ignore-not-found kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \ get configmap nvcf-upgrade-receipt -o yaml python3 "$UPGRADE_SCRIPTS/check-upgrade-receipt.py" ``` Require the receipt to match the selected Self-Managed version. The ConfigMap persists after the hook job expires. To correct a stale receipt, verify the installed package and workloads, then rerun that package's receipt stage; do not patch the ConfigMap. ## 6. Resume workloads Invoke an existing function with its known request and expected response: ```bash CANARY_REQUEST='{"input":"your-known-canary-input"}' "$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \ --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \ --request-body "$CANARY_REQUEST" --timeout 120 ``` For LLM functions, use the [LLM invocation options](/nvcf/dev/overview/cli#invoke-function). Check secret access, then resume submissions and monitor queue progress. Existing workers keep their images until redeployed; stack defaults apply to new workers unless their deployment specifies an image override. ## Rollback boundary Stop on failure and follow your reviewed recovery plan: - Retain patched NVCA, target Tasks and the corrected hostname policy. Do not apply an old bundle unchanged. - For an old core apply, exclude `ingress` and `nvct-api` with `--selector 'name!=ingress,name!=nvct-api' --skip-needs`. Old Tasks settings restore unpublished worker images. - Keep the OpenBao bridge image and both plugins. New mounts may reference commands absent from old images; plugin rollback/cleanup belongs to the separate plugin migration procedure. - Verify original identities, existing workloads and durable data after recovery. If you run tasks, launch a new task before ending recovery. Application recovery does not validate server/agent downgrades or reverse stored data. Use the [OpenBao recovery boundary](/nvcf/dev/self-managed/open-bao-patch-upgrade#verify-servers-and-agents) and your datastore recovery plan for restores.