Upgrade from NVCF 1.0.2 or 1.0.3 to NVCF 1.0.4

Use this procedure for NVCF 1.0.2 or the docs-only 1.0.3 edition.

Prerequisites

• Python 3, Helm 3.21.4, Helmfile 1.1.9, kubectl, jq, curl, Mike Farah’s yq v4, and the helm-diff plugin.
• The Python cryptography package, to verify JWT canary signatures against the mount’s JWKS.
• nvcf-cli 1.16.8 or later from the NGC download instructions.
• Download the NVCF 1.0.4 packaged release and manifest. Mirror the required artifacts to your configured registry; see image mirroring. Include both task worker images if you use tasks and the compute-plane Samba image. The upstream supporting images (NATS server/reloader/box and account-bootstrap/receipt helpers) need their own mirror settings; they do not inherit global.image.
• cert-manager images default to quay.io/jetstack and also do not inherit global.image. For a private mirror, set each component’s image.repository; see upstream cert-manager images.
• Extract the stacks under deploy/stacks/self-managed, deploy/stacks/nvcf-compute-plane and deploy/stacks/observability in a new bundle directory.

Known issues in 1.0.2

• NVCA 3.10.0 can skip new deployments after a NATS restart. Fixed in Compute Plane 1.0.5.
• Tasks select unpublished worker images. Fixed in Self-Managed 1.0.10.

Before starting

1. Back up configuration, databases and persistent volumes.
2. Prepare the target environment and secrets files. Transfer deployment settings and secrets from environments/<name>.yaml and secrets/<name>-secrets.yaml; keep the target chart/image versions.
3. Retain the compute-plane registration file and account ID. Do not run register-cluster again.
4. Locate the kubeconfig for each control-plane and compute cluster. Step 1 exports them as CONTROL_KUBECONFIG and COMPUTE_KUBECONFIG, and every later step uses those variables. In a new shell, export the step 1 values again before continuing.
5. Schedule a maintenance window based on a rehearsal of this procedure.

1. Set deployment values

Download these read-only scripts into one directory:

Use Bash with set -euo pipefail so a failed command stops the procedure. Fill in your existing deployment values below. Each kubeconfig must already select its intended context. For separate clusters, replace the two COMPUTE_* values; for a single cluster, keep them equal to the control plane.

umask 077
export TARGET_BUNDLE="/absolute/path/to/stack-files"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export HELMFILE_ENV="production"
export CLUSTER_NAME="existing-gpu-cluster"
export NCA_ID="existing-account-id"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export CONTROL_KUBECONFIG="/secure/path/to/control-kubeconfig"
export CONTROL_CONTEXT="existing-control-context"
export COMPUTE_KUBECONFIG="$CONTROL_KUBECONFIG"
export COMPUTE_CONTEXT="$CONTROL_CONTEXT"

REGISTRATION_VALUES_DIR contains <cluster-name>-register-values.yaml. Set the stack paths and define the helper used below:

export CONTROL_STACK="$TARGET_BUNDLE/deploy/stacks/self-managed"
export COMPUTE_STACK="$TARGET_BUNDLE/deploy/stacks/nvcf-compute-plane"
export KUBECONFIG="$CONTROL_KUBECONFIG"

control_helmfile() {
  (cd "$CONTROL_STACK" &&
    HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
      --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}

2. Refresh the hostname policy

Apply this before the core stage changes the admin issuer route. Stop if the policy check fails; keep admission enabled.

make -C "$CONTROL_STACK" apply-hostname-policy HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
helm --kubeconfig "$CONTROL_KUBECONFIG" -n envoy-gateway-system status ingress
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A

Require deployed ingress and accepted routes before continuing.

3. Upgrade the existing compute plane

Apply Compute Plane 1.0.5 on each GPU cluster with its retained registration and account ID. Use a separate output directory from the registration file:

make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$COMPUTE_STACK/out-upgrade"
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$COMPUTE_KUBECONFIG" "$COMPUTE_CONTEXT" nvca-operator
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-system get pods
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-operator get nvcfbackends

Wait for ready operator/agent pods and a healthy backend. Keep the existing Observability 1.0.0 configuration on separate compute clusters.

4. Drain pending work

Pause new production requests and task submissions. Check every affected function/version before rolling NATS or dependent services.

Configure CLI authentication: NVCF_API_KEY needs queue_details; NVCF_NVCT_API_KEY needs list_tasks. Point base_nvct_url and nvct_host at this control plane’s Tasks endpoint.

NVCF_CLI="/absolute/path/to/nvcf-cli"
NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
FUNCTION_ID="existing-function-id"
VERSION_ID="existing-version-id"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Require queue size zero for every affected queue; a missing response blocks progress. Let running tasks finish; do not discard work to clear queues.

Check single-replica memory streams on every NATS server too. Choose an unused local port for monitoring port 8222 and keep submissions paused. The temporary directory holds working files for this check and the OpenBao transition:

umask 077
export UPGRADE_RECORD="$(mktemp -d)"
NATS_MONITOR_PORT=18223
NATS_PODS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
  get pods -o json | jq -r '.items[] |
    select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
mkdir "$UPGRADE_RECORD/nats-drain"
for pod in $NATS_PODS; do
  (
    kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
      port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
      > "$UPGRADE_RECORD/nats-drain/$pod.log" 2>&1 &
    forward_pid=$!
    trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
    for attempt in {1..30}; do
      kill -0 "$forward_pid"
      if grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"; then
        break
      fi
      sleep 1
    done
    grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"
    curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
      "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
      > "$UPGRADE_RECORD/nats-drain/$pod.json"
    jq -e '.account_details | type == "array"' "$UPGRADE_RECORD/nats-drain/$pod.json"
    jq -e 'all(.account_details[];
      (.stream_detail | type == "array") and
      all(.stream_detail[];
        (.config.storage | type == "string") and
        (.state.messages | type == "number")))' "$UPGRADE_RECORD/nats-drain/$pod.json"
  )
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
  select(.config.storage == "memory" and (.config.num_replicas // 1) == 1) |
  {name: .name, messages: .state.messages}] |
  if length > 0 and all(.[]; .messages == 0)
  then . else error("Memory streams missing or not drained; do not roll NATS") end' \
  "$UPGRADE_RECORD/nats-drain/"*.json

Require all single-replica memory streams to be present and empty. Failed reads, missing streams or nonzero counts block the rollout. If responses are paginated, retrieve every account/stream page before accepting the result. Use your configured monitoring endpoint if it differs.

Keep NATS running while work drains. For stuck deployments or retained creation messages, follow the read-only recovery steps. For a retry, use a new nats-drain directory; keep the OpenBao baseline.

5. Apply control-plane stages in order

Keep submissions paused. Review each Helmfile diff before confirming the apply. Wait for each stage to succeed before continuing.

OpenBao bridge before dependency migrations

The target uses chart 0.32.9, server/agent 2.6.3-nv-1.3.9 and migrations 0.19.11. The bridge retains JWT plugin v1.3.7 alongside v1.4.1. Existing mounts keep their plugin selection; new mounts use v1.4.1. Existing mounts receive its fixes through a separate plugin migration.

Before changing OpenBao:

1. Complete the snapshot and health checks and define check_openbao. Require three healthy, unsealed members on the same architecture, the original Raft cluster ID and retained unseal material. The active member is the pod whose check_openbao output reports is_self: true.
2. Follow the JWT preflight to define bao_admin. Set BAO_POD to the active member from the previous item. Use the control-plane KUBECONFIG and verify a working JWT canary. For the preflight BRIDGE_IMAGE, use the target server image from your configured registry, for example nvcr.io/nvidia/nvcf/nvcf-openbao:2.6.3-nv-1.3.9.
3. Pause external mount creation and tuning. Capture the baseline once:

(set -o noclobber; bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-before.json")
python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" "$UPGRADE_RECORD/jwt-before.json"

Stop on missing metadata or an incomplete plugin transition. The bridge hook must match the retained binary to the stored catalog SHA. On mismatch, obtain a qualified bridge with the matching binary; do not rewrite the SHA or bypass hooks.

Set bridge mode and apply only OpenBao, leaving NATS running:

export CONTROL_VALUES="$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
yq -i '.openbao.jwtPluginLifecycle.mode = "bridge"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server diff --suppress-secrets --skip-needs
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server apply --skip-needs

Bridge mode pauses migrations and catalog refresh. OnDelete leaves server pods on the old image until you replace them.

Follow the standby replacement and leadership transfer for every member, one at a time. Before the next replacement, verify unsealed health, the original cluster ID, the selected bridge digest and JWT issuance. Update BAO_POD after leadership changes.

With OnDelete, the StatefulSet can keep reporting the old currentRevision after every member is replaced. Confirm the rollout with updatedReplicas equal to replicas instead:

kubectl -n vault-system get statefulset openbao-server \
  -o jsonpath='{.status.updatedReplicas}/{.spec.replicas}{"\n"}'

Once all members run the bridge, check retained mount versions/SHAs and enable migrations. New mounts must use the target catalog entry:

check_upgrade_jwt_mounts() {
  check_openbao
  bao_admin jwt-plugin-lifecycle preflight
  bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-current.json"
  bao_admin sh -ec '. /openbao/jwt-plugin-versions.env
    bao plugin info -format=json -version="$JWT_PLUGIN_TARGET_VERSION" secret vault-plugin-secrets-jwt' \
    > "$UPGRADE_RECORD/jwt-target-catalog.json"
  python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" \
    "$UPGRADE_RECORD/jwt-before.json" "$UPGRADE_RECORD/jwt-current.json" \
    "$UPGRADE_RECORD/jwt-target-catalog.json"
}
check_upgrade_jwt_mounts
yq -i '.openbao.jwtPluginLifecycle.mode = "versioned"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server diff --suppress-secrets --skip-needs
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server apply --skip-needs
check_upgrade_jwt_mounts

Keep versioned mode for subsequent applies. It enables migrations and new mounts without retuning existing ones. Verify existing and new secret consumers, including JWT issuance and signature checks after normal failover. Retain both binaries and catalog entries.

After a shell failure, redefine check_openbao, bao_admin and check_upgrade_jwt_mounts, then reselect BAO_POD. Keep the original baseline.

Dependencies and migrations

Proceed only after the OpenBao checks pass. Use this staged apply instead of the patch runbook’s whole-stack make apply:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
  python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n "$namespace" get jobs
done
check_upgrade_jwt_mounts

Require successful dependency releases and migration jobs before continuing.

Core services

control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
  python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
done
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A

Require deployed releases, ready workloads and accepted routes, including any additional enabled addon namespaces.

Upgrade receipt

After all stages pass, write and check the receipt:

export EXPECTED_CONTROL_STACK_VERSION="1.0.10"
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
  get job upgrade-receipt --ignore-not-found
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
  get configmap nvcf-upgrade-receipt -o yaml
python3 "$UPGRADE_SCRIPTS/check-upgrade-receipt.py"

Require the receipt to match the selected Self-Managed version. The ConfigMap persists after the hook job expires. To correct a stale receipt, verify the installed package and workloads, then rerun that package’s receipt stage; do not patch the ConfigMap.

6. Resume workloads

Invoke an existing function with its known request and expected response:

CANARY_REQUEST='{"input":"your-known-canary-input"}'
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
  --request-body "$CANARY_REQUEST" --timeout 120

For LLM functions, use the LLM invocation options. Check secret access, then resume submissions and monitor queue progress. Existing workers keep their images until redeployed; stack defaults apply to new workers unless their deployment specifies an image override.

Rollback boundary

Stop on failure and follow your reviewed recovery plan:

• Retain patched NVCA, target Tasks and the corrected hostname policy. Do not apply an old bundle unchanged.
• For an old core apply, exclude ingress and nvct-api with --selector 'name!=ingress,name!=nvct-api' --skip-needs. Old Tasks settings restore unpublished worker images.
• Keep the OpenBao bridge image and both plugins. New mounts may reference commands absent from old images; plugin rollback/cleanup belongs to the separate plugin migration procedure.
• Verify original identities, existing workloads and durable data after recovery. If you run tasks, launch a new task before ending recovery.

Application recovery does not validate server/agent downgrades or reverse stored data. Use the OpenBao recovery boundary and your datastore recovery plan for restores.