Skip to navigation

Upgrade from NVCF 1.1.0 to NVCF 1.1.1

View as MarkdownOpen in Claude

This upgrade procedure and its validation are still in progress. Follow the 1.1.1 qualification issue for the remaining work. Do not treat the publication of NVCF 1.1.1 as a qualified upgrade path until that issue records it.

Use this procedure for NVCF 1.1.0. NVCF 1.1.1 is a patch release: it fixes the emSign CA issue (CVE-2026-111943), adds the optional worker callback proxy for multi-cluster installs and moves OpenBao to a versioned JWT plugin lifecycle.

NVCF 1.1.0Target: NVCF 1.1.1
Self-Managed1.1.51.1.9
Compute Plane1.1.01.1.1
Observability1.1.01.1.0

Plan a maintenance window based on a rehearsal of this procedure. NATS, gRPC proxy and Invocation Service rollovers, and the one-at-a-time OpenBao member replacement, can interrupt in-flight invocations and make the control-plane API unavailable for short periods.

Choose your upgrade path

NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach NVCF 1.1.1:

You runDo this
NVCF 1.0.1, 1.0.2 or 1.0.3Upgrade to NVCF 1.0.4 first with the 1.0.2 to 1.0.4 upgrade, then use the 1.0.4 to 1.1.1 upgrade.
NVCF 1.0.4Use the 1.0.4 to 1.1.1 upgrade.
NVCF 1.1.0You have already passed the 1.0.4 stop. Use the 1.1.0 to 1.1.1 upgrade.

Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.

What changes in 1.1.1

Component1.1.01.1.1
OpenBao chart0.36.10.37.1
OpenBao server and agent image2.6.3-nv-1.3.72.6.3-nv-1.3.9
OpenBao migrations image0.23.10.23.4
cert-manager chart0.1.00.2.0
cert-manager imagesv1.20.2v1.20.5 (upstream)
NATS chart0.9.00.9.1
NATS server image2.14.6-alpine3.222.14.9-alpine3.22
NATS config reloader0.24.00.25.0
Invocation service chart1.9.01.10.0
Invocation service image0.14.20.15.1
Function autoscaler chart0.8.00.8.1
Function autoscaler image1.23.11.23.2
ICMS0.8.50.8.6
Event Ledger0.21.00.21.2
Vanity gateway1.42.01.42.2
LLM request router (stargate) and pylon0.29.20.29.3
Worker LLM credentials image1.2.21.2.3
NICLLS worker2.118.22.118.5

Compute Plane 1.1.1 changes NVCA and the NVCA operator from 3.18.6 to 3.18.10, the collectors from 0.160.0-nv-0.4.7 to 0.160.0-nv-0.4.9, and the NATS reloader from 0.24.0 to 0.25.0, which matches Self-Managed. Observability stays on 1.1.0. Upgrade the control plane first, then each GPU cluster to Compute Plane 1.1.1.

Other changes you may notice:

  • Component images are rebuilt with Go 1.26.10 and the patched CA bundle.
  • The worker callback proxy is new and off by default. See Optional: enable the worker callback proxy.
  • api.accountBootstrap.enabled lets an install whose account is provisioned another way turn off the account-bootstrap hook Job. Unset, nothing changes.
  • You can set operator pod annotations per release, for example api.podAnnotations and nats.podAnnotations. Unset, nothing is rendered.
  • ingress.gatewayApi.routes.llmApiGateway.enabled and ingress.gatewayApi.routes.llmInvocation.enabled turn off an LLM route while the LLM add-on is on.
  • The NVCF UI add-on and the LLS HMAC rotation hook now use the shared global.openbaoMigrations.image.tag (0.23.4). The UI add-on pinned 0.19.10 in 1.1.0, so its image changes too. If you pin addons.nvcfUi.openbaoMigrations.image.tag, review that pin against 0.23.4.

cert-manager images come from upstream

1.1.0 defaulted the cert-manager controller, webhook, cainjector and acmesolver images to your configured NVCF registry, <global.image.registry>/<global.image.repository>/cert-manager-<component>. 1.1.1 uses the repositories of cert-manager chart 0.2.0, which are upstream (quay.io/jetstack), and those images do not inherit global.image.

If the cluster pulls from a private mirror or has no access to the upstream registry, do this before you apply the dependencies stage. Otherwise the cert-manager pods try to pull from the upstream registry and fail.

  1. Mirror the upstream images for the five components: controller, webhook, cainjector, acmesolver and startupapicheck. See upstream cert-manager images.
  2. Set the repository of each component in environments/<name>.yaml, for example certManager.image.repository for the controller and certManager.webhook.image.repository, certManager.cainjector.image.repository, certManager.acmesolver.image.repository and certManager.startupapicheck.image.repository.
  3. After the dependencies stage, list the cert-manager pod images and confirm they come from your mirror.

Prerequisites

  • Python 3, Helm 3.21.4, Helmfile 1.1.9, kubectl, jq, curl, Mike Farah’s yq v4, and the helm-diff plugin.
  • The Python cryptography package, to verify JWT canary signatures against the mount’s JWKS.
  • nvcf-cli 1.16.8 or later from the NGC download instructions.
  • Download the NVCF 1.1.1 packaged release and manifest. Mirror the required artifacts to your configured registry; see image mirroring. Include both task worker images if you use tasks and the compute-plane Samba image. The upstream supporting images (NATS server/reloader/box and account-bootstrap/receipt helpers) need their own mirror settings; they do not inherit global.image.
  • Extract the stacks under deploy/stacks/self-managed, deploy/stacks/nvcf-compute-plane and deploy/stacks/observability in a new bundle directory.
  • Spare capacity for rolling updates. A rollout starts one extra pod per Deployment before it removes an old one. Keep free CPU and memory on at least one node equal to the largest pod request, or a stage can stall.

Before starting

  1. Back up configuration, databases and persistent volumes.
  2. Prepare the target environment and secrets files. Transfer deployment settings and secrets from environments/<name>.yaml and secrets/<name>-secrets.yaml; keep the target chart/image versions. Do not copy image tags or chart versions from your 1.1.0 environment file over the target defaults unless you mean to pin them.
  3. Retain the compute-plane registration file and account ID. Do not run register-cluster again.
  4. Locate the kubeconfig for each control-plane and compute cluster. Step 1 exports them as CONTROL_KUBECONFIG and COMPUTE_KUBECONFIG, and every later step uses those variables. In a new shell, export the step 1 values again before continuing.
  5. Complete the cert-manager image settings if the cluster uses a mirror.
  6. Schedule a maintenance window based on a rehearsal of this procedure.

1. Set deployment values

Download these read-only scripts into one directory:

Workload readiness check Receipt check JWT mount preservation check

Use Bash with set -euo pipefail so a failed command stops the procedure. Fill in your existing deployment values below. Each kubeconfig must already select its intended context. For separate clusters, replace the two COMPUTE_* values; for a single cluster, keep them equal to the control plane.

set -euo pipefail
umask 077
export TARGET_BUNDLE="/absolute/path/to/stack-files"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export HELMFILE_ENV="production"
export CLUSTER_NAME="existing-gpu-cluster"
export NCA_ID="existing-account-id"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export CONTROL_KUBECONFIG="/secure/path/to/control-kubeconfig"
export CONTROL_CONTEXT="existing-control-context"
export COMPUTE_KUBECONFIG="$CONTROL_KUBECONFIG"
export COMPUTE_CONTEXT="$CONTROL_CONTEXT"

REGISTRATION_VALUES_DIR contains <cluster-name>-register-values.yaml. Set the stack paths and define the helper used below:

export CONTROL_STACK="$TARGET_BUNDLE/deploy/stacks/self-managed"
export COMPUTE_STACK="$TARGET_BUNDLE/deploy/stacks/nvcf-compute-plane"
export KUBECONFIG="$CONTROL_KUBECONFIG"
control_helmfile() {
(cd "$CONTROL_STACK" &&
HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
--kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
redact_diff() {
awk '
/^[-+ ]+- name: / { sensitive = (toupper($0) ~ /PASSWORD|SECRET|TOKEN|KEY|CREDENTIAL/); print; next }
sensitive && /^[-+ ]+value: / { sub(/value: .*/, "value: [REDACTED]"); print; next }
{ sensitive = 0; print }'
}

--suppress-secrets hides Kubernetes Secret data only. The OpenBao migration Job carries the Cassandra password and the registry credential as plain environment values, so its diff prints them. redact_diff replaces the value of any environment entry whose name contains PASSWORD, SECRET, TOKEN, KEY or CREDENTIAL. Pipe every OpenBao and dependency diff through it, and keep that output private.

2. Drain pending work

Pause new production requests and task submissions. Check every affected function/version before rolling NATS or dependent services.

The checks in this step take under one minute, even for large clusters. The total drain time depends on your longest in-flight request or running task. Plan the maintenance window around that duration.

Configure CLI authentication: NVCF_API_KEY needs queue_details; NVCF_NVCT_API_KEY needs list_tasks. Point base_nvct_url and nvct_host at this control plane’s Tasks endpoint.

NVCF_CLI="/absolute/path/to/nvcf-cli"
NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
FUNCTION_ID="existing-function-id"
VERSION_ID="existing-version-id"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING

Require queue size zero for every affected queue; a missing response blocks progress. Let running tasks finish; do not discard work to clear queues.

Check the NATS memory streams on every NATS server too. Choose an unused local port for monitoring port 8222 and keep submissions paused. The temporary directory holds working files for this check and the OpenBao transition:

umask 077
export UPGRADE_RECORD="$(mktemp -d)"
NATS_MONITOR_PORT=18223
NATS_PODS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
get pods -o json | jq -r '.items[] |
select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
mkdir "$UPGRADE_RECORD/nats-drain"
for pod in $NATS_PODS; do
(
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
> "$UPGRADE_RECORD/nats-drain/$pod.log" 2>&1 &
forward_pid=$!
trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
for attempt in {1..30}; do
kill -0 "$forward_pid"
if grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"; then
break
fi
sleep 1
done
grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"
curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
"http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
> "$UPGRADE_RECORD/nats-drain/$pod.json"
jq -e '.account_details | type == "array"' "$UPGRADE_RECORD/nats-drain/$pod.json"
jq -e 'all(.account_details[];
((.stream_detail // []) | type == "array") and
all((.stream_detail // [])[];
(.config.storage | type == "string") and
(.state.messages | type == "number")))' "$UPGRADE_RECORD/nats-drain/$pod.json"
)
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
select(.config.storage == "memory") |
{name: .name, replicas: (.config.num_replicas // 1), messages: .state.messages}] as $s |
if ($s | length) == 0 then error("No memory streams found; check the NATS monitoring endpoint")
elif any($s[]; .messages > 0) then error("Memory streams not drained; do not roll NATS")
else {memory_streams: ($s | length), single_replica: ([$s[] | select(.replicas == 1)] | length)} end' \
"$UPGRADE_RECORD/nats-drain/"*.json

Require every memory stream to be present and empty. A replicated stream (under preferred or enforced HA) survives a single restart, but draining it keeps the result unambiguous. Failed reads, missing streams or nonzero counts block the rollout. If responses are paginated, retrieve every account/stream page before accepting the result. Use your configured monitoring endpoint if it differs.

Keep NATS running while work drains. For stuck deployments or retained creation messages, follow the read-only recovery steps. For a retry, use a new nats-drain directory; keep the OpenBao baseline.

3. Apply control-plane stages in order

Keep submissions paused. Review each Helmfile diff before confirming the apply. Wait for each stage to succeed before continuing.

OpenBao bridge before dependency migrations

NVCF 1.1.0 runs OpenBao image 2.6.3-nv-1.3.7 and does not set a JWT plugin lifecycle mode in its environment file. The 1.1.1 stack defaults to versioned and the target uses chart 0.37.1, server/agent 2.6.3-nv-1.3.9 and migrations 0.23.4. An existing installation must not jump straight to that default. Move to it in stages: set bridge mode, replace every member, then switch to versioned. The bridge retains the installed JWT plugin alongside the new one. Existing mounts keep their plugin selection; new mounts use the target plugin. Existing mounts receive its fixes through a separate plugin migration.

Before changing OpenBao:

  1. Complete the snapshot and health checks and define check_openbao. Require three healthy, unsealed members on the same architecture, the original Raft cluster ID and retained unseal material. The active member is the pod whose check_openbao output reports is_self: true.
  2. Follow the JWT preflight to define bao_admin. Set BAO_POD to the active member from the previous item. Use the control-plane KUBECONFIG and verify a working JWT canary with the JWT canary. For the preflight BRIDGE_IMAGE, use the target server image from your configured registry, for example nvcr.io/nvidia/nvcf/nvcf-openbao:2.6.3-nv-1.3.9.
  3. Pause external mount creation and tuning. Capture the baseline once:
(set -o noclobber; bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-before.json")
python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" "$UPGRADE_RECORD/jwt-before.json"

Stop on missing metadata or an incomplete plugin transition. The bridge hook must match the retained binary to the stored catalog SHA. On mismatch, obtain a qualified bridge with the matching binary; do not rewrite the SHA or bypass hooks.

Set bridge mode and review the OpenBao diff, leaving NATS running:

export CONTROL_VALUES="$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
yq -i '.openbao.jwtPluginLifecycle.mode = "bridge"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
--selector name=openbao-server diff --suppress-secrets --skip-needs | redact_diff

Stop on unexpected changes. Then apply only OpenBao:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
--selector name=openbao-server apply --skip-needs

Bridge mode pauses migrations and catalog refresh. OnDelete leaves server pods on the old image until you replace them.

Follow the standby replacement and leadership transfer for every member, one at a time. Before the next replacement, verify unsealed health, the original cluster ID, the selected bridge digest and JWT issuance. Update BAO_POD after leadership changes.

With OnDelete, the StatefulSet can keep reporting the old currentRevision after every member is replaced. Confirm the rollout with updatedReplicas equal to replicas instead:

kubectl -n vault-system get statefulset openbao-server \
-o jsonpath='{.status.updatedReplicas}/{.spec.replicas}{"\n"}'

Once all members run the bridge, check retained mount versions/SHAs and enable migrations. New mounts must use the target catalog entry:

check_upgrade_jwt_mounts() {
check_openbao
bao_admin jwt-plugin-lifecycle preflight
bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-current.json"
bao_admin sh -ec '. /openbao/jwt-plugin-versions.env
bao plugin info -format=json -version="$JWT_PLUGIN_TARGET_VERSION" secret vault-plugin-secrets-jwt' \
> "$UPGRADE_RECORD/jwt-target-catalog.json"
python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" \
"$UPGRADE_RECORD/jwt-before.json" "$UPGRADE_RECORD/jwt-current.json" \
"$UPGRADE_RECORD/jwt-target-catalog.json"
}
check_upgrade_jwt_mounts
yq -i '.openbao.jwtPluginLifecycle.mode = "versioned"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
--selector name=openbao-server diff --suppress-secrets --skip-needs | redact_diff

Review the diff, then apply the versioned mode and check the mounts again:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
--selector name=openbao-server apply --skip-needs
check_upgrade_jwt_mounts

Keep versioned mode for subsequent applies. It enables migrations and new mounts without retuning existing ones. Verify existing and new secret consumers, including JWT issuance and signature checks after normal failover. Retain both binaries and catalog entries.

After a shell failure, redefine check_openbao, bao_admin and check_upgrade_jwt_mounts, then reselect BAO_POD. Keep the original baseline.

Dependencies and migrations

Proceed only after the OpenBao checks pass. Use this staged apply instead of the patch runbook’s whole-stack make apply:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets | redact_diff

Stop on unexpected deletions or version overrides. Then apply it:

control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n "$namespace" get jobs
done
check_upgrade_jwt_mounts

Require successful dependency releases and migration jobs before continuing. The NATS server rolls to 2.14.9 here. Confirm that the cert-manager pods run the images you selected in cert-manager images come from upstream.

Core services

Review the diff first, and stop on unexpected deletions or version overrides:

control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets

Then apply it:

control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
done
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A

Require deployed releases, ready workloads and accepted routes, including any additional enabled addon namespaces. The worker-callback-proxy release is off by default and does not appear until you enable it.

If a release fails or a rollout stalls, see A release stays failed after a stalled rollout.

State metrics and autoscaling

Review the diff first:

control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl diff --suppress-secrets

Stop on unexpected changes. Then apply the stage:

control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
-f helmfile.d/03-observability.yaml.gotmpl status
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" nvcf

Keep your existing observability profile. Enabled state-metrics and autoscaler releases must be deployed and healthy. The function autoscaler chart moves to 0.8.1. The Observability stack stays on 1.1.0, so skip its infrastructure stage.

Upgrade receipt

After all stages pass, write and check the receipt:

export EXPECTED_CONTROL_STACK_VERSION="1.1.9"
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
get job upgrade-receipt --ignore-not-found
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
get configmap nvcf-upgrade-receipt -o yaml
python3 "$UPGRADE_SCRIPTS/check-upgrade-receipt.py"

Require the receipt to match the selected Self-Managed version. The ConfigMap persists after the hook job expires. To correct a stale receipt, verify the installed package and workloads, then rerun that package’s receipt stage; do not patch the ConfigMap.

4. Verify the control plane

Do not upgrade the compute plane until the control plane is healthy. Check the releases, the receipt and the routes, and repeat your canary invocation:

helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
get configmap nvcf-upgrade-receipt -o jsonpath='{.data.installed_stack_version}{"\n"}'

Every release must be deployed, the routes Accepted, and the receipt must show 1.1.9. Then run the canary invocation from Resume workloads against a function that is already deployed; it must return the result you expect.

5. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Apply Compute Plane 1.1.1 with the cluster’s retained registration and account ID. Use a separate output directory from the registration file, and repeat the command for each cluster with its own CLUSTER_NAME, NCA_ID, registration directory and kube context:

make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
OUTPUT_DIR="$COMPUTE_STACK/out-upgrade"
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$COMPUTE_KUBECONFIG" "$COMPUTE_CONTEXT" nvca-operator
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-system get pods
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-operator get nvcfbackends

Wait for ready operator/agent pods and a healthy backend. Keep the existing Observability 1.1.0 configuration on separate compute clusters.

6. Resume workloads

Invoke an existing function with its known request and expected response:

CANARY_REQUEST='{"input":"your-known-canary-input"}'
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
--function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
--request-body "$CANARY_REQUEST" --timeout 120

For LLM functions, use the LLM invocation options. Check secret access, then resume submissions and monitor queue progress. Existing workers keep their images until redeployed; stack defaults apply to new workers unless their deployment specifies an image override. Redeploy functions and tasks to pick up the patched NICLLS worker and worker images.

Optional: enable the worker callback proxy

The proxy is off by default, and this upgrade does not turn it on. Enable it after the upgrade passes if you run the control plane and the compute plane in different clusters and want more than one replica of the invocation service or gRPC proxy. Set workerCallbackProxy.enabled: true and workerCallbackProxy.workerDomain in your environment file. The proxy needs its own DNS name, ports and certificate; follow Worker callback proxy, including Expose the proxy and its DNS name. The stack refuses to render if the proxy is on while a fixed callback URL such as global.workerEndpoints.invocationServiceURL is set. For the replica rules, see Multi-cluster installs.

Troubleshooting

A release stays failed after a stalled rollout

A release can fail with context deadline exceeded when a rolling update cannot start its extra pod, usually for lack of CPU or memory on one node. Stop and free capacity, then rerun the same stage apply. If the release then stays failed in helm list because Helmfile sees no diff, record it as deployed before you continue, with the name of the failed release:

control_helmfile -f helmfile.d/02-core.yaml.gotmpl --selector name=<release> sync

Use the stage file that owns the release. Then confirm that every release is deployed and every workload is ready before the next stage.

Rollback boundary

Stop on failure and follow your reviewed recovery plan:

  • Retain patched NVCA and the target Tasks settings. Do not apply an old bundle unchanged.
  • Keep the OpenBao bridge image and both plugins. OpenBao cannot return to image 2.6.3-nv-1.3.7 or chart 0.36.1 after the lifecycle change, and database migrations are not reversed. Plugin rollback and cleanup belong to the separate plugin migration procedure.
  • Verify original identities, existing workloads and durable data after recovery. If you run tasks, launch a new task before ending recovery.

Application recovery does not validate server/agent downgrades or reverse stored data. Use the OpenBao recovery boundary and your datastore recovery plan for restores.