> This page is for version Development.
> For other versions, use one of these documentation indexes:
> - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt
> - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt
> - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.

# Upgrade from NVCF 1.1.0 to NVCF 1.1.1

<Warning>
This upgrade procedure and its validation are still in progress. Follow
[the 1.1.1 qualification issue](https://github.com/dsx-ai-factory/nvcf/issues/3428)
for the remaining work. Do not treat the publication of NVCF 1.1.1 as a
qualified upgrade path until that issue records it.
</Warning>

Use this procedure for NVCF 1.1.0. NVCF 1.1.1 is a patch release: it fixes the
emSign CA issue (CVE-2026-111943), adds the optional worker callback proxy for
multi-cluster installs and moves OpenBao to a versioned JWT plugin lifecycle.

|               | NVCF 1.1.0 | Target: NVCF 1.1.1 |
| ------------- | ---------- | ------------------ |
| Self-Managed  | 1.1.5      | 1.1.9              |
| Compute Plane | 1.1.0      | 1.1.1              |
| Observability | 1.1.0      | 1.1.0              |

<Warning>
Plan a maintenance window based on a rehearsal of this procedure. NATS, gRPC
proxy and Invocation Service rollovers, and the one-at-a-time OpenBao member
replacement, can interrupt in-flight invocations and make the control-plane API
unavailable for short periods.
</Warning>

## Choose your upgrade path

NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach
NVCF 1.1.1:

| You run | Do this |
| --- | --- |
| NVCF 1.0.1, 1.0.2 or 1.0.3 | Upgrade to NVCF 1.0.4 first with the [1.0.2 to 1.0.4 upgrade](/nvcf/dev/release-notes/1-0-2-to-1-0-4-upgrade), then use the [1.0.4 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-0-4-to-1-1-1-upgrade). |
| NVCF 1.0.4 | Use the [1.0.4 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-0-4-to-1-1-1-upgrade). |
| NVCF 1.1.0 | You have already passed the 1.0.4 stop. Use the [1.1.0 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-1-0-to-1-1-1-upgrade). |

Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do
not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.

## What changes in 1.1.1

| Component | 1.1.0 | 1.1.1 |
| --------- | ----- | ----- |
| OpenBao chart | 0.36.1 | 0.37.1 |
| OpenBao server and agent image | 2.6.3-nv-1.3.7 | 2.6.3-nv-1.3.9 |
| OpenBao migrations image | 0.23.1 | 0.23.4 |
| cert-manager chart | 0.1.0 | 0.2.0 |
| cert-manager images | v1.20.2 | v1.20.5 (upstream) |
| NATS chart | 0.9.0 | 0.9.1 |
| NATS server image | 2.14.6-alpine3.22 | 2.14.9-alpine3.22 |
| NATS config reloader | 0.24.0 | 0.25.0 |
| Invocation service chart | 1.9.0 | 1.10.0 |
| Invocation service image | 0.14.2 | 0.15.1 |
| Function autoscaler chart | 0.8.0 | 0.8.1 |
| Function autoscaler image | 1.23.1 | 1.23.2 |
| ICMS | 0.8.5 | 0.8.6 |
| Event Ledger | 0.21.0 | 0.21.2 |
| Vanity gateway | 1.42.0 | 1.42.2 |
| LLM request router (stargate) and pylon | 0.29.2 | 0.29.3 |
| Worker LLM credentials image | 1.2.2 | 1.2.3 |
| NICLLS worker | 2.118.2 | 2.118.5 |

Compute Plane 1.1.1 changes NVCA and the NVCA operator from 3.18.6 to 3.18.10,
the collectors from `0.160.0-nv-0.4.7` to `0.160.0-nv-0.4.9`, and the NATS
reloader from 0.24.0 to 0.25.0, which matches Self-Managed. Observability stays
on 1.1.0. Upgrade the control plane first, then each GPU cluster to Compute
Plane 1.1.1.

Other changes you may notice:

- Component images are rebuilt with Go 1.26.10 and the patched CA bundle.
- The [worker callback proxy](/nvcf/dev/self-managed/worker-callback-proxy) is new
  and off by default. See [Optional: enable the worker callback proxy](#optional-enable-the-worker-callback-proxy).
- `api.accountBootstrap.enabled` lets an install whose account is provisioned
  another way turn off the account-bootstrap hook Job. Unset, nothing changes.
- You can set operator pod annotations per release, for example
  `api.podAnnotations` and `nats.podAnnotations`. Unset, nothing is rendered.
- `ingress.gatewayApi.routes.llmApiGateway.enabled` and
  `ingress.gatewayApi.routes.llmInvocation.enabled` turn off an LLM route while
  the LLM add-on is on.
- The NVCF UI add-on and the LLS HMAC rotation hook now use the shared
  `global.openbaoMigrations.image.tag` (`0.23.4`). The UI add-on pinned `0.19.10`
  in 1.1.0, so its image changes too. If you pin
  `addons.nvcfUi.openbaoMigrations.image.tag`, review that pin against `0.23.4`.

## cert-manager images come from upstream

1.1.0 defaulted the cert-manager controller, webhook, cainjector and acmesolver
images to your configured NVCF registry, `<global.image.registry>/<global.image.repository>/cert-manager-<component>`.
1.1.1 uses the repositories of cert-manager chart 0.2.0, which are upstream
(`quay.io/jetstack`), and those images do not inherit `global.image`.

If the cluster pulls from a private mirror or has no access to the upstream
registry, do this **before** you apply the dependencies stage. Otherwise the
cert-manager pods try to pull from the upstream registry and fail.

1. Mirror the upstream images for the five components: `controller`, `webhook`,
   `cainjector`, `acmesolver` and `startupapicheck`. See
   [upstream cert-manager images](/nvcf/dev/manifest/artifacts#use-upstream-cert-manager-images).
2. Set the repository of each component in `environments/<name>.yaml`, for
   example `certManager.image.repository` for the controller and
   `certManager.webhook.image.repository`, `certManager.cainjector.image.repository`,
   `certManager.acmesolver.image.repository` and `certManager.startupapicheck.image.repository`.
3. After the dependencies stage, list the cert-manager pod images and confirm
   they come from your mirror.

## Prerequisites

- Python 3, Helm 3.21.4, Helmfile 1.1.9, `kubectl`, `jq`, `curl`, Mike Farah's
  `yq` v4, and the [helm-diff plugin](/nvcf/dev/self-managed/helmfile-installation#required-tools-and-software).
- The Python `cryptography` package, to verify JWT canary signatures against
  the mount's JWKS.
- `nvcf-cli` 1.16.8 or later from the [NGC download instructions](/nvcf/dev/manifest/image-mirroring#downloading-nvcf-cli).
- Download the NVCF 1.1.1 packaged release and manifest. Mirror the required
  artifacts to your configured registry; see [image mirroring](/nvcf/dev/manifest/image-mirroring).
  Include both task worker images if you use tasks and the compute-plane Samba
  image. The [upstream supporting images](/nvcf/dev/manifest/artifacts#supporting-images-that-ship-from-docker-hub)
  (NATS server/reloader/box and account-bootstrap/receipt helpers) need their own
  mirror settings; they do not inherit `global.image`.
- Extract the stacks under `deploy/stacks/self-managed`,
  `deploy/stacks/nvcf-compute-plane` and `deploy/stacks/observability` in a new
  bundle directory.
- Spare capacity for rolling updates. A rollout starts one extra pod per
  Deployment before it removes an old one. Keep free CPU and memory on at least
  one node equal to the largest pod request, or a stage can stall.

## Before starting

1. Back up configuration, databases and persistent volumes.
2. Prepare the target [environment and secrets files](/nvcf/dev/self-managed/helmfile-installation).
   Transfer deployment settings and secrets from `environments/<name>.yaml`
   and `secrets/<name>-secrets.yaml`; keep the target chart/image versions. Do
   not copy image tags or chart versions from your 1.1.0 environment file over
   the target defaults unless you mean to pin them.
3. Retain the compute-plane registration file and account ID. Do not run
   `register-cluster` again.
4. Locate the kubeconfig for each control-plane and compute cluster. Step 1
   exports them as `CONTROL_KUBECONFIG` and `COMPUTE_KUBECONFIG`, and every
   later step uses those variables. In a new shell, export the step 1 values
   again before continuing.
5. Complete the [cert-manager image settings](#cert-manager-images-come-from-upstream)
   if the cluster uses a mirror.
6. Schedule a maintenance window based on a rehearsal of this procedure.

## 1. Set deployment values

Download these read-only scripts into one directory:

<Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/dd5b8cce87225ba59eded42f03db7dd225dfb9c1834803ec0aa8b572f02937bf/assets/upgrade-checks/wait-upgrade-workloads.py" filename="wait-upgrade-workloads.py">Workload readiness check</Download>
<Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/14cd65e335917934d739c15ec14825fbbd8c35c48b7fc7aad126067a2f56de83/assets/upgrade-checks/check-upgrade-receipt.py" filename="check-upgrade-receipt.py">Receipt check</Download>
<Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/f452c468639eca7b6ffe5d08835be0b6977f96fa6691c1bf2a02e22c492af851/assets/upgrade-checks/check-upgrade-jwt-mounts.py" filename="check-upgrade-jwt-mounts.py">JWT mount preservation check</Download>

Use Bash with `set -euo pipefail` so a failed command stops the procedure.
Fill in your existing deployment values below. Each kubeconfig must already
select its intended context. For separate clusters, replace the two `COMPUTE_*`
values; for a single cluster, keep them equal to the control plane.

```bash
set -euo pipefail
umask 077
export TARGET_BUNDLE="/absolute/path/to/stack-files"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export HELMFILE_ENV="production"
export CLUSTER_NAME="existing-gpu-cluster"
export NCA_ID="existing-account-id"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export CONTROL_KUBECONFIG="/secure/path/to/control-kubeconfig"
export CONTROL_CONTEXT="existing-control-context"
export COMPUTE_KUBECONFIG="$CONTROL_KUBECONFIG"
export COMPUTE_CONTEXT="$CONTROL_CONTEXT"
```

`REGISTRATION_VALUES_DIR` contains `<cluster-name>-register-values.yaml`.
Set the stack paths and define the helper used below:

```bash
export CONTROL_STACK="$TARGET_BUNDLE/deploy/stacks/self-managed"
export COMPUTE_STACK="$TARGET_BUNDLE/deploy/stacks/nvcf-compute-plane"
export KUBECONFIG="$CONTROL_KUBECONFIG"
```

```bash
control_helmfile() {
  (cd "$CONTROL_STACK" &&
    HELMFILE_ENV="$HELMFILE_ENV" helmfile --environment default \
      --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
redact_diff() {
  awk '
    /^[-+ ]+- name: / { sensitive = (toupper($0) ~ /PASSWORD|SECRET|TOKEN|KEY|CREDENTIAL/); print; next }
    sensitive && /^[-+ ]+value: / { sub(/value: .*/, "value: [REDACTED]"); print; next }
    { sensitive = 0; print }'
}
```

`--suppress-secrets` hides Kubernetes Secret data only. The OpenBao migration
Job carries the Cassandra password and the registry credential as plain
environment values, so its diff prints them. `redact_diff` replaces the value of
any environment entry whose name contains `PASSWORD`, `SECRET`, `TOKEN`, `KEY` or
`CREDENTIAL`. Pipe every OpenBao and dependency diff through it, and keep that
output private.

## 2. Drain pending work

Pause new production requests and task submissions. Check every affected
function/version before rolling NATS or dependent services.

<Note>

The checks in this step take under one minute, even for large clusters. The
total drain time depends on your longest in-flight request or running task.
Plan the maintenance window around that duration.

</Note>

Configure [CLI authentication](/nvcf/dev/overview/cli): `NVCF_API_KEY` needs `queue_details`;
`NVCF_NVCT_API_KEY` needs `list_tasks`. Point `base_nvct_url` and `nvct_host`
at this control plane's Tasks endpoint.

```bash
NVCF_CLI="/absolute/path/to/nvcf-cli"
NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
FUNCTION_ID="existing-function-id"
VERSION_ID="existing-version-id"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING
```

Require queue size zero for every affected queue; a missing response blocks
progress. Let running tasks finish; do not discard work to clear queues.

Check the NATS memory streams on every NATS server too. Choose an unused
local port for monitoring port 8222 and keep submissions paused. The temporary
directory holds working files for this check and the OpenBao transition:

```bash
umask 077
export UPGRADE_RECORD="$(mktemp -d)"
NATS_MONITOR_PORT=18223
NATS_PODS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
  get pods -o json | jq -r '.items[] |
    select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
mkdir "$UPGRADE_RECORD/nats-drain"
for pod in $NATS_PODS; do
  (
    kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
      port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
      > "$UPGRADE_RECORD/nats-drain/$pod.log" 2>&1 &
    forward_pid=$!
    trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
    for attempt in {1..30}; do
      kill -0 "$forward_pid"
      if grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"; then
        break
      fi
      sleep 1
    done
    grep -q 'Forwarding from 127.0.0.1:' "$UPGRADE_RECORD/nats-drain/$pod.log"
    curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
      "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
      > "$UPGRADE_RECORD/nats-drain/$pod.json"
    jq -e '.account_details | type == "array"' "$UPGRADE_RECORD/nats-drain/$pod.json"
    jq -e 'all(.account_details[];
      ((.stream_detail // []) | type == "array") and
      all((.stream_detail // [])[];
        (.config.storage | type == "string") and
        (.state.messages | type == "number")))' "$UPGRADE_RECORD/nats-drain/$pod.json"
  )
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
  select(.config.storage == "memory") |
  {name: .name, replicas: (.config.num_replicas // 1), messages: .state.messages}] as $s |
  if ($s | length) == 0 then error("No memory streams found; check the NATS monitoring endpoint")
  elif any($s[]; .messages > 0) then error("Memory streams not drained; do not roll NATS")
  else {memory_streams: ($s | length), single_replica: ([$s[] | select(.replicas == 1)] | length)} end' \
  "$UPGRADE_RECORD/nats-drain/"*.json
```

Require every memory stream to be present and empty. A replicated stream (under
`preferred` or `enforced` HA) survives a single restart, but draining it keeps
the result unambiguous. Failed reads, missing streams or nonzero counts block
the rollout. If responses are paginated,
retrieve every account/stream page before accepting the result. Use your configured
monitoring endpoint if it differs.

Keep NATS running while work drains. For stuck deployments or retained creation
messages, follow the [read-only recovery steps](/nvcf/dev/self-managed/upgrade-recovery#inspect-retained-creation-messages).
For a retry, use a new `nats-drain` directory; keep the OpenBao baseline.

## 3. Apply control-plane stages in order

Keep submissions paused. Review each Helmfile diff before confirming the apply. Wait for each stage to succeed before continuing.

### OpenBao bridge before dependency migrations

NVCF 1.1.0 runs OpenBao image `2.6.3-nv-1.3.7` and does not set a JWT plugin
lifecycle mode in its environment file. The 1.1.1 stack defaults to `versioned`
and the target uses chart `0.37.1`, server/agent `2.6.3-nv-1.3.9` and
migrations `0.23.4`. An existing installation must not jump straight to that
default. Move to it in stages: set bridge mode, replace every member, then
switch to `versioned`. The bridge retains the installed JWT
plugin alongside the new one. Existing mounts keep their plugin selection; new
mounts use the target plugin. Existing mounts receive its fixes through a
separate [plugin migration](/nvcf/dev/self-managed/open-bao-jwt-plugin-upgrade#pin-rotate-then-upgrade).

Before changing OpenBao:

1. Complete the [snapshot and health checks](/nvcf/dev/self-managed/open-bao-patch-upgrade#before-you-start)
   and define `check_openbao`. Require three healthy, unsealed members on the
   same architecture, the original Raft cluster ID and retained unseal material.
   The active member is the pod whose `check_openbao` output reports
   `is_self: true`.
2. Follow the [JWT preflight](/nvcf/dev/self-managed/open-bao-jwt-plugin-upgrade#preflight)
   to define `bao_admin`. Set `BAO_POD` to the active member from the previous
   item. Use the control-plane `KUBECONFIG` and verify a working JWT canary with
   the [JWT canary](/nvcf/dev/self-managed/open-bao-jwt-plugin-upgrade#jwt-canary).
   For the preflight `BRIDGE_IMAGE`, use the target server image from your
   configured registry, for example
   `nvcr.io/nvidia/nvcf/nvcf-openbao:2.6.3-nv-1.3.9`.
3. Pause external mount creation and tuning. Capture the baseline once:

```bash
(set -o noclobber; bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-before.json")
python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" "$UPGRADE_RECORD/jwt-before.json"
```

Stop on missing metadata or an incomplete plugin transition. The bridge hook
must match the retained binary to the stored catalog SHA. On mismatch, obtain
a qualified bridge with the matching binary; do not rewrite the SHA or bypass hooks.

Set bridge mode and review the OpenBao diff, leaving NATS running:

```bash
export CONTROL_VALUES="$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
yq -i '.openbao.jwtPluginLifecycle.mode = "bridge"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server diff --suppress-secrets --skip-needs | redact_diff
```

Stop on unexpected changes. Then apply only OpenBao:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server apply --skip-needs
```

Bridge mode pauses migrations and catalog refresh. `OnDelete` leaves server
pods on the old image until you replace them.

Follow the [standby replacement and leadership transfer](/nvcf/dev/self-managed/open-bao-patch-upgrade#replace-standbys-one-at-a-time)
for every member, one at a time. Before the next replacement, verify unsealed
health, the original cluster ID, the selected bridge digest and JWT issuance.
Update `BAO_POD` after leadership changes.

With `OnDelete`, the StatefulSet can keep reporting the old `currentRevision`
after every member is replaced. Confirm the rollout with `updatedReplicas`
equal to `replicas` instead:

```bash
kubectl -n vault-system get statefulset openbao-server \
  -o jsonpath='{.status.updatedReplicas}/{.spec.replicas}{"\n"}'
```

Once all members run the bridge, check retained mount versions/SHAs and enable
migrations. New mounts must use the target catalog entry:

```bash
check_upgrade_jwt_mounts() {
  check_openbao
  bao_admin jwt-plugin-lifecycle preflight
  bao_admin bao secrets list -detailed -format=json > "$UPGRADE_RECORD/jwt-current.json"
  bao_admin sh -ec '. /openbao/jwt-plugin-versions.env
    bao plugin info -format=json -version="$JWT_PLUGIN_TARGET_VERSION" secret vault-plugin-secrets-jwt' \
    > "$UPGRADE_RECORD/jwt-target-catalog.json"
  python3 "$UPGRADE_SCRIPTS/check-upgrade-jwt-mounts.py" \
    "$UPGRADE_RECORD/jwt-before.json" "$UPGRADE_RECORD/jwt-current.json" \
    "$UPGRADE_RECORD/jwt-target-catalog.json"
}
check_upgrade_jwt_mounts
yq -i '.openbao.jwtPluginLifecycle.mode = "versioned"' "$CONTROL_VALUES"
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server diff --suppress-secrets --skip-needs | redact_diff
```

Review the diff, then apply the `versioned` mode and check the mounts again:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl \
  --selector name=openbao-server apply --skip-needs
check_upgrade_jwt_mounts
```

Keep `versioned` mode for subsequent applies. It enables migrations and new
mounts without retuning existing ones. Verify existing and new secret consumers,
including JWT issuance and signature checks after normal failover. Retain both
binaries and catalog entries.

After a shell failure, redefine `check_openbao`, `bao_admin` and
`check_upgrade_jwt_mounts`, then reselect `BAO_POD`. Keep the original baseline.

### Dependencies and migrations

Proceed only after the OpenBao checks pass. Use this staged apply instead of
the patch runbook's whole-stack `make apply`:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets | redact_diff
```

Stop on unexpected deletions or version overrides. Then apply it:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
  python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
  kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n "$namespace" get jobs
done
check_upgrade_jwt_mounts
```

Require successful dependency releases and migration jobs before continuing.
The NATS server rolls to 2.14.9 here. Confirm that the cert-manager pods run the
images you selected in [cert-manager images come from upstream](#cert-manager-images-come-from-upstream).

### Core services

Review the diff first, and stop on unexpected deletions or version overrides:

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets
```

Then apply it:

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
  python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" "$namespace"
done
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A
```

Require deployed releases, ready workloads and accepted routes, including
any additional enabled addon namespaces. The `worker-callback-proxy` release is
off by default and does not appear until you enable it.

If a release fails or a rollout stalls, see
[A release stays failed after a stalled rollout](#a-release-stays-failed-after-a-stalled-rollout).

### State metrics and autoscaling

Review the diff first:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl diff --suppress-secrets
```

Stop on unexpected changes. Then apply the stage:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl status
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$CONTROL_KUBECONFIG" "$CONTROL_CONTEXT" nvcf
```

Keep your existing observability profile. Enabled state-metrics and autoscaler
releases must be `deployed` and healthy. The function autoscaler chart moves to
0.8.1. The Observability stack stays on 1.1.0, so skip its infrastructure stage.

### Upgrade receipt

After all stages pass, write and check the receipt:

```bash
export EXPECTED_CONTROL_STACK_VERSION="1.1.9"
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
  get job upgrade-receipt --ignore-not-found
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
  get configmap nvcf-upgrade-receipt -o yaml
python3 "$UPGRADE_SCRIPTS/check-upgrade-receipt.py"
```

Require the receipt to match the selected Self-Managed version. The ConfigMap
persists after the hook job expires. To correct a stale receipt, verify the
installed package and workloads, then rerun that package's receipt stage;
do not patch the ConfigMap.

## 4. Verify the control plane

Do not upgrade the compute plane until the control plane is healthy. Check the
releases, the receipt and the routes, and repeat your canary invocation:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kubectl --kubeconfig "$CONTROL_KUBECONFIG" get httproutes -A
kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf \
  get configmap nvcf-upgrade-receipt -o jsonpath='{.data.installed_stack_version}{"\n"}'
```

Every release must be `deployed`, the routes `Accepted`, and the receipt must
show `1.1.9`. Then run the canary invocation from [Resume workloads](#6-resume-workloads)
against a function that is already deployed; it must return the result you
expect.

## 5. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Apply
Compute Plane 1.1.1 with the cluster's retained registration and account ID.
Use a separate output directory from the registration file, and repeat the
command for each cluster with its own `CLUSTER_NAME`, `NCA_ID`, registration
directory and kube context:

```bash
make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$COMPUTE_STACK/out-upgrade"
python3 "$UPGRADE_SCRIPTS/wait-upgrade-workloads.py" "$COMPUTE_KUBECONFIG" "$COMPUTE_CONTEXT" nvca-operator
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-system get pods
kubectl --kubeconfig "$COMPUTE_KUBECONFIG" -n nvca-operator get nvcfbackends
```

Wait for ready operator/agent pods and a healthy backend. Keep the existing
Observability 1.1.0 configuration on separate compute clusters.

## 6. Resume workloads

Invoke an existing function with its known request and expected response:

```bash
CANARY_REQUEST='{"input":"your-known-canary-input"}'
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
  --request-body "$CANARY_REQUEST" --timeout 120
```

For LLM functions, use the [LLM invocation options](/nvcf/dev/overview/cli#invoke-function).
Check secret access, then resume submissions and monitor queue progress.
Existing workers keep their images until redeployed; stack defaults apply to
new workers unless their deployment specifies an image override. Redeploy
functions and tasks to pick up the patched NICLLS worker and worker images.

## Optional: enable the worker callback proxy

The proxy is off by default, and this upgrade does not turn it on. Enable it
after the upgrade passes if you run the control plane and the compute plane in
different clusters and want more than one replica of the invocation service or
gRPC proxy. Set `workerCallbackProxy.enabled: true` and
`workerCallbackProxy.workerDomain` in your environment file. The proxy needs its
own DNS name, ports and certificate; follow
[Worker callback proxy](/nvcf/dev/self-managed/worker-callback-proxy), including
[Expose the proxy and its DNS name](/nvcf/dev/self-managed/worker-callback-proxy#expose-the-proxy-and-its-dns-name).
The stack refuses to render if the proxy is on while a fixed callback URL such
as `global.workerEndpoints.invocationServiceURL` is set. For the replica rules,
see [Multi-cluster installs](/nvcf/dev/self-managed/high-availability).

## Troubleshooting

### A release stays failed after a stalled rollout

A release can fail with `context deadline exceeded` when a rolling update cannot
start its extra pod, usually for lack of CPU or memory on one node. Stop and
free capacity, then rerun the same stage apply. If the release then stays
`failed` in `helm list` because Helmfile sees no diff, record it as deployed
before you continue, with the name of the failed release:

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl --selector name=<release> sync
```

Use the stage file that owns the release. Then confirm that every release is
`deployed` and every workload is ready before the next stage.

## Rollback boundary

Stop on failure and follow your reviewed recovery plan:

- Retain patched NVCA and the target Tasks settings. Do not apply an old bundle
  unchanged.
- Keep the OpenBao bridge image and both plugins. OpenBao cannot return to image
  `2.6.3-nv-1.3.7` or chart `0.36.1` after the lifecycle change, and database
  migrations are not reversed. Plugin rollback and cleanup belong to the separate
  plugin migration procedure.
- Verify original identities, existing workloads and durable data after recovery.
  If you run tasks, launch a new task before ending recovery.

Application recovery does not validate server/agent downgrades or reverse
stored data. Use the [OpenBao recovery boundary](/nvcf/dev/self-managed/open-bao-patch-upgrade#verify-servers-and-agents)
and your datastore recovery plan for restores.