> This page is for version 1.1.0.
> For other versions, use one of these documentation indexes:
> - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt
> - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt
> - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.

# Upgrade from NVCF 1.0.1 to NVCF 1.1.0

<Warning>
This upgrade procedure and its validation are still in progress. Publication
of NVCF 1.1.0 does not establish a qualified upgrade path. Follow
[upgrade completion](https://github.com/dsx-ai-factory/nvcf/issues/3375)
for the remaining work. The 1.1.0 docs edition is available for selection;
1.0.4 remains the default until this procedure and its validation are complete.
This procedure does not cover NVCF 1.0.4 deployments. Do not use it to upgrade
from 1.0.4 to 1.1.0.
</Warning>

This guide describes an upgrade of a running deployment from NVCF 1.0.1 to
NVCF 1.1.0. NVCF 1.1.0 selects Self-Managed 1.1.5 for the control plane,
Compute Plane 1.1.0 for each GPU cluster, and Observability 1.1.0. Read the
whole guide, including [What changes](#what-changes), before planning an upgrade.

| NVCF edition | Self-Managed | Compute Plane | Observability |
| --- | --- | --- | --- |
| 1.0.1 | 1.0.1 | 1.0.0 | 1.0.0 |
| 1.1.0 (target) | 1.1.5 | 1.1.0 | 1.1.0 |

The stack releases this guide covers are:

| Stack | Source | Target | Target tag |
| --- | --- | --- | --- |
| Self-Managed (control plane) | 1.0.1 | 1.1.5 | `deploy/stacks/self-managed/v1.1.5` |
| Compute Plane | 1.0.0 | 1.1.0 | `deploy/stacks/nvcf-compute-plane/v1.1.0` |
| Observability | 1.0.0 | 1.1.0 | `deploy/stacks/observability/v1.1.0` |

## Cross-stack upgrade order

These declarations apply to the supported starting releases in this guide:

| Transition | Required peer |
| --- | --- |
| Compute Plane 1.1.0 | Self-Managed 1.1.0 or later, subject to the compatibility and maintenance policy |
| Self-Managed 1.1.5 during the upgrade window | Existing Compute Plane 1.0.0 can remain until its upgrade |

After the prerequisite NVCA patch below, upgrade the control plane first,
then each GPU cluster to Compute Plane 1.1.0.

Starting from 0.6.x? Complete the
[0.6.2 to 1.0.1 upgrade](/nvcf/1.1.0/release-notes/0-6-1-to-1-0-1-upgrade) first. Do not skip its
schema or backfill stops.

<Warning>
Do not run a whole-stack `make apply` or `helmfile sync` first. The control
plane has stops that must happen in order: hostname policy, dependencies and
migrations, OpenBao pod replacement, core services, then Cassandra replication
for high availability. Highly available (HA) mode is on by default in 1.1.5.
Decide your mode in [Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration)
before you apply anything.
</Warning>

## What changes

### Control plane: charts

The chart versions the 1.1.5 stack pins, compared with 1.0.1. The `cert-manager`
release (0.1.0) is unchanged.

| Release | 1.0.1 | 1.1.5 |
| --- | --- | --- |
| `admin-issuer-proxy` | 1.5.3 | 1.7.0 |
| `api` | 1.27.1 | 1.35.0 |
| `api-keys` | 1.8.0 | 1.9.4 |
| `cassandra` | 0.21.3 | 0.26.0 |
| `ess-api` | 1.8.2 | 1.11.1 |
| `event-ledger` | not installed | 0.10.0 (new) |
| `function-autoscaler` | 0.5.2 | 0.8.0 |
| `grpc-proxy` | 1.7.4 | 1.14.0 |
| `ingress` | 1.18.2 | 1.18.3 |
| `invocation-service` | 1.6.1 | 1.9.0 |
| `llm-api-gateway` | 1.4.3 | 1.12.0 |
| `llm-request-router` | 1.14.1 | 1.18.2 |
| `nats` | 0.8.4 | 0.9.0 |
| `nats-auth-callout-service` | 1.2.1 | 1.4.2 |
| `notary-service` | 1.6.0 | 1.9.1 |
| `nvcf-pki` | 0.1.0 | 0.2.0 |
| `nvcf-ui` | 1.1.2 | 1.1.4 |
| `nvct-api` | 1.6.0 | 1.11.0 |
| `openbao-server` | 0.32.6 | 0.36.1 |
| `ratelimiter` | 1.2.1 | 1.4.1 |
| `reval` | 1.4.1 | 1.6.0 |
| `sis` | 2.4.0 | 2.6.3 |
| `state-metrics` | 1.0.2 | 1.0.7 |
| `vanity-gateway` | 0.5.0 | 0.11.0 |

The LLM, vanity gateway and UI releases matter only if you enable those add-ons.

### Control plane: data and security components

The data and security components below are the ones that matter for an in-place
upgrade. Versions are the ones each release deploys, taken from the published
stack inventories.

| Component | 1.0.1 | Target 1.1.5 |
| --- | --- | --- |
| Cassandra chart | 0.21.3 | 0.26.0 |
| Cassandra image | 5.0.9-nv-2.0.5 | 5.0.9-nv-2.0.7 |
| Cassandra migrations image | 0.17.6 | 0.19.4 |
| OpenBao chart | 0.32.6 | 0.36.1 |
| OpenBao image | 2.6.2-nv-1.3.4 | 2.6.3-nv-1.3.7 |
| OpenBao migrations images | 0.19.5 | 0.19.10 and 0.23.1 |
| NATS chart | 0.8.4 | 0.9.0 |
| NATS server image | 2.14.6-alpine3.22 | 2.14.6-alpine3.22 |
| API chart | 1.27.1 | 1.35.0 |
| API service image | 1.18.0 | 1.28.4 |
| NVCT API chart | 1.6.0 | 1.11.0 |
| Notary chart | 1.6.0 | 1.9.1 |

The target's main OpenBao migrations image is 0.23.1; 0.19.10
is used only by the optional UI add-on.

### Control plane: behavior that changes

| Area | What happens on upgrade | What you do |
| --- | --- | --- |
| High availability | `highAvailability.mode` is new and defaults to `preferred`. An environment file that does not set it activates HA: replica-safe services scale to 2, Cassandra and NATS scale to 3 members, and PodDisruptionBudgets and placement rules apply. | Choose `none`, `preferred` or `enforced` in [Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration). |
| Cassandra | The chart adds a config setting (`range_request_timeout: 5000ms`) and the server image moves to `5.0.9-nv-2.0.7`. The pod template changes, so the StatefulSet restarts its pods one at a time. The migrations image moves to `nvcf-cassandra-migrations:0.19.4`, whose new migrations only add schema ([Apply the dependency stage](#apply-the-dependency-stage)). | Plan a short Cassandra interruption if you run a single member. |
| OpenBao | Server image `2.6.2-nv-1.3.4` becomes `2.6.3-nv-1.3.7`. The migrations image used by the server hooks moves to 0.23.1. The StatefulSet uses `OnDelete`, so Helm success does not replace the running pods. | Replace the pods yourself in [Replace the OpenBao pods](#replace-the-openbao-pods). |
| Gateway hostname policy | The 1.0.1 gateway-routes chart (1.18.2) has an older hostname policy that blocks the new admin issuer route patch. 1.18.3 carries the corrected policy. | Run the one-time `apply-hostname-policy` step ([Refresh the hostname policy](#3-refresh-the-hostname-policy)). |
| Event Ledger | New release `event-ledger`, on by default (`addons.eventLedger.enabled: true`). It uses the existing `event_ledger` keyspace, and its OpenBao setup runs in the OpenBao migrations job. | Keep it on, or set `addons.eventLedger.enabled: false` before you apply. |
| NATS worker authentication | New `nats.workerAuth` setting. The default, `nkey`, keeps today's behavior. | Nothing, unless you switch to `token`. See "Worker NATS Authentication" in [control-plane operations](/nvcf/1.1.0/self-managed/control-plane-operations). |
| Admin issuer proxy | Now optional (`adminIssuerProxy.enabled`), still on by default. | Nothing, unless you want to disable it. |
| NVCT remote config | New `nvctApi.remoteConfig`, empty by default. | Nothing. |
| Upgrade receipt | The final stage records the installed stack version in `nvcf-upgrade-receipt`. | Validate package stamps first, then compare the receipt with the selected release. |
| NATS images | The NATS server and nats-box images default to public registries (`docker.io/synadia/nats-server`, `docker.io/natsio/nats-box:0.20.1-nonroot`), and the config reloader to `docker.io/natsio/nats-server-config-reloader`. Release 1.0.1 pulled the server and nats-box from the NVCF catalog. | If your cluster cannot pull from public registries, set the registry and repository for all three images to your mirror in [Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration). |
| Worker sidecars | API and NVCT remote config pin the ESS agent sidecar image to `ess-agent:1.4.5`. Running functions keep the images they started with. | If you mirror sidecar images, mirror `ess-agent:1.4.5`. |
| LLM add-on image tags | With `addons.llm.enabled`, the request router and Pylon tag must match, and `addons.llm.requestRouter.image.tag` is now required. `base.yaml` sets `0.29.2`. | Keep the default unless you mirror or pin it. |

### Observability

Observability 1.1.0 is optional. Self-Managed 1.1.5 also works with the
Observability 1.0.0 stack you already run, so you can upgrade Self-Managed on
its own and keep Observability at 1.0.0. The 1.1.0 release changes only two
charts and no images. In the default monitors it sets the `job` label on the
worker and DCGM series and adds pod target labels (`function-id`,
`function-version-id` and `nca-id`) to the worker series. The chart notes say
federation clients such as the BYOO collector select series by that job name,
and the autoscaler queries worker metrics by function identity. Upgrade
Observability if you rely on either. Function scaling and federation were not
exercised with Observability 1.0.0. The control-plane apply reads an
Observability package when the observability profile is enabled, so either
1.0.0 or 1.1.0 must be in place before the render
(see [Get the 1.1.5 packages and check artifacts](#get-the-115-packages-and-check-artifacts)).

| Chart | 1.0.0 | 1.1.0 |
| --- | --- | --- |
| `nvcf-default-monitors` | 0.2.0 | 0.3.0 |
| `nvcf-otel-collector` | 0.2.0 | 0.3.0 |

| Behavior | What changes | What to do |
| --- | --- | --- |
| PodMonitor job labels | The worker and DCGM PodMonitors now set the `job` label to `nvcf-worker` and `nvidia-dcgm-exporter`. | If your dashboards or alerts select the old job names for these series, update them. |
| Worker pod labels | The worker monitor also carries the `function-id`, `function-version-id` and `nca-id` pod labels onto its series. | Nothing, unless you limit label cardinality downstream. |

### Compute plane

The values that differ between Compute Plane 1.0.0 and 1.1.0:

| Component | 1.0.0 | 1.1.0 |
| --- | --- | --- |
| `nvca-operator` chart | 1.28.0 | 1.37.1 |
| NVCA agent (`nvcaVersion`) | 3.10.0 | 3.18.6 |
| Image credential helper (`imageCredHelper.imageTag`) | chart default | 0.11.4 |
| Shared storage image (`sharedStorage.imageTag`) | chart default | 1.0.9 |
| BYOO collector sidecar (`byooOtelCollector.imageTag`) | chart default (0.160.0-nv-0.2.5) | 0.160.0-nv-0.4.7 |

The `byooOtelCollector` setting first appears in 1.0.2 and `sharedStorage` in
1.0.3, so if you start from an earlier release those keys are new to your
environment file.

New optional settings: `global.nvcaOperator.agent.workerCallbackCaPem` (only
for TLS worker callbacks), `byoo.*` sidecar resources, `utils.resources`,
`byooFluentBit` and `eventLedgerServiceURL`. Dynamo, Grove and KAI Scheduler
pins are unchanged. Running function pods keep the worker images they started
with; the new defaults apply to deployments created or replaced afterwards.

## Known issues

- Historical packages can retain the train version `1.1.0` instead of the
  selected patch `1.1.5`. The package and receipt checks above reject this
  mismatch. Obtain a correctly stamped, qualified package before maintenance.
- On editions 1.0.0 through 1.0.3 (Self-Managed 1.0.1), tasks fail before the
  upgrade when images come from `nvcr.io/nvidia/nvcf`. Self-Managed 1.0.1
  selects `nvcf_worker_init:2.107.0`, `nvcf_worker_task:2.107.0` and
  `ess-agent:1.1.0`, which that catalog does not publish, so task pods stop in
  `ImagePullBackOff`. Self-Managed 1.1.5 selects published task worker images;
  complete the [task worker image checks](/nvcf/1.1.0/self-managed/upgrade-recovery#task-worker-images)
  before maintenance. Those checks cover the two worker images only; tasks with
  secrets also pull the ESS agent image set in `ess-agent-container`
  (`ess-agent:1.4.5` unless you override it), so confirm each compute node pool
  can pull that image from your registry too. Tasks created after the upgrade use those
  images.
- A single-member OpenBao installation is not covered; see
  [Replace the OpenBao pods](#replace-the-openbao-pods).
- Whether Compute Plane 1.0.0 keeps working against Self-Managed
  1.1.5 during the upgrade window was not checked live.

## Before starting

1. Schedule a maintenance window. You will pause new function invocations and
   task submissions, and a single-member Cassandra restarts once.
1. You need an operator workstation with `kubectl`, `helm` 3.12 or later,
   `helmfile` 1.1.x (not 1.2.x), the `helm-diff` plugin 3.11 or later, `jq`,
   `curl`, Mike Farah's `yq` v4, and `ngc` if you download packages from NGC.
   Helmfile 1.2.0 removed sequential execution, which these stacks need. See
   [required tools](/nvcf/1.1.0/self-managed/helmfile-installation#required-tools-and-software).
1. Keep your existing control-plane and compute-plane directories, your
   `environments/<environment-name>.yaml`, your
   `secrets/<environment-name>-secrets.yaml`, and the registration file for each
   GPU cluster (`<cluster-name>-register-values.yaml`). The upgrade reuses each
   cluster's registration. Do not run `register-cluster` again.
1. Choose canary workloads: one serving function, one function that reads a
   secret, and one running task. You will test them before and after.
1. If you mirror images and charts to a private registry, mirror the 1.1.5
   artifacts first. [Get the 1.1.5 packages and check artifacts](#get-the-115-packages-and-check-artifacts) checks availability.
1. Before NATS restarts in [Apply the dependency stage](#apply-the-dependency-stage), patch every compute cluster running NVCA
   3.10.0 through 3.10.2 to a compatible 3.10.3 or later 3.10 patch. Preserve
   registration; see [NATS recovery](/nvcf/1.1.0/self-managed/upgrade-recovery#deployments-stuck-after-a-nats-restart).
   The Compute Plane 1.1.0 upgrade still waits until [Upgrade each compute cluster](#7-upgrade-each-compute-cluster).

## 1. Set deployment values

### Set up the session

Download this read-only script into one directory. [Check the starting state](#check-the-starting-state) and [Render and review the changes](#render-and-review-the-changes) use it to
stop an upgrade that would move an image to an older version:

<Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/4193cc1855f7ca7f0138a657422979b46a6eb61f05b493fc8dc50acdfc228003/versions/1.1.0/fern/assets/upgrade-checks/check-upgrade-image-moves.py" filename="check-upgrade-image-moves.py">Image move check</Download>

Use a dedicated shell. Replace every value in angle brackets. Run this block
first, and again after any shell restart.

```bash
bash
set -euo pipefail
umask 077

export WORKDIR="$HOME/nvcf-upgrade-1.1.5"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export OLD_CONTROL_STACK="/absolute/path/to/existing/nvcf-self-managed-stack"
export OLD_COMPUTE_STACK="/absolute/path/to/existing/nvcf-compute-plane-stack"
export HELMFILE_ENV="<environment-name>"
export CONTROL_CONTEXT="<control-plane-kube-context>"
export BACKUP_DIR="$WORKDIR/backup"
mkdir -p "$WORKDIR" "$BACKUP_DIR"

# One kubeconfig per cluster, so a stray context switch cannot redirect a command.
export CONTROL_KUBECONFIG="$WORKDIR/control.kubeconfig"
kubectl config view --minify --flatten --raw --context "$CONTROL_CONTEXT" > "$CONTROL_KUBECONFIG"

kc() { kubectl --kubeconfig "$CONTROL_KUBECONFIG" "$@"; }
control_helmfile() {
  (cd "$CONTROL_STACK" && HELMFILE_ENV="$HELMFILE_ENV" \
    helmfile --environment default --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
wait_workloads() {
  local kubeconfig="$1" namespace="$2" workload strategy
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get deployments,statefulsets -o name |
    while IFS= read -r workload; do
      strategy=$(kubectl --kubeconfig "$kubeconfig" -n "$namespace" \
        get "$workload" -o jsonpath='{.spec.updateStrategy.type}')
      if [ "$strategy" = "OnDelete" ]; then
        printf '%s uses OnDelete; replace its pods with the procedure in this guide\n' "$workload"
        continue
      fi
      kubectl --kubeconfig "$kubeconfig" -n "$namespace" rollout status "$workload" --timeout=15m
    done
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get pods
}
declare -f kc control_helmfile wait_workloads > "$WORKDIR/helpers.sh"
declare -p WORKDIR UPGRADE_SCRIPTS OLD_CONTROL_STACK OLD_COMPUTE_STACK HELMFILE_ENV \
  CONTROL_CONTEXT BACKUP_DIR CONTROL_KUBECONFIG > "$WORKDIR/session.sh"

kc config current-context
kc get nodes
```

All nodes must be `Ready`. To resume later, start a new `bash`, run
`set -euo pipefail`, set `WORKDIR` to the directory you used above
(`export WORKDIR="$HOME/nvcf-upgrade-1.1.5"`), then
`source "$WORKDIR/session.sh"` and `source "$WORKDIR/helpers.sh"`. The saved kubeconfig holds credentials, so keep
`$WORKDIR` private.

### Get the 1.1.5 packages and check artifacts

Both exact target packages must be published and available to your customer
account before maintenance. Stop if either download or package-stamp check
fails. The current 1.0.4 manifest does not select or qualify these targets.

Download the three target packages next to each other, in a new directory, so
the relative paths in the Helmfiles resolve the same way they did for your current release.

```bash
export STACK_VERSION="1.1.5"
export COMPUTE_STACK_VERSION="1.1.0"
# Observability is optional: export OBSERVABILITY_STACK_VERSION=1.0.0 before
# this block to stay on the version you run now. An existing value is kept.
export OBSERVABILITY_STACK_VERSION="${OBSERVABILITY_STACK_VERSION:-1.1.0}"
cd "$WORKDIR"

ngc registry resource download-version "nvidia/nvcf/nvcf-self-managed-stack:${STACK_VERSION}"
mkdir -p nvcf-self-managed-stack
tar -xzf "nvcf-self-managed-stack_v${STACK_VERSION}/nvcf-self-managed-stack-${STACK_VERSION}.tar.gz" \
  -C nvcf-self-managed-stack
rm -rf "nvcf-self-managed-stack_v${STACK_VERSION}"

ngc registry resource download-version "nvidia/nvcf/nvcf-compute-plane-stack:${COMPUTE_STACK_VERSION}"
mkdir -p nvcf-compute-plane-stack
tar -xzf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}/nvcf-compute-plane-stack-${COMPUTE_STACK_VERSION}.tar.gz" \
  -C nvcf-compute-plane-stack
rm -rf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}"

ngc registry resource download-version "nvidia/nvcf/nvcf-observability-stack:${OBSERVABILITY_STACK_VERSION}"
mkdir -p observability
tar -xzf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}/nvcf-observability-stack-${OBSERVABILITY_STACK_VERSION}.tar.gz" \
  -C observability
rm -rf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}"

export CONTROL_STACK="$WORKDIR/nvcf-self-managed-stack"
export COMPUTE_STACK="$WORKDIR/nvcf-compute-plane-stack"
export OBSERVABILITY_STACK="$WORKDIR/observability"
declare -p STACK_VERSION COMPUTE_STACK_VERSION OBSERVABILITY_STACK_VERSION CONTROL_STACK COMPUTE_STACK OBSERVABILITY_STACK >> "$WORKDIR/session.sh"
ls "$CONTROL_STACK" "$COMPUTE_STACK" "$OBSERVABILITY_STACK"
```

If you obtain the packages another way, extract them to these three
directories. If the observability profile is enabled (`observability.profile` is
not `disabled`), the control-plane Helmfile reads
`../../observability/helmfile.d/01-observability.yaml.gotmpl`, which resolves to
`$WORKDIR/observability`. That directory must hold an Observability package,
either 1.1.0 or your existing 1.0.0. To keep 1.0.0, export
`OBSERVABILITY_STACK_VERSION=1.0.0` before running the download block, or copy
your existing 1.0.0 stack to that directory and skip its download. Leaving the stack only in
its original location makes the render fail. With the profile `disabled` the
directory is not read.

Check both new package stamps against the selected versions before maintenance:

```bash
check_selected_stack_versions() {
  local actual
  actual=$(cat "$CONTROL_STACK/VERSION") || return 1
  if [ "$actual" != "$STACK_VERSION" ]; then
    printf 'Self-Managed package must report %s; found %s\n' "$STACK_VERSION" "$actual" >&2
    return 1
  fi
  actual=$(cat "$COMPUTE_STACK/VERSION") || return 1
  if [ "$actual" != "$COMPUTE_STACK_VERSION" ]; then
    printf 'Compute Plane package must report %s; found %s\n' "$COMPUTE_STACK_VERSION" "$actual" >&2
    return 1
  fi
}
check_selected_stack_versions
declare -f check_selected_stack_versions >> "$WORKDIR/helpers.sh"
```

Stop on a missing or mismatched stamp. Obtain a correctly stamped, qualified
package through the release process; do not edit `VERSION`, replace immutable
artifacts or substitute another stack version.

Confirm the charts and images are reachable from your registry. Set the
repositories to the ones in your environment file (`global.helm.sources` and
`global.image`), or to your mirror:

If your environment file uses an HTTPS Helm repository
(`global.helm.sources.url`) instead of an OCI registry, add it under an alias
first and use the alias as `HELM_REPO` below. The password is read from
standard input so it stays out of the process list:

```bash
export HELM_REPOSITORY_CONFIG="$WORKDIR/helm-repositories.yaml"
read -r -s -p "Repository password: " HELM_PASSWORD; echo
printf '%s' "$HELM_PASSWORD" | helm repo add nvcf-check "<https-repository-url>" \
  --username "<username>" --password-stdin
unset HELM_PASSWORD
helm repo update nvcf-check
export HELM_REPO="nvcf-check"
```

For an OCI registry, set `HELM_REPO` to its address instead:

```bash
export HELM_REPO="oci://<registry>/<repository>"
export IMAGE_REPO="<registry>/<repository>"
artifact_missing=0

while IFS=: read -r chart version; do
  if helm show chart "$HELM_REPO/$chart" --version "$version" > /dev/null 2>&1; then
    echo "ok       $chart:$version"
  else
    echo "MISSING  $chart:$version"
    artifact_missing=1
  fi
done <<'EOF'
helm-nvcf-cassandra:0.26.0
helm-nvcf-nats:0.9.0
helm-nvcf-openbao-server:0.36.1
helm-nvcf-pki:0.2.0
nvcf-gateway-routes:1.18.3
helm-nvcf-api:1.35.0
helm-nvcf-api-keys:1.9.4
helm-admin-token-issuer-proxy:1.7.0
helm-nvcf-ess-api:1.11.1
helm-nvcf-sis:2.6.3
helm-nvcf-nvct-api:1.11.0
helm-nvcf-notary-service:1.9.1
helm-nvcf-invocation-service:1.9.0
helm-nvcf-grpc-proxy:1.14.0
helm-nvcf-rate-limiter:1.4.1
helm-reval:1.6.0
helm-nvcf-nats-auth-callout-service:1.4.2
helm-nvcf-event-ledger:0.10.0
helm-nvcf-state-metrics:1.0.7
helm-nvcf-function-autoscaler:0.8.0
EOF

while IFS= read -r image; do
  if docker manifest inspect "$IMAGE_REPO/$image" > /dev/null 2>&1; then
    echo "ok       $image"
  else
    echo "MISSING  $image"
    artifact_missing=1
  fi
done <<'EOF'
nvcf-service-oss:1.28.4
nvcf-api-keys-service:1.10.10
nvct-service-oss:1.70.4
nvcf-ess:0.5.7
icms-service-oss:0.8.5
nvcf-notary:1.14.8
nvcf-invocation-service:0.14.2
nvcf-grpc-proxy:1.35.2
nvcf-event-ledger:0.21.0
admin-token-issuer-proxy:1.1.3
nvcf-ratelimiter:1.18.3
reval-server:0.20.4
nvcf-nats-auth-callout-service:0.8.8
nvcf-state-metrics-service:1.24.1
nvcf-function-autoscaler:1.23.1
nvcf-openbao:2.6.3-nv-1.3.7
nvcf-openbao-migrations:0.23.1
nvcf-cassandra-migrations:0.19.4
cassandra:5.0.9-nv-2.0.7
EOF
test "$artifact_missing" -eq 0
```

The list covers the images that change plus the data-service images. The
[artifact manifest](/nvcf/1.1.0/manifest/artifacts) has the complete list, including optional
add-ons. Fix every `MISSING` line before continuing; a missing image shows up
later as `ImagePullBackOff` in the middle of a stage.

### Decide the HA mode and prepare configuration

#### Choose the HA mode

| Mode | Result | Needs |
| --- | --- | --- |
| `none` | Keeps your current footprint: no extra replicas, no new PodDisruptionBudgets. | Nothing. |
| `preferred` (default) | Replica-safe services run 2 replicas. Cassandra, NATS and OpenBao run 3 members with soft anti-affinity and zone spread. Pods still schedule when capacity is short. | At least 3 schedulable nodes. |
| `enforced` | Same sizing with hard placement. Pods stay `Pending` rather than co-locate. | At least 3 nodes, 3 zones with capacity, `topology.kubernetes.io/zone` labels, and a `WaitForFirstConsumer` StorageClass. |

If this is a single-node cluster, or you are not ready to add capacity, choose
`none`. You can move to HA later. If your Cassandra runs a single member, first
follow [Expand a single-member Cassandra first](#expand-a-single-member-cassandra-first).
The full behavior, including zone spread, is in the
[high availability guide](/nvcf/1.1.0/self-managed/high-availability).

Check what your cluster offers:

```bash
kc get nodes -L topology.kubernetes.io/zone
kc get storageclass
kc get nodes --no-headers | wc -l
```

For `enforced`, the StorageClass your Cassandra, NATS and OpenBao volumes use
must show `WaitForFirstConsumer` as its volume binding mode. Replica-safe
services roll with `maxUnavailable: 0` and `maxSurge: 1`, so leave room for one
extra pod per service.

#### Carry your settings into the new packages

The new `base.yaml` files contain the 1.1.5 defaults and chart settings. Copy
only your own changes into new environment files. Do not copy the whole old
file, because that would overwrite new defaults with old ones.

```bash
# Your customizations: the lines in your environment file that differ from the old base.
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_CONTROL_STACK/environments/base.yaml" \
  "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/control-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/control-overrides.diff"

cp "$CONTROL_STACK/environments/base.yaml" "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml"
```

Open `$WORKDIR/control-overrides.diff` and re-apply each `+` line in
`$CONTROL_STACK/environments/$HELMFILE_ENV.yaml`. Typical lines are your
registry and chart source (`global.helm.sources`, `global.image`), Gateway
settings, storage class and size, node selectors and `imagePullSecrets`. Do not
carry over image tag or chart version overrides unless you deliberately pin
them; the 1.1.5 defaults are the qualified set. The 1.1.5 `base.yaml` no longer
pins `cassandra.migrations.image.tag`, so the chart default (0.19.4) applies.
If you pin that tag, remove the pin or set 0.19.4 or later: an older pin skips
the schema changes in [Apply the dependency stage](#apply-the-dependency-stage).

The 1.1.5 defaults pull the NATS server, nats-box and config reloader images from
public registries (`docker.io/synadia/nats-server`,
`docker.io/natsio/nats-box` and `docker.io/natsio/nats-server-config-reloader`).
Your release pulled the server and nats-box from the NVCF catalog. If your cluster cannot
pull from public registries, mirror the three images and set them in the new
environment file, as in this example (the reloader uses `nats.reloader.image.*`):

```yaml
nats:
  container:
    image:
      registry: <your-registry>
      repository: <your-repository>/nats-server
  natsBox:
    container:
      image:
        registry: <your-registry>
        repository: <your-repository>/nats-box
  reloader:
    image:
      registry: <your-registry>
      repository: <your-repository>/nats-server-config-reloader
```

If you set none of these settings and your cluster can pull from `docker.io`,
there is nothing to do. The 1.1.5 NATS pods run as a non-root user, so confirm in
[Apply the dependency stage](#apply-the-dependency-stage) that each NATS pod becomes Ready.

Then set the HA mode and, if you want to turn Event Ledger off, its switch. In
the environment file:

```yaml
highAvailability:
  mode: preferred   # none | preferred | enforced
addons:
  eventLedger:
    enabled: true   # set false to skip the new Event Ledger release
```

Merge these keys into the existing `addons:` section; do not duplicate it.

#### Check the secrets file for replica overrides

Under `preferred` and `enforced`, the render fails if the secrets file sets a
value that conflicts with the HA sizing, for example `replicaCount: 1`. Look
for them now:

```bash
grep -nE 'replica(Count|s):' "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  || echo "no replica overrides"
```

Remove any replica override the HA sizing should control, or choose
`highAvailability.mode: none`. Record the mode you chose:

```bash
yq '.highAvailability' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
```

## 2. Check the starting state and take backups

### Check the starting state

Do not continue unless every check passes.

```bash
# Releases deployed
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json > "$WORKDIR/control-before.json"
jq -r '.[] | "\(.namespace)\t\(.name)\t\(.chart)\t\(.status)"' "$WORKDIR/control-before.json" | sort
test -z "$(jq -r '.[] | select(.status != "deployed") | .name' "$WORKDIR/control-before.json")" \
  && echo "all releases deployed"

# Cassandra: every member UN
for pod in $(kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra \
  -o jsonpath='{.items[*].metadata.name}'); do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool status
  break
done

# Workloads ready everywhere; sis only when the ICMS release is installed
namespaces="cassandra-system nats-system vault-system nvcf api-keys ess"
if jq -e '.[] | select(.name == "sis")' "$WORKDIR/control-before.json" > /dev/null; then
  namespaces="$namespaces sis"
fi
for namespace in $namespaces; do
  kc -n "$namespace" get pods -o json | jq -e '
    (.items | length > 0) and
    any(.items[]; any(.status.conditions[]?; .type == "Ready" and .status == "True")) and
    all(.items[]; .metadata.deletionTimestamp == null and
      (.status.phase == "Succeeded" or
        any(.status.conditions[]?; .type == "Ready" and .status == "True")))'
done

# Record replica counts and images before the upgrade
kc get deploy,sts -A -o wide > "$WORKDIR/workloads-before.txt"
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" before \
  "$CONTROL_KUBECONFIG" "$WORKDIR/images-before.tsv"
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml > "$WORKDIR/receipt-before.yaml" 2>/dev/null || true
```

Cassandra must report every member `UN`. The OpenBao check function used in
this guide, and again in [Apply the dependency stage](#apply-the-dependency-stage), is:

```bash
check_openbao() {
  local pod pods
  pods=$(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}') || return 1
  [ -n "$pods" ] || { echo "No OpenBao server pods found" >&2; return 1; }
  for pod in $pods; do
    echo "== $pod =="
    kc -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json' || return 1
  done
}
declare -f check_openbao >> "$WORKDIR/helpers.sh"
check_openbao
```

Every OpenBao member must be `initialized: true` and `sealed: false`, with the
same `cluster_id`. Exactly one reports `is_self: true`; standbys report
`is_self: false` or omit the field. Record the cluster ID.

Run your canary workloads now and record their results. With the configured
`nvcf-cli`:

```bash
export NVCF_CLI="/absolute/path/to/nvcf-cli"
export NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
export FUNCTION_ID="<existing-function-id>"
export VERSION_ID="<existing-version-id>"
export CANARY_REQUEST='{"input":"your-known-canary-input"}'
declare -p NVCF_CLI NVCF_CLI_CONFIG FUNCTION_ID VERSION_ID CANARY_REQUEST >> "$WORKDIR/session.sh"

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
  --request-body "$CANARY_REQUEST" --timeout 120
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING
```

#### Back up persistent state

Create restorable backups before you change anything. Follow your storage
provider's procedure for volume snapshots as well; the commands below create
the application-level backups.

```bash
CASSANDRA_REPLICAS=$(kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool snapshot -t pre-1-1-5
done
```

The Cassandra snapshot stays on each persistent volume. It is not an off-cluster
backup, so also copy or snapshot the volumes.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from
a standby fails with `incomplete snapshot`.

```bash
OPENBAO_ACTIVE=""
for pod in $(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  if kc -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
    OPENBAO_ACTIVE="$pod"
  fi
done
echo "active member: $OPENBAO_ACTIVE"

kc -n vault-system get secret openbao-server-root-token -o jsonpath='{.data.root_token}' | base64 -d |
  kc -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
  'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-1-5.snap'
kc -n vault-system cp -c openbao "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-1-5.snap" \
  "$BACKUP_DIR/openbao-pre-1-1-5.snap"
kc -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-1-5.snap

# The snapshot restores only with the unseal key and root token.
kc -n vault-system get secret openbao-server-unseal -o yaml > "$BACKUP_DIR/openbao-server-unseal.yaml"
kc -n vault-system get secret openbao-server-root-token -o yaml > "$BACKUP_DIR/openbao-server-root-token.yaml"

# Helm values of every release, and your configuration files.
mkdir -p "$BACKUP_DIR/helm-values"
jq -r '.[] | "\(.namespace) \(.name)"' "$WORKDIR/control-before.json" |
  while read -r namespace release; do
    helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" -n "$namespace" -o yaml \
      > "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
  done
cp "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" "$BACKUP_DIR/old-environment.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" "$BACKUP_DIR/old-secrets.yaml"
ls -l "$BACKUP_DIR"
```

The unseal and root-token files, the secrets copy and the exported values
contain credentials. Protect them like the snapshot. NVCF NATS streams use
memory storage, so a NATS backup holds stream metadata only. Do not continue
until the Cassandra and OpenBao backups exist.

### Render and review the changes

Render into a new directory, then review the diff the first stage would apply.

```bash
make -C "$CONTROL_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" OUTPUT_DIR="$WORKDIR/control-render"

# Images the 1.1.5 render pulls; make sure each one is in your registry or mirror.
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/control-render" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u
```

The render can include secrets, so keep it private. Check that the namespaces
match your cluster and that no image lacks the registry you expect.

Compare the render with the images that run now. The check fails if any image
would move to an older version, which happens when an old pin survives in your
environment file:

```bash
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" compare \
  "$WORKDIR/images-before.tsv" "$WORKDIR/control-render"
```

Every `BACKWARD` line names an image and both versions. Stop on any of them. For
a pin you did not set on purpose, remove it as described in
[Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration), render again and
re-run the check.

Before maintenance, verify the receipt image from the rendered Job. Its default
is `docker.io/alpine/k8s:1.33.1`; it does not follow `global.image`. For a mirror,
set `upgradeReceipt.image.registry` and `upgradeReceipt.image.repository`, then
render again. Use the resulting image reference for the check.

Run a temporary pull-test Pod in the Job's namespace on each eligible node pool,
using that image, its pull credentials and scheduling settings. Set
`imagePullPolicy: Always` and run `true` instead of the receipt script. Confirm
the image pulls and the Pod succeeds, then remove the test Pod. A host-side
`docker manifest inspect` alone does not verify node access.

If you run tasks, complete the [task worker image checks](/nvcf/1.1.0/self-managed/upgrade-recovery#task-worker-images)
using this target render and the Self-Managed 1.1.5 inventory before proceeding.

## 3. Refresh the hostname policy

This is a required one-time step for release 1.0.1, because
it runs gateway routes chart 1.18.2.

The 1.18.2 chart installed a hostname policy that denies the admin issuer proxy
route patch, and that release applies before `ingress`. The target checks that
the pinned chart (1.18.3) carries the corrected policy and then applies only the
`ingress` release.

```bash
make -C "$CONTROL_STACK" apply-hostname-policy HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
helm --kubeconfig "$CONTROL_KUBECONFIG" -n envoy-gateway-system status ingress
kc get httproutes -A
```

The `ingress` release must be `deployed`, and your existing routes and
canaries must still work. Stop on any error; do not disable admission to get
past it. If the target reports that the pinned chart "still reserves hostnames",
the packages do not match the selected release; recheck [Get the 1.1.5 packages and check artifacts](#get-the-115-packages-and-check-artifacts).

## 4. Drain pending work

Stop new production requests and task submissions at their callers. NATS
request queues use memory storage, and the dependency stage can restart NATS.
Wait for queued work to finish; do not cancel it to get an empty queue.

For every affected function and version, use a key with `queue_details` scope
(`NVCF_API_KEY`). The task command below uses a separate key with `list_tasks`
scope (`NVCF_NVCT_API_KEY`) and your Tasks endpoint (`base_nvct_url` and
`nvct_host` in the CLI configuration). Verify both with the
[CLI authentication settings](/nvcf/1.1.0/overview/cli) before you start:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING
```

Queue sizes must reach zero. Record the identities and results of running
tasks. Then check the control plane's own single-replica memory streams on every
NATS server, through the monitoring port 8222:

```bash
NATS_MONITOR_PORT=18223
NATS_PODS=$(kc -n nats-system get pods -o json | jq -r '.items[] |
  select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
NATS_DRAIN_DIR="$WORKDIR/nats-drain-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir "$NATS_DRAIN_DIR"
for pod in $NATS_PODS; do
  (
    kc -n nats-system port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
      > "$NATS_DRAIN_DIR/$pod.log" 2>&1 &
    forward_pid=$!
    trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
    for attempt in {1..30}; do
      kill -0 "$forward_pid"
      grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log" && break
      sleep 1
    done
    grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log"
    curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
      "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
      > "$NATS_DRAIN_DIR/$pod.json"
    jq -e '.account_details | type == "array"' "$NATS_DRAIN_DIR/$pod.json"
    jq -e 'all(.account_details[];
      ((.stream_detail // []) | type == "array") and
      all((.stream_detail // [])[];
        (.config.storage | type == "string") and
        (.state.messages | type == "number")))' "$NATS_DRAIN_DIR/$pod.json"
  )
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
  select(.config.storage == "memory" and (.config.num_replicas // 1) == 1) |
  {name: .name, messages: .state.messages}] |
  if length > 0 and all(.[]; .messages == 0)
  then . else error("Memory streams missing or not drained; do not roll NATS") end' \
  "$NATS_DRAIN_DIR/"*.json
```

Every single-replica memory stream must show zero messages. A failed or
slow read, a missing stream or a nonzero count blocks the upgrade. A server whose
accounts hold no streams reports no stream details, which counts as empty; the
final check still requires at least one single-replica memory stream. Each attempt writes to a new directory, so files from an
earlier attempt cannot satisfy the check. Wait for the work to drain
and repeat; do not discard messages. If your NATS monitoring is configured
differently, use its endpoint and keep the same per-server check.
For retained creation messages or stuck deployments, follow the
[recovery steps](/nvcf/1.1.0/self-managed/upgrade-recovery#inspect-retained-creation-messages)
before proceeding.

## 5. Apply control-plane stages in order

### Expand a single-member Cassandra first

Do this only if `CASSANDRA_REPLICAS` from [Back up persistent state](#back-up-persistent-state)
is `1` and you chose `preferred` or `enforced`. Skip it if Cassandra already runs
3 members, or if you chose `none`. Run it in the maintenance window, after you
drain pending work: while `system_auth` is being raised and repaired, new logins
can fail.

Do not let the upgrade expand Cassandra from one member to three. The chart
raises the `system_auth` replication factor to the new member count in a hook that
runs after the StatefulSet scales, and it does not repair the data. The new
members then hold none of the existing login data, so administrator and service
logins can fail while every member shows `UN`, and the dependency stage times out.
Expand and repair now, on the release you run today, so the upgrade starts from a
normal three-member cluster.

Place the new members in different zones. A member's volume stays in the zone
where the member first starts, and the placement rules that the upgrade applies
later cannot move it. Under `enforced`, two members in one zone leave a pod
`Pending` during the upgrade. If every schedulable node has a
`topology.kubernetes.io/zone` label, add a zone spread rule to the StatefulSet
before you scale. The rule is hard under `enforced` and soft under `preferred`, so
a zone without capacity never leaves Cassandra `Pending` under `preferred`. This
restarts the single member once, and the upgrade replaces the rule with the
chart's own. If any node has no zone label, skip this rule: a node without the
label cannot take a pod that has it.

```bash
kc get nodes -L topology.kubernetes.io/zone
HA_MODE=$(yq -r '.highAvailability.mode // "preferred"' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml")
case "$HA_MODE" in
  enforced) SPREAD=DoNotSchedule ;;
  preferred) SPREAD=ScheduleAnyway ;;
  *) echo "Unsupported highAvailability.mode: $HA_MODE" >&2; exit 1 ;;
esac
kc -n cassandra-system patch statefulset cassandra --type merge -p "{\"spec\":{\"template\":{\"spec\":{\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"$SPREAD\",\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/name\":\"cassandra\"}}}]}}}}"
kc -n cassandra-system rollout status statefulset/cassandra --timeout=10m
```

Scale the StatefulSet directly. Do not set `cassandra.replicaCount` and apply the
old stack, because that runs the chart hook with the same problem. The block reads
the member count that the backup step saved, so it also works after a shell
restart, and it shows the live count. Scaling to 3 is safe to repeat. Wait for all
three members to be `UN`:

```bash
CASSANDRA_REPLICAS=$(cat "$BACKUP_DIR/cassandra-replicas")
test "$CASSANDRA_REPLICAS" -eq 1
kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}{" members now\n"}'
kc -n cassandra-system scale statefulset cassandra --replicas=3
kc -n cassandra-system rollout status statefulset/cassandra --timeout=20m
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
```

Every member must report `UN`. If a pod stays `Pending`, check the node and volume
capacity before you continue. Confirm that each member runs in a different zone:

```bash
kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra -o wide
kc get nodes -L topology.kubernetes.io/zone
```

Raise the `system_auth` replication factor to 3, repair it on every member, then
confirm that the administrator can log in on every member. The password is that of
the `cassandra` administrator role, not the service role password in your secrets
file. The commands read it from standard input into a temporary protected file, so
do not enable shell tracing.

```bash
CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
cql_on cassandra-0 "ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full system_auth
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  cql_on "$pod" "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
  echo "administrator login ok on $pod"
done
```

All three members must print `administrator login ok`. If any login fails, stop.
Do not continue to the upgrade. Restore from the backups in
[Back up persistent state](#back-up-persistent-state) with your reviewed recovery
procedure, and contact NVIDIA support with the output.

Then raise the application keyspaces to 3 and repair them on every member. The
loop alters only the keyspaces your installation has:

```bash
EXISTING_KEYSPACES=$(cql_on cassandra-0 "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql_on cassandra-0 "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD
```

Repeat your canary invocation from [Check the starting state](#check-the-starting-state)
and confirm that the services report no authentication errors:

```bash
AUTH_LOGS=$(kc -n nvcf logs deploy/nvcf-api --since=10m)
printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true
```

The count must be `0`. [Raise replication for HA](#raise-replication-for-ha) later in
this guide repeats the application keyspace step, which is safe to run again.

### Apply the dependency stage

This stage updates Cassandra, NATS, cert-manager, OpenBao and the PKI. Review
the diff first, and stop on unexpected deletions or version overrides:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets
```

Apply it:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
  kc -n "$namespace" get jobs
done
```

What you will see:

- Cassandra restarts one pod at a time because of the config change and the
  new server image (2.0.7), and grows to 3 members under `preferred` or
  `enforced`. A single-member cluster is unavailable while its one pod restarts.
- Migrations run as Helm hook jobs. They must complete successfully. The
  Cassandra migrations image moves to 0.19.4 from 0.17.6. The
  new migrations only add schema: nullable columns on `sis_api.requests`,
  `sis_api.instances` and the `event_ledger` tables, and indexes on the new
  `sis_api` `creation_bucket` column. They drop nothing. Existing rows keep
  null in the new columns, and nothing backfills them. Building the indexes
  reads the existing rows, so a large `sis_api` table can make that job run
  longer.
- The OpenBao hooks (image 0.23.1) refresh the service, admin issuer and Event
  Ledger configuration, and add a signing role for gRPC proxy events. They
  only add or merge configuration and keep existing roles.
- OpenBao shows the new template, but its pods still run the old image
  until [Replace the OpenBao pods](#replace-the-openbao-pods). `wait_workloads` reports it as `OnDelete`.

Check that the data is intact, not only that pods are Ready:

```bash
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
kc -n cassandra-system get jobs
kc -n vault-system get jobs
```

All Cassandra members must be `UN` and every migration job `Complete`.
Investigate any failed job before continuing; do not disable hooks.

### Replace the OpenBao pods

The OpenBao StatefulSet uses `OnDelete`: Helm updated its template, but the
running pods keep their old image. Replace them one at a time, standbys first.
Never delete a persistent volume claim, reinitialize OpenBao, or replace more
than one member at once. This procedure is the 2.6.2 to 2.6.3 patch procedure
from the [OpenBao patch upgrade runbook](/nvcf/1.1.0/self-managed/open-bao-patch-upgrade).
Use that runbook if a step fails or if you customized plugin registration,
policies or PKI.

This procedure is written for the supported OpenBao deployment, a Raft cluster
of several members. Replace a standby first so that leadership does not move
twice. Check the member count:

```bash
kc -n vault-system get statefulset openbao-server -o jsonpath='{.spec.replicas}{"\n"}'
```

If it prints `1`, stop. A single-member OpenBao is not a supported deployment and
this guide has no procedure for it: there is no standby to replace first and no
leadership handover, and OpenBao is unavailable while its one pod restarts and
unseals. Do not continue on your own. Confirm the Raft snapshot from the backup
step and use the
[OpenBao patch upgrade runbook](/nvcf/1.1.0/self-managed/open-bao-patch-upgrade)
to plan the change with your NVCF contact.

Confirm the new template and the update strategy:

```bash
kc -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}'
check_openbao
```

The server image must be `nvcf-openbao:2.6.3-nv-1.3.7` and the strategy
`OnDelete`. If either is different, stop. Identify the standbys (the members
that do not report `is_self: true`) and replace one:

```bash
POD="<standby-pod-name>"
kc -n vault-system delete pod "$POD" --wait=true --timeout=3m
kc -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=5m
check_openbao
```

If the new pod has not appeared yet, wait for it before repeating the wait. The
replacement must report `version: "2.6.3"`, `sealed: false` and the original
`cluster_id`. If it is Ready but still sealed, let auto-unseal finish and run
`check_openbao` again. Repeat for the second standby only after all three
members are healthy.

Then replace the former active member. Transfer leadership first:

```bash
ACTIVE_POD="<current-active-pod-name>"
set -o pipefail
kc -n vault-system get secret openbao-server-root-token -o json |
  jq -er '.data.root_token | @base64d' |
  kc -n vault-system exec -i "$ACTIVE_POD" -c openbao -- sh -ec '
    IFS= read -r BAO_TOKEN
    test -n "$BAO_TOKEN"
    export BAO_TOKEN
    export BAO_ADDR=http://127.0.0.1:8200
    bao operator step-down
  '
check_openbao
```

Wait until a 2.6.3 member reports `is_self: true` and the old active reports
standby. A successful request alone does not confirm the transfer. Then replace
the former active with the same `delete`, `wait`, `check_openbao` sequence. Do
not enable shell tracing or print the token.

Verify all three members and their images:

```bash
check_openbao
kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'
```

All three must be unsealed on 2.6.3, with one active member and the original
cluster ID. Updating the injector affects only new pods. The services the core
stage rolls in the next step pick up the new agent; roll any other injected
workload through its normal deployment procedure, one service at a time, and
check its authentication and rendered secrets. A single-replica service can be
interrupted during that roll, so never restart all consumers together.

### Apply the core services

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets
control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
done
kc get httproutes -A
```

Core releases must be `deployed`, their workloads ready and their routes
`Accepted`. The new `event-ledger` release appears in the `nvcf` namespace
unless you turned it off. Under `preferred` or `enforced`, services scale to 2
replicas and roll with `maxUnavailable: 0`, so a rollout needs room for one
extra pod per service. Check any add-on namespaces you enable (LLM, vanity
gateway, UI) the same way.

Repeat your canary invocation now. It must return the result you recorded.

### Apply state metrics and autoscaling

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl status
wait_workloads "$CONTROL_KUBECONFIG" nvcf
```

Keep your existing observability profile. Enabled state-metrics and autoscaler
releases must be `deployed` and healthy. Do not turn off state metrics to get
past a validation failure; the `control` and `all` profiles require it. Also
reapply the observability infrastructure stage. It applies the Observability
version in `$WORKDIR/observability`. With 1.0.0, changed configuration can still
update releases. With 1.1.0, the new charts replace the 1.0.0 ones. Review the
diff first:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl diff --suppress-secrets
```

Stop on unexpected changes. Then apply the stage:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl apply
```

### Raise replication for HA

Skip this step if you chose `highAvailability.mode: none`.

After the dependency stage, Cassandra has 3 members, but your application
keyspaces keep the replication factor they were created with, so losing one
Cassandra node can still make data unavailable. Wait until all three Cassandra
pods are Ready, then raise each keyspace to RF=3 and repair, one member at a
time. The loop below alters only the keyspaces your installation has and skips
the rest. It validates the deployed data-center name against the live database
before any ALTER. Credentials travel over stdin into a temporary protected
[cqlsh credentials file](https://cassandra.apache.org/doc/stable/cassandra/managing/tools/cqlsh.html#credentials).
Do not enable shell tracing.

```bash
CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i cassandra-0 -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$1"
)
LIVE_DC=$(cql "SELECT JSON data_center FROM system.local;" |
  sed -n '/^[[:space:]]*{/{s/^[[:space:]]*//;p;}' |
  jq -ers 'if length == 1 then .[0].data_center else error("Missing data-center row") end')
test "$LIVE_DC" = "$CASSANDRA_DC"
cql "SELECT keyspace_name, replication FROM system_schema.keyspaces;"
EXISTING_KEYSPACES=$(cql "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD
```

Until the repairs finish, reads at `LOCAL_ONE` can miss rows that exist only on
the original replica. The application keyspaces must now report RF=3 for the validated data center.

JetStream streams are created with the replication factor current at creation
time. The `icms` streams raise themselves to the configured factor when it
starts. Streams created earlier by `nvcf-api` and `invocation-service` stay at
1 replica until you edit them. List them and check:

```bash
kc -n nats-system exec deploy/nats-box -- nats stream report
```

The `Replicas` column lists each member that holds a copy of the stream. A
highly available stream shows three (`nats-0`, `nats-1`, `nats-2`); a stream
created before HA shows one. Raise each stream that shows one:

```bash
STREAM="<stream-name>"
kc -n nats-system exec deploy/nats-box -- nats stream edit "$STREAM" --replicas=3 --force
kc -n nats-system exec deploy/nats-box -- nats stream info "$STREAM"
```

Replace `<stream-name>` with the name from the report, run both commands for
each such stream, and confirm `nats stream info` shows `Replicas: 3`. Use 3, not
2: a 2-replica stream loses quorum when either member is down. The NATS
commands run in the `nats-box` deployment because the `nats` server container
does not include the CLI.

### Write the upgrade receipt

Run this only after every earlier stage succeeded.

Complete the [receipt image pull check](#render-and-review-the-changes)
before reaching this stage. Recheck it if the image or pull credentials changed.

```bash
check_selected_stack_versions
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kc -n nvcf get job upgrade-receipt --ignore-not-found
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml
test "$(kc -n nvcf get configmap nvcf-upgrade-receipt \
  -o jsonpath='{.data.installed_stack_version}')" = "$STACK_VERSION"
echo "receipt matches the selected Self-Managed release"
```

The receipt must match the selected Self-Managed 1.1.5 release. It does not
record the outcome of workload checks. The hook job is removed after ten
minutes; the ConfigMap is the lasting record.

## 6. Verify the control plane

Check releases, versions and images:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kc -n nvcf get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n api-keys get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n sis get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n ess get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
```

Key images you should see:

| Service | Image |
| --- | --- |
| nvcf-api | `nvcf-service-oss:1.28.4` |
| api-keys | `nvcf-api-keys-service:1.10.10` |
| nvct-api | `nvct-service-oss:1.70.4` |
| ess-api | `nvcf-ess:0.5.7` |
| sis (ICMS) | `icms-service-oss:0.8.5` |
| notary-service | `nvcf-notary:1.14.8` |
| invocation-service | `nvcf-invocation-service:0.14.2` |
| grpc-proxy | `nvcf-grpc-proxy:1.35.2` |
| event-ledger | `nvcf-event-ledger:0.21.0` |
| admin-issuer-proxy | `admin-token-issuer-proxy:1.1.3` |
| OpenBao server | `nvcf-openbao:2.6.3-nv-1.3.7` |
| Cassandra | `cassandra:5.0.9-nv-2.0.7` |

If you chose HA, confirm the replicas and spread (these are the same checks the
[high availability guide](/nvcf/1.1.0/self-managed/high-availability#validation) uses):

```bash
kc get nodes -L topology.kubernetes.io/zone
kc -n cassandra-system get pods -o wide
kc -n nats-system get pods -o wide
kc -n vault-system get pods -o wide
kc -n nvcf get deploy nvcf-api -o wide
kc -n nvcf get deploy invocation-service grpc-proxy-deployment -o wide
kc -n ess get deploy ess-api-deployment -o wide
kc get pdb -A
```

Replica-safe services show 2 replicas, and Cassandra, NATS and OpenBao show 3.
`invocation-service` and `grpc-proxy` stay at 1 replica if you set a fixed
worker callback URL. Do not resume traffic until the checks pass. Repeat the
canary from [Check the starting state](#check-the-starting-state), create and invoke a new function with a secret, and confirm
queues make progress. Existing workers keep the images they started with.

## 7. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Use the
cluster's existing registration; the cluster, group and account identity must
not change.

### Set the variables

```bash
export COMPUTE_CONTEXT="<gpu-cluster-kube-context>"
export CLUSTER_NAME="<gpu-cluster-name>"
export NCA_ID="<existing-account-id>"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export COMPUTE_KUBECONFIG="$WORKDIR/compute-$CLUSTER_NAME.kubeconfig"
kubectl config view --minify --flatten --raw --context "$COMPUTE_CONTEXT" > "$COMPUTE_KUBECONFIG"
kg() { kubectl --kubeconfig "$COMPUTE_KUBECONFIG" "$@"; }
record_identity() {
  kg -n nvca-operator get configmap nvcfbackend-self-managed \
    -o jsonpath='{.data.cluster-dto\.yaml}' |
    yq -o=json '{"clusterId": .clusterId, "clusterGroupId": .clusterGroupId,
      "clusterName": .clusterName, "ncaID": .ncaID}' |
    jq -eS 'if all(.[]; type == "string" and length > 0)
      then . else error("Missing cluster identity") end'
}
kg get nodes
record_identity
```

If the cluster name or account ID was unknown, set `CLUSTER_NAME` and `NCA_ID`
from this output before continuing. The registration file must exist and
match that identity; every node and the NVCA backend must be healthy:

```bash
test -s "$REGISTRATION_VALUES_DIR/$CLUSTER_NAME-register-values.yaml"
declare -p COMPUTE_CONTEXT CLUSTER_NAME NCA_ID REGISTRATION_VALUES_DIR COMPUTE_KUBECONFIG >> "$WORKDIR/session.sh"
declare -f kg record_identity >> "$WORKDIR/helpers.sh"
record_identity | tee "$WORKDIR/identity-before-$CLUSTER_NAME.json"
kg -n nvca-operator get nvcfbackends
kg -n nvca-operator get pods
kg -n nvca-system get pods
```

### Carry your settings into the compute package

```bash
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_COMPUTE_STACK/environments/base.yaml" \
  "$OLD_COMPUTE_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/compute-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/compute-overrides.diff"
cp "$COMPUTE_STACK/environments/base.yaml" "$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml"
```

Re-apply each `+` line from `compute-overrides.diff` in
`$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml` (registry, image settings,
observability profile and similar). Do not copy old `nvcaVersion`, `imageTag`
or chart pins over the 1.1.0 defaults. Leave `workerCallbackCaPem` empty unless
you serve worker callbacks over TLS with a private CA. If your clusters share
one environment file, do this once.

### Check images, render and review

```bash
make -C "$COMPUTE_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/compute-render-$CLUSTER_NAME" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u
```

Mirror every image the list shows (NVCA 3.18.6, shared storage 1.0.9, image
credential helper 0.11.4 and the BYOO collector 0.160.0-nv-0.4.7 are new
defaults) before you apply. Then review the diff:

```bash
(cd "$COMPUTE_STACK" && HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME="$CLUSTER_NAME" \
  NCA_ID="$NCA_ID" OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME" \
  helmfile --environment default --kubeconfig "$COMPUTE_KUBECONFIG" \
  --kube-context "$COMPUTE_CONTEXT" diff --suppress-secrets)
```

Expect the `nvca-operator` chart to move from 1.28.0 to 1.37.1 and the NVCA
agent to 3.18.6. Stop if the diff shows the registration or cluster identity
changing, or unexpected deletions.

### Apply

```bash
make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
wait_workloads "$COMPUTE_KUBECONFIG" nvca-operator
kg -n nvca-system get pods
kg -n nvca-operator get nvcfbackends
record_identity > "$WORKDIR/identity-after-$CLUSTER_NAME.json"
diff -u "$WORKDIR/identity-before-$CLUSTER_NAME.json" "$WORKDIR/identity-after-$CLUSTER_NAME.json" \
  && echo "cluster identity unchanged"
```

The operator and agent must be ready, the backend must report healthy, and
the identity files must match. If they differ, stop and inspect them before
doing anything else. Do not install a second copy of shared observability
resources on separate compute clusters; keep your existing configuration.

Confirm from the control plane that the cluster is registered and healthy:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" cluster list-registered
```

Existing function and task canaries must still work with their old worker
images. To move a function to the new worker defaults, replace its deployment
through the [deployment API or CLI](/nvcf/1.1.0/overview/cli#deploy-function); updating the
stack defaults alone does not change running workers or explicit per-function
image overrides. Plan for the interruption while a replacement rolls.

Repeat this step for each remaining GPU cluster, with its own `CLUSTER_NAME`,
`NCA_ID`, registration directory and kube context.

## 8. Resume workloads

Reapply the control plane and confirm it is stable:

```bash
make -C "$CONTROL_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
control_helmfile status
set +e
control_helmfile diff --suppress-secrets --detailed-exitcode
DIFF_EXIT=$?
set -e
echo "diff exit code: $DIFF_EXIT"
if [ "$DIFF_EXIT" -ne 0 ]; then
  echo "STOP: the final diff exited $DIFF_EXIT. Inspect it before you resume traffic." >&2
  exit "$DIFF_EXIT"
fi
```

The final diff must report no changes and exit 0. A nonzero exit means there
are pending changes or a command error. Resolve them and rerun this gate
before continuing:

1. Repeat the canary invocations and compare IDs, results and persistent data
   with the values from [Check the starting state](#check-the-starting-state).
1. Create and invoke a new function and a new task with the inputs you normally
   use, including one function that reads a secret.
1. Resume production submissions and watch queue progress and telemetry.

Keep `$WORKDIR`, the diffs, the render directories and your notes with the
installed configuration as the upgrade record.

## Troubleshooting

| Symptom | Likely cause | What to do |
| --- | --- | --- |
| Render fails with `overrides the highAvailability sizing` | The secrets file sets a replica count that conflicts with the HA sizing. | Remove the override, or set `highAvailability.mode: none`. See [Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration). |
| `ImagePullBackOff` in a stage | An image is not in your registry or mirror. | Re-run the [Get the 1.1.5 packages and check artifacts](#get-the-115-packages-and-check-artifacts) checks, mirror the missing image, and retry the stage. |
| Pods `Pending` under `enforced` | Fewer than 3 nodes or zones with capacity, or a StorageClass without `WaitForFirstConsumer`. | Add capacity or fix the StorageClass. `preferred` schedules them but does not give zone protection. |
| `apply-hostname-policy` reports the chart still reserves hostnames | The package does not match the selected release. | Recheck the extracted package against [Get the 1.1.5 packages and check artifacts](#get-the-115-packages-and-check-artifacts), then rerun. |
| The admin issuer route patch is denied | The hostname policy step was skipped. | Run [Refresh the hostname policy](#3-refresh-the-hostname-policy), then rerun the core stage. |
| A migration job failed | A dependency was not ready, or a hook could not reach its service. | Read the job logs (`kc -n <namespace> logs job/<name>`), fix the cause, and rerun the same stage. Do not disable hooks. |
| OpenBao member sealed after replacement | Auto-unseal has not finished. | Wait, then rerun `check_openbao`. If it stays sealed after five minutes, follow the runbook's recovery guidance. |
| Cassandra member not `UN` | The pod is restarting or joining. | Wait for it to become Ready and rerun `nodetool status`; do not continue while any member is down. |
| The image check prints `BACKWARD` | An old image pin survived in your environment file. | Stop. Remove pins you did not set on purpose ([Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration)) and render again. |
| A NATS pod is not Ready after [Apply the dependency stage](#apply-the-dependency-stage) | The 1.1.5 NATS pods run as a non-root user (uid 1000), or an image could not be pulled. | Read the pod events and logs. Confirm the images pull from your registry or mirror ([Decide the HA mode and prepare configuration](#decide-the-ha-mode-and-prepare-configuration)) and that the volume is writable by group 1000. |
| Compute plane identity changed | The registration file or `NCA_ID` was wrong. | Stop. Restore the original registration file and rerun the diff before applying. |

## Rollback boundary

If an upgrade fails, stop and use your deployment's reviewed recovery
procedure. Do not apply your old packages unchanged as a blanket rollback.
Helm rollback does not reverse stored data, does not validate a database or
OpenBao downgrade, and does not revert a Cassandra replication factor you
raised. The Cassandra migrations in 1.1.5 add columns and indexes, and a rollback
does not remove them. Older services have not been tested against that schema. Restore data only through the reviewed procedure for that datastore,
using the backups from [Back up persistent state](#back-up-persistent-state). After any recovery, verify the original cluster
identities, old-image canaries, durable data and availability. Changing which
package you use does not roll back a running deployment.

## Related

- [Compatibility matrix](/nvcf/1.1.0/release-notes/compatibility-matrix)
- [Artifact manifest](/nvcf/1.1.0/manifest/artifacts)
- [High availability guide](/nvcf/1.1.0/self-managed/high-availability)
- [OpenBao patch upgrade runbook](/nvcf/1.1.0/self-managed/open-bao-patch-upgrade)
- [Control-plane operations](/nvcf/1.1.0/self-managed/control-plane-operations)
- [0.6.2 to 1.0.1 upgrade](/nvcf/1.1.0/release-notes/0-6-1-to-1-0-1-upgrade)