> This page is for version Development.
> For other versions, use one of these documentation indexes:
> - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt
> - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt
> - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt
> - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt
> - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt
> - Development: https://docs.nvidia.com/nvcf/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.

# Upgrade from NVCF 1.0.4 to NVCF 1.1.1

<Warning>
This upgrade procedure and its validation are still in progress. Follow
[the 1.1.1 qualification issue](https://github.com/dsx-ai-factory/nvcf/issues/3428)
for the remaining work. Do not treat the publication of NVCF 1.1.1 as a
qualified upgrade path until that issue records it.
</Warning>

This guide describes an upgrade of a running deployment from NVCF 1.0.4 to
NVCF 1.1.1. NVCF 1.1.1 selects Self-Managed 1.1.9 for the control plane,
Compute Plane 1.1.1 for each GPU cluster, and Observability 1.1.0. Read the
whole guide, including [What changes](#what-changes), before planning an upgrade.

This guide is for deployments on NVCF 1.0.4.

## Choose your upgrade path

NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach
NVCF 1.1.1:

| You run | Do this |
| --- | --- |
| NVCF 1.0.1, 1.0.2 or 1.0.3 | Upgrade to NVCF 1.0.4 first with the [1.0.2 to 1.0.4 upgrade](/nvcf/dev/release-notes/1-0-2-to-1-0-4-upgrade), then use the [1.0.4 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-0-4-to-1-1-1-upgrade). |
| NVCF 1.0.4 | Use the [1.0.4 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-0-4-to-1-1-1-upgrade). |
| NVCF 1.1.0 | You have already passed the 1.0.4 stop. Use the [1.1.0 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-1-0-to-1-1-1-upgrade). |

Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do
not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.

| NVCF edition | Self-Managed | Compute Plane | Observability |
| --- | --- | --- | --- |
| 1.0.4 | 1.0.10 | 1.0.5 | 1.0.0 |
| 1.1.1 (target) | 1.1.9 | 1.1.1 | 1.1.0 |

The stack releases this guide covers are:

| Stack | Source | Target | Target tag |
| --- | --- | --- | --- |
| Self-Managed (control plane) | 1.0.10 | 1.1.9 | `deploy/stacks/self-managed/v1.1.9` |
| Compute Plane | 1.0.5 | 1.1.1 | `deploy/stacks/nvcf-compute-plane/v1.1.1` |
| Observability | 1.0.0 | 1.1.0 | `deploy/stacks/observability/v1.1.0` |

## Cross-stack upgrade order

These declarations apply to the supported starting release in this guide:

| Transition | Required peer |
| --- | --- |
| Compute Plane 1.1.1 | Self-Managed 1.1.0 or later, subject to the compatibility and maintenance policy |
| Self-Managed 1.1.9 during the upgrade window | Existing Compute Plane 1.0.5 can remain until its upgrade |

Upgrade the control plane first, then each GPU cluster to Compute Plane 1.1.1.

<Warning>
Do not run a whole-stack `make apply` or `helmfile sync` first. The control
plane has stops that must happen in order: dependencies and migrations, the
OpenBao check, core services, then Cassandra replication for high availability.
Choose the high availability (HA) mode in
[Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration)
before you apply anything. The default, `preferred`, needs little preparation;
`enforced` needs the full preparation in that section.
</Warning>

## What changes

### Control plane: charts

The chart versions the 1.1.9 stack pins, compared with 1.0.10. The `cert-manager`
(0.2.0), `ingress` (1.18.3) and `nvcf-ui` (1.1.4) releases are unchanged.

| Release | 1.0.10 | 1.1.9 |
| --- | --- | --- |
| `admin-issuer-proxy` | 1.5.3 | 1.7.0 |
| `api` | 1.27.5 | 1.35.0 |
| `api-keys` | 1.8.2 | 1.9.4 |
| `cassandra` | 0.21.4 | 0.26.0 |
| `ess-api` | 1.8.3 | 1.11.1 |
| `event-ledger` | not installed | 0.10.2 (new) |
| `function-autoscaler` | 0.5.2 | 0.8.1 |
| `grpc-proxy` | 1.7.4 | 1.14.0 |
| `invocation-service` | 1.6.2 | 1.10.0 |
| `llm-api-gateway` | 1.4.3 | 1.12.0 |
| `llm-request-router` | 1.14.3 | 1.18.2 |
| `nats` | 0.8.6 | 0.9.1 |
| `nats-auth-callout-service` | 1.2.1 | 1.4.2 |
| `notary-service` | 1.6.1 | 1.9.1 |
| `nvcf-pki` | 0.1.0 | 0.2.0 |
| `nvct-api` | 1.6.1 | 1.11.0 |
| `openbao-server` | 0.32.9 | 0.37.1 |
| `ratelimiter` | 1.2.1 | 1.4.1 |
| `reval` | 1.4.1 | 1.6.0 |
| `sis` | 2.4.2 | 2.6.3 |
| `state-metrics` | 1.0.2 | 1.0.7 |
| `vanity-gateway` | 0.5.0 | 0.11.0 |
| `worker-callback-proxy` | not installed | 0.1.0 (new, off by default) |

The LLM and vanity gateway releases matter only if you enable those add-ons.

### Control plane: data and security components

The data and security components below are the ones that matter for an in-place
upgrade. Versions are the ones each release deploys, taken from the published
stack inventories.

| Component | 1.0.10 | Target 1.1.9 |
| --- | --- | --- |
| Cassandra chart | 0.21.4 | 0.26.0 |
| Cassandra image | 5.0.9-nv-2.0.7 | 5.0.9-nv-2.0.7 |
| Cassandra migrations image | 0.17.8 | 0.19.4 |
| OpenBao chart | 0.32.9 | 0.37.1 |
| OpenBao image | 2.6.3-nv-1.3.9 | 2.6.3-nv-1.3.9 |
| OpenBao migrations image | 0.19.11 | 0.23.4 |
| NATS chart | 0.8.6 | 0.9.1 |
| NATS server image | 2.14.9-alpine3.22 | 2.14.9-alpine3.22 |
| NATS config reloader | 0.25.0 | 0.25.0 |
| cert-manager images | v1.20.5 | v1.20.5 |
| API service image | 1.18.2 | 1.28.4 |
| Invocation service image | 0.12.3 | 0.15.1 |
| ICMS | 0.7.7 | 0.8.6 |

The NATS, cert-manager and OpenBao server images are the same as in 1.0.4, and
1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle. This upgrade
does not repeat that plugin update.

### Control plane: behavior that changes

| Area | What happens on upgrade | What you do |
| --- | --- | --- |
| High availability | `highAvailability.mode` is new and defaults to `preferred`. An environment file that does not set it activates HA: replica-safe services scale to 2, Cassandra, NATS and OpenBao run 3 members, and PodDisruptionBudgets and placement rules apply. | Choose `none`, `preferred` or `enforced` in [Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration). |
| Cassandra | The chart adds a config setting (`range_request_timeout: 5000ms`). The pod template changes, so the StatefulSet restarts its pods one at a time. The migrations image moves to `nvcf-cassandra-migrations:0.19.4`, whose new migrations only add schema ([Apply the dependency stage](#apply-the-dependency-stage)). | Plan a short Cassandra interruption if you run a single member. |
| OpenBao | The chart moves to 0.37.1 and the migrations image to 0.23.4. The server image and the JWT plugin lifecycle are unchanged. The StatefulSet uses `OnDelete`, so Helm success does not replace the running pods. | Check whether the pods need replacing in [Check the OpenBao pods](#check-the-openbao-pods). |
| Event Ledger | New release `event-ledger`, on by default (`addons.eventLedger.enabled: true`). It uses the existing `event_ledger` keyspace, and its OpenBao setup runs in the OpenBao migrations job. | Keep it on, or set `addons.eventLedger.enabled: false` before you apply. |
| NATS worker authentication | New `nats.workerAuth` setting. The default, `nkey`, keeps today's behavior. | Nothing, unless you switch to `token`. See "Worker NATS Authentication" in [control-plane operations](/nvcf/dev/self-managed/control-plane-operations). |
| Admin issuer proxy | Now optional (`adminIssuerProxy.enabled`), still on by default. | Nothing, unless you want to disable it. |
| NVCT remote config | New `nvctApi.remoteConfig`, empty by default. | Nothing. |
| Upgrade receipt | The final stage records the installed stack version in `nvcf-upgrade-receipt`. | Validate package stamps first, then compare the receipt with the selected release. |
| Worker sidecars | API and NVCT remote config pin the ESS agent sidecar image to `ess-agent:1.4.5` and the patched NICLLS worker to `nvcf_worker_niclls:2.118.5`. Running functions keep the images they started with. | If you mirror sidecar images, mirror both. |
| LLM add-on image tags | With `addons.llm.enabled`, the request router and Pylon tag must match, and `addons.llm.requestRouter.image.tag` is required. `base.yaml` sets `0.29.3`. | Keep the default unless you mirror or pin it. |
| LLM routes | `ingress.gatewayApi.routes.llmApiGateway.enabled` and `ingress.gatewayApi.routes.llmInvocation.enabled` turn off an LLM route while the add-on is on. | Nothing, unless callers reach the gateway in-cluster. |
| Account bootstrap | `api.accountBootstrap.enabled` lets an install whose account is provisioned another way turn off the bootstrap hook Job. Unset, nothing changes. | Nothing, unless you provision the account another way. |
| Pod annotations | Operator pod annotations per release, for example `api.podAnnotations`. Unset, nothing is rendered. | Nothing. |
| Worker callback proxy | New, off by default. See [Optional: enable the worker callback proxy](#optional-enable-the-worker-callback-proxy). | Nothing, unless you need it. |

## cert-manager images come from upstream

NVCF 1.0.4 already pulls the cert-manager controller, webhook, cainjector,
acmesolver and startupapicheck images from the cert-manager chart's upstream
repositories (`quay.io/jetstack`), and these images do not inherit
`global.image`. NVCF 1.1.0 had briefly defaulted them to the NVCF registry, and
1.1.1 returns to upstream. Coming from 1.0.4, the images stay the same
(`v1.20.5`).

Keep the `image.repository` settings you already use for a private mirror or an
air-gapped registry. Verify them **before** you apply the dependencies stage;
otherwise the cert-manager pods try to pull from the upstream registry and
fail. The settings are `certManager.image.repository` for the controller and
`certManager.webhook.image.repository`, `certManager.cainjector.image.repository`,
`certManager.acmesolver.image.repository` and `certManager.startupapicheck.image.repository`.
See [upstream cert-manager images](/nvcf/dev/manifest/artifacts#use-upstream-cert-manager-images).

### Observability

Observability 1.1.0 is optional. Self-Managed 1.1.9 also works with the
Observability 1.0.0 stack you already run, so you can upgrade Self-Managed on
its own and keep Observability at 1.0.0. The 1.1.0 release changes only two
charts and no images. In the default monitors it sets the `job` label on the
worker and DCGM series and adds pod target labels (`function-id`,
`function-version-id` and `nca-id`) to the worker series. The chart notes say
federation clients such as the BYOO collector select series by that job name,
and the autoscaler queries worker metrics by function identity. Upgrade
Observability if you rely on either. Function scaling and federation were not
exercised with Observability 1.0.0. The control-plane apply reads an
Observability package when the observability profile is enabled, so either
1.0.0 or 1.1.0 must be in place before the render
(see [Get the 1.1.9 packages and check artifacts](#get-the-119-packages-and-check-artifacts)).

| Chart | 1.0.0 | 1.1.0 |
| --- | --- | --- |
| `nvcf-default-monitors` | 0.2.0 | 0.3.0 |
| `nvcf-otel-collector` | 0.2.0 | 0.3.0 |

| Behavior | What changes | What to do |
| --- | --- | --- |
| PodMonitor job labels | The worker and DCGM PodMonitors now set the `job` label to `nvcf-worker` and `nvidia-dcgm-exporter`. | If your dashboards or alerts select the old job names for these series, update them. |
| Worker pod labels | The worker monitor also carries the `function-id`, `function-version-id` and `nca-id` pod labels onto its series. | Nothing, unless you limit label cardinality downstream. |

### Compute plane

The values that differ between Compute Plane 1.0.5 and 1.1.1:

| Component | 1.0.5 | 1.1.1 |
| --- | --- | --- |
| `nvca-operator` chart | 1.28.0 | 1.37.1 |
| NVCA agent and operator (`nvcaVersion`) | 3.10.3 | 3.18.10 |
| Image credential helper (`imageCredHelper.imageTag`) | 0.11.3 | 0.11.4 |
| BYOO collector sidecar (`byooOtelCollector.imageTag`) | 0.160.0-nv-0.2.9 | 0.160.0-nv-0.4.9 |
| NVCF collector | 0.160.0-nv-0.2.9 | 0.160.0-nv-0.4.9 |

New optional settings: `global.nvcaOperator.agent.workerCallbackCaPem` (only
for TLS worker callbacks), `byoo.*` sidecar resources, `utils.resources`,
`byooFluentBit` and `eventLedgerServiceURL`. `global.nvcaOperator.byooFluentBit`
is off by default and its image defaults to `docker.io/fluent/fluent-bit:3.2`.
If you turn on bring-your-own observability in a disconnected install, mirror
that image and set `image.registry`, `image.repository` and `image.tag`. Dynamo, Grove and KAI Scheduler
pins are unchanged. Running function pods keep the worker images they started
with; the new defaults apply to deployments created or replaced afterwards.

## Known issues

- A single-member OpenBao installation is not covered; see
  [Check the OpenBao pods](#check-the-openbao-pods).
- Whether Compute Plane 1.0.5 keeps working against Self-Managed
  1.1.9 during the upgrade window was not checked live.

## Before starting

1. Schedule a maintenance window. You will pause new function invocations and
   task submissions, and a single-member Cassandra restarts once.
1. You need an operator workstation with `kubectl`, `helm` 3.12 or later,
   `helmfile` 1.1.x (not 1.2.x), the `helm-diff` plugin 3.11 or later, `jq`,
   `curl`, Mike Farah's `yq` v4, and `ngc` if you download packages from NGC.
   Helmfile 1.2.0 removed sequential execution, which these stacks need. See
   [required tools](/nvcf/dev/self-managed/helmfile-installation#required-tools-and-software).
1. Keep your existing control-plane and compute-plane directories, your
   `environments/<environment-name>.yaml`, your
   `secrets/<environment-name>-secrets.yaml`, and the registration file for each
   GPU cluster (`<cluster-name>-register-values.yaml`). The upgrade reuses each
   cluster's registration. Do not run `register-cluster` again.
1. Choose canary workloads: one serving function, one function that reads a
   secret, and one running task. You will test them before and after.
1. If you mirror images and charts to a private registry, mirror the 1.1.9
   artifacts first. [Get the 1.1.9 packages and check artifacts](#get-the-119-packages-and-check-artifacts) checks availability.
1. Check the cert-manager image settings in
   [cert-manager images come from upstream](#cert-manager-images-come-from-upstream)
   if the cluster uses a mirror.

## 1. Set deployment values

### Set up the session

Download this read-only script into one directory. [Check the starting state](#check-the-starting-state) and [Render and review the changes](#render-and-review-the-changes) use it to
stop an upgrade that would move an image to an older version:

<Download src="/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/4193cc1855f7ca7f0138a657422979b46a6eb61f05b493fc8dc50acdfc228003/assets/upgrade-checks/check-upgrade-image-moves.py" filename="check-upgrade-image-moves.py">Image move check</Download>

Use a dedicated shell. Replace every value in angle brackets. Run this block
first, and again after any shell restart.

```bash
bash
set -euo pipefail
umask 077

export WORKDIR="$HOME/nvcf-upgrade-1.1.9"
export UPGRADE_SCRIPTS="/absolute/path/to/downloaded-scripts"
export OLD_CONTROL_STACK="/absolute/path/to/existing/nvcf-self-managed-stack"
export OLD_COMPUTE_STACK="/absolute/path/to/existing/nvcf-compute-plane-stack"
export HELMFILE_ENV="<environment-name>"
export CONTROL_CONTEXT="<control-plane-kube-context>"
export BACKUP_DIR="$WORKDIR/backup"
mkdir -p "$WORKDIR" "$BACKUP_DIR"

# One kubeconfig per cluster, so a stray context switch cannot redirect a command.
export CONTROL_KUBECONFIG="$WORKDIR/control.kubeconfig"
kubectl config view --minify --flatten --raw --context "$CONTROL_CONTEXT" > "$CONTROL_KUBECONFIG"

kc() { kubectl --kubeconfig "$CONTROL_KUBECONFIG" "$@"; }
control_helmfile() {
  (cd "$CONTROL_STACK" && HELMFILE_ENV="$HELMFILE_ENV" \
    helmfile --environment default --kubeconfig "$CONTROL_KUBECONFIG" "$@")
}
redact_diff() {
  awk '
    /^[-+ ]+- name: / { sensitive = (toupper($0) ~ /PASSWORD|SECRET|TOKEN|KEY|CREDENTIAL/); print; next }
    sensitive && /^[-+ ]+value: / { sub(/value: .*/, "value: [REDACTED]"); print; next }
    { sensitive = 0; print }'
}
wait_workloads() {
  local kubeconfig="$1" namespace="$2" workload strategy
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get deployments,statefulsets -o name |
    while IFS= read -r workload; do
      strategy=$(kubectl --kubeconfig "$kubeconfig" -n "$namespace" \
        get "$workload" -o jsonpath='{.spec.updateStrategy.type}')
      if [ "$strategy" = "OnDelete" ]; then
        printf '%s uses OnDelete; replace its pods with the procedure in this guide\n' "$workload"
        continue
      fi
      kubectl --kubeconfig "$kubeconfig" -n "$namespace" rollout status "$workload" --timeout=15m
    done
  kubectl --kubeconfig "$kubeconfig" -n "$namespace" get pods
}
declare -f kc control_helmfile redact_diff wait_workloads > "$WORKDIR/helpers.sh"
declare -p WORKDIR UPGRADE_SCRIPTS OLD_CONTROL_STACK OLD_COMPUTE_STACK HELMFILE_ENV \
  CONTROL_CONTEXT BACKUP_DIR CONTROL_KUBECONFIG > "$WORKDIR/session.sh"

kc config current-context
kc get nodes
```

`--suppress-secrets` hides Kubernetes Secret data only. The OpenBao migration
Job carries the Cassandra password and the registry credential as plain
environment values, so a diff that includes it prints them. `redact_diff`
replaces the value of any environment entry whose name contains `PASSWORD`,
`SECRET`, `TOKEN`, `KEY` or `CREDENTIAL`. Pipe the dependency diff through it,
and keep that output private.

All nodes must be `Ready`. To resume later, start a new `bash`, run
`set -euo pipefail`, set `WORKDIR` to the directory you used above
(`export WORKDIR="$HOME/nvcf-upgrade-1.1.9"`), then
`source "$WORKDIR/session.sh"` and `source "$WORKDIR/helpers.sh"`. The saved kubeconfig holds credentials, so keep
`$WORKDIR` private.

### Get the 1.1.9 packages and check artifacts

Both exact target packages must be published and available to your customer
account before maintenance. Stop if either download or package-stamp check
fails. The NVCF 1.1.1 manifest selects these three targets together; do not mix them
with other stack versions.

Download the three target packages next to each other, in a new directory, so
the relative paths in the Helmfiles resolve the same way they did for your current release.

```bash
export STACK_VERSION="1.1.9"
export COMPUTE_STACK_VERSION="1.1.1"
# Observability is optional: export OBSERVABILITY_STACK_VERSION=1.0.0 before
# this block to stay on the version you run now. An existing value is kept.
export OBSERVABILITY_STACK_VERSION="${OBSERVABILITY_STACK_VERSION:-1.1.0}"
cd "$WORKDIR"

ngc registry resource download-version "nvidia/nvcf/nvcf-self-managed-stack:${STACK_VERSION}"
mkdir -p nvcf-self-managed-stack
tar -xzf "nvcf-self-managed-stack_v${STACK_VERSION}/nvcf-self-managed-stack-${STACK_VERSION}.tar.gz" \
  -C nvcf-self-managed-stack
rm -rf "nvcf-self-managed-stack_v${STACK_VERSION}"

ngc registry resource download-version "nvidia/nvcf/nvcf-compute-plane-stack:${COMPUTE_STACK_VERSION}"
mkdir -p nvcf-compute-plane-stack
tar -xzf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}/nvcf-compute-plane-stack-${COMPUTE_STACK_VERSION}.tar.gz" \
  -C nvcf-compute-plane-stack
rm -rf "nvcf-compute-plane-stack_v${COMPUTE_STACK_VERSION}"

ngc registry resource download-version "nvidia/nvcf/nvcf-observability-stack:${OBSERVABILITY_STACK_VERSION}"
mkdir -p observability
tar -xzf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}/nvcf-observability-stack-${OBSERVABILITY_STACK_VERSION}.tar.gz" \
  -C observability
rm -rf "nvcf-observability-stack_v${OBSERVABILITY_STACK_VERSION}"

export CONTROL_STACK="$WORKDIR/nvcf-self-managed-stack"
export COMPUTE_STACK="$WORKDIR/nvcf-compute-plane-stack"
export OBSERVABILITY_STACK="$WORKDIR/observability"
declare -p STACK_VERSION COMPUTE_STACK_VERSION OBSERVABILITY_STACK_VERSION CONTROL_STACK COMPUTE_STACK OBSERVABILITY_STACK >> "$WORKDIR/session.sh"
ls "$CONTROL_STACK" "$COMPUTE_STACK" "$OBSERVABILITY_STACK"
```

If you obtain the packages another way, extract them to these three
directories. If the observability profile is enabled (`observability.profile` is
not `disabled`), the control-plane Helmfile reads
`../../observability/helmfile.d/01-observability.yaml.gotmpl`, which resolves to
`$WORKDIR/observability`. That directory must hold an Observability package,
either 1.1.0 or your existing 1.0.0. To keep 1.0.0, export
`OBSERVABILITY_STACK_VERSION=1.0.0` before running the download block, or copy
your existing 1.0.0 stack to that directory and skip its download. Leaving the stack only in
its original location makes the render fail. With the profile `disabled` the
directory is not read.

Check both new package stamps against the selected versions before maintenance:

```bash
check_selected_stack_versions() {
  local actual
  actual=$(cat "$CONTROL_STACK/VERSION") || return 1
  if [ "$actual" != "$STACK_VERSION" ]; then
    printf 'Self-Managed package must report %s; found %s\n' "$STACK_VERSION" "$actual" >&2
    return 1
  fi
  actual=$(cat "$COMPUTE_STACK/VERSION") || return 1
  if [ "$actual" != "$COMPUTE_STACK_VERSION" ]; then
    printf 'Compute Plane package must report %s; found %s\n' "$COMPUTE_STACK_VERSION" "$actual" >&2
    return 1
  fi
}
check_selected_stack_versions
declare -f check_selected_stack_versions >> "$WORKDIR/helpers.sh"
```

Stop on a missing or mismatched stamp. Obtain a correctly stamped, qualified
package through the release process; do not edit `VERSION`, replace immutable
artifacts or substitute another stack version.

Confirm the charts and images are reachable from your registry. Set the
repositories to the ones in your environment file (`global.helm.sources` and
`global.image`), or to your mirror:

If your environment file uses an HTTPS Helm repository
(`global.helm.sources.url`) instead of an OCI registry, add it under an alias
first and use the alias as `HELM_REPO` below. The password is read from
standard input so it stays out of the process list:

```bash
export HELM_REPOSITORY_CONFIG="$WORKDIR/helm-repositories.yaml"
read -r -s -p "Repository password: " HELM_PASSWORD; echo
printf '%s' "$HELM_PASSWORD" | helm repo add nvcf-check "<https-repository-url>" \
  --username "<username>" --password-stdin
unset HELM_PASSWORD
helm repo update nvcf-check
export HELM_REPO="nvcf-check"
```

For an OCI registry, set `HELM_REPO` to its address instead:

```bash
export HELM_REPO="oci://<registry>/<repository>"
export IMAGE_REPO="<registry>/<repository>"
artifact_missing=0

while IFS=: read -r chart version; do
  if helm show chart "$HELM_REPO/$chart" --version "$version" > /dev/null 2>&1; then
    echo "ok       $chart:$version"
  else
    echo "MISSING  $chart:$version"
    artifact_missing=1
  fi
done <<'EOF'
helm-nvcf-cassandra:0.26.0
helm-nvcf-nats:0.9.1
helm-nvcf-openbao-server:0.37.1
helm-nvcf-pki:0.2.0
nvcf-gateway-routes:1.18.3
helm-nvcf-api:1.35.0
helm-nvcf-api-keys:1.9.4
helm-admin-token-issuer-proxy:1.7.0
helm-nvcf-ess-api:1.11.1
helm-nvcf-sis:2.6.3
helm-nvcf-nvct-api:1.11.0
helm-nvcf-notary-service:1.9.1
helm-nvcf-invocation-service:1.10.0
helm-nvcf-grpc-proxy:1.14.0
helm-nvcf-rate-limiter:1.4.1
helm-reval:1.6.0
helm-nvcf-nats-auth-callout-service:1.4.2
helm-nvcf-event-ledger:0.10.2
helm-nvcf-state-metrics:1.0.7
helm-nvcf-function-autoscaler:0.8.1
EOF

while IFS= read -r image; do
  if docker manifest inspect "$IMAGE_REPO/$image" > /dev/null 2>&1; then
    echo "ok       $image"
  else
    echo "MISSING  $image"
    artifact_missing=1
  fi
done <<'EOF'
nvcf-service-oss:1.28.4
nvcf-api-keys-service:1.10.10
nvct-service-oss:1.70.4
nvcf-ess:0.5.7
icms-service-oss:0.8.6
nvcf-notary:1.14.8
nvcf-invocation-service:0.15.1
nvcf-grpc-proxy:1.35.2
nvcf-event-ledger:0.21.2
admin-token-issuer-proxy:1.1.3
nvcf-ratelimiter:1.18.3
reval-server:0.20.4
nvcf-nats-auth-callout-service:0.8.8
nvcf-state-metrics-service:1.24.1
nvcf-function-autoscaler:1.23.2
nvcf-openbao:2.6.3-nv-1.3.9
nvcf-openbao-migrations:0.23.4
nvcf-cassandra-migrations:0.19.4
cassandra:5.0.9-nv-2.0.7
EOF
test "$artifact_missing" -eq 0
```

The list covers the images that change plus the data-service images. The
[artifact manifest](/nvcf/dev/manifest/artifacts) has the complete list, including optional
add-ons. Fix every `MISSING` line before continuing; a missing image shows up
later as `ImagePullBackOff` in the middle of a stage.

### Prepare the HA mode and configuration

#### Choose the HA mode

| Mode | Result | Needs |
| --- | --- | --- |
| `none` | Keeps your current footprint: no extra replicas, no new PodDisruptionBudgets. | Nothing. |
| `preferred` (default) | Replica-safe services run 2 replicas. Cassandra, NATS and OpenBao run 3 members with soft anti-affinity and zone spread. Pods still schedule when capacity is short. | At least 3 schedulable nodes. |
| `enforced` | Same sizing with hard placement. Pods stay `Pending` rather than co-locate. | At least 3 nodes, 3 zones with capacity, `topology.kubernetes.io/zone` labels, and a `WaitForFirstConsumer` StorageClass. |

`preferred` is the default and needs only enough schedulable nodes, so the
checks below are enough for it. `enforced` refuses to place a pod when a
requirement is missing, so complete every step under
[Prepare for enforced mode](#prepare-for-enforced-mode) before you apply
anything. If this is a single-node cluster, or you are not ready to add
capacity, choose `none`. You can move to HA later. If your Cassandra runs a
single member, first follow
[Expand a single-member Cassandra first](#expand-a-single-member-cassandra-first).
The full behavior, including zone spread, is in the
[high availability guide](/nvcf/dev/self-managed/high-availability).

Check what your cluster offers:

```bash
kc get nodes -L topology.kubernetes.io/zone
kc get storageclass
kc get nodes --no-headers | wc -l
```

Replica-safe services roll with `maxUnavailable: 0` and `maxSurge: 1`, so leave
room for one extra pod per service.

#### Prepare for enforced mode

Skip this section for `none` and `preferred`. With `enforced`, pods stay
`Pending` instead of co-locating, and an unmet requirement blocks the stage that
creates them. Complete every step before the first apply, and keep the results in
`$WORKDIR`:

1. **Three nodes with capacity.** Each Cassandra, NATS and OpenBao member
   needs its own node. Check that three nodes are `Ready`, schedulable and not
   tainted against these workloads (`kc get nodes`), and that they have the
   CPU, memory and volume capacity for the extra replicas. Replica-safe services
   also run 2 replicas, so keep room for one extra pod per service during a
   rollout: free CPU and memory on at least one node must equal the largest pod
   request.
2. **Three zones.** The nodes must be spread over three zones, each with
   capacity, and every node must carry the `topology.kubernetes.io/zone` label.
   List the label with the first command above. A node without the label cannot
   take a zone-spread pod.
3. **A `WaitForFirstConsumer` StorageClass.** The StorageClass your Cassandra,
   NATS and OpenBao volumes use must show `WaitForFirstConsumer` as its volume
   binding mode (`kc get storageclass`). With `Immediate` binding, a volume can
   be created in a zone that has no node for its pod, and the pod stays
   `Pending`. Existing volumes keep their zone, so check where each existing
   Cassandra, NATS and OpenBao volume lives, and that the zone has a schedulable
   node:

   ```bash
   kc get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone
   ```

4. **Single-member Cassandra.** If Cassandra runs one member, complete
   [Expand a single-member Cassandra first](#expand-a-single-member-cassandra-first)
   before the dependency stage. The new members land in other zones, so check
   steps 1 to 3 for them too.
5. **Replica overrides.** Remove replica overrides from the secrets file, as in
   [Check the secrets file for replica overrides](#check-the-secrets-file-for-replica-overrides).
   An override conflicts with the `enforced` sizing and fails the render.
6. **Render first.** After you set the mode, render and review the diff in
   [Render and review the changes](#render-and-review-the-changes). Stop if a
   workload would not fit.

If a pod stays `Pending` after the apply, run `kc describe pod` on it and read
the scheduling events before you change the mode. Moving back to `preferred`
later is an environment change and a re-apply of the affected stage.

#### Carry your settings into the new packages

The new `base.yaml` files contain the 1.1.9 defaults and chart settings. Copy
only your own changes into new environment files. Do not copy the whole old
file, because that would overwrite new defaults with old ones.

```bash
# Your customizations: the lines in your environment file that differ from the old base.
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_CONTROL_STACK/environments/base.yaml" \
  "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/control-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/control-overrides.diff"

cp "$CONTROL_STACK/environments/base.yaml" "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml"
```

Open `$WORKDIR/control-overrides.diff` and re-apply each `+` line in
`$CONTROL_STACK/environments/$HELMFILE_ENV.yaml`. Typical lines are your
registry and chart source (`global.helm.sources`, `global.image`), Gateway
settings, storage class and size, node selectors and `imagePullSecrets`. Do not
carry over image tag or chart version overrides unless you deliberately pin
them; the 1.1.9 defaults are the qualified set. The 1.1.9 `base.yaml` no longer
pins `cassandra.migrations.image.tag`, so the chart default (0.19.4) applies.
If you pin that tag, remove the pin or set 0.19.4 or later: an older pin skips
the schema changes in [Apply the dependency stage](#apply-the-dependency-stage).

NVCF 1.0.4 and the 1.1.9 defaults pull the same NATS server, nats-box and
config reloader images from public registries (`docker.io/synadia/nats-server`,
`docker.io/natsio/nats-box` and `docker.io/natsio/nats-server-config-reloader`).
If you mirror them, keep the `nats.container.image`, `nats.natsBox.container.image`
and `nats.reloader.image` settings from your environment file. If you set none of
these and your cluster can pull from `docker.io`, there is nothing to do. The NATS
pods run as a non-root user, so confirm in
[Apply the dependency stage](#apply-the-dependency-stage) that each NATS pod becomes Ready.

Then set the HA mode and, if you want to turn Event Ledger off, its switch. In
the environment file. This sample keeps the default, `preferred`; set `enforced`
only after you complete [Prepare for enforced mode](#prepare-for-enforced-mode):

```yaml
highAvailability:
  mode: preferred   # none | preferred | enforced
addons:
  eventLedger:
    enabled: true   # set false to skip the new Event Ledger release
```

Merge these keys into the existing `addons:` section; do not duplicate it.

#### Check the secrets file for replica overrides

Under `preferred` and `enforced`, the render fails if the secrets file sets a
value that conflicts with the HA sizing, for example `replicaCount: 1`. Look
for them now:

```bash
grep -nE 'replica(Count|s):' "$CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" \
  || echo "no replica overrides"
```

Remove any replica override the HA sizing should control, or choose
`highAvailability.mode: none`. Record the mode you chose:

```bash
yq '.highAvailability' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml"
```

## 2. Check the starting state and take backups

### Check the starting state

Do not continue unless every check passes.

```bash
# Releases deployed
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A -o json > "$WORKDIR/control-before.json"
jq -r '.[] | "\(.namespace)\t\(.name)\t\(.chart)\t\(.status)"' "$WORKDIR/control-before.json" | sort
test -z "$(jq -r '.[] | select(.status != "deployed") | .name' "$WORKDIR/control-before.json")" \
  && echo "all releases deployed"

# Cassandra: every member UN
for pod in $(kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra \
  -o jsonpath='{.items[*].metadata.name}'); do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool status
  break
done

# Workloads ready everywhere; sis only when the ICMS release is installed
namespaces="cassandra-system nats-system vault-system nvcf api-keys ess"
if jq -e '.[] | select(.name == "sis")' "$WORKDIR/control-before.json" > /dev/null; then
  namespaces="$namespaces sis"
fi
for namespace in $namespaces; do
  kc -n "$namespace" get pods -o json | jq -e '
    (.items | length > 0) and
    any(.items[]; any(.status.conditions[]?; .type == "Ready" and .status == "True")) and
    all(.items[]; .metadata.deletionTimestamp == null and
      (.status.phase == "Succeeded" or
        any(.status.conditions[]?; .type == "Ready" and .status == "True")))'
done

# Record replica counts and images before the upgrade
kc get deploy,sts -A -o wide > "$WORKDIR/workloads-before.txt"
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" before \
  "$CONTROL_KUBECONFIG" "$WORKDIR/images-before.tsv"
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml > "$WORKDIR/receipt-before.yaml" 2>/dev/null || true
```

Cassandra must report every member `UN`. The OpenBao check function used in
this guide, and again in [Apply the dependency stage](#apply-the-dependency-stage), is:

```bash
check_openbao() {
  local pod pods
  pods=$(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
    -o jsonpath='{.items[*].metadata.name}') || return 1
  [ -n "$pods" ] || { echo "No OpenBao server pods found" >&2; return 1; }
  for pod in $pods; do
    echo "== $pod =="
    kc -n vault-system exec "$pod" -c openbao -- sh -c \
      'BAO_ADDR=http://127.0.0.1:8200 bao status -format=json' || return 1
  done
}
declare -f check_openbao >> "$WORKDIR/helpers.sh"
check_openbao
```

Every OpenBao member must be `initialized: true` and `sealed: false`, with the
same `cluster_id`. Exactly one reports `is_self: true`; standbys report
`is_self: false` or omit the field. Record the cluster ID.

Run your canary workloads now and record their results. With the configured
`nvcf-cli`:

```bash
export NVCF_CLI="/absolute/path/to/nvcf-cli"
export NVCF_CLI_CONFIG="/secure/path/to/nvcf-cli.yaml"
export FUNCTION_ID="<existing-function-id>"
export VERSION_ID="<existing-version-id>"
export CANARY_REQUEST='{"input":"your-known-canary-input"}'
declare -p NVCF_CLI NVCF_CLI_CONFIG FUNCTION_ID VERSION_ID CANARY_REQUEST >> "$WORKDIR/session.sh"

"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function invoke \
  --function-id "$FUNCTION_ID" --version-id "$VERSION_ID" \
  --request-body "$CANARY_REQUEST" --timeout 120
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING
```

#### Back up persistent state

Create restorable backups before you change anything. Follow your storage
provider's procedure for volume snapshots as well; the commands below create
the application-level backups.

```bash
CASSANDRA_REPLICAS=$(kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}')
echo "$CASSANDRA_REPLICAS" > "$BACKUP_DIR/cassandra-replicas"
for ordinal in $(seq 0 $((CASSANDRA_REPLICAS - 1))); do
  pod="cassandra-${ordinal}"
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool flush
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool snapshot -t pre-1-1-9
done
```

The Cassandra snapshot stays on each persistent volume. It is not an off-cluster
backup, so also copy or snapshot the volumes.

Record the migration ledger too. Each application keyspace keeps its applied
migration version and dirty flag in a table of the `schema_migrations` keyspace.
The migration jobs read it to apply only the pending migrations, so a member that
cannot see a row makes a job replay a migration that already ran. The block saves
the ledger from the original member now, while that member holds all of it:

```bash
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
LEDGER_TABLES=$(cql_on cassandra-0 "SELECT table_name FROM system_schema.tables WHERE keyspace_name = 'schema_migrations';" |
  tr -d ' \r' | grep -E '^[a-z][a-z0-9_]*$' | grep -vx table_name)
test -n "$LEDGER_TABLES"
for table in $LEDGER_TABLES; do
  printf '%s: ' "$table"
  cql_on cassandra-0 "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|'
done > "$BACKUP_DIR/schema-migrations-ledger.txt"
cat "$BACKUP_DIR/schema-migrations-ledger.txt"
unset CASSANDRA_PASSWORD
```

Every line must show a version and a dirty flag of `False`. Resolve a dirty ledger
before you continue, then run this block again so the saved file records the
corrected state. Do not run it again after the expansion: it would overwrite the
saved ledger with what the new members can see.

Take the OpenBao Raft snapshot from the active member. A snapshot taken from
a standby fails with `incomplete snapshot`.

```bash
OPENBAO_ACTIVE=""
for pod in $(kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{.items[*].metadata.name}'); do
  if kc -n vault-system exec "$pod" -c openbao -- sh -c \
    'BAO_ADDR=http://127.0.0.1:8200 bao status' | grep -Eq '^HA Mode[[:space:]]+active$'; then
    OPENBAO_ACTIVE="$pod"
  fi
done
echo "active member: $OPENBAO_ACTIVE"

kc -n vault-system get secret openbao-server-root-token -o jsonpath='{.data.root_token}' | base64 -d |
  kc -n vault-system exec -i "$OPENBAO_ACTIVE" -c openbao -- sh -c \
  'read -r BAO_TOKEN; export BAO_TOKEN BAO_ADDR=http://127.0.0.1:8200; bao operator raft snapshot save /tmp/openbao-pre-1-1-9.snap'
kc -n vault-system cp -c openbao "$OPENBAO_ACTIVE:/tmp/openbao-pre-1-1-9.snap" \
  "$BACKUP_DIR/openbao-pre-1-1-9.snap"
kc -n vault-system exec "$OPENBAO_ACTIVE" -c openbao -- rm -f /tmp/openbao-pre-1-1-9.snap

# The snapshot restores only with the unseal key and root token.
kc -n vault-system get secret openbao-server-unseal -o yaml > "$BACKUP_DIR/openbao-server-unseal.yaml"
kc -n vault-system get secret openbao-server-root-token -o yaml > "$BACKUP_DIR/openbao-server-root-token.yaml"

# Helm values of every release, and your configuration files.
mkdir -p "$BACKUP_DIR/helm-values"
jq -r '.[] | "\(.namespace) \(.name)"' "$WORKDIR/control-before.json" |
  while read -r namespace release; do
    helm --kubeconfig "$CONTROL_KUBECONFIG" get values "$release" -n "$namespace" -o yaml \
      > "$BACKUP_DIR/helm-values/control-$namespace-$release.yaml"
  done
cp "$OLD_CONTROL_STACK/environments/$HELMFILE_ENV.yaml" "$BACKUP_DIR/old-environment.yaml"
cp "$OLD_CONTROL_STACK/secrets/$HELMFILE_ENV-secrets.yaml" "$BACKUP_DIR/old-secrets.yaml"
ls -l "$BACKUP_DIR"
```

The unseal and root-token files, the secrets copy and the exported values
contain credentials. Protect them like the snapshot. NVCF NATS streams use
memory storage, so a NATS backup holds stream metadata only. Do not continue
until the Cassandra and OpenBao backups exist.

### Render and review the changes

Render into a new directory, then review the diff the first stage would apply.

```bash
make -C "$CONTROL_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  KUBECONFIG_FILE="$CONTROL_KUBECONFIG" OUTPUT_DIR="$WORKDIR/control-render"

# Images the 1.1.9 render pulls; make sure each one is in your registry or mirror.
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/control-render" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u
```

The render can include secrets, so keep it private. Check that the namespaces
match your cluster and that no image lacks the registry you expect.

Compare the render with the images that run now. The check fails if any image
would move to an older version, which happens when an old pin survives in your
environment file:

```bash
python3 "$UPGRADE_SCRIPTS/check-upgrade-image-moves.py" compare \
  "$WORKDIR/images-before.tsv" "$WORKDIR/control-render"
```

Every `BACKWARD` line names an image and both versions. Stop on any of them. For
a pin you did not set on purpose, remove it as described in
[Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration), render again and
re-run the check.

Before maintenance, verify the receipt image from the rendered Job. Its default
is `docker.io/alpine/k8s:1.37.0`; it does not follow `global.image`. For a mirror,
set `upgradeReceipt.image.registry` and `upgradeReceipt.image.repository`, then
render again. Use the resulting image reference for the check.

Run a temporary pull-test Pod in the Job's namespace on each eligible node pool,
using that image, its pull credentials and scheduling settings. Set
`imagePullPolicy: Always` and run `true` instead of the receipt script. Confirm
the image pulls and the Pod succeeds, then remove the test Pod. A host-side
`docker manifest inspect` alone does not verify node access.

If you run tasks, complete the [task worker image checks](/nvcf/dev/self-managed/upgrade-recovery#task-worker-images)
using this target render and the Self-Managed 1.1.9 inventory before proceeding.

## 3. Drain pending work

Stop new production requests and task submissions at their callers. NATS
request queues use memory storage, and the dependency stage can restart NATS.
Wait for queued work to finish; do not cancel it to get an empty queue.

For every affected function and version, use a key with `queue_details` scope
(`NVCF_API_KEY`). The task command below uses a separate key with `list_tasks`
scope (`NVCF_NVCT_API_KEY`) and your Tasks endpoint (`base_nvct_url` and
`nvct_host` in the CLI configuration). Verify both with the
[CLI authentication settings](/nvcf/dev/overview/cli) before you start:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" function queue status "$FUNCTION_ID" "$VERSION_ID"
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" task list --status RUNNING
```

Queue sizes must reach zero. Record the identities and results of running
tasks. Then check the control plane's own memory streams on every NATS server,
through the monitoring port 8222:

```bash
NATS_MONITOR_PORT=18223
NATS_PODS=$(kc -n nats-system get pods -o json | jq -r '.items[] |
  select(any(.spec.containers[]; .name == "nats")) | .metadata.name')
test -n "$NATS_PODS"
NATS_DRAIN_DIR="$WORKDIR/nats-drain-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir "$NATS_DRAIN_DIR"
for pod in $NATS_PODS; do
  (
    # Background kubectl itself, not the kc function: $! must be the port-forward
    # process, or the trap leaves it running and the next pod cannot bind the port.
    kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nats-system \
      port-forward --address 127.0.0.1 "pod/$pod" "$NATS_MONITOR_PORT:8222" \
      > "$NATS_DRAIN_DIR/$pod.log" 2>&1 &
    forward_pid=$!
    trap 'kill "$forward_pid" 2>/dev/null || true; wait "$forward_pid" 2>/dev/null || true' EXIT
    for attempt in {1..30}; do
      kill -0 "$forward_pid"
      grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log" && break
      sleep 1
    done
    grep -q 'Forwarding from 127.0.0.1:' "$NATS_DRAIN_DIR/$pod.log"
    curl --fail --silent --show-error --connect-timeout 5 --max-time 30 \
      "http://127.0.0.1:$NATS_MONITOR_PORT/jsz?accounts=true&streams=true&config=true" \
      > "$NATS_DRAIN_DIR/$pod.json"
    jq -e '.account_details | type == "array"' "$NATS_DRAIN_DIR/$pod.json"
    jq -e 'all(.account_details[];
      ((.stream_detail // []) | type == "array") and
      all((.stream_detail // [])[];
        (.config.storage | type == "string") and
        (.state.messages | type == "number")))' "$NATS_DRAIN_DIR/$pod.json"
  )
done
jq -se '[.[] | .account_details[]? | .stream_detail[]? |
  select(.config.storage == "memory") |
  {name: .name, replicas: (.config.num_replicas // 1), messages: .state.messages}] as $s |
  if ($s | length) == 0 then error("No memory streams found; check the NATS monitoring endpoint")
  elif any($s[]; .messages > 0) then error("Memory streams not drained; do not roll NATS")
  else {memory_streams: ($s | length), single_replica: ([$s[] | select(.replicas == 1)] | length)} end' \
  "$NATS_DRAIN_DIR/"*.json
```

Every memory stream must show zero messages. A replicated stream (under
`preferred` or `enforced` HA) survives a single restart, but draining it keeps
the result unambiguous. A failed or slow read, a missing stream or a nonzero
count blocks the upgrade. A server whose accounts hold no streams reports no
stream details, which counts as empty; the final check still requires at least
one memory stream across the servers. Each attempt writes to a new directory, so files from an
earlier attempt cannot satisfy the check. Wait for the work to drain
and repeat; do not discard messages. If your NATS monitoring is configured
differently, use its endpoint and keep the same per-server check.
For retained creation messages or stuck deployments, follow the
[recovery steps](/nvcf/dev/self-managed/upgrade-recovery#inspect-retained-creation-messages)
before proceeding.

## 4. Apply control-plane stages in order

### Expand a single-member Cassandra first

Do this only if `CASSANDRA_REPLICAS` from [Back up persistent state](#back-up-persistent-state)
is `1` and you chose `preferred` or `enforced`. Skip it if Cassandra already runs
3 members, or if you chose `none`. Run it in the maintenance window, after you
drain pending work: while `system_auth` is being raised and repaired, new logins
can fail.

Do not let the upgrade expand Cassandra from one member to three. The chart
raises the `system_auth` replication factor to the new member count in a hook that
runs after the StatefulSet scales, and it does not repair the data. The new
members then hold none of the existing login data, so administrator and service
logins can fail while every member shows `UN`, and the dependency stage times out.
Expand and repair now, on the release you run today, so the upgrade starts from a
normal three-member cluster.

First, make `cassandra-0` the only seed. Cassandra does not stream the existing
data to a member that lists itself as a seed, so a new member that appears in the
old seed list joins without the data. Set the seed list to the original member
only, for the expansion, so that the new members bootstrap. The upgrade later
restores the chart's default two-seed list. The member turns `Ready` a few seconds
before it accepts CQL connections, so the block also waits for the CQL port. It
stops unless Cassandra is a single member:

```bash
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system get statefulset cassandra \
  -o jsonpath='{.spec.replicas}')" = "1" || {
  echo "Expected a single-member Cassandra deployment; stop here."
  exit 1
}

kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system patch statefulset cassandra \
  --type strategic -p '{
    "spec": {
      "template": {
        "spec": {
          "containers": [{
            "name": "cassandra",
            "env": [{
              "name": "CASSANDRA_SEEDS",
              "value": "cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local"
            }]
          }]
        }
      }
    }
  }'

kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system rollout status \
  statefulset/cassandra --timeout=10m

for attempt in $(seq 1 60); do
  if [ "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
    nodetool statusbinary)" = "running" ]; then break; fi
  sleep 5
done
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
  nodetool statusbinary)" = "running"

kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
  printenv CASSANDRA_SEEDS

kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
  nodetool status
```

This restarts the original single member. Before you continue, check four things:
the seed list contains only `cassandra-0`, the member is `UN`, the administrator
and service logins still work, and the migration ledger is unchanged from the copy
that [Back up persistent state](#back-up-persistent-state) saved. The block asks
for the password and defines the helper itself, so it also runs in a new shell:

```bash
SEED_LIST=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
  printenv CASSANDRA_SEEDS)
test "$SEED_LIST" = "cassandra-0.cassandra-headless.cassandra-system.svc.cluster.local"
test "$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec cassandra-0 -c cassandra -- \
  nodetool status | grep -c '^UN')" = "1"
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
cql_on cassandra-0 "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
echo "administrator login ok on cassandra-0"
for table in $(cut -d: -f1 "$BACKUP_DIR/schema-migrations-ledger.txt"); do
  printf '%s: ' "$table"
  cql_on cassandra-0 "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|' || echo "no row"
done > "$BACKUP_DIR/schema-migrations-ledger-after-seed.txt"
diff "$BACKUP_DIR/schema-migrations-ledger-after-seed.txt" "$BACKUP_DIR/schema-migrations-ledger.txt"
echo "migration ledger unchanged after the seed change"
unset CASSANDRA_PASSWORD
```

Repeat your canary invocation from [Check the starting state](#check-the-starting-state),
which checks the service logins, and confirm that the services report no
authentication errors:

```bash
AUTH_LOGS=$(kubectl --kubeconfig "$CONTROL_KUBECONFIG" -n nvcf logs deploy/nvcf-api --since=10m)
test "$(printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true)" = "0"
```

Stop if any check fails, and restore from the backups in
[Back up persistent state](#back-up-persistent-state) with your reviewed recovery
procedure. Recovering an expansion that has already failed was not validated.

Place the new members in different zones. A member's volume stays in the zone
where the member first starts, and the placement rules that the upgrade applies
later cannot move it. Under `enforced`, two members in one zone leave a pod
`Pending` during the upgrade. If every schedulable node has a
`topology.kubernetes.io/zone` label, add a zone spread rule to the StatefulSet
before you scale. The rule is hard under `enforced` and soft under `preferred`, so
a zone without capacity never leaves Cassandra `Pending` under `preferred`. This
restarts the single member once, and the upgrade replaces the rule with the
chart's own. If any node has no zone label, skip this rule: a node without the
label cannot take a pod that has it.

```bash
kc get nodes -L topology.kubernetes.io/zone
HA_MODE=$(yq -r '.highAvailability.mode // "preferred"' "$CONTROL_STACK/environments/$HELMFILE_ENV.yaml")
case "$HA_MODE" in
  enforced) SPREAD=DoNotSchedule ;;
  preferred) SPREAD=ScheduleAnyway ;;
  *) echo "Unsupported highAvailability.mode: $HA_MODE" >&2; exit 1 ;;
esac
kc -n cassandra-system patch statefulset cassandra --type merge -p "{\"spec\":{\"template\":{\"spec\":{\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"$SPREAD\",\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/name\":\"cassandra\"}}}]}}}}"
kc -n cassandra-system rollout status statefulset/cassandra --timeout=10m
```

Scale the StatefulSet directly. Do not set `cassandra.replicaCount` and apply the
old stack, because that runs the chart hook with the same problem. The block reads
the member count that the backup step saved, so it also works after a shell
restart, and it shows the live count. Scaling to 3 is safe to repeat. Wait for all
three members to be `UN`:

```bash
CASSANDRA_REPLICAS=$(cat "$BACKUP_DIR/cassandra-replicas")
test "$CASSANDRA_REPLICAS" -eq 1
kc -n cassandra-system get statefulset cassandra -o jsonpath='{.spec.replicas}{" members now\n"}'
kc -n cassandra-system scale statefulset cassandra --replicas=3
kc -n cassandra-system rollout status statefulset/cassandra --timeout=20m
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
```

Every member must report `UN`. If a pod stays `Pending`, check the node and volume
capacity before you continue. Confirm that each member runs in a different zone:

```bash
kc -n cassandra-system get pods -l app.kubernetes.io/name=cassandra -o wide
kc get nodes -L topology.kubernetes.io/zone
```

Each `ALTER KEYSPACE` below replaces the whole replication map with the one data
center that this guide validates against the running cluster. If your keyspaces
also replicate to another data center, add that entry to every `ALTER` here and
in the application keyspace loop that follows, so the expansion does not remove it.

Raise the `system_auth` replication factor to 3, repair it on every member, then
confirm that the administrator can log in on every member. The password is that of
the `cassandra` administrator role, not the service role password in your secrets
file. The commands read it from standard input into a temporary protected file, so
do not enable shell tracing.

```bash
CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
cql_on cassandra-0 "ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full system_auth
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  cql_on "$pod" "SELECT role FROM system_auth.roles LIMIT 1;" > /dev/null
  echo "administrator login ok on $pod"
done
```

All three members must print `administrator login ok`. If any login fails, stop.
Do not continue to the upgrade. Restore from the backups in
[Back up persistent state](#back-up-persistent-state) with your reviewed recovery
procedure, and contact NVIDIA support with the output.

Then raise the application keyspaces to 3 and repair them on every member. The
loop alters only the keyspaces your installation has:

```bash
EXISTING_KEYSPACES=$(cql_on cassandra-0 "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in schema_migrations nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql_on cassandra-0 "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
```

Repeat your canary invocation from [Check the starting state](#check-the-starting-state)
and confirm that the services report no authentication errors:

```bash
AUTH_LOGS=$(kc -n nvcf logs deploy/nvcf-api --since=10m)
printf '%s\n' "$AUTH_LOGS" | grep -ci 'AuthenticationException' || true
```

The count must be `0`. [Raise replication for HA](#raise-replication-for-ha) later in
this guide repeats the application keyspace step, which is safe to run again.

The loop above also raises `schema_migrations`. Its replication factor is `1` on a
single-member cluster, so after the expansion the migration ledger can sit on a
different member from the one the migration job reads, and the job replays
migrations that already ran. Compare the ledger on every member with the copy you
saved in [Back up persistent state](#back-up-persistent-state). The block asks for
the password and defines the helper itself, so it also runs in a new shell:

```bash
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql_on() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i "$1" -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$2"
)
for pod in cassandra-0 cassandra-1 cassandra-2; do
  for table in $(cut -d: -f1 "$BACKUP_DIR/schema-migrations-ledger.txt"); do
    printf '%s: ' "$table"
    cql_on "$pod" "SELECT version, dirty FROM schema_migrations.$table;" | tr -d ' \r' | grep -E '^[0-9]+\|' || echo "no row"
  done > "$BACKUP_DIR/schema-migrations-ledger-$pod.txt"
  if diff "$BACKUP_DIR/schema-migrations-ledger-$pod.txt" "$BACKUP_DIR/schema-migrations-ledger.txt"; then
    echo "migration ledger matches on $pod"
  else
    echo "migration ledger differs on $pod" >&2
    exit 1
  fi
done
unset CASSANDRA_PASSWORD
```

All three members must print `migration ledger matches`. If any differs, stop. Do not
continue to the upgrade; repair `schema_migrations` again and run this block again,
or restore from the backups.

### Apply the dependency stage

This stage updates Cassandra, NATS, cert-manager, OpenBao and the PKI. Review
the diff first, and stop on unexpected deletions or version overrides:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl diff --suppress-secrets | redact_diff
```

Apply it:

```bash
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl apply
control_helmfile -f helmfile.d/01-dependencies.yaml.gotmpl status
for namespace in nats-system cert-manager vault-system cassandra-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
  kc -n "$namespace" get jobs
done
```

What you will see:

- Cassandra restarts one pod at a time because of the config change (the server
  image stays at 2.0.7), and grows to 3 members under `preferred` or
  `enforced`. A single-member cluster is unavailable while its one pod restarts.
- Migrations run as Helm hook jobs. They must complete successfully. The
  Cassandra migrations image moves to 0.19.4 from 0.17.8. The
  new migrations only add schema: nullable columns on `sis_api.requests`,
  `sis_api.instances` and the `event_ledger` tables, and indexes on the new
  `sis_api` `creation_bucket` column. They drop nothing. Existing rows keep
  null in the new columns, and nothing backfills them. Building the indexes
  reads the existing rows, so a large `sis_api` table can make that job run
  longer.
- The OpenBao hooks (image 0.23.4) refresh the service, admin issuer and Event
  Ledger configuration, and add a signing role for gRPC proxy events. They
  only add or merge configuration and keep existing roles.
- OpenBao shows the new template, but its pods keep running the previous
  template until you check them in [Check the OpenBao pods](#check-the-openbao-pods).
  `wait_workloads` reports it as `OnDelete`.

Check that the data is intact, not only that pods are Ready:

```bash
kc -n cassandra-system exec cassandra-0 -c cassandra -- nodetool status
kc -n cassandra-system get jobs
kc -n vault-system get jobs
```

All Cassandra members must be `UN` and every migration job `Complete`.
Investigate any failed job before continuing; do not disable hooks.

### Check the OpenBao pods

The server image is the same as in 1.0.4 (`nvcf-openbao:2.6.3-nv-1.3.9`), and
1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle, so this
upgrade does not repeat the plugin update or the bridge step. The chart does
change, from 0.32.9 to 0.37.1, and the StatefulSet uses `OnDelete`: Helm updates
the template, but the running pods keep the template they started with. Check
whether any member still needs replacing:

```bash
kc -n vault-system get statefulset openbao-server \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}{"updated: "}{.status.updatedReplicas}{"/"}{.spec.replicas}{"\n"}'
check_openbao
```

The server image must be `nvcf-openbao:2.6.3-nv-1.3.9` and the strategy
`OnDelete`. If either is different, stop. With `OnDelete`, the StatefulSet can
keep reporting the old `currentRevision` after every member is replaced, so read
`updated:` instead. When it shows `3/3`, every member already runs the new
template; do not replace anything and continue with the core services. When it
shows fewer, replace the members that are not updated, one at a time, standbys
first, as below. Never delete a persistent volume claim, reinitialize OpenBao,
or replace more than one member at once. This is the member replacement
procedure from the [OpenBao patch upgrade runbook](/nvcf/dev/self-managed/open-bao-patch-upgrade).
Use that runbook if a step fails or if you customized plugin registration,
policies or PKI.

This procedure is written for the supported OpenBao deployment, a Raft cluster
of several members. Replace a standby first so that leadership does not move
twice. Check the member count:

```bash
kc -n vault-system get statefulset openbao-server -o jsonpath='{.spec.replicas}{"\n"}'
```

If it prints `1`, stop. A single-member OpenBao is not a supported deployment and
this guide has no procedure for it: there is no standby to replace first and no
leadership handover, and OpenBao is unavailable while its one pod restarts and
unseals. Do not continue on your own. Confirm the Raft snapshot from the backup
step and use the
[OpenBao patch upgrade runbook](/nvcf/dev/self-managed/open-bao-patch-upgrade)
to plan the change with your NVCF contact.

Identify the standbys (the members that do not report `is_self: true`) and
replace one:

```bash
POD="<standby-pod-name>"
kc -n vault-system delete pod "$POD" --wait=true --timeout=3m
kc -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=5m
check_openbao
```

If the new pod has not appeared yet, wait for it before repeating the wait. The
replacement must report `version: "2.6.3"`, `sealed: false` and the original
`cluster_id`. If it is Ready but still sealed, let auto-unseal finish and run
`check_openbao` again. Repeat for the second standby only after all three
members are healthy.

Then replace the former active member. Transfer leadership first:

```bash
ACTIVE_POD="<current-active-pod-name>"
set -o pipefail
kc -n vault-system get secret openbao-server-root-token -o json |
  jq -er '.data.root_token | @base64d' |
  kc -n vault-system exec -i "$ACTIVE_POD" -c openbao -- sh -ec '
    IFS= read -r BAO_TOKEN
    test -n "$BAO_TOKEN"
    export BAO_TOKEN
    export BAO_ADDR=http://127.0.0.1:8200
    bao operator step-down
  '
check_openbao
```

Wait until a 2.6.3 member reports `is_self: true` and the old active reports
standby. A successful request alone does not confirm the transfer. Then replace
the former active with the same `delete`, `wait`, `check_openbao` sequence. Do
not enable shell tracing or print the token.

Verify all three members and their images:

```bash
check_openbao
kc -n vault-system get pods -l app.kubernetes.io/name=openbao,component=server \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{"  "}{.name}{": "}{.image}{"\n"}{end}{end}'
```

All three must be unsealed on 2.6.3, with one active member and the original
cluster ID, and `updated:` must show `3/3`. Updating the injector affects only new pods. The services the core
stage rolls in the next step pick up the new agent; roll any other injected
workload through its normal deployment procedure, one service at a time, and
check its authentication and rendered secrets. A single-replica service can be
interrupted during that roll, so never restart all consumers together.

### Apply the core services

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl diff --suppress-secrets
control_helmfile -f helmfile.d/02-core.yaml.gotmpl apply
control_helmfile -f helmfile.d/02-core.yaml.gotmpl status
for namespace in api-keys sis nvcf ess nats-system; do
  wait_workloads "$CONTROL_KUBECONFIG" "$namespace"
done
kc get httproutes -A
```

Core releases must be `deployed`, their workloads ready and their routes
`Accepted`. The new `event-ledger` release appears in the `nvcf` namespace
unless you turned it off. Under `preferred` or `enforced`, services scale to 2
replicas and roll with `maxUnavailable: 0`, so a rollout needs room for one
extra pod per service. Check any add-on namespaces you enable (LLM, vanity
gateway, UI) the same way.

If a release fails or a rollout stalls, see
[A release stays failed after a stalled rollout](#a-release-stays-failed-after-a-stalled-rollout).

Repeat your canary invocation now. It must return the result you recorded.

### Apply state metrics and autoscaling

Review the diff first, and stop on unexpected changes:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl diff --suppress-secrets
```

Then apply it:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl apply
control_helmfile --allow-no-matching-release \
  -f helmfile.d/03-observability.yaml.gotmpl status
wait_workloads "$CONTROL_KUBECONFIG" nvcf
```

Keep your existing observability profile. Enabled state-metrics and autoscaler
releases must be `deployed` and healthy. Do not turn off state metrics to get
past a validation failure; the `control` and `all` profiles require it. Also
reapply the observability infrastructure stage. It applies the Observability
version in `$WORKDIR/observability`. With 1.0.0, changed configuration can still
update releases. With 1.1.0, the new charts replace the 1.0.0 ones. Review the
diff first:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl diff --suppress-secrets
```

Stop on unexpected changes. Then apply the stage:

```bash
control_helmfile --allow-no-matching-release \
  -f helmfile.d/00-observability-infrastructure.yaml.gotmpl apply
```

### Raise replication for HA

Skip this step if you chose `highAvailability.mode: none`.

After the dependency stage, Cassandra has 3 members, but your application
keyspaces keep the replication factor they were created with, so losing one
Cassandra node can still make data unavailable. Wait until all three Cassandra
pods are Ready, then raise each keyspace to RF=3 and repair, one member at a
time. The loop below alters only the keyspaces your installation has and skips
the rest. It validates the deployed data-center name against the live database
before any ALTER. Credentials travel over stdin into a temporary protected
[cqlsh credentials file](https://cassandra.apache.org/doc/stable/cassandra/managing/tools/cqlsh.html#credentials).
Do not enable shell tracing.

Each `ALTER KEYSPACE` in this loop replaces the whole replication map with the
data center it validates. If your keyspaces also replicate to another data center,
add that entry to every `ALTER`, as described in
[Expand a single-member Cassandra first](#expand-a-single-member-cassandra-first).

```bash
CASSANDRA_DC=$(kc -n cassandra-system get statefulset cassandra -o json |
  jq -er '.spec.template.spec.containers[] | select(.name == "cassandra") |
    .env[] | select(.name == "CASSANDRA_DC") | .value')
case "$CASSANDRA_DC" in
  ''|*[!A-Za-z0-9_-]*) echo "Invalid Cassandra data-center name" >&2; exit 1 ;;
esac
read -rs -p "Cassandra administrator password: " CASSANDRA_PASSWORD
printf '\n'
cql() (
  set +x
  printf '%s\n' "$CASSANDRA_PASSWORD" |
    kc -n cassandra-system exec -i cassandra-0 -c cassandra -- sh -ec '
      umask 077
      auth_dir=$(mktemp -d)
      trap '\''rm -rf "$auth_dir"'\'' EXIT
      IFS= read -r password
      printf "[auth_provider]\nmodule = cassandra.auth\nclassname = PlainTextAuthProvider\n" > "$auth_dir/cqlshrc"
      printf "[PlainTextAuthProvider]\nusername = cassandra\npassword = %s\n" "$password" > "$auth_dir/credentials"
      cqlsh --cqlshrc="$auth_dir/cqlshrc" --credentials="$auth_dir/credentials" --no-color -e "$1"
    ' sh "$1"
)
LIVE_DC=$(cql "SELECT JSON data_center FROM system.local;" |
  sed -n '/^[[:space:]]*{/{s/^[[:space:]]*//;p;}' |
  jq -ers 'if length == 1 then .[0].data_center else error("Missing data-center row") end')
test "$LIVE_DC" = "$CASSANDRA_DC"
cql "SELECT keyspace_name, replication FROM system_schema.keyspaces;"
EXISTING_KEYSPACES=$(cql "SELECT keyspace_name FROM system_schema.keyspaces;" | tr -d ' \r')
for ks in schema_migrations nvcf_api nvct_api ess_api api_keys_api sis_api event_ledger nvcf_autoscaler; do
  if printf '%s\n' "$EXISTING_KEYSPACES" | grep -qx "$ks"; then
    cql "ALTER KEYSPACE $ks WITH replication = {'class': 'NetworkTopologyStrategy', '$CASSANDRA_DC': '3'};"
  else
    echo "skipping $ks: not present in this installation"
  fi
done
for pod in cassandra-0 cassandra-1 cassandra-2; do
  kc -n cassandra-system exec "$pod" -c cassandra -- nodetool repair --full
done
unset CASSANDRA_PASSWORD
```

Until the repairs finish, reads at `LOCAL_ONE` can miss rows that exist only on
the original replica. The application keyspaces must now report RF=3 for the validated data center.

JetStream streams are created with the replication factor current at creation
time. The `icms` streams raise themselves to the configured factor when it
starts. Streams created earlier by `nvcf-api` and `invocation-service` stay at
1 replica until you edit them. List them and check:

```bash
kc -n nats-system exec deploy/nats-box -- nats stream report
```

The `Replicas` column lists each member that holds a copy of the stream. A
highly available stream shows three (`nats-0`, `nats-1`, `nats-2`); a stream
created before HA shows one. Raise each stream that shows one:

```bash
STREAM="<stream-name>"
kc -n nats-system exec deploy/nats-box -- nats stream edit "$STREAM" --replicas=3 --force
kc -n nats-system exec deploy/nats-box -- nats stream info "$STREAM"
```

Replace `<stream-name>` with the name from the report, run both commands for
each such stream, and confirm `nats stream info` shows `Replicas: 3`. Use 3, not
2: a 2-replica stream loses quorum when either member is down. The NATS
commands run in the `nats-box` deployment because the `nats` server container
does not include the CLI.

### Write the upgrade receipt

Run this only after every earlier stage succeeded.

Complete the [receipt image pull check](#render-and-review-the-changes)
before reaching this stage. Recheck it if the image or pull credentials changed.

```bash
check_selected_stack_versions
control_helmfile -f helmfile.d/04-upgrade-receipt.yaml.gotmpl apply
kc -n nvcf get job upgrade-receipt --ignore-not-found
kc -n nvcf get configmap nvcf-upgrade-receipt -o yaml
test "$(kc -n nvcf get configmap nvcf-upgrade-receipt \
  -o jsonpath='{.data.installed_stack_version}')" = "$STACK_VERSION"
echo "receipt matches the selected Self-Managed release"
```

The receipt must match the selected Self-Managed 1.1.9 release. It does not
record the outcome of workload checks. The hook job is removed after ten
minutes; the ConfigMap is the lasting record.

## 5. Verify the control plane

Check releases, versions and images:

```bash
helm --kubeconfig "$CONTROL_KUBECONFIG" list -A
kc -n nvcf get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n api-keys get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n sis get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
kc -n ess get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image
```

Key images you should see:

| Service | Image |
| --- | --- |
| nvcf-api | `nvcf-service-oss:1.28.4` |
| api-keys | `nvcf-api-keys-service:1.10.10` |
| nvct-api | `nvct-service-oss:1.70.4` |
| ess-api | `nvcf-ess:0.5.7` |
| sis (ICMS) | `icms-service-oss:0.8.6` |
| notary-service | `nvcf-notary:1.14.8` |
| invocation-service | `nvcf-invocation-service:0.15.1` |
| grpc-proxy | `nvcf-grpc-proxy:1.35.2` |
| event-ledger | `nvcf-event-ledger:0.21.2` |
| admin-issuer-proxy | `admin-token-issuer-proxy:1.1.3` |
| OpenBao server | `nvcf-openbao:2.6.3-nv-1.3.9` |
| Cassandra | `cassandra:5.0.9-nv-2.0.7` |

If you chose HA, confirm the replicas and spread (these are the same checks the
[high availability guide](/nvcf/dev/self-managed/high-availability#validation) uses):

```bash
kc get nodes -L topology.kubernetes.io/zone
kc -n cassandra-system get pods -o wide
kc -n nats-system get pods -o wide
kc -n vault-system get pods -o wide
kc -n nvcf get deploy nvcf-api -o wide
kc -n nvcf get deploy invocation-service grpc-proxy-deployment -o wide
kc -n ess get deploy ess-api-deployment -o wide
kc get pdb -A
```

Replica-safe services show 2 replicas, and Cassandra, NATS and OpenBao show 3.
`invocation-service` and `grpc-proxy` stay at 1 replica if you set a fixed
worker callback URL. Do not resume traffic until the checks pass. Repeat the
canary from [Check the starting state](#check-the-starting-state), create and invoke a new function with a secret, and confirm
queues make progress. Existing workers keep the images they started with.

## 6. Upgrade each compute cluster

Do this for every GPU cluster, after the control plane is verified. Use the
cluster's existing registration; the cluster, group and account identity must
not change.

### Set the variables

```bash
export COMPUTE_CONTEXT="<gpu-cluster-kube-context>"
export CLUSTER_NAME="<gpu-cluster-name>"
export NCA_ID="<existing-account-id>"
export REGISTRATION_VALUES_DIR="/secure/path/to/existing-registration"
export COMPUTE_KUBECONFIG="$WORKDIR/compute-$CLUSTER_NAME.kubeconfig"
kubectl config view --minify --flatten --raw --context "$COMPUTE_CONTEXT" > "$COMPUTE_KUBECONFIG"
kg() { kubectl --kubeconfig "$COMPUTE_KUBECONFIG" "$@"; }
record_identity() {
  kg -n nvca-operator get configmap nvcfbackend-self-managed \
    -o jsonpath='{.data.cluster-dto\.yaml}' |
    yq -o=json '{"clusterId": .clusterId, "clusterGroupId": .clusterGroupId,
      "clusterName": .clusterName, "ncaID": .ncaID}' |
    jq -eS 'if all(.[]; type == "string" and length > 0)
      then . else error("Missing cluster identity") end'
}
kg get nodes
record_identity
```

If the cluster name or account ID was unknown, set `CLUSTER_NAME` and `NCA_ID`
from this output before continuing. The registration file must exist and
match that identity; every node and the NVCA backend must be healthy:

```bash
test -s "$REGISTRATION_VALUES_DIR/$CLUSTER_NAME-register-values.yaml"
declare -p COMPUTE_CONTEXT CLUSTER_NAME NCA_ID REGISTRATION_VALUES_DIR COMPUTE_KUBECONFIG >> "$WORKDIR/session.sh"
declare -f kg record_identity >> "$WORKDIR/helpers.sh"
record_identity | tee "$WORKDIR/identity-before-$CLUSTER_NAME.json"
kg -n nvca-operator get nvcfbackends
kg -n nvca-operator get pods
kg -n nvca-system get pods
```

### Carry your settings into the compute package

```bash
# diff exits 1 when the files differ, which is expected; any other status is an error.
diff -u "$OLD_COMPUTE_STACK/environments/base.yaml" \
  "$OLD_COMPUTE_STACK/environments/$HELMFILE_ENV.yaml" \
  > "$WORKDIR/compute-overrides.diff" || [ "$?" -eq 1 ]
cat "$WORKDIR/compute-overrides.diff"
cp "$COMPUTE_STACK/environments/base.yaml" "$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml"
```

Re-apply each `+` line from `compute-overrides.diff` in
`$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml` (registry, image settings,
observability profile and similar). Do not copy old `nvcaVersion`, `imageTag`
or chart pins over the 1.1.1 defaults. Leave `workerCallbackCaPem` empty unless
you serve worker callbacks over TLS with a private CA. If your clusters share
one environment file, do this once.

### Check images, render and review

```bash
make -C "$COMPUTE_STACK" template HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
grep -rhoE 'image:[[:space:]]*"?[A-Za-z0-9._/:@-]+' "$WORKDIR/compute-render-$CLUSTER_NAME" \
  | sed -E 's/image:[[:space:]]*"?//' | sort -u
```

Mirror every image the list shows (NVCA 3.18.10, image credential helper
0.11.4 and the collectors 0.160.0-nv-0.4.9 are new defaults) before you apply. Then review the diff:

```bash
(cd "$COMPUTE_STACK" && HELMFILE_ENV="$HELMFILE_ENV" CLUSTER_NAME="$CLUSTER_NAME" \
  NCA_ID="$NCA_ID" OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME" \
  helmfile --environment default --kubeconfig "$COMPUTE_KUBECONFIG" \
  --kube-context "$COMPUTE_CONTEXT" diff --suppress-secrets)
```

Expect the `nvca-operator` chart to move from 1.28.0 to 1.37.1 and the NVCA
agent to 3.18.10. Stop if the diff shows the registration or cluster identity
changing, or unexpected deletions.

### Apply

```bash
make -C "$COMPUTE_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" \
  CLUSTER_NAME="$CLUSTER_NAME" NCA_ID="$NCA_ID" \
  REGISTRATION_VALUES_DIR="$REGISTRATION_VALUES_DIR" \
  KUBECONFIG_FILE="$COMPUTE_KUBECONFIG" COMPUTE_KUBE_CONTEXT="$COMPUTE_CONTEXT" \
  OUTPUT_DIR="$WORKDIR/compute-render-$CLUSTER_NAME"
wait_workloads "$COMPUTE_KUBECONFIG" nvca-operator
kg -n nvca-system get pods
kg -n nvca-operator get nvcfbackends
record_identity > "$WORKDIR/identity-after-$CLUSTER_NAME.json"
diff -u "$WORKDIR/identity-before-$CLUSTER_NAME.json" "$WORKDIR/identity-after-$CLUSTER_NAME.json" \
  && echo "cluster identity unchanged"
```

The operator and agent must be ready, the backend must report healthy, and
the identity files must match. If they differ, stop and inspect them before
doing anything else. Do not install a second copy of shared observability
resources on separate compute clusters; keep your existing configuration.

Confirm from the control plane that the cluster is registered and healthy:

```bash
"$NVCF_CLI" --config "$NVCF_CLI_CONFIG" cluster list-registered
```

Existing function and task canaries must still work with their old worker
images. To move a function to the new worker defaults, replace its deployment
through the [deployment API or CLI](/nvcf/dev/overview/cli#deploy-function); updating the
stack defaults alone does not change running workers or explicit per-function
image overrides. Plan for the interruption while a replacement rolls.

Repeat this step for each remaining GPU cluster, with its own `CLUSTER_NAME`,
`NCA_ID`, registration directory and kube context.

## 7. Resume workloads

Reapply the control plane and confirm it is stable:

```bash
make -C "$CONTROL_STACK" apply HELMFILE_ENV="$HELMFILE_ENV" KUBECONFIG_FILE="$CONTROL_KUBECONFIG"
control_helmfile status
set +e
control_helmfile diff --suppress-secrets --detailed-exitcode
DIFF_EXIT=$?
set -e
echo "diff exit code: $DIFF_EXIT"
if [ "$DIFF_EXIT" -ne 0 ]; then
  echo "STOP: the final diff exited $DIFF_EXIT. Inspect it before you resume traffic." >&2
  exit "$DIFF_EXIT"
fi
```

The final diff must report no changes and exit 0. A nonzero exit means there
are pending changes or a command error. Resolve them and rerun this gate
before continuing:

1. Repeat the canary invocations and compare IDs, results and persistent data
   with the values from [Check the starting state](#check-the-starting-state).
1. Create and invoke a new function and a new task with the inputs you normally
   use, including one function that reads a secret.
1. Resume production submissions and watch queue progress and telemetry.

Keep `$WORKDIR`, the diffs, the render directories and your notes with the
installed configuration as the upgrade record.

## Troubleshooting

| Symptom | Likely cause | What to do |
| --- | --- | --- |
| Render fails with `overrides the highAvailability sizing` | The secrets file sets a replica count that conflicts with the HA sizing. | Remove the override, or set `highAvailability.mode: none`. See [Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration). |
| `ImagePullBackOff` in a stage | An image is not in your registry or mirror. | Re-run the [Get the 1.1.9 packages and check artifacts](#get-the-119-packages-and-check-artifacts) checks, mirror the missing image, and retry the stage. |
| Pods `Pending` under `enforced` | Fewer than 3 nodes or zones with capacity, or a StorageClass without `WaitForFirstConsumer`. | Add capacity or fix the StorageClass. `preferred` schedules them but does not give zone protection. |
| A migration job failed | A dependency was not ready, or a hook could not reach its service. | Read the job logs (`kc -n <namespace> logs job/<name>`), fix the cause, and rerun the same stage. Do not disable hooks. |
| OpenBao member sealed after replacement | Auto-unseal has not finished. | Wait, then rerun `check_openbao`. If it stays sealed after five minutes, follow the runbook's recovery guidance. |
| Cassandra member not `UN` | The pod is restarting or joining. | Wait for it to become Ready and rerun `nodetool status`; do not continue while any member is down. |
| The image check prints `BACKWARD` | An old image pin survived in your environment file. | Stop. Remove pins you did not set on purpose ([Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration)) and render again. |
| A NATS pod is not Ready after [Apply the dependency stage](#apply-the-dependency-stage) | The NATS pods run as a non-root user (uid 1000), or an image could not be pulled. | Read the pod events and logs. Confirm the images pull from your registry or mirror ([Prepare the HA mode and configuration](#prepare-the-ha-mode-and-configuration)) and that the volume is writable by group 1000. |
| A release fails with `context deadline exceeded` or stays `failed` | A rolling update could not start its extra pod, usually for lack of CPU or memory on one node. | Free capacity and rerun the stage. See [A release stays failed after a stalled rollout](#a-release-stays-failed-after-a-stalled-rollout). |
| Compute plane identity changed | The registration file or `NCA_ID` was wrong. | Stop. Restore the original registration file and rerun the diff before applying. |

### A release stays failed after a stalled rollout

A release can fail with `context deadline exceeded` when a rolling update cannot
start its extra pod, usually for lack of CPU or memory on one node. Stop and
free capacity, then rerun the same stage apply. If the release then stays
`failed` in `helm list` because Helmfile sees no diff, record it as deployed
before you continue, with the name of the failed release:

```bash
control_helmfile -f helmfile.d/02-core.yaml.gotmpl --selector name=<release> sync
```

Use the stage file that owns the release. Then confirm that every release is
`deployed` and every workload is ready before the next stage.

## Optional: enable the worker callback proxy

The proxy is off by default, and this upgrade does not turn it on. Enable it
after the upgrade passes if you run the control plane and the compute plane in
different clusters and want more than one replica of the invocation service or
gRPC proxy. Set `workerCallbackProxy.enabled: true` and
`workerCallbackProxy.workerDomain` in your environment file. The proxy needs its
own DNS name, ports and certificate; follow
[Worker callback proxy](/nvcf/dev/self-managed/worker-callback-proxy), including
[Expose the proxy and its DNS name](/nvcf/dev/self-managed/worker-callback-proxy#expose-the-proxy-and-its-dns-name).
The stack refuses to render if the proxy is on while a fixed callback URL such
as `global.workerEndpoints.invocationServiceURL` is set. For the replica rules,
see [Multi-cluster installs](/nvcf/dev/self-managed/high-availability).

## Rollback boundary

If an upgrade fails, stop and use your deployment's reviewed recovery
procedure. Do not apply your old packages unchanged as a blanket rollback.
Helm rollback does not reverse stored data, does not validate a database or
OpenBao downgrade, and does not revert a Cassandra replication factor you
raised. Keep the OpenBao image and both JWT plugins that 1.0.4 installed; do not
return to an older OpenBao image. The Cassandra migrations in 1.1.9 add columns and indexes, and a rollback
does not remove them. Older services have not been tested against that schema. Restore data only through the reviewed procedure for that datastore,
using the backups from [Back up persistent state](#back-up-persistent-state). After any recovery, verify the original cluster
identities, old-image canaries, durable data and availability. Changing which
package you use does not roll back a running deployment.

## Related

- [Compatibility matrix](/nvcf/dev/release-notes/compatibility-matrix)
- [Artifact manifest](/nvcf/dev/manifest/artifacts)
- [High availability guide](/nvcf/dev/self-managed/high-availability)
- [OpenBao patch upgrade runbook](/nvcf/dev/self-managed/open-bao-patch-upgrade)
- [Control-plane operations](/nvcf/dev/self-managed/control-plane-operations)
- [1.0.2 to 1.0.4 upgrade](/nvcf/dev/release-notes/1-0-2-to-1-0-4-upgrade)
- [1.1.0 to 1.1.1 upgrade](/nvcf/dev/release-notes/1-1-0-to-1-1-1-upgrade)
- [Worker callback proxy](/nvcf/dev/self-managed/worker-callback-proxy)