Upgrade from NVCF 1.0.4 to NVCF 1.1.1
This upgrade procedure and its validation are still in progress. Follow the 1.1.1 qualification issue for the remaining work. Do not treat the publication of NVCF 1.1.1 as a qualified upgrade path until that issue records it.
This guide describes an upgrade of a running deployment from NVCF 1.0.4 to NVCF 1.1.1. NVCF 1.1.1 selects Self-Managed 1.1.9 for the control plane, Compute Plane 1.1.1 for each GPU cluster, and Observability 1.1.0. Read the whole guide, including What changes, before planning an upgrade.
This guide is for deployments on NVCF 1.0.4.
Choose your upgrade path
NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach NVCF 1.1.1:
Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.
The stack releases this guide covers are:
Cross-stack upgrade order
These declarations apply to the supported starting release in this guide:
Upgrade the control plane first, then each GPU cluster to Compute Plane 1.1.1.
Do not run a whole-stack make apply or helmfile sync first. The control
plane has stops that must happen in order: dependencies and migrations, the
OpenBao check, core services, then Cassandra replication for high availability.
Choose the high availability (HA) mode in
Prepare the HA mode and configuration
before you apply anything. The default, preferred, needs little preparation;
enforced needs the full preparation in that section.
What changes
Control plane: charts
The chart versions the 1.1.9 stack pins, compared with 1.0.10. The cert-manager
(0.2.0), ingress (1.18.3) and nvcf-ui (1.1.4) releases are unchanged.
The LLM and vanity gateway releases matter only if you enable those add-ons.
Control plane: data and security components
The data and security components below are the ones that matter for an in-place upgrade. Versions are the ones each release deploys, taken from the published stack inventories.
The NATS, cert-manager and OpenBao server images are the same as in 1.0.4, and 1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle. This upgrade does not repeat that plugin update.
Control plane: behavior that changes
cert-manager images come from upstream
NVCF 1.0.4 already pulls the cert-manager controller, webhook, cainjector,
acmesolver and startupapicheck images from the cert-manager chart’s upstream
repositories (quay.io/jetstack), and these images do not inherit
global.image. NVCF 1.1.0 had briefly defaulted them to the NVCF registry, and
1.1.1 returns to upstream. Coming from 1.0.4, the images stay the same
(v1.20.5).
Keep the image.repository settings you already use for a private mirror or an
air-gapped registry. Verify them before you apply the dependencies stage;
otherwise the cert-manager pods try to pull from the upstream registry and
fail. The settings are certManager.image.repository for the controller and
certManager.webhook.image.repository, certManager.cainjector.image.repository,
certManager.acmesolver.image.repository and certManager.startupapicheck.image.repository.
See upstream cert-manager images.
Observability
Observability 1.1.0 is optional. Self-Managed 1.1.9 also works with the
Observability 1.0.0 stack you already run, so you can upgrade Self-Managed on
its own and keep Observability at 1.0.0. The 1.1.0 release changes only two
charts and no images. In the default monitors it sets the job label on the
worker and DCGM series and adds pod target labels (function-id,
function-version-id and nca-id) to the worker series. The chart notes say
federation clients such as the BYOO collector select series by that job name,
and the autoscaler queries worker metrics by function identity. Upgrade
Observability if you rely on either. Function scaling and federation were not
exercised with Observability 1.0.0. The control-plane apply reads an
Observability package when the observability profile is enabled, so either
1.0.0 or 1.1.0 must be in place before the render
(see Get the 1.1.9 packages and check artifacts).
Compute plane
The values that differ between Compute Plane 1.0.5 and 1.1.1:
New optional settings: global.nvcaOperator.agent.workerCallbackCaPem (only
for TLS worker callbacks), byoo.* sidecar resources, utils.resources,
byooFluentBit and eventLedgerServiceURL. global.nvcaOperator.byooFluentBit
is off by default and its image defaults to docker.io/fluent/fluent-bit:3.2.
If you turn on bring-your-own observability in a disconnected install, mirror
that image and set image.registry, image.repository and image.tag. Dynamo, Grove and KAI Scheduler
pins are unchanged. Running function pods keep the worker images they started
with; the new defaults apply to deployments created or replaced afterwards.
Known issues
- A single-member OpenBao installation is not covered; see Check the OpenBao pods.
- Whether Compute Plane 1.0.5 keeps working against Self-Managed 1.1.9 during the upgrade window was not checked live.
Before starting
- Schedule a maintenance window. You will pause new function invocations and task submissions, and a single-member Cassandra restarts once.
- You need an operator workstation with
kubectl,helm3.12 or later,helmfile1.1.x (not 1.2.x), thehelm-diffplugin 3.11 or later,jq,curl, Mike Farah’syqv4, andngcif you download packages from NGC. Helmfile 1.2.0 removed sequential execution, which these stacks need. See required tools. - Keep your existing control-plane and compute-plane directories, your
environments/<environment-name>.yaml, yoursecrets/<environment-name>-secrets.yaml, and the registration file for each GPU cluster (<cluster-name>-register-values.yaml). The upgrade reuses each cluster’s registration. Do not runregister-clusteragain. - Choose canary workloads: one serving function, one function that reads a secret, and one running task. You will test them before and after.
- If you mirror images and charts to a private registry, mirror the 1.1.9 artifacts first. Get the 1.1.9 packages and check artifacts checks availability.
- Check the cert-manager image settings in cert-manager images come from upstream if the cluster uses a mirror.
1. Set deployment values
Set up the session
Download this read-only script into one directory. Check the starting state and Render and review the changes use it to stop an upgrade that would move an image to an older version:
Image move checkUse a dedicated shell. Replace every value in angle brackets. Run this block first, and again after any shell restart.
--suppress-secrets hides Kubernetes Secret data only. The OpenBao migration
Job carries the Cassandra password and the registry credential as plain
environment values, so a diff that includes it prints them. redact_diff
replaces the value of any environment entry whose name contains PASSWORD,
SECRET, TOKEN, KEY or CREDENTIAL. Pipe the dependency diff through it,
and keep that output private.
All nodes must be Ready. To resume later, start a new bash, run
set -euo pipefail, set WORKDIR to the directory you used above
(export WORKDIR="$HOME/nvcf-upgrade-1.1.9"), then
source "$WORKDIR/session.sh" and source "$WORKDIR/helpers.sh". The saved kubeconfig holds credentials, so keep
$WORKDIR private.
Get the 1.1.9 packages and check artifacts
Both exact target packages must be published and available to your customer account before maintenance. Stop if either download or package-stamp check fails. The NVCF 1.1.1 manifest selects these three targets together; do not mix them with other stack versions.
Download the three target packages next to each other, in a new directory, so the relative paths in the Helmfiles resolve the same way they did for your current release.
If you obtain the packages another way, extract them to these three
directories. If the observability profile is enabled (observability.profile is
not disabled), the control-plane Helmfile reads
../../observability/helmfile.d/01-observability.yaml.gotmpl, which resolves to
$WORKDIR/observability. That directory must hold an Observability package,
either 1.1.0 or your existing 1.0.0. To keep 1.0.0, export
OBSERVABILITY_STACK_VERSION=1.0.0 before running the download block, or copy
your existing 1.0.0 stack to that directory and skip its download. Leaving the stack only in
its original location makes the render fail. With the profile disabled the
directory is not read.
Check both new package stamps against the selected versions before maintenance:
Stop on a missing or mismatched stamp. Obtain a correctly stamped, qualified
package through the release process; do not edit VERSION, replace immutable
artifacts or substitute another stack version.
Confirm the charts and images are reachable from your registry. Set the
repositories to the ones in your environment file (global.helm.sources and
global.image), or to your mirror:
If your environment file uses an HTTPS Helm repository
(global.helm.sources.url) instead of an OCI registry, add it under an alias
first and use the alias as HELM_REPO below. The password is read from
standard input so it stays out of the process list:
For an OCI registry, set HELM_REPO to its address instead:
The list covers the images that change plus the data-service images. The
artifact manifest has the complete list, including optional
add-ons. Fix every MISSING line before continuing; a missing image shows up
later as ImagePullBackOff in the middle of a stage.
Prepare the HA mode and configuration
Choose the HA mode
preferred is the default and needs only enough schedulable nodes, so the
checks below are enough for it. enforced refuses to place a pod when a
requirement is missing, so complete every step under
Prepare for enforced mode before you apply
anything. If this is a single-node cluster, or you are not ready to add
capacity, choose none. You can move to HA later. If your Cassandra runs a
single member, first follow
Expand a single-member Cassandra first.
The full behavior, including zone spread, is in the
high availability guide.
Check what your cluster offers:
Replica-safe services roll with maxUnavailable: 0 and maxSurge: 1, so leave
room for one extra pod per service.
Prepare for enforced mode
Skip this section for none and preferred. With enforced, pods stay
Pending instead of co-locating, and an unmet requirement blocks the stage that
creates them. Complete every step before the first apply, and keep the results in
$WORKDIR:
-
Three nodes with capacity. Each Cassandra, NATS and OpenBao member needs its own node. Check that three nodes are
Ready, schedulable and not tainted against these workloads (kc get nodes), and that they have the CPU, memory and volume capacity for the extra replicas. Replica-safe services also run 2 replicas, so keep room for one extra pod per service during a rollout: free CPU and memory on at least one node must equal the largest pod request. -
Three zones. The nodes must be spread over three zones, each with capacity, and every node must carry the
topology.kubernetes.io/zonelabel. List the label with the first command above. A node without the label cannot take a zone-spread pod. -
A
WaitForFirstConsumerStorageClass. The StorageClass your Cassandra, NATS and OpenBao volumes use must showWaitForFirstConsumeras its volume binding mode (kc get storageclass). WithImmediatebinding, a volume can be created in a zone that has no node for its pod, and the pod staysPending. Existing volumes keep their zone, so check where each existing Cassandra, NATS and OpenBao volume lives, and that the zone has a schedulable node: -
Single-member Cassandra. If Cassandra runs one member, complete Expand a single-member Cassandra first before the dependency stage. The new members land in other zones, so check steps 1 to 3 for them too.
-
Replica overrides. Remove replica overrides from the secrets file, as in Check the secrets file for replica overrides. An override conflicts with the
enforcedsizing and fails the render. -
Render first. After you set the mode, render and review the diff in Render and review the changes. Stop if a workload would not fit.
If a pod stays Pending after the apply, run kc describe pod on it and read
the scheduling events before you change the mode. Moving back to preferred
later is an environment change and a re-apply of the affected stage.
Carry your settings into the new packages
The new base.yaml files contain the 1.1.9 defaults and chart settings. Copy
only your own changes into new environment files. Do not copy the whole old
file, because that would overwrite new defaults with old ones.
Open $WORKDIR/control-overrides.diff and re-apply each + line in
$CONTROL_STACK/environments/$HELMFILE_ENV.yaml. Typical lines are your
registry and chart source (global.helm.sources, global.image), Gateway
settings, storage class and size, node selectors and imagePullSecrets. Do not
carry over image tag or chart version overrides unless you deliberately pin
them; the 1.1.9 defaults are the qualified set. The 1.1.9 base.yaml no longer
pins cassandra.migrations.image.tag, so the chart default (0.19.4) applies.
If you pin that tag, remove the pin or set 0.19.4 or later: an older pin skips
the schema changes in Apply the dependency stage.
NVCF 1.0.4 and the 1.1.9 defaults pull the same NATS server, nats-box and
config reloader images from public registries (docker.io/synadia/nats-server,
docker.io/natsio/nats-box and docker.io/natsio/nats-server-config-reloader).
If you mirror them, keep the nats.container.image, nats.natsBox.container.image
and nats.reloader.image settings from your environment file. If you set none of
these and your cluster can pull from docker.io, there is nothing to do. The NATS
pods run as a non-root user, so confirm in
Apply the dependency stage that each NATS pod becomes Ready.
Then set the HA mode and, if you want to turn Event Ledger off, its switch. In
the environment file. This sample keeps the default, preferred; set enforced
only after you complete Prepare for enforced mode:
Merge these keys into the existing addons: section; do not duplicate it.
Check the secrets file for replica overrides
Under preferred and enforced, the render fails if the secrets file sets a
value that conflicts with the HA sizing, for example replicaCount: 1. Look
for them now:
Remove any replica override the HA sizing should control, or choose
highAvailability.mode: none. Record the mode you chose:
2. Check the starting state and take backups
Check the starting state
Do not continue unless every check passes.
Cassandra must report every member UN. The OpenBao check function used in
this guide, and again in Apply the dependency stage, is:
Every OpenBao member must be initialized: true and sealed: false, with the
same cluster_id. Exactly one reports is_self: true; standbys report
is_self: false or omit the field. Record the cluster ID.
Run your canary workloads now and record their results. With the configured
nvcf-cli:
Back up persistent state
Create restorable backups before you change anything. Follow your storage provider’s procedure for volume snapshots as well; the commands below create the application-level backups.
The Cassandra snapshot stays on each persistent volume. It is not an off-cluster backup, so also copy or snapshot the volumes.
Record the migration ledger too. Each application keyspace keeps its applied
migration version and dirty flag in a table of the schema_migrations keyspace.
The migration jobs read it to apply only the pending migrations, so a member that
cannot see a row makes a job replay a migration that already ran. The block saves
the ledger from the original member now, while that member holds all of it:
Every line must show a version and a dirty flag of False. Resolve a dirty ledger
before you continue, then run this block again so the saved file records the
corrected state. Do not run it again after the expansion: it would overwrite the
saved ledger with what the new members can see.
Take the OpenBao Raft snapshot from the active member. A snapshot taken from
a standby fails with incomplete snapshot.
The unseal and root-token files, the secrets copy and the exported values contain credentials. Protect them like the snapshot. NVCF NATS streams use memory storage, so a NATS backup holds stream metadata only. Do not continue until the Cassandra and OpenBao backups exist.
Render and review the changes
Render into a new directory, then review the diff the first stage would apply.
The render can include secrets, so keep it private. Check that the namespaces match your cluster and that no image lacks the registry you expect.
Compare the render with the images that run now. The check fails if any image would move to an older version, which happens when an old pin survives in your environment file:
Every BACKWARD line names an image and both versions. Stop on any of them. For
a pin you did not set on purpose, remove it as described in
Prepare the HA mode and configuration, render again and
re-run the check.
Before maintenance, verify the receipt image from the rendered Job. Its default
is docker.io/alpine/k8s:1.37.0; it does not follow global.image. For a mirror,
set upgradeReceipt.image.registry and upgradeReceipt.image.repository, then
render again. Use the resulting image reference for the check.
Run a temporary pull-test Pod in the Job’s namespace on each eligible node pool,
using that image, its pull credentials and scheduling settings. Set
imagePullPolicy: Always and run true instead of the receipt script. Confirm
the image pulls and the Pod succeeds, then remove the test Pod. A host-side
docker manifest inspect alone does not verify node access.
If you run tasks, complete the task worker image checks using this target render and the Self-Managed 1.1.9 inventory before proceeding.
3. Drain pending work
Stop new production requests and task submissions at their callers. NATS request queues use memory storage, and the dependency stage can restart NATS. Wait for queued work to finish; do not cancel it to get an empty queue.
For every affected function and version, use a key with queue_details scope
(NVCF_API_KEY). The task command below uses a separate key with list_tasks
scope (NVCF_NVCT_API_KEY) and your Tasks endpoint (base_nvct_url and
nvct_host in the CLI configuration). Verify both with the
CLI authentication settings before you start:
Queue sizes must reach zero. Record the identities and results of running tasks. Then check the control plane’s own memory streams on every NATS server, through the monitoring port 8222:
Every memory stream must show zero messages. A replicated stream (under
preferred or enforced HA) survives a single restart, but draining it keeps
the result unambiguous. A failed or slow read, a missing stream or a nonzero
count blocks the upgrade. A server whose accounts hold no streams reports no
stream details, which counts as empty; the final check still requires at least
one memory stream across the servers. Each attempt writes to a new directory, so files from an
earlier attempt cannot satisfy the check. Wait for the work to drain
and repeat; do not discard messages. If your NATS monitoring is configured
differently, use its endpoint and keep the same per-server check.
For retained creation messages or stuck deployments, follow the
recovery steps
before proceeding.
4. Apply control-plane stages in order
Expand a single-member Cassandra first
Do this only if CASSANDRA_REPLICAS from Back up persistent state
is 1 and you chose preferred or enforced. Skip it if Cassandra already runs
3 members, or if you chose none. Run it in the maintenance window, after you
drain pending work: while system_auth is being raised and repaired, new logins
can fail.
Do not let the upgrade expand Cassandra from one member to three. The chart
raises the system_auth replication factor to the new member count in a hook that
runs after the StatefulSet scales, and it does not repair the data. The new
members then hold none of the existing login data, so administrator and service
logins can fail while every member shows UN, and the dependency stage times out.
Expand and repair now, on the release you run today, so the upgrade starts from a
normal three-member cluster.
First, make cassandra-0 the only seed. Cassandra does not stream the existing
data to a member that lists itself as a seed, so a new member that appears in the
old seed list joins without the data. Set the seed list to the original member
only, for the expansion, so that the new members bootstrap. The upgrade later
restores the chart’s default two-seed list. The member turns Ready a few seconds
before it accepts CQL connections, so the block also waits for the CQL port. It
stops unless Cassandra is a single member:
This restarts the original single member. Before you continue, check four things:
the seed list contains only cassandra-0, the member is UN, the administrator
and service logins still work, and the migration ledger is unchanged from the copy
that Back up persistent state saved. The block asks
for the password and defines the helper itself, so it also runs in a new shell:
Repeat your canary invocation from Check the starting state, which checks the service logins, and confirm that the services report no authentication errors:
Stop if any check fails, and restore from the backups in Back up persistent state with your reviewed recovery procedure. Recovering an expansion that has already failed was not validated.
Place the new members in different zones. A member’s volume stays in the zone
where the member first starts, and the placement rules that the upgrade applies
later cannot move it. Under enforced, two members in one zone leave a pod
Pending during the upgrade. If every schedulable node has a
topology.kubernetes.io/zone label, add a zone spread rule to the StatefulSet
before you scale. The rule is hard under enforced and soft under preferred, so
a zone without capacity never leaves Cassandra Pending under preferred. This
restarts the single member once, and the upgrade replaces the rule with the
chart’s own. If any node has no zone label, skip this rule: a node without the
label cannot take a pod that has it.
Scale the StatefulSet directly. Do not set cassandra.replicaCount and apply the
old stack, because that runs the chart hook with the same problem. The block reads
the member count that the backup step saved, so it also works after a shell
restart, and it shows the live count. Scaling to 3 is safe to repeat. Wait for all
three members to be UN:
Every member must report UN. If a pod stays Pending, check the node and volume
capacity before you continue. Confirm that each member runs in a different zone:
Each ALTER KEYSPACE below replaces the whole replication map with the one data
center that this guide validates against the running cluster. If your keyspaces
also replicate to another data center, add that entry to every ALTER here and
in the application keyspace loop that follows, so the expansion does not remove it.
Raise the system_auth replication factor to 3, repair it on every member, then
confirm that the administrator can log in on every member. The password is that of
the cassandra administrator role, not the service role password in your secrets
file. The commands read it from standard input into a temporary protected file, so
do not enable shell tracing.
All three members must print administrator login ok. If any login fails, stop.
Do not continue to the upgrade. Restore from the backups in
Back up persistent state with your reviewed recovery
procedure, and contact NVIDIA support with the output.
Then raise the application keyspaces to 3 and repair them on every member. The loop alters only the keyspaces your installation has:
Repeat your canary invocation from Check the starting state and confirm that the services report no authentication errors:
The count must be 0. Raise replication for HA later in
this guide repeats the application keyspace step, which is safe to run again.
The loop above also raises schema_migrations. Its replication factor is 1 on a
single-member cluster, so after the expansion the migration ledger can sit on a
different member from the one the migration job reads, and the job replays
migrations that already ran. Compare the ledger on every member with the copy you
saved in Back up persistent state. The block asks for
the password and defines the helper itself, so it also runs in a new shell:
All three members must print migration ledger matches. If any differs, stop. Do not
continue to the upgrade; repair schema_migrations again and run this block again,
or restore from the backups.
Apply the dependency stage
This stage updates Cassandra, NATS, cert-manager, OpenBao and the PKI. Review the diff first, and stop on unexpected deletions or version overrides:
Apply it:
What you will see:
- Cassandra restarts one pod at a time because of the config change (the server
image stays at 2.0.7), and grows to 3 members under
preferredorenforced. A single-member cluster is unavailable while its one pod restarts. - Migrations run as Helm hook jobs. They must complete successfully. The
Cassandra migrations image moves to 0.19.4 from 0.17.8. The
new migrations only add schema: nullable columns on
sis_api.requests,sis_api.instancesand theevent_ledgertables, and indexes on the newsis_apicreation_bucketcolumn. They drop nothing. Existing rows keep null in the new columns, and nothing backfills them. Building the indexes reads the existing rows, so a largesis_apitable can make that job run longer. - The OpenBao hooks (image 0.23.4) refresh the service, admin issuer and Event Ledger configuration, and add a signing role for gRPC proxy events. They only add or merge configuration and keep existing roles.
- OpenBao shows the new template, but its pods keep running the previous
template until you check them in Check the OpenBao pods.
wait_workloadsreports it asOnDelete.
Check that the data is intact, not only that pods are Ready:
All Cassandra members must be UN and every migration job Complete.
Investigate any failed job before continuing; do not disable hooks.
Check the OpenBao pods
The server image is the same as in 1.0.4 (nvcf-openbao:2.6.3-nv-1.3.9), and
1.0.4 already moved OpenBao to the versioned JWT plugin lifecycle, so this
upgrade does not repeat the plugin update or the bridge step. The chart does
change, from 0.32.9 to 0.37.1, and the StatefulSet uses OnDelete: Helm updates
the template, but the running pods keep the template they started with. Check
whether any member still needs replacing:
The server image must be nvcf-openbao:2.6.3-nv-1.3.9 and the strategy
OnDelete. If either is different, stop. With OnDelete, the StatefulSet can
keep reporting the old currentRevision after every member is replaced, so read
updated: instead. When it shows 3/3, every member already runs the new
template; do not replace anything and continue with the core services. When it
shows fewer, replace the members that are not updated, one at a time, standbys
first, as below. Never delete a persistent volume claim, reinitialize OpenBao,
or replace more than one member at once. This is the member replacement
procedure from the OpenBao patch upgrade runbook.
Use that runbook if a step fails or if you customized plugin registration,
policies or PKI.
This procedure is written for the supported OpenBao deployment, a Raft cluster of several members. Replace a standby first so that leadership does not move twice. Check the member count:
If it prints 1, stop. A single-member OpenBao is not a supported deployment and
this guide has no procedure for it: there is no standby to replace first and no
leadership handover, and OpenBao is unavailable while its one pod restarts and
unseals. Do not continue on your own. Confirm the Raft snapshot from the backup
step and use the
OpenBao patch upgrade runbook
to plan the change with your NVCF contact.
Identify the standbys (the members that do not report is_self: true) and
replace one:
If the new pod has not appeared yet, wait for it before repeating the wait. The
replacement must report version: "2.6.3", sealed: false and the original
cluster_id. If it is Ready but still sealed, let auto-unseal finish and run
check_openbao again. Repeat for the second standby only after all three
members are healthy.
Then replace the former active member. Transfer leadership first:
Wait until a 2.6.3 member reports is_self: true and the old active reports
standby. A successful request alone does not confirm the transfer. Then replace
the former active with the same delete, wait, check_openbao sequence. Do
not enable shell tracing or print the token.
Verify all three members and their images:
All three must be unsealed on 2.6.3, with one active member and the original
cluster ID, and updated: must show 3/3. Updating the injector affects only new pods. The services the core
stage rolls in the next step pick up the new agent; roll any other injected
workload through its normal deployment procedure, one service at a time, and
check its authentication and rendered secrets. A single-replica service can be
interrupted during that roll, so never restart all consumers together.
Apply the core services
Core releases must be deployed, their workloads ready and their routes
Accepted. The new event-ledger release appears in the nvcf namespace
unless you turned it off. Under preferred or enforced, services scale to 2
replicas and roll with maxUnavailable: 0, so a rollout needs room for one
extra pod per service. Check any add-on namespaces you enable (LLM, vanity
gateway, UI) the same way.
If a release fails or a rollout stalls, see A release stays failed after a stalled rollout.
Repeat your canary invocation now. It must return the result you recorded.
Apply state metrics and autoscaling
Review the diff first, and stop on unexpected changes:
Then apply it:
Keep your existing observability profile. Enabled state-metrics and autoscaler
releases must be deployed and healthy. Do not turn off state metrics to get
past a validation failure; the control and all profiles require it. Also
reapply the observability infrastructure stage. It applies the Observability
version in $WORKDIR/observability. With 1.0.0, changed configuration can still
update releases. With 1.1.0, the new charts replace the 1.0.0 ones. Review the
diff first:
Stop on unexpected changes. Then apply the stage:
Raise replication for HA
Skip this step if you chose highAvailability.mode: none.
After the dependency stage, Cassandra has 3 members, but your application keyspaces keep the replication factor they were created with, so losing one Cassandra node can still make data unavailable. Wait until all three Cassandra pods are Ready, then raise each keyspace to RF=3 and repair, one member at a time. The loop below alters only the keyspaces your installation has and skips the rest. It validates the deployed data-center name against the live database before any ALTER. Credentials travel over stdin into a temporary protected cqlsh credentials file. Do not enable shell tracing.
Each ALTER KEYSPACE in this loop replaces the whole replication map with the
data center it validates. If your keyspaces also replicate to another data center,
add that entry to every ALTER, as described in
Expand a single-member Cassandra first.
Until the repairs finish, reads at LOCAL_ONE can miss rows that exist only on
the original replica. The application keyspaces must now report RF=3 for the validated data center.
JetStream streams are created with the replication factor current at creation
time. The icms streams raise themselves to the configured factor when it
starts. Streams created earlier by nvcf-api and invocation-service stay at
1 replica until you edit them. List them and check:
The Replicas column lists each member that holds a copy of the stream. A
highly available stream shows three (nats-0, nats-1, nats-2); a stream
created before HA shows one. Raise each stream that shows one:
Replace <stream-name> with the name from the report, run both commands for
each such stream, and confirm nats stream info shows Replicas: 3. Use 3, not
2: a 2-replica stream loses quorum when either member is down. The NATS
commands run in the nats-box deployment because the nats server container
does not include the CLI.
Write the upgrade receipt
Run this only after every earlier stage succeeded.
Complete the receipt image pull check before reaching this stage. Recheck it if the image or pull credentials changed.
The receipt must match the selected Self-Managed 1.1.9 release. It does not record the outcome of workload checks. The hook job is removed after ten minutes; the ConfigMap is the lasting record.
5. Verify the control plane
Check releases, versions and images:
Key images you should see:
If you chose HA, confirm the replicas and spread (these are the same checks the high availability guide uses):
Replica-safe services show 2 replicas, and Cassandra, NATS and OpenBao show 3.
invocation-service and grpc-proxy stay at 1 replica if you set a fixed
worker callback URL. Do not resume traffic until the checks pass. Repeat the
canary from Check the starting state, create and invoke a new function with a secret, and confirm
queues make progress. Existing workers keep the images they started with.
6. Upgrade each compute cluster
Do this for every GPU cluster, after the control plane is verified. Use the cluster’s existing registration; the cluster, group and account identity must not change.
Set the variables
If the cluster name or account ID was unknown, set CLUSTER_NAME and NCA_ID
from this output before continuing. The registration file must exist and
match that identity; every node and the NVCA backend must be healthy:
Carry your settings into the compute package
Re-apply each + line from compute-overrides.diff in
$COMPUTE_STACK/environments/$HELMFILE_ENV.yaml (registry, image settings,
observability profile and similar). Do not copy old nvcaVersion, imageTag
or chart pins over the 1.1.1 defaults. Leave workerCallbackCaPem empty unless
you serve worker callbacks over TLS with a private CA. If your clusters share
one environment file, do this once.
Check images, render and review
Mirror every image the list shows (NVCA 3.18.10, image credential helper 0.11.4 and the collectors 0.160.0-nv-0.4.9 are new defaults) before you apply. Then review the diff:
Expect the nvca-operator chart to move from 1.28.0 to 1.37.1 and the NVCA
agent to 3.18.10. Stop if the diff shows the registration or cluster identity
changing, or unexpected deletions.
Apply
The operator and agent must be ready, the backend must report healthy, and the identity files must match. If they differ, stop and inspect them before doing anything else. Do not install a second copy of shared observability resources on separate compute clusters; keep your existing configuration.
Confirm from the control plane that the cluster is registered and healthy:
Existing function and task canaries must still work with their old worker images. To move a function to the new worker defaults, replace its deployment through the deployment API or CLI; updating the stack defaults alone does not change running workers or explicit per-function image overrides. Plan for the interruption while a replacement rolls.
Repeat this step for each remaining GPU cluster, with its own CLUSTER_NAME,
NCA_ID, registration directory and kube context.
7. Resume workloads
Reapply the control plane and confirm it is stable:
The final diff must report no changes and exit 0. A nonzero exit means there are pending changes or a command error. Resolve them and rerun this gate before continuing:
- Repeat the canary invocations and compare IDs, results and persistent data with the values from Check the starting state.
- Create and invoke a new function and a new task with the inputs you normally use, including one function that reads a secret.
- Resume production submissions and watch queue progress and telemetry.
Keep $WORKDIR, the diffs, the render directories and your notes with the
installed configuration as the upgrade record.
Troubleshooting
A release stays failed after a stalled rollout
A release can fail with context deadline exceeded when a rolling update cannot
start its extra pod, usually for lack of CPU or memory on one node. Stop and
free capacity, then rerun the same stage apply. If the release then stays
failed in helm list because Helmfile sees no diff, record it as deployed
before you continue, with the name of the failed release:
Use the stage file that owns the release. Then confirm that every release is
deployed and every workload is ready before the next stage.
Optional: enable the worker callback proxy
The proxy is off by default, and this upgrade does not turn it on. Enable it
after the upgrade passes if you run the control plane and the compute plane in
different clusters and want more than one replica of the invocation service or
gRPC proxy. Set workerCallbackProxy.enabled: true and
workerCallbackProxy.workerDomain in your environment file. The proxy needs its
own DNS name, ports and certificate; follow
Worker callback proxy, including
Expose the proxy and its DNS name.
The stack refuses to render if the proxy is on while a fixed callback URL such
as global.workerEndpoints.invocationServiceURL is set. For the replica rules,
see Multi-cluster installs.
Rollback boundary
If an upgrade fails, stop and use your deployment’s reviewed recovery procedure. Do not apply your old packages unchanged as a blanket rollback. Helm rollback does not reverse stored data, does not validate a database or OpenBao downgrade, and does not revert a Cassandra replication factor you raised. Keep the OpenBao image and both JWT plugins that 1.0.4 installed; do not return to an older OpenBao image. The Cassandra migrations in 1.1.9 add columns and indexes, and a rollback does not remove them. Older services have not been tested against that schema. Restore data only through the reviewed procedure for that datastore, using the backups from Back up persistent state. After any recovery, verify the original cluster identities, old-image canaries, durable data and availability. Changing which package you use does not roll back a running deployment.