Upgrade from NVCF 1.1.0 to NVCF 1.1.1
This upgrade procedure and its validation are still in progress. Follow the 1.1.1 qualification issue for the remaining work. Do not treat the publication of NVCF 1.1.1 as a qualified upgrade path until that issue records it.
Use this procedure for NVCF 1.1.0. NVCF 1.1.1 is a patch release: it fixes the emSign CA issue (CVE-2026-111943), adds the optional worker callback proxy for multi-cluster installs and moves OpenBao to a versioned JWT plugin lifecycle.
Plan a maintenance window based on a rehearsal of this procedure. NATS, gRPC proxy and Invocation Service rollovers, and the one-at-a-time OpenBao member replacement, can interrupt in-flight invocations and make the control-plane API unavailable for short periods.
Choose your upgrade path
NVCF 1.0.4 is the upgrade stop for every customer on a 1.0.x release. To reach NVCF 1.1.1:
Do not upgrade from a 1.0.x release other than 1.0.4 straight to 1.1.1, and do not use the 1.0.1 to 1.1.0 procedure for a 1.0.4 deployment.
What changes in 1.1.1
Compute Plane 1.1.1 changes NVCA and the NVCA operator from 3.18.6 to 3.18.10,
the collectors from 0.160.0-nv-0.4.7 to 0.160.0-nv-0.4.9, and the NATS
reloader from 0.24.0 to 0.25.0, which matches Self-Managed. Observability stays
on 1.1.0. Upgrade the control plane first, then each GPU cluster to Compute
Plane 1.1.1.
Other changes you may notice:
- Component images are rebuilt with Go 1.26.10 and the patched CA bundle.
- The worker callback proxy is new and off by default. See Optional: enable the worker callback proxy.
api.accountBootstrap.enabledlets an install whose account is provisioned another way turn off the account-bootstrap hook Job. Unset, nothing changes.- You can set operator pod annotations per release, for example
api.podAnnotationsandnats.podAnnotations. Unset, nothing is rendered. ingress.gatewayApi.routes.llmApiGateway.enabledandingress.gatewayApi.routes.llmInvocation.enabledturn off an LLM route while the LLM add-on is on.- The NVCF UI add-on and the LLS HMAC rotation hook now use the shared
global.openbaoMigrations.image.tag(0.23.4). The UI add-on pinned0.19.10in 1.1.0, so its image changes too. If you pinaddons.nvcfUi.openbaoMigrations.image.tag, review that pin against0.23.4.
cert-manager images come from upstream
1.1.0 defaulted the cert-manager controller, webhook, cainjector and acmesolver
images to your configured NVCF registry, <global.image.registry>/<global.image.repository>/cert-manager-<component>.
1.1.1 uses the repositories of cert-manager chart 0.2.0, which are upstream
(quay.io/jetstack), and those images do not inherit global.image.
If the cluster pulls from a private mirror or has no access to the upstream registry, do this before you apply the dependencies stage. Otherwise the cert-manager pods try to pull from the upstream registry and fail.
- Mirror the upstream images for the five components:
controller,webhook,cainjector,acmesolverandstartupapicheck. See upstream cert-manager images. - Set the repository of each component in
environments/<name>.yaml, for examplecertManager.image.repositoryfor the controller andcertManager.webhook.image.repository,certManager.cainjector.image.repository,certManager.acmesolver.image.repositoryandcertManager.startupapicheck.image.repository. - After the dependencies stage, list the cert-manager pod images and confirm they come from your mirror.
Prerequisites
- Python 3, Helm 3.21.4, Helmfile 1.1.9,
kubectl,jq,curl, Mike Farah’syqv4, and the helm-diff plugin. - The Python
cryptographypackage, to verify JWT canary signatures against the mount’s JWKS. nvcf-cli1.16.8 or later from the NGC download instructions.- Download the NVCF 1.1.1 packaged release and manifest. Mirror the required
artifacts to your configured registry; see image mirroring.
Include both task worker images if you use tasks and the compute-plane Samba
image. The upstream supporting images
(NATS server/reloader/box and account-bootstrap/receipt helpers) need their own
mirror settings; they do not inherit
global.image. - Extract the stacks under
deploy/stacks/self-managed,deploy/stacks/nvcf-compute-planeanddeploy/stacks/observabilityin a new bundle directory. - Spare capacity for rolling updates. A rollout starts one extra pod per Deployment before it removes an old one. Keep free CPU and memory on at least one node equal to the largest pod request, or a stage can stall.
Before starting
- Back up configuration, databases and persistent volumes.
- Prepare the target environment and secrets files.
Transfer deployment settings and secrets from
environments/<name>.yamlandsecrets/<name>-secrets.yaml; keep the target chart/image versions. Do not copy image tags or chart versions from your 1.1.0 environment file over the target defaults unless you mean to pin them. - Retain the compute-plane registration file and account ID. Do not run
register-clusteragain. - Locate the kubeconfig for each control-plane and compute cluster. Step 1
exports them as
CONTROL_KUBECONFIGandCOMPUTE_KUBECONFIG, and every later step uses those variables. In a new shell, export the step 1 values again before continuing. - Complete the cert-manager image settings if the cluster uses a mirror.
- Schedule a maintenance window based on a rehearsal of this procedure.
1. Set deployment values
Download these read-only scripts into one directory:
Workload readiness check Receipt check JWT mount preservation checkUse Bash with set -euo pipefail so a failed command stops the procedure.
Fill in your existing deployment values below. Each kubeconfig must already
select its intended context. For separate clusters, replace the two COMPUTE_*
values; for a single cluster, keep them equal to the control plane.
REGISTRATION_VALUES_DIR contains <cluster-name>-register-values.yaml.
Set the stack paths and define the helper used below:
--suppress-secrets hides Kubernetes Secret data only. The OpenBao migration
Job carries the Cassandra password and the registry credential as plain
environment values, so its diff prints them. redact_diff replaces the value of
any environment entry whose name contains PASSWORD, SECRET, TOKEN, KEY or
CREDENTIAL. Pipe every OpenBao and dependency diff through it, and keep that
output private.
2. Drain pending work
Pause new production requests and task submissions. Check every affected function/version before rolling NATS or dependent services.
The checks in this step take under one minute, even for large clusters. The total drain time depends on your longest in-flight request or running task. Plan the maintenance window around that duration.
Configure CLI authentication: NVCF_API_KEY needs queue_details;
NVCF_NVCT_API_KEY needs list_tasks. Point base_nvct_url and nvct_host
at this control plane’s Tasks endpoint.
Require queue size zero for every affected queue; a missing response blocks progress. Let running tasks finish; do not discard work to clear queues.
Check the NATS memory streams on every NATS server too. Choose an unused local port for monitoring port 8222 and keep submissions paused. The temporary directory holds working files for this check and the OpenBao transition:
Require every memory stream to be present and empty. A replicated stream (under
preferred or enforced HA) survives a single restart, but draining it keeps
the result unambiguous. Failed reads, missing streams or nonzero counts block
the rollout. If responses are paginated,
retrieve every account/stream page before accepting the result. Use your configured
monitoring endpoint if it differs.
Keep NATS running while work drains. For stuck deployments or retained creation
messages, follow the read-only recovery steps.
For a retry, use a new nats-drain directory; keep the OpenBao baseline.
3. Apply control-plane stages in order
Keep submissions paused. Review each Helmfile diff before confirming the apply. Wait for each stage to succeed before continuing.
OpenBao bridge before dependency migrations
NVCF 1.1.0 runs OpenBao image 2.6.3-nv-1.3.7 and does not set a JWT plugin
lifecycle mode in its environment file. The 1.1.1 stack defaults to versioned
and the target uses chart 0.37.1, server/agent 2.6.3-nv-1.3.9 and
migrations 0.23.4. An existing installation must not jump straight to that
default. Move to it in stages: set bridge mode, replace every member, then
switch to versioned. The bridge retains the installed JWT
plugin alongside the new one. Existing mounts keep their plugin selection; new
mounts use the target plugin. Existing mounts receive its fixes through a
separate plugin migration.
Before changing OpenBao:
- Complete the snapshot and health checks
and define
check_openbao. Require three healthy, unsealed members on the same architecture, the original Raft cluster ID and retained unseal material. The active member is the pod whosecheck_openbaooutput reportsis_self: true. - Follow the JWT preflight
to define
bao_admin. SetBAO_PODto the active member from the previous item. Use the control-planeKUBECONFIGand verify a working JWT canary with the JWT canary. For the preflightBRIDGE_IMAGE, use the target server image from your configured registry, for examplenvcr.io/nvidia/nvcf/nvcf-openbao:2.6.3-nv-1.3.9. - Pause external mount creation and tuning. Capture the baseline once:
Stop on missing metadata or an incomplete plugin transition. The bridge hook must match the retained binary to the stored catalog SHA. On mismatch, obtain a qualified bridge with the matching binary; do not rewrite the SHA or bypass hooks.
Set bridge mode and review the OpenBao diff, leaving NATS running:
Stop on unexpected changes. Then apply only OpenBao:
Bridge mode pauses migrations and catalog refresh. OnDelete leaves server
pods on the old image until you replace them.
Follow the standby replacement and leadership transfer
for every member, one at a time. Before the next replacement, verify unsealed
health, the original cluster ID, the selected bridge digest and JWT issuance.
Update BAO_POD after leadership changes.
With OnDelete, the StatefulSet can keep reporting the old currentRevision
after every member is replaced. Confirm the rollout with updatedReplicas
equal to replicas instead:
Once all members run the bridge, check retained mount versions/SHAs and enable migrations. New mounts must use the target catalog entry:
Review the diff, then apply the versioned mode and check the mounts again:
Keep versioned mode for subsequent applies. It enables migrations and new
mounts without retuning existing ones. Verify existing and new secret consumers,
including JWT issuance and signature checks after normal failover. Retain both
binaries and catalog entries.
After a shell failure, redefine check_openbao, bao_admin and
check_upgrade_jwt_mounts, then reselect BAO_POD. Keep the original baseline.
Dependencies and migrations
Proceed only after the OpenBao checks pass. Use this staged apply instead of
the patch runbook’s whole-stack make apply:
Stop on unexpected deletions or version overrides. Then apply it:
Require successful dependency releases and migration jobs before continuing. The NATS server rolls to 2.14.9 here. Confirm that the cert-manager pods run the images you selected in cert-manager images come from upstream.
Core services
Review the diff first, and stop on unexpected deletions or version overrides:
Then apply it:
Require deployed releases, ready workloads and accepted routes, including
any additional enabled addon namespaces. The worker-callback-proxy release is
off by default and does not appear until you enable it.
If a release fails or a rollout stalls, see A release stays failed after a stalled rollout.
State metrics and autoscaling
Review the diff first:
Stop on unexpected changes. Then apply the stage:
Keep your existing observability profile. Enabled state-metrics and autoscaler
releases must be deployed and healthy. The function autoscaler chart moves to
0.8.1. The Observability stack stays on 1.1.0, so skip its infrastructure stage.
Upgrade receipt
After all stages pass, write and check the receipt:
Require the receipt to match the selected Self-Managed version. The ConfigMap persists after the hook job expires. To correct a stale receipt, verify the installed package and workloads, then rerun that package’s receipt stage; do not patch the ConfigMap.
4. Verify the control plane
Do not upgrade the compute plane until the control plane is healthy. Check the releases, the receipt and the routes, and repeat your canary invocation:
Every release must be deployed, the routes Accepted, and the receipt must
show 1.1.9. Then run the canary invocation from Resume workloads
against a function that is already deployed; it must return the result you
expect.
5. Upgrade each compute cluster
Do this for every GPU cluster, after the control plane is verified. Apply
Compute Plane 1.1.1 with the cluster’s retained registration and account ID.
Use a separate output directory from the registration file, and repeat the
command for each cluster with its own CLUSTER_NAME, NCA_ID, registration
directory and kube context:
Wait for ready operator/agent pods and a healthy backend. Keep the existing Observability 1.1.0 configuration on separate compute clusters.
6. Resume workloads
Invoke an existing function with its known request and expected response:
For LLM functions, use the LLM invocation options. Check secret access, then resume submissions and monitor queue progress. Existing workers keep their images until redeployed; stack defaults apply to new workers unless their deployment specifies an image override. Redeploy functions and tasks to pick up the patched NICLLS worker and worker images.
Optional: enable the worker callback proxy
The proxy is off by default, and this upgrade does not turn it on. Enable it
after the upgrade passes if you run the control plane and the compute plane in
different clusters and want more than one replica of the invocation service or
gRPC proxy. Set workerCallbackProxy.enabled: true and
workerCallbackProxy.workerDomain in your environment file. The proxy needs its
own DNS name, ports and certificate; follow
Worker callback proxy, including
Expose the proxy and its DNS name.
The stack refuses to render if the proxy is on while a fixed callback URL such
as global.workerEndpoints.invocationServiceURL is set. For the replica rules,
see Multi-cluster installs.
Troubleshooting
A release stays failed after a stalled rollout
A release can fail with context deadline exceeded when a rolling update cannot
start its extra pod, usually for lack of CPU or memory on one node. Stop and
free capacity, then rerun the same stage apply. If the release then stays
failed in helm list because Helmfile sees no diff, record it as deployed
before you continue, with the name of the failed release:
Use the stage file that owns the release. Then confirm that every release is
deployed and every workload is ready before the next stage.
Rollback boundary
Stop on failure and follow your reviewed recovery plan:
- Retain patched NVCA and the target Tasks settings. Do not apply an old bundle unchanged.
- Keep the OpenBao bridge image and both plugins. OpenBao cannot return to image
2.6.3-nv-1.3.7or chart0.36.1after the lifecycle change, and database migrations are not reversed. Plugin rollback and cleanup belong to the separate plugin migration procedure. - Verify original identities, existing workloads and durable data after recovery. If you run tasks, launch a new task before ending recovery.
Application recovery does not validate server/agent downgrades or reverse stored data. Use the OpenBao recovery boundary and your datastore recovery plan for restores.