Upgrading NICo v2.1

setup.sh is designed to be idempotent for supported deployment topologies: running it against an existing NICo installation upgrades each component in place. The same script and values files used for initial installation are the mechanism for upgrades — there is no separate upgrade script. For a predecessor Flow Deployment that bundles PSM or NSM, first choose whether to preserve it or follow the manual Flow-only overwrite guidance.

This page documents how each component behaves when you re-run setup.sh against a live cluster, what to prepare before upgrading, and version-specific considerations for the 2.0-to-2.1 upgrade path.

How setup.sh handles upgrades

After any required manual Flow overwrite, every installation phase is safe to re-run:

What is preserved across upgrades

• Vault cluster: init state, unseal keys, PKI chain, AppRole credentials, all secrets. Vault is never re-initialized on an upgrade run.
• PostgreSQL data: all NICo Core and NICo REST database state, including host records, machine state, and site config in nico-pg-cluster. Keycloak realm data and Temporal workflow history are preserved too. They stay on nico-pg-cluster, or on the deprecated standalone postgres StatefulSet in the same namespace for a Site that still uses it. Upgrading with auto never moves a deployed database from one to the other. Refer to Standalone Temporal/Keycloak PostgreSQL deprecated.
• MetalLB site config: IPAddressPool, BGPPeer, BGPAdvertisement, and L2Advertisement instances. These are re-applied on every run so any manual changes outside setup.sh are reconciled back to the values in values/metallb-config.yaml.
• SSH host key: the cluster SSH identity is preserved so BMC consoles do not require known-host updates.
• Site UUID: the REST site identity and site-agent registration are preserved — as long as NICO_SITE_UUID is unset or identical to the initial install; a different value deletes and re-creates the site-registration Secret.
• Certificates: all cert-manager-managed certificates remain valid until their natural expiry; they are not reissued on upgrade unless the new release changes a Certificate spec.

What changes during an upgrade

• All Helm release images are updated to the new tags set in NICO_CORE_IMAGE_TAG, NICO_REST_IMAGE_TAG, etc.
• CRD schemas are updated to their new versions via server-side apply.
• ConfigMaps and Secrets produced by Helm are updated to reflect new chart values.
• The NICo Core and REST database schemas are migrated forward by their respective pre-upgrade Jobs.
• The DPF operator chart is reinstalled from the helm-prereqs/doca-platform commit the new setup.sh pins, with the operator image tag from NICO_DPF_IMAGE_TAG (default: that release). The DPUServices (dts, doca_hbn, and NICo’s dpu_agent, dhcp_server, fmds, otel) are versioned independently by [dpf.services.*] in the NICo site config (dpf.md §3.5); the pin does not change them.

Pre-upgrade checklist

Complete every item before running setup.sh. Missing any of these can cause the upgrade to fail or leave the cluster in a partially upgraded state.

If you are upgrading to asynchronous VPC peering deletion for the first time, plan an API maintenance window. You must stop the old API before starting the new one; the default rolling update does not do this.

If flow/flow or an active Flow Pod still contains a psm or nsm container, do not run setup yet. Preserve that release, or follow the manual Flow-only overwrite guidance. Setup rejects that predecessor topology or an incomplete Flow-only rollout before making any cluster change.

# Verify the backup Secret exists
kubectl get secret vault-cluster-keys -n vault -o jsonpath='{.metadata.name}'

umask 077   # keep the backup readable only by you
kubectl get secret vault-cluster-keys -n vault -o json > vault-cluster-keys-backup.json

tmp=$(mktemp nico-secrets-kek-backup.XXXXXX)   # created with mode 0600
kubectl get secret nico-secrets-kek -n nico-system -o json > "$tmp" \
  && mv "$tmp" nico-secrets-kek-backup.json   # keep the old backup if the export fails

umask 077
# nico-pg-cluster: nico_system_nico, nico_rest, flow, any retained predecessor psm/nsm databases,
# and temporal, temporal_visibility, keycloak when they use nico-pg-cluster
kubectl exec -n postgres \
    "$(kubectl get pods -n postgres -l application=spilo,spilo-role=master -o jsonpath='{.items[0].metadata.name}')" \
    -- su postgres -c "pg_dumpall" > nico_pg_pre_upgrade.sql
# standalone `postgres` StatefulSet (skip if the Site doesn't run it): temporal, temporal_visibility, keycloak
kubectl exec -n postgres postgres-0 -- pg_dumpall -U postgres > nico_rest_pg_pre_upgrade.sql

kubectl get pods -n nico-system
kubectl get pods -n nico-rest
kubectl get pods -n temporal
kubectl get pods -n vault
kubectl get pods -n postgres
kubectl get pods -n metallb-system
kubectl get pods -n dpf-operator-system   # if DPF is enabled — phase 5b upgrades it

kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.status.loadBalancer.ingress[0].ip}{"\n"}{end}' \
    > pre-upgrade-vips.txt

kubectl exec -n vault vault-0 -c vault -- vault status -tls-skip-verify | grep -E "Sealed|Initialized"

git fetch upstream
git checkout upstream/release/v2.1   # or the specific release tag

git diff upstream/release/v2.0..upstream/release/v2.1 -- helm-prereqs/values.yaml \
    helm-prereqs/values/nico-core.yaml \
    helm-prereqs/values/nico-rest.yaml \
    helm-prereqs/values/metallb-config.yaml

export NICO_IMAGE_REGISTRY=registry.example.com/nico   # your registry
export NICO_CORE_IMAGE_TAG=v2.1.0                      # new Core tag
export NICO_REST_IMAGE_TAG=v2.1.0                      # new REST tag

export NICO_DPF_DPU_INTERFACE=<DPU high-speed interface, e.g. ens1f0np0>
export NICO_DPF_DPU_CLUSTER_VIP=<VIP for the DPU cluster control plane>

cd helm-prereqs/
source ./preflight.sh

Running the upgrade

With the checklist complete — including the version-specific notes for your upgrade path, which describe behavior that happens during the run — run setup.sh exactly as you would for a fresh install:

cd helm-prereqs/
./setup.sh -y

setup.sh processes all phases in order. Phases that find their components already at the correct state complete quickly. Phases that detect a version delta or config change apply the update.

Upgrade-specific flags

Estimated upgrade time

Total: typically 15–45 minutes for a full upgrade with DPF.

Post-upgrade verification

Run the same checks as after initial installation:

kubectl get pods -n nico-system
kubectl get pods -n nico-rest
kubectl get pods -n temporal
kubectl get pods -n vault
kubectl get pods -n postgres
kubectl get pods -n metallb-system
kubectl get pods -n dpf-operator-system   # if DPF enabled

Verify the deployed image versions match the target tags:

kubectl get deployment -n nico-system nico-api \
    -o jsonpath='{.spec.template.spec.containers[0].image}'

Verify every LoadBalancer service in the cluster kept its VIP — an upgrade must not reassign them. Diff against the baseline captured in the pre-upgrade checklist (same stable namespace/name/VIP fields):

kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.status.loadBalancer.ingress[0].ip}{"\n"}{end}' \
    | diff pre-upgrade-vips.txt - && echo "VIPs unchanged"

Any diff output or a missing address means MetalLB reassigned or stopped advertising a VIP — check the MetalLB CRDs and site config objects (see the 2.0→2.1 note below) and the pins in values/nico-core.yaml.

Verify PostgreSQL has an elected leader and the cluster is running:

kubectl get postgresql -n postgres nico-pg-cluster -o jsonpath='{.status.PostgresClusterStatus}'
kubectl get pods -n postgres -l application=spilo,spilo-role=master

The first command should print Running; the second should show exactly one master pod in Running state.

Verify NICo Core is serving — hit the same HTTP health port the liveness probe uses (1080, exposed via the metrics Service):

kubectl run -i --rm --restart=Never --image=curlimages/curl upgrade-check \
  -n nico-system --quiet -- \
  -sf http://nico-api-metrics.nico-system.svc.cluster.local:1080/ >/dev/null && echo "nico-api healthy"

Then run the included health check, which covers the full stack (Vault, cert-manager, ESO, MetalLB, .forge DNS records):

cd helm-prereqs/
./health-check.sh

Moving credentials to Postgres

Migrating an existing site’s credentials from Vault to Postgres is a configuration walk described in Secrets Storage, and it can be folded into an upgrade window. Two points matter when you migrate:

• Every step changes the nico-api config and, in the reference installation, needs kubectl -n nico-system rollout restart deployment/nico-api: the chart carries the Stakater Reloader annotation, so installations that run Reloader restart automatically, but the reference installation does not install it.
• The default rolling update also keeps the old pod running until the new one is ready, so finish each step’s rollout before starting the next. Keep bmc_rotation_enabled and uefi_rotation_enabled at their default false until the whole fleet runs one config.

Once the writer is Postgres, the KEK backup above belongs in every pre-upgrade checklist.

Version-specific upgrade notes

Upgrades That Introduce Asynchronous Peering Deletion

Follow these steps the first time you upgrade to the peering deletion behavior added in #6917. You can upgrade directly; no intermediate release is required.

The new API waits for DPUs to stop allowing a peering before deleting it. An old API can still tell those DPUs to allow the connection, making the new API mistakenly treat deletion as finished. Stop every old API before starting any new one. The Helm chart’s default RollingUpdate can run both versions at once, even with just one replica.

During your API maintenance window:

1. Record the namespace, Deployment name, and desired replica count for every nico-api Deployment serving the site. Pause any automation that would restore the old replicas during the maintenance window.

2. Scale those Deployments to zero. Wait for all old API pods and any API processes outside those Deployments to exit and their requests to drain.

3. Run the normal upgrade with the new image and desired replica count. With setup.sh, set nico-api.replicas in the Core values file to that count. Phase 6 runs the Core migration Job and starts the new API pods. Keep the old processes stopped.

4. Check the API pods in each namespace before restoring automation:

   kubectl get pods -n <namespace> -l app.kubernetes.io/name=nico-api \
     -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image'

   Every expected API pod must be present, and its nico-api container must use the new image. The selector uses the chart’s default name; replace nico-api if you set nameOverride. For API processes outside those Deployments, check their container image or executable version through their service manager, and confirm that every old process has stopped. Follow the peering deletion guide to confirm any pending deletions finish.

Do not roll back to an API that lacks this behavior. It deletes peerings without waiting for the DPUs, and restoring a database backup does not undo their network changes. For the network-version and migration details, refer to the configuration reference.

2.3 → 2.4: Standalone Temporal/Keycloak PostgreSQL deprecated

Before 2.4, temporal.useHaPostgres and keycloak.useHaPostgres in helm-prereqs/values.yaml defaulted to false, so every new Site got the StatefulSet. They now default to auto. With auto, both databases are created in nico-pg-cluster, but their location is left unchanged for an existing Site. So an existing Site needs no changes for the upgrade, and a new Site only gets the StatefulSet when a value is set to false. With postgresql.enabled: false, nico-pg-cluster is an external or out-of-band PostgreSQL, so auto means false unless the databases are already there. Such a Site can move them with true once it provisions them, as Using an external or out-of-band PostgreSQL describes. Copying existing data from the StatefulSet onto an external or out-of-band PostgreSQL isn’t supported in 2.4. Support is tracked in #7989. While either database stays on the StatefulSet, preflight warns that it’s deprecated, and setup.sh repeats the warning at the end of the run.

On an existing Site, the new temporal, temporal_visibility, and keycloak databases on nico-pg-cluster stay empty and unused until it migrates. To migrate, set temporal.useHaPostgres, keycloak.useHaPostgres, or both to true, because auto never moves an existing database. Then run the migration script with the matching --db temporal, --db keycloak, or --db both, as the migration guide describes. setup.sh won’t move a database off the StatefulSet until the script has copied it. A values file copied from an earlier release may still set false. That keeps the databases on the StatefulSet and creates nothing on nico-pg-cluster. Once a database is on nico-pg-cluster, preflight refuses false, because there’s no supported way back. After both have moved, setup.sh leaves the StatefulSet and its data in place, so remove them as Removing the standalone StatefulSet describes.

2.2 → 2.3: Core components are mandatory

Impact: The core NICo subcharts (nico-api, nico-bmc-proxy, nico-dhcp, nico-dns, nico-hardware-health, nico-pxe, nico-ssh-console-rs) no longer have <chart>.enabled toggles - they are unconditional dependencies in helm/Chart.yaml and always install. A site values file (including one passed via setup.sh --core-values) that still sets a core <chart>.enabled key, whether true or false, keeps working: the key is ignored and the component installs. No values changes are required for the upgrade. If a site previously disabled a core component (for example, a DHCP or BMC proxy provided externally), plan for the in-cluster component to appear after the upgrade. Optional services (nico-dsx-exchange-consumer, nico-ntp, unbound) keep their toggles.

NICo Flow is also mandatory within the REST install: --skip-flow and the helm-prereqs flow.enabled value were removed, phase 7h has no skip flag of its own and runs whenever REST is installed (only --skip-rest skips it, together with all of REST), and the Flow chart moved from helm/charts/nico-flow to helm/nico-flow. A leftover nico-flow.enabled key in umbrella values is ignored, because the umbrella no longer deploys Flow; setup.sh installs it. Sites running an external PostgreSQL (postgresql.enabled: false in helm-prereqs values) must provision the flow database and the flow.nico.nico-pg-cluster.credentials Secret in the flow namespace before running setup.sh, or phase 7h fails after its 120s credential wait.

2.0 → 2.1: MetalLB CRD ownership migration

Impact: This upgrade path requires special handling that setup.sh performs automatically. If you skip MetalLB’s phase in your upgrade (for example, by removing it from the helmfile run), your site config objects (IPAddressPool, BGPPeer, BGPAdvertisement) will be deleted.

Root cause: In NICo 2.0, MetalLB was deployed with crds.enabled: true (the Helm chart default), which places all seven MetalLB CRDs inside the Helm release manifest. In NICo 2.1, crds.enabled: false is set explicitly so that the MetalLB cert rotator can take SSA field ownership of the CRD caBundle without conflicting with Helm on every re-sync. When Helm sees crds.enabled change from true to false, it removes the CRDs from its manifest — and Kubernetes garbage-collects every IPAddressPool, BGPPeer, and BGPAdvertisement instance stored as CRD resources, permanently deleting your site config.

How setup.sh handles this: Before running helmfile sync for MetalLB, setup.sh strips the app.kubernetes.io/managed-by: Helm label and the meta.helm.sh/release-name / meta.helm.sh/release-namespace annotations from any existing MetalLB CRDs. With the labels removed, Helm does not consider the CRDs part of its managed set and does not delete them during the sync. CRDs are then applied directly (server-side, --force-conflicts) before and after the helmfile sync.

If you are upgrading manually (not via setup.sh), you must strip Helm ownership from all MetalLB CRDs before running helmfile sync or helm upgrade:

for crd in $(kubectl get crd -o name | grep '\.metallb\.io$'); do
    kubectl annotate "${crd}" meta.helm.sh/release-name- meta.helm.sh/release-namespace- --overwrite
    kubectl label  "${crd}" app.kubernetes.io/managed-by- --overwrite
done

Then apply the CRDs directly before sync:

METALLB_VERSION="0.14.5"   # match the version in helmfile.yaml
helm template metallb metallb/metallb --version "${METALLB_VERSION}" \
    -n metallb-system --include-crds \
    | awk '/^---[[:space:]]*$/ { if (doc ~ /kind: CustomResourceDefinition/) printf "%s---\n", doc; doc = ""; next } { doc = doc $0 "\n" } END { if (doc ~ /kind: CustomResourceDefinition/) printf "%s", doc }' \
    | kubectl apply --server-side --force-conflicts -f -

2.0 → 2.1: DPF version update

The helm-prereqs/doca-platform submodule pin is updated with each NICo minor release to the tested DOCA Platform Framework version. On a 2.0→2.1 upgrade, DPF is upgraded from its 2.0 version to the 2.1 version automatically as part of phase 5b.

DPF manages DPU provisioning state in DPUCluster, DPUService, and DPF CRs, all of which persist across the upgrade. In-flight DPU provisioning workflows may pause while the DPF operator restarts; they resume automatically when the new operator pod comes up.

2.0 → 2.1: NICo Core startupProbe

NICo 2.1 requires startupProbe to be explicitly configured in the machine-a-tron deployment (issue #4298). The chart now validates this at render time and fails with a clear error if startupProbe is absent.

2.1 → 2.2: NICo REST postgres volume size

NICo 2.2 raises the postgres StatefulSet’s volumeClaimTemplates storage request from 1Gi to 10Gi. Kubernetes forbids changing that field on an existing StatefulSet, so phase 7c of setup.sh deletes the StatefulSet with --cascade=orphan and re-applies it; the postgres-0 pod and its postgres-data-postgres-0 PVC are kept. The PVC of an upgraded site stays at 1Gi. It can be left as is, or grown in place if the StorageClass has allowVolumeExpansion: true and its storage provisioner supports expansion:

kubectl patch pvc postgres-data-postgres-0 -n postgres \
    -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'

2.2 → 2.3: Machine-a-Tron startupProbe Default

NICo 2.3 raises the default startupProbe.failureThreshold from 20 to 120 (60 minutes), sized for a 250-rack site spread over ten pods (issue 5968). The threshold applies to each pod on its own, so size it for the pod that registers the most records. For larger sites, raise startupProbe.failureThreshold following the sizing rule in the chart’s values.yaml, as helm-prereqs/values/machine-a-tron-scale.yaml does.

2.2 → 2.3: Vault Probe Settings

NICo 2.3 raises the Vault server probe timeoutSeconds from 3 to 10 and failureThreshold from 2 to 5 in helm-prereqs/operators/values/vault.yaml.

For example, on a 250-rack site, the active node missed two 3-second liveness probes under load, was killed, came back sealed, and the standby nodes stayed leaderless.

The Vault StatefulSet uses updateStrategy: OnDelete. This means that helmfile sync updates the pod template, but running pods keep the old probes until they are deleted. Phase 3 prints a warning naming the pods still running the previous revision. If the StatefulSet status cannot be read or has not caught up, the warning indicates that the revision could not be verified.

After the upgrade finishes, find the active node, then roll the pods one at a time, standby nodes first, and the active node last. The active node is the pod whose HA Mode is active:

for pod in vault-0 vault-1 vault-2; do
    echo -n "${pod}: "; kubectl exec -n vault "${pod}" -c vault -- vault status -tls-skip-verify | grep 'HA Mode'
done

From the repository root, run the following commands for each standby in turn, and only then for the active node. Replace <pod> with the pod being rolled and <active> with the pod whose HA Mode is active at that point. A recreated pod starts sealed and is not Ready until it is unsealed, so the first wait is for Running:

kubectl delete pod -n vault <pod>
kubectl wait pod/<pod> -n vault --for=jsonpath='{.status.phase}'=Running --timeout=300s
helm-prereqs/unseal_vault.sh
kubectl wait pod/<pod> -n vault --for=condition=Ready --timeout=300s
kubectl -n vault get secret vaultroottoken -o jsonpath='{.data.token}' | base64 -d \
    | kubectl exec -i -n vault <active> -c vault -- \
        sh -c 'VAULT_TOKEN=$(cat) vault operator raft autopilot state -tls-skip-verify' \
    | sed '/^Servers:/,$d'

The root token travels on standard input; it does not appear in the kubectl exec arguments that the API server records in audit events. The sed operation trims the output to the cluster summary: Healthy, Failure Tolerance, Leader, and Voters.

unseal_vault.sh returns once vault status reports the pod unsealed. Unsealing only opens the pod’s own storage, and does not show that the pod has rejoined the cluster as a healthy voter. Continue to the next pod only when Healthy is true, Failure Tolerance is 1 on a three-node cluster, and <pod> is listed under Voters.

Right after the unseal, the summary can still show Healthy false and Failure Tolerance 0 until the leader hears from the pod. Rerun the check after a few seconds. A pod that comes back without its Raft data joins as a non-voter and appears under Voters only after autopilot promotes it. Deleting the next standby before the promotion can leave the three-node cluster below quorum. When the former active node is rolled, a standby has taken over; run the status loop again to find <active>.

If kubectl wait reports the pod as not found, the StatefulSet has not recreated it yet. Run the wait again.

2.2 → 2.3: Kustomize deployment deprecated

The Kustomize deployment under deploy/ (deploy/kustomization.yaml, deploy/nico-base, deploy/nico-system, deploy/nico-unbound-base) is deprecated in 2.3 and will be removed in 2.4. Every service it deploys has a Helm chart under helm/, and setup.sh installs NICo Core from those charts only. Kustomize deployments keep working in 2.3 but receive no new configuration. Before upgrading to 2.4, install with the Helm charts as described in the quick start and retire the Kustomize overlays. The rest-api/deploy/kustomize bases that setup.sh uses for cert-manager, PostgreSQL, and Temporal are not affected.

2.3 → 2.4: Kustomize deployment removed

The Kustomize deployment under deploy/ is removed in 2.4. Install and upgrade NICo Core with the Helm charts under helm/, either through setup.sh as described in the quick start or directly as described in helm/README.md. Sites still running the Kustomize manifests must move to the Helm charts before upgrading to 2.4. The annotated site configuration example now lives at helm/examples/nico-api-site-config.toml, and the .nico service endpoint reference moved to DNS Service Endpoints. The rest-api/deploy/kustomize bases that setup.sh uses are not affected.

Rollback

setup.sh does not have a built-in rollback mechanism. Rollback consists of:

1. Checking out the prior release branch or tag.
2. Re-running setup.sh with the prior image tags.

For NICo Core and REST, the Helm release is rolled back in-place, and the database migration Jobs for the prior version run on startup. NICo’s database migrations are designed to be forward-compatible; rolling back does not guarantee schema compatibility if the new version added non-nullable columns. This is why a pre-upgrade database backup is essential.

The database dumps from the pre-upgrade checklist are the rollback foundation: nico-pg-cluster holds nico_system_nico (NICo Core), nico_rest (NICo REST), flow, and any retained psm or nsm databases from a predecessor bundled-manager deployment. The temporal, temporal_visibility, and keycloak databases are there too, unless the Site still runs the standalone postgres StatefulSet. Restoring means replaying the SQL against a clean instance (psql -f <dump>.sql).

Re-running setup.sh with the prior image tags alone is not a complete rollback if the new version’s migrations already ran. Restore the database dumps first, then deploy the prior version.

If you have upgraded to asynchronous peering deletion, do not return to an API that lacks it, even after restoring the database. Fix the newer version instead.

For DPF, rolling back to a prior DPF version is not supported by NVIDIA. If DPF fails to upgrade, reach out to NVIDIA support rather than attempting a downgrade.

Using setup.sh for individual component upgrades

You can narrow an upgrade to particular components with the --skip-* flags. --skip-core and --skip-rest skip exactly the phase they name - none of them skip the prerequisite stack, and there is no --skip-prereqs. NICo Flow (Phase 7h) has no skip flag of its own; it upgrades together with REST and is skipped only by --skip-rest. (--skip-dpf is the exception: refer to its caveat in the flag table above. --skip-core also suppresses the imagepullsecret upsert that the Core migration Job uses.)

# Prerequisites + NICo Core; leave the REST stack untouched (skips Phase 7)
./setup.sh -y --skip-rest
# Prerequisites + NICo REST; leave Core untouched (skips Phase 6)
./setup.sh -y --skip-core
# Prerequisites only, fully non-interactive
./setup.sh -y --skip-core --skip-rest

The prerequisite phases therefore run on every invocation. That is by design and is cheap: each one is idempotent, and a phase whose inputs have not changed reconciles to the same state and exits quickly.

To upgrade only the NICo Core release without running the prerequisite phases, use a chart and Core image built from the same repository revision, following the image compatibility policy. Update both together when moving to a new revision. Run from the repository root, matching what setup.sh itself executes:

helm upgrade --install nico ./helm \
    -n nico-system \
    -f helm-prereqs/values/nico-core.yaml \
    --set-string global.image.repository="${NICO_IMAGE_REGISTRY}/nvmetal-carbide" \
    --set-string global.image.tag="${NICO_CORE_IMAGE_TAG}" \
    --timeout 600s --wait