Upgrading NICo v2.1 New

setup.sh is designed to be idempotent: running it against an existing NICo installation upgrades each component in place. The same script and values files used for initial installation are the mechanism for upgrades — there is no separate upgrade script.

This page documents how each component behaves when you re-run setup.sh against a live cluster, what to prepare before upgrading, and version-specific considerations for the 2.0-to-2.1 upgrade path.

How setup.sh handles upgrades

Every installation phase is safe to re-run:

What is preserved across upgrades

• Vault cluster: init state, unseal keys, PKI chain, AppRole credentials, all secrets. Vault is never re-initialized on an upgrade run.
• PostgreSQL data: all NICo Core and NICo REST database state, including host records, machine state, and site config in nico-pg-cluster. Keycloak realm data and Temporal workflow history live in a separate plain postgres StatefulSet in the same namespace (created in phase 7c) and are also preserved.
• MetalLB site config: IPAddressPool, BGPPeer, BGPAdvertisement, and L2Advertisement instances. These are re-applied on every run so any manual changes outside setup.sh are reconciled back to the values in values/metallb-config.yaml.
• SSH host key: the cluster SSH identity is preserved so BMC consoles do not require known-host updates.
• Site UUID: the REST site identity and site-agent registration are preserved — as long as NICO_SITE_UUID is unset or identical to the initial install; a different value deletes and re-creates the site-registration Secret.
• Certificates: all cert-manager-managed certificates remain valid until their natural expiry; they are not reissued on upgrade unless the new release changes a Certificate spec.

What changes during an upgrade

• All Helm release images are updated to the new tags set in NICO_CORE_IMAGE_TAG, NICO_REST_IMAGE_TAG, etc.
• CRD schemas are updated to their new versions via server-side apply.
• ConfigMaps and Secrets produced by Helm are updated to reflect new chart values.
• The NICo Core and REST database schemas are migrated forward by their respective pre-upgrade Jobs.
• DPF operator and DPUService images are updated to the new NICO_DPF_VERSION.

Pre-upgrade checklist

Complete every item before running setup.sh. Missing any of these can cause the upgrade to fail or leave the cluster in a partially upgraded state.

$
# Verify the backup Secret exists
$
kubectl get secret vault-cluster-keys -n vault -o jsonpath='{.metadata.name}'

$
umask 077   # keep the backup readable only by you
$
kubectl get secret vault-cluster-keys -n vault -o json > vault-cluster-keys-backup.json

$
umask 077
$
# nico-pg-cluster: nico_system_nico, nico_rest, and (with Flow) flow/psm/nsm
$
kubectl exec -n postgres \
>
    "$(kubectl get pods -n postgres -l application=spilo,spilo-role=master -o jsonpath='{.items[0].metadata.name}')" \
>
    -- su postgres -c "pg_dumpall" > nico_pg_pre_upgrade.sql
$
$
# plain `postgres` StatefulSet: temporal, temporal_visibility, keycloak
$
kubectl exec -n postgres postgres-0 -- pg_dumpall -U postgres > nico_rest_pg_pre_upgrade.sql

$
kubectl get pods -n nico-system
$
kubectl get pods -n nico-rest
$
kubectl get pods -n temporal
$
kubectl get pods -n vault
$
kubectl get pods -n postgres
$
kubectl get pods -n metallb-system
$
kubectl get pods -n dpf-operator-system   # if DPF is enabled — phase 5b upgrades it

$
kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.status.loadBalancer.ingress[0].ip}{"\n"}{end}' \
>
    > pre-upgrade-vips.txt

$
kubectl exec -n vault vault-0 -c vault -- vault status -tls-skip-verify | grep -E "Sealed|Initialized"

$
git fetch upstream
$
git checkout upstream/release/v2.1   # or the specific release tag

$
git diff upstream/release/v2.0..upstream/release/v2.1 -- helm-prereqs/values.yaml \
>
    helm-prereqs/values/nico-core.yaml \
>
    helm-prereqs/values/nico-rest.yaml \
>
    helm-prereqs/values/metallb-config.yaml

$
export NICO_IMAGE_REGISTRY=registry.example.com/nico   # your registry
$
export NICO_CORE_IMAGE_TAG=v2.1.0                      # new Core tag
$
export NICO_REST_IMAGE_TAG=v2.1.0                      # new REST tag

$
export NICO_DPF_DPU_INTERFACE=<DPU high-speed interface, e.g. ens1f0np0>
$
export NICO_DPF_DPU_CLUSTER_VIP=<VIP for the DPU cluster control plane>

$
cd helm-prereqs/
$
source ./preflight.sh

Running the upgrade

With the checklist complete — including the version-specific notes for your upgrade path, which describe behavior that happens during the run — run setup.sh exactly as you would for a fresh install:

$
cd helm-prereqs/
$
./setup.sh -y

setup.sh processes all phases in order. Phases that find their components already at the correct state complete quickly. Phases that detect a version delta or config change apply the update.

Upgrade-specific flags

Estimated upgrade time

Total: typically 15–45 minutes for a full upgrade with DPF.

Post-upgrade verification

Run the same checks as after initial installation:

$
kubectl get pods -n nico-system
$
kubectl get pods -n nico-rest
$
kubectl get pods -n temporal
$
kubectl get pods -n vault
$
kubectl get pods -n postgres
$
kubectl get pods -n metallb-system
$
kubectl get pods -n dpf-operator-system   # if DPF enabled

Verify the deployed image versions match the target tags:

$
kubectl get deployment -n nico-system nico-api \
>
    -o jsonpath='{.spec.template.spec.containers[0].image}'

Verify every LoadBalancer service in the cluster kept its VIP — an upgrade must not reassign them. Diff against the baseline captured in the pre-upgrade checklist (same stable namespace/name/VIP fields):

$
kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.status.loadBalancer.ingress[0].ip}{"\n"}{end}' \
>
    | diff pre-upgrade-vips.txt - && echo "VIPs unchanged"

Any diff output or a missing address means MetalLB reassigned or stopped advertising a VIP — check the MetalLB CRDs and site config objects (see the 2.0→2.1 note below) and the pins in values/nico-core.yaml.

Verify PostgreSQL has an elected leader and the cluster is running:

$
kubectl get postgresql -n postgres nico-pg-cluster -o jsonpath='{.status.PostgresClusterStatus}'
$
kubectl get pods -n postgres -l application=spilo,spilo-role=master

The first command should print Running; the second should show exactly one master pod in Running state.

Verify NICo Core is serving — hit the same HTTP health port the liveness probe uses (1080, exposed via the metrics Service):

$
kubectl run -i --rm --restart=Never --image=curlimages/curl upgrade-check \
>
  -n nico-system --quiet -- \
>
  -sf http://nico-api-metrics.nico-system.svc.cluster.local:1080/ >/dev/null && echo "nico-api healthy"

Then run the included health check, which covers the full stack (Vault, cert-manager, ESO, MetalLB, .forge DNS records):

$
cd helm-prereqs/
$
./health-check.sh

Version-specific upgrade notes

2.0 → 2.1: MetalLB CRD ownership migration

Impact: This upgrade path requires special handling that setup.sh performs automatically. If you skip MetalLB’s phase in your upgrade (for example, by removing it from the helmfile run), your site config objects (IPAddressPool, BGPPeer, BGPAdvertisement) will be deleted.

Root cause: In NICo 2.0, MetalLB was deployed with crds.enabled: true (the Helm chart default), which places all seven MetalLB CRDs inside the Helm release manifest. In NICo 2.1, crds.enabled: false is set explicitly so that the MetalLB cert rotator can take SSA field ownership of the CRD caBundle without conflicting with Helm on every re-sync. When Helm sees crds.enabled change from true to false, it removes the CRDs from its manifest — and Kubernetes garbage-collects every IPAddressPool, BGPPeer, and BGPAdvertisement instance stored as CRD resources, permanently deleting your site config.

How setup.sh handles this: Before running helmfile sync for MetalLB, setup.sh strips the app.kubernetes.io/managed-by: Helm label and the meta.helm.sh/release-name / meta.helm.sh/release-namespace annotations from any existing MetalLB CRDs. With the labels removed, Helm does not consider the CRDs part of its managed set and does not delete them during the sync. CRDs are then applied directly (server-side, --force-conflicts) before and after the helmfile sync.

If you are upgrading manually (not via setup.sh), you must strip Helm ownership from all MetalLB CRDs before running helmfile sync or helm upgrade:

$
for crd in $(kubectl get crd -o name | grep '\.metallb\.io$'); do
$
    kubectl annotate "${crd}" meta.helm.sh/release-name- meta.helm.sh/release-namespace- --overwrite
$
    kubectl label  "${crd}" app.kubernetes.io/managed-by- --overwrite
$
done

Then apply the CRDs directly before sync:

$
METALLB_VERSION="0.14.5"   # match the version in helmfile.yaml
$
helm template metallb metallb/metallb --version "${METALLB_VERSION}" \
>
    -n metallb-system --include-crds \
>
    | awk '/^---[[:space:]]*$/ { if (doc ~ /kind: CustomResourceDefinition/) printf "%s---\n", doc; doc = ""; next } { doc = doc $0 "\n" } END { if (doc ~ /kind: CustomResourceDefinition/) printf "%s", doc }' \
>
    | kubectl apply --server-side --force-conflicts -f -

2.0 → 2.1: DPF version update

The default NICO_DPF_VERSION in setup.sh is updated with each NICo minor release to the tested DOCA Platform Framework version. On a 2.0→2.1 upgrade, DPF is upgraded from its 2.0 version to the 2.1 version automatically as part of phase 5b.

DPF manages DPU provisioning state in DPUCluster, DPUService, and DPF CRs, all of which persist across the upgrade. In-flight DPU provisioning workflows may pause while the DPF operator restarts; they resume automatically when the new operator pod comes up.

2.0 → 2.1: NICo Core startupProbe

NICo 2.1 requires startupProbe to be explicitly configured in the machine-a-tron deployment (issue #4298). The chart now validates this at render time and fails with a clear error if startupProbe is absent. The default values provide a suitable probe scaled to ~2,300 hosts; for larger sites, refer to helm-prereqs/values/machine-a-tron-scale.yaml for recommended parameters scaled to 13,500 hosts.

Rollback

setup.sh does not have a built-in rollback mechanism. Rollback consists of:

1. Checking out the prior release branch or tag.
2. Re-running setup.sh with the prior image tags.

For NICo Core and REST, the Helm release is rolled back in-place, and the database migration Jobs for the prior version run on startup. NICo’s database migrations are designed to be forward-compatible; rolling back does not guarantee schema compatibility if the new version added non-nullable columns. This is why a pre-upgrade database backup is essential.

The database dumps from the pre-upgrade checklist are the rollback foundation: nico-pg-cluster holds nico_system_nico (NICo Core), nico_rest (NICo REST), and — when Flow is enabled — flow, psm, and nsm, while the plain postgres StatefulSet holds temporal, temporal_visibility, and keycloak. Restoring means replaying the SQL against a clean instance (psql -f <dump>.sql).

Re-running setup.sh with the prior image tags alone is not a complete rollback if the new version’s migrations already ran. Restore the database dumps first, then deploy the prior version.

For DPF, rolling back to a prior DPF version is not supported by NVIDIA. If DPF fails to upgrade, reach out to NVIDIA support rather than attempting a downgrade.

Using setup.sh for individual component upgrades

You can narrow an upgrade to particular components with the --skip-* flags. --skip-core, --skip-rest, and --skip-flow skip exactly the phase they name — none of them skip the prerequisite stack, and there is no --skip-prereqs. (--skip-dpf is the exception: refer to its caveat in the flag table above. --skip-core also suppresses the imagepullsecret upsert that the Core migration Job uses.)

$
# Prerequisites + NICo Core; leave the REST stack untouched (skips Phase 7)
$
./setup.sh -y --skip-rest
$
$
# Prerequisites + NICo REST; leave Core untouched (skips Phase 6)
$
./setup.sh -y --skip-core
$
$
# Prerequisites only, fully non-interactive
$
./setup.sh -y --skip-core --skip-rest

The prerequisite phases therefore run on every invocation. That is by design and is cheap: each one is idempotent, and a phase whose inputs have not changed reconciles to the same state and exits quickly.

For a single-Helm-chart upgrade (such as rotating the NICo Core image tag without going through the full script), run from the repository root, matching what setup.sh itself executes:

$
helm upgrade --install nico ./helm \
>
    -n nico-system \
>
    -f helm-prereqs/values/nico-core.yaml \
>
    --set-string global.image.repository="${NICO_IMAGE_REGISTRY}/nvmetal-carbide" \
>
    --set-string global.image.tag="${NICO_CORE_IMAGE_TAG}" \
>
    --timeout 600s --wait