Operator lifecycle
Installing, upgrading, downgrading and uninstalling the operator itself. The GPU stack it manages is never removed by any of these: it is owned by Argo CD Applications that survive the operator.
CI checks this page against the published artifacts, on the newest supported Kubernetes, using the in-cluster registry. Each operator change, and each release before it publishes, installs the newest published release, then:
- upgrades it to the new build mid-rollout;
- replaces the in-cluster registry pod;
- upgrades it again with validation enabled and requires deployment validation for a rollout, which fails on kind, without a GPU, and promotes nothing;
- downgrades it with
helm rollback; - uninstalls it in the order below, checking what is kept.
See qualify-published in
operator-e2e.yaml.
An operator release defaults to its own release’s AICR (earlier charts, such
as v1.2.0’s, to an earlier qualified AICR release), so when the two releases’
defaults differ, that upgrade also moves AICR within v1, unless your values
set the AICR keys (operations).
Install
The overview installs the published chart, and
installing on EKS covers ECR. Each chart version
defaults to its operator image and to the AICR and validator images the
operator was qualified against, pinned by digest. The operator namespace is the Helm release namespace;
create it first with the restricted Pod Security label, because Helm cannot
template its own release namespace.
Upgrade
An operator upgrade is a chart upgrade, and it moves the image, CRDs, RBAC and admission policies together:
- The CRDs are chart templates, not
crds/files, sohelm upgradeupdates them. Their schema reaches existing objects at once. --reuse-valuesis not supported. Helm then renders the new chart with the previous chart’s defaults, so every image your values do not set, such as the operator’s, stays at the previous release’s. The chart still renders an operator that starts. A key the previous chart did not have is left unset, and the operator uses its own default. To carry the previous release’s values forward, use--reset-then-reuse-values, which applies them over the new chart’s defaults.- With
aicr.versionv1.0.x, add--force-conflicts: AICR v1.0.x relabels the discovery and validation namespaces, and Helm 4 then reports a server-side apply conflict on theirapp.kubernetes.io/managed-bylabel (troubleshooting). - A rollout in progress resumes from its durable state under the new operator with the same approval, and its in-flight step is not written twice.
- The upgrade itself creates no rollout unless the new operator renders different output. An operator version is a resolution input, so the resolver runs again; an identical render is the same plan.
- The new operator reads plans and rollouts written by the previous version. Existing plan ConfigMaps are immutable and never rewritten; new plans use the new schema.
v1alpha1carries no conversion guarantee. A breaking API change ships with a manual migration in its release notes, never a conversion webhook.
Upgrading AICR is a values change inside the same chart (operations).
Downgrade
Downgrading is supported only while no plan or rollout carries a schema
version newer than the older operator knows. Otherwise the older operator
reports Stalled with PlanUnreadable and writes nothing to Argo CD; for
rollouts, the message names them. Recovery is to upgrade again; the newer objects are never
rewritten down.
helm rollback is a downgrade: it also returns the CRDs to the older
release’s schema.
Uninstall
The supported order:
- Delete the
ClusterRuntimeand wait until it is gone. WithdeletionPolicy: Orphan(default), the Applications are annotated withoperator.aicr.run/orphaned-fromand keep reconciling. WithDelete, an approved removal rollout deletes every release first (stuck finalizers). - Uninstall the chart:
helm uninstall aicr-operator --namespace aicr-system. To reinstall later after AICR v1.0.x ran, first give the kept discovery and validation namespaces theirmanaged-by=Helmlabel back (troubleshooting). - Optionally, remove what Helm keeps, once nothing will adopt the stack
again: the operator, Job, discovery and validation namespaces (the
operator namespace holds the in-cluster registry and its volume), the
Argo CD
repo-credsSecret, the policiesaicr-operator-stateandaicr-operator-approvalwith their bindings, and the CRDs. Delete the CRDs last; deleting a CRD deletes every object of its kind.
What helm uninstall keeps
After uninstall, Argo CD keeps reconciling the last synced state. The
in-cluster registry keeps serving it until you delete the operator namespace.
With the aws provider, nothing refreshes the token in repo-creds, so within
12 hours Argo CD can no longer fetch charts: comparisons fail with
ComparisonError, and running workloads are unaffected. To keep the stack
unmanaged for longer, replace the Secret’s credentials with your own.
A reinstall with the same release name adopts the kept resources when they still match the chart.
Uninstalling with the runtime still present
If you uninstall first, the ClusterRuntime and its finalizer stay, and
nothing changes in the cluster. Reinstalling the operator resumes managing it
without a new rollout; it re-verifies every plan ConfigMap it reads, so nothing
written while it was absent is trusted unchecked. A runtime delete issued while
no operator runs waits for the operator to return. Removing the finalizer by
hand instead cascades to nothing, and a later runtime
adopts the Applications
by the old UID.
Data retention
The operator-namespace ConfigMaps are caches. Losing them, even by deleting
the namespace, loses provenance (the recipe and the AICR version a plan came
from) but not the stack: the active plan can be rebuilt from the Applications’
annotations, and adoption needs nothing else. Evidence stays in the registry
and is referenced from the succeeded rollout’s operator.aicr.run/evidence
annotation.