Upgrading a Deployed Stack
Moving a cluster from one AICR release to a newer one. The short version: regenerate, check, then apply.
The middle step is the one this page is about. The first step takes one more flag this page also covers, --inherit-from, for when a component’s namespace, chart, source, path, manifest files or pre-manifest files moved between the two AICR releases. See When a component moves namespace.
Why a check step exists
A new AICR release moves chart version pins. Most of those moves are ordinary and a deployer absorbs them. Some are not: a CRD is renamed, a default flips, an API group changes, and the upgrade damages a running cluster in a way no deployer reports as a failure.
Nothing in aicr recipe or aicr bundle can tell you which kind you are looking at. bundle has a recipe and no source version, so it cannot compute a transition at all. A pin says what a new deployment gets; it says nothing about moving an existing one. The most bundle can do is carry the guidance for boundaries whose to range contains its own pins, conditioned on where you might be starting from; see Upgrade guidance in the bundle. That guidance is not a verdict for your move.
aicr upgrade-check compares two artifacts and answers that question per component, from transition records written by whoever bumped the pin.
Running it
Keep the bundle you deployed, or at least the recipe you deployed from. --from always needs one of them: it is the only record of where your cluster came from, and without it there is nothing to compare. Prefer the bundle. Both compare versions and namespaces, but only a bundle records the values it installed with, which is what the object-name comparison reads; given a recipe, that comparison is skipped and the report says so.
If you have already generated the recipe you intend to move to, name both:
If you have not, omit --to and ask the other useful question:
That re-resolves your artifact’s own criteria against the running binary’s pins, answering “am I behind, and does catching up hurt?” rather than “is this specific move safe?”. It is usually the question you actually have.
--deployer is required whenever a component carries steps, because steps differ per deployer and the tool will not guess. Pass the same value you pass to aicr bundle.
What is actually running here?
Both forms above compare artifacts, so they answer for the recipe you think you deployed. If somebody applied a chart by hand, or a prior upgrade only half landed, the artifact in git no longer says where the cluster is. Ask the cluster instead:
That reads the record your deployer leaves behind, and only that one: Helm’s release records for helm, helmfile and flux, and Argo CD’s Application objects for argocd and argocd-helm, which write no per-component Helm release at all. The --to side is still an artifact; there is nothing in a cluster to upgrade to.
Two flags stop being optional here. --to, because a cluster carries no criteria to re-resolve, so the “am I behind?” form has nothing to work from. And --deployer, whether or not any component turns out to carry steps: a release name encodes the deployer that wrote it (flux composes <targetNamespace>-<name>, Argo CD prepends a prefix you set), so without one nothing installed maps to a component and the read could only report an empty cluster.
Permissions it needs. The read lists across all namespaces, so the identity in your kubeconfig needs cluster-scoped list on what your deployer writes. For helm, helmfile and flux, that is secrets and configmaps: Helm stores each release record in one or the other, and the read reads both. It selects only Helm release records (label owner=helm), but list secrets at cluster scope lets that identity read every Secret in the cluster, so grant it knowingly. For argocd and argocd-helm, that is applications.argoproj.io, plus secrets when an Application names its destination cluster instead of giving its server address, because the name is resolved through Argo CD’s cluster Secrets. A denied list on any of these fails the run and names the permission to grant. The at-risk scan, which --from cluster turns on, also lists each resource kind it scans cluster-wide; where that is denied, its section reports the scan as failed and the comparison still runs.
What the read is, and what it is not. It answers which version is installed, authoritatively, including where that has drifted from git. It answers nothing else. Only an established version counts: a Helm release whose newest revision reached deployed, and an Argo CD Application revision that a sync completed, never the pin it is configured to reach. A pending or failed Helm upgrade reads as unversioned, because the old version may still be running; an Argo CD Application still syncing reads at its last completed sync, or as unversioned if it has none. A component the read places but cannot version, such as one whose release record cannot be read, also reads as unversioned rather than disappearing, so it fails the run instead of reading as newly installed. Both are records of what was applied, not observations. A resource somebody edited by hand leaves both untouched, and reading them will not tell you it happened. Treat the answer as “what was installed here”, not “what this cluster looks like”.
The report grows a READ FROM CLUSTER block above the rows, naming the kubeconfig it read and accounting for each reader. On an Argo CD management cluster, Applications deploying to other clusters are left out and counted as remote; only what deploys into the cluster you read counts as installed. Its context line reads - because the current context is not yet reported, so check it yourself with kubectl config current-context. Read it before you read the verdicts. A run that recognized nothing is reported rather than failed, and says so explicitly, because every row then reads “added” and three causes look identical from the rows alone: a genuinely bare cluster, the wrong cluster, or components installed by a deployer other than the one you named.
A component installed outside its registry default namespace is still found: the read also tries the namespace your --to recipe puts it in (an overlay or mixin can move it, and --inherit-from keeps it where it was), and under flux the namespace the release itself records. A strict run also fails when the block reports AICR-stamped releases that match no component, since that means the read lost track of something AICR installed.
One axis the cluster read does not cover is the namespace. When a component moves namespace is an artifact-to-artifact comparison only: the read recovers a version and no namespace, because the namespace is an input to the attribution rather than something the records hand back, and three of the five deployers could not report one at all. A relocation row therefore never appears against --from cluster. Keep the artifact comparison for that question.
Objects the upgrade could destroy
A different question about the same cluster: not what is installed, but what is sitting there that an upgrade might take with it. --scan-cluster answers it.
The scan is its own axis rather than a mode. It needs a cluster wherever the --from side came from, so scanning a live cluster while comparing two artifacts, as above, is a normal thing to do. --from cluster turns it on for you, and --scan-cluster=false turns it back off if you do not want it there.
For the resource kinds the crossed transition records name, it lists each kind cluster-wide and reports every object carrying neither Helm’s ownership markers (app.kubernetes.io/managed-by=Helm together with a meta.helm.sh/release-name annotation) nor Argo CD’s argocd.argoproj.io/tracking-id. What it is trying to stop is your own custom resources going away with a CRD that a component removes: AICR did not create them and cannot put them back.
Because the test is positive, an object carrying no marker AICR recognizes is reported. Something a fourth tool owns will show up here. That is the intended direction of error: a spurious line of output costs you a moment, a missing one costs the object.
Two cases follow from that test. Any Helm release or Argo CD Application counts as an owner, not only AICR’s, so your own custom resources managed by your own chart or Application are treated as owned and not reported: your deployer can usually re-apply them, but only once the kind exists again. And Argo CD 2.x tracks resources by the app.kubernetes.io/instance label by default rather than the annotation, so on such a cluster every Argo-managed object of a scanned kind is reported, including ones your own deployer manages. The scan stays conservative there on purpose, since treating that generic label as ownership would hide real findings.
Findings never fail the run. Blocking an upgrade over resources AICR does not own is a claim it has not earned, so the section is advisory and the exit code ignores it entirely. Acting on it is yours: confirm each object is expected to survive this upgrade, or back it up, before you apply.
An empty section is not an all-clear. The AT RISK section prints on every run, including runs that touched no cluster, and states which case it is: no cluster access was requested, the scan was explicitly turned off, no crossed record names a resource kind, or the scan read objects and found every one of them owned. Check which one you are looking at before reading it as a clean bill of health.
Acting on the report
Full flag and verdict reference lives in the CLI reference. What to do:
safe: nothing. Apply the new bundle.
manual: read the steps under the table. They are scoped to the deployer you named, and they are ordered. Check the PRECONDITION line first: it states a cluster state that must hold before you start, and it is prose for you to verify, not something the tool evaluates. Run the steps, then apply the bundle.
blocked: do not make this jump in one step. The report always names the boundary it stops at, and the detail block under the row says which of four things happened. The first of them comes with instructions; the other three deliberately do not.
Blocked, and here is how to do it safely. One record describes exactly your move, and its author marked it blocked, meaning it must not be done in a single step. That record’s steps are under the row, scoped to the deployer you named, saying what to do instead: typically land on an intermediate version, do some work there, and continue. Read them the way you read a manual row, and re-run the check when you get there.
Blocked because you would skip a boundary. Either the jump crosses two or more recorded boundaries, so no single record describes it, or a blocked record sits between you and your target but was written for a different starting point. Either way the report names a version to land on: the first boundary you would have flown over. Upgrade to it, apply, then re-run.
Blocked because nothing describes your starting version. A boundary is in the way, but no record covers an upgrade from where you are. Usually that is because your version is below the lowest one anybody has written a record for, and the report names that earliest recorded starting point: upgrade to it first, then re-run. Where your version instead falls in a gap between recorded from domains there is no such landing point to name, so the report says to author a record for your starting point instead.
Blocked because your target is past what the record assessed. One record does describe a move from where you are, but it stops assessing before your target: its to names a ceiling, and you are asking to land above it. Whoever wrote it cannot have read the migration notes for releases that did not exist yet, so it says nothing about the ground past that ceiling. Upgrade no further than the ceiling the report names, then re-run; the other remedy is for someone to widen that record.
The last three show no steps, and that is the point. The record carrying them describes a different move than the one you asked about, and running one migration’s steps without the work that precedes them is how data gets destroyed.
Those last two cases are stricter than a tool that simply had no record for you, and the strictness is deliberate. upgrade-check is opt-in: nothing in aicr recipe or aicr bundle invokes it, so a check somebody chose to run should not also be quietly permissive. When a boundary is in the way and the records reach neither back to your starting point nor forward to your target, saying so is more useful than a verdict nobody authored for your situation.
unknown: AICR has nothing to tell you about this transition. That is a gap in its data, not a verdict of safe, and it always fails the run. Read the component’s own upstream release notes and decide for yourself. Four different things produce it, and they differ in what would close the gap:
- No record at all (
no-record). Nobody has assessed this component. Consider authoring the first record so the next operator does not repeat the work. - A record exists but is silent here (
no-boundary-crossed). Somebody has assessed this component, but wrote no boundary in the range you are moving through. Decide whether one belongs there, and widen the record if it does. - The component’s identity moved (
identity-changed). Its namespace, chart, source, kustomize path, deployment type, manifest files, pre-manifest files or object names changed. Records assess version boundaries, so none of them assesses a relocation, a replacement or a rename. See When a component moves namespace and When a component’s objects are renamed. - You are rolling back (
downgrade). See below: this one can never become known.
The difference from blocked is worth holding onto. blocked means AICR has something to tell you and a version to stop at, so read it and act on it. unknown means AICR has nothing, so the investigation is yours. Neither is permission to proceed.
unversioned: one side’s version is not comparable, so no boundary can be classified at all. Pin something comparable and re-run.
A component that appears on only one side is reported too. An added component is simply installed. A removed component stays installed: AICR dropping a component from a recipe says what AICR now ships, not that your running workload should be torn down. Removing it is your call.
When a component moves namespace
The check compares two axes, not one. Beside the version it compares each component’s identity: the namespace each artifact resolves for it, its chart, source, path, deployment type and manifest sets, and the values that name the objects its chart owns. This section covers the namespace; object names have their own below.
The reason is at the top of this page. You regenerate the recipe from scratch on every AICR upgrade, and a component’s namespace comes from recipes/registry.yaml in the binary doing the regenerating. If a default namespace moved between the two AICR releases, the new recipe names the new namespace. Helm cannot move a release between namespaces, so applying the resulting bundle does not relocate anything. What it does instead depends on what the chart owns, and neither outcome is one you want: it either installs a second copy of the component beside the one already running with nothing reconciling the two, or it fails outright partway through the bundle. Both are covered below. A version-only comparison reports the move as no change at all, which is why the check used to pass it in silence.
Two shapes of row come out of this:
- The component held its version and moved anyway. You get a row where you previously got none: change kind
identity, verdictunknown, reasonidentity-changed. It fails a strict run. - The component moved on both axes in one hop. The relocation is carried on the version row, and a
safeverdict there is withdrawn tounknown. The record assessed a version boundary; nobody asked its author about a relocation, and reading a claim about one axis as evidence about the other is exactly the false confidence a wrongsafebuys. Any other verdict is left as it was, because it already stops the run and already sends you to the row.
unknown rather than blocked is deliberate. A blocked verdict is an author’s judgement recorded against a version boundary, and the record vocabulary has no way to express one about where a release lives, so no author can record it here. The gap is in what the vocabulary covers, not in somebody’s diligence.
What applying the moved bundle actually does
Whether you get a second copy or a hard failure turns on whether the chart owns cluster-scoped resources, because those carry Helm ownership annotations naming the release’s namespace.
A chart whose resources are entirely namespaced has nothing to collide over. The install into the new namespace succeeds and you get the second copy described above.
A chart that ships cluster-scoped resources, CRDs most commonly, cannot have them adopted by a release in a different namespace. Helm refuses before creating anything:
This is the better of the two outcomes: the running component is untouched, there is no second operator and no competing reconcilers. But three things are worth knowing before you meet it at three in the morning:
- The bundle applies partially. Components ordered before the relocated one are already upgraded when the failure hits, so the cluster is left in a mixed state. Re-running after you fix the cause is safe, since every component’s install is
helm upgrade --install. - The error names nothing you can act on. It talks about ownership annotations, not about namespaces moving, not about AICR, and not about this page. Seeing
must equal "<new-namespace>": current value is "<old-namespace>"is the tell that you are in this situation. - It depends on the deployer reaching Helm.
helm,helmfile, and FluxHelmReleaseall perform this ownership validation. A deployer that renders manifests and applies them directly does not, so it is not protected by this check.
Either way the fix is the same, and upgrade-check flags the move before you apply anything: pin the namespace with --inherit-from, below, or move the release deliberately and re-run the check.
Structured output carries the move alongside the verdict, on both shapes of row:
field is one of namespace, type, chart, source, path, manifestFiles or preManifestFiles. For the two file sets, from and to hold the whole sorted sets joined by commas.
Acting on it is its own piece of work, not an upgrade step: move the release deliberately, then re-run the check. Or take the relocation out of the hop entirely, below.
When a component’s objects are renamed
Half the components in the registry pin the names of the objects their chart creates, with fullnameOverride or nameOverride in their values file. Those two values are what Helm’s chart.fullname and chart.name templates read, so they rename objects rather than reconfigure them: they change what a component’s Deployment, ServiceAccount, Service and webhook configurations are called, where every other value changes how it behaves.
Those two keys are the whole of what is compared, at any depth. A chart can also name an object it owns through a value of its own — serviceAccount.name is the common one — and that produces no row. So a clean report means neither override key moved, not that nothing was renamed. Widening the set is not obviously right either: the paths differ per chart, and a comparison that guessed at them would report ordinary configuration changes as renames.
Edit or remove one and every object the chart owns is renamed at once. Helm applies that as delete-and-recreate, so expect a service gap, and an orphan for anything referenced by name or not owned by the release. It is worse where the moved key changes the selector labels of an object whose name does not change, for example nameOverride moving while fullnameOverride is pinned: spec.selector is immutable, so the upgrade fails outright rather than replacing anything. A key that renames the object as well, such as nameOverride with no fullnameOverride in the standard scaffold, is plain delete-and-recreate with no error. Which you get is a property of the chart, so check the chart before you assume.
The worked example is nodewright-operator. Its values pin fullnameOverride: skyhook-operator, and that single line is the only thing holding its objects at stable names across the upstream skyhook → nodewright chart rename: upstream’s chart.fullname falls back to .Chart.Name, which the rename changes. Dropping the line as a tidy-up renames the lot.
These moves report on the same rows and with the same verdicts as a namespace move, and field carries the dotted value path rather than namespace:
An empty from or to means the name appeared or disappeared, which is a rename either way: a chart with no fullnameOverride names its objects after itself. This is the opposite of how the namespace axis reads an empty value, and deliberately so — an absent namespace is a fact the artifact did not record, while an absent object name is a fact it did.
This axis needs a bundle on the --from side. A resolved recipe records valuesFile as a path, resolved against whichever binary reads it, so comparing two recipe files would read today’s values twice and see nothing move. Only a bundle writes the merged result down, in the per-release values.yaml four deployers emit and the HelmRelease spec.values flux inlines. When the source cannot supply them the report says so once, above the table, rather than reporting that nothing moved:
--format json carries the same fact as objectNamesCompared and objectNamesSkipped. A bundle built before AICR stamped bundle-info.yaml reports its own reason: there is no record locating the values it installed.
Pinning the namespaces you already deployed into
Regenerating from scratch is what introduces the move, so the way to avoid it is to tell the new recipe where the old one put things:
--inherit-from takes the bundle directory you deployed, which is read through the recipe.yaml every deployer writes at the bundle root, or failing that the recipe you deployed from. The resolved recipe keeps that artifact’s namespace, chart name, source, kustomize path, manifest file set and pre-manifest file set. Everything else, version pins and values included, comes from the new binary as usual. A file set the prior artifact lists is restored whole, so a file the registry dropped is kept and a file the registry added is left out. A set the prior artifact leaves empty is not restored, so a file the new release adds is kept. A component whose deployment type changed between Helm and Kustomize keeps only its namespace, since no chart, source, path or manifest set carries across that flip, and upgrade-check reports the type move. The relocation rows then disappear from the check, leaving the version axis to be assessed on its own.
Chart and source are pinned beside the new version. The chart name and source come from the prior artifact while the version pin comes from the new binary, and nothing checks that the old source serves the new version. If a release moves a chart to a new repository and pins a version published only there, install fails against the inherited source. In that case resolve without --inherit-from and let upgrade-check report the move.
Object names are pinned too, from a bundle. Given a bundle directory, the flag also carries forward the fullnameOverride and nameOverride values that bundle installed with, so a values-file edit in the new AICR release does not rename a running release’s objects. It writes an override only where the inherited name differs from what the new binary resolves, so a steady-state inherit adds nothing to the recipe and an override appears exactly where a rename was prevented:
Only those two keys are carried, never arbitrary values. Inheriting general configuration would freeze a component against registry updates you do want; these are different because they name objects rather than configure them. A name the new values add where the prior bundle pinned none is written as an explicit null, because letting it apply would rename the running objects just as surely.
A recipe file cannot supply this half: it records valuesFile as a path, so its values are whatever the reading binary ships. Passing one still pins namespaces, and logs a warning that object names were left alone. Pass the bundle directory to get both.
Inheriting from a bundle older than v0.22.0 works only for helm. Writing recipe.yaml for every deployer landed in v0.22.0; before that only the helm deployer wrote one. So a bundle built by v0.21.1 or earlier with helmfile, argocd, argocd-helm or flux has no recipe.yaml at its root, and pointing --inherit-from at it is rejected:
This is the one upgrade where it bites, because the artifact you are inheriting from is by definition built by the older release. Pass the recipe file you generated instead, which every version writes. From a v0.22.0 bundle onward, either works. Note that falling back to the recipe costs you the object-name half above, so a component whose fullnameOverride moved in the same hop needs the rename performed deliberately.
The same is true of a helm bundle built before v0.22.0, which --inherit-from does accept. It carries no bundle-info.yaml (that record first shipped in v0.22.0), so it pins and compares namespaces only, and logs a warning that object names were left alone; perform any object-name change deliberately. This is the hop existing skyhook-era installs take.
A component the prior artifact does not name keeps the registry default, because as far as that artifact knows it is a first deploy. Two cases land there and are worth telling apart: a component the new AICR release adds, which genuinely is a first deploy, and a component you excluded at bundle time with --set <component>:enabled=false, which a bundle’s recipe.yaml records post-filter and therefore does not carry. Inheriting from a filtered bundle gives the excluded components registry defaults. Inherit from the recipe rather than the bundle if you want them pinned.
The flag fails closed rather than quietly resolving as a first deploy. A path that does not exist, a directory with no recipe.yaml in it, and a cm:// URI (not supported yet) are each rejected with INVALID_REQUEST. So is an artifact resolved for a different service, accelerator, intent or OS than the new recipe, because same-named components differ across them. A dimension either side leaves unset or any, and the platform and node count, are not compared.
aicr query and aicr mirror list carry the same flag, because all three share aicr recipe’s resolution flags. The REST API does not: --inherit-from names a path on the machine running the CLI, so it is CLI-only for now, as aicr recipe --snapshot and --data already are.
Gating a pipeline
upgrade-check exits non-zero by default when any component needs attention, so a pipeline can consume the result without parsing output:
Add --fail-on-error=false to report without gating. The report prints in full either way; the exit code is orthogonal to it.
Anything other than safe exits non-zero, unknown included. Adopting this as a hard gate today will fail on most comparisons, because records are still being authored (see Coverage starts near zero below); --fail-on-error=false plus the JSON payload is the usable shape until coverage grows.
Both a bad invocation and a failing check exit 2. To tell them apart, write JSON and branch on the payload:
Rolling back
Point the check the other way:
Records are directional. A record describing a forward upgrade never applies in reverse, so a downgrade reports unknown, never safe. That is deliberate: undoing a migration is rarely the same work as doing it, and a CRD deleted on the way up does not come back on the way down.
A downgrade always fails the run, whatever the versions are and whatever records exist. It is the one unknown that can never become known: the well-formedness rules reject every reverse record, so no amount of authoring closes it. It is reported as unknown rather than blocked because blocked names an intermediate version to stop at, and you cannot partially roll back.
So a rollback needs human review before you run it. Read the component’s own downgrade guidance, and if you accept the risk deliberately, --fail-on-error=false gives you the report without the gate.
What this does not cover
- An artifact comparison reads no cluster state. Nothing is inspected, deployed or modified. (A
cm://path is an artifact location like a file path, so reading or writing one does contact that cluster’s API for the ConfigMap itself.) If your cluster has drifted from the recipe you think you deployed, the check compares the artifacts you gave it, not reality. Ask the cluster when that is the question. - The cluster read is a read of declarations, not of live state. A Helm release answers only from a newest revision that reached
deployed, and an Argo CDApplicationonly from a revision its sync status or history shows was synced, never from the pin it is configured to reach. An upgrade that is still pending, or failed, therefore never reads as made: under Argo CD it reads at the last synced revision, or asunversionedif there is none, and under Helm it reads asunversioned. A hand-edited resource changes neither, so the read is authoritative about installed versions and silent about everything else. It also reports no namespace or object-name move, having neither to compare. - Argo CD bundles built with
--vendor-chartsread asunversioned. Vendoring turns every chart into a path-basedApplication, which carries no payload version anywhere in the cluster, so every component readsunversionedand a strict run always fails. Compare the vendored bundle as an artifact instead:--from <bundle>. - You still name the deployer. A bundle records the deployer that built it in
bundle-info.yaml.upgrade-checkreads that file only to locate release values and does not take the deployer from it yet, so--deployeris required whenever a component carries steps, even when reading a bundle, and unconditionally when reading the cluster. - Coverage starts near zero, so expect red. Only five components ship a record today, so most transitions report
unknownand the check exits non-zero on most comparisons. Absence of a record is absence of assessment, and the tool says so rather than rounding it up to approval. This is a coverage problem with an owner (#2535 makes a record mandatory for every pin bump), and it shrinks as records land. Use--fail-on-error=falsefor the report without the gate in the meantime. - A namespace move is seen only when both artifacts state one. An empty namespace is read as a fact the artifact did not carry, not as a move to or from the default, so a component that gains or loses an explicit namespace between the two artifacts produces no relocation row at all. Reading it the other way would report a move nobody performed for every component the moment one of the two artifacts stopped carrying the field. The same holds for chart, source and path. The manifest and pre-manifest file sets are the exception. Each is compared as a set, so a set that empties is a move. Object names are another, read the opposite way to the scalar fields, and that section says why. The release name is not compared, because it is derived from the component name.
- Object names need a bundle on the
--fromside, for the reason given in that section. With a recipe file or--from clusterthere, the object names are not compared and the report says so; it does not report that nothing moved. - The object-name comparison covers
fullnameOverrideandnameOverrideonly. A chart that names an object through some other value of its own, such asserviceAccount.name, produces no row when that value moves. - A pinned object name is not carried into what is derived from it. When
--inherit-fromholds a component at an old name, its health check and any other component’s hardcoded reference to that name still come from the new release. A namespace pin rebinds the health check; a name pin does not yet (#3024). --dynamicvalues are not all visible. A flux release that takes--dynamicvalues throughspec.valuesFromwithdraws the object-name axis instead of guessing, and pins no names from that bundle. Anargocd-helmbundle moves--dynamicpaths into its root chart values, which nothing in the bundle records, so a dynamic name key there reads as unset (#3025). Avoid putting a name key under--dynamiconargocd-helm.- Records are human assertions. A
safeverdict names what verified it, but it is somebody’s reading of the migration notes plus a test lane, not a proof.
See Also
aicr upgrade-checkreference- Upgrade Notes: per-component migration prose, including the ones with no record yet
- Authoring transition records: for whoever bumps the pin
- Generating Bundles: including the DRA driver eviction caveat, which is upgrade-relevant and predates this check