AICR Operator
A Kubernetes operator that runs AICR in your cluster and rolls the GPU stack it renders out through Argo CD: in dependency order, with approval for risky changes and validation evidence for each rollout. The operator is opt-in; the CLI, API server and bundles work as they always have without it (ADR-027).
Status: preview. The operator is in preview: its API
(operator.aicr.run/v1alpha1) may change without a conversion path, and no
hardware profile is qualified yet. The configuration it deploys is not a preview. The
operator renders nothing itself: every bundle it rolls out is the one the
pinned AICR release renders, with zero diff, from the same validated and
optimized recipes as the AICR CLI. The operator supports AICR >=1.0.0, <2.0.0,
Kubernetes 1.34 or later, Argo CD as the deployment engine, and one homogeneous
accelerator and OS profile per cluster. A GB200 profile, a Flux adapter and
greenfield bootstrap follow later (support policy).
Why an operator
AICR decides what a GPU cluster should run: it resolves a validated recipe for your hardware and renders it into a bundle. Without the operator you run AICR from a workstation or CI and hand the bundle to Argo CD or Flux through Git. That gets you a first install. Keeping the stack correct as the cluster and AICR change also takes deciding when each change goes in, whether it is safe, and whether it worked. Neither AICR nor GitOps decides that, so each team scripts it.
Same configuration as the AICR CLI. The operator adds coordination, not
configuration. It runs aicr snapshot, aicr recipe and aicr bundle from
the pinned AICR image with inputs from your ClusterRuntime, so each bundle it
deploys has zero diff with the one aicr bundle --deployer helm renders from
the same recipe and flags, and Argo CD installs each release with the values
that bundle’s deploy.sh would pass to Helm. CI checks this on every change,
and you can check it on your cluster.
Validated scenarios
Each scenario below ran on two H100 nodes (a3-megagpu-8g, GPUDirect-TCPXO) on
GKE with a training job running throughout.
Responsibilities
The operator does not replace GPU Operator, allocate tenant GPUs, infer training performance from healthy pods, roll back automatically, or run its own node drains.
Installation
Prerequisites
- Kubernetes 1.34 or later, with Argo CD installed at the pinned release.
- Storage for the runtime registry. By default the chart runs the registry in the operator namespace, on a 10 GiB volume from the cluster’s default StorageClass and with a login it generates. That StorageClass and its CSI driver must exist before install. EKS has neither until its recipe installs them through the operator, so on EKS use an external registry (choosing a registry).
- The namespaces your recipe deploys into, listed in the chart values; the chart creates them with their Pod Security labels and keeps them on uninstall.
The chart README details each prerequisite, what the chart installs, and its admission policies; installing on EKS walks through an ECR registry, the operator’s AWS identity and the component namespaces.
Install the chart
Each AICR release publishes the operator image, the in-cluster registry image
and the chart, tagged with the release version. The chart defaults to that
release’s operator and registry images and to that release’s own AICR and
validator images, all pinned by digest. Before it publishes, the release runs
the operator with that AICR through install, rollout and deployment
validation, runs the validator images, and upgrades the previous release’s
operator to this one. An earlier chart, such as v1.2.0’s,
can default to an earlier AICR release. helm show values shows the default, and
operations moves it.
Create values.yaml with your cluster’s inputs:
Deployment validation runs AICR’s validators with cluster-admin. If you
leave validation.enabled off, set spec.validation.deployment: Disabled on
the ClusterRuntime below; otherwise the runtime stalls with
ValidationUnavailable before any step runs.
Then verify the chart and install it:
Required: namespaces.components, and with an external registry,
registry.repository and its credentials. Set approvers.users or
approvers.groups, or bind the aicr-operator-approver roles yourself: with no
approver, a rollout that needs approval waits in AwaitingApproval and expires
at its deadline. Everything else has a working default; validation.enabled is
recommended. Every value,
with its default and when to set it, is in the
chart’s values reference.
Choosing a registry
The operator keeps the charts it mirrors, the bundles AICR renders and validation evidence in an OCI registry, and Argo CD pulls every chart from it. The registry is storage, not something the operator trusts: every plan pins each chart by digest, so a lost or altered artifact fails the pull instead of deploying other content.
The in-cluster registry runs in the operator namespace under the restricted
Pod Security profile and serves plain HTTP inside the cluster. Only the
operator, its AICR Jobs in the Job and validation namespaces and Argo CD’s
repo server reach it, where the CNI enforces NetworkPolicy; kubectl port-forward
bypasses the policy, so for anyone allowed to port-forward in the operator
namespace the login is the only control. It keeps serving through uninstall,
and it deletes nothing, so size the volume for every plan the cluster will
resolve. Security covers the trust model.
Choose before you install: plans name the registry they were mirrored to, so
changing registry.mode on a running installation is not supported. An
installation from a release before the in-cluster default keeps its registry
with registry.mode: external; the chart refuses to render until it is set.
Install the kubectl plugin
Approvers use the kubectl aicr plugin. Each AICR
GitHub release carries its binaries
for Linux and macOS (kubectl-aicr-<os>-<arch>):
Usage
Describe the cluster
The operator acts on one ClusterRuntime named cluster:
Approve rollouts
The operator discovers the node groups, resolves the matching AICR recipe and plans the rollout. Risky transitions wait for an approver:
A manual transition whose record has steps after the apply pauses once that
release is applied; do those steps, then run kubectl aicr acknowledge on the
rollout.
Day-2 operations covers reading status, approvals, pausing, rollback and AICR upgrades, and troubleshooting every stalled state. The API reference describes every field.
Verify the bundle yourself
kubectl aicr reproduce writes a plan’s recipe (by default the active plan’s)
to aicr-reproduce/recipe.json and prints the commands that render the same
bundle with the AICR CLI and compare it with the operator’s. With the
placeholders filled in from its output:
The plugin only reads the plan’s record; the AICR image renders and diff
compares, so a record that did not match what the operator rendered would show
up as a diff. Each Argo CD Application’s spec.source.helm.valuesObject is
that release’s values.yaml merged with the bundle’s cluster-values.yaml.
The plugin needs get on the ClusterRuntime and on ConfigMaps in the
operator namespace.
Upgrade and uninstall
An operator upgrade is a chart upgrade (helm upgrade with the new
--version). helm uninstall leaves CRDs, namespaces and the Argo CD project
in place, so it never removes the GPU stack. Delete the ClusterRuntime first;
operator lifecycle covers upgrade, downgrade, uninstall order
and what each keeps.