API reference
The operator.aicr.run/v1alpha1 API and the operator’s installation values.
Field names may change while the API is v1alpha1; ownership, mutability and
durability rules do not change without a design record.
Conventions
- Group
operator.aicr.run, notaicr.run. AICR usesaicr.run/<version>for its own artifactapiVersionvalues (AICR ADR-011) andaicr.run/*label keys; a separate group keeps the two version policies apart. v1alpha1carries no conversion guarantee. A breaking change beforev1beta1ships with a documented manual migration, not a conversion webhook.- Admission uses CRD validation rules and ValidatingAdmissionPolicy only; no webhooks and no mutating admission.
- Both kinds are cluster-scoped. Durable decisions live in immutable
specfields, the set-once approval, operator-set outcome labels and immutable plan ConfigMaps;statusholds observations that can be rebuilt from those and live objects. - Plans are referenced by
desiredStateDigesteverywhere, never by object name. - Engine objects, plan ConfigMaps and rollouts carry no
ownerReferences. The one exception is an AICR Job’s run-scoped objects (its output ConfigMap, input copies and credential Secret), which are owned by the Job in the same namespace so garbage collection removes them even when the operator is down. - Every list, map and string in both CRDs has
maxItems,maxPropertiesormaxLength, so validation rules stay within the CEL cost budget. Set-once and immutability rules sit onspec, which always exists, sooldSelfis always available. - Infrastructure (AICR image, deployment engine, registry, namespaces,
identities) is installation configuration in Helm values.
ClusterRuntimeholds only runtime intent and policy.
Installation values
The installation chart is charts/aicr-operator
(README); its values.schema.json
rejects unknown keys. The operator namespace is the release namespace.
The chart fails to render when aicr.version is outside >=1.0.0, <2.0.0.
The operator also checks the version each AICR Job reports and stops on a
mismatch. engine.type accepts only ArgoCD in this release; the chart schema
and the operator reject any other value. CRDs, namespaces, the AppProject and
the repo-creds Secret survive helm uninstall.
ClusterRuntime
Validation rules (CRD x-kubernetes-validations, stable since Kubernetes 1.29):
- Root:
self.metadata.name == 'cluster'(root rules may readmetadata.name). spec.placement.*.nodeSelector: at least one entry; system and accelerated selectors must differ.spec.selection:pinnedPlanDigestis non-empty if and only ifpolicy == 'Pinned'.spec.overrides: unique(component, path). Encoded value size is enforced at reconcile time.spec.selection.pinnedPlanDigest, when set, matches^sha256:[a-f0-9]{64}$;spec.adoption.runtimeUID, when set, is a UUID.
Bounds: intent, platform, profile, criteria values and overrides[].component
at most 63 characters; overrides at most 64 items with path at most 256
characters; nodeSelector at most 16 entries within Kubernetes label limits;
tolerations at most 16 items; configuration.gke.tcpxoInterfaces at most 16
items with unique interfaceName (at most 15 characters) and network at most
63 characters; conditions at most 16 items with messages at most 1024
characters.
Recipe configuration. spec.configuration mirrors AICR’s recipe
configuration: cluster-specific desired state AICR records in the recipe
without a criteria axis. Today it holds the GKE GPUDirect-TCPXO interface
mapping, which recipes that ship the TCPXO training runtime
(h100-gke-cos-training-kubeflow) require. Provisioning decides it, so the
administrator copies it from the cluster:
The CRD bounds only its shape. AICR validates the mapping when it resolves
the recipe (on a3-megagpu-8g: eth1 to eth8 exactly once, eight distinct
Device Network names containing gpu-nic), and a rejected mapping fails the
recipe Job with AICR’s message. A change resolves again; it does not take a
new snapshot.
Conditions follow kstatus so Argo CD or other tools can health-check the runtime itself:
While a rollout is in progress, Ready=False has reason RollingOut and a
message that names the rollout and where it is
(rollout <name>: step 9/18 gpu-operator (Upgrade),
rollout <name> awaiting approval,
rollout <name>: deployment validation run <id>,
rollout <name> stalled: <message>); before
the rollout exists it says the target plan is not rolled out yet. Available
observes a release whose Application already carries another plan at that
plan, since a rollout moved it, whether the rollout is in progress or failed,
and no longer counts a release whose Remove step started; a release a rollout
installs counts once the rollout succeeds. After a failed rollout Stalled
reports the failure, and Available stays true while every release is healthy
at the plan it carries. The release of the step in flight leaves Available alone while it
progresses, as a rolling update leaves a Deployment available; the step’s
action timeout reports one that never settles.
Each condition carries observedGeneration. The two controllers write disjoint
condition types with separate server-side apply field managers, which works only
because:
status.conditionsis a map list keyed by type (+listType=map,+listMapKey=type), so each entry has its own owner;- each controller applies only the condition types it owns, never the full list read back from the object. Applying the full list would transfer ownership of every entry, and a later apply without an entry would remove it.
The resolver owns only Resolved and reports its failures there, with AICR’s
exit category as the reason. The rollout controller derives the kstatus
Reconciling and Stalled conditions from its own state and from Resolved,
so no condition type has two writers.
Plan ConfigMap
Plans are operator-internal: only the resolver writes them, only the operator
reads them, and they have no status. They are immutable ConfigMaps in the
operator namespace rather than an API kind, so their format evolves inside the
operator through schemaVersion, without CRD versions or conversion.
They are ConfigMaps, not Secrets,
because plans and values never hold credentials.
The plan document, shown as YAML:
Bounds: at most 128 releases, with release IDs of at most 53 characters (the
Helm release-name limit), stored as plain JSON (no compression) so
kubectl get -o yaml stays readable; a plan that does not fit one ConfigMap
(1 MiB) is rejected. The recipe and values live in their own immutable
ConfigMaps, so the plan stays small. A plan keeps two recipes: the one AICR
resolved, which upgrade-check compares and kubectl aicr reproduce renders
from, and the one the bundle records after AICR applied the render flags,
which deployment validation runs against, so a component an override turns
off is not required. A plan written before it recorded validationConfigMapRef
validates against configMapRef. Neither reference is part of the plan’s
digest. Every chart is served from the runtime
registry and pinned by manifest digest; mirrored charts also record the verified
upstream archive digest.
Rules:
- The resolver creates the ConfigMap once.
AlreadyExistswith a matching plan-digest annotation is success; a mismatch is a conflict. - A plan belongs to the runtime UID it records. A matching plan that records
another runtime’s UID, left by an earlier runtime, is replaced rather than
reused, and a plan recording another UID is never rolled out (
StalledwithPlanOfAnotherRuntime). - A resolved plan replaces a
Rebuiltplan of its digest; a rebuilt plan never replaces a resolved one. A plan without a recipe is not rolled out while deployment validation isRequired(StalledwithPlanWithoutRecipe). immutable: truefreezesdata; labels and annotations stay writable. A ValidatingAdmissionPolicy limits creating and updating operator-labeled ConfigMaps in the operator namespace to the operator’s ServiceAccount.- The operator reads a plan only when the
schema-versionlabel is a version it supports and equalsschemaVersionin the payload, and the payload’sdesiredStateDigestequals the plan-digest annotation. Anything else setsStalledwith reasonPlanUnreadable; the operator never guesses. - A plan ConfigMap is a cache of state the engine objects also carry. If it is lost, the plan is re-resolved (candidate) or rebuilt from the live engine objects (active, adopted), as re-adoption describes.
- List plans with
kubectl get configmaps -n aicr-system -l operator.aicr.run/role=plan.
RuntimeRollout
Naming. <transition> is the first 16 hex characters of SHA-256 over the
runtime UID, fromPlanDigest and toPlanDigest. Before creating a rollout,
the controller lists the runtime’s rollouts through the uncached reader,
keeping only those whose spec.runtimeUID is the current runtime’s. A
non-terminal rollout is the one in progress, and no new one is created.
Otherwise <n> is one more than the highest suffix among the listed rollouts
carrying that transition label. A retried create after a lost response finds
the first through the uncached list. Two concurrent creators collide on the
name even when they read live state at different times, because neither the
name nor the suffix depends on it: a creator whose list predates another’s
create derives that rollout’s name. AlreadyExists triggers a fresh list,
never a second object. The live inventory is recorded in
startingInventoryDigest, not in the name.
Outcome. When a rollout ends, the operator sets the outcome label
(Succeeded, Failed, Expired, Revoked, Superseded or Orphaned) and, for a
validated success, the evidence annotation. Both are set once, only by the
operator, and survive the loss of
status, so the active plan and terminal states can be rebuilt after a restore.
CRD validation rules:
- Every
specfield other thanapprovalis immutable (self == oldSelf). approval.approvedBy,expiresAtandreasonare set together and once.approval.revokedBy,revokedAtandrevokeReasonare set together and once, only afterapprovedBy. An approval can be neither cleared nor extended.approval.acknowledgementsare append-only, one per release, and need an existing approval.
Bounds: steps at most 256 items (128 target releases plus removals from the
active plan); status.releases at most 256 items; release IDs at most 53
characters; approvedBy and revokedBy at most 1024 characters, which covers
IAM-derived usernames; reason, revokeReason and message at most 1024
characters; conditions at most 16 items.
Approval
Approval is the set-once spec.approval on the rollout it authorizes. The rest
of the spec already fixes the target plan digest, starting inventory, steps and
deadline, so the approval is bound to all of them without copying them.
CRDs support only the
status and scale subresources, so there is no /approve endpoint. RBAC grants
update on the virtual subresource runtimerollouts/approve, which RBAC matches
as a string, and a ValidatingAdmissionPolicy checks it on every UPDATE that
changes spec.approval:
authorizer.group('operator.aicr.run').resource('runtimerollouts').subresource('approve').name(object.metadata.name).check('update').allowed(). Approvers are granted this separately fromClusterRuntimeediting, so editors of the runtime cannot approve their own changes.- A newly set
approvedByorrevokedByequalsrequest.userInfo.username, so the object carries a verified identity without an audit-log lookup. - Revocation is rejected when the rollout carries an
outcomelabel or its status phase isValidating. Status can be empty after a restore, so the controller also ignores a revocation that arrives after the last step finished: a revoked or expired approval ends the rollout only before a step whose mutation this rollout has not made, judged from the engine objects (the spec written, or a sync since the rollout was created). - The request changes nothing but
spec.approval; the operator’s own identity can never write it, and a rollout cannot be created with it. - Only the operator’s identity creates rollouts, so nobody else can create one
that skips approval with
approvalRequired: falseor anoutcomelabel already set.
To restore RuntimeRollouts from a backup under another identity, as Velero does:
- Stop the operator, so no rollout advances while admission is relaxed:
kubectl -n aicr-system scale deployment/aicr-operator --replicas=0. - Delete the
aicr-operator-approvalValidatingAdmissionPolicyBinding. - Restore the rollouts.
- Run
helm upgradeon the operator chart with the same values. It recreates the binding and restores the operator’s replica count.
A restored rollout comes back without its status. The operator rebuilds its
phase from the outcome label and records each release in its steps as
Unknown, because which ones it wrote is lost. Until a later rollout succeeds,
a missing Application of such a release is not repaired from the active plan.
While the target differs from the active plan, the runtime stalls with
MovedReleaseMissing; with the target at the active plan, the operator creates
a rollout to it that reinstalls the release, with a fresh approval.
Approvers also need get, list, watch and patch on runtimerollouts,
and get on ConfigMaps in the operator namespace to read assessment reports.
The chart’s aicr-operator-approver roles grant exactly that plus reading the
runtime, bound to approvers.users and approvers.groups; the policy
(aicr-operator-approval) is kept on uninstall. They
write their own username: a ValidatingAdmissionPolicy cannot set fields, and
MutatingAdmissionPolicy is beta and disabled by default at the Kubernetes 1.34
floor (GA in 1.36). The kubectl aicr plugin fills it from kubectl auth whoami:
Rollout names are computed at runtime, so approvals are made against the live object, not committed through GitOps.
Engine object metadata
The operator uses one Argo CD Application per plan release in the engine
namespace.
Every engine object, and every
operator-owned ConfigMap or Job, carries:
Engine objects also carry the plan graph, so a plan can be rebuilt from live state:
Every other input to desiredStateDigest is in the engine object’s own spec:
namespace, chart repository and manifest digest, values and engine policy
fields.
Labels select objects for watches. Annotations plus the recorded UID authorize
mutation; a label match alone never does. Argo CD Applications never carry the
resources-finalizer.argocd.argoproj.io finalizer except during an approved
Remove step.
Request annotations on ClusterRuntime
Retention
The operator deletes its own objects by label; only AICR Jobs’ run-scoped objects rely on garbage collection:
- Plans: keep active, candidate and pinned plan ConfigMaps plus the five most recent superseded ones, together with their recipe and values ConfigMaps.
- Rollouts: keep, per runtime UID, the newest rollout labeled
outcome: Succeeded, any rollout without an outcome, and the ten most recent terminal ones. - AICR Jobs: delete a Job with
propagationPolicy: Foregroundafter its outputs are recorded; thebatch/v1default orphans the Job’s pods. Every Job also setsttlSecondsAfterFinished(24 hours, raised to seven days on a stalled stage’s failed Jobs) as a backstop while the operator is down or uninstalled. A Job’s run-scoped objects are owned by the Job and go with it, so a result the operator keeps (snapshot, recipe, assessment report) is first copied into a ConfigMap without an owner. Keep those copies while a retained plan or rollout references them, plus the newest successful snapshot.