> This page is for AI Cluster Runtime, version v1.3.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.3.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt
> - v1.3.0: https://docs.nvidia.com/aicr/v1.3.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt.

# AICR Operator

A Kubernetes operator that runs AICR in your cluster and rolls the GPU stack it
renders out through Argo CD: in dependency order, with approval for risky
changes and validation evidence for each rollout. The operator is opt-in; the
CLI, API server and bundles work as they always have without it
([ADR-027](https://github.com/NVIDIA/aicr/blob/main/docs/design/027-aicr-operator.md)).

**Status: preview.** The operator is in preview: its API
(`operator.aicr.run/v1alpha1`) may change without a conversion path, and no
hardware profile is qualified yet. The configuration it deploys is not a preview. The
operator renders nothing itself: every bundle it rolls out is the one the
pinned AICR release renders, with zero diff, from the same validated and
optimized recipes as the AICR CLI. The operator supports AICR `>=1.0.0, <2.0.0`,
Kubernetes 1.34 or later, Argo CD as the deployment engine, and one homogeneous
accelerator and OS profile per cluster. A GB200 profile, a Flux adapter and
greenfield bootstrap follow later ([support policy](/aicr/v1.3.0/operator-preview/support-policy)).

## Why an operator

AICR decides what a GPU cluster should run: it resolves a validated recipe for
your hardware and renders it into a bundle. Without the operator you run AICR
from a workstation or CI and hand the bundle to Argo CD or Flux through Git.
That gets you a first install. Keeping the stack correct as the cluster and
AICR change also takes deciding when each change goes in, whether it is safe,
and whether it worked. Neither AICR nor GitOps decides that, so each team
scripts it.

| With the AICR CLI and Argo CD | With the operator |
| --- | --- |
| `aicr snapshot` captures one moment; new nodes or drivers go unseen until someone reruns it | Discovery reruns as the cluster changes and resolves the matching recipe |
| Sync waves advance on each Application's last health, which can still describe the previous revision | Releases roll out in dependency order; each advances only when ready at the revision just written, including AICR's readiness gates |
| `aicr upgrade-check` reports `manual` and `blocked` transitions; enforcing them is up to you | Risky transitions wait for an approver; `blocked` transitions never start |
| `aicr validate` runs on its own; its evidence names no rollout | Validation gates each rollout, and its evidence is bound to the deployed plan and its approval |
| Removing a component from Git deletes its Application and leaves the component running, unmanaged | Removing a component is an approved step that uninstalls it in reverse dependency order |

**Same configuration as the AICR CLI.** The operator adds coordination, not
configuration. It runs `aicr snapshot`, `aicr recipe` and `aicr bundle` from
the pinned AICR image with inputs from your `ClusterRuntime`, so each bundle it
deploys has zero diff with the one `aicr bundle --deployer helm` renders from
the same recipe and flags, and Argo CD installs each release with the values
that bundle's `deploy.sh` would pass to Helm. CI checks this on every change,
and you can [check it on your cluster](#verify-the-bundle-yourself).

## Validated scenarios

Each scenario below ran on two H100 nodes (`a3-megagpu-8g`, GPUDirect-TCPXO) on
GKE with a training job running throughout.

| Scenario | What you see |
| --- | --- |
| **Install on bare GPU nodes** | One approval installs 18 components in dependency order, each only after the one before is healthy at the revision just written. Deployment validation passes, and its evidence records the plan and the approver. 15 to 20 minutes |
| **Upgrade AICR by changing one value** | A new plan resolves, `aicr upgrade-check` rates each component change (`safe`, `manual`), and one approval rolls it out and validates it |
| **Roll back** | Pinning an earlier plan is assessed and approved like any other change; nothing rolls back on its own |
| **Stop a rollout mid-way** | Revoking the approval lets the step in flight finish and starts nothing else; a retry needs a fresh approval |
| **Upgrade or crash the operator mid-rollout** | The same rollout resumes where it stopped, and no write is repeated |
| **Lose the operator, keep the stack** | Deleting the operator, its namespace and every saved plan leaves the GPU stack and the training job running. A reinstall rebuilds the plan from what is running, confirms nothing was edited by hand, and takes the stack back with one approval in under a minute |
| **Freeze everything during an incident** | `suspendEngine` stops every sync and drift repair within seconds and deletes nothing; clearing it resumes syncs and repairs the drift |
| **Nobody approves their own change** | Approving without the approver role, approving your own runtime edit, approving in someone else's name, the operator approving itself, and tenants reaching admin resources are all refused |
| **Status you can act on** | While a rollout runs, `Ready` names the step, such as `step 9/18 gpu-operator (Upgrade)`. A node reset or a registry outage shows up in the conditions with its cause and clears on recovery |

## Responsibilities

| Concern | Owner |
| --- | --- |
| Cluster snapshot and recipe selection | AICR (`aicr snapshot`, `aicr recipe`) |
| Chart versions, values, manifests and readiness gates | AICR (`aicr bundle`) |
| Transition risk between the active and the new plan's recipes | AICR (`aicr upgrade-check`) |
| Deployment validation and its evidence | AICR (`aicr validate`) |
| Installing charts and repairing drift | Argo CD |
| Device lifecycles | Component operators, such as GPU Operator |
| When to resolve and roll out, in what order, approvals, readiness at the current revision, removal | The operator |
| Serving every chart from the runtime registry, pinned by digest | The operator |

The operator does not replace GPU Operator, allocate tenant GPUs, infer
training performance from healthy pods, roll back automatically, or run its own
node drains.

## Installation

### Prerequisites

- Kubernetes 1.34 or later, with Argo CD installed at the pinned release.
- Storage for the runtime registry. By default the chart runs the registry in
  the operator namespace, on a 10 GiB volume from the cluster's default
  StorageClass and with a login it generates. That StorageClass and its CSI
  driver must exist before install. EKS has neither until its recipe installs
  them through the operator, so on EKS use an external registry
  ([choosing a registry](#choosing-a-registry)).
- The namespaces your recipe deploys into, listed in the chart values; the
  chart creates them with their Pod Security labels and keeps them on
  uninstall.

The [chart README](https://github.com/NVIDIA/aicr/blob/main/charts/aicr-operator/README.md)
details each prerequisite, what the chart installs, and its admission policies;
[installing on EKS](/aicr/v1.3.0/operator-preview/installing-on-eks) walks through an ECR registry, the
operator's AWS identity and the component namespaces.

### Install the chart

Each AICR release publishes the operator image, the in-cluster registry image
and the chart, tagged with the release version. The chart defaults to that
release's operator and registry images and to that release's own AICR and
validator images, all pinned by digest. Before it publishes, the release runs
the operator with that AICR through install, rollout and deployment
validation, runs the validator images, and upgrades the previous release's
operator to this one. An earlier chart, such as v1.2.0's,
can default to an earlier AICR release. `helm show values` shows the default, and
[operations](/aicr/v1.3.0/operator-preview/day-2-operations#upgrading-aicr-within-v1) moves it.

```shell
VERSION="<release>"   # an AICR release that ships the operator, without the leading v
SIGNER=NVIDIA/aicr/.github/workflows/attest-operator.yaml   # for v1.2.0 and its release candidates: NVIDIA/aicr/.github/workflows/on-tag.yaml

kubectl create namespace aicr-system
kubectl label namespace aicr-system pod-security.kubernetes.io/enforce=restricted
```

Create `values.yaml` with your cluster's inputs:

```yaml
namespaces:
  components:          # every namespace the recipe deploys into
    - name: gpu-operator
      podSecurity: privileged
approvers:
  groups: [gpu-platform-approvers]
validation:
  enabled: true        # AICR deployment validation, which gates every rollout by default
```

Deployment validation runs AICR's validators with `cluster-admin`. If you
leave `validation.enabled` off, set `spec.validation.deployment: Disabled` on
the `ClusterRuntime` below; otherwise the runtime stalls with
`ValidationUnavailable` before any step runs.

Then verify the chart and install it:

```shell
gh attestation verify "oci://ghcr.io/nvidia/charts/aicr-operator:${VERSION}" --repo NVIDIA/aicr \
  --signer-workflow "${SIGNER}" --source-ref "refs/tags/v${VERSION}" \
  && helm install aicr-operator oci://ghcr.io/nvidia/charts/aicr-operator \
  --version "${VERSION}" --namespace aicr-system -f values.yaml
```

Required: `namespaces.components`, and with an external registry,
`registry.repository` and its credentials. Set `approvers.users` or
`approvers.groups`, or bind the `aicr-operator-approver` roles yourself: with no
approver, a rollout that needs approval waits in `AwaitingApproval` and expires
at its deadline. Everything else has a working default; `validation.enabled` is
recommended. Every value,
with its default and when to set it, is in the
[chart's values reference](https://github.com/NVIDIA/aicr/blob/main/charts/aicr-operator/README.md#values).

### Choosing a registry

The operator keeps the charts it mirrors, the bundles AICR renders and
validation evidence in an OCI registry, and Argo CD pulls every chart from it.
The registry is storage, not something the operator trusts: every plan pins
each chart by digest, so a lost or altered artifact fails the pull instead of
deploying other content.

| `registry.mode` | What you provide | Use it for |
| --- | --- | --- |
| `inCluster`, the default | A default StorageClass that exists before install, or `registry.inCluster.storage.storageClassName` | A cluster with storage in place and no registry of its own to manage |
| `external` | `registry.repository` and its credentials: ECR through the operator's AWS identity (`aws`), or Secrets you create (`static`) | A registry you already run, with your own retention, replication or access control; EKS, whose storage the recipe installs |

The in-cluster registry runs in the operator namespace under the restricted
Pod Security profile and serves plain HTTP inside the cluster. Only the
operator, its AICR Jobs in the Job and validation namespaces and Argo CD's
repo server reach it, where the CNI enforces NetworkPolicy; `kubectl port-forward`
bypasses the policy, so for anyone allowed to port-forward in the operator
namespace the login is the only control. It keeps serving through uninstall,
and it deletes nothing, so size the volume for every plan the cluster will
resolve. [Security](/aicr/v1.3.0/operator-preview/privilege-boundaries#the-runtime-registry) covers the trust model.

Choose before you install: plans name the registry they were mirrored to, so
changing `registry.mode` on a running installation is not supported. An
installation from a release before the in-cluster default keeps its registry
with `registry.mode: external`; the chart refuses to render until it is set.

### Install the kubectl plugin

Approvers use the `kubectl aicr` plugin. Each AICR
[GitHub release](https://github.com/NVIDIA/aicr/releases) carries its binaries
for Linux and macOS (`kubectl-aicr-<os>-<arch>`):

```shell
gh release download "v${VERSION}" --repo NVIDIA/aicr --pattern kubectl-aicr-darwin-arm64
gh attestation verify kubectl-aicr-darwin-arm64 --repo NVIDIA/aicr \
  --signer-workflow "${SIGNER}" --source-ref "refs/tags/v${VERSION}" \
  && install -m 0755 kubectl-aicr-darwin-arm64 /usr/local/bin/kubectl-aicr
```

## Usage

### Describe the cluster

The operator acts on one `ClusterRuntime` named `cluster`:

```yaml
apiVersion: operator.aicr.run/v1alpha1
kind: ClusterRuntime
metadata:
  name: cluster
spec:
  intent: training
  platform: kubeflow
  # Declare what the cluster is when its GPU nodes have no driver yet; the
  # snapshot taken after the GPU stack is installed must confirm it.
  criteria:
    service: eks
    accelerator: h100
    os: ubuntu
  placement:
    system:
      nodeSelector: {example.com/pool: system}
    accelerated:
      nodeSelector: {example.com/pool: gpu}
      tolerations:
        - {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}
```

### Approve rollouts

The operator discovers the node groups, resolves the matching AICR recipe and
plans the rollout. Risky transitions wait for an approver:

```shell
kubectl get clusterruntime cluster
kubectl get runtimerollouts
ROLLOUT="<rollout>"   # the name from the list
kubectl aicr describe "${ROLLOUT}"
kubectl aicr approve "${ROLLOUT}" --expires 2h --reason "Maintenance window"
```

A `manual` transition whose record has steps after the apply pauses once that
release is applied; do those steps, then run `kubectl aicr acknowledge` on the
rollout.

[Day-2 operations](/aicr/v1.3.0/operator-preview/day-2-operations) covers reading status, approvals, pausing,
rollback and AICR upgrades, and [troubleshooting](/aicr/v1.3.0/operator-preview/troubleshooting) every
stalled state. The [API reference](/aicr/v1.3.0/operator-preview/api-reference) describes every field.

### Verify the bundle yourself

`kubectl aicr reproduce` writes a plan's recipe (by default the active plan's)
to `aicr-reproduce/recipe.json` and prints the commands that render the same
bundle with the AICR CLI and compare it with the operator's. With the
placeholders filled in from its output:

```shell
kubectl aicr reproduce

# From its output: the AICR image the operator ran, the plan's flags, and the bundle.
AICR_IMAGE="<aicr image>"
PLAN_FLAGS=("<flag>" "<value>")
BUNDLE="<registry>/bundles/cluster@sha256:<digest>"

# Render with that AICR image, from the plan's recipe and flags.
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/aicr-reproduce:/work" -w /work \
  "${AICR_IMAGE}" bundle --recipe recipe.json "${PLAN_FLAGS[@]}" --output cli-bundle

# Pull the bundle the operator rendered: log in to an external registry first.
# For the in-cluster registry, run the port-forward and pull reproduce prints.
oras pull "${BUNDLE}" -o aicr-reproduce/operator-bundle

diff -r aicr-reproduce/operator-bundle aicr-reproduce/cli-bundle && echo "zero diff"
```

The plugin only reads the plan's record; the AICR image renders and `diff`
compares, so a record that did not match what the operator rendered would show
up as a diff. Each Argo CD Application's `spec.source.helm.valuesObject` is
that release's `values.yaml` merged with the bundle's `cluster-values.yaml`.
The plugin needs `get` on the `ClusterRuntime` and on ConfigMaps in the
operator namespace.

## Upgrade and uninstall

An operator upgrade is a chart upgrade (`helm upgrade` with the new
`--version`). `helm uninstall` leaves CRDs, namespaces and the Argo CD project
in place, so it never removes the GPU stack. Delete the `ClusterRuntime` first;
[operator lifecycle](/aicr/v1.3.0/operator-preview/operator-lifecycle) covers upgrade, downgrade, uninstall order
and what each keeps.

## Pages

| Page | Covers |
| --- | --- |
| [Installing on EKS](/aicr/v1.3.0/operator-preview/installing-on-eks) | ECR, the operator's AWS identity, the operator image, component namespaces and the first `ClusterRuntime` |
| [Day-2 operations](/aicr/v1.3.0/operator-preview/day-2-operations) | Reading status; approving, revoking and retrying; pausing versus suspending the engine; rolling back by pinning; changing inputs and the AICR version |
| [Troubleshooting](/aicr/v1.3.0/operator-preview/troubleshooting) | Runbooks by symptom: blocked plans, approvals, registry and credential outages, version mismatches, stuck finalizers, orphaned objects and re-adoption |
| [Operator lifecycle](/aicr/v1.3.0/operator-preview/operator-lifecycle) | Upgrading, downgrading and uninstalling the operator, what each keeps, and data retention |
| [Privilege boundaries](/aicr/v1.3.0/operator-preview/privilege-boundaries) | Every identity, what it may do, and the admission policies that hold it there |
| [Argo CD](/aicr/v1.3.0/operator-preview/argo-cd) | The Argo CD release and project the operator needs, its per-Application comparison exceptions, and what not to change by hand |
| [kubectl aicr](/aicr/v1.3.0/operator-preview/kubectl-aicr) | The plugin's commands, flags, output and permissions |
| [Support policy](/aicr/v1.3.0/operator-preview/support-policy) | Supported AICR, Kubernetes and Argo CD versions, qualified profiles, and when qualification goes stale |
| [API reference](/aicr/v1.3.0/operator-preview/api-reference) | Every field of `ClusterRuntime` and `RuntimeRollout`, the plan ConfigMap, and the installation values |