> This page is for AI Cluster Runtime, version Latest · v1.0.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v1.0.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt > - v1.0.0: https://docs.nvidia.com/aicr/v1.0.0/llms.txt > - v0.22.0: https://docs.nvidia.com/aicr/v0.22.0/llms.txt > - v0.21.1: https://docs.nvidia.com/aicr/v0.21.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server. # CLI Reference Complete reference for the `aicr` command-line interface. For details on which CLI verbs and critical user journeys are exercised by tests, on what hardware, and at what cadence, see the [Coverage Matrix](/aicr/user-guide/coverage-matrix). > Version numbers in examples (component, chart, and driver versions) are illustrative and may not match any released recipe. The authoritative, current versions live in the [Component Catalog](/aicr/user-guide/component-catalog) and the [Container Images BOM](https://github.com/NVIDIA/aicr/blob/main/docs/user/container-images.md). ## Overview AICR provides a four-step workflow for optimizing GPU infrastructure: ``` ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Snapshot │─────▶│ Recipe │─────▶│ Validate │─────▶│ Bundle │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ ``` **Step 1**: Capture system configuration **Step 2**: Generate optimization recipes **Step 3**: Validate constraints against cluster **Step 4**: Create deployment bundles ## Global Flags Available for all commands: | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--debug` | | bool | false | Enable debug logging (text mode with full metadata) | | `--log-json` | | bool | false | Enable JSON logging (structured output for machine parsing) | | `--help` | `-h` | bool | false | Show help | | `--version` | `-v` | bool | false | Show version | ### Logging Modes AICR supports three logging modes: 1. **CLI Mode (default)**: Minimal user-friendly output - Just message text without timestamps or metadata - Error messages display in red (ANSI color) - Example: `Snapshot captured successfully` 2. **Text Mode (`--debug`)**: Debug output with full metadata - Key=value format with time, level, source location - Example: `time=2025-01-06T10:30:00.123Z level=INFO module=aicr version=v1.0.0 msg="snapshot started"` 3. **JSON Mode (`--log-json`)**: Structured JSON for automation - Machine-readable format for log aggregation - Example: `{"time":"2025-01-06T10:30:00.123Z","level":"INFO","msg":"snapshot started"}` **Examples:** ```shell # Default: Clean CLI output aicr snapshot # Debug mode: Full metadata aicr --debug snapshot # JSON mode: Structured logs aicr --log-json snapshot # Combine with other flags (--debug is global; --output is a snapshot flag, so it follows the subcommand) aicr --debug snapshot --output system.yaml ``` ## Commands ### aicr snapshot Capture comprehensive system configuration including OS, GPU, Kubernetes, and SystemD settings. **Synopsis:** ```shell aicr snapshot [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--output` | `-o` | string | stdout | Output destination: file path, ConfigMap URI (cm://namespace/name), or stdout | | `--format` | `-t` | string | yaml | Output format: json, yaml, table | | `--config` | | string | | Path or HTTP/HTTPS URL to an AICRConfig file (YAML/JSON) that populates `spec.snapshot.*`. CLI flags below always win over the corresponding config field. | | `--kubeconfig` | `-k` | string | ~/.kube/config | Path to kubeconfig file (overrides KUBECONFIG env). Also used when `--output` is a ConfigMap URI so reads and writes target the same cluster. | | `--namespace` | `-n` | string | default | Kubernetes namespace for agent deployment. Reads `AICR_NAMESPACE` env when unset. | | `--image` | | string | matches CLI version | Container image for agent Job. Release builds default to `ghcr.io/nvidia/aicr:v`; dev and `-next` snapshot builds default to `ghcr.io/nvidia/aicr:latest`. Reads `AICR_IMAGE` env when unset. | | `--job-name` | | string | aicr | Prefix for the agent Job name; the run ID is always appended (`-`) | | `--service-account-name` | | string | aicr | ServiceAccount the agent pod runs as. **Exact-if-exists:** when a ServiceAccount of exactly this name already exists in `--namespace`, it is used verbatim and the run creates **no** ServiceAccount, Role, RoleBinding, ClusterRole, or ClusterRoleBinding — and deletes none at cleanup. Otherwise the value is a name prefix and the run ID is appended (`-`). Exact-if-exists mode needs **fewer** caller permissions — the run's pre-flight gate stops demanding `create`/`delete` on the five RBAC kinds — but requires the ServiceAccount to already carry the agent's rules, which the gate verifies with a `SubjectAccessReview`. See [Using an existing ServiceAccount](/aicr/user-guide/agent-deployment#using-an-existing-serviceaccount-irsa-and-workload-identity) and [Pre-flight permission gate](/aicr/user-guide/agent-deployment#pre-flight-permission-gate) | | `--add-roles-to-service-account` | | string | | **Writes manifests and applies nothing.** Renders the `Role`/`RoleBinding` (`aicr-agent--rbac`) and `ClusterRole`/`ClusterRoleBinding` (`aicr-agent-.-rbac`) that grant the agent's permissions to the named ServiceAccount into `./snapshot-rbac-/`, one object per file with a comment header explaining what it grants, then exits **without taking a snapshot**. **No cluster is contacted** — no kubeconfig or privileges needed, and the ServiceAccount is not checked for existence. Review the files, then apply with `kubectl apply -f /` and revoke with `kubectl delete -f /` yourself; no run cleanup ever touches them. Fails with `CONFLICT` if the directory already exists. Combine with `--discover-network` to also render the mutating live-discovery rules | | `--node-selector` | | string[] | auto | Node selector for agent scheduling (key=value, repeatable). When omitted (and neither `--require-gpu` nor `--runtime-class` is set), the agent auto-targets GPU nodes labeled `nvidia.com/gpu.present=true` if the cluster has any — see [GPU Node Auto-Targeting](/aicr/user-guide/agent-deployment#gpu-node-auto-targeting). Pass an explicit selector to override. | | `--toleration` | | string[] | all taints | Tolerations for agent scheduling (key=value:effect, repeatable). **Default: all taints tolerated** (uses `operator: Exists`). Only specify to restrict which taints are tolerated. | | `--timeout` | | duration | 5m | Timeout for agent Job completion | | `--no-cleanup` | | bool | false | Skip removal of Job and RBAC resources on completion. **Warning:** leaves both the agent's run-scoped `aicr-node-reader-` ClusterRole and the identically named ClusterRoleBinding active. By default the ClusterRole grants only read-only access; with `--discover-network` it also carries the mutating rules live network discovery needs (CRD/namespace/daemonset create, pod exec, node patch, NicClusterPolicy). Delete both when you are done — removing only the binding leaves the grant definition behind. | | `--privileged` | | bool | true | Run agent in privileged mode (required for GPU/SystemD collectors). Set to false for PSS-restricted namespaces. | | `--image-pull-secret` | | string[] | | Image pull secrets for private registries (repeatable) | | `--require-gpu` | | bool | false | In privileged mode (the default), requests an `nvidia.com/gpu` resource for the agent pod, so it schedules only onto GPU nodes. With `--privileged=false` it adds no GPU request, but still disables automatic GPU-node selection, so the agent can land on a CPU node unless you pass `--node-selector` (mutually exclusive with `--runtime-class`). Reads `AICR_REQUIRE_GPU` env when unset. | | `--runtime-class` | | string | | Runtime class for GPU access without consuming a GPU allocation (e.g., `nvidia`). Mutually exclusive with `--require-gpu`. Reads `AICR_RUNTIME_CLASS` env when unset. | | `--template` | | string | | Path to Go template file for custom output formatting (requires YAML format) | | `--max-nodes-per-entry` | | int | 0 | Maximum node names per taint/label entry in topology collection (0 = unlimited) | | `--os` | | string | | Node OS family (`ubuntu`, `rhel`, `cos`, `amazonlinux`, `ol`, `talos`). Selects the per-OS pod configuration and in-pod service collector backend. `talos` skips the `/run/systemd` and `/etc/os-release` hostPath mounts and uses the Kubernetes-API service backend. Reads `AICR_OS` env when unset. | | `--requests` | | string | | Override agent container resource requests as a comma-separated list of `name=quantity` pairs (e.g. `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`). Unspecified resources keep the built-in privileged or restricted defaults. Reads `AICR_REQUESTS` env when unset. | | `--limits` | | string | | Override agent container resource limits as a comma-separated list of `name=quantity` pairs (e.g. `cpu=1,memory=2Gi,ephemeral-storage=2Gi`). Unspecified resources keep the built-in defaults. With `--require-gpu`, the default `nvidia.com/gpu=1` is applied only when `--limits` does not already contain that key — an explicit `--limits nvidia.com/gpu=N` wins. Reads `AICR_LIMITS` env when unset. | | `--cluster-config` | | string | | Path to a pre-existing k8s-launch-kit (l8k) `cluster-config.yaml`. Ingests the file's per-hardware-group network topology (PFs, capabilities, kernel modules, machine/GPU type, fabric type) into the snapshot as a `NetworkTopology` Measurement. **Local agent mode only for now** (`AICR_AGENT_MODE=true`) — Job-mode rejects this flag with an `INVALID_REQUEST` error until ConfigMap mounting is implemented. Mutually exclusive with `--discover-network` at the collector level — file path wins when both are set, so callers can default discovery from a flag without inadvertent cluster contact. Reads `AICR_CLUSTER_CONFIG_PATH` env when unset. | | `--oke-addons` | | string | | Path to an `oci ce cluster list-addons --cluster-id --all --output json` dump on the local filesystem. Projects the `NvidiaGpuPlugin` add-on's control-plane state into the `K8s.oke-addons.nvidia-gpu-plugin` snapshot reading (`installed` / `absent`); any other add-on lifecycle state projects a value no profile constraint accepts, so profile-qualified resolution fails closed with the observed state. The projection runs controller-side and is merged into the snapshot in both agent Job mode and local mode; a bad file fails the command before any cluster work. Input is capped at 1 MiB and must be a regular file. Reads `AICR_OKE_ADDONS_PATH` env when unset. Also accepted by `aicr validate` for its live-capture path. Example: `oci ce cluster list-addons --cluster-id --all --output json > addons.json && aicr snapshot --oke-addons addons.json -o snapshot.yaml`. | | `--aks-gpu-pools` | | string | | Path to an `az aks nodepool list -o json` dump on the local filesystem. Projects each NVIDIA GPU agent pool's `gpuProfile.driver` into the `K8s.aks-gpu-pools.gpu-driver` snapshot reading (`Install` / `None`); mixed or AKS-managed pools project a value no profile constraint accepts, so profile-qualified resolution fails closed with the observed state (ADR-015 DD3). AMD GPU pools (NG family, MI300X-class ND sizes, Radeon NV sizes) are excluded. The projection runs controller-side and is merged into the snapshot in both agent Job mode and local mode; a bad file fails the command before any cluster work. Input is capped at 1 MiB and must be a regular file. Reads `AICR_AKS_GPU_POOLS_PATH` env when unset. Also accepted by `aicr validate` for its live-capture path. Example: `az aks nodepool list -g --cluster-name -o json > pools.json && aicr snapshot --aks-gpu-pools pools.json -o snapshot.yaml`. | | `--gke-gpu-pools` | | string | | Path to a `gcloud container node-pools list --cluster --format=json` dump on the local filesystem. Projects each GPU pool's `gpuDriverInstallationConfig.gpuDriverVersion` into the `K8s.gke-gpu-pools.gpu-driver-installation` snapshot reading (`Installed` / `Disabled`). Mixed or unrecognized driver-version values project a value no profile constraint accepts, so profile-qualified resolution fails closed with the observed state. Only the GKE `bundle-installer` gpuStack value declares a constraint on this reading. The default `gke-default` value resolves from the opt-out node label alone and needs no pool dump. The projection runs controller-side and is merged into the snapshot in both agent Job mode and local mode. A bad file fails the command before any cluster work. Input is capped at 1 MiB and must be a regular file. Reads `AICR_GKE_GPU_POOLS_PATH` env when unset. Also accepted by `aicr validate` for its live-capture path. Example: `gcloud container node-pools list --cluster --format=json > pools.json && aicr snapshot --gke-gpu-pools pools.json -o snapshot.yaml`. | | `--discover-network` | | bool | false | Opt into live k8s-launch-kit (l8k) discovery: bootstraps an in-cluster nic-configuration daemon, walks the cluster's NICs, and emits a `NetworkTopology` Measurement. **NOT read-only** — writes `nvidia.kubernetes-launch-kit.machine` / `.gpu` labels on matched nodes and patches `NicClusterPolicy` via server-side apply. Job-mode is supported (the snapshot Job's ClusterRole gains discovery-specific RBAC when this flag is set). Reads `AICR_DISCOVER_NETWORK` env when unset. | **Output Destinations:** - **stdout**: Default when no `-o` flag specified - **File**: Local file path (`/path/to/snapshot.yaml`) - **ConfigMap**: Kubernetes ConfigMap URI (`cm://namespace/configmap-name`) **Output Formats:** `--format` applies to every destination. - `yaml` (default) delivers the agent's document byte-for-byte to a file or stdout, so fields emitted by a newer `--image` than the CLI survive. - `json` re-encodes that document with the same keys, which is what `aicr diff --target snapshot.json` and `jq` expect. Since `aicr diff` picks its decoder from the file extension, pair `--format json` with a `.json` path. - `table` is a flattened `FIELD`/`VALUE` rendering for humans and cannot be read back by `aicr diff`, `aicr validate --snapshot`, or `aicr recipe --snapshot`. - ConfigMap destinations store the rendering under the `snapshot.yaml`, `snapshot.json`, or `snapshot.txt` data key alongside a `format` key; AICR's ConfigMap readers follow that key, so `yaml` and `json` are both consumable from `cm://`. Because those keys and the resource labels are derived from the document, a `cm://` destination re-serializes it (deterministically, and without dropping unmodeled fields) rather than storing the agent's exact bytes — use a file or stdout when you need byte-identical YAML. - `--template` supplies its own rendering and therefore requires `--format yaml` (or no `--format`). **What it captures:** - **SystemD Services**: containerd, docker, kubelet configurations - **OS Configuration**: grub, kmod, sysctl, release info - **Kubernetes**: server version, images, ClusterPolicy - **GPU**: driver-free PCI/NFD hardware detection — GPU presence, GPU count, accelerator model/SKU (resolved from the PCI device ID), nvidia kernel-module loaded state, and detection source. Does not require the NVIDIA driver or `nvidia-smi`, and does not emit driver version, CUDA version, or MIG settings. - **NodeTopology**: node topology (cluster-wide taints and labels across all nodes) - **NetworkTopology**: per-hardware-group network topology (PFs, capabilities, kernel modules, machine/GPU type, fabric type) — emitted only when `--cluster-config` or `--discover-network` is set **Examples:** ```shell # Output to stdout (YAML) aicr snapshot # Save to file (JSON) aicr snapshot --output system.json --format json # Save to Kubernetes ConfigMap (requires cluster access) # The agent Job writes the ConfigMap directly, using the Role created in its # deployment namespace (--namespace, default `default`). To write into another # namespace, deploy the agent there so it has the matching ConfigMap RBAC: aicr snapshot --namespace gpu-operator --output cm://gpu-operator/aicr-snapshot # Debug mode aicr --debug snapshot # Table format (human-readable) aicr snapshot --format table # With custom kubeconfig aicr snapshot --kubeconfig ~/.kube/prod-cluster # Auto-target GPU nodes (default when no placement flag is set) # On a cluster with nvidia.com/gpu.present=true nodes, the agent is # steered onto a GPU node automatically — see GPU Node Auto-Targeting. aicr snapshot --namespace gpu-operator # Targeting specific nodes (explicit selector disables auto-targeting) aicr snapshot \ --namespace gpu-operator \ --node-selector accelerator=nvidia-h100 \ --node-selector zone=us-west1-a # With tolerations for tainted nodes # (By default all taints are tolerated - only needed to restrict tolerations) aicr snapshot \ --toleration nvidia.com/gpu=present:NoSchedule # Full example with all options. --image defaults to the agent image that # matches the CLI version; set AICR_VERSION to your CLI release. AICR_VERSION=vX.Y.Z aicr snapshot \ --kubeconfig ~/.kube/config \ --namespace gpu-operator \ --image ghcr.io/nvidia/aicr:${AICR_VERSION} \ --job-name snapshot-gpu-nodes \ --node-selector accelerator=nvidia-h100 \ --toleration nvidia.com/gpu:NoSchedule \ --timeout 10m \ --output cm://gpu-operator/aicr-snapshot \ --no-cleanup # Write the RBAC manifests that grant the agent's permissions to an existing # ServiceAccount, then exit. Applies nothing and contacts no cluster; takes no # snapshot. Writes ./snapshot-rbac-/. aicr snapshot \ --namespace gpu-operator \ --add-roles-to-service-account irsa-snapshotter # Review what each file grants, then apply them yourself. kubectl apply -f snapshot-rbac-/ # Capture as that ServiceAccount. Because it already exists, the name is used # verbatim and this run creates and deletes no RBAC of its own. aicr snapshot \ --namespace gpu-operator \ --service-account-name irsa-snapshotter \ --output cm://gpu-operator/aicr-snapshot # Custom template formatting aicr snapshot --template examples/templates/snapshot-template.md.tmpl # Template with file output aicr snapshot --template examples/templates/snapshot-template.md.tmpl --output report.md # With custom template aicr snapshot \ --namespace gpu-operator \ --template examples/templates/snapshot-template.md.tmpl \ --output cluster-report.yaml ``` #### Snapshot Config File Mode Drive `aicr snapshot` from an `AICRConfig` document so the snapshot inputs version-control alongside the recipe, bundle, and validate steps in an end-to-end workflow. ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 metadata: name: gke-h100-training spec: snapshot: output: path: snapshot.yaml # written to disk; same shape as -o format: yaml # yaml | json | table template: "" # optional Go template path agent: namespace: aicr-validation image: "" # default: matches the CLI version (:latest on dev/-next builds) imagePullSecrets: [] # jobName is an optional PREFIX, not a name — the run ID is always # appended. serviceAccountName is exact-if-exists: an existing # ServiceAccount of exactly that name is used verbatim and the run # then creates and deletes NO RBAC; otherwise it is a prefix too. # Omit both to take the run-scoped defaults. nodeSelector: nodeGroup: gpu-worker tolerations: - dedicated=gpu-workload:NoSchedule - nvidia.com/gpu=present:NoSchedule requireGpu: false runtimeClassName: "" # mutually exclusive with requireGpu os: "" # ubuntu | rhel | cos | amazonlinux | ol | talos requests: "" # "cpu=500m,memory=1Gi" limits: "" # "cpu=1,memory=2Gi" execution: timeout: 5m noCleanup: false privileged: true # set false for PSS-restricted namespaces maxNodesPerEntry: 0 # 0 = unlimited topology entries ``` Precedence: a CLI flag always wins over the matching config field. Selectors and tolerations omitted entirely inherit the snapshotter's compiled-in defaults (`tolerations` defaults to *tolerate all taints*); an explicit empty list (`tolerations: []`) clears the tolerate-all default — the same nil-vs-empty semantics used by `spec.validate.agent`. ```shell # Run snapshot driven entirely by config aicr snapshot --config aicr-config.yaml # Reuse the same config but write to a one-off path aicr snapshot --config aicr-config.yaml -o /tmp/snapshot.yaml ``` #### Custom Templates The `--template` flag enables custom output formatting using Go templates with [Sprig functions](https://masterminds.github.io/sprig/). Templates receive the full Snapshot struct: ```text # Available template data structure: .Kind # Resource kind ("Snapshot") .APIVersion # API version string .Metadata # Map of key-value pairs (timestamp, version, source-node) .Measurements # Array of Measurement objects .Type # Measurement type (K8s, GPU, OS, SystemD, NodeTopology, NetworkTopology) .Subtypes # Array of Subtype objects .Name # Subtype name (e.g., "server", "hardware", "grub") .Data # Map of readings (key -> Reading with .String method) # NodeTopology measurement type has subtypes: summary, taint, label # Taint encoding: effect|value|node1,node2,... (parseable with Sprig splitList "|") # Label encoding: value|node1,node2,... ``` Example template extracting key cluster info: ```go cluster: kubernetes: {{ with index .Measurements 0 }}{{ range .Subtypes }}{{ if eq .Name "server" }} version: {{ (index .Data "version").String }}{{ end }}{{ end }}{{ end }} gpu: {{ range .Measurements }}{{ if eq .Type.String "GPU" }}{{ range .Subtypes }}{{ if eq .Name "hardware" }} model: {{ (index .Data "model").String }} count: {{ (index .Data "gpu-count").String }}{{ end }}{{ end }}{{ end }}{{ end }} ``` See `examples/templates/snapshot-template.md.tmpl` for a complete example template that generates a concise cluster report. #### Agent Deployment When running against a cluster, AICR deploys a Kubernetes Job to capture the snapshot. For the RBAC the agent creates, the in-cluster Job lifecycle, and ConfigMap storage, see [Agent Deployment](/aicr/user-guide/agent-deployment). For GPU-node auto-targeting (proactive selector injection plus the reactive placement-mismatch warning), see [GPU Node Auto-Targeting](/aicr/user-guide/agent-deployment#gpu-node-auto-targeting). #### ConfigMap Output When using ConfigMap URIs (`cm://namespace/name`), the snapshot is stored directly in Kubernetes: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aicr-snapshot namespace: gpu-operator labels: app.kubernetes.io/name: aicr app.kubernetes.io/component: Snapshot app.kubernetes.io/version: data: snapshot.yaml: | # Full snapshot content format: yaml timestamp: "2025-12-31T10:30:00Z" ``` **Snapshot Structure:** ```yaml apiVersion: aicr.run/v1 kind: Snapshot metadata: timestamp: "2025-12-31T10:30:00Z" version: v1.0.0 source-node: gpu-node-1 measurements: - type: SystemD subtypes: [...] - type: OS subtypes: [...] - type: K8s subtypes: [...] - type: GPU subtypes: [...] ``` --- ### aicr recipe Generate optimized configuration recipes from query parameters or captured snapshots. **Synopsis:** ```shell aicr recipe [flags] ``` **Modes:** `aicr recipe` resolves a recipe from **criteria** — `service`, `accelerator`, `os`, `intent`, `platform`. (`nodes` is accepted but is advisory metadata — it does not select or filter overlays.) You can supply those criteria three ways, composed with the precedence **CLI flags > `--config` file > `--snapshot`**: - **Snapshot** (`--snapshot`) — criteria are auto-detected from a captured cluster snapshot: accelerator from the `nvidia.com/gpu.product` GFD label (primary — cluster-wide, so it surfaces heterogeneous clusters) or the per-node PCI device ID (fallback — maps the device ID to a SKU, e.g. `h100`, with no driver or GFD label required); service from the node's cloud-provider ID; OS from the node's OS release; node count from cluster topology. Use this when you already have a running cluster: you don't hand-specify the hardware, AICR reads it. (A detected SKU fills `criteria.accelerator` only when it's in the supported accelerator set; an unsupported GPU is recorded descriptively but not as a recipe criterion.) On the snapshot path AICR also reads the sampled GPU node's `driver-loaded` reading; on unprofiled/legacy compositions whose resolved values already carry the coordinated preinstalled-driver configuration, it injects `gpu-operator.overrides.driver.enabled=false` into the resolved recipe, while on a profiled AKS composition the snapshot only *qualifies* the explicit-or-default `gpuStack` selection — the injector skips profile-owned paths and never mutates them; the inverse mismatch — a preinstalled-driver overlay resolved against a snapshot with *no* driver loaded — is handled per family: on profiled AKS it either fails closed at the profile constraint (pools not reading `Install`) or records `gpuDriverState: absent` for the bundle-time gate whose remedy is pool repair/recreation + recapture + `--profile`; on non-profiled families and legacy artifacts it warns with the bundle-time override set. See [Component Catalog › GPU Operator Driver Auto-Detect](/aicr/user-guide/component-catalog#gpu-operator-driver-auto-detect) for the gate, warnings, and the pre-deploy-snapshot requirement. - **Config file** (`--config`) — criteria (and bundle settings) from an `AICRConfig` document; good for reproducible, version-controlled workflows. - **Query flags** (`--service`, `--accelerator`, …) — state the criteria directly. Use this when there is **no cluster to snapshot yet** — the common case when you generate a recipe in order to *provision* a cluster, or build one offline/ahead of time for hardware you can't reach. **Why you sometimes specify criteria the snapshot could detect:** recipe generation **does not collect live cluster state or deploy/modify workloads** — it reads inputs (criteria, embedded/`--data` catalog, an optional snapshot) and does **not** snapshot a cluster for you. Its only cluster interaction is explicit `cm://` ConfigMap reads/writes (snapshot input or recipe output). This keeps generation hermetic and reproducible (same inputs → same recipe) and lets it run before a target cluster even exists. So you state criteria explicitly when there's no cluster to read; when a cluster is available, capture it first and let detection fill them in: ```shell aicr snapshot -o cm://default/snapshot # captures the cluster (deploys the collector agent) aicr recipe -s cm://default/snapshot --intent training ``` When both a snapshot and explicit criteria are given, the explicit values win (e.g. `--snapshot … --service gke` overrides the detected service). This is also the only way to select a `generic` recipe from a snapshot: a bare-metal cluster fingerprints as its provisioner (`metal3`, `rke2`), never as `generic`, so pass the override explicitly: ```bash aicr recipe --snapshot snapshot.yaml --service generic --intent training ``` **Configuration profiles:** an overlay composition may declare one named configuration choice. Select a non-default value with `--profile name=value`; omitting the flag applies the declaration's required default. A selection against a composition with no declaration, a wrong name, or an unknown value fails closed. Profile-bearing output records `metadata.selectedProfile`, uses recipe apiVersion `aicr.run/v1beta2`, and locks every declared owned path: divergent `aicr bundle`/`aicr mirror` static overrides are rejected (identical values accepted), and argocd-helm install-time values are rejected on key *presence* alone — even when the value is identical to the selected one. The AKS family is the first embedded adopter: `gpuStack` with values `azure-managed` (default) and `operator-managed` — see [AKS GPU setup](/aicr/integrator-guide/aks-gpu-setup#gpu-driver-setup). The GKE family declares `gpuStack` with values `gke-default` (default; GKE's managed plugin stays the advertiser — recorded as `advertiser: external` — for default-provisioned clusters with no node label) and `bundle-installer` (the GPU Operator's device plugin owns `nvidia.com/gpu`; GPU node pools carry `gke-no-default-nvidia-gpu-device-plugin=true` and, because that label forfeits GKE's managed driver install, are created `gpu-driver-version=disabled` — the bundle's `gcp-driver-installer` component supplies the driver with a recipe-pinned version, and a snapshot carrying the label without a matching `K8s.gke-gpu-pools.gpu-driver-installation` reading (`--gke-gpu-pools`) fails the `bundle-installer` selection closed); because the GKE values govern advertisement, the #1327 allocation-policy paths are closure-locked in addition to the declared owned paths — see [GKE GPU setup](/aicr/integrator-guide/gke-gpu-setup#gpu-device-plugin-ownership) and [Component Catalog › GKE Device-Plugin Ownership](/aicr/user-guide/component-catalog#gke-device-plugin-ownership). The OKE family declares `gpuStack` with values `oci-managed` (default; Oracle's GPU node image supplies the driver and OKE's `NvidiaGpuPlugin` add-on advertises — `advertiser: external`) and `operator-managed` (bring-your-own driverless image with the add-on removed; the operator installs driver, toolkit, and plugin, with the DRA driver root in lockstep). Each value is qualified by the add-on's control-plane state, supplied as an `oci ce cluster list-addons --cluster-id --all --output json` dump via `--oke-addons` on `aicr snapshot` and `aicr validate` — see [OKE GPU setup](/aicr/integrator-guide/oke-gpu-setup). Profiles can also be exercised through a versioned external overlay. Selection and verification are independent: `--profile` (or the default) always decides the selected value — never the snapshot — and a supplied `--snapshot` always verifies the selection against the cluster's recorded readings (fail-closed on mismatch or a missing reading; on AKS, the pool mode from `--aks-gpu-pools`; on GKE `bundle-installer`, the pool mode from `--gke-gpu-pools`). Without a snapshot no check can run at generation; the recorded constraint is enforced at `aicr validate` readiness instead. See the with/without matrices in [AKS GPU setup](/aicr/integrator-guide/aks-gpu-setup#gpu-driver-setup) and [GKE GPU setup](/aicr/integrator-guide/gke-gpu-setup#gpu-device-plugin-ownership). **Every stated criteria dimension must be honored:** for `service`, `accelerator`, `intent`, `os`, and `platform`, resolution now enforces a coverage post-condition — if you state a value for one of these dimensions, at least one applied overlay must carry that exact value, or the request fails with an actionable `INVALID_REQUEST` error instead of silently returning a recipe that ignores what you asked for. The error names the uncovered dimension and, where possible, the additional criteria that would make the request resolvable (e.g. `platform 'kubeflow' ... requires os (valid: ubuntu)`). `--nodes` is exempt from this check — it is advisory only, since no overlay gates on node count, and stating it never requires matching node-count-specific recipe content to exist. **Snapshot-detected dimensions are advisory, not strict:** on the `--snapshot` path, `service`, `accelerator`, and `os` can be auto-detected from the cluster fingerprint rather than stated by you. If the coverage post-condition above fails and every uncovered dimension came from that detection (none of them were set via a CLI flag or `--config`), AICR relaxes those dimensions back to unstated and retries resolution once, logging a warning that names each relaxed dimension and its detected value — this handles overlay trees that are deliberately agnostic to a dimension the snapshot still reports (e.g. an OS-agnostic Kind overlay tree observing `os=ubuntu` on the node). Dimensions you passed explicitly via a flag are never relaxed: if any uncovered dimension was explicitly stated, the error still fails the request. #### Config File Mode (Recommended) Generate recipes using an `AICRConfig` document. The same file format also drives the `bundle` command, so a single file can describe an end-to-end recipe-to-bundle workflow. **Flags:** | Flag | Short | Type | Description | |------|-------|------|-------------| | `--config` | | string | Path or HTTP/HTTPS URL to an AICRConfig file (YAML/JSON) | | `--output` | `-o` | string | Output file (default: stdout) | | `--format` | `-t` | string | Format: json, yaml, table (default: yaml) | | `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) | The config file uses a Kubernetes-style envelope: ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 metadata: name: gb200-eks-ubuntu-training spec: recipe: criteria: service: eks os: ubuntu accelerator: gb200 intent: training nodes: 8 platform: slurm configuration: slurm: accounting: mode: disabled output: path: recipe.yaml format: yaml ``` Individual CLI flags always override config file values. For slice/map flags, presence on the CLI replaces the file's value (no append). For a composition that declares a profile, the equivalent config field is `spec.recipe.profile`. The CLI flag takes precedence. For example, on the AKS family (the embedded adopter): ```yaml spec: recipe: criteria: service: aks os: ubuntu accelerator: h100 intent: training profile: gpuStack=operator-managed ``` ```shell # Load criteria from config file aicr recipe --config config.yaml # Override service from file aicr recipe --config config.yaml --service gke # Save output to file aicr recipe --config config.yaml -o recipe.yaml # Load config from a URL (e.g. CI shared template) aicr recipe --config https://team.example.com/configs/eks-h100-training.yaml ``` `--config` accepts a local file path or an HTTP/HTTPS URL. ConfigMap (`cm://`) sources are not supported; export the data with `kubectl get cm -o yaml` and pass the resulting file. #### Query Mode Generate recipes using direct system parameters: **Flags:** | Flag | Short | Type | Description | |------|-------|------|-------------| | `--service` | | string | K8s service: eks, gke, aks, oke, ocp, kind, lke, bcm, metal3, rke2, generic, k0s. `generic` is a concrete value (self-managed Kubernetes with no distinguishing distro or provisioner; `self-managed`, `self`, and `vanilla` are accepted aliases) — unlike the `any` wildcard, which matches every service and does not select `generic` recipes. `generic` is never detected from a snapshot (the fingerprint reports the provisioner it sees, such as `metal3` or `rke2`), so `generic` recipes require this flag as an explicit opt-in, also alongside `--snapshot`; recipe evidence records the service dimension as `not-inferable` with the observed provider, which does not disqualify the evidence | | `--accelerator` | `--gpu` | string | Accelerator/GPU type: h100, h200, gb200, gb300, b200, a100, l40, l40s, rtx-pro-6000, vr200 | | `--intent` | | string | Workload intent: training, inference | | `--os` | | string | OS family: ubuntu, rhel, cos, amazonlinux, ol, talos | | `--platform` | | string | Platform/framework type: dynamo, kubeflow, nim, runai, slurm | | `--profile` | | string | Profile selection in exact `name=value` form (e.g. `gpuStack=operator-managed` on AKS/OKE or `gpuStack=bundle-installer` on GKE); omit to use the declaration's default (`gpuStack=azure-managed` on AKS, `gpuStack=gke-default` on GKE, `gpuStack=oci-managed` on OKE) | | `--slurm-accounting-mode` | | string | Slurm accounting ownership: disabled (default), customer-managed, aicr-provided | | `--runtime-inventory` | | string | Runtime AI inventory (`k8s-aibom`) selection: `enabled` grants the component on a GKE recipe that neither declares nor declines it, confirms it where the recipe already declares it, and is rejected over an explicit recipe decline (any service) or against a non-GKE recipe that does not declare it; `disabled` is unchanged and always requires the recipe to already declare the component. Recorded in the generated recipe | | `--gke-tcpxo-interfaces` | | string | Ordered `eth1=,...,eth8=` GPU-NIC Network mapping for the `torch-distributed-tcpxo` runtime. Required when the resolved recipe ships it (h100 GKE kubeflow training); recorded in the generated recipe | | `--nodes` | | int | Number of GPU nodes in the cluster | | `--inherit-from` | | string | Prior recipe file, or bundle directory, whose component namespace, chart, source, path, manifest files and pre-manifest files the resolved recipe keeps instead of re-deriving them from the registry. When both artifacts state a deployment type and they differ, such as a Helm to Kustomize change, only the namespace is kept for that component and `aicr upgrade-check` reports the type move. Given a bundle directory it also keeps that bundle's `fullnameOverride` / `nameOverride` object names, writing an override only where one differs. Use on an AICR upgrade so a moved registry default does not relocate, replace or rename a component that is already running. See [Upgrading a Deployed Stack](/aicr/user-guide/upgrading-a-deployed-stack#pinning-the-namespaces-you-already-deployed-into). A component the prior artifact does not name keeps the registry default. `cm://` locations are not supported yet | | `--output` | `-o` | string | Output file (default: stdout) | | `--format` | `-t` | string | Format: json, yaml, table (default: yaml) | | `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) | | `--criteria-strict` | | bool | Reject criteria values not in the embedded OSS catalog; ignores values registered from `--data`. Also honored via `AICR_CRITERIA_STRICT=1` or `spec.recipe.criteriaStrict: true` in `--config`. Intended for OSS CI gates. | **Accelerator values name a GPU model, not a machine type.** A provider usually offers several machine types for the same GPU, and the machine type — not the GPU — determines the fabric, the NIC count, and which components a recipe can use. `--accelerator h100` therefore does not, on its own, say which node shape the resolved recipe targets. See [Qualified Machine Types](#qualified-machine-types) below. > **Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set.** When `--data` registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see [Data Extension](/aicr/integrator-guide/data-extension). `--criteria-strict` restores the OSS-only set regardless of what `--data` contributes. > **`--service rke2` and `--accelerator vr200` are Preview.** They publish an early-adopter recipe path without the full production support and lifecycle qualification required for Supported status. See the published validation evidence at [validation.aicr.run](https://validation.aicr.run/) for current coverage. > **`--service k0s` is Preview**, covering the single `k0s / h200 / ubuntu / training` coordinate. See [k0s H200 Setup](/aicr/integrator-guide/k-0-s-h-200-setup) for its prerequisites and known gaps. **Examples:** ```shell # Basic recipe for Ubuntu on EKS with H100 aicr recipe --os ubuntu --service eks --accelerator h100 --intent training # Training workload with multiple GPU nodes aicr recipe \ --service eks \ --accelerator gb200 \ --intent training \ --os ubuntu \ --nodes 8 \ --format yaml # Kubeflow training workload aicr recipe \ --service eks \ --accelerator h100 \ --intent training \ --os ubuntu \ --platform kubeflow # Save to file (--gpu is an alias for --accelerator) aicr recipe --os ubuntu --gpu h100 --service eks --intent training --output recipe.yaml # Slurm with an AICR-provided accounting database installation aicr recipe \ --service eks \ --accelerator h100 \ --intent training \ --os ubuntu \ --platform slurm \ --slurm-accounting-mode aicr-provided ``` The AICR-provided example above is criteria-only: because it does not use `--snapshot`, the resulting recipe carries no target-cluster MariaDB Operator conflict evidence. Bundling warns that conflicts were not evaluated and proceeds for compatibility. Use a current snapshot before deployment when you need conflict detection. See [Conflict detection requires snapshot evidence](/aicr/user-guide/slurm-accounting#conflict-detection-requires-snapshot-evidence). #### Qualified Machine Types Each recipe is qualified against a specific node shape. Criteria resolution does not reject another machine type of the same GPU model — there is no axis to reject it on — so a recipe always resolves. What differs by family is what happens afterwards: on some, deployment validation fails; on others it succeeds and only the performance gates are affected. | Accelerator | Service / intent | Qualified machine type | On other shapes of the same GPU | |---|---|---|---| | `h100` | `gke`, `training` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. AICR ships no GPUDirect-TCPX component for the shapes that need one — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). | | `h100` | `gke`, `inference` | not machine-type-bound (`dynamo` floors calibrated on `a3-megagpu-8g`) | Deploys. The inference lineage carries no `gke-nccl-tcpxo` component, so the hard failure above does not apply. Plain `inference` declares no performance gates at all; the `dynamo` variant adds floors calibrated on the 8-GPU node, so smaller shapes such as `a3-highgpu-1g/2g/4g` can false-fail there. | | `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. | | `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM, InfiniBand) | **Deployment fails on the non-IB NCads shapes.** The AKS chain wires `network-operator` with a NicClusterPolicy unconditionally, so the deployment-phase `expected-resources` check runs an RDMA-fabric readiness gate that fails closed. `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) are PCIe H100 with no InfiniBand, so they never advertise the shared RDMA resource and the gate fails before any performance gate runs. To run these recipes on a non-IB shape, disable the component in the recipe itself — set `overrides.enabled: false` on the `network-operator` componentRef (or use an overlay that omits the NicClusterPolicy manifest). A bundle-time `--set` does not help: `aicr validate` has no `--set` flag, and the gate reads the recipe's componentRefs, not the bundle's Helm values. | | `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape and are themselves provisional pending production NVL72 data. | | `a100` | `gke` | the whole `a2` family (`a2-highgpu-*`, `a2-ultragpu-*`) | Family-level by construction, not per-shape: GPUDirect-TCPXO targets H100 `a3-megagpu-8g`, so the `gke-nccl-tcpxo` component is inapplicable to every `a2` shape and is intentionally omitted. No shape in the family carries a machine-type-bound component. | | `b200` | `gke` | the `a4` family — **specific machine type not recorded** | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE `a4`'s GCP-managed multi-NIC, so nothing here is machine-type-bound. The overlay records a production reference cluster but no machine type, so this row cannot name one. | A row that names no intent applies to every intent for that accelerator and service. Where a row names a family rather than a machine type, the entry is a family-level statement — either because no component in that family binds to a machine type, or because the specific shape is not recorded in-repo. The row says which. Two distinct failure modes are worth separating: - **Component-level (hard).** Two families fail deployment outright, by different mechanisms. The GKE H100 **training** lineage pins artifacts to a machine type — `h100-gke-cos-training` and the leaves inheriting it — so on a non-matching shape the DaemonSets have nowhere to land and a Chainsaw health check fails. The AKS H100 **training** lineage instead wires an RDMA fabric unconditionally, and a Go readiness gate in the deployment phase fails closed when no node advertises the shared RDMA resource — which is every non-IB NCads shape. Neither is a degradation; both stop the deployment phase. - **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the NCCL and inference floors are fixed absolute values calibrated on full, high-bandwidth nodes. They are not normalized for GPU count or fabric class, so a smaller shape can fail a gate while being perfectly healthy. This is the EKS and GB200 case; on AKS the deployment gate above bites first. See [Validation › Node-shape assumption](/aicr/user-guide/validation). Normalizing these floors per GPU or per fabric class was considered and declined ([#1256](https://github.com/NVIDIA/aicr/issues/1256), [#1254](https://github.com/NVIDIA/aicr/issues/1254), both closed as not planned) — the floors are deliberately fixed absolute full-node values, so running a qualified shape is the supported way to pass them. The table lists the accelerator/service pairs that have a qualified shape; a pair or a shape absent from it is **undocumented rather than known-broken**. It has not been qualified, and the criteria model has no axis that would distinguish it from one that has. Whether AICR should gain one — finer-grained accelerator values, a machine-type axis, or a fabric class — is tracked in [#2377](https://github.com/NVIDIA/aicr/issues/2377). #### Snapshot Mode Generate recipes from captured snapshots: Snapshots derive infrastructure criteria such as service, accelerator, and OS, but `intent` and `platform` remain explicit choices. `K8s.slinky-slurm` reports declared Slinky Controller presence and, for a single Controller, a secret-safe associated-resource topology; it does not select Slurm or reconstruct the installed chart's Helm values. `K8s.mariadb-operator` records official MariaDB Operator API/CR conflict evidence but does not infer database availability, accounting intent, or database source. Pass `--platform slurm` to resolve a Slurm leaf. Older snapshots without either subtype remain compatible. For `aicr-provided`, missing MariaDB Operator conflict evidence causes bundling to warn that conflicts were not evaluated and proceed without target-cluster conflict detection. **Flags:** | Flag | Short | Type | Description | |------|-------|------|-------------| | `--snapshot` | `-s` | string | Path/URI to snapshot (file path, URL, or cm://namespace/name) | | `--intent` | | string | Workload intent: training, inference | | `--platform` | | string | Explicit platform/framework type, including slurm | | `--profile` | | string | Profile selection in exact `name=value` form; omit to use the declaration's default | | `--slurm-accounting-mode` | | string | Slurm accounting ownership: disabled (default), customer-managed, aicr-provided | | `--runtime-inventory` | | string | Runtime AI inventory (`k8s-aibom`) selection: `enabled` grants the component on a GKE recipe that neither declares nor declines it, confirms it where the recipe already declares it, and is rejected over an explicit recipe decline (any service) or against a non-GKE recipe that does not declare it; `disabled` is unchanged and always requires the recipe to already declare the component. Recorded in the generated recipe | | `--gke-tcpxo-interfaces` | | string | Ordered `eth1=,...,eth8=` GPU-NIC Network mapping for the `torch-distributed-tcpxo` runtime. Required when the resolved recipe ships it (h100 GKE kubeflow training); recorded in the generated recipe | | `--output` | `-o` | string | Output destination (file, ConfigMap URI, or stdout) | | `--format` | `-t` | string | Format: json, yaml, table (default: yaml) | | `--kubeconfig` | `-k` | string | Path to kubeconfig file (used when `--snapshot` or `--output` is a ConfigMap URI; overrides KUBECONFIG env) | **Snapshot Sources:** - **File**: Local file path (`./snapshot.yaml`) - **URL**: HTTP/HTTPS URL (`https://example.com/snapshot.yaml`) - **ConfigMap**: Kubernetes ConfigMap URI (`cm://namespace/configmap-name`) **Examples:** ```shell # Generate recipe from local snapshot file aicr recipe --snapshot system.yaml --intent training # Resolve the Slurm leaf from snapshot-derived infrastructure criteria aicr recipe --snapshot system.yaml --intent training --platform slurm \ --slurm-accounting-mode customer-managed # From ConfigMap (requires cluster access) aicr recipe --snapshot cm://gpu-operator/aicr-snapshot --intent training # From ConfigMap with custom kubeconfig aicr recipe \ --snapshot cm://gpu-operator/aicr-snapshot \ --kubeconfig ~/.kube/prod-cluster \ --intent training # Output to ConfigMap aicr recipe -s system.yaml -o cm://gpu-operator/aicr-recipe # Chain snapshot → recipe with ConfigMaps aicr snapshot -o cm://default/snapshot aicr recipe -s cm://default/snapshot -o cm://default/recipe # With custom output aicr recipe -s system.yaml --intent inference -o recipe.yaml --format yaml ``` **Output structure:** ```yaml apiVersion: aicr.run/v1beta2 kind: RecipeResult metadata: version: v1.0.0 appliedOverlays: - base - monitoring-hpa - gb200-any - eks - eks-ubuntu - eks-training - gb200-eks-training - gb200-eks-ubuntu-training - gb200-eks-ubuntu-training-slurm criteria: service: eks accelerator: gb200 intent: training os: ubuntu platform: slurm configuration: slurm: accounting: mode: disabled componentRefs: - name: gpu-operator type: Helm version: vXX.Y.Z # illustrative; see Component Catalog for current pins source: https://helm.ngc.nvidia.com/nvidia chart: gpu-operator deploymentOrder: - gpu-operator constraints: - name: K8s.server.version value: ">= 1.34" - name: OS.release.ID value: ubuntu - name: OS.release.VERSION_ID value: "24.04" - name: OS.sysctl./proc/sys/kernel/osrelease value: ">= 6.8" ``` A profiled result applies the selected value's constraints and component overrides to the normal recipe fields. It also uses a profile-aware recipe apiVersion and records the selected identity and declaration-wide lock surface: ```yaml apiVersion: aicr.run/v1beta2 kind: RecipeResult metadata: selectedProfile: name: gpuStack value: azure-managed ownedPaths: gpu-operator: - driver.enabled - enabled - operator.runtimeClass - toolkit.enabled nvidia-dra-driver-gpu: - enabled - nvidiaDriverRoot ``` --- ### aicr recipe list Enumerate overlay recipes in the catalog. Useful for discovering which criteria combinations have a dedicated leaf overlay versus an intermediate shared recipe. Each leaf overlay also carries a structural-health verdict (ADR-009 §4): a rolled-up status and a per-phase declared-coverage summary, computed by resolving the recipe and inspecting the result. Intermediate (non-leaf) overlays are not scored — only leaf overlays resolve to a concrete combination. **Synopsis:** ```shell aicr recipe list [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--service` | | string | | Filter by Kubernetes service type (e.g. eks, gke, aks) | | `--accelerator` | `--gpu` | string | | Filter by accelerator/GPU type (e.g. h100, gb200) | | `--intent` | | string | | Filter by workload intent (e.g. training, inference) | | `--os` | | string | | Filter by worker-node OS (e.g. ubuntu, rhel, cos) | | `--platform` | | string | | Filter by platform/framework type (e.g. dynamo, kubeflow, nim) | | `--format` | `-t` | string | table | Output format: json, yaml, table | | `--data` | | string | | External data directory to include alongside embedded overlays | | `--no-health` | `--skip-health` | bool | false | Skip per-leaf structural-health computation; render only the enumeration columns/fields | Filter flags narrow the output to overlays whose criteria carry that exact value. Unspecified flags match all overlays for that dimension. Multiple filters are combined with AND. **Output fields:** | Field | Description | |-------|-------------| | `name` | Overlay name (e.g. `h100-eks-ubuntu-training`) | | `criteria` | The full criteria dimensions the overlay targets | | `is_leaf` | `true` when the overlay is a leaf — no other overlay inherits from it | | `source` | Data provenance: `embedded` (built-in) or `external` (from `--data`) | | `profile` | Effective inherited profile summary (`name`, `description`, `default`, sorted `values`); structured output only and omitted when none is declared | | `health.status` | Rolled-up structural verdict for the leaf overlay: `pass`, `warn`, `fail`, or `unknown` (omitted for non-leaf overlays) | | `health.dimensions` | Per-dimension status map (e.g. `resolves`, `chart_pinned`) feeding the rollup | | `health.coverage` | Declared validation coverage per phase (`readiness`, `deployment`, `performance`, `conformance`): the named checks and phase-level constraint count each declares | In `table` format the health axis is rendered as two extra columns: `STATUS` (the rolled-up verdict) and `COVERAGE`, a compact per-phase named-check summary of the form `R:2 D:4 P:1 C:10` (Readiness / Deployment / Performance / Conformance). Non-leaf overlays render `-` in both columns. A dimension whose grader cannot reach a confident verdict surfaces as `unknown`, and the status column still renders. Resolving every leaf overlay to compute its health verdict adds latency on each invocation. For purely interactive "what overlays exist?" lookups — or scripted callers that only consume `name`/`criteria`/`is_leaf`/`source` — pass `--no-health` (alias `--skip-health`) to skip the health computation entirely. With the flag set the `table` format omits the `STATUS`/`COVERAGE` columns and the `json`/`yaml` formats omit the `health` block, leaving only the enumeration columns/fields. **Examples:** ```shell # List all overlays as a table (default) aicr recipe list # List all overlays as JSON aicr recipe list --format json # Filter to EKS training overlays aicr recipe list --service eks --intent training # Filter to H100 overlays and emit JSON aicr recipe list --accelerator h100 --format json # Include external overlays from a custom data directory aicr recipe list --data /etc/aicr/custom-recipes --format yaml # Skip the structural-health computation for a faster enumeration-only listing aicr recipe list --no-health ``` **Example table output:** Intermediate (non-leaf) overlays render `-` in the `STATUS`/`COVERAGE` columns; only leaf overlays are scored. ```text NAME SERVICE ACCELERATOR INTENT OS PLATFORM IS_LEAF STATUS COVERAGE SOURCE gb200-eks-training eks gb200 training any any false - - embedded gb200-any any gb200 any any any true pass R:0 D:4 P:0 C:0 embedded ``` **Example JSON output:** The `criteria` keys are lowercase. Unset dimensions are omitted. The `health` block is present only for leaf overlays — non-leaf overlays omit it. ```json [ { "name": "gb200-any", "criteria": {"service": "any", "accelerator": "gb200"}, "is_leaf": true, "source": "embedded", "health": { "status": "pass", "dimensions": {"resolves": "pass", "chart_pinned": "pass"}, "coverage": { "readiness": {"declared": false, "constraints": 0}, "deployment": { "declared": true, "checks": ["check-nvidia-smi", "expected-resources", "gpu-operator-version", "operator-health"], "constraints": 1 }, "performance": {"declared": false, "constraints": 0}, "conformance": {"declared": false, "constraints": 0} } } } ] ``` --- ### aicr recipe verify-catalog Verify the embedded recipe catalog (`registry.yaml` + `validators/catalog.yaml`) against its Sigstore bundle. The bundle is distributed as `recipe-catalog.sigstore.json` release asset alongside each tagged `aicr` binary. `aicr recipe verify-catalog` recomputes a deterministic SHA-256 over the embedded catalog content using a length-prefixed encoding of the two raw files, then verifies the digest against the Sigstore bundle using NVIDIA CI identity pinning. Exit code is 0 on success, non-zero on any verification failure. **Synopsis:** ```shell aicr recipe verify-catalog [flags] ``` **Flags:** | Flag | Type | Default | Description | |------|------|---------|-------------| | `--identity-pattern` | string | | Override the NVIDIA CI certificate identity regexp. Must *begin with* `https://github.com/NVIDIA/aicr/` (a leading `^` is allowed; `github\.com` also accepted) and must not use top-level alternation, so the pattern stays confined to the repository. Put any alternatives after the prefix, e.g. `.../aicr/\.github/workflows/(on-tag\|release)\.yaml@.*`. Also reads `AICR_CATALOG_IDENTITY_PATTERN`. | **Examples:** ```shell # Download the catalog signature for a tagged release and verify it. curl -Lo recipe-catalog.sigstore.json \ https://github.com/NVIDIA/aicr/releases/download/vX.Y.Z/recipe-catalog.sigstore.json aicr recipe verify-catalog recipe-catalog.sigstore.json ``` **Output:** On success, prints the verified content digest and the Fulcio certificate identity (the GitHub Actions workflow that signed the release): ```text catalog verified digest: sha256: identity: https://github.com/NVIDIA/aicr/.github/workflows/on-tag.yaml@refs/tags/vX.Y.Z ``` --- ### aicr query Query a specific value from the fully hydrated recipe configuration. Resolves a recipe from criteria (same as `aicr recipe`), merges all base, overlay, and inline value overrides, then extracts the value at the given dot-path selector. Because `aicr query` resolves criteria the same way `aicr recipe` does, it is subject to the same coverage post-condition: a stated `service`/`accelerator`/`intent`/`os`/`platform` value that no overlay honors fails the query with an actionable error rather than silently extracting a value from an unrelated recipe. `--nodes` remains advisory and is never required to be covered. **Synopsis:** ```shell aicr query --selector [flags] ``` **Flags:** All `aicr recipe` flags except `--output` are supported (query always prints to stdout), including `--profile` and `--inherit-from`, plus: | Flag | Type | Description | |------|------|-------------| | `--selector` | string | **Required.** Dot-path to the configuration value to extract | #### Selector Syntax Uses dot-delimited paths consistent with Helm `--set` and `yq`: | Selector | Returns | |----------|---------| | `components..values.` | Hydrated Helm value (scalar or subtree) | | `components..chart` | Component metadata field | | `components.` | Entire hydrated component block | | `criteria.` | Recipe criteria field | | `metadata.selectedProfile` | Selected profile identity and lock surface; present only on profiled recipes | | `deploymentOrder` | Component deployment order list | | `constraints` | Merged constraint list | | `.` or empty | Entire hydrated recipe | Leading dots are optional (yq-style): `.components.gpu-operator.chart` and `components.gpu-operator.chart` are equivalent. **Output:** - **Scalar values** (string, number, bool) are printed as plain text — no YAML wrapper - **Complex values** (maps, lists) are printed as YAML (default) or JSON (`--format json`) **Examples:** ```shell # Get a specific Helm value aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator.values.driver.version # stdout: # illustrative; actual value comes from the resolved recipe # Get a value subtree aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator.values.driver # stdout (excerpt): # enabled: true # version: # Get the full hydrated component aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator # Get deployment order aicr query --service eks --accelerator h100 --intent training \ --selector deploymentOrder # Use in shell scripts VERSION=$(aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator.values.driver.version) echo "Driver version: $VERSION" # JSON output for complex values aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator.values --format json # Query from snapshot aicr query --snapshot snapshot.yaml \ --selector components.gpu-operator.values.driver.version # Full hydrated recipe aicr query --service eks --accelerator h100 --intent training --selector . ``` **Advanced Examples:** ```shell # Cross-cloud comparison: Prometheus storage across providers # EKS provisions a 50Gi persistent EBS volume (gp3, via the cluster-default ebs-csi-default-sc) aicr query --service eks --intent training \ --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec # GKE also provisions a 50Gi persistent volume, but only under the COS overlay — # GKE has no OS-agnostic recipe, so os=cos must be stated explicitly aicr query --service gke --os cos --intent training \ --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec # Compare deployment order across clouds # EKS deploys 15 components (includes aws-ebs-csi-driver, aws-efa, nodewright-customizations) aicr query --service eks --accelerator h100 --intent training --selector deploymentOrder # GKE (COS) deploys 15 components (includes gke-nccl-tcpxo and gcp-driver-installer; storage is otherwise platform-managed) aicr query --service gke --os cos --accelerator h100 --intent training --selector deploymentOrder # Pin the exact driver version into Terraform/Pulumi variables DRIVER_VERSION=$(aicr query --service eks --accelerator h100 --intent training \ --selector components.gpu-operator.values.driver.version) echo "gpu_driver_version = \"${DRIVER_VERSION}\"" # Compare nodewright tuning parameters across accelerators # H100: real tuning packages (kernel setup, nvidia-tuned, full setup) aicr query --service eks --accelerator h100 --intent training \ --selector components.nodewright-customizations.values # GB200: same manifest and value structure; the accelerator/intent overrides # select the gb200 profile instead aicr query --service eks --accelerator gb200 --intent training \ --selector components.nodewright-customizations.values # GB300 on EKS: same shared setup-and-tuning manifest as gb200 aicr query --service eks --accelerator gb300 --intent training \ --selector components.nodewright-customizations.values # Watch constraints tighten as you add specificity # Just "EKS" → 1 constraint (K8s >= 1.32) aicr query --service eks --selector constraints # Add GPU + intent + OS → 4 constraints (K8s >= 1.32.4, Ubuntu 24.04, kernel >= 6.8) aicr query --service eks --accelerator h100 --intent training --os ubuntu \ --selector constraints ``` --- ### aicr validate Validate a system snapshot against the constraints defined in a recipe to verify cluster compatibility. Supports multi-phase validation with different validation stages. For a task-oriented walkthrough (capture snapshot → generate recipe → run each phase, with worked training and inference examples), see [Validation](/aicr/user-guide/validation). **Synopsis:** ```shell aicr validate [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--recipe` | `-r` | string | (required) | Path/URI to recipe file containing constraints (or via `spec.validate.input.recipe` in `--config`) | | `--snapshot` | `-s` | string | | Path/URI to snapshot file containing measurements (omit to capture live) | | `--config` | | string | | Path or HTTP/HTTPS URL to an AICRConfig file (YAML/JSON). CLI flags override values from this file. See [Validate Config File Mode](#validate-config-file-mode). | | `--phase` | | string[] | all | Validation phase to run: deployment, performance, conformance, all (repeatable) | | `--skip-check` | | string[] | | Check to withhold from every phase that runs, one level below `--phase` (repeatable). For a caller that cannot satisfy a check the recipe declares, e.g. a lane deploying a subset of the recipe. Each named check is **reported as skipped**, not dropped, so the CTRF report and the recipe-evidence bundle still account for it. Rejected before any validation resource is created when a name matches no check, when the list would leave a requested phase with nothing to run, or when it is combined with `--evidence-dir` (the CNCF renderer omits skipped checks, so a submission would silently lose the requirement). A `cm://` recipe is read from the cluster first, so that form contacts the API server before the list is judged. Mirrors `spec.validate.execution.skipChecks`. | | `--fail-on-error` | | bool | true | Exit with non-zero status if any phase check reports `failed` or `other` (crash/OOM/timeout). Scopes to phase checks only — the readiness pre-flight always fails closed with exit 2 regardless of this flag (see the readiness note under [Validation Phases](#validation-phases)). | | `--fail-fast` | | bool | false | Stop after the first phase that fails. By default all phases run and produce results. | | `--output` | `-o` | string | stdout | Output destination: file path, ConfigMap URI (`cm://namespace/name`), or stdout | | `--kubeconfig` | `-k` | string | ~/.kube/config | Path to kubeconfig file selecting the target cluster for **every** Kubernetes operation in the invocation: `cm://` recipe/snapshot/output I/O, snapshot-agent deployment, validation namespace and RBAC, validator Jobs, and cleanup. One invocation targets one cluster. When omitted, default discovery applies (`KUBECONFIG` env, then `~/.kube/config`, then in-cluster). An invalid path fails any run that performs Kubernetes operations; `--no-cluster` dry-runs over local files do not load it. **Changed in v0.18:** validator-engine operations, including validator Jobs, now honor this flag instead of using the default cluster. | | `--namespace` | `-n` | string | aicr-validation | Kubernetes namespace for validation Job deployment. Reads `AICR_NAMESPACE` when unset. | | `--image` | | string | matches CLI version | Container image for the **snapshot-capture agent** Job, not the validator Jobs. Release builds default to `ghcr.io/nvidia/aicr:v`; dev and `-next` builds default to `ghcr.io/nvidia/aicr:latest`. Reads `AICR_VALIDATOR_IMAGE` when unset. Validator catalog images are resolved separately — override them with `AICR_VALIDATOR_IMAGE_TAG` / `AICR_VALIDATOR_IMAGE_REGISTRY`. | | `--image-pull-secret` | | string[] | | Image pull secrets for private registries (repeatable) | | `--job-name` | | string | aicr-validate | Prefix for the **live snapshot-capture agent's** Job name; the run ID is always appended (`-`). Inert when `--snapshot` is supplied — no agent is deployed. Does not name the validator Jobs (`aicr--`) | | `--service-account-name` | | string | aicr | ServiceAccount the **live snapshot-capture agent** runs as. Leaving the flag unset is not the same as passing that default: an unset value is never probed, and the agent's run-scoped names are derived from the `aicr-validate` base instead. **Exact-if-exists:** an existing ServiceAccount of exactly this name in `--namespace` is used verbatim and the agent creates no RBAC for the run; otherwise the value is a prefix for the agent's ServiceAccount, Role, and RoleBinding and the run ID is appended (`-`). Inert when `--snapshot` is supplied. Does not name the validator Jobs' ServiceAccount (`aicr-validator-`), whose RBAC is always run-scoped. Generate that ServiceAccount's RBAC manifests with `aicr snapshot --namespace --add-roles-to-service-account ` (matching this command's `--namespace`) and apply them yourself — that command applies nothing | | `--node-selector` | | string[] | | Override GPU node selection for the live snapshot agent (when `--snapshot` is omitted) and inner validation workloads. Replaces platform-specific selectors (e.g., `cloud.google.com/gke-accelerator`, `node.kubernetes.io/instance-type`) on inner workloads like NCCL benchmark pods. Use when GPU nodes have non-standard labels. Does not affect the validator orchestrator Job. (format: key=value, repeatable) | | `--toleration` | | string[] | | Override tolerations for the live snapshot agent (when `--snapshot` is omitted) and inner validation workloads. When omitted, the snapshot agent tolerates all taints. Does not affect the validator orchestrator Job. (format: key=value:effect, repeatable) | | `--timeout` | | duration | 5m | Timeout for the **live snapshot-capture agent** Job to complete (inert when `--snapshot` is supplied). Validator Jobs use their own per-check timeouts | | `--no-cleanup` | | bool | false | Skip removal of Job and RBAC resources on completion | | `--require-gpu` | | bool | false | Request an `nvidia.com/gpu` resource for the **live snapshot-capture agent** pod. Reads `AICR_REQUIRE_GPU` when unset | | `--oke-addons` | | string | | Path to an `oci ce cluster list-addons --cluster-id --all --output json` dump on the local filesystem, projected into the `K8s.oke-addons.nvidia-gpu-plugin` reading when validate captures a live snapshot. Ignored when `--snapshot` supplies a pre-captured snapshot — capture that snapshot with the same flag instead. Reads `AICR_OKE_ADDONS_PATH` env when unset. | | `--aks-gpu-pools` | | string | | Path to an `az aks nodepool list -o json` dump on the local filesystem, projected into the `K8s.aks-gpu-pools.gpu-driver` reading when validate captures a live snapshot (ADR-015 DD3). Ignored when `--snapshot` supplies a pre-captured snapshot — capture that snapshot with the same flag instead. Reads `AICR_AKS_GPU_POOLS_PATH` env when unset. | | `--gke-gpu-pools` | | string | | Path to a `gcloud container node-pools list --cluster --format=json` dump on the local filesystem, projected into the `K8s.gke-gpu-pools.gpu-driver-installation` reading when validate captures a live snapshot. Ignored when `--snapshot` supplies a pre-captured snapshot. Capture that snapshot with the same flag instead. Reads `AICR_GKE_GPU_POOLS_PATH` env when unset. | | `--no-cluster` | | bool | false | Skip cluster access (test mode): skips RBAC and Job deployment, reports checks as skipped. A `cm://` `--recipe` or `--snapshot` is still read through the Kubernetes API, so only file inputs make the run fully offline. An offline dry-run does not sign or push a recipe-evidence attestation: combining it with `--emit-attestation` or `--push` is rejected with `INVALID_REQUEST` (exit 2), and a config-supplied `spec.validate.evidence.attestation` is skipped with a warning. Cannot be combined with `--cncf-submission` (that collector requires a live cluster); `--evidence-dir` conformance markdown is still rendered locally | | `--evidence-dir` | | string | | Directory to write conformance evidence artifacts | | `--cncf-submission` | | bool | false | Generate CNCF conformance submission artifacts | | `--feature` | `-f` | string[] | | CNCF evidence-collection feature(s) to scope (repeatable). Valid names: `dra-support`, `gang-scheduling`, `secure-access`, `accelerator-metrics`, `ai-service-metrics`, `inference-gateway`, `robust-operator`, `pod-autoscaling`, `cluster-autoscaling`. Empty selects all features. | | `--emit-attestation` | | string | | Directory to write a recipe-evidence attestation bundle, predicateType v3, with a content-only recipe digest for the recipe content resolved by the selected profile. The digest excludes `metadata.version`, so it is independent of the aicr binary version that produced the recipe (signed when `--push` is set, unless `--no-sign`). The bundle is minimized by default, see `--full`. See [ADR-007](https://github.com/NVIDIA/aicr/blob/main/docs/design/007-recipe-evidence.md). | | `--full` | | bool | false | Emit the full (unredacted) evidence bundle. By default the bundle is minimized: `snapshot.yaml` is reduced to an allowlisted set of fields (dropping node names, provider instance IDs, the node label/taint set, OS tuning, loaded modules, systemd config) and per-test CTRF `stdout`/`message` are omitted. `--full` ships the raw payloads. The cryptographic verification story holds either way; minimal bundles record the applied policy in `predicate.redaction` and self-verify with `aicr evidence verify`. | | `--bom` | | string | | Path to a CycloneDX BOM (`bom.cdx.json`) to embed. Optional with `--emit-attestation`; when omitted, aicr synthesizes a recipe-bound BOM from the recipe's component refs + validator catalog images. Pass `make bom`'s output for an exhaustive BOM. | | `--push` | | string | | OCI registry reference to push the signed summary bundle to. Triggers Sigstore keyless signing via the precedence chain documented under `--identity-token`. The `sha256:` digest is the canonical address, so the tag is only a human-readable label — tag choice never affects verification. Omit the tag and aicr derives a unique per-recipe one, `-` (e.g. `ghcr.io/myorg/aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e`), so distinct attestations never collide on a shared tag. Pass an explicit tag to override. | | `--no-sign` | | bool | false | Push the evidence bundle **unsigned** (requires `--emit-attestation` and `--push`) and write a `pointer.yaml` with an empty `signer` block. No-op unless both `--emit-attestation` and `--push` are set. Commit the flat unsigned pointer; the fork CI signing leg signs it and relocates it to its nested per-source path (or sign locally with `aicr evidence sign --relocate` on a Sigstore-reachable host). The blocking *Evidence Pointer Contract* gate requires the final committed pointer to be **signed and nested** — see [Publishing Recipe Evidence](/aicr/contributor-guide/publishing-recipe-evidence#recommended-path-split-the-legs-sign-in-ci). | | `--allow-mutable-validator-tags` | | bool | false | Emit the attestation even when a validator image resolves to a mutable tag. Emission otherwise fails closed with `INVALID_REQUEST`: the predicate identifies the validators that ran by tag alone (`validatorImages[].digest` is empty by design), so a moving tag such as `:edge` or `:latest` leaves the bundle naming a reference that can later resolve to different validator code. A reference qualifies as immutable only if it is digest-pinned (`name@sha256:...`, any registry), or carries a CI-frozen tag — `:vX.Y.Z` and pre-releases, `:sha-`, `:uat-` — **under `ghcr.io/nvidia/aicr-validators/`**, the only namespace whose tag conventions AICR's workflows enforce. Tag syntax alone is not proof: a third-party or mirrored registry may repoint its own `:v1.0.0` freely, so images from one must be digest-pinned. The usual cause is a stale `AICR_VALIDATOR_IMAGE_TAG` override — unset it rather than reaching for this flag. Has no env var or config equivalent by design: the opt-out must be passed per invocation so it cannot persist into a later run. See [Validator image provenance](/aicr/contributor-guide/publishing-recipe-evidence#validator-image-provenance). | | `--plain-http` | | bool | false | Use HTTP instead of HTTPS for evidence push (local registry tests). | | `--insecure-tls` | | bool | false | Skip TLS verification for evidence push (self-signed registries). | | `--identity-token` | | string | | Pre-fetched OIDC identity token for `--push` keyless signing. Skips ambient/browser/device-code flows. Reads `COSIGN_IDENTITY_TOKEN` from env. Same precedence chain as `aicr bundle --attest`. | | `--oidc-device-flow` | | bool | false | Use the OAuth 2.0 device authorization grant for `--push` OIDC instead of opening a browser callback. Reads `AICR_OIDC_DEVICE_FLOW`. | | `--yes` | `--assume-yes` | bool | false | Skip the interactive confirmation shown before keyless signing publishes your OIDC identity (browser/device-code paths only; the banner is still printed). Reads `AICR_ASSUME_YES`. See [Privacy: identity in keyless signatures](#privacy-identity-in-keyless-signatures). | | `--data` | | string | | External data directory to overlay on embedded data | **Input Sources:** - **File**: Local file path (`./recipe.yaml`, `./snapshot.yaml`) - **URL**: HTTP/HTTPS URL (`https://example.com/recipe.yaml`) - **ConfigMap**: Kubernetes ConfigMap URI (`cm://namespace/configmap-name`) #### Validation Phases Validation can be run in different phases to validate different aspects of the deployment: | Phase | Description | When to Run | |-------|-------------|-------------| | `deployment` | Validates component deployment completeness plus post-install GPU readiness signals (see below) | After deploying components | | `performance` | Validates system performance and network fabric health | After components are running | | `conformance` | Validates workload-specific requirements and conformance | Before running production workloads | | `all` | Runs all phases sequentially; results collected regardless of failures | Complete end-to-end validation | > **Note:** Readiness constraints (K8s version, OS, kernel) are always evaluated implicitly before any phase runs. If readiness fails, validation stops before deploying any Jobs and exits 2 (`INVALID_REQUEST`). This gate always fails closed — `--fail-on-error=false` scopes to phase check results and does not downgrade a readiness failure. > > **Declared-check pre-flight:** Every check named under a phase's `checks` list must resolve to exactly one catalog validator in that phase. Before any Job is deployed, `validate` fails closed with exit 2 (`INVALID_REQUEST`) if a declared check matches no validator (a typo, or a check missing from the loaded `--data` catalog), exists only under a different phase, or is declared more than once — reporting every offender at once. This runs in `--no-cluster` mode too, and like readiness it is independent of `--fail-on-error`. It replaces the previous warn-and-continue behavior, which let a phase with only unresolved checks report `skipped` and exit `0`. > > **Version skew:** Snapshots and recipes record the `aicr` version that produced them. When the recipe, the snapshot, and the running binary report different release versions, `validate` logs a single advisory warning (`version skew detected across validate inputs`) naming all three. This is a debugging breadcrumb — mixing artifacts from different versions can surface as confusing failures — and does **not** fail the command. Dev (`dev`) and pre-release (`-next`) builds are ignored to avoid noise. > > **apiVersion gate:** AICR emits `aicr.run/v1` for snapshots and default > recipes, `aicr.run/v1beta1` for config and ordinary catalog inputs, and > `aicr.run/v1beta2` for profile-bearing recipes, and as of v1.0.0 those are the > only values it reads. The superseded `aicr.run/v1alpha2` and > `aicr.run/v1alpha3` were retired in v1.0.0 (ADR-022 N+2), along with the empty > header the snapshot, recipe and criteria readers had tolerated. v0.22 was the > last release that read them, and it warned; v1.0.0 rejects instead, naming the > observed value, the expected value and the release that withdrew it. > Unsupported artifact headers fail fast; raw external catalog headers are > checked before merge or hydration. Recapture, regenerate, or update the > authored header with a version supported by the running AICR release. See > [ADR-011](https://github.com/NVIDIA/aicr/blob/main/docs/design/011-artifact-apiversion-policy.md) > and > [ADR-022](https://github.com/NVIDIA/aicr/blob/main/docs/design/022-artifact-maturity-and-deprecation.md). > [Catalog and binary compatibility](/aicr/integrator-guide/data-extension#catalog-and-binary-compatibility) > has the release-by-release table. Phases run sequentially with `--phase all` and all phases run by default, producing results regardless of earlier failures; use `--fail-fast` to stop after the first failing phase. For what each phase actually checks (deployment-phase readiness signals, graceful-skip semantics, RBAC, Day-N re-verification, and evidence), see [Validation](/aicr/user-guide/validation). Within a phase, `--skip-check` withholds individual checks. It is for a caller that cannot satisfy a check the recipe declares (a lane that deploys only part of the recipe, or runs on simulated devices), and it narrows the *run*, never the recipe, so every other consumer of that recipe still gets the check. Two guards apply before any cluster work: a name matching no check in the catalog fails the run, and a list that would leave a requested phase with nothing to run fails it too (that phase would otherwise report `passed` while running nothing, because the skipped entries keep its test count above zero). A skipped check appears in the CTRF report as `skipped` with its reason in `message` and as the code `extra.skipReason: named-in-skip-checks`. The default (minimal) recipe-evidence bundle (`--emit-attestation`) carries the report redacted rather than verbatim (every test's `message` and `stdout` is blanked), so it is the `extra` code that carries the reason into the attestation, unless `--full` is passed. The CNCF conformance evidence path does NOT: its renderer drops skipped entries, so a withheld requirement would leave no file and no index entry, and `--skip-check` is therefore refused together with `--evidence-dir` (and so with `--cncf-submission`, which requires it). A check that is *added* to a recipe later is not silenced by an existing list: it runs, which is the direction that forces a decision rather than hiding one. #### Constraint paths and operators Constraints use fully qualified measurement paths: `{Type}.{Subtype}.{Key}` | Constraint Path | Description | |-----------------|-------------| | `K8s.server.version` | Kubernetes server version | | `OS.release.ID` | Operating system identifier (ubuntu, rhel) | | `OS.release.VERSION_ID` | OS version (24.04, 22.04) | | `OS.sysctl./proc/sys/kernel/osrelease` | Kernel version | | `GPU.hardware.model` | GPU model (e.g. `h100`, `l40s`) | Constraint paths are validated against the measurement catalog when recipe data is loaded, so a typo such as `K8s.server.verison` fails immediately — naming the file, the field, and the nearest matching key — instead of silently evaluating as a missing reading. See [Recipe Development › Constraints](/aicr/integrator-guide/recipe-development) for the addressing rules. Supported operators: | Operator | Example | Description | |----------|---------|-------------| | `>=` | `>= 1.32` | Greater than or equal (version comparison) | | `<=` | `<= 1.33` | Less than or equal (version comparison) | | `>` | `> 1.30` | Greater than (version comparison) | | `<` | `< 2.0` | Less than (version comparison) | | `==` | `== ubuntu` | Explicit equality | | `!=` | `!= rhel` | Not equal | | (none) | `ubuntu` | Exact string match | **Examples:** ```shell # Validate snapshot against recipe (readiness constraints run implicitly) aicr validate --recipe recipe.yaml --snapshot snapshot.yaml # Validate specific phase aicr validate \ --recipe recipe.yaml \ --snapshot snapshot.yaml \ --phase deployment # Run all validation phases aicr validate \ --recipe recipe.yaml \ --snapshot snapshot.yaml \ --phase all # Load snapshot from ConfigMap aicr validate \ --recipe recipe.yaml \ --snapshot cm://gpu-operator/aicr-snapshot # Save results to file aicr validate \ --recipe recipe.yaml \ --snapshot cm://gpu-operator/aicr-snapshot \ --output validation-results.json # Validate deployment phase after components are installed; the bundle's # recipe.yaml records the component set it deployed aicr validate \ --recipe ./bundles/recipe.yaml \ --snapshot snapshot.yaml \ --phase deployment # Run performance validation aicr validate \ --recipe recipe.yaml \ --snapshot snapshot.yaml \ --phase performance # With custom kubeconfig — selects the cluster for the whole run: # cm:// I/O, agent deployment, validator Jobs, and cleanup (#1787) aicr validate \ --recipe recipe.yaml \ --snapshot cm://gpu-operator/aicr-snapshot \ --kubeconfig ~/.kube/prod-cluster # Write a recipe-evidence attestation bundle (unsigned, on disk). # --bom is optional: when omitted, aicr synthesizes a recipe-bound BOM from # the recipe's component refs and validator catalog images. aicr validate \ --recipe recipe.yaml --snapshot snapshot.yaml \ --emit-attestation ./out # Writes ./out/summary-bundle/ and ./out/pointer.yaml. # The bundle is minimized by default: sensitive snapshot fields and CTRF # logs are removed, and predicate.redaction records the applied policy. # Ship the full (unredacted) bundle instead — raw snapshot + CTRF stdout. aicr validate \ --recipe recipe.yaml --snapshot snapshot.yaml \ --emit-attestation ./out --full # Use an exhaustive BOM (e.g., `make bom`-produced) instead of the auto-generated one aicr validate \ --recipe recipe.yaml --snapshot snapshot.yaml \ --emit-attestation ./out --bom dist/bom/bom.cdx.json # Sign and push a recipe-evidence bundle to OCI (cosign keyless via Sigstore public-good). # Token acquisition follows the same precedence chain as `aicr bundle --attest`: # pre-fetched COSIGN_IDENTITY_TOKEN > ambient GitHub Actions OIDC > --oidc-device-flow > interactive browser. aicr validate \ --recipe recipe.yaml --snapshot snapshot.yaml \ --emit-attestation ./out \ --push ghcr.io/myorg/aicr-evidence # tag optional; aicr derives :- # After this, copy ./out/pointer.yaml to the per-source path printed in the # 'copyTo' hint: recipes/evidence///.yaml # NOTE: keyless --push signing publishes the signer's identity (email + issuer) # to the public Rekor log. On a TTY, aicr pauses for confirmation first (--yes skips it). # Validate on a cluster with custom GPU node labels (non-standard labels that AICR doesn't # recognize by default, e.g., using a custom node pool label instead of cloud-provider defaults) aicr validate \ --recipe recipe.yaml \ --node-selector my-org/gpu-pool=true \ --phase performance # Override both node selector and tolerations for a non-standard taint setup aicr validate \ --recipe recipe.yaml \ --node-selector gpu-type=h100 \ --toleration gpu-type=h100:NoSchedule ``` #### Validate Config File Mode `aicr validate --config ` reads inputs from an AICRConfig YAML/JSON file under `spec.validate`. CLI flags always override values loaded from `--config`; override events are logged at INFO so users can see which input won. The OIDC identity token used for `--push` signing stays out of the schema by design (short-lived tokens must not be committed); the CLI resolves it at sign time through the precedence chain described on `--identity-token`. **Supported schema:** ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 metadata: name: prod-validate spec: validate: input: recipe: ./recipe.yaml snapshot: ./snapshot.yaml # optional; omit to capture live agent: # only used when input.snapshot is empty namespace: aicr-validation image: ghcr.io/nvidia/aicr:vX.Y.Z # default: matches the CLI version imagePullSecrets: [registry-secret] # Optional prefixes for the live-capture agent, not names — the run ID # is always appended. Omitted here so the defaults apply # (both aicr-validate). nodeSelector: my-org/gpu-pool: "true" tolerations: # [] clears the live snapshot agent's tolerate-all default - "gpu-type=h100:NoSchedule" requireGpu: true execution: phases: [deployment, conformance] # skipChecks: # --skip-check; withheld and reported as skipped. # - gpu-operator-health # Shown commented out because it cannot be combined # # with evidence.cncf.dir below. See the next example. failOnError: true # default; false = don't fail on phase-check results (readiness pre-flight still exits 2) noCluster: false noCleanup: false timeout: 10m # live-capture agent Job only; inert when input.snapshot is set evidence: cncf: # --evidence-dir / --cncf-submission / --feature dir: ./out/cncf cncfSubmission: false features: [] # empty = all features attestation: # --emit-attestation / --bom / --push / ... out: ./out/attestation bom: dist/bom/bom.cdx.json # optional; auto-generated from recipe + validators when absent push: ghcr.io/myorg/aicr-evidence # tag optional; aicr derives :- plainHTTP: false insecureTLS: false ``` **Withholding checks (`execution.skipChecks`):** A lane that deploys only part of a recipe can withhold the checks it cannot satisfy. Each named check is still reported, as skipped, so the run accounts for it; a name matching no check in the recipe's catalog, or a list that would leave a requested phase with nothing to run, is rejected before any validation resource is created. The recipe is loaded first, so a `cm://` recipe is read from the cluster before the list is judged. `skipChecks` cannot be combined with `evidence.cncf.dir`. The CNCF evidence renderer drops skipped checks entirely, so a withheld requirement would produce no file and no index entry and the submission would read as complete. That is why the schema above shows the field commented out, and why it gets its own config here: ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 metadata: name: partial-lane-validate spec: validate: input: recipe: ./recipe.yaml snapshot: ./snapshot.yaml execution: phases: [deployment] skipChecks: - gpu-operator-health ``` **Examples:** ```shell # Use a config file aicr validate --config validate.yaml # Override a single config value from the CLI aicr validate --config validate.yaml --phase deployment # Validate the same recipe across two clusters using two different agent # configs (config-bound) without retyping flags aicr validate --config validate-cluster-a.yaml aicr validate --config validate-cluster-b.yaml ``` The `--node-selector` and `--toleration` flags control scheduling for the inner validation workloads (NCCL benchmark workers, conformance test pods). When `--snapshot` is omitted, they also configure the preliminary live snapshot agent. They do not configure the validator orchestrator Job. For when to use them with non-standard GPU labels or taints, see [Validation](/aicr/user-guide/validation#non-standard-gpu-labels-or-taints). **Output Structure ([CTRF](https://ctrf.io/) JSON):** Results are output in CTRF (Common Test Report Format) — an industry-standard schema for test reporting. ```json { "reportFormat": "CTRF", "specVersion": "0.0.1", "timestamp": "2026-03-10T20:10:44Z", "generatedBy": "aicr", "results": { "tool": { "name": "aicr", "version": "v0.22.0" }, "summary": { "tests": 16, "passed": 13, "failed": 0, "skipped": 3, "pending": 0, "other": 0, "start": 1773173400872, "stop": 1773173799002 }, "tests": [ { "name": "operator-health", "status": "passed", "duration": 0, "suite": ["deployment"], "stdout": ["Found 1 gpu-operator pod(s)", "Running: 1/1"] }, { "name": "expected-resources", "status": "passed", "duration": 0, "suite": ["deployment"], "stdout": ["All deployment resources and required readiness signals are healthy"] }, { "name": "nccl-all-reduce-bw", "status": "passed", "duration": 234000, "suite": ["performance"], "stdout": ["NCCL All Reduce bandwidth: 488.37 GB/s", "Constraint: >= 100 → true"] }, { "name": "inference-perf", "status": "passed", "duration": 612000, "suite": ["performance"], "stdout": [ "RESULT: Inference throughput: 108789.87 tokens/sec", "RESULT: Inference TTFT p99: 687.50 ms", "Throughput constraint: >= 50000 → PASS", "TTFT p99 constraint: <= 2000 → PASS" ] }, { "name": "dra-support", "status": "passed", "duration": 8000, "suite": ["conformance"], "stdout": ["DRA support verified (driver healthy, ResourceSlices validated)"] }, { "name": "cluster-autoscaling", "status": "skipped", "duration": 0, "suite": ["conformance"], "stdout": ["SKIP reason=\"Karpenter not found\""] } ] } } ``` > **Note:** The `tests` array above is truncated for brevity. A full validation run produces one entry per check across all phases. Each entry includes `stdout` with detailed diagnostic output. **Test Statuses:** | Status | Description | |--------|-------------| | `passed` | Check or constraint passed | | `failed` | Check or constraint failed | | `skipped` | Check could not be evaluated (missing data, no-cluster mode) | | `other` | Indeterminate outcome — the check produced no usable verdict (crash, OOM, or a validator Job that failed for a non-deadline reason with no inspectable pod) | **Exit Codes:** | Code | Description | |------|-------------| | `0` | All phases passed or were skipped (also returned under `--fail-on-error=false` even when phases report `failed`/`other`) | | `2` | Invalid input (bad flags, missing recipe), a readiness pre-flight constraint not met, or a declared check that does not resolve to exactly one catalog validator in its phase (unmatched, cross-phase, or duplicate) — these pre-flight gates always fail closed here regardless of `--fail-on-error` | | `5` | A structured timeout reached the top-level CLI (recipe/snapshot load, snapshot-agent wait, evidence signing). A per-validator wait deadline becomes a check result instead and exits `8` — see [validation: CI/CD integration](/aicr/user-guide/validation#cicd-integration) | | `8` | One or more phase checks reported `failed` (including a validator Job killed on its `activeDeadlineSeconds`) or `other` (crash/OOM) — when `--fail-on-error` is set | --- ### aicr diff Compare two snapshots field-by-field to surface configuration drift between cluster states. Reports added, removed, and modified readings across every measurement type (K8s, GPU, OS, SystemD, NodeTopology, NetworkTopology). **Synopsis:** ```shell aicr diff --baseline --target [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--baseline` | `-b` | string | | Baseline snapshot (file path or ConfigMap URI). **Required.** | | `--target` | | string | | Target snapshot (file path or ConfigMap URI). **Required.** | | `--fail-on-drift` | | bool | false | Exit with non-zero status (`ErrCodeConflict`) if any drift is detected. Useful for CI/CD gating. | | `--output` | `-o` | string | stdout | Output destination: file path, ConfigMap URI (`cm://namespace/name`, JSON/YAML only), or stdout. **Note:** ConfigMap destinations are rejected for `--format table` (a structured format is required for ConfigMap storage). | | `--format` | `-t` | string | yaml | Output format: `json`, `yaml`, or `table`. | | `--kubeconfig` | `-k` | string | ~/.kube/config | Path to kubeconfig (used when `--baseline`, `--target`, or `--output` is a ConfigMap URI). | **Inputs:** - File paths (`./baseline.yaml`, `/tmp/snap.json`) - ConfigMap URIs (`cm://gpu-operator/aicr-snapshot`) - Both inputs may mix freely; e.g., a local baseline file vs. a live ConfigMap target. **Output Semantics:** - A nil reading is rendered as the literal `` so it cannot be confused with an empty-string value (`""`). Both forms surface as drift when one side is nil and the other is a concrete value. - Changes are emitted in deterministic order (sorted by `Path`) so the diff is reproducible across runs and machines. - The `Result` envelope includes `baselineSource` and `targetSource` (the supplied paths), a `changes` array, and a `summary` with `added`, `removed`, `modified`, and `total` counts. | Field | Stable path | |-------|-------------| | Existing subtype Data | `..` | | Subtype Context | `..context.` | | Item cardinality | `..items.length` | | Item Context | `..items[].context.` | | Item Data | `..items[].data.` | Items are ordered and compared by zero-based position. Reordering distinct records is drift. A list-length change emits `items.length` and field-level additions or removals. Ordinary existing Data paths remain unchanged for compatibility. Data keys equal to `context` or `items`, or beginning with `context.`, `items.`, or `items[`, use JSON-style bracket quoting so they cannot collide with structured paths; for example, subtype Data key `context.node` is emitted as `.["context.node"]`, and the same key in Item Data is emitted as `..items[].data["context.node"]`. **Examples:** ```shell # Local-file diff in default YAML aicr diff --baseline before.yaml --target after.yaml # Human-readable table to stdout aicr diff -b before.yaml --target after.yaml --format table # CI/CD gate: non-zero exit on drift, JSON to a file aicr diff -b before.yaml --target after.yaml \ --format json --output drift.json --fail-on-drift # Compare two ConfigMaps in the cluster aicr diff \ --baseline cm://gpu-operator/aicr-snapshot-baseline \ --target cm://gpu-operator/aicr-snapshot # Mix file + ConfigMap (golden baseline vs live cluster) aicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot ``` **Exit Codes:** | Code | Description | |------|-------------| | `0` | Diff completed; no drift, or `--fail-on-drift` not set | | `2` | Invalid input (missing flags, bad format, ConfigMap output for `--format table`) **or** drift detected with `--fail-on-drift` (mapped from `ErrCodeConflict`) | > **Note on CI gating:** A non-zero exit identifies *that* drift was detected, but doesn't by itself distinguish drift from malformed input — both map to exit `2`. To differentiate without relying on stderr format (text by default; JSON only with `--log-json`), inspect the diff payload directly: write the result with `--output drift.json --format json` and branch on the presence of the file plus its `summary.total` field. That signal is format-stable regardless of logging mode. --- ### aicr upgrade-check Compare two recipes or bundles component by component and report, for each component whose version or identity (namespace, chart, source, path, deployment type, manifest files, pre-manifest files or object names) changed, whether moving between them is safe to apply. Verdicts come from the [transition records](/aicr/contributor-guide/upgrade-transition-records) the running `aicr` release ships. An artifact comparison inspects no cluster state, which makes it the CI and GitOps path: it reads two artifacts and nothing else. A `cm://` path is an artifact location like a file path, so reading or writing one does contact that cluster's API for the ConfigMap itself. Two opt-in flags do read a cluster: `--from cluster` takes the source side from what the deployers recorded they installed, and `--scan-cluster` adds an advisory pass over live objects the upgrade might disturb. **Synopsis:** ```shell aicr upgrade-check --from [--to ] [--deployer ] [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--from` | `-f` | string | | Source: recipe file, bundle directory, ConfigMap URI, or the literal `cluster` to read the installed inventory instead of an artifact. **Required.** | | `--to` | | string | re-resolve | Target artifact. When omitted, `--from`'s own criteria are re-resolved against this binary's registry. **Required with `--from cluster`**, which carries no criteria to re-resolve. | | `--deployer` | `-d` | string | | Deployer the reported steps are scoped to: `argocd`, `argocd-helm`, `flux`, `helm`, `helmfile`. **Always required with `--from cluster`**; otherwise required whenever any component needs steps. | | `--scan-cluster` | | bool | off (on with `--from cluster`) | Also report live objects the upgrade could disturb that carry no deployer ownership marker. Implied by `--from cluster`; pass `--scan-cluster=false` there to skip the scan. Advisory: findings never change the exit code. | | `--fail-on-error` | | bool | **true** | Exit non-zero when any component needs attention. | | `--output` | `-o` | string | stdout | Output destination: file path, ConfigMap URI (`cm://namespace/name`, JSON/YAML only), or stdout. | | `--format` | `-t` | string | **table** | Output format: `json`, `yaml`, or `table`. | | `--kubeconfig` | `-k` | string | | Kubeconfig used for `cm://` artifact reads, a `cm://` `--output`, `--from cluster`, and `--scan-cluster`. Overrides `KUBECONFIG` and `~/.kube/config`. No cluster is contacted unless one of those is in play. | Note the two defaults that differ from sibling commands. `--format` defaults to `table` rather than `yaml`, because the report's payload is a list of operator steps that folded YAML scalars make unreadable. `--fail-on-error` defaults to **true**, the opposite of `aicr diff --fail-on-drift`: you chose to run this check, so its exit code is what makes running it worth something in a pipeline. **The two questions it answers:** `--from X --to Y` asks *"is this specific move safe?"* and presumes you already know your target. Omitting `--to` asks *"am I behind, and does catching up hurt?"*, which is usually the real question: what an operator holds is an old artifact, not a chosen destination. The `--from` artifact's embedded criteria are re-resolved against the running binary's pins to synthesize the target. **`--from cluster` takes the source side from what is installed.** Instead of a source artifact, the check reads the record the named deployer leaves behind, and only that one: Helm's release records, out of the Kubernetes objects Helm stores them in, for `helm`, `helmfile` and `flux`; Argo CD's `Application` objects for `argocd` and `argocd-helm`, which write no per-component Helm release at all. The `--to` side is still an artifact; there is nothing in a cluster to upgrade *to*. That answers *which version is installed*, authoritatively, including where a cluster has drifted from the recipe in git. It answers nothing else. A version counts only once it is established: a Helm release whose newest revision reached `deployed`, and an `Application` revision its sync status or history shows was synced, never the pin it is configured to reach. A Helm upgrade that is pending or failed reads as `unversioned` rather than at its target, since the old version may still be what runs; an `Application` that has not finished a sync reads at its last completed one, or as `unversioned` if there is none. A component the read places but cannot version, such as one whose release record cannot be decoded, reads as `unversioned` too rather than dropping out, where it would read as newly installed. Both sources are records of what was applied rather than observations, so a resource somebody edited by hand leaves both untouched and reading them will not say so. It is not a view of live cluster state. Two flags stop being optional: - `--to`, because a cluster carries no criteria to re-resolve, so the single-argument "am I behind?" form has nothing to work from. - `--deployer`, whether or not any component turns out to carry steps, and checked before any cluster I/O. A release name encodes the deployer that wrote it (`flux` composes `-`, Argo CD prepends a prefix the user sets), so without one no installed release maps to a component and the read could only report an empty cluster. `cluster` is a sentinel value rather than a path. A directory of that name is reached as `./cluster`. The report gains a `READ FROM CLUSTER` block above the rows, naming the kubeconfig that was read and accounting for each reader separately. Its `context` line reads `-`: the current context is not yet reported, so confirm it with `kubectl config current-context` against the same kubeconfig before trusting the rows. The two sets of counts are never summed: the Helm side counts storage records, so one release with ten retained revisions contributes ten, while the Argo side counts `Application` objects, of which a component has one. Only an `Application` that deploys into the cluster being read counts as installed. One addressed to another cluster is excluded and counted as `remote`, so on an Argo CD management cluster a remote install never becomes the local baseline. A destination given by name is resolved through Argo CD's cluster Secrets, and one that cannot be resolved is refused rather than guessed. A component need not sit in its registry default namespace to be found. The read also tries the namespace the `--to` recipe resolves it into, which an overlay, a mixin or `--inherit-from` can change, and under `flux` the namespace the release itself records. AICR-stamped releases that still match no component are counted as `stamped but unmatched`, and any such count fails a strict run: the read has lost track of something AICR installed, and the component it belongs to would otherwise read as newly installed. **Permissions.** A cluster read lists at cluster scope, so it needs `list` on `secrets` and `configmaps` under `helm`, `helmfile` and `flux`, and `list` on `applications.argoproj.io` under `argocd` and `argocd-helm`, plus `list` on `secrets` when an `Application` names its destination cluster rather than addressing it. The at-risk scan additionally needs API discovery and `list` on each kind it scans. Nothing is written. `--format json` and `--format yaml` carry the same thing as `source`. A read that recognizes nothing is reported, never failed. Every row then reads "added", and the block says so in as many words, because from the rows alone a cluster with none of these components installed looks identical to the two likelier causes: the wrong cluster, or components installed by a deployer other than the one you named. **A cluster read reports no namespace move.** The identity comparison below is artifact-to-artifact only. The read recovers a version and no namespace, and that is not an omission it could fill: attribution composes the release name *from* the registry's namespace, so a namespace is an input to the read rather than a fact recovered from it, and only `helm` and `helmfile` match on the component's bare name at all. A namespace stated here would come from two deployers of five and be silently absent under the other three, which is a worse report than none. **`--scan-cluster` warns about objects the upgrade could destroy.** It is an axis of its own rather than a property of `--from`: the scan needs a cluster wherever the `from` table came from, so comparing two bundles while scanning a live cluster is a legitimate and useful combination. `--from cluster` implies it, and an explicit `--scan-cluster=false` wins over that implication. For the group and kind pairs the *crossed* transition records name, the scan lists each kind cluster-wide and reports every object carrying neither Helm ownership (the `app.kubernetes.io/managed-by=Helm` label together with a `meta.helm.sh/release-name` annotation, because the label alone is written by anything) nor Argo CD's `argocd.argoproj.io/tracking-id` annotation. Only crossed records contribute, because an upgrade nobody is making cannot put anything at risk. That test is positive, so an object carrying no marker AICR recognizes is reported by default. The direction is deliberate: the failure being guarded against is an operator's own custom resources being cascade-deleted when a component removes a CRD, and AICR cannot restore what it does not own. Over-warning about an object some fourth tool owns costs a line of output; under-warning costs the object. Findings **never change the exit code**. A kind the cluster does not serve is reported as not installed rather than raised, since the CRD a record names may simply not be there, but only once discovery has enumerated that kind's API group; if it could not, the kind cannot be called absent and the scan fails. A scan that fails (an RBAC gap, an apiserver that went away, an API group discovery could not reach) fills its own section with the reason and leaves the comparison alone. **An empty `AT RISK` section is not an all-clear.** It renders on every run, including one that contacted no cluster, and states which case it is: no cluster access was requested, the scan was explicitly turned off, no crossed record names a resource kind so nothing was examined, or the scan read objects and found every one of them owned. `--format json` and `--format yaml` carry `atRisk` unconditionally for the same reason. **Verdicts:** | Verdict | Meaning | Fails a strict run | |---------|---------|--------------------| | `safe` | Upgrade in place. Nothing to do. | no | | `manual` | Operator steps are required first. | yes | | `blocked` | Do not make this jump in one step. The report names the boundary it stops at. | yes | | `unknown` | No record covers this transition. A gap in the **data**. | yes | | `unversioned` | One side's version is not comparable. A gap in the **inputs**, closed by pinning something comparable. | yes | Anything other than `safe` exits non-zero. `unknown` is included deliberately: a transition nobody assessed is not a transition anyone approved, and the distance moved does not change that. The report still reports a **breaking boundary** (a major bump, a minor bump while the major version is `0`, or a changed prerelease identifier over an otherwise unchanged `major.minor.patch`) in the NOTES cell and the JSON `breaking` field, because the size of a move tells you how hard to look. It no longer affects the exit code. **Four kinds of `unknown`.** They differ in what would close the gap, so they carry different `reason` codes and different report text: | `reason` | Situation | Closed by | |---|---|---| | `no-record` | No record exists for this component | Somebody authoring the first record | | `no-boundary-crossed` | A record exists but says nothing about this range | Widening it, or confirming no boundary belongs there | | `downgrade` | You are rolling back | Nothing. Records describe forward moves only, so this can never become known | | `identity-changed` | The component's namespace, chart, source, path, deployment type, manifest files, pre-manifest files or object names moved between the two artifacts | Performing the relocation, replacement or rename as its own piece of work, or resolving the target recipe with `aicr recipe --inherit-from` so it does not happen | `blocked` and `unknown` say opposite things. `blocked` means AICR has something to tell you and a version to stop at: read it and act on it. `unknown` means AICR has nothing for you: read the component's own upstream release notes and decide. Neither is a pass. **Rollout note: expect red today.** Only five registry components ship a transition record so far, so most components that change version report `unknown` and the check exits non-zero on most comparisons. That is a coverage problem being worked ([#2535](https://github.com/NVIDIA/aicr/issues/2535) makes records mandatory per pin bump), not a tool limitation, and it shrinks as records are authored. Use `--fail-on-error=false` if you want the report without the gate in the meantime. Components whose version *and* identity are identical on both sides produce no row. Added components are reported with nothing to do. Removed components are reported and **stay installed**, because AICR does not uninstall them. **Identity is compared too.** A recipe is regenerated from scratch on every AICR upgrade, and each component's namespace comes from `recipes/registry.yaml`, so a moved registry default lands in the regenerated recipe. Helm cannot move a release between namespaces, so applying the resulting bundle installs a **second copy** of the component beside the running one, and nothing reconciles the two. A version-only comparison reports that as no change at all, so the identity is compared as its own axis. Besides the namespace, that is the chart name, the chart or kustomize source, the kustomize path, the deployment type, and the sets of manifest files and pre-manifest files. A chart, source or path move can leave the version untouched while pointing at different content, and a manifest file that leaves the set deletes a live object when the deployer prunes. A scalar field (namespace, chart, source, path, deployment type) counts only when both artifacts state it. The manifest and pre-manifest file sets are compared even when one side is empty, so a set that empties or fills is a move. Reordering a set is not a move. A move is reported like this: | What moved | Change kind | What the row says | |---|---|---| | Identity only | `identity` | A row where a version comparison produced none. Verdict `unknown`, reason `identity-changed`, and it fails a strict run. | | Version and identity in the same hop | `version` | The change rides on the version row. A `safe` verdict there is **withdrawn** to `unknown`. Every other verdict stands, because it already stops the run and already sends you to the row. | `unknown` rather than `blocked` is deliberate. `blocked` is an author's judgement recorded against a version boundary, and the record vocabulary has no way to express one about where a release lives, so no author can record it. `--format json` and `--format yaml` carry the move as `identityChanges`, a list of `{field, from, to}` objects, on both kinds of row. For `manifestFiles` and `preManifestFiles`, `from` and `to` hold the whole sorted sets, and the row's `explanation` names the entries dropped and added. The remedy is not an upgrade step. Either move the release deliberately and re-run the check, or resolve the target recipe with [`aicr recipe --inherit-from`](#aicr-recipe) so it keeps the identity the prior artifact deployed and the change never enters the hop. See [Upgrading a Deployed Stack](/aicr/user-guide/upgrading-a-deployed-stack#when-a-component-moves-namespace). **Object names are compared too, from a bundle.** Half the registry's components pin the names of the objects their chart creates, with `fullnameOverride` or `nameOverride` in their values file. Those two keys are part of the identity compared above and are reported on the same rows, with `field` carrying the dotted value path (`fullnameOverride`, `grafana.fullnameOverride`) where an identity field carries its own name. **Exactly those two keys are compared, at any depth, and nothing else.** They are the ones Helm's `chart.fullname` and `chart.name` templates read, so they rename an object rather than reconfigure it. A chart can still name an object it owns through some other value of its own — `serviceAccount.name` is the common one — and no row is produced for that. Read a clean report as "neither override key moved", not as "nothing was renamed." An empty `from` or `to` means the name appeared or disappeared, which is a rename either way — the opposite of how the namespace axis reads an empty value, because a chart with no `fullnameOverride` names its objects after itself. Renaming is applied by Helm as delete-and-recreate: expect a service gap, and an orphan for anything referenced by name or not owned by the release. Where the moved key changes the selector labels of an object whose name does not change (for example `nameOverride` moving while `fullnameOverride` is pinned), `spec.selector` is immutable and the upgrade fails outright instead. This axis needs a **bundle** as `--from`. A resolved recipe records `valuesFile` as a path resolved against whichever binary reads it, so comparing two recipe files would read today's values twice and see nothing move; only a bundle writes the merged result down. `--from cluster` cannot supply them either, because the cluster read recovers versions alone. When the source cannot supply them, the report says so once above the table rather than reporting that nothing moved, and `--format json` carries `objectNamesCompared: false` with `objectNamesSkipped` naming the reason. See [Upgrading a Deployed Stack](/aicr/user-guide/upgrading-a-deployed-stack#when-a-components-objects-are-renamed). **The four routes to `blocked`.** A record is *crossed* when your source version sits below the boundary its `to` names and your target reaches it. That is a property of the jump alone, so a record still counts even when the jump flies straight over it: | Route | When | `reason` | Renders steps | |---|---|---|---| | A record describes this move and blocks it | One record is crossed and its `from` covers your source | `recorded` | yes | | You would skip a boundary | Two or more records that ask something of you are crossed, or a crossed `blocked` record is not the only boundary in the way, or one was written for a different starting point | `multiple-boundaries`, `record-blocks` | no | | Nothing describes your starting version | One record is crossed, but its `from` does not cover your source, usually because you are below the lowest recorded starting point | `undefined-origin` | no | | Your target is past what the record assessed | One record is crossed and its `from` covers your source, but your target sits above the ceiling that record's `to` names | `beyond-record-ceiling` | no | A crossed `safe` boundary does not count toward the second row: it carries no steps by construction, so crossing one composes nothing and skips nothing, and a jump whose only *substantive* boundary describes the whole move reports that record rather than stopping. The first row renders its record's steps, deployer-scoped, exactly as a `manual` row does: the author marked the move `blocked` and then wrote what to do instead. The other three render none, because the record that carries them describes a different move than the one you asked about. All four name a stopping point. The fourth exists because a record vouches only as far as its own `to` ceiling. A record claiming `>=0.18.0 <0.19.0` says nothing about `0.25.0`, and letting it lend its verdict there would report eight minors as safe on the strength of a two-minor claim. Stop at the assessed ceiling and re-run, or have the record widened. This is also how a component deliberately held below a breaking release reports: its record's ceiling sits at that release, so any target above it is told to stop there and take the boundary on its own. Every row states its reason in the detail block under the table, and `--format json` carries the same thing as `reason` (a stable code: `recorded`, `record-blocks`, `multiple-boundaries`, `undefined-origin`, `beyond-record-ceiling`, `no-record`, `no-boundary-crossed`, `downgrade`, `identity-changed`, `not-comparable`) plus `explanation`, the sentence naming your versions. **Why `--deployer` is required rather than defaulted:** Steps are deployer-scoped. Showing an Argo CD operator an imperative "delete the legacy CRDs" step is the exact failure deployer-scoping exists to prevent, so the command asks rather than guessing, and never renders every deployer's path. On an artifact comparison it is only required when some component actually carries steps; `--from cluster` requires it unconditionally, for the separate reason that the release names it maps encode it. A bundle now records the deployer that built it in [`bundle-info.yaml`](/aicr/user-guide/generating-bundles#bundle-info), so `upgrade-check` can stop asking once it reads that record ([#2528](https://github.com/NVIDIA/aicr/issues/2528)). **Example:** ```console $ aicr upgrade-check --from old-recipe.yaml --to new-recipe.yaml --deployer helm UPGRADE CHECK from old-recipe.yaml to new-recipe.yaml deployer helm COMPONENT FROM TO VERDICT NOTES --------- ---- -- ------- ----- grove v0.1.0-alpha.8 v0.1.0-alpha.12 manual 3 steps grove v0.1.0-alpha.8 -> v0.1.0-alpha.12 (manual) alpha.12 drops the clustertopologies.grove.io CRD (kind ClusterTopology) in favor of clustertopologybindings.grove.io, which reuses the shortname ct. PRECONDITION No ClusterTopology objects exist in the cluster. STEPS (deployer: helm) 1. delete-legacy-crd kubectl delete crd clustertopologies.grove.io ... 1 component change, 1 needs attention ``` **More examples:** ```shell # Two recipes, for a pipeline that already knows its deployer aicr upgrade-check --from old-recipe.yaml --to new-recipe.yaml --deployer argocd # Am I behind, and does catching up hurt? aicr upgrade-check --from ./bundles-v0.16.0 --deployer helm # What is actually installed on this cluster, and does moving it hurt? aicr upgrade-check --from cluster --to ./bundles-v0.17.0 --deployer argocd # Compare two bundles, and scan the live cluster while doing it aicr upgrade-check --from ./bundles-v0.16.0 --to ./bundles-v0.17.0 \ --deployer helm --scan-cluster # Read the cluster without the advisory scan it would otherwise imply aicr upgrade-check --from cluster --to ./bundles-v0.17.0 \ --deployer flux --scan-cluster=false # JSON for a pipeline, reporting without gating aicr upgrade-check --from old.yaml --to new.yaml \ --format json --output report.json --fail-on-error=false ``` **Exit Codes:** | Code | Description | |------|-------------| | `0` | No component needs attention, or `--fail-on-error=false` | | `2` | Invalid input (missing `--from`, unknown deployer, a bundle with no `recipe.yaml`, a missing `--deployer` where steps are needed, `--from cluster` without `--deployer` or without `--to`, `--to cluster`) **or** a component needs attention, AICR-stamped releases match no component, or a cluster read finds two installs of one component it cannot choose between (all mapped from `ErrCodeConflict`) | | `4` | `--from cluster`: a List the read needs was forbidden. The message names the permission to grant | | `5` | `--from cluster`: the run's own deadline expired | | `6` | `--from cluster`: the apiserver went away mid-read, or a paged List outlived its continue token. Re-run | | `8` | `--from cluster`: a release record this read answers for could not be read: an unknown storage format, an undecodable payload, or a malformed revision label. The message names the object | | `9` | The run was canceled | > **Note on CI gating:** as with `aicr diff`, a bad invocation and a failing check both exit `2`. To tell them apart without parsing stderr, write the report with `--format json --output report.json` and branch on the file's presence plus its `summary.failing` count. **Limitations:** - **A bundle is read through the `recipe.yaml` at its root.** Every deployer writes one as of [#2759](https://github.com/NVIDIA/aicr/pull/2759); a directory without it is neither a recipe nor a bundle and is rejected rather than misread. - **Coverage starts near zero.** Every transition without an authored record reports `unknown`. See the [authoring guide](/aicr/contributor-guide/upgrade-transition-records). - **A namespace move is seen only when both artifacts state one.** An absent namespace is read as a field the artifact did not carry, not as the default, so a component that gains or loses an explicit namespace between the two artifacts produces no relocation row. Reading it the other way would report a move nobody performed for every component the moment one side stopped carrying the field. - **A cluster read answers the version question and nothing else.** What it reads are the deployers' own records of what they applied, so it is authoritative about installed versions and silent about everything a hand edit changed. It also reports no relocation, because it recovers no namespace to compare. - **Argo CD bundles built with `--vendor-charts` read as `unversioned`.** Vendoring turns every chart into a path-based `Application`, which carries no payload version anywhere in the cluster, so a strict run always fails. Compare the vendored bundle as an artifact instead: `--from `. - **The at-risk scan sees ownership markers, not use.** An object is reported for carrying no marker the scan recognizes, which says nothing about whether anything depends on it. Confirming that is yours to do. - **Any Helm release or Argo CD `Application` counts as an owner, not only AICR's.** Your own custom resources managed by your own chart or `Application` are treated as owned and not reported. Conversely, Argo CD 2.x tracks resources by the `app.kubernetes.io/instance` label by default rather than the annotation the scan reads, so there every Argo-managed object of a scanned kind is reported. --- ### aicr bundle Generate deployment-ready bundles from recipes containing Helm values, manifests, scripts, and documentation. **Synopsis:** ```shell aicr bundle [flags] ``` **Flags:** | Flag | Short | Type | Description | |---------------------------------|-------|------|-------------| | `--recipe` | `-r` | string | Path to recipe file (required, or via `spec.bundle.input.recipe` in `--config`) | | `--config` | | string | Path or HTTP/HTTPS URL to an AICRConfig file (YAML/JSON). CLI flags override values from this file. See [Bundle Config File Mode](#bundle-config-file-mode). | | `--output` | `-o` | string | Local output directory or `oci://` registry URI (default: current directory) | | `--deployer` | `-d` | string | Deployment method: `helm` (default), `argocd`, `argocd-helm`, `flux`, or `helmfile` | | `--repo` | | string | Git/OCI repository URL baked into GitOps sources: Argo CD Application sources with `--deployer argocd`, and the Flux `GitRepository` source for local-chart components with `--deployer flux` (a placeholder URL is written when omitted). Ignored with `--deployer argocd-helm` (that bundle is URL-portable — the URL is supplied at `helm install` time via `--set repoURL=...`); a warning is logged if passed. | | `--set` | | string[] | Override **scalar** values in bundle files (repeatable, format: `component:path=value`). Use `enabled` key to include/exclude components (e.g., `--set awsebscsidriver:enabled=false`). Scalar-only — for list/object values use `--set-json` / `--set-file`. An override whose component is absent from the generated bundle is rejected rather than silently discarded; the scalar `enabled=false` spelling is exempt on a declared component (it is the removal mechanism). See [Overrides that cannot take effect are rejected](/aicr/user-guide/generating-bundles#overrides-that-cannot-take-effect-are-rejected). | | `--set-json` | | string[] | Override values with a JSON-encoded **list or object** (repeatable, format: `component:path=`, e.g. `--set-json agentgateway:allowedSourceRanges='["216.228.127.128/30"]'`). Object values deep-merge into existing maps; lists and scalars replace. Takes precedence over `--set` on the same path. An override whose component is absent from the generated bundle is rejected — no `enabled` exemption on the typed path (`enabled` is honored only via scalar `--set`); see [Overrides that cannot take effect are rejected](/aicr/user-guide/generating-bundles#overrides-that-cannot-take-effect-are-rejected). See [List and Object Value Overrides](#list-and-object-value-overrides). | | `--set-file` | | string[] | Override a value by reading JSON/YAML from a file (repeatable, format: `component:path=`). For larger structures than `--set-json`; same merge and absent-component-rejection semantics (no `enabled` exemption on the typed path). | | `--dynamic` | | string[] | Declare value paths as install-time parameters (repeatable, format: `component:path`). Supported with `helm`, `argocd-helm`, `flux`, and `helmfile` deployers. A declaration whose component is absent from the generated bundle is rejected (no path is exempt — a dynamic path is never a removal idiom); see [Overrides that cannot take effect are rejected](/aicr/user-guide/generating-bundles#overrides-that-cannot-take-effect-are-rejected). Certain gate- or contract-owned paths on **present** components cannot be declared dynamic either — driver-ownership paths (e.g. `gpuoperator:driver.enabled`), GPU allocation-policy keys, the DRA eviction paths `kubeletPlugin.nodeSelector` and `driver.manager.env` when both contract components are enabled **and** the eviction contract is opted into with `--dra-eviction-node-label`, and, where the corresponding NVSentinel gate applies on the recipe's platform and configuration, the NVSentinel remedy/consumer/runtime-class paths — because an install-time edit there would undo what AICR verified or made consistent; see [NVSentinel on provider-installed-driver platforms](/aicr/user-guide/component-catalog#nvsentinel-on-provider-installed-driver-platforms). See [Dynamic Install-Time Values](#dynamic-install-time-values). | | `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) | | `--system-node-selector` | | string[] | Node selector for system components (format: key=value, repeatable). Optional in general, but some components (e.g. `slinky-slurm`, `slurm-accounting-mariadb`) declare `requireNodeSelector` in the registry and fail the bundle if this is omitted and no overlay opts their paths out. `kube-prometheus-stack` declares the conditional `requireNodeSelectorIfStorageClassSet` instead, so it only fails once the component ends up with a non-empty value at a declared `storageClassPaths`/`sharedStorageClassPaths` entry, whether from `--storage-class`, a per-component `--set` override, or an overlay's own `storageClassName` default. See [`nodeScheduling.system` vs `accelerated`](/aicr/contributor-guide/components#nodeschedulingsystem-vs-accelerated). | | `--system-node-toleration` | | string[] | Toleration for system components (format: key=value:effect, repeatable). With `--readiness-hooks`, this (keyed tolerations only) and `--system-node-selector` also place the readiness gate Jobs; see [Readiness Gates](#readiness-gates). | | `--accelerated-node-selector` | | string[] | Node selector for accelerated/GPU nodes (format: key=value, repeatable). Same `requireNodeSelector` caveat as `--system-node-selector` above applies to components that declare it on their accelerated paths. | | `--accelerated-node-toleration` | | string[] | Toleration for accelerated/GPU nodes (format: key=value:effect, repeatable) | | `--dra-eviction-node-label` | | string | Opt in to DRA kubelet-plugin eviction coordination with GPU Operator driver upgrades (format: `key=value`; no default — unset means AICR injects nothing). Applied only when both components are enabled. Also deploys `dra-node-labeler`, which applies the label to every GPU node from GFD's `nvidia.com/gpu.present`; pass `--set dra-node-labeler:enabled=false` to provision the label yourself instead. | | `--workload-gate` | | string | Taint for nodewright-operator runtime required (format: key=value:effect or key:effect). This is a day 2 option for cluster scaling operations. | | `--workload-selector` | | string[] | Label selector for nodewright-customizations to prevent eviction of running training jobs (format: key=value, repeatable). Required when nodewright-customizations is enabled with training intent. | | `--nodes` | | int | Estimated number of GPU nodes (default: 0 = unset). At bundle time, written to Helm value paths declared in the registry under `nodeScheduling.nodeCountPaths`. | | `--storage-class` | | string | Kubernetes StorageClass name to inject at bundle time. Written to registry-declared `storageClassPaths` for each component. Overrides any `storageClassName` set in recipe overlays. For a component declaring the conditional `requireNodeSelectorIfStorageClassSet` (e.g. `kube-prometheus-stack`), setting this to a non-empty value (or providing a non-empty value through a per-component `--set` override or an overlay's own `storageClassName` default) also starts requiring `--system-node-selector`/`--accelerated-node-selector`; see the `--system-node-selector` row above. | | `--shared-storage-class` | | string | RWX-capable Kubernetes StorageClass for opt-in shared filesystem PVCs. Written to registry-declared `sharedStorageClassPaths`; never falls back to `--storage-class`. | | `--vendor-charts` | | bool | Pull upstream Helm chart bytes into the bundle at bundle time so the artifact is fully self-contained and air-gap deployable. Requires `helm` on `$PATH`. See [Vendoring Charts for Air-Gap](#vendoring-charts-for-air-gap). | | `--readiness-hooks` | | bool | Emit a per-component readiness gate (`NNN--readiness/`) for each component that ships a `recipes/components//readiness.yaml` Chainsaw test. The gate runs as a post-component Job so the deploy blocks on component-specific readiness signals (e.g. `ClusterPolicy` state). Supported with `--deployer helm`, `argocd`, and `argocd-helm`. Off by default. See [Readiness Gates](#readiness-gates). | | `--serial` | | bool | Sequence components strictly one at a time in deployment order, disabling the parallel rollout of independent components. Affects `--deployer argocd`, `argocd-helm`, `flux`, and `helmfile` (helm is already serial): argocd falls back to a linear sync-wave per folder, flux chains each `HelmRelease` `dependsOn` to the previous component, and helmfile chains every release via `needs:` into one linear apply chain. An escape hatch for reproducing the pre-parallelism ordering or bisecting a rollout. Off by default. | | `--flux-oci-source-name` | | string | Name of the OCIRepository CR that Flux uses to pull the bundle (default: `aicr-bundle`). Used with `--deployer flux` and OCI output. Must match the OCIRepository deployed in the target cluster. See [Flux OCI Mode](#flux-oci-mode). | | `--flux-namespace` | | string | Kubernetes namespace where Flux CRs (HelmRelease, sources, ArtifactGenerator) are deployed (default: `flux-system`). Must match the namespace of the Flux installation in the target cluster. | | `--app-name` | | string | Parent Argo Application name (default: `aicr-stack` for `--deployer argocd-helm`, `nvidia-stack` for `--deployer argocd`). Must be a DNS-1123 subdomain. Required when deploying multiple non-overlapping AICR bundles to the same Argo CD namespace so the parent Applications do not collide. For `--deployer argocd-helm`, the value is the chart default and can still be overridden at install time via `helm install --set appName=...`. Rejected on other deployers (`helm`, `flux`, `helmfile`). | | `--kubeconfig` | `-k` | string | Path to kubeconfig file | | `--insecure-tls` | | bool | Skip TLS verification for OCI registry connections | | `--plain-http` | | bool | Use plain HTTP for OCI registry connections | | `--image-refs` | | string | External file to receive the published OCI digest. Valid only with OCI `--output`; local output is rejected. The parent must be an existing real directory, and the target must be outside and not aliased to the planned or completed bundle. | | `--attest` | | bool | Enable bundle attestation and binary provenance verification. Requires OIDC authentication, or a KMS key via `--signing-key` for environments without OIDC. See [Bundle Attestation](#bundle-attestation). | | `--certificate-identity-regexp` | | string | Override the certificate identity pattern for binary attestation verification. Must *begin with* `https://github.com/NVIDIA/aicr/` (a leading `^` is allowed) and must not use top-level alternation. For testing only. | | `--identity-token` | | string | Pre-fetched OIDC identity token for `--attest` keyless signing. Skips ambient/browser/device-code flows. Prefer `COSIGN_IDENTITY_TOKEN` on shared hosts — flag values are visible in `ps` and `/proc//cmdline`. | | `--oidc-device-flow` | | bool | Use the OAuth 2.0 device authorization grant for `--attest` instead of opening a browser callback. Useful on headless hosts that can still reach Sigstore (`--identity-token` and CI ambient OIDC are alternatives). Also reads `AICR_OIDC_DEVICE_FLOW`. | | `--fulcio-url` | | string | Override the Fulcio CA URL for `--attest` keyless signing, pointing at a private Sigstore instance. Must be an absolute `https://` URL with no embedded credentials. Defaults to the public-good Fulcio when omitted. Also reads `AICR_FULCIO_URL`. | | `--rekor-url` | | string | Sign the `--attest` bundle to **Rekor v1** at this URL instead of the **Rekor v2 default** (a private Sigstore instance, or the public-good v1 URL). Must be an absolute `https://` URL with no embedded credentials. Also reads `AICR_REKOR_URL`. Independent of `--fulcio-url`. Mutually exclusive with `--signing-config`. | | `--signing-config` | | string | Sign the `--attest` bundle with a custom Sigstore signing config JSON instead of the default Rekor v2 config (advanced — e.g. an edited config or a private v2 instance). Also reads `AICR_SIGNING_CONFIG`. Mutually exclusive with `--rekor-url`. | | `--signing-key` | | string | Sign the `--attest` bundle with a KMS-backed key instead of keyless OIDC, for CI/CD environments without OIDC (Jenkins, internal pipelines). Takes a KMS URI; supported schemes are `awskms://`, `gcpkms://`, `azurekms://`, and `hashivault://`. Like keyless signing, KMS signs to Rekor v2 by default; opt out with `--rekor-url` (v1) or `--signing-config` (custom). Mutually exclusive with `--identity-token`, `--oidc-device-flow`, and `--fulcio-url` (the keyless-only flags); passing both is a validation error. See [KMS-Backed Signing](#kms-backed-signing). | | `--tlog-upload` | | bool | Upload the signature to the Rekor transparency log (default: `true`). Set `--tlog-upload=false` to skip the Rekor upload for fully offline / air-gapped signing; this requires `--signing-key` (KMS), because keyless OIDC signing needs Fulcio and Rekor network access to mint a verifiable certificate. Mutually exclusive with `--rekor-url` and `--signing-config` (both select where the transparency-log entry goes, which contradicts writing none). Verify the resulting bundle offline with `aicr verify --key --insecure-ignore-tlog`; use a local PEM public key for a fully offline verify, since a KMS `--key` URI still makes a live `GetPublicKey` call. See [KMS-Backed Signing](#kms-backed-signing). | | `--yes` | `--assume-yes` | bool | Skip the interactive confirmation shown before keyless signing publishes your OIDC identity (browser/device-code paths only; the banner is still printed). Reads `AICR_ASSUME_YES`. See [Privacy: identity in keyless signatures](#privacy-identity-in-keyless-signatures). | `--image-refs` writes the published digest through a mode-`0600` temporary file and an anchored same-directory rename. Its target may be absent or an existing retained regular file; directories, symlinks, other non-regular files, and aliases of the bundle tree are rejected. The final validation and rename are ordered but are not one atomic identity-conditioned filesystem operation, so no other process should mutate the target directory while the command runs. For every CLI OCI publication, AICR revalidates a private bundle snapshot and publishes only its closed-world inventory; it never packages the mutable caller tree directly. For Argo CD Helm OCI output, AICR keeps the raw Distribution tag in the registry reference and derives a strict Helm semantic version for chart metadata and consumers. For example, registry tag `1.2.3_build.5` remains unchanged, while `Chart.yaml`, Argo CD `targetRevision`, and Helm's `--version` use `1.2.3+build.5`. The OCI repository basename must equal the generated chart name. #### Bundle Config File Mode The bundle command accepts the same `AICRConfig` format used by `aicr recipe`. A single file can populate both `spec.recipe` and `spec.bundle`, capturing an end-to-end workflow that can be committed to git, fetched from CI, or shared across environments. When both `spec.recipe.output.path` and `spec.bundle.input.recipe` are set, they must reference the same path; otherwise loading fails fast. ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 spec: bundle: input: recipe: ./recipe.yaml output: target: oci://ghcr.io/example/bundle:v1.0.0 imageRefs: ./publish/bundle.digest # ./publish must already be a real directory deployment: deployer: argocd repo: https://example.git/charts set: - gpuoperator:driver.version=570.86.16 scheduling: systemNodeSelector: role: system acceleratedNodeTolerations: - "nvidia.com/gpu=present:NoSchedule" draEvictionNodeLabel: nvidia.com/dra-kubelet-plugin=true nodes: 8 storageClass: gp3 attestation: enabled: false # Optional: target a private Sigstore instead of the public-good # endpoints. Each defaults to public Sigstore when omitted; both must # be absolute https:// URLs with no embedded credentials. fulcioURL: https://fulcio.internal.example.com rekorURL: https://rekor.internal.example.com # Optional KMS-backed signing (awskms:// | gcpkms:// | azurekms:// | # hashivault://): a durable, non-secret key reference that replaces # keyless OIDC. Mirrors --signing-key (the flag wins when both are set). # Mutually exclusive with the keyless-only inputs above (fulcioURL, # oidcDeviceFlow) and the runtime-only --identity-token, so it is shown # commented out here as the alternative to the keyless block. # signingKey: awskms://alias/aicr-signing registry: insecureTLS: false plainHTTP: false ``` ```shell # Drive the bundle entirely from a config file aicr bundle --config bundle.yaml # Override the deployer for a one-off run aicr bundle --config bundle.yaml --deployer helm ``` CLI flags always override values loaded from `--config`. For slice/map flags (`--set`, `--dynamic`, `--system-node-selector`, etc.), CLI presence replaces the config's value rather than appending. Override events are logged at INFO so users can see which input won. **Secrets:** the cosign identity token is never read from a config file; supply it via `--identity-token` or `COSIGN_IDENTITY_TOKEN`. #### Node Scheduling The `--accelerated-node-selector` and `--accelerated-node-toleration` flags control scheduling for GPU-specific components: | Flag | GPU DaemonSets | NFD Workers | |------|---------------|-------------| | `--accelerated-node-selector` | Applied (restricts to GPU nodes) — **except gpu-operator operands**, which self-place via the operator's GPU detection | **Not applied** (NFD runs on all nodes) | | `--accelerated-node-toleration` | Applied (including gpu-operator operands, via `daemonsets.tolerations`) | Applied | | `--system-node-selector` | Not applied | Not applied | | `--system-node-toleration` | Not applied | Not applied | NFD (Node Feature Discovery) workers must run on **all nodes** (GPU, CPU, and system) to detect hardware features. This matches the gpu-operator default behavior where NFD workers also run on control-plane nodes. The `--accelerated-node-selector` is intentionally not applied to NFD workers so they are not restricted to GPU nodes. > **Note:** When no `--accelerated-node-toleration` is specified, a default toleration (`operator: Exists`) is applied to both GPU DaemonSets and NFD workers, allowing them to run on nodes with any taint. **Example:** ```bash aicr bundle --recipe recipe.yaml \ --accelerated-node-selector nodeGroup=gpu-worker \ --accelerated-node-toleration dedicated=worker-workload:NoSchedule \ --accelerated-node-toleration dedicated=worker-workload:NoExecute \ --system-node-selector nodeGroup=system-worker \ --system-node-toleration dedicated=system-workload:NoSchedule \ --system-node-toleration dedicated=system-workload:NoExecute \ --output bundle ``` > **Cluster node requirements:** This example assumes the cluster has nodes labeled `nodeGroup=system-worker` with taints `dedicated=system-workload:NoSchedule,NoExecute` for system infrastructure, and GPU nodes labeled `nodeGroup=gpu-worker` with taints `dedicated=worker-workload:NoSchedule,NoExecute`. This results in: - **gpu-operator operand DaemonSets** (driver, device-plugin, toolkit, dcgm): **no `nodeSelector` is applied** — the gpu-operator chart and its ClusterPolicy CRD have no `daemonsets.nodeSelector` field, so the selector cannot constrain them; the operator places these operands on GPU nodes itself via its GFD/NFD-driven `nvidia.com/gpu.deploy.*` labels. They **do** receive the tolerations for `dedicated=worker-workload` (both `NoSchedule` and `NoExecute`) through the real `daemonsets.tolerations` value. - **NFD workers**: no nodeSelector (runs on all nodes) + tolerations for `dedicated=worker-workload` with both `NoSchedule` and `NoExecute` - **System components** (gpu-operator controller, NFD gc/master, dynamo grove, agentgateway proxy): `nodeSelector=nodeGroup=system-worker` + tolerations for `dedicated=system-workload` with both `NoSchedule` and `NoExecute` **Behavior:** - All components from the recipe are bundled automatically - Each component creates a subdirectory in the output directory - Components are deployed in the order specified by `deploymentOrder` in the recipe #### DRA Driver Upgrade Eviction **This is opt-in.** By default AICR injects nothing here, so the DRA kubelet plugin runs on every accelerated node with no extra node label required. Set `--dra-eviction-node-label` (or `scheduling.draEvictionNodeLabel`) to opt in. **What the opt-in does.** When a recipe includes both `nvidia-dra-driver-gpu` and `gpu-operator` and a label is configured, AICR merges that `key=value` into `kubeletPlugin.nodeSelector` and sets the GPU Operator `driver.manager.env` entry `NODE_LABEL_FOR_GPU_POD_EVICTION` to the same key, so GPU Operator's Driver Manager can deschedule the plugin ahead of a driver container restart. The same applies to the `-ocp` components. Existing accelerated-node selectors and unrelated Driver Manager environment variables are preserved. **The label is derived, not provisioned.** The same opt-in keeps the `dra-node-labeler` component in the bundle: a small DaemonSet that runs on every node GFD reports as `nvidia.com/gpu.present=true` and applies the configured `key=value` once, after GPU Operator is up. It never rewrites an existing value, so the Driver Manager's `paused-for-driver-upgrade` and a hand-set `false` opt-out both survive a labeler restart. Nodes added later by autoscaling, replacement, or a pool scaled from zero are labeled the same way as soon as GFD labels them. Without the flag, or when a `bundlers` filter leaves out the labeler, the labeler is left out of the bundle entirely (a `bundlers` selection that names the labeler without the flag or without both components it serves is rejected as invalid); node labeling is then required only if both components remain and eviction is still opted in. OpenShift recipes are not wired for the labeler yet (#2828) and keep the provisioning workflow. To provision the label yourself (node-pool labels, Karpenter `NodePool` labels) pass `--set dra-node-labeler:enabled=false`; the requirements in the next paragraphs then apply. ```bash aicr bundle --recipe recipe.yaml \ --dra-eviction-node-label nvidia.com/dra-kubelet-plugin=true \ --output bundle ``` **What you give up by not opting in.** The plugin is not descheduled before a driver container restart. On a driver upgrade the module unload can fail with `failed to uninstall nvidia driver components`, leaving the replacement uninstalled and the old driver possibly partially torn down. On a restart with unchanged driver configuration the stale driver rootfs is unmounted underneath the running plugin; upstream documents this as leaving `NodePrepareResources` unable to build CDI specs, with no error at restart time. That second shape concerns the **full-GPU allocation** path, and we have not observed it in AICR's default configuration — see the note below. `aicr bundle` warns about this when GPU Operator manages the driver. This does not apply where the driver is provider-installed (`driver.enabled=false` — AKS `azure-managed`, GKE COS, OKE). Those deploy no GPU Operator driver pod and therefore no Driver Manager, so there is nothing to coordinate with and no warning is emitted. **If you disable the labeler, every GPU node must carry the label.** Set it in the **node pool definition** — an EKS managed nodegroup `labels` entry, a Karpenter `NodePool` `spec.template.metadata.labels` entry, or the equivalent for your provisioner. An ad hoc `kubectl label node` is a repair, not a configuration: it does not survive node replacement, recycling, autoscaling, or a nodegroup scaled from zero. With the labeler in the bundle (the default once opted in) none of this is required; the remaining gap is a GPU node GFD has not labeled that does not already carry the configured `key=value` (for example from a node pool that still provisions it): such a node runs no kubelet plugin until `nvidia.com/gpu.present` appears. ```bash kubectl label node nvidia.com/dra-kubelet-plugin=true kubectl get nodes -l nvidia.com/dra-kubelet-plugin=true ``` **The opt-in failure mode is silent, and partial coverage is the dangerous shape.** An unlabeled GPU node then runs no DRA kubelet plugin and publishes no `ResourceSlices` for itself. If *no* GPU node carries the label, the `nvidia-dra-driver-gpu-kubelet-plugin` DaemonSet sits at `DESIRED=0`. If *some* do, those nodes work normally while the rest silently lack DRA. Helm and the bundle's `deploy.sh` report success in every case, so check the DaemonSet after applying. **Recovering from the two opt-out failures.** Both are recoverable, and the procedures are different — only the first needs the plugin suppressed. *Driver upgrade stopped with `failed to uninstall nvidia driver components`.* The plugin still holds the driver, so the replacement is not installed and the old driver may be partially torn down. Recovery means clearing the node's DRA claim holders and then removing the plugin before retrying the driver. Whether the removal can be confined to the failed node depends on the deployed tolerations, below. **`kubectl cordon` does not keep the plugin off the node.** The DaemonSet controller adds a `node.kubernetes.io/unschedulable:NoSchedule` toleration to its pods, so a cordoned node still gets one and deleting the pod simply recreates it. Suspending a GitOps controller does not help either; the DaemonSet controller is what recreates the pod. **Do not patch the DaemonSet to exclude a node.** Two properties of the shipped DaemonSet rule out per-node *template* edits: its affinity is five OR-ed `nodeSelectorTerms`, so excluding a node requires editing every term rather than one; and it uses `RollingUpdate` with `maxUnavailable: 100%`, so *any* change to `.spec.template` rolls pods on every node it covers — and reverting the change rolls them again. A node-scoped-looking patch is therefore cluster-wide in effect. **A node taint is the exception, and it is worth checking for.** A taint acts on the node, not the DaemonSet template, so it rolls nothing. Whether it can work depends on the tolerations of your *deployed* DaemonSet — read them before choosing a procedure: ```bash kubectl -n nvidia-dra-driver get ds nvidia-dra-driver-gpu-kubelet-plugin \ -o jsonpath='{.spec.template.spec.tolerations}' ``` **Expect a wildcard, because that is AICR's default.** When `--accelerated-node-toleration` is not passed, AICR applies a keyless `{operator: Exists}` toleration, which accepts every taint and leaves no key to exclude a node with. A default bundle therefore has no node-scoped option and must use the cluster-wide sequence below. This was the deployed state on an EKS GB300 cluster whose bundle was generated without toleration flags. **A taint works only when the deployed list is narrow** — a bundle built with explicit `--accelerated-node-toleration` flags, or the upstream chart default of `nvidia.com/gpu` alone. Then a taint whose key appears nowhere in that list excludes the plugin from exactly that node. **"Node-scoped" describes the plugin suppression, not the blast radius of the whole procedure.** The taint affects one node, and the DaemonSet keeps running everywhere else. But step 2 quiesces the *controllers* that own claim holders, and a controller is rarely node-scoped: scaling a Deployment, StatefulSet or multi-replica `NodeSet` to zero terminates its Pods on **every** node it runs on, not just the failed one. Scope that step as narrowly as your workloads allow — suspending a single Job, or cordoning and draining only what holds claims here — and treat wider quiescing as a maintenance window, not a node-local repair. Three invariants govern the order, and every step below exists to preserve one of them: - **Fence before you drain.** Terminating a claim holder while a controller can still schedule onto the node re-fills what you just cleared. - **The plugin outlives its claim holders.** The kubelet routes `NodeUnprepareResources` through the plugin, so a holder still terminating after the plugin is gone will hang. - **The driver recovers before the plugin returns.** A plugin that comes back early reopens the driver mid-recovery, reintroducing the original failure. 1. **Fence the node. This is the very first operation — before touching any workload.** ```bash kubectl taint node aicr.nvidia.com/dra-recovery=true:NoSchedule ``` `NoSchedule` blocks *new* pods while leaving running ones alone, so the taint fences the node without disturbing the plugin, which is still needed. Every later step assumes this fence is up. 2. **Quiesce the controllers that own claim holders on that node.** The invariant is that *no owning controller can create a replacement Pod*; confirm reconciliation is actually quiesced before relying on it. The action is workload-specific — set `.spec.suspend: true` on Jobs, scale Deployments, StatefulSets and **standalone** ReplicaSets to zero, scale operator-owned workloads (a Slinky Slurm `NodeSet`, for instance) to zero replicas. A Deployment-owned ReplicaSet must be controlled through its Deployment, which will otherwise recreate it. **`kubectl rollout pause` is not sufficient**: it halts rollout progression while the ReplicaSet keeps reconciling replicas, so a deleted claim holder is recreated immediately. These actions *terminate* Pods — Job suspension deletes active Pods, scaling to zero removes them — and, as noted above, they do so wherever that controller runs. That is why the fence goes up first: the terminations on this node cannot then be undone by a controller rescheduling onto it. 3. **Confirm every claim holder on that node has completed `NodeUnprepareResources`.** Allocated ComputeDomain claims can legitimately persist cluster-wide, so a cluster-wide claim listing does not establish that *this* node is clear — resolve holders to the node before proceeding. This is stated as an invariant rather than a command because the holder-to-node mapping depends on your workload shape and no single command was verified here. 4. **Only then delete the plugin on that node.** ```bash kubectl -n nvidia-dra-driver delete pod \ -l nvidia-dra-driver-gpu-component=kubelet-plugin \ --field-selector spec.nodeName= ``` Select on `nvidia-dra-driver-gpu-component=kubelet-plugin`, the DaemonSet's own selector. The chart-wide `app.kubernetes.io/name=nvidia-dra-driver-gpu` label also matches the DRA *controller*, which can be colocated on a GPU node. 5. **Confirm at the node's runtime that the plugin container is actually gone**, not merely that the Pod object was deleted. Aggregate `kubectl get ds` counts prove nothing about this node, and a stuck `FailedKillPod` leaves the container — and the driver handle — alive after the API object disappears. 6. **Retry the driver, and wait for it to finish.** Driver Manager must report success and the driver Pod must reach `Ready`. Do not proceed while it is still retrying. 7. **Only now remove the taint.** ```bash kubectl taint node aicr.nvidia.com/dra-recovery- ``` 8. **Confirm the plugin returns `Ready` on that node and its `ResourceSlices` are republished**, then restore the controllers quiesced in step 2. Until the slices are back the node cannot serve DRA allocations, even though the pod is running. `NoSchedule` does not evict running pods, so the driver pod stays put and its driver-manager init container keeps retrying while the plugin is gone. The DRA plugin on other nodes is untouched; whether their *workloads* are depends entirely on how wide step 2 had to reach. The *suppression* mechanism in steps 1, 7 and 8 was verified on a two-GPU-node AKS cluster built with explicit tolerations, whose plugin tolerated only `nvidia.com/gpu=present:NoSchedule`: tainting one node took the DaemonSet from `DESIRED=2` to `DESIRED=1` with no replacement pod on the tainted node, while the second node's pod kept its original start time — no roll. Removing the taint returned it to `DESIRED=2 READY=2` with both `ResourceSlices` restored. That cluster uses a host-installed driver with no GPU Operator Driver Manager, so steps 3, 5 and 6 — the driver-recovery half — were not exercised there. **Cluster-wide sequence** — required for the default wildcard-toleration bundle, and for any cluster where the check above shows no usable taint key: 1. Stop workload reconciliation across **every** node the DaemonSet covers. Same invariant as the node-scoped path — no owning controller may create a replacement Pod, confirmed quiesced before you drain — and the same caveat: `kubectl rollout pause` leaves the ReplicaSet reconciling replicas, so scale replica controllers to zero and suspend Jobs instead. This matters more here than in the node-scoped path, because there is no taint fence to fall back on: quiescing the controllers is the only lever. 2. Clear DRA claim holders on **every** node the DaemonSet covers, not only the failed one, and confirm none remain. The kubelet needs the plugin to complete `NodeUnprepareResources`, so any claim holder still terminating when the plugin goes away will hang. 3. Suppress the DaemonSet by deleting it. A DaemonSet has no replica count to scale, and editing its `nodeSelector` to match nothing is itself the cluster-wide template rollout — acceptable here only because this sequence has already accepted cluster-wide impact. Suspend any GitOps reconciliation first, or it will recreate the object underneath you. 4. Confirm every plugin pod has terminated. Pod-object deletion is not proof the container is gone; check the nodes. 5. Retry the driver on the affected node and **wait for it to finish** — Driver Manager reporting success and the driver Pod `Ready`. Restoring the DaemonSet while the driver is still retrying recreates the plugin and lets it reopen the driver mid-recovery, which is the same race the node-scoped path guards against. 6. Only then restore the DaemonSet, and confirm the plugin reaches `Ready` with its `ResourceSlices` republished. 7. Finally restore the workload controllers quiesced in step 1. If a GitOps controller reconciles the DaemonSet, suspend it for the duration and resume afterwards. *Claim preparation failing after a restart with unchanged driver configuration.* The stale rootfs was unmounted underneath a plugin that kept running, so upstream's comment describes it as no longer able to build CDI specs. This one needs no DaemonSet suppression and no claim-holder clearing: once the replacement driver pod is `Ready`, restart the plugin pod on the affected node and the recreated pod binds the new rootfs. **Not reproduced in AICR's default configuration.** An attempt on an EKS GB300 cluster running the default ComputeDomain-only DRA setup (`resources.gpus.enabled: false`) reached the exact conditions — driver-manager logged `skipping the uninstallation` and `Unmounting NVIDIA driver rootfs` while the kubelet plugin was never evicted — and then a fresh ComputeDomain claim still prepared successfully, with the plugin not restarted. An existing claim holder also terminated cleanly, so `NodeUnprepareResources` was still being serviced. The degradation upstream describes is about CDI specs, which is the full-GPU allocation path that AICR disables by default, so this shape may be unreachable in the default configuration. Treat that as one negative result in one configuration, not proof: full-GPU DRA was not tested, and only the unchanged-config restart path was exercised. The procedure above is retained because it is the correct response if the degradation does occur. **Known limitation of the opt-in.** Under `k8s-driver-manager` v0.12 the configured label is paused in the same batch as other GPU operands, and no wait covers the standalone DRA kubelet plugin, so ordering against DRA claim holders and completion of plugin teardown are not guaranteed. The mechanism is the one NVIDIA documents, but it is best-effort here. See [NVIDIA/k8s-driver-manager#250](https://github.com/NVIDIA/k8s-driver-manager/issues/250). **Support posture by GPU Operator version.** The 26.3 DRA installation guide requires this label for its **full-GPU allocation** workflow; its ComputeDomain-only procedure does not. AICR's default disables full-GPU DRA (`resources.gpus.enabled: false`) while keeping ComputeDomains, so an unlabeled default bundle does not satisfy the 26.3 full-GPU workflow — it does not put every bundle outside the guide. GPU Operator 26.7 documents the GPUCluster stack instead, which needs no separate label; AICR has not adopted that path. The flag accepts exactly one Kubernetes label in `key=value` form; AICR uses the full pair for DRA placement and the key for GPU Operator: ```bash aicr bundle --recipe recipe.yaml \ --dra-eviction-node-label example.com/dra-ready=enabled \ --output bundle kubectl label node example.com/dra-ready=enabled ``` GPU Operator's Driver Manager receives only the label key; it does not receive or compare the configured value. The cluster's node-labeling convention must therefore preserve the configured key/value pair when the label is restored. The wiring is absent when either component is disabled, and when no eviction label is configured. Once opted in, direct value overrides for the managed selector key or `NODE_LABEL_FOR_GPU_POD_EVICTION` are overwritten so the cross-chart contract cannot drift, and a `--dynamic` declaration intersecting `kubeletPlugin.nodeSelector` or `driver.manager.env` is rejected because install-time editing would split the same contract. Without the opt-in AICR owns neither path, so both remain freely overridable and declarable. See NVIDIA's [GPU Operator DRA installation guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.3/dra-intro-install.html) for the upstream driver-upgrade requirement. #### Storage Class The `--storage-class` flag injects a Kubernetes StorageClass name into components at bundle time. StorageClass is a cluster infrastructure detail — the right value depends on what the target cluster has provisioned, not on the recipe. When provided, the value is written to all Helm value paths declared in the component registry under `storageClassPaths`, overriding any `storageClassName` set in recipe overlays. If a per-component `--set :=` explicitly targets the same path, that value takes precedence over `--storage-class`. **Example:** ```bash # Pin an explicit StorageClass on EKS (AICR's gp3-backed class) instead of relying on the cluster default aicr bundle --recipe recipe.yaml \ --storage-class ebs-csi-default-sc \ --output bundle # Use a custom storage class on an on-prem cluster aicr bundle --recipe recipe.yaml \ --storage-class local-path \ --output bundle ``` When `--storage-class` is not set, any `storageClassName` values already defined in the recipe overlays are preserved as defaults. When it is set, `--set :=` on the same path still wins — `--storage-class` only fills in paths that were not explicitly overridden. If a rendered component creates a PVC at a registry-declared `storageClassPaths` entry and no usable `storageClassName` is set after overlay, `--storage-class`, and `--set` precedence is resolved, `aicr bundle` emits a non-blocking warning. The bundle still relies on the target cluster's default StorageClass in that case. `aicr bundle` reports cluster-state dependencies it cannot verify as non-blocking warnings of this kind. Two more concern DRA eviction. Without `--dra-eviction-node-label`, and only where GPU Operator manages the driver, the bundle warns that automatic eviction was not configured and that a driver restart carries the documented risks. With the flag set, it reports that `dra-node-labeler` derives the label from `nvidia.com/gpu.present` and that a GPU node GFD has not labeled, and that does not already carry the configured label, runs without DRA; with the labeler disabled it warns instead that every GPU node must carry the configured label, that the label belongs in the node pool definition rather than an ad hoc `kubectl label`, and that unlabeled nodes silently run without DRA — no `ResourceSlices`, and `DESIRED=0` if no GPU node matches at all. These describe state AICR deliberately does not own — StorageClasses and node labels are cluster infrastructure. See [DRA Driver Upgrade Eviction](#dra-driver-upgrade-eviction). `--shared-storage-class` is a separate input for registry-declared `sharedStorageClassPaths`. It is used by opt-in Slinky Slurm PVCs mounted at `/home` and `/scratch/fsw`, which request `ReadWriteMany`; it does not affect existing generic storage paths. Enable the feature with `--set slinkyslurm:storage.enabled=true`. If either shared PVC has no effective class after per-component overrides and `--shared-storage-class` are applied, bundle creation fails instead of falling back to a potentially RWO-only class. See [Slurm Shared Storage](/aicr/user-guide/slurm-shared-storage). In contrast, when a bundle includes the `agentgateway` component with an empty or unset `allowedSourceRanges`, `aicr bundle` is **private by default**: it injects the RFC1918 private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`) into the inference-gateway's `loadBalancerSourceRanges` and records a bundle note, so the deployed gateway is reachable from inside the cluster/VPC but denied to the public internet — it is never emitted open to `0.0.0.0/0`. (Kubernetes treats an empty `loadBalancerSourceRanges` as allow-all, so the safe default has to be a real list.) An invalid value — a bare-string `--set`, a non-list, an unparseable CIDR, or a non-canonical CIDR such as `1.2.3.4/24` — is rejected with an error. To admit specific clients (e.g. a corporate VPN, which egresses from a public IP not covered by the default), scope it via a recipe `componentRef` override or the list-aware [`--set-json`](#list-and-object-value-overrides) flag (`agentgateway:allowedSourceRanges='[""]'`); to deliberately expose it publicly, opt in explicitly with `'["0.0.0.0/0"]'`, which generates with a loud warning. See [Inference Gateway Network Exposure](/aicr/user-guide/component-catalog#inference-gateway-network-exposure). #### Deployment Methods The `--deployer` flag controls how deployment artifacts are generated: | Method | Description | |--------|-------------| | `helm` | (Default) Generates Helm charts with values for deployment. Supports `--dynamic`. | | `argocd` | Generates Argo CD Application manifests for GitOps deployment. Does **not** support `--dynamic`. | | `argocd-helm` | Generates a Helm chart app-of-apps for Argo CD. All non-profile-owned values overridable at install time via `helm --set`; a profiled recipe ships a lock template that rejects overrides on profile-owned paths. Use `--dynamic` to pre-populate specific paths for components that resolve to remote Helm charts. `--dynamic` naming a local-chart or non-Helm component is rejected (those components bake values at bundle time and have no install-time stub surface). | | `flux` | Generates Flux HelmRelease manifests for GitOps deployment. Supports `--dynamic` via ConfigMap `valuesFrom`. | | `helmfile` | Generates a `helmfile.yaml` release graph driven by the upstream [helmfile](https://helmfile.readthedocs.io/) CLI (`helmfile apply` / `diff` / `destroy`). Supports `--dynamic` via per-release `cluster-values.yaml`. Requires the `helmfile` binary at deploy time. | > **Note:** `--dynamic` is not supported with `--deployer argocd`. Use `--deployer argocd-helm` instead, which produces a Helm chart where all non-profile-owned values are overridable at install time (a profiled recipe ships a lock template that rejects install-time values on profile-owned paths). > **Note:** `--dynamic` declarations targeting the GPU allocation-policy keys (`nvidia-dra-driver-gpu` `resources.gpus.enabled` / `gpuResourcesEnabledOverride`, `gpu-operator`(`-ocp`) `devicePlugin.enabled`, or those components' `enabled` toggle) are rejected: validators verify the recipe-resolved allocation policy, so its value cannot be deferred to install time. See [Configured GPU allocation policy](/aicr/user-guide/validation#configured-gpu-allocation-policy). > **Note:** `--dynamic` is rejected on a path that equals, contains, or is contained by a path a component's `requireNodeSelector` (or its conditional `requireNodeSelectorIfStorageClassSet` counterpart, e.g. `kube-prometheus-stack`) marks as required (e.g. `slinky-slurm`, `slurm-accounting-mariadb`), or by a declared `storageClassPaths`/`sharedStorageClassPaths` path that conditions `requireNodeSelectorIfStorageClassSet`. This holds regardless of whether a storage class is configured yet, since one could be added later without rebuilding the bundle. The flag defers the value to install time, the same unpinned state `requireNodeSelector` exists to reject. Supply `--system-node-selector` / `--accelerated-node-selector` instead. See [`nodeScheduling.system` vs `accelerated`](/aicr/contributor-guide/components#nodeschedulingsystem-vs-accelerated). **Deployment Order:** Ordering follows each component's declared dependencies (`dependencyRefs`), not its position in the bundle. Components with no dependency between them deploy **in parallel**; a component with a real dependency waits for that dependency to become healthy. The `NNN-/` folder numbers reflect a valid serialization for readability only. - **Helm**: `deploy.sh` installs one component at a time in deployment order (intentionally serial). - **Argo CD** / **Argo CD (Helm)**: `argocd.argoproj.io/sync-wave` annotation assigned by dependency depth. Independent components share a wave and sync together; a dependent lands in a later wave band that Argo starts only after the prior tier (including any readiness gate) is healthy. - **Flux**: `dependsOn` references in each `HelmRelease` mirror the component's declared `dependencyRefs` directly (a component with no dependencies has no `dependsOn` and reconciles in parallel). Pre/post manifests preserve the per-component chain `-pre → → -post`. The bundle's root `kustomization.yaml` is a plain Kustomize file (not a Flux Kustomization CR). - **Helmfile**: emits one `level-N.yaml` sub-helmfile per dependency tier, processed in sequence (so each tier's CRDs register before the next tier renders). Within a tier, `needs:` chains only a component's own `-pre → primary → -post` releases; independent components carry no edge, so helmfile applies them concurrently. Pass `--serial` to force every deployer to install strictly one component at a time in deployment order (reverts argocd/argocd-helm/flux/helmfile to a linear chain; helm is already serial). For the full model and per-deployer rationale, see [Deployment ordering](/aicr/contributor-guide/components#deployment-ordering). #### Value Overrides Override any value in the generated bundle files using dot notation: ```shell --set bundler:path.to.field=value ``` **Format:** `bundler:path=value` where: - `bundler` - Bundler name (e.g., `gpuoperator`, `networkoperator`, `certmanager`, `nodewright-operator`, `nvsentinel`) - `path` - Dot-separated path to the field - `value` - New value to set **Behavior:** - **Duplicate keys**: When the same `bundler:path` is specified multiple times, the **last value wins** - **Overlapping scalar paths**: Scalar paths are applied shallowest-first. If one `--set` assigns a non-map parent and another assigns one of its descendants (for example, `comp:driver=true` plus `comp:driver.enabled=false`), the request is rejected regardless of flag order. Use one coherent object through `--set-json` / `--set-file` when a parent and its nested keys must be supplied together. - **Array values**: Individual array elements cannot be overridden (no `[0]` index syntax). `--set` is **scalar-only** — pointing it at a list/object field writes a bare string and produces type-invalid output. To replace an entire array or object from the CLI, use [`--set-json` / `--set-file`](#list-and-object-value-overrides); recipe-level overrides in `componentRefs[].overrides` are the alternative. - **Type conversion**: String values are automatically converted to appropriate types (`true`/`false` → bool, numeric strings → numbers) - **Component enable/disable**: The special `enabled` key controls whether a component is included in the bundle. `--set :enabled=false` excludes a component the recipe enabled. A component the recipe **disabled** (`overrides.enabled: false`) cannot be re-enabled this way — `--set :enabled=true` on such a component is rejected, since re-enabling a platform-provided component would install a conflicting second copy. The `enabled` key is consumed by the bundler and not passed to Helm chart values. - **Aliases merge**: overrides supplied under both a component's canonical name and a registered alias (e.g. `gpu-operator` and `gpuoperator`) are **combined, not dropped**; the canonical name wins on any shared path. (Same alias-merge behavior as [`--set-json` / `--set-file`](#list-and-object-value-overrides).) - **GPU allocation-policy keys are deprecated at bundle time**: static overrides of the nested policy values — `nvidia-dra-driver-gpu` `resources.gpus.enabled` / `gpuResourcesEnabledOverride` and `gpu-operator`(`-ocp`) `devicePlugin.enabled` — still work via `--set`, `--set-json`, or `--set-file` but log a deprecation warning; the component-level `enabled` toggle of those components is honored **only via scalar `--set`** (the typed `--set-json`/`--set-file` path rejects `enabled` for every component, as described above) and likewise warns. Validators verify the recipe-resolved allocation policy, so a bundle-time change surfaces as recipe/cluster drift; move the allocation mode to a recipe overlay. `--dynamic` on any of these keys is **rejected** (the value would be unknowable when the policy is resolved). This boundary covers only what the bundler renders: **post-generation changes — `argocd-helm` / Argo CD parameter overrides, install-time `helm --set`, and manual edits to generated bundles — cannot be intercepted by AICR** and are outside the guarantee; they surface later as recipe/cluster drift when validation verifies the recipe-resolved policy. See [Configured GPU allocation policy](/aicr/user-guide/validation#configured-gpu-allocation-policy). - **Profile-owned paths are locked**: on a recipe carrying `metadata.selectedProfile` (ADR-015; the AKS, GKE, and OKE `gpuStack` families), a static override on a profile-owned path is accepted only when identical to the selected value — a divergent value is rejected, `--set-json`/`--set-file` are always rejected for an owned component's `enabled` presence key, and `--dynamic` is rejected on any intersection with an owned path. On GKE the lock additionally covers the closure-locked allocation-policy paths (`devicePlugin.enabled`, DRA `resources.gpus.enabled` / `gpuResourcesEnabledOverride`) — divergent overrides there are rejected rather than deprecation-warned; see [GKE GPU setup](/aicr/integrator-guide/gke-gpu-setup#gpu-device-plugin-ownership). - **Repeat to add; commas are literal**: To supply multiple overrides, repeat the flag (`--set a:x=1 --set b:y=2`). On the `bundle` command, commas inside a single slice-flag value are taken **literally** (not treated as a value separator), so a value containing a comma — and the comma-heavy JSON passed to `--set-json` — is preserved intact. This applies to all repeatable `bundle` flags (`--set`, `--set-json`, `--set-file`, `--dynamic`, `--*-node-selector`, `--*-node-toleration`, `--workload-selector`). **Examples:** ```shell # Generate all bundles aicr bundle --recipe recipe.yaml --output ./bundles # Override values in GPU Operator bundle aicr bundle -r recipe.yaml \ --set gpuoperator:gds.enabled=true \ --set gpuoperator:driver.version=570.86.16 \ -o ./bundles # Override multiple components aicr bundle -r recipe.yaml \ --set gpuoperator:mig.strategy=mixed \ --set networkoperator:sriovNetworkOperator.enabled=true \ -o ./bundles # Override cert-manager resources aicr bundle -r recipe.yaml \ --set certmanager:resources.limits.memory=512Mi \ --set certmanager:webhook.resources.limits.cpu=200m \ -o ./bundles # Override Nodewright manager resources aicr bundle -r recipe.yaml \ --set nodewright-operator:controllerManager.manager.resources.requests.cpu=500m \ --set nodewright-operator:controllerManager.manager.resources.limits.cpu=500m \ -o ./bundles # Disable a component at bundle time (e.g., EBS CSI already installed as EKS addon) aicr bundle -r recipe.yaml \ --set awsebscsidriver:enabled=false \ -o ./bundles ``` #### Argo CD Deployer Options The `deployer` prefix is reserved: with `--deployer argocd` or `--deployer argocd-helm`, `--set deployer:=` configures the generated Argo CD Applications instead of component chart values. Unknown `deployer:` keys are rejected, and the prefix is rejected entirely with any other `--deployer` type (`helm`, `flux`, `helmfile`). | Key | Applies to | Default | Example | |-----|------------|---------|---------| | `namePrefix` | Child Application names | (none) | `--set deployer:namePrefix=tenant-a-` | | `destinationServer` | Child Applications' `spec.destination.server` | `https://kubernetes.default.svc` | `--set deployer:destinationServer=https://prod.example.com:6443` | | `project` | Child Applications' `spec.project` | `default` | `--set deployer:project=gpu-infra` | | `cascadeDelete` | Parent and child Applications | `false` | `--set deployer:cascadeDelete=true` | **Child Applications only:** `namePrefix`, `destinationServer`, and `project` affect the per-component child Applications, not the parent app-of-apps. Application CRs are reconciled only from the cluster running Argo CD, so the parent stays on the control-plane cluster in project `default` — see the Argo CD [cluster bootstrapping guide](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/). The parent's name is set with [`--app-name`](#aicr-bundle). **Prerequisites for remote destinations:** AICR only writes `deployer:destinationServer` and `deployer:project` into the generated Applications — it does not configure Argo CD. The destination cluster must already be registered with Argo CD (`argocd cluster add ` or a [declarative cluster Secret](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters)), and the [Argo CD project](https://argo-cd.readthedocs.io/en/stable/user-guide/projects/) referenced by `deployer:project` must permit the emitted destinations and source repositories; otherwise the child Applications fail to sync with a permission or unknown-cluster error. **Name limits:** the composed child Application name (`namePrefix` + component name) must be at most 53 characters — Argo CD uses the Application name as the Helm release name, which Helm caps at 53 — and must not equal the parent Application's name (set with `--app-name`). Both violations are rejected at bundle time, and argocd-helm bundles re-check them at `helm template`/`helm install` time for install-time `--set deployer.namePrefix=...` overrides. **Install-time overrides with argocd-helm:** for `--deployer argocd-helm`, the three string keys ship as defaults in the bundle chart's root `values.yaml` and can also be overridden at install time via `helm install --set deployer.=...` (note the dot, not colon). The bundle ships a `values.schema.json` that Helm applies on install/upgrade/template/lint: unknown `deployer.*` keys (e.g. a `destinationSever` typo) and malformed values are rejected at install time instead of silently falling back to defaults. `cascadeDelete` is bundle-time only — it adds the [`resources-finalizer.argocd.argoproj.io` finalizer](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/) (a list field, not overridable via `--set`) so deleting an Application also deletes its deployed resources. **Children-only rendering (`deployer.includeRootApp`, install-time only):** argocd-helm bundles render the parent app-of-apps Application as a chart template, so an *externally managed* root Application pointed at the published chart — for example one created by a controller such as a Cluster API addon provider — would otherwise render a second parent (`aicr-stack`) that owns the same child Applications and fights the external root through its automated prune/selfHeal sync policy. Set `deployer.includeRootApp: false` in that root Application's `spec.source.helm.valuesObject` (or `helm install --set deployer.includeRootApp=false`) to render children-only. The default (`true`) keeps the standalone `helm install` flow unchanged; the same published bundle serves both consumption modes. This is a **boolean** key — pass it with plain `--set` (not `--set-string`), and note it is not a bundle-time `deployer:` key. **Use `--set-string` for values Helm would type-infer:** apart from the boolean `deployer.includeRootApp`, the schema is intentionally string-typed, and Helm's plain `--set` parses booleans and bare numbers into their inferred types — `helm install ... --set deployer.project=true` delivers a boolean, which the schema rejects. Pass such values with `--set-string` so they stay strings: `helm install ... --set-string deployer.project=true`. ```shell # Deploy child Applications to a remote cluster under a tenant prefix aicr bundle -r recipe.yaml --deployer argocd \ --set deployer:namePrefix=tenant-a- \ --set deployer:destinationServer=https://prod.example.com:6443 \ --set deployer:project=gpu-infra \ -o ./bundles # Enable cascading deletion on the app-of-apps and its children aicr bundle -r recipe.yaml --deployer argocd-helm \ --set deployer:cascadeDelete=true \ -o ./bundle # argocd-helm: override the shipped defaults at install time instead helm install aicr-stack ./bundle \ --set deployer.namePrefix=tenant-a- \ --set deployer.project=gpu-infra # argocd-helm: children-only — an external root Application already # points at the published chart (e.g. created by a controller) helm template aicr-bundle oci://ghcr.io/myorg/aicr-bundle \ --set repoURL=oci://ghcr.io/myorg \ --set deployer.includeRootApp=false ``` #### List and Object Value Overrides `--set` is scalar-only: it cannot express a list or object value. Pointing it at a list field such as `agentgateway.allowedSourceRanges` writes a **bare string** at that path, which the bundler rejects with an invalid-request error (it must be a list of CIDR strings). Use `--set-json` (inline) or `--set-file` (from a file) for any list or object override. **Format:** `component:path=` where: - `component` / `path` — same component name and dot-separated path as `--set` - `` — for `--set-json`, a JSON-encoded value; for `--set-file`, a path to a **regular file** containing a **single** JSON or YAML value (one document — in a multi-document YAML file only the first document is read). A non-regular path (directory, FIFO/named pipe, device, socket) is rejected with a fast validation error rather than hanging or failing later. **Behavior:** - **Object values deep-merge** into any existing map at the path (partial object overrides compose with recipe/base values), matching how inline recipe `componentRefs[].overrides` merge. Within a merged object, a JSON `null` **deletes** that key from the result (same explicit-null semantics as recipe overrides). - **Lists and scalars replace** the value at the path. - **Precedence**: applied after `--set`, so a `--set-json` / `--set-file` entry wins over a scalar `--set` on the same path. Between the two typed flags, an inline `--set-json` wins over a `--set-file` on the same `component:path` (mirroring Helm's `--set` taking precedence over `-f` value files). Within a single flag, the last entry for a given `component:path` wins. - **Overlapping nested paths**: when one override targets a parent object and another targets a key beneath it (e.g. `comp:driver.env=` plus `comp:driver.env.HTTPS_PROXY=`), the deeper, more-specific path wins on the keys they share — regardless of the order the flags are given. The parent object's other keys are preserved. - **Aliases merge**: overrides supplied under both a component's canonical name and a registered alias (e.g. `gpu-operator` and `gpuoperator`) are combined, not dropped; the canonical name wins on any shared path. - **Node-scheduling paths deep-merge with CLI injection (asymmetric with `--set`)**: a typed override on a node-scheduling path (e.g. `--set-json gpu-operator:nodeSelector=`) **deep-merges into** the selectors and tolerations injected by `--accelerated-node-selector` / `--system-node-selector` / `--*-node-toleration`, rather than suppressing that injection. This is intentional — deep-merge is the point of the typed path — but it differs from scalar `--set`: `--set comp:nodeSelector.x=y` marks that path as user-populated and suppresses CLI injection on it, whereas a typed override composes with the injected keys (so system-injected selector keys remain present alongside it). Use scalar `--set`, or omit the node-scheduling flags, when you need to fully replace an injected selector instead of merging into it. - **CLI-only**: `--set-json` / `--set-file` have no `AICRConfig` (`spec.bundle.deployment.set`) or HTTP API (`?set=`) equivalent — those surfaces remain scalar-only. To set a list or object value outside the CLI, use a recipe overlay or `componentRefs[].overrides`. - **Not for `enabled`**: the special `enabled` component toggle is honored only via scalar `--set`; passing it through `--set-json` / `--set-file` is rejected with an error (it would not toggle the component and would leak a stray `enabled:` value into the chart). **Examples:** ```shell # Scope the agentgateway inference-gateway to trusted CIDRs (inline JSON list) aicr bundle -r recipe.yaml \ --set-json agentgateway:allowedSourceRanges='["216.228.127.128/30","10.0.0.0/8"]' \ -o ./bundles # Same override, read from a file (JSON or YAML) cat > ranges.yaml <<'EOF' - 216.228.127.128/30 - 10.0.0.0/8 EOF aicr bundle -r recipe.yaml \ --set-file agentgateway:allowedSourceRanges=ranges.yaml \ -o ./bundles # Override object keys (deep-merges into the existing map) aicr bundle -r recipe.yaml \ --set-json gpuoperator:driver.env='{"HTTPS_PROXY":"http://proxy:3128"}' \ -o ./bundles ``` ```shell # Schedule system components on specific node pool aicr bundle -r recipe.yaml \ --system-node-selector nodeGroup=system-pool \ --system-node-toleration dedicated=system:NoSchedule \ -o ./bundles # Schedule GPU workloads on labeled GPU nodes aicr bundle -r recipe.yaml \ --accelerated-node-selector nvidia.com/gpu.present=true \ --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \ -o ./bundles # Combined: separate system and GPU scheduling aicr bundle -r recipe.yaml \ --system-node-selector nodeGroup=system-pool \ --system-node-toleration dedicated=system:NoSchedule \ --accelerated-node-selector accelerator=nvidia-h100 \ --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \ -o ./bundles # Set estimated GPU node count (writes to nodeCountPaths in registry) aicr bundle -r recipe.yaml --nodes 8 -o ./bundles # Day 2 options: workload-gate and workload-selector for nodewright aicr bundle -r recipe.yaml \ --workload-gate nodewright.nvidia.com=runtime-required:NoSchedule \ --workload-selector workload-type=training \ -o ./bundles # Generate an attested bundle (opens browser for OIDC auth) aicr bundle -r recipe.yaml --attest -o ./bundles # In GitHub Actions (OIDC token detected automatically) aicr bundle -r recipe.yaml --attest -o ./bundles # Sign with a KMS-backed key in CI/CD without OIDC (Jenkins, internal pipelines) aicr bundle --recipe recipe.yaml --attest \ --signing-key awskms://arn:aws:kms:us-east-1:123456789012:key/abcd-1234 \ --output ./bundles # Generate Argo CD Application manifests for GitOps aicr bundle -r recipe.yaml --deployer argocd -o ./bundles # Argo CD with Git repository URL (avoids placeholder in app-of-apps.yaml) aicr bundle -r recipe.yaml --deployer argocd \ --repo https://github.com/my-org/my-gitops-repo.git \ -o ./bundles # Combine deployer with value overrides aicr bundle -r recipe.yaml \ --deployer argocd \ -o ./bundles ``` #### Vendoring Charts for Air-Gap The `--vendor-charts` flag pulls upstream Helm chart bytes into the bundle at bundle time. With the flag set, every Helm-typed component becomes a local chart inside the generated bundle and the resulting artifact deploys end-to-end with zero registry egress. Without the flag, deploy-time `helm upgrade --install` calls fetch from the upstream repository — which works for connected clusters but breaks in air-gapped environments. **Bundle-time requirement:** the `helm` binary must be on `$PATH` when `aicr bundle --vendor-charts` runs. Authentication for private chart registries flows through Helm's own conventions: - **HTTP(S) repositories** — `HELM_REPOSITORY_USERNAME` / `HELM_REPOSITORY_PASSWORD` environment variables. - **OCI registries** — standard docker config (`~/.docker/config.json` or `$DOCKER_CONFIG`); run `docker login ` ahead of time. **Tradeoff: CVE-yank fail-loud signal is lost.** Non-vendored bundles fail loudly when an upstream chart version is yanked at registry time, which prompts a rebundle with a fixed recipe. Vendored bundles freeze the chart bytes at bundle creation and silently install the frozen version even after upstream yank. Treat `provenance.yaml` (below) as the audit surface for cross-referencing yank lists. **Bundle-time costs.** Vendoring adds bundle-time network egress (the chart pull), bundle-time auth surface (private registries need credentials at the bundle host), and bundle size (typically 0.5–5 MB unpacked per chart). Users who don't need air-gap shouldn't set `--vendor-charts` and shouldn't pay these costs. **Bundle layout with `--vendor-charts`** — every Helm component emits a wrapper folder holding the vendored tarball. Mixed components keep the primary + `-post` split they have on the non-vendored path, so recipe-side manifests stay tracked members of their own Helm release: ```text my-bundle/ 001-kube-prometheus-stack/ Chart.yaml # wrapper, declares the vendored subchart charts/kube-prometheus-stack-vXX.Y.Z.tgz # vendored upstream tarball values.yaml # values nested under the subchart name cluster-values.yaml # dynamic values, also nested install.sh # helm upgrade --install ./ ... 002-gpu-operator/ Chart.yaml charts/gpu-operator-v26.7.1.tgz values.yaml cluster-values.yaml install.sh 003-gpu-operator-post/ # mixed component: recipe-side manifests, Chart.yaml # emitted immediately after its primary templates/ # plain local chart, no vendored tarball clusterrole.yaml # ordinary template, no helm.sh/hook values.yaml cluster-values.yaml install.sh provenance.yaml # bundle-time audit log ... ``` **`provenance.yaml`** sits at the bundle root and lists one entry per vendored chart, using the same K8s-style `apiVersion`/`kind` shape as the rest of AICR's persisted formats: ```yaml apiVersion: aicr.run/v1 kind: BundleProvenance vendoredCharts: - name: gpu-operator chart: gpu-operator version: v26.7.1 repository: https://helm.ngc.nvidia.com/nvidia sha256: abc123... tarballName: gpu-operator-v26.7.1.tgz pullerVersion: helm-cli v3.20.2 ``` The `sha256` field is the digest of the bytes copied into `charts/`, suitable for yank-list lookups and cross-bundle drift comparisons. Pipe through `yq -o=json provenance.yaml` if your scanner expects JSON. **Examples:** ```bash # Vendor everything for an air-gap deployment aicr bundle --recipe recipe.yaml --vendor-charts -o ./bundle # Vendor with private OCI registry credentials docker login nvcr.io aicr bundle --recipe recipe.yaml --vendor-charts -o ./bundle # Vendor with private HTTP(S) chart repo credentials HELM_REPOSITORY_USERNAME=robot \ HELM_REPOSITORY_PASSWORD=secret \ aicr bundle --recipe recipe.yaml --vendor-charts -o ./bundle ``` #### Readiness Gates The `--readiness-hooks` flag makes a deploy block on **component-specific readiness signals** rather than just the chart's own resources reporting Ready. A component opts in by shipping a `recipes/components//readiness.yaml` Chainsaw test that asserts the signal that actually means "ready" — for example, `gpu-operator` waits for its `ClusterPolicy` to reach `status.state: ready`, which Helm and Argo CD cannot assess natively. With the flag set, the bundler emits an extra folder, `NNN--readiness/`, immediately after each opted-in component. The folder is a small chart containing a Kubernetes `Job` (plus the ServiceAccount/RBAC and a ConfigMap holding the Chainsaw test). The Job runs the `gate` CLI (`ghcr.io/nvidia/aicr-gate`), which polls the test until it passes continuously for a stability window or a `--max-wait` ceiling elapses. The deploy blocks on that Job: - **`helm`** — `deploy.sh` runs the readiness folder with `helm upgrade --install --wait`. The gate Job is a `post-install,post-upgrade` hook, and `--wait` blocks on hook completion regardless of `--wait-for-jobs`, so the latter is not needed. Helm's own `--timeout` is derived by the bundler from the gate's `--max-wait` plus a buffer, so the gate owns the deadline (Helm never preempts it). - **`argocd` / `argocd-helm`** — the readiness folder inherits the next sync-wave after its component, and Argo CD blocks that wave on the gate Job via its built-in `batch/Job` health (Progressing → Healthy on success, Degraded on failure). No custom health Lua and no direct `ClusterPolicy` watch — the readiness logic stays encapsulated in the Chainsaw test the Job runs. `flux` and `helmfile` are not yet supported and `--readiness-hooks` is rejected for them. Components without a `readiness.yaml` are unaffected. The gate only reads API objects, so it needs a healthy node rather than the gated component's node. Its pod carries the bundle's system node selector (`--system-node-selector`, or `scheduling.systemNodeSelector` in the config file) and its keyed system tolerations (`--system-node-toleration`, or `scheduling.systemNodeTolerations`). Keyless tolerations are not applied to the gate — that includes the tolerate-all default the CLI and API use when no toleration is set, and an explicit `*` — because a toleration with no key also matches the not-ready, unreachable and cordoned taints, which would let the gate bind to an unhealthy node. On a cluster whose every node is tainted, pass a keyed `--system-node-toleration` for the system nodes' taint so the gate can schedule ([#2590](https://github.com/NVIDIA/aicr/issues/2590)). Placement does not follow the gated component's own `--set` or `--dynamic` scheduling values. Go SDK callers get only what they set with `config.WithSystemNodeSelector` / `config.WithSystemNodeTolerations`. Selector and toleration values are checked against the API server's rules when the bundle is generated, and an invalid one fails the bundle; `Lt` and `Gt` tolerations are rejected. The gate Job also sets `activeDeadlineSeconds` (the gate's maximum wait plus 3 minutes), so a gate pod that cannot schedule or start fails the Job instead of holding the Helm hook or Argo CD sync open. The gate evaluates the test **in-process**: it reads cluster state through its own ServiceAccount and applies the assertions itself, using the same executor `aicr validate --phase deployment` uses. The image ships no Chainsaw binary. Two independent controls keep a readiness test read-only. First, the executor honors only the `assert` and `error` operations — every state-changing or side-effecting operation is rejected before evaluation, and the check fails. Second, the gate's ServiceAccount is bound to a ClusterRole granting only `get`, `list`, and `watch`, so a mutating call would be denied by the API server even if one were somehow issued. ```bash # Deploy and block on each component's readiness gate (helm) aicr bundle --recipe recipe.yaml --readiness-hooks -o ./bundle # Same, generating an Argo CD app-of-apps that gates per sync-wave aicr bundle --recipe recipe.yaml --readiness-hooks --deployer argocd -o ./bundle ``` #### Dynamic Install-Time Values The `--dynamic` flag declares value paths that are cluster-specific and should be provided at install time rather than baked into the bundle at build time. This enables building a single bundle that can be deployed to multiple clusters with different configurations. Use `--dynamic` for values that genuinely vary per cluster — cluster names, subnet IDs, endpoint URLs, region-specific settings. For values that are static per bundle but differ from the recipe default (e.g., a specific driver version), use `--set` instead. | Use case | Flag | Example | |----------|------|---------| | Cluster-specific value (varies per deployment) | `--dynamic` | `--dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.cluster` | | Static override (same for all deployments of this bundle) | `--set` | `--set gpuoperator:driver.version=580.105.08` | > **Attestation scope:** Dynamic values are supplied at install time and are > **not covered by `--attest`**. Attestation binds the generated closed-world > inventory, including `recipe.yaml`, not operator-provided overrides. If you > need to constrain dynamic values at deploy time, use admission control or > Argo sync hooks — see [Attestation Scope](#attestation-scope). ```shell --dynamic component:path.to.field ``` **Format:** `component:path` where: - `component` - Component name or override key (same keys as `--set`, e.g., `gpuoperator`, `kubeprometheusstack`) - `path` - Dot-separated path to the value that varies per cluster **Helm deployer behavior:** Dynamic paths are removed from `values.yaml` and written to the component's `cluster-values.yaml`. Every Helm folder carries a `cluster-values.yaml` (an empty document when none of its paths are dynamic), and each folder's `install.sh` passes both files to Helm: ```shell helm upgrade --install gpu-operator ... \ -f values.yaml \ -f cluster-values.yaml ``` Before deploying, fill in `cluster-values.yaml` with cluster-specific values. **Argo CD deployer behavior:** The `--deployer argocd-helm` generates a Helm chart app-of-apps where all non-profile-owned values are overridable at install time. When the recipe carries a selected profile, `templates/aicr-profile-lock.yaml` fails the install if a value is supplied for a profile-owned path. Static values are baked into the chart as files; dynamic overrides are merged on top at render time. Use `--dynamic` to pre-populate specific paths in the root `values.yaml` for components that resolve to remote Helm charts. Local-chart components (Helm with no upstream `Source`, common on OCP overlays) and non-Helm components have no install-time stub surface — `--dynamic` naming them is rejected rather than silently dropped. ```shell helm install aicr-bundle ./bundle \ --set prometheus.prometheus.prometheusSpec.externalLabels.cluster=prod-east \ --set prometheus.prometheus.prometheusSpec.externalLabels.region=us-east-1 ``` **Examples:** ```shell # Helm: declare cluster name as install-time parameter aicr bundle -r recipe.yaml \ --dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.cluster \ -o ./bundles # Helm: multiple dynamic paths across components aicr bundle -r recipe.yaml \ --dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.cluster \ --dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.region \ -o ./bundles # Helm: combine with --set (static overrides + dynamic cluster-specific values) aicr bundle -r recipe.yaml \ --set gpuoperator:driver.version=580.105.08 \ --dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.cluster \ -o ./bundles # Argo CD Helm chart: all non-profile-owned values overridable, --dynamic pre-populates specific paths aicr bundle -r recipe.yaml \ --deployer argocd-helm \ --dynamic kubeprometheusstack:prometheus.prometheusSpec.externalLabels.cluster \ -o ./bundles # Argo CD Helm chart: without --dynamic, non-profile-owned values still overridable via helm --set aicr bundle -r recipe.yaml \ --deployer argocd-helm \ -o ./bundles ``` **Bundle structure with `--dynamic`** (Helm deployer): ``` bundles/ ├── 008-kube-prometheus-stack/ │ ├── install.sh # helm upgrade --install ... -f values.yaml -f cluster-values.yaml │ ├── values.yaml # Static values (the dynamic path removed) │ └── cluster-values.yaml # Dynamic values (override before deploying) ├── 009-gpu-operator/ │ ├── install.sh │ ├── values.yaml │ └── cluster-values.yaml # No dynamic values: empty document ├── deploy.sh # Runs each folder's install.sh └── ... ``` **Bundle structure with `--dynamic`** (Flux deployer): The `--deployer flux` bundle uses Flux's native `spec.valuesFrom` to reference ConfigMaps containing dynamic values. Dynamic paths are removed from the inline `spec.values` and placed into a ConfigMap per component. Flux merges `valuesFrom` first, then inline values on top — since dynamic paths are stripped from inline values, the ConfigMap values take effect without conflicts. ```text bundles/ ├── gpu-operator/ │ ├── helmrelease.yaml # HelmRelease with valuesFrom + inline values │ └── configmap-values.yaml # Dynamic values ConfigMap (edit before applying) ├── cert-manager/ │ └── helmrelease.yaml # No dynamic values, no ConfigMap ├── sources/ │ └── ... ├── kustomization.yaml └── README.md ``` Before applying the bundle to your cluster, edit each `configmap-values.yaml` with the correct per-cluster values: ```shell # 1. Generate the bundle aicr bundle -r recipe.yaml --deployer flux \ --dynamic gpuoperator:driver.version \ --repo https://github.com/my-org/gitops.git \ -o ./bundles # 2. Edit dynamic ConfigMaps vim bundles/gpu-operator/configmap-values.yaml # 3. Push to your Git repository and let Flux reconcile git add bundles/ && git commit -m "Add AICR bundle" && git push ``` **Bundle structure with `--dynamic`** (Helmfile deployer): The `--deployer helmfile` bundle references both `values.yaml` (static) and `cluster-values.yaml` (dynamic stubs) per release. `helmfile` merges value files in declaration order, so `cluster-values.yaml` overrides on top of the generated `values.yaml`. Edit `cluster-values.yaml` per component before `helmfile apply`: ```text bundles/ ├── helmfile.yaml # Release graph; per-release values: [./NNN-/values.yaml, ./NNN-/cluster-values.yaml] ├── 001-cert-manager/ │ ├── values.yaml # Generated static values │ └── cluster-values.yaml # Dynamic stubs (edit before apply) ├── 002-gpu-operator/ │ ├── values.yaml │ └── cluster-values.yaml └── README.md ``` ```shell # 1. Generate the bundle aicr bundle -r recipe.yaml --deployer helmfile \ --dynamic gpuoperator:driver.version \ -o ./bundles # 2. Edit per-cluster overrides vim bundles/002-gpu-operator/cluster-values.yaml # 3. Preview and apply cd ./bundles && helmfile diff && helmfile apply ``` **Argo CD Helm chart structure with `--dynamic`:** The `--deployer argocd-helm` bundle is itself a Helm chart whose `templates/` create per-component Argo Applications. Each application's `helm.values` block merges static values (loaded via `.Files.Get` for upstream-helm components, or read from the wrapped chart's own `values.yaml` for local-chart components) with dynamic overrides from the parent chart's `.Values`. The same uniform `NNN-/` folder layout used by `--deployer argocd` is included at the bundle root so that path-based Argo Applications (manifest-only, kustomize-wrapped, mixed `-post`) can resolve their `path:` references against the OCI-published bundle. ```text bundles/ ├── Chart.yaml # Parent chart metadata ├── values.yaml # Dynamic stubs only (per-cluster surface) ├── templates/ │ ├── aicr-stack.yaml # Parent Argo Application (renders all children) │ ├── cert-manager.yaml # Argo App, multi-source (upstream-helm) │ ├── gpu-operator.yaml # Argo App, multi-source │ ├── gpu-operator-post.yaml # Argo App, path-based (mixed -post) │ └── nodewright-customizations.yaml # Argo App, path-based (manifest-only) ├── static/ │ ├── cert-manager.yaml # Static values for upstream-helm Applications │ └── gpu-operator.yaml ├── 001-cert-manager/ # NNN-folder content (KindUpstreamHelm) │ └── values.yaml ├── 002-gpu-operator/ # KindUpstreamHelm (mixed primary) │ └── values.yaml ├── 003-gpu-operator-post/ # KindLocalHelm (mixed -post) │ ├── Chart.yaml │ ├── templates/ │ └── values.yaml ├── 004-nodewright-customizations/ # KindLocalHelm (manifest-only) │ ├── Chart.yaml │ ├── templates/ │ └── values.yaml └── README.md ``` Manifest-only components and mixed-component raw manifests are supported by `--deployer argocd-helm` via the path-based Application shape. `static/` holds per-component values for upstream-helm Applications only. A recipe whose components are all local charts contributes no such files, and the directory is omitted from the bundle entirely rather than emitted empty. Most OpenShift (OCP) overlay components are local charts (OLM installer + CR pairs), but a few — components with no certified OCP operator, such as `prometheus-adapter-ocp` and `nvidia-dra-driver-gpu-ocp` — reuse their upstream Helm chart and do contribute a `static/` values file. **The bundle's `repoURL` defaults to the registry it was pushed to.** No `--repo` flag is needed (and is ignored if passed with `--deployer argocd-helm`). When pushed to an OCI registry, the parent namespace is baked into `values.yaml` as the default `repoURL` — a plain `helm install` works with no `--set repoURL` needed. Override with `--set repoURL=oci://mirror` when deploying from a different registry. **Recommended deploy flow:** ```shell # 1. Generate the bundle (URL-agnostic) aicr bundle -r recipe.yaml --deployer argocd-helm --dynamic gpuoperator:driver.version -o ./bundle # 2. Publish to your chart registry (any HTTPS-capable OCI / Helm chart repo) helm package ./bundle -d /tmp/ helm push /tmp/aicr-bundle-*.tgz oci:/// # 3. Install — the URL is supplied here, not at bundle time # `--set repoURL` is the PARENT NAMESPACE (no trailing chart name). # The parent Application appends `.Chart.Name` into its OCI # `source.repoURL`, and path-based children append it directly into # their rendered `source.repoURL`. For non-OCI Helm repositories, the # parent uses `source.chart` instead. Including the chart name in # --set repoURL double-appends it and the children fail to resolve. helm install aicr-bundle oci:////aicr-bundle --version \ -n argocd \ --set repoURL=oci:/// \ --set targetRevision= ``` The chart's `templates/aicr-stack.yaml` renders the parent Argo Application with `.Values.repoURL` and `.Values.targetRevision` substituted in. The parent Application then triggers Argo to render the chart again from the OCI source, creating the per-component child Applications with sync-wave ordering preserved. Child Applications whose source is path-based (manifest-only and mixed-component `-pre` / `-post` folders) inherit `.Values.repoURL` and append `.Chart.Name` so they pull from the same published artifact as the parent. **Argo CD OCI prerequisites.** Path-based child Applications use Argo CD's generic OCI artifact source type (introduced in Argo CD v2.13). The argocd-helm bundle therefore requires: - Argo CD **≥ v2.13** on the target cluster. - A registry that serves Helm-pushed OCI artifacts through the generic OCI manifest fetch path (most modern registries — ECR, GHCR, GAR, Harbor, Artifactory, plain `oras`-compatible registries — support this). If the recipe is pure-Helm (no manifest-only / mixed components), path-based children are not exercised and the bundle can work on Argo CD versions older than v2.13. If path-based children are present, Argo CD v2.13+ is required. See the troubleshooting section below if `Failed to load target state` appears on `aicr-stack` or any `-pre` / `-post` Application. **`helm install ./bundle` from a local directory** *also* works, but with a caveat: child Applications whose source is path-based require Argo's repo-server to fetch the bundle from a remote (git or OCI) — there is no local-filesystem source type for an Argo Application. Local `helm install` is therefore end-to-end only when the recipe contains pure-Helm components. For everything else, publish first. **Bundle structure** (with default Helm deployer): ``` bundles/ ├── README.md # Deployment guide with ordered steps ├── UPGRADING.md # Present only when a pinned version needs upgrade guidance ├── deploy.sh # Generic install loop + name-matched blocks ├── recipe.yaml # Recipe used to generate bundle ├── bundle-info.yaml # Deployer and generation metadata ├── checksums.txt # SHA256 checksums ├── attestation/ # Present when --attest is used │ ├── bundle-attestation.sigstore.json # SLSA Build Provenance v1 │ └── aicr-attestation.sigstore.json # Binary SLSA provenance chain ├── 001-cert-manager/ # Upstream-helm folder: no Chart.yaml │ ├── install.sh # Rendered: helm upgrade --install ... --repo ${REPO} │ ├── values.yaml │ ├── cluster-values.yaml # Dynamic-path overrides (operator-edited) │ └── upstream.env # CHART, REPO, VERSION (sourced by install.sh) ├── 002-network-operator/ # Mixed component primary (upstream-helm) │ ├── install.sh │ ├── values.yaml │ ├── cluster-values.yaml │ └── upstream.env └── 003-network-operator-post/ # Injected -post wrapped chart (mixed component's raw manifests) ├── Chart.yaml # Local-helm folder: Chart.yaml + templates/ present ├── install.sh # Rendered: helm upgrade --install ... ./ ├── values.yaml ├── cluster-values.yaml └── templates/ └── nfd-network-rule.yaml ``` **Folder layout rules:** - Folders are numbered `NNN-/` (1-based, zero-padded). Numbering is regenerated on every bundle. - Each folder is one of two **kinds**, distinguished by the presence of `Chart.yaml`: - **upstream-helm** — no `Chart.yaml`; `upstream.env` carries `CHART`/`REPO`/`VERSION`; `install.sh` installs the upstream chart. - **local-helm** — `Chart.yaml` + `templates/`; `install.sh` installs the local chart (`helm upgrade --install ./`). - **Mixed components** (Helm chart + raw manifests) emit **two adjacent folders**: a primary upstream-helm `NNN-/` and an injected `(NNN+1)--post/` local-helm wrapper carrying the raw manifests. Subsequent components shift by one. - Components with pre-install manifests (`preManifestFiles`) also get an injected `NNN--pre/` local-helm wrapper immediately before the primary folder. - Manifest-only components (no upstream Helm chart, just raw manifests) become a single local-helm wrapped chart. - Kustomize-typed components run `kustomize build` at bundle time; the output becomes a single `templates/manifest.yaml` inside a local-helm folder. **Breaking change vs. earlier releases:** Previous releases used a flat `/` layout with `manifests/` siblings and a `--deployer helm` script that branched on component kind. The new format is uniform: - All folders carry a rendered `install.sh`. The top-level `deploy.sh` is a generic loop with no per-component branching — name-matched special-case blocks (nodewright-operator taint cleanup, kai-scheduler async timeout, orphan-CRD scan, DRA kubelet-plugin restart) live around the loop, not inside it. - Raw manifests for mixed components now apply **post-install only**, via the injected `-post` wrapped chart. The earlier pre-apply mechanism with a CRD-race retry wrapper is gone — Helm now owns CRD ordering for mixed components natively. - Tooling that parsed bundle paths by bare component name must account for the `NNN-` prefix. **Argo CD bundle structure** (with `--deployer argocd`): The argocd deployer uses the same uniform `NNN-/` folder layout as `--deployer helm`. Each folder carries an `application.yaml` whose Application shape is decided by the folder kind: - **`Chart.yaml` absent** (KindUpstreamHelm — pure Helm components): today's multi-source Application pointing at the upstream Helm repository plus a values $ref to the user's git repo. Unchanged for current users. - **`Chart.yaml` present** (KindLocalHelm — manifest-only, kustomize-wrapped, mixed `-post`): single-source path-based Application with `source.path: NNN-` against the user's repo. The argocd deployer emits only what Argo CD's repo-server consumes: `application.yaml`, `values.yaml` (multi-source `helm.valueFiles` for upstream-helm, or local-chart Helm rendering for KindLocalHelm), and `Chart.yaml`/`templates/` for KindLocalHelm. The helm-deployer orchestration files (`install.sh`, `upstream.env`, `cluster-values.yaml`) are stripped — Argo doesn't run shell scripts or source shell env, and `--dynamic` is rejected with `--deployer argocd` (use `--deployer argocd-helm` for install-time values). ```text bundles/ ├── app-of-apps.yaml # Parent Application (recurses *.application.yaml) ├── 001-cert-manager/ # KindUpstreamHelm — no Chart.yaml │ ├── values.yaml # Static Helm values (consumed via multi-source) │ └── application.yaml # Multi-source Application (sync-wave 1, tier 0) ├── 002-gpu-operator/ # KindUpstreamHelm — primary of mixed │ ├── values.yaml │ └── application.yaml ├── 003-gpu-operator-post/ # KindLocalHelm — injected mixed -post │ ├── Chart.yaml # Synthesized wrapper for raw manifests │ ├── templates/ # Rendered manifests │ ├── values.yaml │ └── application.yaml # Path-based Application (sync-wave 2) ├── 004-nodewright-customizations/ # KindLocalHelm — manifest-only │ ├── Chart.yaml │ ├── templates/ │ ├── values.yaml │ └── application.yaml └── README.md ``` Manifest-only components (e.g., `nodewright-customizations`) and mixed-component raw manifests (the `-post` injection) are now deployed by `--deployer argocd`. Previously they were silently dropped. Set `--repo ` to populate the `repoURL` on path-based Applications so Argo can resolve them. **Day 2 Options:** The `--workload-gate` and `--workload-selector` flags are day 2 operational options for cluster scaling operations: - **`--workload-gate`**: Specifies a taint for nodewright-operator's runtime required feature. This ensures nodes are properly configured before workloads can schedule on them during cluster scaling. The taint is configured in the nodewright-operator Helm values file at `controllerManager.manager.env.runtimeRequiredTaint`. When the flag is omitted the chart default applies: `nodewright.nvidia.com=runtime-required:NoSchedule` from operator v0.18.0 (previously `skyhook.nvidia.com=runtime-required:NoSchedule`, which the operator still recognizes and removes until v0.20.0). Node pools that pre-taint with the legacy key should pass `--workload-gate skyhook.nvidia.com=runtime-required:NoSchedule` so auto-tainted and pre-tainted nodes carry the same key. `aicr validate --phase deployment` reads the configured taint from the operator Deployment, so whatever value is passed here is what the readiness gate waits to see cleared. For more information about runtime required, see the [Nodewright documentation](https://github.com/NVIDIA/nodewright/blob/main/docs/user-guide/runtime-required.md). - **`--workload-selector`**: Specifies a label selector for nodewright-customizations to prevent nodewright from evicting running training jobs. This is critical for training workloads where job eviction would cause significant disruption. The selector is set in the Skyhook CR manifest (tuning.yaml) in the `spec.workloadSelector.matchLabels` field. **Estimated node count (`--nodes`):** The `--nodes` flag is a **bundle-time** option: it is applied when you run `aicr bundle`, not when you run `aicr recipe`. The value is written to each component's Helm values at the paths declared in the registry under `nodeScheduling.nodeCountPaths`. - **When to use**: Pass the expected or typical number of GPU nodes (e.g. size of your node pool). Use `0` (default) to leave the value unset. - **Where it goes**: Components that define `nodeCountPaths` in the registry receive the value at those paths in their generated `values.yaml`. - **Example**: `aicr bundle -r recipe.yaml --nodes 8 -o ./bundles` writes `8` to every path listed in each component's `nodeScheduling.nodeCountPaths`. **Component Validation System:** AICR includes a component-driven validation system that automatically checks bundle configuration and displays warnings or errors during bundle generation. Validations are defined in the component registry and run automatically when components are included in a recipe. **How Validations Work:** 1. **Automatic Execution**: When generating a bundle, validations are automatically executed for each component in the recipe 2. **Condition-Based**: Validations can be configured to run only when specific conditions are met (e.g., intent, service, accelerator) 3. **Severity Levels**: Each validation can be configured as a "warning" (non-blocking) or "error" (blocking) 4. **Custom Messages**: Each validation can include an optional detail message that provides actionable guidance **Validation Warnings:** When generating bundles with nodewright-customizations enabled, validation warnings are displayed for missing configuration: 1. **Workload Selector Warning**: When nodewright-customizations is enabled with training intent, if `--workload-selector` is not set, a warning will be displayed: ``` Warning: nodewright-customizations is enabled but --workload-selector is not set. This may cause nodewright to evict running training jobs. Consider setting --workload-selector to prevent eviction. ``` 2. **Accelerated Selector Warning**: When nodewright-customizations is enabled with training or inference intent, if `--accelerated-node-selector` is not set, a warning will be displayed: ``` Warning: nodewright-customizations is enabled but --accelerated-node-selector is not set. Without this selector, the customization will run on all nodes. Consider setting --accelerated-node-selector to target specific nodes. ``` **Viewing Validation Warnings:** Validation warnings are displayed in the bundle output after successful generation: ```shell Note: ⚠ Warning: nodewright-customizations is enabled but --workload-selector is not set. This may cause nodewright to evict running training jobs. Consider setting --workload-selector to prevent eviction. ⚠ Warning: nodewright-customizations is enabled but --accelerated-node-selector is not set. Without this selector, the customization will run on all nodes. Consider setting --accelerated-node-selector to target specific nodes. ``` **Resolving Validation Warnings:** To resolve the warnings, include the appropriate flags when generating the bundle: ```shell # Resolve workload selector warning aicr bundle -r recipe.yaml \ --workload-selector workload-type=training \ -o ./bundle # Resolve accelerated selector warning aicr bundle -r recipe.yaml \ --accelerated-node-selector nodeGroup=gpu-worker \ -o ./bundle # Resolve both warnings aicr bundle -r recipe.yaml \ --workload-selector workload-type=training \ --accelerated-node-selector nodeGroup=gpu-worker \ -o ./bundle ``` **Examples:** ```shell # Generate bundle with day 2 options for training workloads aicr bundle -r recipe.yaml \ --workload-gate nodewright.nvidia.com=runtime-required:NoSchedule \ --workload-selector workload-type=training \ --workload-selector intent=training \ --accelerated-node-selector accelerator=nvidia-h100 \ -o ./bundles # Generate bundle for inference workloads with accelerated selector aicr bundle -r recipe.yaml \ --accelerated-node-selector accelerator=nvidia-h100 \ -o ./bundles ``` Argo CD Applications use multi-source to: 1. Pull Helm charts from upstream repositories 2. Apply values.yaml from your GitOps repository Raw manifests (manifest-only components and mixed-component `-post` folders) are deployed by separate path-based Applications, as described above. #### Flux OCI Mode When using `--deployer flux` with OCI output (`--output oci://...`), AICR generates ArtifactGenerator and ExternalArtifact CRs instead of GitRepository sources for local-chart components. This allows Flux to reconcile HelmReleases directly from OCI artifacts without a Git repository. **Prerequisites (Flux v2.7+):** - **source-watcher controller** must be deployed (`source.extensions.fluxcd.io`). This controller watches ArtifactGenerator CRs and creates ExternalArtifact objects. - **ExternalArtifact=true feature gate** must be enabled on helm-controller. This allows HelmRelease CRs to reference ExternalArtifact objects via `spec.chartRef`. Without both prerequisites, bundles generate successfully but HelmReleases will not reconcile at deploy time. **Configuration flags:** | Flag | Default | Description | |------|---------|-------------| | `--flux-oci-source-name` | `aicr-bundle` | Name of the OCIRepository CR in the target cluster. Every generated ArtifactGenerator references this name in `spec.sources[0].name`. | | `--flux-namespace` | `flux-system` | Namespace where all Flux CRs (HelmRelease, sources, ArtifactGenerator) are placed. | ```shell # Generate an OCI bundle with a custom OCIRepository name and namespace aicr bundle -r recipe.yaml --deployer flux \ --output oci://ghcr.io/my-org/aicr-bundle:v1.0.0 \ --flux-oci-source-name my-oci-repo \ --flux-namespace gitops ``` The generated ArtifactGenerator CRs extract per-component chart directories from the outer OCIRepository into ExternalArtifact objects. Each HelmRelease then references the ExternalArtifact via `spec.chartRef` instead of the traditional `spec.chart.spec.sourceRef` pointing at a GitRepository. #### Bundle Attestation > **Prerequisite:** `--attest` needs the binary attestation `aicr-attestation.sigstore.json` beside the `aicr` binary. The install script and the release archives both include it (keep it next to `aicr` when installing manually); binaries from `go install` lack it and cannot use `--attest`. When `--attest` is passed, the bundle command performs five steps: 1. **Verifies the binary attestation file exists** — The running `aicr` binary must have a valid SLSA provenance file (`aicr-attestation.sigstore.json`) alongside it, included by the install script from a release archive. If missing, the command fails immediately with guidance on how to install correctly. 2. **Acquires a signing credential** — in the default keyless mode this is an OIDC token (see [OIDC Token Sources](#oidc-token-sources) below); with `--signing-key` this step instead resolves the KMS key and no OIDC token is acquired (see [KMS-Backed Signing](#kms-backed-signing)). 3. **Verifies the binary's own attestation** — Cryptographically verifies the SLSA provenance binds to the running binary and was signed by NVIDIA CI. This ensures only NVIDIA-built binaries can produce attested bundles. 4. **Signs the bundle** — Creates a SLSA Build Provenance v1 in-toto statement binding the creator's identity to the generated closed-world inventory, including `recipe.yaml`, and the binary that produced it. 5. **Writes attestation files** — `attestation/bundle-attestation.sigstore.json` and `attestation/aicr-attestation.sigstore.json` are added to the bundle output. Attestation is opt-in; bundles are unsigned by default. By default, signing uses Sigstore keyless signing (Fulcio CA + Rekor transparency log) and records the entry in **Rekor v2** (the signing config is fetched from Sigstore's TUF repository, so shard rotation is handled automatically; a cold cache is fetched on demand). Verifying such bundles with `aicr verify` needs only the `aicr` binary; verifying them with `cosign verify-blob-attestation` needs Cosign v3.0.1+. For CI/CD environments without OIDC, pass `--signing-key` to sign with a KMS key instead; see [KMS-Backed Signing](#kms-backed-signing) below. For verification, see [`aicr verify`](#aicr-verify). **Rekor v1 / private Sigstore:** pass `--rekor-url` to sign to Rekor **v1** at a specific URL — a private instance, or the public-good v1 URL — instead of the v2 default. Organizations running their own Fulcio CA can also redirect with `--fulcio-url` (both must be absolute `https://` URLs with no embedded credentials); the two are independent. For a fully custom v2 setup, `--signing-config` takes a Sigstore signing config JSON (mutually exclusive with `--rekor-url`). > **Verification:** these flags redirect **signing** only. Verify the resulting bundles with `aicr verify --trust-root `, supplying the `trusted_root.json` your self-hosted Fulcio/Rekor emits. That root is unioned with AICR's built-in public-good root, so privately-signed and NVIDIA-signed bundles both verify; see the [`aicr verify`](#aicr-verify) `--trust-root` flag. ##### OIDC Token Sources `--attest` resolves an OIDC identity token from the first matching source, in order: 1. `--identity-token` flag (or `COSIGN_IDENTITY_TOKEN` env) — a pre-fetched token. Use this when a token is obtained out of band (e.g., from a cloud workload-identity exchange or another `cosign` invocation). On shared hosts prefer the env var: a flag value is visible in `ps` and `/proc//cmdline` to any user on the same machine. 2. `ACTIONS_ID_TOKEN_REQUEST_URL` + `ACTIONS_ID_TOKEN_REQUEST_TOKEN` — the ambient GitHub Actions OIDC credential. Used automatically in CI. 3. `--oidc-device-flow` flag (or `AICR_OIDC_DEVICE_FLOW` env) — OAuth 2.0 Device Authorization Grant (RFC 8628). The CLI prints a verification URL and short code; the user enters the code in a browser **on a separate device**. Use on headless hosts (bastions, remote build boxes) where the default browser callback cannot reach the machine running `aicr`. The host still needs outbound network access to Sigstore's OIDC and signing endpoints. 4. Interactive browser flow — opens the default browser and listens on a random `localhost` port for the redirect. Default on workstations. Both interactive flows time out after 5 minutes. Attestation works with all deployers (`helm`, `argocd`, `argocd-helm`, `flux`, `helmfile`). External `--data` files copied into the bundle are included in `checksums.txt` and listed as resolved dependencies in the attestation. ##### Privacy: identity in keyless signatures Keyless signing trades a long-lived key for a short-lived Fulcio certificate minted from **your identity** — and that identity becomes part of the public record. Before any of the interactive sources above (device-code or browser) opens a login, understand what is published: - The Fulcio certificate embeds the **authenticated OIDC identity** — typically your **email address** plus the OIDC **issuer** — in the certificate's subject alternative name. - With the **public-good** Sigstore defaults (`fulcio.sigstore.dev` / `rekor.sigstore.dev`), the signature and certificate are recorded in the **Rekor transparency log**, which is **public, append-only, and permanent** — entries cannot be deleted, so the identity is **globally searchable forever**. - The same identity-bearing Sigstore bundle is **attached to the pushed OCI artifact** as a referrer, visible to anyone who can pull it. This applies equally to `aicr bundle --attest`, `aicr validate --emit-attestation --push`, and `aicr evidence publish --push`. **Controlling exposure.** To avoid publishing a personal identity: - Sign from a **service / CI ambient identity** (e.g. GitHub Actions OIDC) rather than a personal browser login. - Supply a **pre-fetched `--identity-token`** minted from a non-personal identity (a workload-identity exchange, a CI bot account). - Point `--fulcio-url` / `--rekor-url` at **private Sigstore infrastructure** so the identity stays inside your organization rather than the public commons. (Note: the `validate --emit-attestation` / `evidence publish` paths always use the public-good endpoints today; only `bundle --attest` exposes these flags.) - Use `--signing-key` (KMS-backed signing) instead of keyless — no OIDC identity is embedded. See [KMS-Backed Signing](#kms-backed-signing). **Interactive confirmation gate.** Because the consequence is irreversible on public Sigstore, the CLI gates the interactive login behind an explicit confirmation. When `aicr` is about to open a **browser or device-code** flow it prints a disclosure banner naming the Fulcio/Rekor endpoints in effect and what will be published, then: - **On a TTY**, it pauses for a `y/N` confirmation (default **no**) and aborts cleanly — no browser opens — if you decline. - **On non-interactive stdin** (CI, pipes), it prints the banner and proceeds without blocking, so scripted/CI signing is never wedged. - Pre-fetched `--identity-token`, ambient GitHub Actions OIDC, and `--signing-key` paths are **not** gated — they neither open a browser nor publish a surprise identity. Pass `--yes` (alias `--assume-yes`, env `AICR_ASSUME_YES`) to skip the prompt for trusted interactive automation; the banner is still printed. ##### KMS-Backed Signing Keyless signing depends on an OIDC identity provider. Some CI/CD environments (Jenkins, internal pipelines, air-gapped build hosts) have no OIDC issuer that Sigstore trusts. For those, `--signing-key` signs the `--attest` bundle with a KMS-backed key instead of a short-lived Fulcio certificate. The flag takes a KMS URI; the supported schemes are: - `awskms://`: AWS Key Management Service - `gcpkms://`: Google Cloud KMS - `azurekms://`: Azure Key Vault - `hashivault://`: HashiCorp Vault (Transit secrets engine) ```shell aicr bundle --recipe recipe.yaml --attest \ --signing-key awskms://arn:aws:kms:us-east-1:123456789012:key/abcd-1234 \ --output ./bundles ``` Because a KMS key reference is durable and non-secret, it can live in a version-controlled `--config` file as `spec.bundle.attestation.signingKey` instead of on the command line (the `--signing-key` flag wins when both are set). As with the flag, signing only runs when attestation is enabled, so a config-only KMS workflow must also set `spec.bundle.attestation.enabled: true` (the config equivalent of `--attest`); `signingKey` alone with `enabled: false` produces no attestation. See [Bundle Config File Mode](#bundle-config-file-mode). `--signing-key` is mutually exclusive with the keyless-only flags `--identity-token`, `--oidc-device-flow`, and `--fulcio-url`. Passing `--signing-key` together with any of them is a validation error, since they select incompatible signing modes (KMS key versus Fulcio-issued certificate). Like keyless signing, KMS signs to **Rekor v2** by default. Opt out with `--rekor-url` to log to a Rekor v1 instance (private or public-good), or `--signing-config` to use a custom signing config; the two opt-outs are mutually exclusive with each other but both compose with `--signing-key`. For a fully offline / air-gapped signing host that cannot reach any Rekor instance, add `--tlog-upload=false`. KMS signing then skips the transparency-log upload entirely and the bundle carries no Rekor entry: ```shell aicr bundle --recipe recipe.yaml --attest \ --signing-key awskms://arn:aws:kms:us-east-1:123456789012:key/abcd-1234 \ --tlog-upload=false \ --output ./bundles ``` `--tlog-upload=false` requires `--signing-key`; it is rejected on the keyless path, because keyless OIDC signing needs Fulcio and Rekor network access to mint a verifiable certificate. Verify an air-gapped bundle offline with `aicr verify --key --insecure-ignore-tlog`, which skips the transparency-log lookup that would otherwise require network access. Use a local PEM public key (exported once with `cosign public-key --key `) for a fully offline verify: a KMS `--key` URI still makes a live `GetPublicKey` call. The resulting bundle uses the same Sigstore bundle format as keyless signing, but its verification material is the signing key's public key rather than a Fulcio certificate. > **Verification:** verify a KMS-signed bundle with `aicr verify --key `, > supplying the same KMS URI used to sign (or a local PEM public-key file). See > the [`aicr verify`](#aicr-verify) flags below. cosign's public-key path > (`cosign verify-blob-attestation --key ...`) also works, since > the bundle uses the standard Sigstore bundle format. The `hashivault://` scheme signs through HashiCorp Vault's Transit secrets engine (or an API-compatible server such as OpenBAO). It reads the server address and token from the standard `VAULT_ADDR` and `VAULT_TOKEN` environment variables (the OpenBAO equivalents `BAO_ADDR` and `BAO_TOKEN` are honored as fallbacks), so no additional flags are required; set `TRANSIT_SECRET_ENGINE_PATH` if the Transit engine is mounted somewhere other than the default `transit/`. Verify the resulting bundle with the same URI via `aicr verify --key hashivault://`. ##### Attestation Scope Attestation binds a closed-world bundle inventory. `checksums.txt` contains one SHA256 entry for every regular payload file — including `recipe.yaml`, defaults, dynamic-value stubs, and external `--data` files copied into the bundle. Verification derives the required directories and rejects every additional file or directory, symlink, and other non-regular object. Only `checksums.txt`, `attestation/bundle-attestation.sigstore.json`, and `attestation/aicr-attestation.sigstore.json` may exist outside the manifest; they remain part of the verified inventory and the attestation files are verified separately. Manifest parsing is order-independent and AICR generates entries sorted by canonical slash-relative path, but reordering an already signed `checksums.txt` changes the signed bytes and invalidates its existing attestation. The attestation does **not** bind install-time values supplied via `helm --set`, a user-provided `-f extra.yaml`, or Argo `Application.spec.source.helm.parameters`. That boundary is intentional: dynamic values are the operator's domain by design. If you need to enforce specific install-time values (e.g., pinning `driver.version`), that is a **policy concern**, not an attestation one. Use admission control (Kyverno, Gatekeeper) or Argo sync hooks to reject deployments that violate the policy. `aicr verify` checks bundle integrity and provenance; it does not evaluate install-time value constraints. #### Deploying a bundle ```shell # Navigate to bundle cd bundles # Review root README and a component's values cat README.md cat 009-gpu-operator/values.yaml # folder numbers depend on the recipe # Verify the complete closed-world inventory and any available attestations aicr verify . # Deploy to cluster chmod +x deploy.sh && ./deploy.sh ``` > **Note:** `deploy.sh` is a convenience script — not the only deployment path. Each `NNN-/` folder contains a rendered `install.sh` that runs the exact `helm upgrade --install` command for manual or pipeline-driven deployment. For teardown, bundles delegate to the deployer-native uninstall path (see [Bundle Uninstall](#bundle-uninstall) below). #### Deploy Script Behavior (`deploy.sh`) The deploy script installs components in the order specified by `deploymentOrder` in the recipe. **Flags:** | Flag | Description | |------|-------------| | `--no-wait` | Skip Helm chart-level wait (`helm --wait`) where AICR uses it. Keeps `--timeout` for hooks. | | `--best-effort` | Continue past individual component failures instead of exiting | | `--retries N` | Retry failed helm/kubectl operations N times with exponential backoff (default: 5) | Unknown flags are rejected with an error to catch typos (e.g., `--bes-effort` or `--retires N`). > **Note on install completion vs. workload readiness.** By default, `deploy.sh` waits on Helm chart readiness where AICR uses `helm --wait`. Some components are intentionally installed without Helm chart-level waiting, and the script does not wait for bundle-level workload readiness such as Nodewright node tuning, GPU operator operand rollout (driver, toolkit, device-plugin DaemonSets), or NVIDIA DRA kubelet plugin registration. Those continue asynchronously after the script exits. When `--best-effort` is used, the script may also finish with non-fatal component failures; check warning lines and logs before treating the install/apply pass as fully successful. `--no-wait` only skips the Helm chart-level wait where AICR uses it; it does not affect bundle-level convergence. ##### Cluster connection environment `deploy.sh` and each component's `install.sh` act on whichever cluster the environment selects. Both are standalone entry points, so the same variables apply whether you run the whole bundle or a single component by hand. | Variable | Effect | |----------|--------| | `KUBE_CONTEXT` | Context to act on. Rendered as `--kube-context` for `helm` and `--context` for `kubectl`, and exported to each component's `install.sh`. | | `KUBECONFIG` | Path to a kubeconfig. Read natively by both `helm` and `kubectl`, so no flag is derived from it. | | `KUBECONFIG_FLAG` | Deprecated. A literal `helm` flag string; only `--kube-context` and `--kubeconfig` are translated, with a warning. | ```bash KUBE_CONTEXT=my-cluster ./deploy.sh # Or a single component, from its own folder: cd 009-gpu-operator && KUBE_CONTEXT=my-cluster bash install.sh ``` Prefer `KUBE_CONTEXT`. Setting it alongside `KUBECONFIG_FLAG=--kube-context` is accepted while both name the same context; only a mismatch is rejected. A `KUBECONFIG_FLAG` carrying an option that is not translated, or naming a different context than `KUBE_CONTEXT`, fails before the first cluster call rather than falling back to the ambient context. When an unsupported or malformed option is rejected, the message names the option but not its argument, so a flag carrying a credential does not reach the log. The context-mismatch message is the exception: it names both contexts, which identify clusters rather than authenticate to them. **Retry behavior:** The deploy script retries failed `helm upgrade --install` and `kubectl apply` operations with exponential backoff. By default, each operation is retried up to 5 times (6 total attempts). The backoff delay increases quadratically: 5s, 20s, 45s, 80s, 120s (capped) between retries. Use `--retries 0` to disable retries (fail-fast behavior). When `--best-effort` is also set, retries are exhausted first before falling through to best-effort handling. **Exit codes:** | Code | Meaning | |------|---------| | `0` | All components installed. With `--best-effort`, the run finished; check the `✗` and warning lines for components that failed. | | `1` | A component failed (without `--best-effort`), the arguments were invalid, or a `--best-effort` run had component failures and also needs a retry. | | `2` | All components installed, but the DRA kubelet plugin restart was withheld because a GPU driver migration was in progress. Re-run `deploy.sh` once the GPU Operator has converged (driver DaemonSet present and nodes labeled `nvidia.com/gpu.deploy.driver=true`). | | `130` | Interrupted (SIGINT or SIGTERM). | **Pre-install manifests and CRD ordering:** `deploy.sh` does not `kubectl apply` component manifests itself. A component's pre-install manifests (`preManifestFiles`) are wrapped in an injected `NNN--pre/` local chart that installs just before the component's chart, and its raw manifests (`manifestFiles`) in an injected `NNN--post/` local chart that installs just after it, so the chart's CRDs are registered before the custom resources that use them. Both are ordinary folders in the install loop, installed by their own `install.sh` with `helm upgrade --install`. **Async components:** Components that use operator patterns with custom resources that reconcile asynchronously (e.g., `kai-scheduler`) are installed without `--wait` to avoid Helm timing out on CR readiness. ##### DRA kubelet plugin registration After installing `nvidia-dra-driver-gpu`, the script automatically restarts the DRA kubelet plugin DaemonSet. This is a best-effort mitigation for a known issue: after uninstall/reinstall, the kubelet's plugin watcher (`fsnotify`) may not detect new registration sockets, causing `DRA driver gpu.nvidia.com is not registered` errors. If the GPU Operator is still migrating the driver when this step runs, the script withholds the restart rather than restart the plugin against a driver that is changing underneath it, and exits `2` (see [Exit codes](#deploy-script-behavior-deploysh) above). Re-run `deploy.sh` once the migration finishes. If DRA pods fail with this error after redeployment, the DaemonSet restart alone may not be sufficient — a **node reboot** is required to reset the kubelet's plugin registration state. To reboot GPU nodes: ```bash # Cordon, drain, and reboot the affected node kubectl cordon kubectl drain --ignore-daemonsets --delete-emptydir-data # Reboot via cloud provider (e.g., AWS EC2 console or CLI) aws ec2 reboot-instances --instance-ids # Uncordon after node returns kubectl uncordon ``` #### Bundle Uninstall AICR bundles do **not** ship a generated `undeploy.sh`. Teardown is delegated to the deployer-native uninstall path; AICR's role ends at design-time generation. Pick the walkthrough that matches the deployer used to generate your bundle. ##### helm Uninstall releases in **reverse** deployment order — the same order the generated `README.md` lists under `## Uninstall`: ```bash # For each NNN-/ folder in descending order: helm uninstall -n ``` Helm intentionally does not delete CRDs declared under `crds/`. PVC lifecycle depends on how the claim is created: StatefulSet-created claims normally outlive the StatefulSet, but standalone PVCs rendered as release resources are deleted unless the chart marks them with `helm.sh/resource-policy: keep`. Review the chart and the bound PersistentVolume reclaim policy before removing storage: ```bash # CRDs — review first; deletion cascades to every custom resource cluster-wide kubectl get crd -o name | grep -E '' kubectl delete crd # PVCs — list and select explicitly; deletion can destroy backing data kubectl -n get pvc kubectl -n delete pvc # Namespace kubectl delete namespace ``` Resources created by Helm hooks are not tracked as ordinary release resources and can remain after `helm uninstall`. Review the component-specific cleanup notes in the [Component Catalog](/aicr/user-guide/component-catalog) and the hook lifecycle guidance in [Bundling](/aicr/user-guide/generating-bundles) before deleting them manually. If a release is stuck in `pending-install` or `pending-upgrade` (interrupted deploy), retry with `--no-hooks`: ```bash helm uninstall -n --no-hooks ``` See [Helm 3 uninstall docs](https://helm.sh/docs/helm/helm_uninstall/) for the full flag reference. ##### argocd Delete the parent `Application` that owns the bundle's child Applications (app-of-apps). By default AICR does **not** set the `resources-finalizer.argocd.argoproj.io` finalizer on generated Applications, so a plain `kubectl delete` removes only the Application CR and leaves the managed resources running. Bundles generated with `--set deployer:cascadeDelete=true` (see [Argo CD Deployer Options](#argo-cd-deployer-options)) are the exception: the finalizer is baked onto the parent and every child Application, so a plain `kubectl delete` on the parent already cascades to the managed resources. For default bundles, use one of the cascade-aware flows instead: ```bash # Argo CD CLI — cascade is the default; foreground waits for resources argocd app delete --cascade --propagation-policy foreground ``` If you can only use `kubectl`, add the finalizer first so the controller performs the cascade for you: ```bash kubectl -n argocd patch application --type=merge \ -p '{"metadata":{"finalizers":["resources-finalizer.argocd.argoproj.io"]}}' kubectl -n argocd delete application ``` The CRD and PVC notes from the **helm** walkthrough above still apply, but Argo CD does not run `helm uninstall` for Helm-templated children. It renders manifests with `helm template` and prunes rendered resources directly. For standalone PVCs, `Delete=false` (or the equivalent `helm.sh/resource-policy: keep`) prevents cleanup during Application deletion, while `Prune=false` prevents pruning during manual or automated sync after a PVC disappears from the desired manifests. StatefulSet-created claims are not rendered as Application resources and normally remain. See [Argo CD app deletion docs](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/) for finalizer behavior, cascade modes, and selective deletion. ##### argocd-helm Same path as plain `argocd`: Argo CD uses Helm only to render charts into Kubernetes manifests (via `helm template`) and then manages those resources itself. Deleting the Application with cascade enabled prunes the resources Argo CD tracks; it does **not** run `helm uninstall`, and `helm ls` will not show the bundle's releases. ```bash argocd app delete --cascade --propagation-policy foreground ``` The kubectl + finalizer-patch fallback from the **argocd** walkthrough applies here too, and CRD / PVC cleanup follows the **helm** notes above. See the [Argo CD Helm user guide](https://argo-cd.readthedocs.io/en/stable/user-guide/helm/) and the [Argo CD FAQ entry on `helm ls`](https://argo-cd.readthedocs.io/en/stable/faq/#after-deploying-my-helm-application-with-argo-cd-i-cannot-see-it-with-helm-ls-and-other-helm-commands) for why Helm CLI tools don't see Argo-deployed releases. ##### flux AICR's `flux` bundle emits one `HelmRelease` per component (plus the `HelmRepository` / `OCIRepository` source objects). Deleting each `HelmRelease` from the cluster triggers `helm-controller` to run `helm uninstall` for the underlying release, honoring the chart's `spec.uninstall` settings (`disableHooks`, `keepHistory`, etc.): ```bash kubectl -n delete helmrelease ``` Delete the bundle's source objects (`HelmRepository` / `OCIRepository`) after the releases are gone. The CRD / PVC notes from the **helm** walkthrough above still apply: `helm-controller` honors Helm resource-policy annotations, while unannotated standalone PVCs are release resources and are deleted during uninstall. See the [Flux helm-controller uninstall reference](https://fluxcd.io/flux/components/helm/helmreleases/#uninstall-configuration) for `spec.uninstall` field semantics. ##### helmfile AICR's `helmfile` bundle emits a single `helmfile.yaml` release graph. The upstream `helmfile` CLI handles teardown: ```bash helmfile -f helmfile.yaml destroy ``` CRD / PVC cleanup follows the **helm** walkthrough above. See the [Helmfile `destroy` documentation](https://github.com/helmfile/helmfile/blob/main/docs/index.md) for flags and behavior. --- ### aicr mirror list Discover container images and Helm charts referenced by a recipe for air-gapped mirroring. Renders each component's Helm chart with recipe-resolved values and scans referenced manifests to produce a deduplicated image and chart list. When the recipe was resolved with `--data `, both values and manifests are read through the overlay so overlay-shadowed paths take precedence over embedded. Recognized mapping-valued image descriptors with null, empty, or non-scalar members are rejected so the command cannot succeed with a known-incomplete image set. For an end-to-end walkthrough covering Hauler and Zarf workflows, see [Air-Gapped Mirroring](/aicr/user-guide/air-gapped-mirroring). **Synopsis:** ```shell aicr mirror list [flags] ``` **Flags:** | Flag | Short | Type | Default | Description | |------|-------|------|---------|-------------| | `--recipe` | `-r` | string | | Path/URI to a previously generated recipe. Supports: file paths, HTTP/HTTPS URLs, or ConfigMap URIs (`cm://namespace/name`). | | `--service` | | string | | Cloud service (e.g., `eks`, `gke`, `aks`, `ocp`). Alternative to `--recipe`. | | `--accelerator` | `--gpu` | string | | GPU accelerator (e.g., `h100`, `gb200`). Alternative to `--recipe`. | | `--intent` | | string | | Workload intent (`training` or `inference`). Alternative to `--recipe`. | | `--os` | | string | | Operating system (e.g., `ubuntu`). Alternative to `--recipe`. | | `--platform` | | string | | Optional platform specialization (e.g., `kubeflow`). | | `--profile` | | string | | Profile selection in exact `name=value` form when resolving from criteria. Cannot be combined with `--recipe`. | | `--inherit-from` | | string | | Prior recipe file, or bundle directory, whose component namespace, chart, source, path, manifest files and pre-manifest files the resolved recipe keeps instead of re-deriving them from the registry; a bundle directory also supplies its object names. When both artifacts state a deployment type and they differ, such as a Helm to Kustomize change, only the namespace is kept for that component and `aicr upgrade-check` reports the type move. Applies to criteria-based resolution only, and cannot be combined with `--recipe`, because that file already records the identity it resolved to. See [Upgrading a Deployed Stack](/aicr/user-guide/upgrading-a-deployed-stack#pinning-the-namespaces-you-already-deployed-into). | | `--set` | | string[] | | Override values that affect image discovery (format: `component:path.to.field=value`). Repeatable. | | `--data` | | string | | External data directory to overlay on embedded data. Overlay-provided component values and manifests both feed image discovery (see [External Data](#external-data-directory)). | | `--format` | `-f` | string | `yaml` | Output format: `yaml`, `json`, `hauler`, `zarf` | | `--output` | `-o` | string | stdout | Output file path | The remaining `aicr recipe` resolution flags (`--snapshot`/`-s`, `--config`, `--nodes`, `--slurm-accounting-mode`, `--runtime-inventory`, `--gke-tcpxo-interfaces`, `--criteria-strict`, `--kubeconfig`/`-k`) are also accepted when resolving from criteria; see [`aicr recipe`](#aicr-recipe). **Examples:** ```shell # List images from a recipe file (YAML to stdout) aicr mirror list --recipe recipe.yaml # Resolve recipe from query parameters aicr mirror list --service eks --accelerator h100 --intent training --os ubuntu # Generate Hauler manifest aicr mirror list --recipe recipe.yaml --format hauler --output hauler-manifest.yaml # Generate Zarf package config aicr mirror list --recipe recipe.yaml --format zarf --output zarf.yaml # Override a value that affects image discovery aicr mirror list --recipe recipe.yaml --set gpuoperator:driver.enabled=false ``` --- ### aicr verify Verify the complete closed-world inventory and attestation chain of a bundle. `aicr verify .` is the full verification command when run from the bundle root. It rejects any additional file or directory, symlink, or other non-regular filesystem object, except the three exact inventory metadata paths. By default verification is offline and makes no network calls. The one exception is `--key` with a KMS URI, which reaches the KMS provider to fetch the public key (see the `--key` network behavior note below). **Synopsis:** ```shell aicr verify [flags] ``` **Flags:** | Flag | Type | Default | Description | |------|------|---------|-------------| | `--min-trust-level` | string | `max` | Minimum required trust level. `max` auto-detects the highest achievable level and verifies against it. Explicit levels: `verified`, `attested`, `unverified`, `unknown`. | | `--require-creator` | string | | Require a specific creator identity, matched against the bundle attestation signing certificate. | | `--cli-version-constraint` | string | | Version constraint for the aicr CLI version in the attestation predicate. Supports `>=`, `>`, `<=`, `<`, `==`, `!=`. A bare version (e.g. `"0.8.0"`) defaults to `>=`. | | `--certificate-identity-regexp` | string | | Override the certificate identity pattern for binary attestation verification. Must *begin with* `https://github.com/NVIDIA/aicr/` (a leading `^` is allowed) and must not use top-level alternation. For testing only. | | `--key` | string | | Verify a key-signed bundle attestation against a KMS key URI (`awskms://` \| `gcpkms://` \| `azurekms://` \| `hashivault://`) or a local PEM public-key file. This is the counterpart to `bundle --signing-key`. It coexists with `--certificate-identity-regexp`, which pins the binary attestation; the two verify different attestations. | | `--trust-root` | string | | Verify the bundle attestation against a private Sigstore trusted root (a `trusted_root.json` from a self-hosted Fulcio/Rekor). Additive to AICR's built-in public-good root, so NVIDIA-signed and privately-signed bundles both verify. Composes with `--key` and `--certificate-identity-regexp`. The verify counterpart to `bundle --fulcio-url`/`--rekor-url`. | | `--insecure-ignore-tlog` | bool | `false` | Offline/air-gapped verification: skip the transparency-log (and observer-timestamp) requirement so a bundle signed with `bundle --signing-key ... --tlog-upload=false` verifies against `--key` with no transparency-log network calls. A local PEM `--key` is then fully offline; a KMS `--key` URI still makes a live `GetPublicKey` call to resolve the key (export a PEM with `cosign public-key` for a truly offline verify). Requires `--key`; the air-gapped path is key-based, not keyless. Named "insecure" because, with no transparency log, there is no trusted timestamp proving when the signature was made. Does not affect the binary attestation, which always requires a transparency log. | | `--format` | string | `text` | Output format: `text` or `json`. | | `--config` | string | | Path or HTTP/HTTPS URL to an AICRConfig file (YAML/JSON) supplying verification policy from `spec.verify`. CLI flags override values from this file. See [Verify Config File Mode](#verify-config-file-mode). | #### Verify Config File Mode `aicr verify --config ` reads verification policy from an AICRConfig YAML/JSON file under `spec.verify`. CLI flags always override values loaded from `--config`; override events are logged at INFO so users can see which input won. This is the one consumer-side section of the schema, so a single committed document can carry both the settings that build an artifact and the trust floor a downstream consumer enforces against it. **Supported schema:** ```yaml kind: AICRConfig apiVersion: aicr.run/v1beta1 metadata: name: prod-verify spec: verify: policy: # assertions checked after verification runs minTrustLevel: verified # or unknown | unverified | attested | max requireCreator: ci@myorg.example.com cliVersionConstraint: ">= 0.16.0" # bare version means ">=" trust: # material verification runs against certificateIdentityRegexp: "https://github.com/NVIDIA/aicr/.+" key: gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k trustRoot: ./trusted_root.json ``` Every field is a durable, non-secret reference or policy value, so the whole section is safe to commit. No private key material is part of the schema. Three `aicr verify` flags are deliberately **not** in the schema: - The bundle directory, which is a positional argument rather than a flag. - `--format`, which is presentation rather than policy. - `--insecure-ignore-tlog`, which weakens the trust floor by dropping the transparency-log requirement. Keeping it command-line-only means a committed file can never silently disable that check, and an air-gap override stays an explicit operator act. It still composes with a config-supplied `key`. Values are validated when the document loads, so a typo fails with its spec path (for example `invalid spec.verify.policy.minTrustLevel`) rather than after a full verification run. One limit is worth knowing: `cliVersionConstraint` is checked at the operator level only, so `">="` with no version is rejected at load time while `">= not-a-version"` is accepted and fails later when the constraint is evaluated. ```shell # Commit the verification policy, then gate a deploy on it. aicr verify ./my-bundle --config aicr-config.yaml # A CLI flag still wins over the committed policy. aicr verify ./my-bundle --config aicr-config.yaml --min-trust-level attested ``` #### Trust Levels | Level | Name | Criteria | |-------|------|----------| | 4 | `verified` | Full chain: closed-world inventory + bundle attestation + binary attestation pinned to NVIDIA CI | | 3 | `attested` | Closed-world inventory and bundle attestation valid; binary attestation missing, or external data (`--data`) used | | 2 | `unverified` | Closed-world inventory valid; `--attest` was not used when creating the bundle | | 1 | `unknown` | Missing, incomplete, or invalid inventory; or an attestation (bundle or binary) present but failing verification | #### Verification steps 1. **Closed-world inventory** — verifies every regular payload file is listed in `checksums.txt`, all listed digests match, required directories are present, and no additional filesystem entries exist beyond the three allowed metadata paths 2. **Bundle attestation** — cryptographic signature verified against Sigstore trusted root 3. **Binary attestation** — provenance chain verified with identity pinned to NVIDIA CI (`on-tag.yaml` workflow) Inventory validation accepts valid manifest entries in any order, while AICR generates canonical slash-relative entries in sorted order. Reordering a signed `checksums.txt` invalidates its existing attestation. Legacy bundles with incomplete manifests report `unknown` trust and must be regenerated. **Examples:** ```shell # From a bundle root, perform full closed-world verification cd ./my-bundle aicr verify . # Enforce a minimum trust level aicr verify ./my-bundle --min-trust-level verified # Require a specific bundle creator aicr verify ./my-bundle --require-creator jdoe@company.com # Require minimum CLI version used to create the bundle aicr verify ./my-bundle --cli-version-constraint ">= 0.8.0" # JSON output for CI pipelines aicr verify ./my-bundle --format json # Sign a bundle with a KMS key, then verify it with the same key aicr bundle -r recipe.yaml --attest --signing-key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k -o ./bundles aicr verify ./bundles/ --key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k # Or verify against an exported PEM public key (no KMS access needed) aicr verify ./bundles/ --key ./bundle-signer.pub # Verify a privately-signed bundle against an org trusted root aicr verify ./my-bundle --trust-root ./trusted_root.json # Verify an offline/air-gapped bundle: no transparency log, no network calls. # Sign on a connected host (signing needs KMS access), then export the public key once: aicr bundle -r recipe.yaml --attest --signing-key awskms://alias/my-key --tlog-upload=false -o ./bundles cosign public-key --key awskms://alias/my-key > bundle-signer.pub # Verify anywhere offline against the exported PEM (a KMS --key URI would make a live GetPublicKey call): aicr verify ./bundles/ --key ./bundle-signer.pub --insecure-ignore-tlog ``` > **`--key` network behavior:** Resolving a **KMS URI** (`awskms://`, `gcpkms://`, `azurekms://`, `hashivault://`) makes network calls to the KMS provider to fetch the public key, so credentials for that provider must be available in the environment. A **local PEM** public-key file is read from disk with no provider calls; export it once with `cosign public-key --key ` (or your provider's console) and verify anywhere. > > Resolving the key is only part of verification: by default the bundle's Rekor transparency-log entry is also checked. Its inclusion proof is embedded in the bundle, so no live Rekor call is made, but the check needs the Sigstore trusted root. That root is loaded from the local cache and falls back to the embedded trusted root on a cache miss, so no network fetch happens on the verify path and `aicr trust update` is not required for offline use. For a bundle signed without a transparency-log entry at all (`bundle --signing-key ... --tlog-upload=false`), pass `--insecure-ignore-tlog` alongside `--key` to drop the transparency-log requirement entirely; verification then runs with no network calls and no trusted timestamp, so use it only for true air-gapped bundles you signed yourself. > > **Stale root:** If verification fails with certificate chain errors, run `aicr trust update` to refresh the Sigstore trusted root. --- ### aicr evidence digest Print the canonical sha256 of a resolved recipe — byte-for-byte the same value recorded in `predicate.recipe.digest` by `aicr validate --emit-attestation`. The input is resolved through the same recipe builder path as `aicr validate -r`, so overlays and mixins are hydrated before hashing. Use this to detect drift between a signed evidence pointer and the current recipe on a PR branch without pulling the OCI artifact. **Synopsis:** ```shell aicr evidence digest -r [flags] ``` **Flags:** | Flag | Alias | Type | Default | Description | |------|-------|------|---------|-------------| | `--recipe` | `-r` | string | | Path/URI to a recipe or overlay file (file, HTTP/HTTPS, or `cm://namespace/name`). Required. | | `--profile` | | string | | Profile selection in exact `name=value` form for overlay inputs on a profiled family (e.g. `gpuStack=operator-managed` on AKS); omit for the declaration default. Rejected when the input is a hydrated `RecipeResult` — its selection is already baked into `metadata.selectedProfile`. | | `--kubeconfig` | `-k` | string | | Kubeconfig path; consulted only when the input is a `cm://` URI. | **Exit codes:** | Code | Meaning | |------|---------| | 0 | Digest printed to stdout. | | non-zero | Input could not be loaded, hydrated, or canonicalized. | **Examples:** ```shell # Print the digest of a hydrated overlay. aicr evidence digest -r recipes/overlays/h100-eks-ubuntu-training.yaml # Profiled family: the selection changes the digest. Omit --profile for the # declaration default; pass it for a non-default value. --profile is rejected # when the input is a hydrated RecipeResult (the selection is already baked). aicr evidence digest -r recipes/overlays/h100-aks-ubuntu-training.yaml \ --profile gpuStack=operator-managed # CI drift gate: compare the digest pinned in a signed evidence bundle # against the recipe currently on the PR branch. For a profiled pointer, # replay its recorded selection — recomputing without it hydrates the # declaration default and false-stales every non-default value. For the # h100 GKE kubeflow training leaf (ships torch-distributed-tcpxo), replay # the recorded configuration.gke.tcpxoInterfaces too — overlay-direct # digest fails closed without it; hydrate the recipe first, then digest it. ptr=recipes/evidence///.yaml prof=$(yq -r '.profile // ""' "$ptr") signed=$(aicr evidence verify "$ptr" --format json \ | jq -r .predicate.recipe.digest) current=$(aicr evidence digest -r recipes/overlays/.yaml ${prof:+--profile "$prof"}) [[ "$signed" == "$current" ]] || echo "evidence is stale" ``` --- ### aicr evidence publish Sign, push, and write the pointer for a recipe-evidence bundle (predicateType v3; v1 and v2 remain verifiable for already-signed evidence) that was produced earlier by `aicr validate --emit-attestation` **without** `--push` (which leaves an unsigned bundle on disk). This decouples the cluster-bound validate step from the Fulcio/Rekor-bound signing step so they can run on different networks: validation must run where the cluster is reachable (often a corporate VPN), but keyless signing must reach `fulcio.sigstore.dev` + `rekor.sigstore.dev`, which corporate networks frequently block. Run `validate --emit-attestation` on the VPN, then `evidence publish` from a host with Sigstore egress (CI runner, jump box, hotspot). The **unsigned** subject/predicate — and therefore the OCI bundle digest — is identical regardless of which host ran which leg, because the predicate (including its baked-in `attestedAt`) is signed verbatim from the bundle on disk. The Sigstore signature, Fulcio certificate, and Rekor entry differ per signing run, so the *signed bytes* themselves are not byte-for-byte reproducible. **Synopsis:** ```shell aicr evidence publish --push [flags] ``` The positional `` is either the directory `--emit-attestation` wrote (holds `summary-bundle/` and receives `pointer.yaml`) or the `summary-bundle/` directory itself. **Flags:** | Flag | Alias | Type | Default | Description | |------|-------|------|---------|-------------| | `--push` | | string | | OCI registry reference to push the signed summary bundle to. Required. Triggers Sigstore keyless signing via the precedence chain documented under `--identity-token`. Omit the tag and aicr derives a unique per-recipe one (`-`); pass an explicit tag to override. See [`aicr validate --push`](#aicr-validate). | | `--no-sign` | | bool | `false` | Push the bundle **unsigned** and write a `pointer.yaml` with an empty `signer` block, instead of signing. Skips all OIDC/Fulcio/Rekor steps (and the identity-disclosure prompt), so it runs even where Sigstore egress is blocked. The bundle's content reference (`bundle.oci`/`bundle.digest`) is still recorded. Intended for the two-leg publish flow — commit the flat unsigned pointer and complete the signing leg via the fork CI workflow or [`aicr evidence sign --relocate`](#aicr-evidence-sign), which signs it and relocates it to the nested per-source path the blocking *Evidence Pointer Contract* gate requires. See [Publishing Recipe Evidence](/aicr/contributor-guide/publishing-recipe-evidence#recommended-path-split-the-legs-sign-in-ci). | | `--identity-token` | | string | | Pre-fetched OIDC identity token for keyless signing. Skips ambient/browser/device-code flows. Reads `COSIGN_IDENTITY_TOKEN` from env. Same precedence chain as `aicr validate --push`. | | `--oidc-device-flow` | | bool | `false` | Use the OAuth 2.0 device authorization grant for OIDC instead of opening a browser callback. Reads `AICR_OIDC_DEVICE_FLOW`. Useful on headless hosts. | | `--yes` | `--assume-yes` | bool | `false` | Skip the interactive confirmation shown before keyless signing publishes your OIDC identity (browser/device-code paths only; the banner is still printed). Reads `AICR_ASSUME_YES`. See [Privacy: identity in keyless signatures](#privacy-identity-in-keyless-signatures). | | `--plain-http` | | bool | `false` | Use HTTP instead of HTTPS when pushing the OCI artifact (local-registry tests). | | `--insecure-tls` | | bool | `false` | Skip TLS verification when pushing the OCI artifact (self-signed registries). | > **Identity disclosure:** `evidence publish` signs unless `--no-sign` is set. > On the interactive (browser / device-code) keyless paths it publishes the > signer's identity (email + issuer) to the public Rekor log, so on a TTY it > pauses for confirmation first (`--yes` skips it). `--no-sign` runs no OIDC > flow, so the prompt is skipped entirely. See > [Privacy: identity in keyless signatures](#privacy-identity-in-keyless-signatures). **Exit codes:** | Code | Meaning | |------|---------| | 0 | Bundle pushed and `pointer.yaml` written (signed, or unsigned with `--no-sign`). | | non-zero | Identity-disclosure prompt declined, or bundle could not be loaded, signed, or pushed. | **Examples:** ```shell # On VPN: produce an unsigned bundle from a passing validation. aicr validate -r recipe.yaml -s snapshot.yaml --emit-attestation ./out # Off VPN: sign, push, and write the pointer. Omit the tag and aicr derives # a unique per-recipe one (-). aicr evidence publish ./out --push ghcr.io/myorg/aicr-evidence ``` --- ### aicr evidence sign Complete the signing leg for a bundle that was already pushed **unsigned** (via `aicr evidence publish --no-sign` or `validate --emit-attestation ./out --push --no-sign`). It reads a **local** pointer file (e.g. `./out/pointer.yaml`), pulls the bundle it references (`bundle.oci` + `bundle.digest` — no recipe-name or bundle-ref input needed), signs the predicate with keyless OIDC, attaches the Sigstore Bundle as an OCI referrer of the existing artifact, and patches the pointer's `signer` block in place. Signing is the only leg that needs Fulcio/Rekor egress, so this command runs wherever Sigstore is reachable (a jump box, CI runner, or hotspot) while the cluster-bound validate/push legs run wherever the cluster lives. The bundle is **not** re-emitted: the predicate is read verbatim from the pulled bundle, so the signature binds the same bytes the unsigned push produced. The fork CI signing leg runs this with `--relocate` on the committed flat pending pointer — signing it and moving it to the nested per-source path the blocking *Evidence Pointer Contract* gate requires; you can also run it directly on a Sigstore-reachable host. See [Publishing Recipe Evidence](/aicr/contributor-guide/publishing-recipe-evidence#recommended-path-split-the-legs-sign-in-ci). **Synopsis:** ```shell aicr evidence sign [flags] ``` The positional `` is the committed flat pending pointer `recipes/evidence/.yaml`. The pointer must carry exactly one attestation that is already pushed (`bundle.oci`/`bundle.digest` set) and not yet signed (empty `signer`); otherwise the command fails closed (an already-signed pointer is never re-signed, except under `--relocate`, which moves an already-signed pointer without re-signing). With `--relocate`, the now-signed pointer is moved from its flat pending path to its canonical per-source path `recipes/evidence///.yaml` — the layout the per-source contract gate requires. A flat pointer is the only committable state for an unsigned pointer, because the `` segment derives from the signer it does not yet have. This is the step the fork-based CI signing leg runs. **Flags:** | Flag | Alias | Type | Default | Description | |------|-------|------|---------|-------------| | `--identity-token` | | string | | Pre-fetched OIDC identity token for keyless signing. Skips ambient/browser/device-code flows. Reads `COSIGN_IDENTITY_TOKEN` from env. Same precedence chain as `aicr evidence publish`. | | `--oidc-device-flow` | | bool | `false` | Use the OAuth 2.0 device authorization grant for OIDC instead of opening a browser callback. Reads `AICR_OIDC_DEVICE_FLOW`. Useful on headless hosts. | | `--yes` | `--assume-yes` | bool | `false` | Skip the interactive confirmation shown before keyless signing publishes your OIDC identity (browser/device-code paths only; the banner is still printed). Reads `AICR_ASSUME_YES`. | | `--plain-http` | | bool | `false` | Use HTTP instead of HTTPS for the registry (pull + referrer attach; local-registry tests). | | `--insecure-tls` | | bool | `false` | Skip TLS verification for the registry (pull + referrer attach; self-signed registries). | | `--relocate` | | bool | `false` | After signing, move the pointer from its flat pending path (`recipes/evidence/.yaml`) to its canonical per-source path (`recipes/evidence///.yaml`). Used by the fork-based CI signing leg to complete the commit-flat → sign → relocate flow. Idempotent: an already-signed flat pointer is moved without re-signing, and a pointer already at its canonical path is a no-op. If a **different** file already occupies the canonical path, the command fails closed with a conflict error rather than overwriting it (per-source pointers are immutable) — resolve by removing the duplicate, or `git pull` to sync a prior relocation, then re-run. | **Exit codes:** | Code | Meaning | |------|---------| | 0 | Bundle signed, referrer attached, and the pointer's `signer` block written back. | | non-zero | Pointer already signed / has nothing pushed to sign, bundle could not be pulled (e.g. a private registry returns 403), identity-disclosure prompt declined, or signing/attach failed. | **Examples:** ```shell # In CI (ambient OIDC), after a contributor committed an unsigned pointer: # sign and relocate it to its canonical per-source path. aicr evidence sign recipes/evidence/h100-eks-ubuntu-training.yaml --relocate ``` --- ### aicr evidence verify Verify a recipe-evidence bundle (predicateType v3 for newly produced evidence; v1 and v2 remain verifiable) produced by `aicr validate --emit-attestation`. When the bundle carries a signature, verifies it against the Sigstore trusted root and extracts the cryptographically anchored predicate. Recomputes every manifest-listed payload file's sha256 against `manifest.json` (which the predicate's `manifest.digest` field anchors), and surfaces the predicate's fingerprint, phase counts, and BOM info. It does not replay the recipe's inline constraints against the bundled snapshot. **Synopsis:** ```shell aicr evidence verify [flags] ``` The positional argument is auto-detected as one of: * `recipes/evidence///.yaml` — **pointer file (preferred)**. The verifier pulls **by digest** — `registry/repo@`, with the registry/repo taken from `bundle.oci` and the digest as the pin — so it fetches the exact attested bytes even if the `bundle.oci` tag has since been moved to a different artifact. This is the input to use in nearly all cases. * `ghcr.io//aicr-evidence@sha256:...` or `oci://...@sha256:...` — a **digest-pinned** OCI reference. A tag-only ref (such as the `bundle.oci` value copied from a pointer, e.g. `...aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e`) is refused by default because tags are registry-rewritable; see `--allow-unpinned-tag`. * `./out/summary-bundle/` (or a parent containing it) — unpacked directory. > **Do not extract `bundle.oci` from a pointer and pass it to `verify` as a raw OCI argument.** As a raw ref it carries no companion `bundle.digest`, so a tag-only ref is refused (tags are registry-rewritable). Pass the pointer file itself — the verifier reads `bundle.digest` from it and pulls `registry/repo@`, ignoring the tag. If you must verify a raw OCI ref, use the digest form (`...@sha256:`), not the tag. **Flags:** | Flag | Alias | Type | Default | Description | |------|-------|------|---------|-------------| | `--output` | `-o` | string | | Write output to this file. When empty, output goes to stdout. | | `--format` | `-t` | string | `text` | Output format: `text` (Markdown) or `json`. Applies regardless of destination. | | `--expected-issuer` | | string | | Pin the OIDC issuer URL on the signing certificate. Empty allows any issuer. | | `--expected-identity-regexp` | | string | | Pin the signer's `SubjectAlternativeName` via regex. Empty allows any identity. | | `--bundle` | | string | | OCI reference override for a local-only pointer that carries no `bundle.oci`. Use a digest-pinned ref (`...@sha256:`); a tag-only ref is refused unless `--allow-unpinned-tag` is set. | | `--registry-plain-http` | | bool | `false` | Use HTTP for registry traffic (local-registry tests only). | | `--registry-insecure-tls` | | bool | `false` | Skip TLS verification for the registry (self-signed certificates). | | `--allow-unpinned-tag` | | bool | `false` | Accept tag-only OCI references. By default the verifier refuses unpinned refs because tags are registry-rewritable; opt in only for one-off debugging. Pointer-driven flows ignore this flag when the pointer carries a `sha256:` digest. | **Verdict codes:** These are the values of the `exit` field in the JSON/Markdown output, mirroring `VerifyResult.Exit` from the library API. They are **not** the process exit code — see the table below them. | Code | Meaning | |------|---------| | 0 | Bundle valid; every check passed (or valid but **unsigned** — see pending below). | | 1 | Bundle valid, but recorded validator phase results show failures (informational). | | 2 | Bundle invalid. The `failureCause.class` field gives the specific reason — registry access (`registry-forbidden`/`not-found`/`registry`), `signature`, `integrity`, `schema`, or `unknown` (see Failure cause below). | | 3 | Verification **did not complete**, so no verdict was reached. `failureCause.class` is `transient` (the bundle was not readable — dead NFS/FUSE mount, unreachable registry) or `canceled` (the operator aborted the run). This is **not** a statement about the bundle — nothing was proven about it either way. For `transient`, retry; do not reject the artifact. | **Process exit codes:** | Process code | Verdicts that map to it | |--------------|-------------------------| | 0 | verdict 0 | | 2 | verdicts 1 **and** 2 | | 5 | verdict 3 with `failureCause.class: transient` | | 9 | verdict 3 with `failureCause.class: canceled` | Verdicts 1 and 2 are indistinguishable at the process level because both map through `pkg/errors` to the same code. To tell them apart, branch on the JSON `exit` field via `jq '.exit'` rather than on `$?`. Verdict 3 is deliberately given its own process code so a CI gate can distinguish an infrastructure fault from an invalid attestation without parsing JSON. **Pending signature.** An unsigned bundle whose pointer carries no `signer` (e.g. one published with `--no-sign`, awaiting the signing leg) is **not** a `verify` failure: it verifies at exit `0` with `pending: true` in the JSON output and a "pending signature" verdict in the Markdown summary. This is a useful local check on a not-yet-signed bundle. The committed flat pending pointer is signed and relocated to its nested per-source path by the fork CI signing leg (`aicr evidence sign --relocate`); the blocking *Evidence Pointer Contract* gate (`pkg/evidence/verifier/discover.go`) requires that final signed, nested pointer. See [Publishing Recipe Evidence](/aicr/contributor-guide/publishing-recipe-evidence#recommended-path-split-the-legs-sign-in-ci). **Failure cause.** On verdict 2 or 3, the JSON output carries a structured `failureCause` object — `class` (one of `registry-forbidden`, `not-found`, `registry`, `signature`, `integrity`, `schema`, `transient`, `canceled`, `unknown`), an optional `httpStatus`, and an actionable `hint`. For example, a private fork registry returns `class: registry-forbidden`, `httpStatus: 403` with a hint to make the package public — so the reason is self-serviceable rather than a bare "invalid". The Markdown summary renders the same as **Cause**/**Hint** lines. **Examples:** ```shell # Verify a pointer that a contributor committed alongside their recipe change. aicr evidence verify recipes/evidence///.yaml # Verify a pushed OCI bundle directly (no repo checkout required). aicr evidence verify ghcr.io/myorg/aicr-evidence@sha256:abc... # Verify a local bundle directory (contributor self-debug before push). aicr evidence verify ./out/summary-bundle # Pin the expected OIDC signer. aicr evidence verify recipes/evidence///.yaml \ --expected-issuer https://token.actions.githubusercontent.com \ --expected-identity-regexp '^https://github\.com/myorg/myrepo/\.github/workflows/release\.yaml@refs/tags/.+$' # CI pipelines: JSON output. aicr evidence verify recipes/evidence///.yaml -o result.json -t json ``` See [`demos/evidence.md`](https://github.com/NVIDIA/aicr/blob/main/demos/evidence.md) for a full producer-and-consumer walkthrough. > **Stale root:** If verification fails with certificate chain errors, run `aicr trust update` to refresh the Sigstore trusted root. --- ### aicr trust update Fetch the latest Sigstore trusted root **and Rekor v2 signing config** from the TUF CDN and update the local cache at `~/.sigstore/root/`. This is needed when Sigstore rotates signing keys (a few times per year). The trusted root is verification material (which signatures are valid); the signing config is sign-side (which Rekor/timestamp endpoints signing writes to — AICR signs to Rekor v2 by default). **Synopsis:** ```shell aicr trust update [--emit-signing-config ] ``` This command contacts `tuf-repo-cdn.sigstore.dev`, verifies the update chain against the embedded TUF root, and writes the result to `~/.sigstore/root/`. **Flags:** | Flag | Type | Description | |------|------|-------------| | `--emit-signing-config` | string | Also write the fetched Rekor v2 signing config to this file path, for tools that take a signing-config path (e.g. `cosign attest-blob --signing-config`). | **When to run:** - After initial installation (the install script runs this automatically) - When `aicr verify` reports a stale or expired trusted root - When Sigstore announces key rotation **Example:** ```shell aicr trust update aicr trust update --emit-signing-config signing-config.json ``` --- ### aicr skill Generate an AI agent skill file that teaches a coding agent how to use the AICR CLI. The generated file is written to the agent's standard configuration directory. **Synopsis:** ```shell aicr skill --agent [flags] ``` **Flags:** | Flag | Type | Default | Description | |------|------|---------|-------------| | `--agent` | string | (required) | Target coding agent: `claude-code`, `codex` | | `--stdout` | bool | false | Print to stdout instead of writing to disk | | `--force` | bool | false | Overwrite an existing skill file without prompting | **Install Locations:** | Agent | Path | |-------|------| | `claude-code` | `~/.claude/skills/aicr/SKILL.md` | | `codex` | `~/.codex/skills/aicr/SKILL.md` | **Behavior:** - Without `--stdout`: writes the file to disk and prints the path - With `--stdout`: prints the generated content to stdout - If the target file already exists: prompts `overwrite? [y/N]` when stdin is a terminal; aborts on non-interactive stdin unless `--force` is set - Creates parent directories as needed **Examples:** ```shell # Install Claude Code skill file aicr skill --agent claude-code # Install Codex skill file aicr skill --agent codex # Overwrite an existing skill file without prompting (e.g., in CI) aicr skill --agent claude-code --force # Print to stdout (e.g., for review before installing) aicr skill --agent claude-code --stdout ``` --- ## Complete Workflow Examples ### File-Based Workflow ```shell # Step 1: Capture system configuration aicr snapshot --output snapshot.yaml # Step 2: Generate optimized recipe for training workloads aicr recipe \ --snapshot snapshot.yaml \ --intent training \ --output recipe.yaml # Step 3: Check readiness constraints (K8s version, OS, kernel) against the snapshot. # --no-cluster deploys no validator Jobs: the phase checks need the components # installed in Step 5, so they are reported as skipped here. aicr validate \ --recipe recipe.yaml \ --snapshot snapshot.yaml \ --no-cluster # Step 4: Create deployment bundle aicr bundle \ --recipe recipe.yaml \ --output ./deployment # Step 5: Deploy to cluster chmod +x deployment/deploy.sh && ./deployment/deploy.sh # Step 6: Verify deployment (exits non-zero on a failed check) aicr validate \ --recipe recipe.yaml \ --snapshot snapshot.yaml \ --phase deployment kubectl get pods -n gpu-operator kubectl logs -n gpu-operator -l app=nvidia-operator-validator ``` ### ConfigMap-Based Workflow (Kubernetes-Native) ```shell # Step 1: Agent captures snapshot to ConfigMap (using CLI deployment) # --namespace gpu-operator deploys the agent there so it has the ConfigMap RBAC # (namespace-scoped Role) to write into the gpu-operator namespace. aicr snapshot --namespace gpu-operator --output cm://gpu-operator/aicr-snapshot # The CLI handles agent deployment automatically # No manual kubectl steps needed # Step 2: Generate recipe from ConfigMap aicr recipe \ --snapshot cm://gpu-operator/aicr-snapshot \ --intent training \ --output recipe.yaml # Alternative: Write recipe to ConfigMap as well aicr recipe \ --snapshot cm://gpu-operator/aicr-snapshot \ --intent training \ --output cm://gpu-operator/aicr-recipe # With custom kubeconfig (if not using default) aicr recipe \ --snapshot cm://gpu-operator/aicr-snapshot \ --kubeconfig ~/.kube/prod-cluster \ --intent training \ --output recipe.yaml # Step 3: Check readiness constraints against the cluster snapshot # (--no-cluster deploys no validator Jobs, so phase checks are skipped until # Step 5 deploys the components; reading the cm:// snapshot still uses the # Kubernetes API) aicr validate \ --recipe recipe.yaml \ --snapshot cm://gpu-operator/aicr-snapshot \ --no-cluster # Step 4: Create bundle from recipe aicr bundle \ --recipe recipe.yaml \ --output ./deployment # Step 5: Deploy to cluster chmod +x deployment/deploy.sh && ./deployment/deploy.sh # Step 6: Verify deployment (exits non-zero on a failed check) aicr validate \ --recipe recipe.yaml \ --snapshot cm://gpu-operator/aicr-snapshot \ --phase deployment kubectl get pods -n gpu-operator kubectl logs -n gpu-operator -l app=nvidia-operator-validator ``` ### E2E Testing Validate the complete workflow: ```shell # Run all CLI integration tests (no cluster needed) make e2e # Run a single chainsaw test (AICR_BIN must be exported and absolute) export AICR_BIN="$(find "$(pwd)/dist" -maxdepth 2 -type f -name aicr | head -n 1)" chainsaw test --no-cluster --test-dir tests/chainsaw/cli/recipe-generation ``` ## Shell Completion Generate shell completion scripts: ```shell # Bash aicr completion bash # Zsh aicr completion zsh # Fish aicr completion fish # PowerShell aicr completion pwsh ``` **Installation:** **Bash:** ```shell source <(aicr completion bash) # Or add to ~/.bashrc for persistence echo 'source <(aicr completion bash)' >> ~/.bashrc ``` **Zsh:** ```shell source <(aicr completion zsh) # Or add to ~/.zshrc echo 'source <(aicr completion zsh)' >> ~/.zshrc ``` ## Environment Variables AICR respects standard environment variables: | Variable | Description | Default | |----------|-------------|---------| | `KUBECONFIG` | Path to Kubernetes config file | `~/.kube/config` | | `AICR_DEBUG` | Same as the global `--debug` flag | unset | | `AICR_LOG_JSON` | Same as the global `--log-json` flag | unset | | `AICR_LOG_LEVEL` | Logging level for the `aicrd` API server: debug, info, warn, error. The `aicr` CLI does not read it; use `--debug` / `AICR_DEBUG` instead | info | | `AICR_LOG_PREFIX` | Override the CLI logger prefix | `cli` | | `AICR_REQUESTS` | Default for `aicr snapshot --requests`. Comma-separated `name=quantity` pairs (e.g. `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`). Unspecified resources keep the built-in privileged or restricted defaults. | unset | | `AICR_LIMITS` | Default for `aicr snapshot --limits`. Comma-separated `name=quantity` pairs (e.g. `cpu=1,memory=2Gi,ephemeral-storage=2Gi`). Unspecified resources keep the built-in defaults. With `--require-gpu`, the default `nvidia.com/gpu=1` is applied only when this list does not already contain that key — explicit `nvidia.com/gpu=N` wins. | unset | | `AICR_CRITERIA_STRICT` | When set to `1` / `true` / `yes` / `on`, equivalent to `--criteria-strict` on every `aicr recipe` invocation: rejects criteria values not in the embedded OSS catalog regardless of `--data` contributions. Intended for OSS CI gates; `make qualify` exports it automatically for the unit-test step. | unset | | `NO_COLOR` | Suppress ANSI color codes in CLI logger output (de-facto standard, see [no-color.org](https://no-color.org/)) | unset | ## Exit Codes | Code | Meaning | |------|---------| | 0 | Success | | 1 | General error (unclassified) | | 2 | Invalid input (bad arguments, validation failure) | | 3 | Not found (requested resource does not exist) | | 4 | Unauthorized (authentication or authorization failure) | | 5 | Timeout (operation exceeded time limit) | | 6 | Unavailable (service temporarily unavailable) | | 7 | Rate limited (client exceeded rate limit) | | 8 | Internal error (unexpected failure) | | 9 | Operation aborted by the operator (SIGINT/SIGTERM) | ## Common Usage Patterns ### Quick Recipe Generation ```shell aicr recipe --os ubuntu --accelerator h100 --service eks --intent training --format json | jq '.componentRefs[]' ``` ### Save All Steps ```shell aicr snapshot -o snapshot.yaml aicr recipe -s snapshot.yaml --intent training -o recipe.yaml aicr bundle -r recipe.yaml -o ./bundles ``` ### JSON Processing ```shell # Extract GPU Operator version from recipe aicr recipe --os ubuntu --accelerator h100 --service eks --intent training --format json | \ jq -r '.componentRefs[] | select(.name=="gpu-operator") | .version' # Get all component versions aicr recipe --os ubuntu --accelerator h100 --service eks --intent training --format json | \ jq -r '.componentRefs[] | "\(.name): \(.version)"' ``` ### Multiple Environments ```shell # Generate recipes for different cloud providers. # os is per-service: GKE only ships a COS-based recipe, EKS/AKS ship Ubuntu. for pair in eks:ubuntu gke:cos aks:ubuntu; do service=${pair%%:*} os=${pair##*:} aicr recipe --os "$os" --service "$service" --gpu h100 --intent training \ --output recipe-${service}.yaml done ``` ## Troubleshooting ### Snapshot Fails ```shell # Check Kubernetes access kubectl cluster-info # Run with debug. The CLI streams the agent Job's logs to stderr and prints # them again if the Job fails. The snapshot needs no NVIDIA driver, so # nvidia-smi is not a prerequisite. aicr --debug snapshot ``` For an agent Job that stays pending, fails, or produces no output, see [Agent Deployment troubleshooting](/aicr/user-guide/agent-deployment#troubleshooting). ### Recipe Not Found ```shell # A partial query can be ambiguous: no single recipe covers h100+ubuntu # without also stating a service (and, for some services, an intent), so this fails with an # actionable error instead of silently returning a partial recipe: aicr recipe --os ubuntu --gpu h100 # error: os 'ubuntu' requires additional criteria; supported combinations: # (service=aks), (service=eks), (service=bcm, intent=training) # Fix by adding one of the combinations the error lists: aicr recipe --os ubuntu --gpu h100 --service eks --intent training ``` ### Bundle Generation Fails ```shell # Verify recipe file cat recipe.yaml # List available flags aicr bundle --help # Run with debug aicr --debug bundle -r recipe.yaml ``` **"helm CLI not found on PATH" with `--vendor-charts`** — the bundle-time vendoring path shells out to `helm pull`. Install Helm v3 or later (`brew install helm` / package manager) and re-run, or drop `--vendor-charts` for a registry-referencing bundle. See [Vendoring Charts for Air-Gap](#vendoring-charts-for-air-gap). **"failed to load manifest \ for component \"** — the recipe references a manifest path that does not exist in the current AICR binary's embedded data. This usually means the recipe was generated by an older binary and a referenced manifest has since been removed or relocated. Regenerate the recipe with the current binary (`aicr recipe ...`) and re-bundle. AICR recipes are a point-in-time artifact of the binary that produced them; bundling a stale recipe against a newer binary is not supported. **`--deployer argocd-helm`: `aicr-stack` or `-pre` / `-post` Application stuck at `Unknown` sync status / "Failed to load target state: ... `/:: not found`"** — Argo CD cannot resolve the OCI artifact the parent or path-based child Application points at. Common causes: 1. **Chart name doubled in `--set repoURL`.** Under the current contract, `--set repoURL` carries the **parent namespace only** (e.g., `oci://ghcr.io/myorg`). The parent Application appends `.Chart.Name` into its OCI `source.repoURL`, and path-based children append it directly into their rendered `source.repoURL`. For non-OCI Helm repositories, the parent uses `source.chart` instead. Passing `--set repoURL=oci://ghcr.io/myorg/aicr-bundle` produces a double-suffixed reference (`.../aicr-bundle/aicr-bundle:`) that does not exist. Drop the trailing chart segment. 2. **Argo CD older than v2.13.** Path-based children rely on Argo CD's generic OCI artifact source type, added in v2.13. Older Argo treats the source as Git and fails to resolve. Check with `kubectl -n argocd get deploy argocd-repo-server -o jsonpath='{.spec.template.spec.containers[0].image}'`. Upgrade Argo, or use `--deployer helm` if Argo upgrade is not an option. 3. **Tag missing from the registry.** Verify the published artifact exists at the exact tag the parent expects: `oras manifest fetch //:`. If `aicr bundle` is invoked without a tag (`oci:////` with no `:` suffix), the CLI version is used as the default — make sure `--set targetRevision=` at install time matches. 4. **Private registry credentials keyed to a different source URL.** Problem: Argo CD matches repository credentials against the source URL it dereferences. Failure case: For this deployer, path-based OCI Applications render full `oci:////` source URLs even though `--set repoURL` is the parent namespace. A Secret keyed only to `/` or to a scheme-less Helm-OCI URL may let local `helm install` succeed while Argo's repo-server still returns 401. Solution: Key the Argo CD repository credential to the rendered `oci://.../` prefix, or to a broader matching prefix allowed by your cluster's credential policy, such as `oci:///` or `oci:////`. ## External Data Directory The `--data` flag enables extending or overriding the embedded recipe data with external files. This allows customization without rebuilding the CLI. ### Overview AICR embeds recipe data (overlays, component values, registry) at compile time. The `--data` flag layers an external directory on top, enabling: - **Custom components**: Add new components to the registry - **Override values**: Replace default component values files - **Custom overlays**: Add new recipe overlays for specific environments - **Registry extensions**: Add custom components while preserving embedded ones ### Directory Structure The external directory must mirror the embedded data structure: ``` my-data/ ├── registry.yaml # REQUIRED - merged with embedded registry ├── overlays/ │ ├── base.yaml # Optional - replaces embedded base.yaml │ └── custom-overlay.yaml # Optional - adds new overlay └── components/ └── gpu-operator/ └── values.yaml # Optional - replaces embedded values ``` ### Requirements 1. **registry.yaml is required**: The external directory must contain a `registry.yaml` file 2. **Security validations**: Symlinks are rejected, file size is limited (10MB default) 3. **No path traversal**: Paths containing `..` are rejected ### Merge Behavior | File Type | Behavior | |-----------|----------| | `registry.yaml` | **Merged** - External components are added to embedded; same-named components are replaced | | Profile-only `overlays/*.yaml` | **Extended**. Adds values to the profile of the embedded overlay at the same path | | All other files | **Replaced** - External file completely replaces embedded if path matches | ### Usage Examples ```shell # Use external data directory for recipe generation aicr recipe --service eks --accelerator h100 --data ./my-data # Use external data directory for bundle generation aicr bundle --recipe recipe.yaml --data ./my-data --output ./bundles # Combine with other flags aicr recipe --service eks --gpu gb200 --intent training \ --data ./custom-recipes \ --output recipe.yaml ``` ### Example: Adding a Custom Component 1. **Create external data directory:** ```shell mkdir -p my-data/components/my-operator ``` 2. **Create registry.yaml with custom component:** ```yaml # my-data/registry.yaml apiVersion: aicr.run/v1beta1 kind: ComponentRegistry components: - name: my-operator displayName: My Custom Operator helm: defaultRepository: https://my-charts.example.com defaultChart: my-operator defaultVersion: v1.0.0 ``` 3. **Create values file for the component:** ```yaml # my-data/components/my-operator/values.yaml replicaCount: 1 image: repository: my-registry/my-operator tag: v1.0.0 ``` 4. **Create overlay that includes the component:** ```yaml # my-data/overlays/my-custom-overlay.yaml kind: RecipeMetadata apiVersion: aicr.run/v1beta1 metadata: name: my-custom-overlay spec: criteria: service: eks intent: training componentRefs: - name: my-operator type: Helm valuesFile: components/my-operator/values.yaml ``` 5. **Generate recipe with external data:** ```shell aicr recipe --service eks --intent training --data ./my-data ``` ### Debugging External Data Use `--debug` flag to see detailed logging about external data loading: ```shell aicr --debug recipe --service eks --data ./my-data ``` Debug logs include: - External files discovered and registered - File source resolution (embedded vs external) - Registry merge details (components added/overridden) ## Example Files The `examples/` directory contains reference files for testing and learning: ### Recipes (`examples/recipes/`) | File | Description | |------|-------------| | `kind.yaml` | Recipe for local Kind cluster with fake GPU | | `eks-training.yaml` | EKS recipe optimized for training workloads | | `aks-training.yaml` | AKS recipe for training workloads | | `eks-gb200-ubuntu-training-with-validation.yaml` | GB200 on EKS with Ubuntu and multi-phase validation | **Usage:** ```shell # Generate bundle from example recipe aicr bundle --recipe examples/recipes/eks-training.yaml --output ./bundles ``` ### Templates (`examples/templates/`) | File | Description | |------|-------------| | `snapshot-template.md.tmpl` | Go template for custom snapshot report formatting | **Usage:** ```shell # Generate custom cluster report aicr snapshot --template examples/templates/snapshot-template.md.tmpl --output report.md ``` ## See Also - [Installation Guide](/aicr/user-guide/installation) - Install aicr - [Agent Deployment](/aicr/user-guide/agent-deployment) - Kubernetes agent setup - [API Reference](/aicr/user-guide/api-reference) - Programmatic access - [Architecture Docs](../contributor/) - Internal architecture - [Data Architecture](/aicr/contributor-guide/recipes-overlays-and-mixins) - Recipe data system details