CLI Reference

Complete reference for the aicr command-line interface.

For details on which CLI verbs and critical user journeys are exercised by tests, on what hardware, and at what cadence, see the Coverage Matrix.

Version numbers in examples (component, chart, and driver versions) are illustrative and may not match any released recipe. The authoritative, current versions live in the Component Catalog and the Container Images BOM.

Overview

AICR provides a four-step workflow for optimizing GPU infrastructure:

┌──────────────┐      ┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│   Snapshot   │─────▶│    Recipe    │─────▶│   Validate   │─────▶│    Bundle    │
└──────────────┘      └──────────────┘      └──────────────┘      └──────────────┘

Step 1: Capture system configuration Step 2: Generate optimization recipes Step 3: Validate constraints against cluster Step 4: Create deployment bundles

Global Flags

Available for all commands:

Logging Modes

AICR supports three logging modes:

1. CLI Mode (default): Minimal user-friendly output

   • Just message text without timestamps or metadata
   • Error messages display in red (ANSI color)
   • Example: Snapshot captured successfully

2. Text Mode (--debug): Debug output with full metadata

   • Key=value format with time, level, source location
   • Example: time=2025-01-06T10:30:00.123Z level=INFO module=aicr version=v1.0.0 msg="snapshot started"

3. JSON Mode (--log-json): Structured JSON for automation

   • Machine-readable format for log aggregation
   • Example: {"time":"2025-01-06T10:30:00.123Z","level":"INFO","msg":"snapshot started"}

Examples:

# Default: Clean CLI output
aicr snapshot
# Debug mode: Full metadata
aicr --debug snapshot
# JSON mode: Structured logs
aicr --log-json snapshot
# Combine with other flags (--debug is global; --output is a snapshot flag, so it follows the subcommand)
aicr --debug snapshot --output system.yaml

Commands

aicr snapshot

Capture comprehensive system configuration including OS, GPU, Kubernetes, and SystemD settings.

Synopsis:

aicr snapshot [flags]

Flags:

Output Destinations:

• stdout: Default when no -o flag specified
• File: Local file path (/path/to/snapshot.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

Output Formats: --format applies to every destination.

• yaml (default) delivers the agent’s document byte-for-byte to a file or stdout, so fields emitted by a newer --image than the CLI survive.
• json re-encodes that document with the same keys, which is what aicr diff --target snapshot.json and jq expect. Since aicr diff picks its decoder from the file extension, pair --format json with a .json path.
• table is a flattened FIELD/VALUE rendering for humans and cannot be read back by aicr diff, aicr validate --snapshot, or aicr recipe --snapshot.
• ConfigMap destinations store the rendering under the snapshot.yaml, snapshot.json, or snapshot.txt data key alongside a format key; AICR’s ConfigMap readers follow that key, so yaml and json are both consumable from cm://. Because those keys and the resource labels are derived from the document, a cm:// destination re-serializes it (deterministically, and without dropping unmodeled fields) rather than storing the agent’s exact bytes — use a file or stdout when you need byte-identical YAML.
• --template supplies its own rendering and therefore requires --format yaml (or no --format).

What it captures:

• SystemD Services: containerd, docker, kubelet configurations
• OS Configuration: grub, kmod, sysctl, release info
• Kubernetes: server version, images, ClusterPolicy
• GPU: driver-free PCI/NFD hardware detection — GPU presence, GPU count, accelerator model/SKU (resolved from the PCI device ID), nvidia kernel-module loaded state, and detection source. Does not require the NVIDIA driver or nvidia-smi, and does not emit driver version, CUDA version, or MIG settings.
• NodeTopology: node topology (cluster-wide taints and labels across all nodes)
• NetworkTopology: per-hardware-group network topology (PFs, capabilities, kernel modules, machine/GPU type, fabric type) — emitted only when --cluster-config or --discover-network is set

Examples:

# Output to stdout (YAML)
aicr snapshot
# Save to file (JSON)
aicr snapshot --output system.json --format json
# Save to Kubernetes ConfigMap (requires cluster access)
# The agent Job writes the ConfigMap directly, using the Role created in its
# deployment namespace (--namespace, default `default`). To write into another
# namespace, deploy the agent there so it has the matching ConfigMap RBAC:
aicr snapshot --namespace gpu-operator --output cm://gpu-operator/aicr-snapshot
# Debug mode
aicr --debug snapshot
# Table format (human-readable)
aicr snapshot --format table
# With custom kubeconfig
aicr snapshot --kubeconfig ~/.kube/prod-cluster
# Auto-target GPU nodes (default when no placement flag is set)
# On a cluster with nvidia.com/gpu.present=true nodes, the agent is
# steered onto a GPU node automatically — see GPU Node Auto-Targeting.
aicr snapshot --namespace gpu-operator
# Targeting specific nodes (explicit selector disables auto-targeting)
aicr snapshot \
  --namespace gpu-operator \
  --node-selector accelerator=nvidia-h100 \
  --node-selector zone=us-west1-a
# With tolerations for tainted nodes
# (By default all taints are tolerated - only needed to restrict tolerations)
aicr snapshot \
  --toleration nvidia.com/gpu=present:NoSchedule
# Full example with all options
aicr snapshot \
  --kubeconfig ~/.kube/config \
  --namespace gpu-operator \
  --image ghcr.io/nvidia/aicr:v0.19.0 \
  --job-name snapshot-gpu-nodes \
  --node-selector accelerator=nvidia-h100 \
  --toleration nvidia.com/gpu:NoSchedule \
  --timeout 10m \
  --output cm://gpu-operator/aicr-snapshot \
  --no-cleanup
# Write the RBAC manifests that grant the agent's permissions to an existing
# ServiceAccount, then exit. Applies nothing and contacts no cluster; takes no
# snapshot. Writes ./snapshot-rbac-<run-id>/.
aicr snapshot \
  --namespace gpu-operator \
  --add-roles-to-service-account irsa-snapshotter
# Review what each file grants, then apply them yourself.
kubectl apply -f snapshot-rbac-<run-id>/
# Capture as that ServiceAccount. Because it already exists, the name is used
# verbatim and this run creates and deletes no RBAC of its own.
aicr snapshot \
  --namespace gpu-operator \
  --service-account-name irsa-snapshotter \
  --output cm://gpu-operator/aicr-snapshot
# Custom template formatting
aicr snapshot --template examples/templates/snapshot-template.md.tmpl
# Template with file output
aicr snapshot --template examples/templates/snapshot-template.md.tmpl --output report.md
# With custom template
aicr snapshot \
  --namespace gpu-operator \
  --template examples/templates/snapshot-template.md.tmpl \
  --output cluster-report.yaml

Snapshot Config File Mode

Drive aicr snapshot from an AICRConfig document so the snapshot inputs version-control alongside the recipe, bundle, and validate steps in an end-to-end workflow.

kind: AICRConfig
apiVersion: aicr.run/v1beta1
metadata:
  name: gke-h100-training
spec:
  snapshot:
    output:
      path: snapshot.yaml          # written to disk; same shape as -o
      format: yaml                 # yaml | json | table
      template: ""                 # optional Go template path
    agent:
      namespace: aicr-validation
      image: ""                    # default: matches the CLI version (:latest on dev/-next builds)
      imagePullSecrets: []
      # jobName is an optional PREFIX, not a name — the run ID is always
      # appended. serviceAccountName is exact-if-exists: an existing
      # ServiceAccount of exactly that name is used verbatim and the run
      # then creates and deletes NO RBAC; otherwise it is a prefix too.
      # Omit both to take the run-scoped defaults.
      nodeSelector:
        nodeGroup: gpu-worker
      tolerations:
        - dedicated=gpu-workload:NoSchedule
        - nvidia.com/gpu=present:NoSchedule
      requireGpu: false
      runtimeClassName: ""         # mutually exclusive with requireGpu
      os: ""                       # ubuntu | rhel | cos | amazonlinux | ol | talos
      requests: ""                 # "cpu=500m,memory=1Gi"
      limits: ""                   # "cpu=1,memory=2Gi"
    execution:
      timeout: 5m
      noCleanup: false
      privileged: true             # set false for PSS-restricted namespaces
      maxNodesPerEntry: 0          # 0 = unlimited topology entries

Precedence: a CLI flag always wins over the matching config field. Selectors and tolerations omitted entirely inherit the snapshotter’s compiled-in defaults (tolerations defaults to tolerate all taints); an explicit empty list (tolerations: []) clears the tolerate-all default — the same nil-vs-empty semantics used by spec.validate.agent.

# Run snapshot driven entirely by config
aicr snapshot --config aicr-config.yaml
# Reuse the same config but write to a one-off path
aicr snapshot --config aicr-config.yaml -o /tmp/snapshot.yaml

Custom Templates

The --template flag enables custom output formatting using Go templates with Sprig functions. Templates receive the full Snapshot struct:

# Available template data structure:
.Kind           # Resource kind ("Snapshot")
.APIVersion     # API version string
.Metadata       # Map of key-value pairs (timestamp, version, source-node)
.Measurements   # Array of Measurement objects
  .Type         # Measurement type (K8s, GPU, OS, SystemD, NodeTopology, NetworkTopology)
  .Subtypes     # Array of Subtype objects
    .Name       # Subtype name (e.g., "server", "hardware", "grub")
    .Data       # Map of readings (key -> Reading with .String method)
# NodeTopology measurement type has subtypes: summary, taint, label
# Taint encoding: effect|value|node1,node2,...  (parseable with Sprig splitList "|")
# Label encoding: value|node1,node2,...

Example template extracting key cluster info:

cluster:
  kubernetes: {{ with index .Measurements 0 }}{{ range .Subtypes }}{{ if eq .Name "server" }}
    version: {{ (index .Data "version").String }}{{ end }}{{ end }}{{ end }}
  gpu: {{ range .Measurements }}{{ if eq .Type.String "GPU" }}{{ range .Subtypes }}{{ if eq .Name "hardware" }}
    model: {{ (index .Data "model").String }}
    count: {{ (index .Data "gpu-count").String }}{{ end }}{{ end }}{{ end }}{{ end }}

See examples/templates/snapshot-template.md.tmpl for a complete example template that generates a concise cluster report.

Agent Deployment

When running against a cluster, AICR deploys a Kubernetes Job to capture the snapshot. For the RBAC the agent creates, the in-cluster Job lifecycle, ConfigMap storage, and GPU-node auto-targeting (proactive selector injection plus the reactive placement-mismatch warning), see Agent Deployment.

ConfigMap Output

When using ConfigMap URIs (cm://namespace/name), the snapshot is stored directly in Kubernetes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aicr-snapshot
  namespace: gpu-operator
  labels:
    app.kubernetes.io/name: aicr
    app.kubernetes.io/component: Snapshot
    app.kubernetes.io/version: <aicr-version>
data:
  snapshot.yaml: |
    # Full snapshot content
  format: yaml
  timestamp: "2025-12-31T10:30:00Z"

Snapshot Structure:

apiVersion: aicr.run/v1
kind: Snapshot
metadata:
  timestamp: "2025-12-31T10:30:00Z"
  version: v1.0.0
  source-node: gpu-node-1
measurements:
  - type: SystemD
    subtypes: [...]
  - type: OS
    subtypes: [...]
  - type: K8s
    subtypes: [...]
  - type: GPU
    subtypes: [...]

aicr recipe

Generate optimized configuration recipes from query parameters or captured snapshots.

Synopsis:

aicr recipe [flags]

Modes:

aicr recipe resolves a recipe from criteria — service, accelerator, os, intent, platform. (nodes is accepted but is advisory metadata — it does not select or filter overlays.) You can supply those criteria three ways, composed with the precedence CLI flags > --config file > --snapshot:

• Snapshot (--snapshot) — criteria are auto-detected from a captured cluster snapshot: accelerator from the nvidia.com/gpu.product GFD label (primary — cluster-wide, so it surfaces heterogeneous clusters) or the per-node PCI device ID (fallback — maps the device ID to a SKU, e.g. h100, with no driver or GFD label required); service from the node’s cloud-provider ID; OS from the node’s OS release; node count from cluster topology. Use this when you already have a running cluster: you don’t hand-specify the hardware, AICR reads it. (A detected SKU fills criteria.accelerator only when it’s in the supported accelerator set; an unsupported GPU is recorded descriptively but not as a recipe criterion.) On the snapshot path AICR also reads the sampled GPU node’s driver-loaded reading; on unprofiled/legacy compositions whose resolved values already carry the coordinated preinstalled-driver configuration, it injects gpu-operator.overrides.driver.enabled=false into the resolved recipe, while on a profiled AKS composition the snapshot only qualifies the explicit-or-default gpuStack selection — the injector skips profile-owned paths and never mutates them; the inverse mismatch — a preinstalled-driver overlay resolved against a snapshot with no driver loaded — is handled per family: on profiled AKS it either fails closed at the profile constraint (pools not reading Install) or records gpuDriverState: absent for the bundle-time gate whose remedy is pool repair/recreation + recapture + --profile; on non-profiled families and legacy artifacts it warns with the bundle-time override set. See Component Catalog › GPU Operator Driver Auto-Detect for the gate, warnings, and the pre-deploy-snapshot requirement.
• Config file (--config) — criteria (and bundle settings) from an AICRConfig document; good for reproducible, version-controlled workflows.
• Query flags (--service, --accelerator, …) — state the criteria directly. Use this when there is no cluster to snapshot yet — the common case when you generate a recipe in order to provision a cluster, or build one offline/ahead of time for hardware you can’t reach.

Why you sometimes specify criteria the snapshot could detect: recipe generation does not collect live cluster state or deploy/modify workloads — it reads inputs (criteria, embedded/--data catalog, an optional snapshot) and does not snapshot a cluster for you. Its only cluster interaction is explicit cm:// ConfigMap reads/writes (snapshot input or recipe output). This keeps generation hermetic and reproducible (same inputs → same recipe) and lets it run before a target cluster even exists. So you state criteria explicitly when there’s no cluster to read; when a cluster is available, capture it first and let detection fill them in:

aicr snapshot -o cm://default/snapshot      # captures the cluster (deploys the collector agent)
aicr recipe   -s cm://default/snapshot --intent training

When both a snapshot and explicit criteria are given, the explicit values win (e.g. --snapshot … --service gke overrides the detected service). This is also the only way to select a generic recipe from a snapshot: a bare-metal cluster fingerprints as its provisioner (metal3, rke2), never as generic, so pass the override explicitly:

aicr recipe --snapshot snapshot.yaml --service generic --intent training

Configuration profiles: an overlay composition may declare one named configuration choice. Select a non-default value with --profile name=value; omitting the flag applies the declaration’s required default. A selection against a composition with no declaration, a wrong name, or an unknown value fails closed. Profile-bearing output records metadata.selectedProfile, uses recipe apiVersion aicr.run/v1beta2, and locks every declared owned path: divergent aicr bundle/aicr mirror static overrides are rejected (identical values accepted), and argocd-helm install-time values are rejected on key presence alone — even when the value is identical to the selected one. The AKS family is the first embedded adopter: gpuStack with values azure-managed (default) and operator-managed — see AKS GPU setup. The GKE family declares gpuStack with values gke-default (default; GKE’s managed plugin stays the advertiser — recorded as advertiser: external — for default-provisioned clusters with no node label) and bundle-installer (the GPU Operator’s device plugin owns nvidia.com/gpu; GPU node pools carry gke-no-default-nvidia-gpu-device-plugin=true and, because that label forfeits GKE’s managed driver install, are created gpu-driver-version=disabled — the bundle’s gcp-driver-installer component supplies the driver with a recipe-pinned version); because the GKE values govern advertisement, the #1327 allocation-policy paths are closure-locked in addition to the declared owned paths — see GKE GPU setup and Component Catalog › GKE Device-Plugin Ownership. The OKE family declares gpuStack with values oci-managed (default; Oracle’s GPU node image supplies the driver and OKE’s NvidiaGpuPlugin add-on advertises — advertiser: external) and operator-managed (bring-your-own driverless image with the add-on removed; the operator installs driver, toolkit, and plugin, with the DRA driver root in lockstep). Each value is qualified by the add-on’s control-plane state, supplied as an oci ce cluster list-addons --cluster-id <cluster-ocid> --all --output json dump via --oke-addons on aicr snapshot and aicr validate — see OKE GPU setup. Profiles can also be exercised through a versioned external overlay.

Selection and verification are independent: --profile (or the default) always decides the selected value — never the snapshot — and a supplied --snapshot always verifies the selection against the cluster’s recorded readings (fail-closed on mismatch or a missing reading; on AKS, the pool mode from --aks-gpu-pools). Without a snapshot no check can run at generation; the recorded constraint is enforced at aicr validate readiness instead. See the with/without matrices in AKS GPU setup and GKE GPU setup.

Every stated criteria dimension must be honored: for service, accelerator, intent, os, and platform, resolution now enforces a coverage post-condition — if you state a value for one of these dimensions, at least one applied overlay must carry that exact value, or the request fails with an actionable INVALID_REQUEST error instead of silently returning a recipe that ignores what you asked for. The error names the uncovered dimension and, where possible, the additional criteria that would make the request resolvable (e.g. platform 'kubeflow' ... requires os (valid: ubuntu)). --nodes is exempt from this check — it is advisory only, since no overlay gates on node count, and stating it never requires matching node-count-specific recipe content to exist.

Snapshot-detected dimensions are advisory, not strict: on the --snapshot path, service, accelerator, and os can be auto-detected from the cluster fingerprint rather than stated by you. If the coverage post-condition above fails and every uncovered dimension came from that detection (none of them were set via a CLI flag or --config), AICR relaxes those dimensions back to unstated and retries resolution once, logging a warning that names each relaxed dimension and its detected value — this handles overlay trees that are deliberately agnostic to a dimension the snapshot still reports (e.g. an OS-agnostic Kind overlay tree observing os=ubuntu on the node). Dimensions you passed explicitly via a flag are never relaxed: if any uncovered dimension was explicitly stated, the error still fails the request.

Config File Mode (Recommended)

Generate recipes using an AICRConfig document. The same file format also drives the bundle command, so a single file can describe an end-to-end recipe-to-bundle workflow.

Flags:

The config file uses a Kubernetes-style envelope:

kind: AICRConfig
apiVersion: aicr.run/v1beta1
metadata:
  name: gb200-eks-ubuntu-training
spec:
  recipe:
    criteria:
      service: eks
      os: ubuntu
      accelerator: gb200
      intent: training
      nodes: 8
      platform: slurm
    configuration:
      slurm:
        accounting:
          mode: disabled
    output:
      path: recipe.yaml
      format: yaml

Individual CLI flags always override config file values. For slice/map flags, presence on the CLI replaces the file’s value (no append).

For a composition that declares a profile, the equivalent config field is spec.recipe.profile. The CLI flag takes precedence. For example, on the AKS family (the embedded adopter):

spec:
  recipe:
    criteria:
      service: aks
      os: ubuntu
      accelerator: h100
      intent: training
    profile: gpuStack=operator-managed

# Load criteria from config file
aicr recipe --config config.yaml
# Override service from file
aicr recipe --config config.yaml --service gke
# Save output to file
aicr recipe --config config.yaml -o recipe.yaml
# Load config from a URL (e.g. CI shared template)
aicr recipe --config https://team.example.com/configs/eks-h100-training.yaml

--config accepts a local file path or an HTTP/HTTPS URL. ConfigMap (cm://) sources are not supported; export the data with kubectl get cm <name> -o yaml and pass the resulting file.

Query Mode

Generate recipes using direct system parameters:

Flags:

Accelerator values name a GPU model, not a machine type. A provider usually offers several machine types for the same GPU, and the machine type — not the GPU — determines the fabric, the NIC count, and which components a recipe can use. --accelerator h100 therefore does not, on its own, say which node shape the resolved recipe targets. See Qualified Machine Types below.

Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set. When --data registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see Data Extension. --criteria-strict restores the OSS-only set regardless of what --data contributes.

--service rke2 and --accelerator vr200 are Preview. They publish an early-adopter recipe path without the full production support and lifecycle qualification required for Supported status. See the published validation evidence at validation.aicr.run for current coverage.

--service k0s is Preview, covering the single k0s / h200 / ubuntu / training coordinate. See k0s H200 Setup for its prerequisites and known gaps.

Examples:

# Basic recipe for Ubuntu on EKS with H100
aicr recipe --os ubuntu --service eks --accelerator h100 --intent training
# Training workload with multiple GPU nodes
aicr recipe \
  --service eks \
  --accelerator gb200 \
  --intent training \
  --os ubuntu \
  --nodes 8 \
  --format yaml
# Kubeflow training workload
aicr recipe \
  --service eks \
  --accelerator h100 \
  --intent training \
  --os ubuntu \
  --platform kubeflow
# Save to file (--gpu is an alias for --accelerator)
aicr recipe --os ubuntu --gpu h100 --service eks --intent training --output recipe.yaml
# Slurm with an AICR-provided accounting database installation
aicr recipe \
  --service eks \
  --accelerator h100 \
  --intent training \
  --os ubuntu \
  --platform slurm \
  --slurm-accounting-mode aicr-provided

The AICR-provided example above is criteria-only: because it does not use --snapshot, the resulting recipe carries no target-cluster MariaDB Operator conflict evidence. Bundling warns that conflicts were not evaluated and proceeds for compatibility. Use a current snapshot before deployment when you need conflict detection. See Conflict detection requires snapshot evidence.

Qualified Machine Types

Each recipe is qualified against a specific node shape. Criteria resolution does not reject another machine type of the same GPU model — there is no axis to reject it on — so a recipe always resolves. What differs by family is what happens afterwards: on some, deployment validation fails; on others it succeeds and only the performance gates are affected.

A row that names no intent applies to every intent for that accelerator and service. Where a row names a family rather than a machine type, the entry is a family-level statement — either because no component in that family binds to a machine type, or because the specific shape is not recorded in-repo. The row says which.

Two distinct failure modes are worth separating:

• Component-level (hard). Two families fail deployment outright, by different mechanisms. The GKE H100 training lineage pins artifacts to a machine type — h100-gke-cos-training and the leaves inheriting it — so on a non-matching shape the DaemonSets have nowhere to land and a Chainsaw health check fails. The AKS H100 training lineage instead wires an RDMA fabric unconditionally, and a Go readiness gate in the deployment phase fails closed when no node advertises the shared RDMA resource — which is every non-IB NCads shape. Neither is a degradation; both stop the deployment phase.
• Performance-gate (soft). Elsewhere the recipe deploys normally, but the NCCL and inference floors are fixed absolute values calibrated on full, high-bandwidth nodes. They are not normalized for GPU count or fabric class, so a smaller shape can fail a gate while being perfectly healthy. This is the EKS and GB200 case; on AKS the deployment gate above bites first. See Validation › Node-shape assumption. Normalizing these floors per GPU or per fabric class was considered and declined (#1256, #1254, both closed as not planned) — the floors are deliberately fixed absolute full-node values, so running a qualified shape is the supported way to pass them.

The table lists the accelerator/service pairs that have a qualified shape; a pair or a shape absent from it is undocumented rather than known-broken. It has not been qualified, and the criteria model has no axis that would distinguish it from one that has. Whether AICR should gain one — finer-grained accelerator values, a machine-type axis, or a fabric class — is tracked in #2377.

Snapshot Mode

Generate recipes from captured snapshots:

Snapshots derive infrastructure criteria such as service, accelerator, and OS, but intent and platform remain explicit choices. K8s.slinky-slurm reports declared Slinky Controller presence and, for a single Controller, a secret-safe associated-resource topology; it does not select Slurm or reconstruct the installed chart’s Helm values. K8s.mariadb-operator records official MariaDB Operator API/CR conflict evidence but does not infer database availability, accounting intent, or database source. Pass --platform slurm to resolve a Slurm leaf. Older snapshots without either subtype remain compatible. For aicr-provided, missing MariaDB Operator conflict evidence causes bundling to warn that conflicts were not evaluated and proceed without target-cluster conflict detection.

Flags:

Snapshot Sources:

• File: Local file path (./snapshot.yaml)
• URL: HTTP/HTTPS URL (https://example.com/snapshot.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

Examples:

# Generate recipe from local snapshot file
aicr recipe --snapshot system.yaml --intent training
# Resolve the Slurm leaf from snapshot-derived infrastructure criteria
aicr recipe --snapshot system.yaml --intent training --platform slurm \
  --slurm-accounting-mode customer-managed
# From ConfigMap (requires cluster access)
aicr recipe --snapshot cm://gpu-operator/aicr-snapshot --intent training
# From ConfigMap with custom kubeconfig
aicr recipe \
  --snapshot cm://gpu-operator/aicr-snapshot \
  --kubeconfig ~/.kube/prod-cluster \
  --intent training
# Output to ConfigMap
aicr recipe -s system.yaml -o cm://gpu-operator/aicr-recipe
# Chain snapshot → recipe with ConfigMaps
aicr snapshot -o cm://default/snapshot
aicr recipe -s cm://default/snapshot -o cm://default/recipe
# With custom output
aicr recipe -s system.yaml --intent inference -o recipe.yaml --format yaml

Output structure:

apiVersion: aicr.run/v1beta2
kind: RecipeResult
metadata:
  version: v1.0.0
  appliedOverlays:
    - base
    - eks
    - eks-training
    - gb200-eks-training
criteria:
  service: eks
  accelerator: gb200
  intent: training
  os: any
  platform: slurm
configuration:
  slurm:
    accounting:
      mode: disabled
componentRefs:
  - name: gpu-operator
    type: Helm
    version: vXX.Y.Z          # illustrative; see Component Catalog for current pins
    source: https://helm.ngc.nvidia.com/nvidia
    chart: gpu-operator
deploymentOrder:
  - gpu-operator
constraints:
  - name: driver.version
    value: "<driver-version>"     # illustrative
  - name: driver.cudaVersion
    value: "<cuda-version>"       # illustrative

A profiled result applies the selected value’s constraints and component overrides to the normal recipe fields. It also uses a profile-aware recipe apiVersion and records the selected identity and declaration-wide lock surface:

apiVersion: aicr.run/v1beta2
kind: RecipeResult
metadata:
  selectedProfile:
    name: gpuStack
    value: azure-managed
    ownedPaths:
      gpu-operator:
        - driver.enabled
        - enabled
        - operator.runtimeClass
        - toolkit.enabled
      nvidia-dra-driver-gpu:
        - enabled
        - nvidiaDriverRoot

aicr recipe list

Enumerate overlay recipes in the catalog. Useful for discovering which criteria combinations have a dedicated leaf overlay versus an intermediate shared recipe.

Each leaf overlay also carries a structural-health verdict (ADR-009 §4): a rolled-up status and a per-phase declared-coverage summary, computed by resolving the recipe and inspecting the result. Intermediate (non-leaf) overlays are not scored — only leaf overlays resolve to a concrete combination.

Synopsis:

aicr recipe list [flags]

Flags:

Filter flags narrow the output to overlays whose criteria carry that exact value. Unspecified flags match all overlays for that dimension. Multiple filters are combined with AND.

Output fields:

In table format the health axis is rendered as two extra columns: STATUS (the rolled-up verdict) and COVERAGE, a compact per-phase named-check summary of the form R:2 D:4 P:1 C:10 (Readiness / Deployment / Performance / Conformance). Non-leaf overlays render - in both columns. A dimension whose grader cannot reach a confident verdict surfaces as unknown, and the status column still renders.

Resolving every leaf overlay to compute its health verdict adds latency on each invocation. For purely interactive “what overlays exist?” lookups — or scripted callers that only consume name/criteria/is_leaf/source — pass --no-health (alias --skip-health) to skip the health computation entirely. With the flag set the table format omits the STATUS/COVERAGE columns and the json/yaml formats omit the health block, leaving only the enumeration columns/fields.

Examples:

# List all overlays as a table (default)
aicr recipe list
# List all overlays as JSON
aicr recipe list --format json
# Filter to EKS training overlays
aicr recipe list --service eks --intent training
# Filter to H100 overlays and emit JSON
aicr recipe list --accelerator h100 --format json
# Include external overlays from a custom data directory
aicr recipe list --data /etc/aicr/custom-recipes --format yaml
# Skip the structural-health computation for a faster enumeration-only listing
aicr recipe list --no-health

Example table output:

Intermediate (non-leaf) overlays render - in the STATUS/COVERAGE columns; only leaf overlays are scored.

NAME                SERVICE  ACCELERATOR  INTENT    OS   PLATFORM  IS_LEAF  STATUS  COVERAGE         SOURCE
gb200-eks-training  eks      gb200        training  any  any       false    -       -                embedded
gb200-any           any      gb200        any       any  any       true     pass    R:0 D:4 P:0 C:0  embedded

Example JSON output:

The criteria keys are capitalized because the criteria struct carries no field tags; the structured output mirrors the Go field names. The health block is present only for leaf overlays — non-leaf overlays omit it.

[
  {
    "name": "gb200-any",
    "criteria": {"Service": "any", "Accelerator": "gb200", "Intent": "", "OS": "", "Platform": "", "Nodes": 0},
    "is_leaf": true,
    "source": "embedded",
    "health": {
      "status": "pass",
      "dimensions": {"resolves": "pass", "chart_pinned": "pass"},
      "coverage": {
        "readiness": {"declared": false, "constraints": 0},
        "deployment": {
          "declared": true,
          "checks": ["check-nvidia-smi", "expected-resources", "gpu-operator-version", "operator-health"],
          "constraints": 1
        },
        "performance": {"declared": false, "constraints": 0},
        "conformance": {"declared": false, "constraints": 0}
      }
    }
  }
]

aicr recipe verify-catalog

Verify the embedded recipe catalog (registry.yaml + validators/catalog.yaml) against its Sigstore bundle. The bundle is distributed as recipe-catalog.sigstore.json release asset alongside each tagged aicr binary.

aicr recipe verify-catalog recomputes a deterministic SHA-256 over the embedded catalog content using a length-prefixed encoding of the two raw files, then verifies the digest against the Sigstore bundle using NVIDIA CI identity pinning. Exit code is 0 on success, non-zero on any verification failure.

Synopsis:

aicr recipe verify-catalog <bundle-path> [flags]

Flags:

Examples:

# Download the catalog signature for a tagged release and verify it.
curl -Lo recipe-catalog.sigstore.json \
  https://github.com/NVIDIA/aicr/releases/download/vX.Y.Z/recipe-catalog.sigstore.json
aicr recipe verify-catalog recipe-catalog.sigstore.json

Output:

On success, prints the verified content digest and the Fulcio certificate identity (the GitHub Actions workflow that signed the release):

catalog verified
  digest:   sha256:<hex>
  identity: https://github.com/NVIDIA/aicr/.github/workflows/on-tag.yaml@refs/tags/vX.Y.Z

aicr query

Query a specific value from the fully hydrated recipe configuration. Resolves a recipe from criteria (same as aicr recipe), merges all base, overlay, and inline value overrides, then extracts the value at the given dot-path selector.

Because aicr query resolves criteria the same way aicr recipe does, it is subject to the same coverage post-condition: a stated service/accelerator/intent/os/platform value that no overlay honors fails the query with an actionable error rather than silently extracting a value from an unrelated recipe. --nodes remains advisory and is never required to be covered.

Synopsis:

aicr query --selector <path> [flags]

Flags:

All aicr recipe flags are supported, including --profile and --inherit-from, plus:

Selector Syntax

Uses dot-delimited paths consistent with Helm --set and yq:

Leading dots are optional (yq-style): .components.gpu-operator.chart and components.gpu-operator.chart are equivalent.

Output:

• Scalar values (string, number, bool) are printed as plain text — no YAML wrapper
• Complex values (maps, lists) are printed as YAML (default) or JSON (--format json)

Examples:

# Get a specific Helm value
aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator.values.driver.version
# stdout: <driver-version>   # illustrative; actual value comes from the resolved recipe
# Get a value subtree
aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator.values.driver
# stdout:
#   version: "<driver-version>"
#   repository: nvcr.io/nvidia
# Get the full hydrated component
aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator
# Get deployment order
aicr query --service eks --accelerator h100 --intent training \
  --selector deploymentOrder
# Use in shell scripts
VERSION=$(aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator.values.driver.version)
echo "Driver version: $VERSION"
# JSON output for complex values
aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator.values --format json
# Query from snapshot
aicr query --snapshot snapshot.yaml \
  --selector components.gpu-operator.values.driver.version
# Full hydrated recipe
aicr query --service eks --accelerator h100 --intent training --selector .

Advanced Examples:

# Cross-cloud comparison: Prometheus storage across providers
# EKS provisions a 50Gi persistent EBS volume (gp2)
aicr query --service eks --intent training \
  --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec
# GKE also provisions a 50Gi persistent volume, but only under the COS overlay —
# GKE has no OS-agnostic recipe, so os=cos must be stated explicitly
aicr query --service gke --os cos --intent training \
  --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec
# Compare deployment order across clouds
# EKS deploys 14 components (includes aws-ebs-csi-driver, aws-efa, nodewright-customizations)
aicr query --service eks --accelerator h100 --intent training --selector deploymentOrder
# GKE (COS) deploys 14 components (includes gke-nccl-tcpxo and gcp-driver-installer; storage is otherwise platform-managed)
aicr query --service gke --os cos --accelerator h100 --intent training --selector deploymentOrder
# Pin the exact driver version into Terraform/Pulumi variables
DRIVER_VERSION=$(aicr query --service eks --accelerator h100 --intent training \
  --selector components.gpu-operator.values.driver.version)
echo "gpu_driver_version = \"${DRIVER_VERSION}\""
# Compare nodewright tuning parameters across accelerators
# H100: real tuning packages (kernel setup, nvidia-tuned, full setup)
aicr query --service eks --accelerator h100 --intent training \
  --selector components.nodewright-customizations.values
# GB200: same manifest and value structure; the accelerator/intent overrides
# select the gb200 profile instead
aicr query --service eks --accelerator gb200 --intent training \
  --selector components.nodewright-customizations.values
# GB300 on EKS: same shared setup-and-tuning manifest as gb200
aicr query --service eks --accelerator gb300 --intent training \
  --selector components.nodewright-customizations.values
# Watch constraints tighten as you add specificity
# Just "EKS" → 1 constraint (K8s >= 1.32)
aicr query --service eks --selector constraints
# Add GPU + intent + OS → 4 constraints (K8s >= 1.32.4, Ubuntu 24.04, kernel >= 6.8)
aicr query --service eks --accelerator h100 --intent training --os ubuntu \
  --selector constraints

aicr validate

Validate a system snapshot against the constraints defined in a recipe to verify cluster compatibility. Supports multi-phase validation with different validation stages.

For a task-oriented walkthrough (capture snapshot → generate recipe → run each phase, with worked training and inference examples), see Validation.

Synopsis:

aicr validate [flags]

Flags:

Input Sources:

• File: Local file path (./recipe.yaml, ./snapshot.yaml)
• URL: HTTP/HTTPS URL (https://example.com/recipe.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

Validation Phases

Validation can be run in different phases to validate different aspects of the deployment:

Note: Readiness constraints (K8s version, OS, kernel) are always evaluated implicitly before any phase runs. If readiness fails, validation stops before deploying any Jobs and exits 2 (INVALID_REQUEST). This gate always fails closed — --fail-on-error=false scopes to phase check results and does not downgrade a readiness failure.

Declared-check pre-flight: Every check named under a phase’s checks list must resolve to exactly one catalog validator in that phase. Before any Job is deployed, validate fails closed with exit 2 (INVALID_REQUEST) if a declared check matches no validator (a typo, or a check missing from the loaded --data catalog), exists only under a different phase, or is declared more than once — reporting every offender at once. This runs in --no-cluster mode too, and like readiness it is independent of --fail-on-error. It replaces the previous warn-and-continue behavior, which let a phase with only unresolved checks report skipped and exit 0.

Version skew: Snapshots and recipes record the aicr version that produced them. When the recipe, the snapshot, and the running binary report different release versions, validate logs a single advisory warning (version skew detected across validate inputs) naming all three. This is a debugging breadcrumb — mixing artifacts from different versions can surface as confusing failures — and does not fail the command. Dev (dev) and pre-release (-next) builds are ignored to avoid noise.

apiVersion gate: As of v0.22, the ADR-022 emitter switch, AICR emits aicr.run/v1 for snapshots and default recipes, aicr.run/v1beta1 for config and ordinary catalog inputs, and aicr.run/v1beta2 for profile-bearing recipes. Readers additionally still accept the superseded aicr.run/v1alpha2 and aicr.run/v1alpha3, so artifacts produced by v0.21 or earlier keep loading. Unsupported artifact headers fail fast; raw external catalog headers are checked before merge or hydration. Recapture, regenerate, or update the authored header with a version supported by the running AICR release. See ADR-011 and ADR-022. v1.0.0 stops accepting the alpha values, along with the empty header that the snapshot, recipe, and criteria readers still tolerate. Reading either now logs a deprecation warning naming the file. AICRConfig and external catalog headers already reject an empty value, so they have no tolerance to retire. Catalog and binary compatibility has the release-by-release table.

Phases run sequentially with --phase all and all phases run by default, producing results regardless of earlier failures; use --fail-fast to stop after the first failing phase. For what each phase actually checks (deployment-phase readiness signals, graceful-skip semantics, RBAC, Day-N re-verification, and evidence), see Validation.

Constraint paths and operators

Constraints use fully qualified measurement paths: {Type}.{Subtype}.{Key}

Constraint paths are validated against the measurement catalog when recipe data is loaded, so a typo such as K8s.server.verison fails immediately — naming the file, the field, and the nearest matching key — instead of silently evaluating as a missing reading. See Recipe Development › Constraints for the addressing rules.

Supported operators:

Examples:

# Validate snapshot against recipe (readiness constraints run implicitly)
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml
# Validate specific phase
aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase deployment
# Run all validation phases
aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase all
# Load snapshot from ConfigMap
aicr validate \
  --recipe recipe.yaml \
  --snapshot cm://gpu-operator/aicr-snapshot
# Save results to file
aicr validate \
  --recipe recipe.yaml \
  --snapshot cm://gpu-operator/aicr-snapshot \
  --output validation-results.json
# Validate deployment phase after components are installed
aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase deployment
# Run performance validation
aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase performance
# With custom kubeconfig — selects the cluster for the whole run:
# cm:// I/O, agent deployment, validator Jobs, and cleanup (#1787)
aicr validate \
  --recipe recipe.yaml \
  --snapshot cm://gpu-operator/aicr-snapshot \
  --kubeconfig ~/.kube/prod-cluster
# Write a recipe-evidence attestation bundle (unsigned, on disk).
# --bom is optional: when omitted, aicr synthesizes a recipe-bound BOM from
# the recipe's component refs and validator catalog images.
aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml \
  --emit-attestation ./out
# Writes ./out/summary-bundle/ and ./out/pointer.yaml.
# The bundle is minimized by default: sensitive snapshot fields and CTRF
# logs are removed, and predicate.redaction records the applied policy.
# Ship the full (unredacted) bundle instead — raw snapshot + CTRF stdout.
aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml \
  --emit-attestation ./out --full
# Use an exhaustive BOM (e.g., `make bom`-produced) instead of the auto-generated one
aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml \
  --emit-attestation ./out --bom dist/bom/bom.cdx.json
# Sign and push a recipe-evidence bundle to OCI (cosign keyless via Sigstore public-good).
# Token acquisition follows the same precedence chain as `aicr bundle --attest`:
# pre-fetched COSIGN_IDENTITY_TOKEN > ambient GitHub Actions OIDC > --oidc-device-flow > interactive browser.
aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/myorg/aicr-evidence  # tag optional; aicr derives :<recipe-slug>-<fingerprint>
# After this, copy ./out/pointer.yaml to the per-source path printed in the
# 'copyTo' hint: recipes/evidence/<recipe>/<src>/<bundle-digest>.yaml
# NOTE: keyless --push signing publishes the signer's identity (email + issuer)
# to the public Rekor log. On a TTY, aicr pauses for confirmation first (--yes skips it).
# Validate on a cluster with custom GPU node labels (non-standard labels that AICR doesn't
# recognize by default, e.g., using a custom node pool label instead of cloud-provider defaults)
aicr validate \
  --recipe recipe.yaml \
  --node-selector my-org/gpu-pool=true \
  --phase performance
# Override both node selector and tolerations for a non-standard taint setup
aicr validate \
  --recipe recipe.yaml \
  --node-selector gpu-type=h100 \
  --toleration gpu-type=h100:NoSchedule

Validate Config File Mode

aicr validate --config <path> reads inputs from an AICRConfig YAML/JSON file under spec.validate. CLI flags always override values loaded from --config; override events are logged at INFO so users can see which input won. The OIDC identity token used for --push signing stays out of the schema by design (short-lived tokens must not be committed); the CLI resolves it at sign time through the precedence chain described on --identity-token.

Supported schema:

kind: AICRConfig
apiVersion: aicr.run/v1beta1
metadata:
  name: prod-validate
spec:
  validate:
    input:
      recipe: ./recipe.yaml
      snapshot: ./snapshot.yaml          # optional; omit to capture live
    agent:                               # only used when input.snapshot is empty
      namespace: aicr-validation
      image: ghcr.io/nvidia/aicr:v0.19.0
      imagePullSecrets: [registry-secret]
      # Optional prefixes for the live-capture agent, not names — the run ID
      # is always appended. Omitted here so the defaults apply
      # (both aicr-validate).
      nodeSelector:
        my-org/gpu-pool: "true"
      tolerations:                         # [] clears the live snapshot agent's tolerate-all default
        - "gpu-type=h100:NoSchedule"
      requireGpu: true
    execution:
      phases: [deployment, conformance]
      failOnError: true                  # default; false = don't fail on phase-check results (readiness pre-flight still exits 2)
      noCluster: false
      noCleanup: false
      timeout: 10m
    evidence:
      cncf:                              # --evidence-dir / --cncf-submission / --feature
        dir: ./out/cncf
        cncfSubmission: false
        features: []                     # empty = all features
      attestation:                       # --emit-attestation / --bom / --push / ...
        out: ./out/attestation
        bom: dist/bom/bom.cdx.json       # optional; auto-generated from recipe + validators when absent
        push: ghcr.io/myorg/aicr-evidence  # tag optional; aicr derives :<recipe-slug>-<fingerprint>
        plainHTTP: false
        insecureTLS: false

Examples:

# Use a config file
aicr validate --config validate.yaml
# Override a single config value from the CLI
aicr validate --config validate.yaml --phase deployment
# Validate the same recipe across two clusters using two different agent
# configs (config-bound) without retyping flags
aicr validate --config validate-cluster-a.yaml
aicr validate --config validate-cluster-b.yaml

The --node-selector and --toleration flags control scheduling for the inner validation workloads (NCCL benchmark workers, conformance test pods). When --snapshot is omitted, they also configure the preliminary live snapshot agent. They do not configure the validator orchestrator Job. For when to use them with non-standard GPU labels or taints, see Validation.

Output Structure (CTRF JSON):

Results are output in CTRF (Common Test Report Format) — an industry-standard schema for test reporting.

{
  "reportFormat": "CTRF",
  "specVersion": "0.0.1",
  "timestamp": "2026-03-10T20:10:44Z",
  "generatedBy": "aicr",
  "results": {
    "tool": {
      "name": "aicr",
      "version": "v0.22.0"
    },
    "summary": {
      "tests": 16,
      "passed": 13,
      "failed": 0,
      "skipped": 3,
      "pending": 0,
      "other": 0,
      "start": 1773173400872,
      "stop": 1773173799002
    },
    "tests": [
      {
        "name": "operator-health",
        "status": "passed",
        "duration": 0,
        "suite": ["deployment"],
        "stdout": ["Found 1 gpu-operator pod(s)", "Running: 1/1"]
      },
      {
        "name": "expected-resources",
        "status": "passed",
        "duration": 0,
        "suite": ["deployment"],
        "stdout": ["All deployment resources and required readiness signals are healthy"]
      },
      {
        "name": "nccl-all-reduce-bw",
        "status": "passed",
        "duration": 234000,
        "suite": ["performance"],
        "stdout": ["NCCL All Reduce bandwidth: 488.37 GB/s", "Constraint: >= 100 → true"]
      },
      {
        "name": "inference-perf",
        "status": "passed",
        "duration": 612000,
        "suite": ["performance"],
        "stdout": [
          "RESULT: Inference throughput: 108789.87 tokens/sec",
          "RESULT: Inference TTFT p99: 687.50 ms",
          "Throughput constraint: >= 50000 → PASS",
          "TTFT p99 constraint: <= 2000 → PASS"
        ]
      },
      {
        "name": "dra-support",
        "status": "passed",
        "duration": 8000,
        "suite": ["conformance"],
        "stdout": ["DRA support verified (driver healthy, ResourceSlices validated)"]
      },
      {
        "name": "cluster-autoscaling",
        "status": "skipped",
        "duration": 0,
        "suite": ["conformance"],
        "stdout": ["SKIP reason=\"Karpenter not found\""]
      }
    ]
  }
}

Note: The tests array above is truncated for brevity. A full validation run produces one entry per check across all phases. Each entry includes stdout with detailed diagnostic output.

Test Statuses:

Exit Codes:

aicr diff

Compare two snapshots field-by-field to surface configuration drift between cluster states. Reports added, removed, and modified readings across every measurement type (K8s, GPU, OS, SystemD, NodeTopology, NetworkTopology).

Synopsis:

aicr diff --baseline <path|cm://...> --target <path|cm://...> [flags]

Flags:

Inputs:

• File paths (./baseline.yaml, /tmp/snap.json)
• ConfigMap URIs (cm://gpu-operator/aicr-snapshot)
• Both inputs may mix freely; e.g., a local baseline file vs. a live ConfigMap target.

Output Semantics:

• A nil reading is rendered as the literal <nil> so it cannot be confused with an empty-string value (""). Both forms surface as drift when one side is nil and the other is a concrete value.
• Changes are emitted in deterministic order (sorted by Path) so the diff is reproducible across runs and machines.
• The Result envelope includes baselineSource and targetSource (the supplied paths), a changes array, and a summary with added, removed, modified, and total counts.

Items are ordered and compared by zero-based position. Reordering distinct records is drift. A list-length change emits items.length and field-level additions or removals. Ordinary existing Data paths remain unchanged for compatibility. Data keys equal to context or items, or beginning with context., items., or items[, use JSON-style bracket quoting so they cannot collide with structured paths; for example, subtype Data key context.node is emitted as <type>.<subtype>["context.node"], and the same key in Item Data is emitted as <type>.<subtype>.items[<index>].data["context.node"].

Examples:

# Local-file diff in default YAML
aicr diff --baseline before.yaml --target after.yaml
# Human-readable table to stdout
aicr diff -b before.yaml --target after.yaml --format table
# CI/CD gate: non-zero exit on drift, JSON to a file
aicr diff -b before.yaml --target after.yaml \
  --format json --output drift.json --fail-on-drift
# Compare two ConfigMaps in the cluster
aicr diff \
  --baseline cm://gpu-operator/aicr-snapshot-baseline \
  --target   cm://gpu-operator/aicr-snapshot
# Mix file + ConfigMap (golden baseline vs live cluster)
aicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot

Exit Codes:

Note on CI gating: A non-zero exit identifies that drift was detected, but doesn’t by itself distinguish drift from malformed input — both map to exit 2. To differentiate without relying on stderr format (text by default; JSON only with --log-json), inspect the diff payload directly: write the result with --output drift.json --format json and branch on the presence of the file plus its summary.total field. That signal is format-stable regardless of logging mode.

aicr upgrade-check

Compare two recipes or bundles component by component and report, for each component whose version or namespace changed, whether moving between them is safe to apply. Verdicts come from the transition records the running aicr release ships. No cluster state is inspected, which makes this the CI and GitOps path: the comparison reads two artifacts and nothing else. A cm:// path is an artifact location like a file path, so reading or writing one does contact that cluster’s API for the ConfigMap itself.

Synopsis:

aicr upgrade-check --from <recipe|bundle> [--to <recipe|bundle>] [--deployer <name>] [flags]

Flags:

Note the two defaults that differ from sibling commands. --format defaults to table rather than yaml, because the report’s payload is a list of operator steps that folded YAML scalars make unreadable. --fail-on-error defaults to true, the opposite of aicr diff --fail-on-drift: you chose to run this check, so its exit code is what makes running it worth something in a pipeline.

The two questions it answers:

--from X --to Y asks “is this specific move safe?” and presumes you already know your target.

Omitting --to asks “am I behind, and does catching up hurt?”, which is usually the real question: what an operator holds is an old artifact, not a chosen destination. The --from artifact’s embedded criteria are re-resolved against the running binary’s pins to synthesize the target.

Verdicts:

Anything other than safe exits non-zero. unknown is included deliberately: a transition nobody assessed is not a transition anyone approved, and the distance moved does not change that.

The report still reports a breaking boundary (a major bump, a minor bump while the major version is 0, or a changed prerelease identifier over an otherwise unchanged major.minor.patch) in the NOTES cell and the JSON breaking field, because the size of a move tells you how hard to look. It no longer affects the exit code.

Four kinds of unknown. They differ in what would close the gap, so they carry different reason codes and different report text:

blocked and unknown say opposite things. blocked means AICR has something to tell you and a version to stop at: read it and act on it. unknown means AICR has nothing for you: read the component’s own upstream release notes and decide. Neither is a pass.

Rollout note: expect red today. Only two registry components ship a transition record so far, so most components that change version report unknown and the check exits non-zero on most comparisons. That is a coverage problem being worked (#2535 makes records mandatory per pin bump), not a tool limitation, and it shrinks as records are authored. Use --fail-on-error=false if you want the report without the gate in the meantime.

Components whose version and namespace are identical on both sides produce no row. Added components are reported with nothing to do; removed components are reported and stay installed, because AICR does not uninstall them.

Namespaces are compared too. A recipe is regenerated from scratch on every AICR upgrade, and each component’s namespace comes from recipes/registry.yaml, so a moved registry default lands in the regenerated recipe. Helm cannot move a release between namespaces, so applying the resulting bundle installs a second copy of the component beside the running one, and nothing reconciles the two. A version-only comparison reports that as no change at all, so the namespace is compared as its own axis:

unknown rather than blocked is deliberate. blocked is an author’s judgement recorded against a version boundary, and the record vocabulary has no way to express one about where a release lives, so no author can record it. --format json and --format yaml carry the move as identityChanges, a list of {field, from, to} objects, on both kinds of row.

The remedy is not an upgrade step. Either move the release deliberately and re-run the check, or resolve the target recipe with aicr recipe --inherit-from so it keeps the namespaces the prior artifact deployed into and the relocation never enters the hop. See Upgrading a Deployed Stack.

The four routes to blocked. A record is crossed when your source version sits below the boundary its to names and your target reaches it. That is a property of the jump alone, so a record still counts even when the jump flies straight over it:

A crossed safe boundary does not count toward the second row: it carries no steps by construction, so crossing one composes nothing and skips nothing, and a jump whose only substantive boundary describes the whole move reports that record rather than stopping. The first row renders its record’s steps, deployer-scoped, exactly as a manual row does: the author marked the move blocked and then wrote what to do instead. The other three render none, because the record that carries them describes a different move than the one you asked about. All four name a stopping point.

The fourth exists because a record vouches only as far as its own to ceiling. A record claiming >=0.18.0 <0.19.0 says nothing about 0.25.0, and letting it lend its verdict there would report eight minors as safe on the strength of a two-minor claim. Stop at the assessed ceiling and re-run, or have the record widened. This is also how a component deliberately held below a breaking release reports: its record’s ceiling sits at that release, so any target above it is told to stop there and take the boundary on its own.

Every row states its reason in the detail block under the table, and --format json carries the same thing as reason (a stable code: recorded, record-blocks, multiple-boundaries, undefined-origin, beyond-record-ceiling, no-record, no-boundary-crossed, downgrade, identity-changed, not-comparable) plus explanation, the sentence naming your versions.

Why --deployer is required rather than defaulted:

Steps are deployer-scoped. Showing an Argo CD operator an imperative “delete the legacy CRDs” step is the exact failure deployer-scoping exists to prevent, so the command asks rather than guessing, and never renders every deployer’s path. It is only required when some component actually carries steps. A bundle now records the deployer that built it in bundle-info.yaml, so upgrade-check can stop asking once it reads that record (#2528).

Example:

$ aicr upgrade-check --from old-recipe.yaml --to new-recipe.yaml --deployer helm
UPGRADE CHECK
  from      old-recipe.yaml
  to        new-recipe.yaml
  deployer  helm
COMPONENT  FROM            TO               VERDICT  NOTES
---------  ----            --               -------  -----
grove      v0.1.0-alpha.8  v0.1.0-alpha.12  manual   3 steps
grove v0.1.0-alpha.8 -> v0.1.0-alpha.12  (manual)
  alpha.12 drops the clustertopologies.grove.io CRD (kind ClusterTopology) in
  favor of clustertopologybindings.grove.io, which reuses the shortname ct.
  PRECONDITION
    No ClusterTopology objects exist in the cluster.
  STEPS (deployer: helm)
    1. delete-legacy-crd
       kubectl delete crd clustertopologies.grove.io
    ...
1 component change, 1 needs attention

More examples:

# Two recipes, for a pipeline that already knows its deployer
aicr upgrade-check --from old-recipe.yaml --to new-recipe.yaml --deployer argocd
# Am I behind, and does catching up hurt?
aicr upgrade-check --from ./bundles-v0.16.0 --deployer helm
# JSON for a pipeline, reporting without gating
aicr upgrade-check --from old.yaml --to new.yaml \
  --format json --output report.json --fail-on-error=false

Exit Codes:

Note on CI gating: as with aicr diff, a bad invocation and a failing check both exit 2. To tell them apart without parsing stderr, write the report with --format json --output report.json and branch on the file’s presence plus its summary.failing count.

Limitations:

• A bundle is read through the recipe.yaml at its root. Every deployer writes one as of #2759; a directory without it is neither a recipe nor a bundle and is rejected rather than misread.
• Coverage starts near zero. Every transition without an authored record reports unknown. See the authoring guide.
• A namespace move is seen only when both artifacts state one. An absent namespace is read as a field the artifact did not carry, not as the default, so a component that gains or loses an explicit namespace between the two artifacts produces no relocation row. Reading it the other way would report a move nobody performed for every component the moment one side stopped carrying the field.
• No cluster comparison yet. --from cluster, which reads installed Helm release inventory, is tracked in #2531.

aicr bundle

Generate deployment-ready bundles from recipes containing Helm values, manifests, scripts, and documentation.

Synopsis:

aicr bundle [flags]

Flags: