Component Catalog

AICR recipes are composed of components — the individual software packages that make up a GPU-accelerated Kubernetes runtime. This page lists every component that can appear in a recipe.

Note: Components are included as appropriate in recipes. Not every component listed here will appear in a recipe.

The source of truth is recipes/registry.yaml. Each entry in the registry defines the component’s Helm chart (or Kustomize source), default version, namespace, and node scheduling configuration. If a component is not listed there, it cannot appear in a recipe.

See also: Recipe Health reports the structural health of every recipe these components compose into — resolvability and chart-pin hygiene across the whole criteria matrix.

Components

How Components Are Selected

Not every component appears in every recipe. The recipe engine selects components based on the overlay chain for your environment:

• Base components (cert-manager, kube-prometheus-stack) appear in most recipes.
• Cloud-specific components (aws-efa, aws-ebs-csi-driver) are added when the service matches. OCP recipes replace base components (gpu-operator, nfd, network-operator, cert-manager) with OLM+CR pairs where a certified operator exists (e.g., gpu-operator-ocp-olm + gpu-operator-ocp, cert-manager-ocp-olm + cert-manager-ocp). Components with no certified OCP operator (for example, prometheus-adapter, nvidia-dra-driver-gpu, k8s-nim-operator) instead reuse the same upstream Helm chart as their base component, with OCP-specific manifests (SCC RoleBindings, RBAC, CA bundle injection) layered on top. k8s-nim-operator-ocp is available on OCP via the ocp-inference-nim overlay and depends on cert-manager-ocp for webhook TLS.
• Intent-specific components (agentgateway, agentgateway-crds) are added based on workload intent (e.g., inference recipes include the inference gateway).
• Platform-specific components (slinky-slurm-operator, slinky-slurm, kubeflow-trainer, dynamo-platform) are added when the recipe selects a matching --platform. For --platform slurm, all three core Slinky pieces (slinky-slurm-operator-crds, slinky-slurm-operator, slinky-slurm) are declared inline per slurm leaf overlay — the same shape dynamo-platform uses across *-inference-dynamo leaves. IMEX-capable Slurm leaves attach a fixed ComputeDomain through slinky-slurm.preManifestFiles so slurmd pods can consume DRA-provisioned IMEX channels. Leaves that want the operator only inline the CRDs + operator and omit the slinky-slurm componentRef. For an end-to-end walkthrough (recipe → bundle → install → validate → srun smoke job on AKS, EKS, GKE, or Kind), see demos/cuj1-slinky-slurm.md.
• Topology-aware optional components (slinky-topograph) are not installed by default. Opting in requires an explicit componentRef entry for slinky-topograph in the leaf overlay — the componentRef is what installs it; dependencyRefs alone does not. That componentRef declares slinky-slurm as a dependencyRef, so Topograph deploys after the Slurm cluster chart, which owns the ConfigMap Topograph patches. See the wiring example in the Recipe Development Guide.
• Accelerator/OS-specific tuning (nodewright-customizations, nvidia-dra-driver-gpu) varies by hardware and OS combination.

NFD Topology Updater

Production GPU leaf recipes (H100, GB200, RTX Pro 6000 on EKS / AKS / GKE / OKE / LKE) enable the NFD Topology Updater. It publishes per-node NodeResourceTopology CRDs that describe NUMA zones, GPU-to-NUMA affinity, and NIC-to-NUMA affinity. Runtime consumers (NUMA-aware schedulers, debugging via kubectl get noderesourcetopologies) can read these CRDs without further configuration.

The Topology Updater requires the kubelet podResources gRPC socket. The KubeletPodResources feature gate has been on by default since Kubernetes 1.15 (Beta) and reached GA in Kubernetes 1.28; AICR’s recipe constraints require K8s ≥ 1.32, so this is satisfied in practice. Recipes targeting Kubernetes < 1.15 must enable the feature gate explicitly. Kind / KWOK simulated clusters do not run a real kubelet and therefore leave the Topology Updater disabled — kind-based recipes will not see NodeResourceTopology CRDs.

See the upstream Topology Updater docs for runtime consumer examples.

GPU Operator Driver Auto-Detect

When a recipe is resolved from a snapshot (aicr recipe --snapshot snap.yaml, or the ResolveRecipeFromSnapshot SDK entry point), AICR reads the sampled GPU node’s driver-loaded measurement and, when the NVIDIA kernel module is already loaded, injects components.gpu-operator.overrides.driver.enabled=false into the resolved recipe. On recipes whose ADR-015 profile owns driver.enabled (the AKS family), the injector is subordinated: the profile fragment owns the path, the injector skips it without mutating (logging the skip), and the fragment’s value is authoritative. Subordination follows path ownership, not mere profile presence — the GKE family is also profiled (gpuStack owns devicePlugin.enabled, not driver.enabled), so the injection/teardown discussion in this section applies to GKE-COS exactly as to unprofiled compositions (OKE, legacy AKS artifacts). The override lands at the top of the merge chain (base values.yaml → ValuesFile → Overrides), so the rendered Helm values a deployer installs carry driver.enabled: false regardless of what the resolved overlay’s values file would default to. This prevents the GPU Operator from installing a second driver on top of one the platform has already provisioned. Explicit --set flags at bundle generation (aicr bundle --set gpuoperator:driver.enabled=true) retain higher precedence and can supersede the injection unless the path is profile-owned — on recipes carrying metadata.selectedProfile (the AKS family’s gpuStack), a --set diverging from the selected value on an owned path fails closed at bundle time. --set is a bundle-time flag, not an aicr recipe flag.

Injection is gated on the resolved overlay already declaring driver.enabled=false in its merged base+valuesFile. That marker check inspects driver.enabled alone; the shipped preinstalled-driver overlays additionally carry coordinated ownership settings (AKS and OKE set toolkit.enabled=false; GKE-COS keeps the toolkit enabled with the COS-specific toolkit.installDir under the host-managed driver root) plus hostPaths.driverInstallDir, and the bundle-time CheckDriverOwnershipCoherence validation enforces full ownership coherence for any recipe. That scopes auto-detect to overlays like AKS, GKE-COS, and OKE where every dependent setting is already aligned. Bare EKS overlays lack the marker; the auto-detect skips them and logs a warning (gpu-operator driver auto-detect: pre-installed driver observed …) telling the operator to use a preinstalled-profile overlay rather than land a half-configured Operator (driver off, toolkit and gdrcopy still enabled with no operator-managed driver root). The case is still tracked as separate work:

• EKS — GPU-optimized AMIs that ship an NVIDIA driver preinstalled on the AMI itself. Today this warns; a full preinstalled EKS overlay is tracked separately.

On preinstalled-driver overlays whose profile does not own driver.enabled (GKE-COS) or that carry no profile (OKE, legacy AKS artifacts), the injection is semantically idempotent — the rendered driver.enabled value is unchanged; the resolved recipe records the override explicitly (visible in aicr recipe -o recipe.yaml) so the reason for the value is auditable end-to-end. On profiled AKS there is no injection at all: the gpuStack fragment writes the value and the injector skips the owned path (see above). The AKS default is the AKS azure-managed profile (driver.enabled=false, toolkit.enabled=false, operator.runtimeClass=nvidia-container-runtime — the Azure default, where the node image preinstalls driver and toolkit); a --gpu-driver none pool selects the operator-managed profile value at recipe time (--profile gpuStack=operator-managed, see the GPU Operator-managed profile).

The inverse mismatch (no NVIDIA driver loaded on the sampled GPU node while the resolved overlay declares the preinstalled-driver profile) is handled differently depending on whether the family carries an ADR-015 configuration profile:

• On AKS (profiled), a pool reading that mismatches the SELECTED value fails closed at resolution. Selection comes from --profile (or the azure-managed default); the K8s.aks-gpu-pools.gpu-driver reading then verifies it before any driver-state post-processing. Pools reading None fail the azure-managed default but qualify --profile gpuStack=operator-managed — rerun with that selection against the same snapshot; no pool change or recapture is needed. Mixed and Managed reject either selection naming the observed state — fix the pools and recapture; a missing reading (snapshot captured without --aks-gpu-pools) rejects with “reading unavailable” — recapture with the pool dump (see the AKS GPU Operator-managed profile).
• On AKS (profiled), Install-mode pools whose sampled node has no driver loaded still enter the record-and-gate flow. Pool mode is the ownership contract, not live state: gpu-driver: Install satisfies the azure-managed constraint even while a failed AKS driver install or a mid-reimage node samples no loaded driver. Resolution succeeds, records metadata.gpuDriverState: absent, and the bundle-time CheckDriverOwnershipCoherence gate blocks aicr bundle with the AKS remedy (repair the pools and recapture, or switch pools to --gpu-driver none, recapture, and regenerate with --profile gpuStack=operator-managed). The driver-ownership paths are profile-owned, so the pre-profile per-path --set override tuple is rejected at bundle time.
• On families without a profile (and legacy pre-profile AKS artifacts), every inverse mismatch takes that same warn-record-gate path: resolution logs a warning, records metadata.gpuDriverState: absent in the recipe, and the bundle-time CheckDriverOwnershipCoherence validation blocks aicr bundle (#1757) unless the values are flipped to operator-managed mode or the GPU pools are reprovisioned with the platform’s default driver install and re-snapshotted. Ownership overrides are bundle-time flags there, so resolution itself cannot fail hard without cutting off the supported override path — bundle generation is the first point where the final effective values are known.

One per-OS limitation is intentionally out of the check’s scope: the check verifies value coherence (driver ownership and driver-root lockstep), not per-OS install capability. On GKE COS node images a deliberate --set gpuoperator:driver.enabled=true clears the gate — the values are internally coherent — but the GPU Operator cannot install a driver on COS, so the deployment fails at deploy time, where the gpu-operator-health deployment-phase check is the backstop. The check’s GKE remedy therefore points COS clusters at the GKE-managed driver install (gpu-driver-version) rather than the override tuple.

The policy is only-false: the auto-detect never forces driver.enabled=true, so recipes resolved without a snapshot (or targeting a node without a loaded driver) fall back to today’s static defaults. Two operational consequences:

• Criteria-only resolves (aicr recipe --service ... --accelerator ...) and no-cluster mode see zero behavior change — no snapshot, no override.
• A stale snapshot from an older CLI that omits the driver-loaded reading is treated as unknown, not absent, so it cannot flip a hardened overlay.

Capture the snapshot BEFORE deploying the GPU Operator (unprofiled compositions; on profiled AKS the pool reading, not driver-loaded, gates resolution, and a post-deploy snapshot cannot flip an owned path). The driver-loaded reading is installer-agnostic — it reports whether the nvidia kernel module is currently loaded, not who loaded it. A snapshot taken after a prior AICR deploy has run the operator’s driver container will still report driver-loaded=true, and a re-resolve from that post-deploy snapshot would flip a working overlay toward driver.enabled=false, tearing the operator-managed driver DaemonSet down and leaving new or rebooted GPU nodes driverless. AICR emits a gpu-operator driver auto-detect: driver-loaded=true AND a ClusterPolicy is already present… warning when both signals appear together in the same snapshot, but the guard is observability, not prevention: a pre-deploy snapshot is the intended workflow.

The signal is a single-node sample: the snapshotter Job runs on one nvidia.com/gpu.present=true node, so its driver-loaded reading is representative only when every GPU pool is in the same driver state. Mixed-pool clusters (some nodes with a preinstalled driver, some without) are out of scope for the auto-detect and tracked in #464; AICR emits a topology reports non-uniform GPU labels… warning when the snapshot’s node-topology labels indicate divergent GPU nodes so the fail-direction (some non-preinstalled pools may come up driverless) is at least observable.

To see exactly which components appear in a given recipe, generate one:

aicr recipe --service eks --accelerator h100 --os ubuntu --intent training -o recipe.yaml

The output lists every component with its pinned version and configuration values.

GKE Device-Plugin Ownership

Device-plugin ownership is a configuration profile. The GKE recipes declare an ADR-015 gpuStack profile with two qualified values, selected at recipe generation and recorded in metadata.selectedProfile:

• gke-default (the default) — GKE’s managed device plugin is the nvidia.com/gpu advertiser (recorded as advertiser: external), and the recipe disables the GPU Operator’s plugin (devicePlugin.enabled: false, profile-owned). Its constraint requires that no GPU node — identified by its cloud.google.com/gke-accelerator label — carries the opt-out label gke-no-default-nvidia-gpu-device-plugin. This is the default GKE cluster shape: create GPU node pools normally (with gpu-driver-version=default or latest for GKE’s managed driver install) and no further cluster setup is required.
• bundle-installer (aicr recipe ... --profile gpuStack=bundle-installer) — the GPU Operator’s device plugin is the sole advertiser (devicePlugin.enabled: true, profile-owned), and the constraint inverts: every GPU node must carry gke-no-default-nvidia-gpu-device-plugin=true on pools created gpu-driver-version=disabled. The bundle’s gcp-driver-installer component supplies the driver — the version is pinned in the recipe and upgrades roll with the bundle; nothing is applied by hand.

For the end-to-end setup flow (snapshot → recipe → validate), the qualification and selection-vs-verification matrices, and troubleshooting, see GKE GPU Setup.

Why exactly one advertiser: two plugins registering nvidia.com/gpu on one node is not a benign overlap. Kubelet’s device manager keys its endpoint and device inventory by resource name, so competing registrations and ListAndWatch updates replace each other. Ownership becomes nondeterministic, and one plugin’s device IDs (GKE uses nvidia0-style names, NVIDIA uses GPU UUIDs) can reach the other plugin’s Allocate. Expect intermittent allocation and runtime failures.

Bundle-installer cluster setup. The opt-out label forfeits GKE’s managed driver install: the managed install (gpu-driver-version=default/latest) is finalized by an init container of the same kube-system DaemonSet the label disables, so a labeled pool paired with gpu-driver-version=default comes up driverless — never combine the label with the managed driver install. Pools for the bundle-installer value must be created with gpu-driver-version=disabled; the bundle’s gcp-driver-installer component carries the installer DaemonSet, so there is nothing to apply out-of-band. (AICR’s GKE-COS overlays keep driver.enabled: false in either mode — the GPU Operator cannot install a driver on COS node images.) A previously hand-applied standalone nvidia-driver-installer DaemonSet must be deleted before deploying the bundle: the bundle’s DaemonSet shares its name in kube-system and Helm will not adopt the pre-existing object. The operational procedures live in GKE GPU Setup.

aicr validate enforces the selected value deterministically, before any phase runs. The selected value’s NodeTopology.gpu-nodes.label constraint (#1755) is verified against the snapshot at recipe generation and re-evaluated by the validate readiness pre-flight. The check fails closed: labels contradicting the selected value, mixed labels, an empty GPU-node set, and readings that --max-nodes-per-entry actually truncated (a cap larger than the node count truncates nothing and validates normally) all fail with exit 2 and remediation text pointing back at this section, before any check Jobs deploy. See Validation for the readiness-gate mechanics.

The profile also locks the ownership tuple: devicePlugin.enabled is profile-owned, and because both values govern advertisement, the #1327 allocation-policy paths (devicePlugin.enabled, DRA resources.gpus.enabled / gpuResourcesEnabledOverride) are closure-locked — a bundle- or install-time override diverging at any of them is rejected rather than warned. Switching modes is a recipe-generation decision (--profile), never a --set.

The selected constraint is the only deterministic detection point. aicr bundle is offline by design and cannot read node labels. The operator-health deployment check passes under the conflict because it verifies only that GPU Operator controller pods are Running — it never inspects the device plugin. Allocation probes such as check-nvidia-smi schedule a pod requesting nvidia.com/gpu on each schedulable GPU node, but skip cordoned nodes and skip entirely when any schedulable GPU node is busy; when they do run, they may fail nondeterministically without identifying the missing label as the cause.

See GKE’s GPU node-pool guide for the authoritative pool-creation procedures. The NVIDIA GPU Operator GKE guide documents the same gpu-driver-version=disabled + installer-DaemonSet combination the bundle-installer value builds on (#1716) — with AICR, that DaemonSet ships inside the bundle rather than being applied by hand, and the two values are distinguished at generation time by the opt-out pool label alone (positive vs negated), which resolved ADR-015 Deferred Decision 5 by construction.

OKE Device-Plugin Ownership

Same profile mechanism as GKE, one control-plane signal. The OKE recipes declare an ADR-015 gpuStack profile with two qualified values; because both ownership axes (driver and device plugin) move together on OKE, a single signal — the NvidiaGpuPlugin cluster add-on’s control-plane state — qualifies a selection:

• oci-managed (the default) — Oracle’s GPU node image supplies the driver and toolkit, and the NvidiaGpuPlugin add-on is the nvidia.com/gpu advertiser (recorded as advertiser: external); the GPU Operator’s driver, toolkit, and plugin are all off (profile-owned). Its constraint requires the add-on installed and ACTIVE.
• operator-managed (aicr recipe ... --profile gpuStack=operator-managed) — bring-your-own driverless image with the add-on removed: the GPU Operator owns driver, toolkit, and plugin, and the DRA driver root moves to the operator install path in lockstep. Its constraint requires the add-on absent.

The signal is supplied as an oci ce cluster list-addons --cluster-id <cluster-ocid> --all --output json dump via --oke-addons on aicr snapshot and aicr validate (projected into K8s.oke-addons.nvidia-gpu-plugin); it is evaluated at snapshot-based generation and re-evaluated by the validate readiness pre-flight, failing closed on any other add-on lifecycle state or a missing reading. Per-node label disablement (oci.oraclecloud.com/disable-gpu-device-plugin) is out of contract — it leaves the add-on installed. The exactly-one-advertiser rationale is the same as GKE’s, and the profile closure-locks the ownership tuple the same way — switching modes is a recipe-generation decision, never a --set.

Legacy device-plugin tripwire. Older OKE clusters ship the plugin through a second, pre-add-on mechanism — a kube-system/nvidia-gpu-device-plugin DaemonSet reconciled by the legacy Kubernetes addon-manager — which list-addons cannot see. The snapshot observes that DaemonSet directly (K8s.oke-legacy-plugin.nvidia-gpu-device-plugin), and operator-managed additionally requires it none (absent or fully disabled), so a legacy cluster whose add-on reads absent still fails closed instead of double-advertising; remediation is the per-pool oci.oraclecloud.com/disable-gpu-device-plugin=true label or migration to the managed add-on. oci-managed is deliberately not gated on it — the installed add-on reconciles the same DaemonSet. For the end-to-end flow and the qualification matrices, see OKE GPU Setup.

NVSentinel Deployment Posture

AICR ships NVSentinel in the upstream chart’s monitoring-only configuration: it detects GPU and node faults and publishes health events, but takes no automatic action on a node. AICR does not disable remediation — the upstream chart ships it off, and AICR inherits that default rather than overriding it.

recipes/components/nvsentinel/values.yaml enables and disables no NVSentinel component. It carries deployment-shaping values only: fullnameOverride, tolerate-all scheduling so GPU-node DaemonSets land on tainted nodes, networkPolicy.enabled: false (the metrics policy otherwise blocks cert-manager webhook traffic in the same namespace — this is the one upstream default AICR overrides here), platformConnector resources, and janitor-provider.csp.provider: generic, which selects the reboot mechanism used if remediation is later enabled but does not enable it. Every component on/off default below is the chart’s.

On by default — the detection path:

Off by default — the datastore and remediation path:

Also off: healthEventsAnalyzer, lifecycleManager, cspHealthMonitor, kubernetesObjectMonitor, nicHealthMonitor, slurmDrainMonitor, preflight, eventExporter, inclusterFileServer, k8sdatastoreCrds. Verified against chart v1.20.0, the version pinned in recipes/registry.yaml.

The practical effect. A stock AICR bundle surfaces GPU faults; it does not act on them. A node that needs a reboot is reported, not rebooted, and an operator intervenes. That is deliberate: janitor can reboot or terminate nodes, and enabling it without the operator having chosen to is not a safe default.

Enabling Remediation

AICR does not support enabling remediation today, and this page does not carry a recipe for it. #1014 tracks adding a qualified opt-in path.

Turning the components on is not a matter of flipping the six enabled flags. Those flags start the pipeline but leave fault-remediation.maintenance.actions at the subchart defaults, where COMPONENT_RESET maps to kind: RebootNode — so a recoverable GPU fault cordons, drains and reboots the whole node. Upstream’s own remediation configuration maps that same action to kind: GPUReset, scoped to the affected GPU UUID, and resets in place instead. A partial enablement is therefore not a milder version of remediation; it is a more destructive one.

If you need remediation before #1014 lands, start from the chart’s self-contained values-remediation.yaml (shipped inside the nvsentinel chart, pinned at the version in recipes/registry.yaml) rather than composing --set flags, and qualify the result on a cluster you can afford to have rebooted. Three further things apply whatever path you take:

• The reboot path is privileged. AICR pins janitor-provider.csp.provider: generic, so remediation reboots run as a privileged Job executing chroot /host /sbin/reboot on the target node. That avoids requiring cloud IAM credentials, but it is a broad grant. Cloud providers are selectable instead, and need the corresponding credentials.
• The datastore is a real dependency. mongodbStore deploys an in-cluster database. The chart also supports an external datastore and a postgresql provider; see the global.datastore block in the chart’s values.
• Check arm64 before enabling on ARM. The chart’s default MongoDB image has no linux/arm64 manifest. arm64 works via the Percona path (NVIDIA/NVSentinel#1328).

NVSentinel is included by default, not universally required. recipes/overlays/base.yaml includes it unconditionally, so every recipe carries it unless something later removes it, and ADR-018 classifies it as core rather than ops on the rule that ops must be GPU-free and verifiable on CPU-only clusters. Two things can still remove it: an overlay that overrides enabled (the OCP overlay does), and a bundle-time exclusion on a platform whose presence is not profile-locked (EKS and OKE). Only the AKS and GKE-COS gpuStack profiles make its presence mandatory — see NVSentinel is mandatory on the profiled families below.

The component values AICR sets are platform-correctness values, not remediation policy. labeler.assumeDriverInstalled and metadata-collector.runtimeClassName describe facts about the target cluster that the chart cannot infer — who installs the driver, and what the RuntimeClass is named. The next section gives the per-platform values, when an explicit value is needed, and what each failure looks like when it is missing. Which components run is a different matter and stays upstream’s call.

NVSentinel on Provider-Installed-Driver Platforms

The recipes configure NVSentinel for you on every platform that needs it. This section explains what they set and why, so the values are recognizable in a generated bundle and the failure signatures are diagnosable if they ever reappear.

Symptom. NVSentinel’s metadata-collector and both syslog-health-monitor DaemonSets report 0 desired pods and never schedule, while everything else looks fine (#2175).

This is easy to miss. A DaemonSet whose node selector matches no node is not unhealthy — it reports no error and emits no event — and gpu-health-monitor keeps running normally because it selects on the DCGM label instead. The stack presents as fully rolled out.

Cause. Those three DaemonSets select on the node label nvsentinel.dgxc.nvidia.com/driver.installed, which the NVSentinel labeler applies by watching for a GPU driver pod. Where the driver ships in the node image and no driver pod exists, the labeler never applies the label — so labeler.assumeDriverInstalled must be set to skip driver-pod detection and label GPU nodes unconditionally.

The recipes now carry that value wherever it is needed (#2181):

The explicit false on the operator-managed variants is deliberate rather than redundant: it keeps the path profile-owned, so it cannot be flipped into an unsafe hybrid later. Do not assume a preinstalled driver where the GPU Operator installs one — skipping detection there would keep the label applied across an unloaded or unhealthy driver.

NVSentinel is mandatory on the profiled families. Because the AKS, GKE-COS, and OKE gpuStack profiles name nvsentinel, its presence is profile-owned: --set nv-sentinel:enabled=false and a bundlers= list that omits it are both rejected on those platforms. That is intended — NVSentinel is a required component for these deployments. It remains optional on platforms with no gpuStack profile, such as EKS.

AKS, GKE-COS, and OKE get the install-time profile lock; Kind sets the value at overlay level, so a bundle-time or declared-dynamic change is still rejected by the gate below, but a manual post-generation edit to the rendered Helm values is not.

If you do need to set it yourself on an unlisted platform, it is an ordinary override:

aicr bundle -r recipe.yaml \
  --set nv-sentinel:labeler.assumeDriverInstalled=true \
  -o ./bundles

The value renders the labeler’s --assume-driver-installed argument. That is the chart-level automation of the Manual Labeling Procedure documented in NVSentinel design 018. Upstream has settled the design question: it is the recommended, permanent mechanism for host-installed drivers — no automatic detection fallback will be added (NVIDIA/NVSentinel#1583).

Do not label the nodes by hand. kubectl label node <node> nvsentinel.dgxc.nvidia.com/driver.installed=true takes effect immediately — the DaemonSets roll out — and then silently reverts. With no driver pod to observe, the labeler computes an empty desired value and removes the label on its next reconcile. Because upstream design 018 documents manual labeling as the procedure for this case, an operator following it will see it work and later find the DaemonSets back at 0 desired.

aicr bundle rejects a configuration that would reintroduce the gap. A recipe that includes nvsentinel with no observable driver pod and without labeler.assumeDriverInstalled fails bundle generation with a blocking error (CheckNVSentinelDriverLabelDetectable). On the profiled families the value is also profile-owned, so a --set diverging from the selected gpuStack value is rejected before the gate even runs.

One exception, for completeness: the gate is silent if you disable both label consumers (--set nv-sentinel:global.metadataCollector.enabled=false and --set nv-sentinel:global.syslogHealthMonitor.enabled=false). Nothing then reads the label, so there is no gap to reintroduce. Disabling only one still requires the value, and no recipe disables either — both default to enabled, and the gate fails closed when it cannot prove otherwise. Everything above therefore applies to every shipped configuration.

A second, distinct failure on AKS azure-managed: RuntimeClass mismatch. The metadata-collector DaemonSet requests a RuntimeClass by name, and the GPU Operator’s ClusterPolicy controller names that object after operator.runtimeClass. The AKS azure-managed profile retargets it to nvidia-container-runtime, so a metadata-collector left on its chart default nvidia finds no such RuntimeClass and the API server rejects every pod at admission (pod rejected: RuntimeClass "nvidia" not found — #2176).

The two signatures differ: the label gap above shows 0 DESIRED pods (never scheduled, no error, no event); the RuntimeClass mismatch shows N desired / 0 CREATED with a FailedCreate event on the DaemonSet and no pod object to describe.

The AKS gpuStack profile now owns both names — gpu-operator.operator.runtimeClass and nvsentinel.metadata-collector.runtimeClassName — in the same profile value, so they agree by construction under either value and no override is needed:

Every other platform leaves operator.runtimeClass at the shared chart default nvidia, so neither side needs a value. CheckNVSentinelRuntimeClassCoherence still compares the two resolved names as defense in depth, treating either side unset as nvidia.

An AKS bundle therefore needs no NVSentinel overrides at all — only the keyed toleration AKS requires independently of NVSentinel (bundling an AKS recipe without one is itself a blocking error, CheckWildcardAcceleratedToleration):

aicr bundle -r recipe.yaml \
  --accelerated-node-toleration nvidia.com/gpu:NoSchedule \
  -o ./bundles

See AKS GPU Setup for the per-profile guidance.

Enabling NVCRE

nvcre is not on any shipped overlay, so enabling it means writing the componentRef yourself. Two requirements are easy to miss, and each one produces a different failure.

dependencyRefs only orders components that are already in componentRefs — it does not add one. Naming a component that is not present fails resolution outright with component "nvcre" references unknown dependency "kubeflow-trainer". NVCRE drives its benchmarks through Kubeflow Trainer (TrainJob / TrainingRuntime) and the chart does not install Trainer, so Trainer has to come from somewhere else — the platform-kubeflow mixin is the cleanest source.

Component values are also never auto-discovered from the component name: a ref with no valuesFile and no inline overrides resolves to an empty map. Omit it and the chart defaults apply, which means a ServiceMonitor you did not ask for (requiring prometheus-operator CRDs) and a release-prefixed Deployment name such as aicr-stack-nvcre-manager, which the shipped health check cannot match.

Add this to an overlay that already inherits a stock AICR base:

spec:
  mixins:
    - platform-kubeflow          # brings in kubeflow-trainer
  componentRefs:
    - name: nvcre
      type: Helm
      valuesFile: components/nvcre/values.yaml
      dependencyRefs:
        - kubeflow-trainer

Do not also declare kubeflow-trainer locally. The mixin’s ref sets type, valuesFile, and dependencyRefs, all of which are prohibited collision fields, and overlay chains merge before mixins — so a local ref collides rather than overrides.

Prerequisites: NVIDIA GPU Operator and cert-manager, both inherited from base.yaml by every stock recipe, plus Kubeflow Trainer, which is not — see the fragment above.

NVCRE v0.2.0 expects Kubeflow Trainer v2.2.1 — it pins kubeflowTrainerVersion = "v2.2.1" and its setup status reports the 2.2.0 that the registry defaults to as unsupported. No functional break is known between the two versions: the CRD delta is documentation text plus one embedded PodSpec field NVCRE does not set. Aligning the global Trainer default is tracked separately.

NIM Workload Credentials

AICR installs the k8s-nim-operator only. It does not create a NIMService and does not create credentials — deploying a workload is an operator step, and there are two ways to supply the model.

Whichever path you take, spec.authSecret is required by the NIMService schema and must name an existing secret in the workload’s namespace. spec.image.pullSecrets is optional; the image block requires only repository and tag.

NGC path

Model artifacts come from NGC, so the secret must carry a valid NGC_API_KEY:

kubectl create secret generic ngc-api-secret \
  --from-literal=NGC_API_KEY="$NGC_API_KEY" -n nim-workload

Add a docker-registry secret and reference it from image.pullSecrets when the image lives in a private or authenticated registry path. See demos/workloads/inference/nimservice-llama-3-2-1b.yaml for a complete example.

Credential-free path (Hugging Face)

Setting NIM_MODEL_NAME to an hf:// URI puts the operator on its Hugging Face path, where it marks NGC_API_KEY optional and injects HF_TOKEN from the same authSecret. With an ungated Hugging Face model and a NIM image that pulls anonymously, no NGC credential is needed anywhere:

kubectl create secret generic hf-secret --from-literal=HF_TOKEN="" -n nim-workload

spec:
  authSecret: hf-secret                                  # holds only HF_TOKEN
  image:
    repository: nvcr.io/nim/meta/llama-3.1-8b-instruct   # pulls anonymously; no pullSecrets
    tag: "2.0.10"                                        # pin a version; avoid the mutable latest
  env:
    - name: NIM_MODEL_NAME
      value: hf://Qwen/Qwen3-0.6B                        # ungated model
    - name: NIM_SERVED_MODEL_NAME
      value: Qwen/Qwen3-0.6B                             # the OpenAI-API `model` id

The HF_TOKEN key must exist in the secret — that reference is not optional — but an empty value is sufficient for an ungated model. A gated Hugging Face repository needs a real token here.

Model-specific NIM repositories (for example nim/meta/llama-3.1-8b-instruct) serve anonymous registry tokens; the generic Multi-LLM image nim/nvidia/llm-nim does not and requires a pull secret.

Note that pairing a model-specific image with an unrelated hf:// model is off-label: the container runs its own profile against the downloaded weights. It works, but nim/nvidia/llm-nim is the image intended for arbitrary Hugging Face models — and because that repository is gated, choosing it trades the credential-free property for a supported pairing. Pin an image tag rather than latest so the pairing you validated is the one you ship.

See demos/workloads/inference/nimservice-hf-nocred.yaml for a complete example.

Inference Gateway Network Exposure

Inference recipes include the agentgateway component, which deploys an inference-gateway Gateway. The agentgateway controller materializes that Gateway into a Service of type LoadBalancer, so on every cloud the platform provisions a load balancer for the (plaintext HTTP, unauthenticated) inference endpoint. Left unrestricted that load balancer is internet-facing, so aicr bundle scopes it to private networks by default — the opt-in path for public exposure and the validation behavior are described below.

aicr bundle is private by default: when a bundle includes agentgateway and agentgateway.allowedSourceRanges is empty or unset, the bundler injects the private RFC1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) into the generated Service’s spec.loadBalancerSourceRanges. The deployed gateway is therefore reachable from inside the cluster/VPC (and from privately-routed peers) but denied to the public internet — it is never emitted open to 0.0.0.0/0 without an explicit opt-in. (Kubernetes treats an empty loadBalancerSourceRanges as allow-all, so a safe default has to be a real list, not an empty one.) A bundle note records when the default was applied.

To restrict it to specific trusted networks instead — for example to allow a corporate VPN, which egresses from a public IP and is therefore not covered by the RFC1918 default — set agentgateway.allowedSourceRanges to a list of CIDR (Classless Inter-Domain Routing) blocks. The values replace the default and are rendered into the generated Service’s spec.loadBalancerSourceRanges, which the AWS, GCP, Azure, and OCI cloud load balancers all honor — so one setting locks the gateway down on every platform.

Do not use plain --set for this key. --set agentgateway:allowedSourceRanges=<cidr> writes loadBalancerSourceRanges as a bare string instead of a list; the bundler rejects that with ErrCodeInvalidRequest (a bare scalar would render a type-invalid Service). Use the list-aware --set-json / --set-file flags from the CLI:

aicr bundle -r recipe.yaml \
  --set-json agentgateway:allowedSourceRanges='["216.228.127.128/30"]'

or scope the gateway through a recipe overlay or componentRef override:

componentRefs:
  - name: agentgateway
    type: Helm
    overrides:
      allowedSourceRanges:
        - 216.228.127.128/30   # e.g. corporate egress

The default is the generic RFC1918 private set rather than a fixed customer CIDR: a baked-in specific range would firewall every downstream deployment to one network and lock other operators out of their own gateway. RFC1918 is universal — it trusts only privately-routed traffic — so it is a safe default that still denies the public internet. Override it whenever you need to admit a specific public client.

If public exposure is genuinely intended, opt in explicitly with an any-source CIDR — bundle generation then succeeds but logs a loud warning that the gateway is open to the entire internet:

aicr bundle -r recipe.yaml \
  --set-json agentgateway:allowedSourceRanges='["0.0.0.0/0"]'

This setting filters by source IP only; it does not add TLS or authentication to the gateway listener.

Exposure guardrails

AICR enforces and surfaces inference-gateway exposure in two places:

• Bundle-time private-by-default. When a bundle includes agentgateway and allowedSourceRanges is empty/unset, aicr bundle injects the RFC1918 private ranges so the deployed gateway denies the public internet, and records a bundle note. An invalid value (a bare-string --set, a non-list, an unparseable CIDR, or a non-canonical CIDR such as 1.2.3.4/24 that Kubernetes’ strict validation would reject at apply time) is rejected with ErrCodeInvalidRequest. A scoped list passes silently; an explicit any-source CIDR (0.0.0.0/0 or ::/0) passes with a loud warning as a deliberate opt-in. See #1373.
• Conformance check. The inference-gateway conformance check (run during aicr validate --phase conformance on a live cluster) inspects the gateway’s LoadBalancer Service and records its exposure as evidence — the source ranges if scoped, or an explicit “open to 0.0.0.0/0” finding if not. Set AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY=true on the validator environment to escalate an open gateway to a check failure.

k8s-aibom Runtime Inventory

AICR qualifies k8s-aibom v1.3.0 as an optional Helm component. It is not in the base or a mixin. Exactly one stock recipe installs it, h100-gke-cos-inference, under ADR-019’s stock-adoption amendment. Decline it at generation time with aicr recipe --runtime-inventory disabled, described below.

h100-gke-cos-inference-dynamo inherits from that recipe and deliberately declines the component, so the Dynamo platform recipe deploys exactly what it did before. Adoption beyond the one recipe is a later decision.

To enable it anywhere else, add this reference to a custom or external overlay and keep that overlay’s criteria as narrow as the intended rollout:

spec:
  componentRefs:
    - name: k8s-aibom
      type: Helm
      valuesFile: components/k8s-aibom/values.yaml

A broad criteria overlay affects every matching recipe. In particular, intent: any is universal across intents; do not use it unless that injection is deliberate. The in-tree recipes/overlays/monitoring-hpa.yaml overlay shows that broad reach: its criteria: intent: any attaches to every matching intent. See Recipe Development for external data and criteria composition.

The qualified artifacts are source tag v1.3.0 at commit 30af41abbe0bed3c41a42289ccf294be8c4779bb, OCI chart oci://ghcr.io/googlecloudplatform/charts/k8s-aibom:1.3.0, and the controller image pinned by digest in the component values. v1.3.0 is the API-graduation release: both v1alpha1 and v1beta1 are served and CRD storage is on v1beta1, while the chart still renders the AIBOMControllerConfig resource itself at v1alpha1. Upstream states Kubernetes support as a policy rather than a fixed range: stable APIs only, no known version ceiling, tested floor 1.27, backed by a weekly CI matrix. The authoritative statement is upstream’s compatibility policy, which is linked rather than restated here so it cannot drift out of date on our side. That link deliberately tracks main: the point is the current policy, not a snapshot of it, which is the opposite of how this page cites qualified artifacts.

AICR also observed the dedicated integration test passing on its Kind 1.36.1 node image; that is qualification evidence, not an extension of upstream’s support statement.

Health and readiness

The deployment-phase check requires the controller Deployment to have at least one desired replica and all desired replicas available. It also requires the cluster-scoped AIBOMControllerConfig/default to report a current Ready=True condition: both top-level status and the condition must have observed the object’s current generation. Missing or stale resources fail closed. Zero AIBOM objects is healthy before any namespace opts in.

The check also requires both shipped CRDs, aiboms.aibom.k8saibom.dev and aibomcontrollerconfigs.aibom.k8saibom.dev, to report the storage version of the chart version pinned in the registry. It matters because Helm and Helmfile skip a chart’s crds/ directory on upgrade, so a cluster can run a new controller against the previous schema while the older version stays served and the controller keeps working.

Flux is the exception for this component: k8s-aibom is marked ownsCRDs in the registry, so its generated HelmRelease sets spec.upgrade.crds: CreateReplace and Flux applies the CRDs itself. Argo CD applies them as ordinary manifests each sync. The assertion is still worth making on every deployer, because it proves the deployed CRDs match the pinned chart rather than merely that some deployer was expected to update them.

Both CRDs are asserted separately, so a failure names which one is stranded and a partially applied CRD set cannot pass. If this check fails after a chart bump, the pre-upgrade CRD step in Upgrade, uninstall, and troubleshooting is the thing to run.

The assertion establishes that the storage-version contract matches the pinned chart. It is not provenance: it reads one field, so it cannot show the CRDs originated from that chart, and it cannot tell apart chart versions that share a storage version. Charts 1.0.0, 1.1.0, and 1.2.0 all declare v1alpha1 as storage. So the check catches a stranded upgrade that crosses a storage-version boundary, such as the 1.2.0 to 1.3.0 move this pin made, and does not catch one within a boundary, such as 1.0.0 to 1.2.0.

Declining the component. h100-gke-cos-inference installs k8s-aibom by default. Decline it at generation time:

aicr recipe --service gke --accelerator h100 --os cos --intent inference \
  --runtime-inventory disabled -o recipe.yaml

The same flag works for a recipe that adds the component through a custom overlay — the shape shown above. Point --data at the directory holding that overlay:

aicr recipe --service gke --accelerator h100 --os cos --intent inference \
  --data ./my-recipes --runtime-inventory disabled -o recipe.yaml

Passing the flag against a recipe that does not declare the component is an error, not a silent no-op. Training recipes do not, so:

$ aicr recipe --service gke --accelerator h100 --os cos --intent training \
    --runtime-inventory disabled
[INVALID_REQUEST] runtime inventory mode "disabled" requires the recipe to
declare component "k8s-aibom"; this recipe does not resolve it

The selection is recorded in the emitted recipe as configuration.runtimeInventory.mode, and the component’s ref carries install: false, so the component and its health check are both absent from the bundle and from deployment validation. A bundle-time --set k8s-aibom:enabled=false is not equivalent and is not a supported way to decline the component: it changes neither the recipe nor its health checks, which is why ADR-019 rejects it as a selection contract.

A wrong --service or a typo therefore surfaces instead of producing a recipe that claims a decision it never applied.

The same selection is available in an AICRConfig document as spec.recipe.configuration.runtimeInventory.mode.

Overriding the chart version requires overriding this assertion. Assert content is static YAML with no templating, so the expected storage version is a literal tied to the registry’s pinned chart, currently v1beta1 for chart 1.3.0. Charts 1.2.0 and earlier declare only v1alpha1. A recipe that sets version on the k8s-aibom componentRef to a chart with a different storage version will therefore fail this step even though the cluster is correct. Such a recipe must supply matching inline healthCheckAsserts on the componentRef, or set healthCheckSkip: true to drop the registry check entirely.

readiness.strictConfig is enabled, so invalid new configuration cannot silently replace the controller’s last-known-good configuration while the pod continues to report ready.

Security, privacy, and retention

Namespace discovery requires the label aibom.k8saibom.dev/enabled=true. AICR does not apply it. No external sink, endpoint, credential, or Secret access is configured by default. BOMs up to 262144 bytes are stored inline in AIBOM.status; larger output is summarized and marked truncated when no sink is configured. Inline status consumes etcd storage, so workload count and document size are part of the cluster control plane footprint. The controller runs non-root with a read-only root filesystem, RuntimeDefault seccomp, no privilege escalation, and no Linux capabilities.

The namespace label limits which workloads produce AIBOMs, not informer read scope. The controller still reads workload and pod specifications cluster-wide—including image references, arguments, and inline environment values—into memory. With the default empty sink list that data does not leave the cluster, but cluster-wide visibility remains part of the privacy boundary. The controller is not read-only: its bounded RBAC permits writes to its own AIBOM API resources and required status subresources, configuration status, and Kubernetes Events. It has no Secret access while sinks are disabled.

An AIBOM is owned by its top-level workload and is garbage-collected when that owner is deleted. Helm does not delete CRDs from a chart’s crds/ directory on uninstall; consequently AIBOM resources for owners that still exist may remain after controller removal. External sinks, if an operator configures one, have their own retention policy outside AICR.

Upgrade, uninstall, and troubleshooting

Upgrade the component by qualifying a new chart and image together, then regenerate the custom recipe and bundle. Do not change only the controller image: chart, CRDs, status API, and image are one qualified set. Quiesce configuration changes during rollback and confirm that AIBOMControllerConfig/default returns to a current Ready=True state.

Apply CRDs before the bundle upgrade — helm and helmfile only. The chart ships its CRDs under crds/. Helm installs that directory on first install and never touches it again on upgrade, so a chart bump whose CRDs changed leaves the previous schema in place and the API server silently prunes the new controller’s writes to added fields.

The flux, argocd, and argocd-helm bundles handle this themselves for this component and need no manual step; see the deployer table below. For helm and helmfile, apply the CRDs from the exact qualified chart first, then upgrade:

CHART="oci://ghcr.io/googlecloudplatform/charts/k8s-aibom"
VERSION="1.3.0"   # replace with the version you are upgrading to
helm show crds "${CHART}" --version "${VERSION}" \
  | sed -n '/^---$/,$p' \
  | kubectl apply --server-side --force-conflicts -f -

Three details in that command are load-bearing. The obvious shorter form — piping helm show crds straight into kubectl apply --server-side — fails on the first two:

• sed -n '/^---$/,$p' drops helm’s progress output. For an OCI chart, helm show crds writes Pulled: and Digest: lines to stdout, and those two lines parse as a valid YAML mapping, so kubectl rejects the stream with error validating data: [apiVersion not set, kind not set].
• --force-conflicts is required because Helm created these CRDs on install and owns their fields. Without it, server-side apply refuses with a field-manager conflict.
• --server-side is required because the CRDs exceed the annotation size limit that client-side apply depends on.

Verified against a live GKE cluster across a 1.2.0 to 1.3.0 upgrade.

Which deployers need that step differs, so check yours:

ownsCRDs is opt-in, and narrow on purpose. Of the 15 registry components that ship CRDs under crds/, 11 share at least one CRD with another component: nfd, gpu-operator, and network-operator all ship the NodeFeature CRDs, and nfd, gpu-operator, and kai-scheduler all appear together in base.yaml. If every release replaced CRDs on upgrade, two or three HelmRelease objects would rewrite the same CRD on every reconcile, each with the schema its own chart pins. The Skip default is what prevents that today, so it stays the default.

A component qualifies only if it solely owns every CRD it ships and ships none using spec.conversion.strategy: Webhook, since replace discards a caBundle injected at runtime. kubeflow-trainer is excluded for that second reason. Currently gatekeeper, k8s-aibom, and nvsentinel qualify.

helm and helmfile always need the step, because skipping crds/ on upgrade is Helm’s own behavior rather than something the generated bundle can change.

Uninstall in this order. Removing the component from the overlay and applying a regenerated bundle does not remove the previously installed release: the helm and helmfile deployers install releases by name, and a release the new bundle no longer mentions is simply left alone. Skipping the explicit uninstall leaves the controller running while the next step deletes the CRs and CRDs underneath it, so it reconciles against resources that are disappearing.

1. Remove the component reference from the custom overlay and regenerate the recipe and bundle.

2. Uninstall the release, scoped to this component only:

   # helm and helmfile bundles alike: helmfile installs through Helm, so the
   # release is an ordinary Helm release and this removes exactly one.
   helm uninstall k8s-aibom -n k8s-aibom-system

   For Argo CD, delete the owning Application; for Flux, the HelmRelease. Confirm the controller Deployment is gone before continuing.

   Do not use helmfile destroy for this. It tears down every release in the bundle in reverse dependency order, not just this component. It is also ineffective here: step 1 regenerated the bundle without k8s-aibom, so the release is no longer declared in it and destroy would not remove the one release you actually want gone while removing all the ones you do not. If you prefer a Helmfile-native command, run it against a bundle that still declares the component and scope it explicitly with helmfile destroy --selector name=k8s-aibom.

3. Only then delete retained AIBOMs and, last, the CRDs.

Deleting the CRDs cascades to every AIBOM stored cluster-wide, including any belonging to a namespace or release you did not intend to touch. Enumerate before deleting rather than passing --all:

# Review what exists and who owns it; delete only what this release should own.
kubectl get aiboms.aibom.k8saibom.dev --all-namespaces \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name
kubectl -n <namespace> delete aiboms.aibom.k8saibom.dev <name>
# CRDs last, and only once no other release uses them — deletion removes every
# stored custom resource of these kinds cluster-wide.
kubectl delete crd \
  aiboms.aibom.k8saibom.dev \
  aibomcontrollerconfigs.aibom.k8saibom.dev

If health validation fails, inspect the Deployment and configuration before looking for AIBOMs:

kubectl rollout status deployment/k8s-aibom -n k8s-aibom-system
kubectl get aibomcontrollerconfig default -o yaml
kubectl logs deployment/k8s-aibom -n k8s-aibom-system

A current Ready=False or stale observedGeneration means the active configuration was not accepted. If the controller is healthy but produces no inventory, verify the namespace label and the workload kind. A truncated BOM with no sink is expected once its canonical document exceeds the inline threshold; configure retention and credentials explicitly before enabling an external sink.

Adding Components

New components are added declaratively in recipes/registry.yaml — no Go code required. See the Contributing Guide and Components docs for details.

Upgrade Notes

Migration steps when upgrading from a prior AICR-generated bundle to a newer one that changes how a component delivers its Kubernetes resources.

A generated recipe is a point-in-time artifact of the AICR binary that produced it: the embedded registry, overlays, manifest paths, and chart pins are part of that binary’s surface. When upgrading AICR, regenerate the recipe from scratch with the new binary (aicr recipe ...) before re-bundling. aicr bundle --recipe <old-file> against a newer binary may fail if the saved recipe references manifest paths the new release has moved or removed (see Bundle Generation Fails for the specific error).

gpu-operator: dcgm-exporter ConfigMap moved into the main release

Earlier bundles shipped the dcgm-exporter ConfigMap as a post-manifest in a separate Helm release named gpu-operator-post. The in-cluster ConfigMap therefore carries ownership annotations pointing at that release:

meta.helm.sh/release-name: gpu-operator-post
meta.helm.sh/release-namespace: gpu-operator

Newer bundles render the ConfigMap directly from the main gpu-operator chart’s dcgmExporter.config.data values. On upgrade, Helm 3 refuses to claim the existing ConfigMap because its annotations point at a different release:

Error: ConfigMap "dcgm-exporter" in namespace "gpu-operator" exists and cannot be
imported into the current release: invalid ownership metadata; annotation
validation error: key "meta.helm.sh/release-name" must equal "gpu-operator":
current value is "gpu-operator-post"

Fresh installs are not affected. To migrate an existing cluster, remove the stale gpu-operator-post release before applying the new bundle.

Raw Helm (per-component bundle / deploy.sh):

helm uninstall gpu-operator-post --namespace gpu-operator

helm uninstall removes the ConfigMap it owns; the next gpu-operator upgrade re-creates it from values.

Helmfile — the new bundle no longer references gpu-operator-post, so helmfile apply will not prune it on its own. Run the helm uninstall above first, then helmfile apply.

Argo CD — delete the stale Application (it will not self-prune unless an ApplicationSet was managing it), then sync the updated gpu-operator application:

argocd app delete gpu-operator-post --cascade

Flux — delete the stale HelmRelease so Flux uninstalls the release and removes the ConfigMap, then reconcile the updated gpu-operator HelmRelease. The example below assumes the Flux control plane runs in flux-system; substitute the namespace where your Flux installation lives:

kubectl delete helmrelease gpu-operator-post --namespace flux-system

After migration, confirm the ConfigMap is owned by the gpu-operator release:

kubectl get configmap dcgm-exporter -n gpu-operator \
  -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}'
# Expected: gpu-operator

grove: v0.1.0-alpha.8 (or earlier) to v0.1.0-alpha.12

alpha.8’s clustertopologies.grove.io CRD (kind ClusterTopology, shortname ct) is dropped by alpha.12 in favor of clustertopologybindings.grove.io, which reuses shortname ct. helm upgrade never applies changed crds/ (see the CRD-upgrade caveat above), so a plain chart bump alone never installs the new CRD and the operator crash-loops on no matches for kind "ClusterTopologyBinding" regardless of deployer. If the new CRD is applied before the old one is deleted, the shortname collision blocks it from ever reaching Established.

Fresh installs are unaffected. To migrate an existing cluster:

1. Check for any live ClusterTopology resources — if any exist, stop for an explicit data migration before touching CRDs:

   kubectl get clustertopologies.grove.io -A
2. If zero exist, delete the obsolete CRD:

   kubectl delete crd clustertopologies.grove.io
3. Apply the new CRDs directly from the chart — this works from any directory, no local checkout needed:

   helm show crds oci://ghcr.io/ai-dynamo/grove/grove-charts --version v0.1.0-alpha.12 \
     | sed -n '/^---$/,$p' \
     | kubectl apply --server-side --force-conflicts -f -
4. Resume whichever deployer you use (helm/helmfile: re-run install.sh or helmfile apply; flux/argocd/argocd-helm: reconcile/sync as usual). AICR’s generated install.sh already passes --force-conflicts on Helm 4.

Verified live end-to-end on a real EKS cluster, 2026-09-02: dynamo-operator, grove-operator, and kai-scheduler gang-scheduling all confirmed healthy together post-migration.

dynamo-platform: opting back into bundled NATS on a standing cluster

Dynamo 1.4+ no longer installs bundled NATS by default (the request plane defaults to TCP and the KV event plane to ZMQ). Re-enabling it needs to be done by regenerating the bundle, not by hand-editing a generated bundle’s values.yaml or cluster-values.yaml — both are covered by the bundle’s checksum manifest, and aicr verify fails closed on any modified file:

aicr verify ./bundle
# ✓ Checksums verified (87 files)
# Bundle verification: PASSED
# Hand-edit 017-dynamo-platform/values.yaml, then:
aicr verify ./bundle
# ✗ Checksum verification failed
# - [INVALID_REQUEST] checksum mismatch for "017-dynamo-platform/values.yaml"
# Bundle verification: FAILED (exit 4)

A modified bundle also invalidates the binding of any existing attestation to the original checksum digest. Regenerate instead, using the same aicr bundle command and flags you used originally, with the NATS overrides added:

aicr bundle --recipe recipe.yaml \
  <your original --accelerated-node-selector / --system-node-selector / etc. flags> \
  --set dynamoplatform:global.nats.install=true \
  --output ./bundle

If your system node group is tainted or the cluster has no default StorageClass, add the scheduling/storage overrides a re-enabled NATS also needs — AICR no longer targets NATS with --system-node-selector, --system-node-toleration, or --storage-class at bundle-generation time (those registry paths were removed along with bundled-by-default NATS), so pass them explicitly or the NATS pod/PVC can stay Pending. Use --set-json for nodeSelector and tolerations — the scalar --set dot-path parser rejects qualified Kubernetes label keys that contain a / (e.g. eks.amazonaws.com/nodegroup), and there is no scalar form for a tolerations list at all:

  --set-json 'dynamoplatform:nats.podTemplate.merge.spec.nodeSelector={"<key>":"<value>"}' \
  --set-json 'dynamoplatform:nats.podTemplate.merge.spec.tolerations=[{"key":"<key>","operator":"Equal","value":"<value>","effect":"NoSchedule"}]' \
  --set dynamoplatform:nats.config.jetstream.fileStore.pvc.storageClassName=<name>

Verified against this exact sequence (generate → aicr verify passes → hand-edit → aicr verify fails with exit 4 → regenerate with the overrides above → rendered nats.podTemplate.merge.spec carries both the qualified nodeSelector and the toleration → aicr verify passes again), 2026-09-02. A scalar --set …nodeSelector.eks.amazonaws.com/nodegroup=<value> was separately confirmed to fail with exit 2 (INVALID_REQUEST: invalid path segment "com/nodegroup"), and --system-node-toleration alone was confirmed to not reach NATS (nats.podTemplate.merge.spec.tolerations rendered empty; only the operator’s controllerManager.tolerations was set) — both are why the overrides above are required rather than optional.

On an in-place helm upgrade from the regenerated bundle, disabling NATS removes its StatefulSet but does not delete its PVC; inspect kubectl get pvc -n dynamo-system -l app.kubernetes.io/name=nats before deciding whether to retain or delete it.

Separately, and more importantly than the NATS opt-out above: bumping the operator without also bumping any standing DynamoGraphDeployment’s runtime image is its own hazard, independent of NATS. An older runtime under a newer operator is the same version-skew class that caused the frontend discovery panic fixed in #1193 — setting DYN_EVENT_PLANE=zmq on the old workload is defense in depth, not a substitute for bumping its image to match the operator.

gpu-operator and nvidia-dra-driver-gpu: ComputeDomain CRD ownership on Argo CD

gpu-operator and nvidia-dra-driver-gpu (and nvidia-dra-driver-gpu-ocp) both ship the computedomains.resource.nvidia.com CRD. As of gpu-operator v26.7.0 the two chart copies disagree on schema (spec.numNodes required vs. optional with a default), so on --deployer argocd and --deployer argocd-helm — where every generated Application syncs with automated.selfHeal: true — Argo CD perpetually reconciles the CRD toward whichever Application last synced.

When a bundle pairs a standalone DRA driver with gpu-operator v26.7.0 or newer, AICR scopes an ignoreDifferences entry to the divergent fields on the gpu-operator Application, so the DRA driver’s copy stays the effective owner and the reconcile loop stops. This is generated automatically — no flag or override is needed. Bundles with only one of the two components, or with a gpu-operator older than v26.7.0 (whose chart ships no computedomains CRD to contend with), are unaffected and carry no such entry.

One side effect worth knowing about. The entry is paired with the RespectIgnoreDifferences=true sync option, without which Argo CD would exclude the fields from its diff but still re-apply them on every sync. That option is Application-wide, not per-entry: Argo builds the sync-time normalizer from this Application’s ignoreDifferences plus any resource.customizations.ignoreDifferences.* configured cluster-wide in argocd-cm. So on an Argo instance carrying global ignore rules (webhook caBundle, HPA-managed replicas, aggregated ClusterRole rules), those fields also stop being enforced at sync time for the gpu-operator Application specifically — drift in them is preserved rather than corrected by selfHeal. Argo CD offers no way to scope the option to a single entry.

This is a stopgap, not a durable fix. Helm and Flux bundles are not affected (both install CRDs once and never re-apply them), and the OLM-based OCP path (gpu-operator-ocp, gpu-operator-ocp-olm) is not covered — those components install no chart crds/ of their own, so their CRDs come from the OLM Subscription/CSV and an Application-level ignoreDifferences has nothing to arbitrate. That conflict is tracked separately. See NVIDIA/aicr#2546.

agentgateway: upgrading across breaking releases

AICR pins the agentgateway and agentgateway-crds charts in the component registry, and a pin bump can cross upstream releases that document breaking changes — to JWT claim enforcement, LLM token accounting, policy merging, cross-namespace route delegation, managed API-key metadata, Istio identity, Gateway API and TCPRoute handling, MCP guardrails, standalone auth, and image base. Whether any of that reaches you depends entirely on which agentgateway resources exist in your cluster, and the answer differs sharply between what AICR generates and what you author yourself.

AICR-generated bundles are unaffected. A bundle creates exactly two agentgateway resources: an AgentgatewayParameters that carries deployment and service shape only, and the inference-gateway Gateway. It ships no AgentgatewayPolicy, AgentgatewayBackend, AgentgatewayModel, or HTTPRoute — so the breaking changes land on surface AICR never populates.

Resources you author yourself are exposed, and AICR can neither detect nor migrate them. Before applying a bundle whose agentgateway pin moved, check whether you have any:

(
  if ! kinds=$(kubectl api-resources --api-group=agentgateway.dev -o name); then
    echo "discovery incomplete — retry; do not read this as clear" >&2; exit 1
  fi
  if [ -z "$kinds" ]; then
    echo "no agentgateway.dev kinds registered — the chart is not installed" >&2; exit 0
  fi
  for kind in $kinds; do
    kubectl get "$kind" -A || { echo "listing $kind failed — retry" >&2; exit 1; }
  done
)

The kinds are discovered rather than named because the API group grows across chart versions — AgentgatewayModel only exists from v1.4.0 — so naming them would fail with the server doesn't have a resource type on exactly the older pins whose operators most need to run this. The status checks matter for the same reason: an unhealthy aggregated APIService (a down metrics-server or custom-metrics adapter, and prometheus-adapter is in AICR’s own component set) makes kubectl api-resources exit non-zero, the substitution yields an empty list, and the loop would silently report clean. Empty output plus an error means retry, not clear.

Routes you authored live in the Gateway API group rather than agentgateway.dev, so the sweep above does not see them — and they are exactly what the cross-namespace route delegation change affects:

(
  kubectl get httproutes,grpcroutes -A \
    -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,PKIND:.spec.parentRefs[*].kind,PNS:.spec.parentRefs[*].namespace,PARENTS:.spec.parentRefs[*].name' \
    || { echo "route listing failed — retry" >&2; exit 1; }
)

Read the rows by their parent:

• PKIND: Gateway naming inference-gateway — a route you attached to the AICR gateway.
• PKIND: HTTPRoute with a PNS that differs from NS — cross-namespace route-to-route delegation. From v1.5.0 this requires a ReferenceGrant in the child’s namespace authorizing the parent’s namespace, where previously none was needed. Confirm kubectl get referencegrants -A covers each one before upgrading, or the delegation stops being accepted.
• PKIND: HTTPRoute with an empty PNS — same-namespace delegation, which the change does not affect. parentRefs[].namespace is optional and defaults to the route’s own namespace, so empty is the same-namespace signal.

One caveat on the command: a route with several parentRefs where only some set namespace will have its PNS column misalign, because JSONPath omits the missing entries rather than padding them. Describe those routes individually with kubectl get <httproute-or-grpcroute> <name> -n <ns> -o yaml rather than trusting the columns.

AICR’s own AgentgatewayParameters named system-proxy in agentgateway-system is expected. If nothing else appears here, nothing you authored in the agentgateway.dev group is affected — but that is not the whole check. A cluster with no custom agentgateway resources and no routes can still hold an agentgateway Gateway outside agentgateway-system, which stops reconciling once the namespace list is scoped. Finish with the Gateway inventory in which namespaces the controller may provision Gateways in before concluding the upgrade needs nothing from you. Anything else returned means read the upstream release notes for every version between the old and new pin and validate off-production first — a multi-version jump has to absorb every breaking change in between, not just the newest one. Use the component version matrix to find which versions those are.

One limit worth stating plainly: AICR CI exercises fresh installs of a pinned chart, not in-place upgrades from an older pin. A green release validates that the new version deploys and passes its health checks. It is not an in-place upgrade certification.

agentgateway: which namespaces the controller may provision Gateways in

From chart v1.5.0 the controller’s write permissions are scoped by rbac.gatewayNamespaces, and AICR sets it to agentgateway-system — the one namespace it provisions the inference-gateway Gateway in. The chart’s own default is an empty list, which binds the write role (Deployments, DaemonSets, Secrets, ServiceAccounts, ConfigMaps, Services, HPAs, PDBs) with a ClusterRoleBinding, letting a network-facing controller write those objects in every namespace on the cluster.

Scoping narrows that write reach to the namespaces you name, and nothing else. Two things it does not contain. The controller’s read role is a separate ClusterRole bound cluster-wide regardless of this value, and it carries get/list/watch on Secrets — so a scoped controller can still read every Secret in the cluster. And the write role still grants daemonsets, so a DaemonSet created in a permitted namespace still schedules pods onto every node. Treat the controller as privileged rather than contained; scoping is worth doing, but it is not the control that keeps it away from your Secrets.

The consequence for you is that only Gateways in the listed namespaces are provisioned. A Gateway elsewhere is accepted by the API server but never gets an address — the controller cannot create its Deployment or Service there. The Gateway reports it: v1.5.0 sets Programmed=False with reason DeploymentFailed when those writes are denied, so kubectl get gateway <name> -n <ns> -o yaml shows the cause in status.conditions, with matching forbidden errors in the controller log. This is about where the Gateway lives; routes are unaffected, and HTTPRoutes in any namespace still attach to the AICR gateway.

Upgrading an existing cluster is where this bites. A Gateway that reconciles today under the chart’s unscoped default stops once the list is scoped without its namespace, so inventory what you have before upgrading:

(
  kubectl get gateways.gateway.networking.k8s.io -A \
    -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.gatewayClassName' \
    || { echo "Gateway listing failed — retry" >&2; exit 1; }
)

Every namespace holding a Gateway with CLASS: agentgateway belongs in the list. To add them:

aicr bundle -r recipe.yaml \
  --set-json agentgateway:rbac.gatewayNamespaces='["agentgateway-system","my-gateways"]'

Two constraints on that edit. The namespaces must already exist — the chart creates a RoleBinding in each, and naming one that has not been created fails the install. And the key is a list, so it needs --set-json: a plain --set agentgateway:rbac.gatewayNamespaces=my-gateways writes a bare string and the chart’s range over it fails at render, the same list-versus-string trap described for allowedSourceRanges above.