> This page is for AI Cluster Runtime, version v1.3.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.3.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt
> - v1.3.0: https://docs.nvidia.com/aicr/v1.3.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt.

# Generating Bundles

`aicr bundle` materializes a recipe into deployment-ready artifacts — one
folder per component with its Helm values, plus a root README, `checksums.txt`,
and `bundle-info.yaml`. This guide covers the common bundling tasks: choosing a
deployer, overriding values, enabling or disabling components, pinning node scheduling, producing offline
bundles, and gating on component readiness.

This is a task-oriented how-to. For the complete flag list and exit codes, see
the `aicr bundle` section of the [CLI Reference](/aicr/v1.3.0/user-guide/cli-reference). For the
recipe → bundle → deploy → validate flow end to end, see the
[End-to-End Tutorial](/aicr/v1.3.0/user-guide/end-to-end-tutorial).

## Choose a deployer

The `--deployer` (`-d`) flag selects the output format. The bundle content is
the same validated configuration; only the serialization differs, so you can
re-render the same recipe for whatever pipeline you run:

| Deployer | Output |
|----------|--------|
| `helm` (default) | Per-component Helm values + a `deploy.sh` that installs in dependency order. |
| `helmfile` | A `helmfile.yaml` release graph. |
| `argocd` | Argo CD `Application` manifests (app-of-apps), published from a Git repo (`--repo`). |
| `argocd-helm` | A Helm chart app-of-apps; `repoURL` defaults to the push-target registry — plain `helm install` works with no `--set repoURL` needed. Override with `--set repoURL=oci://mirror` when mirroring. Bringing your own root Application? Set `deployer.includeRootApp=false` to render children-only — see [Argo CD Deployer Options](/aicr/v1.3.0/user-guide/cli-reference#argo-cd-deployer-options). |
| `flux` | Flux `HelmRelease` manifests plus their source objects, and a plain Kustomize `kustomization.yaml` at the bundle root (not a Flux `Kustomization` CR). |
| `terraform` | A Terraform root module (or child module) calling one module per component, with `depends_on` carrying the recipe's dependency graph. |
| `fleet` | Rancher Fleet: a `fleet.yaml` per component plus a root `gitrepo.yaml` (`--fleet-mode gitrepo`, the default), or a single `helmops.yaml` with values inlined (`--fleet-mode helmop`). |

```bash
# GitOps with Argo CD, sourced from your config repo
aicr bundle --recipe recipe.yaml --deployer argocd \
  --repo https://github.com/my-org/my-gitops-repo.git \
  --output ./bundles
```

## Output directory

`--output` defaults to the current directory. The bundle renders in a private
`.aicr-stage-*` directory, inside the output directory when it exists and
beside it when it does not, and reaches the output directory only once it is
complete. A run that fails while rendering, or that a first Ctrl-C or
`SIGTERM` interrupts before the bundle starts moving into place, leaves the
output directory as it found it. A first interrupt that arrives after that
does not stop the move.

Bundling into a directory that holds an earlier bundle replaces it. Every
top-level folder named like a component folder, three digits and a hyphen such
as `001-gpu-operator/`, is replaced or removed whatever it holds, even one you
created. So is a top-level file the new bundle also writes, such as
`README.md`, along with
`provenance.yaml`, `undeploy.sh` and an empty `static/` or `sources/`. Any
other file, including one under a directory such as `data/`, is replaced or
removed only when the new bundle writes a file at the same path, or the
earlier bundle's `checksums.txt` lists it and its content still matches, such
as a dropped `--data` file. The earlier bundle's attestation files are
replaced or removed whatever they hold, because `checksums.txt` cannot list
the files that sign it. The run fails and changes nothing if the directory
holds anything else, such as a file you added under `data/`, because
`checksums.txt` must list the whole tree. An earlier bundle is recognized only
when its `checksums.txt` lists `bundle-info.yaml` and that file still matches.
One with no `checksums.txt`, such as one made through the Go library with
checksums off, is not recognized, so any of its files the new bundle does not
write, such as a dropped `--data` file, fail the run. Bundle into a new
directory, or remove the earlier bundle first. An `argocd-helm` bundle that
omits the GPU Operator or the NVIDIA DRA driver also fails, before anything is
written, when the directory holds that component's output from an earlier
bundle. Bundle it into a new directory.

A run that is killed outright, for example by `SIGKILL`, the out-of-memory
killer, or a second Ctrl-C or `SIGTERM`, cannot clean up after itself. A run
that fails while moving the finished bundle into an existing output directory
keeps what it has not moved yet, and its error names the directory holding
it. Either can leave a `.aicr-stage-*` directory behind, and the output
directory partly replaced. Delete the `.aicr-stage-*` directory and bundle
again. If that run reports an unexpected file, the partial replacement
removed the `checksums.txt` that listed it, so bundle into a new directory
instead.

A `--split` set needs a path that does not exist yet. See
[Split a bundle by class](#split-a-bundle-by-class).

## Bundle layout

This is the canonical description of what a bundle contains. The trees below
are frozen at v1 and gated by `TestBundleLayoutMatchesManifest`, so a path
shown here will not disappear or be renamed without a deliberate, reviewed
change. Automation may read these paths.

Every deployer writes `bundle-info.yaml`, `checksums.txt`, `README.md` and
`recipe.yaml` at the bundle root. Six of the seven group components into
ordered `NNN-<component>` directories; Flux is the exception and uses a plain
`<component>` directory with shared `sources/`.

A bundle set produced by `--split` (helm deployer only) has a different
root: `recipe.yaml`, `split.yaml`, `bundle-info.yaml` and `checksums.txt`,
with one `<class>/` directory per class that is itself a helm bundle tree
as shown below, minus `bundle-info.yaml` and `checksums.txt`. The root
`bundle-info.yaml` lists every release with its class-prefixed path, so
`aicr upgrade-check --from <set>` and `aicr recipe --inherit-from <set>`
read a set like a bundle. See
[Split a bundle by class](#split-a-bundle-by-class).

```text
helm/                            argocd/                      flux/
  001-cert-manager/               001-cert-manager/            cert-manager/
    values.yaml                     application.yaml             helmrelease.yaml
    cluster-values.yaml             values.yaml                nfd/
    install.sh                    002-nfd/                       helmrelease.yaml
    upstream.env                    application.yaml           sources/
  002-nfd/                          values.yaml                  helmrepo-<host>.yaml
    ...                           app-of-apps.yaml               gitrepo-<host>.yaml
  deploy.sh                       bundle-info.yaml             kustomization.yaml
  bundle-info.yaml                checksums.txt                bundle-info.yaml
  recipe.yaml                     README.md                    checksums.txt
  checksums.txt                   recipe.yaml                  README.md
  README.md                                                    recipe.yaml
```

`helmfile` shares Helm's per-component files but not its root: it writes
`helmfile.yaml` instead of `deploy.sh`. A recipe with dependencies also
produces one `level-N.yaml` per dependency depth, which is derived from the
recipe rather than fixed by the layout.

```text
helmfile/
  001-cert-manager/            same four files as helm
  002-nfd/
  helmfile.yaml
  level-N.yaml                 one per dependency depth; absent when flat
  recipe.yaml
  bundle-info.yaml
  checksums.txt
  README.md
```

`terraform` shares Helm's per-component values and bundled charts — not its
`install.sh` or `upstream.env`, which only the shell path reads — and adds the
Terraform layer above them: one module call per component, with that
component's pre, post and readiness folders chained inside the call, plus the
single generic component module they all share. The bundle does not create a
cluster.
By default it is a root module that configures the `helm` provider from
variables, so it applies against whatever cluster you point it at; with
`--terraform-child-module` it is a child module with no provider block, for
calling from a configuration that creates the cluster itself.

```text
terraform/
  001-cert-manager/            values.yaml and cluster-values.yaml; Chart.yaml +
  002-nfd/                     templates/ for a bundled chart
  main.tf                      one module call per component; the dependency graph
  versions.tf                  required_providers + the helm provider config
  variables.tf                 cluster connection, wait/timeout/atomic
  outputs.tf
  terraform.tfvars.example     root-module form only
  modules/component/           the shared module every call resolves to, holding
                               the component's pre / chart / post / readiness
                               releases chained in order
  bundle-info.yaml
  recipe.yaml
  checksums.txt
  README.md
```

`argocd-helm` renders a Helm chart at the root — `Chart.yaml`, `values.yaml`,
`values.schema.json` — with one template per component.

`fleet` keeps Helm's per-component folders and adds its own files. In the
default `gitrepo` mode each folder gets a `fleet.yaml` (one Fleet bundle) and a
`.fleetignore`, and the root gets `gitrepo.yaml`; in `helmop` mode the root gets
a single `helmops.yaml` instead, with each component's values inlined. In
`helmop` mode the folders' values files are reference copies: Fleet, and
`--inherit-from` and `upgrade-check`, read the values in `helmops.yaml`. Both
trees are gated; the `helmop` tree has its own manifest.

```text
fleet/ (--fleet-mode gitrepo)        fleet/ (--fleet-mode helmop)
  001-cert-manager/                    001-cert-manager/   same four files as helm
    same four files as helm            002-nfd/
    fleet.yaml                         helmops.yaml
    .fleetignore                       recipe.yaml
  002-nfd/                             bundle-info.yaml
  gitrepo.yaml                         checksums.txt
  recipe.yaml                          README.md
  bundle-info.yaml
  checksums.txt
  README.md
```

Two kinds of name appear in these trees, and only one is a promise:

- **Fixed names are contract.** `deploy.sh`, `app-of-apps.yaml`,
  `kustomization.yaml`, `main.tf` with `modules/component/`, `gitrepo.yaml`,
  `helmops.yaml`, `fleet.yaml`, `bundle-info.yaml`, `checksums.txt`,
  `values.yaml`, `helmrelease.yaml`, and the `NNN-<component>` convention itself.
- **Derived names are not.** Flux writes one `helmrepo-<host>.yaml` per chart
  repository and helmfile one `level-N.yaml` per dependency depth, so both sets
  change with the recipe. Discover them by listing the directory rather than
  hardcoding a name.

`UPGRADING.md` is the one root file whose name is fixed but whose presence is
not: a bundle from any deployer may carry it, and only does when it needs it. Test for it
rather than assuming it. See [Upgrade guidance in the
bundle](#upgrade-guidance-in-the-bundle).

### Upgrade guidance in the bundle

A new pin is sometimes one an existing installation cannot simply move to: a
CRD is renamed, or objects have to be migrated by hand. When a component's
pinned version falls inside the `to` range of a `manual` or `blocked`
[transition record](/aicr/v1.3.0/contributor-guide/upgrade-transition-records), the bundle carries that
record in three places:

- **`UPGRADING.md`** at the bundle root: the summary, the precondition, whether
  the change can be reversed, the ordered steps, and references. Steps are
  filtered to the `--deployer` the bundle was built with, so an Argo CD bundle
  shows the GitOps path and a Helm bundle the imperative one.
- **A "Before You Upgrade" table** near the top of `README.md`, naming each
  affected component, its verdict, and the versions it applies to when you
  upgrade from them.
- **A warning** at the top of the `Note:` block that `aicr bundle` prints.

The guidance applies only when you apply the bundle over an existing
installation of an earlier version. A fresh install can skip it. A bundle
knows the versions it pins but not the ones your cluster runs, so every entry
names the `from` range it applies to rather than a verdict for your move. For
that verdict, run [`aicr upgrade-check`](/aicr/v1.3.0/user-guide/upgrading-a-deployed-stack) against the recipe you
deployed from.

Nothing is written when no pinned version needs it, and a `safe` record never
produces any of the three.

Bundling never fails because of a transition record. Checking records is the
job of `aicr upgrade-check` and the project's lint gate, so a version you pin
yourself with `--data` bundles even when no record covers it. If a record
cannot be read at all, `aicr bundle` still writes the bundle, leaves the
guidance out, and says so in its `Note:` block; run `aicr upgrade-check` to see
the upgrade steps instead.

### Bundle info

Every bundle carries a `bundle-info.yaml` at its root, written unconditionally
by every deployer — no flag turns it off. It answers three questions a
bundle cannot answer for itself: which deployer built it, which `aicr` binary
built it, and which Helm release landed in which directory.

`layout.entrypoint` names the file a consumer invokes or applies —
`deploy.sh`, `helmfile.yaml`, `app-of-apps.yaml`, `Chart.yaml`,
`kustomization.yaml`, `main.tf`, `gitrepo.yaml`, or `helmops.yaml` — so automation reads one key instead of branching on
`build.deployer`.

In a bundle set generated with `--split`, the root `bundle-info.yaml`
describes the whole set: `layout.entrypoint` is the first class's script
(`core/deploy.sh`), and it installs only the releases whose `path` starts
with that class directory. Every other release's `path` starts with a later
class directory, whose own `deploy.sh` installs it; `split.yaml` lists the
classes in install order. A consumer can recognize a set from
`bundle-info.yaml` alone by an entrypoint below the root and release paths
under more than one class directory. An explicit set marker in the record
is deferred to a readers-first schema change (ADR-022).

`layout.releases` lists every Helm release the bundle installs, in deployment
order, and that ordering is normative: there is no ordinal field, so a
consumer reads sequence from list position rather than from a number. A
release injected alongside a component — a `-pre` folder when it declares
pre-install manifests, a `-post` folder when a component that also ships an
upstream chart declares post-install manifests, or a `-readiness` folder
under `--readiness-hooks` (none of the three tied to `--vendor-charts`) — has
no recipe component of its own, so it names its parent in `component` while
`name` carries its own suffixed name. One collision is deliberately left
undefined: a recipe that declares a component whose own name ends in `-pre`,
`-post` or `-readiness` alongside the matching base name makes `component`
deployer-dependent for that release, so do not rely on its value there. No
component in the shipped registry has such a name, and resolving this is a
reserved additive change — a later release may add an explicit field
distinguishing a primary release from an injected one.

`bundle-info.yaml` deliberately carries no component *inventory*:
`recipe.yaml` sits beside it at the bundle root and is already the source of
truth for what the recipe resolved to. `build.settings.components` in the
example below is not an inventory — it is the REST API's `?bundlers=` filter
(there is no CLI flag for it), recorded because the filtered `recipe.yaml`
alongside it is written post-filter and would otherwise be indistinguishable
from an unfiltered bundle of a smaller recipe. It is absent unless that
filter was used; see [Overrides that cannot take effect are
rejected](#overrides-that-cannot-take-effect-are-rejected) for the same
`bundlers=` filter from the override side. `bundle-info.yaml` also carries no
timestamp — the record feeds `checksums.txt`, which is the subject of the
bundle attestation, so a wall-clock field would make every bundle
irreproducible.

`warnings` is reserved for the non-blocking warnings `aicr bundle` prints in
its `Note:` block. Each entry has a stable UPPER_SNAKE `code`, an optional
`component`, and a `message` that may change. This release reads the field
but does not write it yet: `bundle-info.yaml` is decoded strictly, and a
release that predates the field rejects a record that carries it, so a release
that reads it ships before one that writes it. Until then, warnings appear
only in the `Note:` block.

Below is a fully populated example, generated from the shipped serializer and
extended with fields (`repoURL`, `provenance`, an injected `-post`/`-readiness`
pair) that a single bundle rarely exercises all at once. Key order is
alphabetical at every nesting level — `serializer.MarshalYAMLDeterministic`
sorts mapping keys before writing — so the top level reads `apiVersion, build,
kind, layout, metadata`, not a hand-arranged order. Expect the same
alphabetical order in your own bundle.

```yaml
apiVersion: aicr.run/v1
build:
  deployer: argocd
  recipe:
    digest: sha256:d429490ec53e902e5a7f5a9b0221ab4a46f233ec70f808f1e62bbade2576e083
    path: recipe.yaml
    version: v0.22.0
  settings:
    appName: gpu-cluster
    attested: true
    checksums: true
    components:
      - cert-manager
      - nfd
      - network-operator
    nodeScheduling:
      accelerated:
        selector:
          nvidia.com/gpu.present: "true"
        tolerations:
          - effect: NoSchedule
            key: nvidia.com/gpu
            operator: Equal
            value: present
      system:
        selector:
          nodeGroup: system
        tolerations:
          - effect: NoSchedule
            key: dedicated
            operator: Equal
            value: system
    readinessHooks: true
    repoURL: https://github.com/my-org/my-gitops-repo.git
    serial: false
    sharedStorageClass: shared-nfs
    storageClass: local-nvme
    targetRevision: main
    vendorCharts: true
kind: BundleInfo
layout:
  entrypoint: app-of-apps.yaml
  provenance: provenance.yaml
  releases:
    - component: cert-manager
      manifest: 001-cert-manager/application.yaml
      name: cert-manager
      namespace: cert-manager
      path: 001-cert-manager
    - component: nfd
      manifest: 002-nfd/application.yaml
      name: nfd
      namespace: node-feature-discovery
      path: 002-nfd
    - component: network-operator
      manifest: 003-network-operator/application.yaml
      name: network-operator
      namespace: nvidia-network-operator
      path: 003-network-operator
    - component: network-operator
      manifest: 004-network-operator-post/application.yaml
      name: network-operator-post
      namespace: nvidia-network-operator
      path: 004-network-operator-post
    - component: network-operator
      manifest: 005-network-operator-readiness/application.yaml
      name: network-operator-readiness
      namespace: nvidia-network-operator
      path: 005-network-operator-readiness
metadata:
  version: v0.22.0
```

`build.recipe.version` and `metadata.version` can diverge: the former is the
`aicr` binary that resolved `recipe.yaml` (stamped by the recipe builder and
never restamped), the latter is the `aicr` binary that ran `bundle`. They read
the same on a one-step workflow and differ whenever recipe generation and
bundling run on different releases. `build.settings` includes only settings
whose effect is already observable elsewhere in the bundle's own files —
Fulcio/Rekor endpoints, certificate identity, and free-form `--set` overrides
are excluded by construction, since `values.yaml` already carries the effect
of the latter.

That test is applied per deployer, so `repoURL`, `targetRevision` and
`appName` appear only where the bundle shows them: all three under `argocd`,
`repoURL` and `targetRevision` under `flux`, `appName` alone under
`argocd-helm` (whose chart is URL-portable and takes the publish location at
`helm install --set repoURL=...` time, which is also why `--repo` warns that
it is ignored there), and none of the three under `helm`, `helmfile` or `terraform`. Where
a key does not apply it is omitted entirely rather than written empty.

Each deployer reports what it resolved, so the recorded value is the one the
bundle carries — including the fallback it applies when you pass no flag. A
`flux` bundle built without `--repo` records
`repoURL: https://github.com/YOUR_ORG/YOUR_REPO.git`, the same placeholder
written into `sources/gitrepo-*.yaml`, and `targetRevision: main`. That is
deliberate: the placeholder is what ships, and a bundle that needs its repo
URL replaced before it can be applied should say so rather than look
unconfigured.

### Generated chart versions

Not every chart in a bundle comes from upstream. AICR generates one for each
Kustomize-derived component, each manifest-only component, each injected
`-pre` / `-post` / `-readiness` release, and — under `--vendor-charts` — as a
wrapper around every vendored upstream chart. A generated `Chart.yaml` has to
answer two different questions, so it answers them in separate fields:

| Field | Carries | Why |
|-------|---------|-----|
| `version` | the AICR version that produced the bundle | The chart's content is AICR's own, so AICR's version identifies the artifact. This is the `aicr` binary that ran `bundle`, which is not necessarily the one that generated the recipe. |
| `appVersion` | the payload version | The upstream chart pin for a Helm component, the git ref for a Kustomize one. Free-form, so a ref like `release-1.4` need not look like SemVer. |
| `aicr.run/component-version` annotation | the payload version | The same value as `appVersion`, under a stable key to read from a live release. |
| `aicr.run/generated-by` annotation | the AICR version | The same value as `version`. |

To read the payload version back out of a cluster, the rule is: **use
`aicr.run/component-version` when it is present, otherwise use the release's
own chart version.** A component installed straight from its upstream chart
carries neither annotation, and its release version already *is* the payload
version. The annotation's presence is exactly the signal that the chart
version describes the wrapper instead of what it wraps.

So for the `gpu-operator-post` release generated alongside gpu-operator, a
`helm list` reports chart `gpu-operator-post-0.22.0` — the AICR version — while
its app version and `aicr.run/component-version` both read `v26.7.2`, the
gpu-operator pin those manifests accompany.

A component with no upstream pin — a manifest-only component, and the injected
releases belonging to one — ships only AICR-authored content, so `appVersion`
and the annotation report the AICR version too.

A build that is not release-stamped (`aicr --version` reports `dev`) reports
`0.0.0-dev`. Helm validates `version:` as SemVer 2 and refuses to load a chart
whose version is not, so `dev` cannot be written through verbatim.

The root chart `argocd-helm` renders is not one of these generated wrappers and
carries neither annotation. Its `version:` tracks the recipe's
`metadata.version` — the AICR build that generated the *recipe*, not the one
that ran `bundle` — so on a two-step workflow where the two binaries differ, it
will not match the wrappers alongside it. Only the `0.0.0-dev` substitution
above is shared.

Verify a bundle you received with `aicr verify` — see
[Artifact verification](/aicr/v1.3.0/user-guide/artifact-verification).

## Override values

Use `--set` for scalar overrides, scoped per component as
`component:path.to.field=value`:

```bash
aicr bundle --recipe recipe.yaml \
  --set gpuoperator:driver.version=570.86.16 \
  --set gpuoperator:gds.enabled=true \
  --output ./bundles
```

On recipes that carry an ADR-015 configuration profile
(`metadata.selectedProfile`, e.g. the AKS family's `gpuStack`), the
profile's owned paths are locked. The lock is enforced per surface:

- **`aicr bundle` static overrides** (`--set`, `--set-json`,
  `--set-file`, or a config-file override, from any of those sources):
  a value identical to the selected one is accepted; a divergent value
  is rejected. The typed sources (`--set-json`, `--set-file`) are
  always rejected for the special `enabled` presence key — even when
  the value matches — because they would write a stray literal
  `enabled:` chart value instead of toggling the component.
- **`aicr mirror list --set` overrides**: mirror exposes only the
  repeatable scalar `--set` (no `--set-json`/`--set-file`), and the
  same identical-accepted / divergent-rejected rule applies to it.
  Note that `mirror list` does not apply a config file's
  `spec.bundle.deployment.set` overrides — pass image-affecting
  overrides to mirror via `--set` explicitly.
- **`--dynamic` exports**: rejected whenever they merely intersect an
  owned path, regardless of the current value — install-time mutability
  of a locked path is itself the violation.
- **argocd-helm install-time values**: any install-time key whose path
  equals, contains, or is contained by an owned path is rejected at Helm
  render time, even when the value is identical. Key presence alone
  trips the guard.
- **argocd-helm driver-ownership paths, profiled or not**: independent of
  any profile, an `argocd-helm` bundle that ships `gpu-operator` or
  `gpu-operator-ocp` rejects install-time values on `driver.enabled`,
  `toolkit.enabled` and `hostPaths.driverInstallDir`, and one that ships
  `nvidia-dra-driver-gpu` or `nvidia-dra-driver-gpu-ocp` rejects
  `nvidiaDriverRoot`. These paths decide who owns the GPU driver, and
  bundle-time checks verify them (`CheckDriverOwnershipCoherence`, when the
  bundle ships a GPU Operator), so an install-time value would bypass that
  verdict. As above, a key that equals, contains, or is contained by a
  guarded path trips the guard on presence alone (`false`, `null`, `[]`,
  a scalar replacing a parent, or an identical value). The guard names
  the path and remedy; when a scalar replaces the whole component, Helm
  may report the component template's type error first. Set ownership at
  bundle time instead (`aicr bundle --set gpuoperator:driver.enabled=...`)
  so bundle-time ownership checks apply; siblings such as
  `gpuoperator.driver.version` stay overridable. When the selected profile
  owns the path (the AKS `gpuStack` profile owns `driver.enabled`,
  `toolkit.enabled` and `nvidiaDriverRoot`), a bundle-time `--set` is
  rejected as well, so choose the profile value with
  `aicr recipe --profile gpuStack=...` instead. A `--dynamic` declaration
  on a guarded path would stub it into the chart's root `values.yaml`,
  which the guard rejects on every render, so `aicr bundle` rejects it.
  The guard lives in `templates/aicr-driver-ownership-lock.yaml`.
  With `--vendor-charts`, a component's wrapper chart forwards values to the
  upstream chart under its dependency name, so the guard also rejects the
  same paths there (for example `gpuoperator.gpu-operator.driver.enabled`, or
  `dradriver.dra-driver-nvidia-gpu.nvidiaDriverRoot`); siblings such as
  `gpuoperator.gpu-operator.driver.version` stay overridable.
  A bundle subset (the REST `bundlers=` filter or the SDK's
  `config.WithBundlers`) that omits these components carries no guard
  for them. When regenerating into an existing directory, stale artifacts
  for an omitted or disabled guarded component are rejected before any
  output is written, even without checksums; generate into a fresh
  directory. An existing install that sets these values at install time
  fails after upgrading; see the
  [upgrade note](/aicr/v1.3.0/user-guide/component-catalog#gpu-operator-and-nvidia-dra-driver-gpu-argocd-helm-rejects-install-time-driver-ownership-values).
- **Component presence (the synthetic `enabled` owned path)**: not
  changeable by selecting a different profile value — profile fragments
  cannot assign `enabled`, so no `--profile` choice adds or removes a
  component. The lock rejects removing (or bundle-subsetting away) an
  owned component; changing which components are present is a
  catalog/composition change. A component the profile references as
  `optional` has no presence lock: it may be removed, and its owned
  value paths are enforced only when it ships.

Change owned **value** paths by regenerating with
`aicr recipe --profile name=value` instead; component presence is not
affected by reselection.

`--set` is scalar-only. For list or object values use `--set-json` (inline JSON)
or `--set-file` (value read from a file); both deep-merge objects and replace
lists/scalars, and take precedence over `--set` on the same path:

```bash
aicr bundle --recipe recipe.yaml \
  --set-json agentgateway:allowedSourceRanges='["216.228.127.128/30"]' \
  --output ./bundles
```

The `agentgateway` inference-gateway is **private by default**: with no
`allowedSourceRanges` override, the bundler scopes the LoadBalancer to private
RFC1918 ranges so it is not exposed to the public internet. Use the override
above to admit specific clients (e.g. a corporate VPN). On a cluster with no
LoadBalancer provider, `--set agentgateway:serviceType=ClusterIP` gives the
gateway a ClusterIP Service instead, to which source ranges do not apply. See
[Inference Gateway Network Exposure](/aicr/v1.3.0/user-guide/component-catalog#inference-gateway-network-exposure).

### Size controllers for large clusters

Some controllers' memory grows with node count. The controllers below ship
default memory **limits** that cover clusters of about 2500 nodes, in the
recipes each row names; a row with no recipe applies to all of them. Their
**requests** are sized for small clusters, so a large cluster schedules them
with far less reserved than they use. On large clusters, raise the requests so
the scheduler reserves memory on the system nodes:

| Controller | Override key | Resources path |
|---|---|---|
| GPU Operator | `gpuoperator` | `operator.resources` |
| Network Operator (generic GB300) | `networkoperator` | `operator.resources` |
| Maintenance Operator (Network Operator subchart, generic GB300) | `networkoperator` | `maintenance-operator-chart.operator.resources` |
| Nodewright manager | `nodewright` | `controllerManager.manager.resources` |

The Network Operator limit is sized for large clusters only in the generic
GB300 recipe. Other recipes keep a lower limit (2Gi on OKE, the chart default
elsewhere), so raise `limits.memory` together with the request there: a
request above its limit is rejected.

The NIC configuration daemon, a per-node DaemonSet whose pods each cache every
`NicDevice` in the cluster, is not sized by AICR. The generic GB300 and NVKE
VR200 recipes' `NicClusterPolicy` sets no resources for it, so the Network
Operator applies its own default (100m CPU and 512Mi memory requests, 5Gi
memory limit). To set them yourself, use the `networkoperator` values path
`nicConfigurationOperator.configurationDaemon.resources`, which the policy
manifest reads:

```bash
aicr bundle --recipe recipe.yaml \
  --set networkoperator:nicConfigurationOperator.configurationDaemon.resources.limits.memory=8Gi \
  --set networkoperator:nicConfigurationOperator.configurationDaemon.resources.requests.memory=1Gi \
  --set networkoperator:nicConfigurationOperator.configurationDaemon.resources.requests.cpu=100m \
  --output ./bundles
```

An override replaces the operator's default as a whole: the operator fills in
no field you leave out. It must set both `requests.memory` and
`limits.memory`, or the bundle fails. Set `requests.cpu` as well (100m is the
operator's default); without it the daemon runs with no CPU request.

Earlier releases documented overriding `limits.memory` alone, which kept
AICR's own requests. That command now fails: add `requests.memory` and
`requests.cpu` to it.

The `helm`, `helmfile`, `terraform`, `argocd`, and `argocd-helm` deployers
render this manifest when the bundle is generated, so the install-time overrides
below do not reach it. A `flux` bundle ships the manifest as a template, and Flux renders
it at install time from the values in `network-operator-post/helmrelease.yaml`.
Set `spec.values.network-operator.nicConfigurationOperator.configurationDaemon.resources`
there to resize the daemon without regenerating the bundle, with the same three
fields: an override missing either memory value fails when Flux renders the
release.

At bundle time, scope each resources path in the table with its override key:

```bash
aicr bundle --recipe recipe.yaml \
  --set gpuoperator:operator.resources.requests.memory=4Gi \
  --set networkoperator:operator.resources.requests.memory=8Gi \
  --set networkoperator:maintenance-operator-chart.operator.resources.requests.memory=8Gi \
  --set nodewright:controllerManager.manager.resources.requests.memory=8Gi \
  --output ./bundles
```

An `argocd-helm` bundle also takes these at install time, without
regenerating. Each child Application merges the root chart's
`.Values.<override key>` over the bundle's static values. Use a `.` after the
override key instead of a `:`. Bundles generated before these limits were
raised still carry the old, lower limits, so set each limit with its request:

```bash
helm install aicr-bundle . \
  --set gpuoperator.operator.resources.requests.memory=4Gi \
  --set gpuoperator.operator.resources.limits.memory=8Gi \
  --set networkoperator.operator.resources.requests.memory=8Gi \
  --set networkoperator.operator.resources.limits.memory=12Gi \
  --set networkoperator.maintenance-operator-chart.operator.resources.requests.memory=8Gi \
  --set networkoperator.maintenance-operator-chart.operator.resources.limits.memory=12Gi \
  --set nodewright.controllerManager.manager.resources.requests.memory=8Gi \
  --set nodewright.controllerManager.manager.resources.limits.memory=12Gi
```

When the root Application installs the chart, put these keys in its Helm
values. Do not edit the child Applications: the root Application's self-heal
reverts them.

These overrides only change requests and limits. Free memory on the system
nodes still decides whether the controllers fit.

### Required and install-time secret inputs

A few components need a value the recipe cannot carry, because it differs per
cluster: `runai-cluster` needs the Run:ai control plane URL, the cluster UID
that control plane assigned, the FQDN it reaches the cluster at, and the
bootstrap token it issued. The registry
lists such paths as `requiredValues`, and `aicr bundle` refuses a bundle that
leaves one empty instead of rendering an install that cannot work:

```text
[INVALID_REQUEST] component "runai-cluster": required values not set:
controlPlane.url, cluster.uid, cluster.url (pass --set runai:<path>=<value>,
or declare --dynamic runai:<path> to supply it at install time);
controlPlane.bootstrapToken (an install-time secret: declare --dynamic
runai:<path> and fill it in cluster-values.yaml before install)
```

A path counts as supplied when it holds a non-empty value after every
override, or when it is declared with `--dynamic`, which moves it into the
component's `cluster-values.yaml` for you to fill in on the install machine.

A path the registry marks as an install-time secret (`installTimeSecretPaths`)
can only be supplied the second way. Passing a value with `--set`, or putting
one in a values file or overlay, is rejected, so no bundle, golden or
evidence ever contains the credential:

```shell
aicr bundle -r recipe.yaml \
  --set runai:controlPlane.url="${CONTROL_PLANE_URL}" \
  --set runai:cluster.uid="${CLUSTER_UID}" \
  --set runai:cluster.url="${CLUSTER_FQDN}" \
  --dynamic runai:controlPlane.bootstrapToken \
  -o ./bundle
```

Two further rules follow from the secret being filled in after bundling:

- **Only the `helm` deployer** may carry such a component. `flux`, `fleet`,
  `terraform` and `argocd-helm` would store the filled value at rest (a
  ConfigMap manifest, a values file committed to Git, Terraform state, the
  Application's `helm.values`); `helmfile` reads `cluster-values.yaml` at
  apply time like `helm` but has no stub check, so an empty token would reach
  the chart. `aicr bundle` rejects all of them for a component whose secret
  path is required or declared dynamic. Plain `argocd` rejects every
  `--dynamic` declaration already.
- **`install.sh` checks the stub.** For a required path declared dynamic, the
  component's install script resolves the path in `cluster-values.yaml` and
  refuses to run while it is missing or empty (`""`, `''`, `~` or `null`,
  a trailing comment allowed), naming the path. Each path is checked at its
  full path, so two required paths sharing a leaf key are checked apart, and
  a declaration on a parent map is judged on the leaves written beneath it.
  `DRY_RUN_FLAG=--dry-run ./deploy.sh` skips the check, so a bundle can be
  previewed before a short-lived value is minted.

The registry side of this contract is described in
[requiredValues and installTimeSecretPaths](/aicr/v1.3.0/contributor-guide/components#requiredvalues-and-installtimesecretpaths).

## Enable or disable components

The special `enabled` key includes or excludes a component at bundle time
without editing the recipe:

```bash
# Skip the AWS EBS CSI driver for this bundle
aicr bundle --recipe recipe.yaml \
  --set awsebscsidriver:enabled=false \
  --output ./bundles
```

A recipe or overlay can also disable a component by default via
`overrides.enabled: false` (for example, a platform that ships its own
cert-manager). Such components are already excluded from the recipe's
`deploymentOrder`.

`--set <component>:enabled=false` disables a component the recipe leaves on.
A component the recipe **disabled** cannot be re-enabled at bundle time —
`--set <component>:enabled=true` on such a component is rejected with an error.
The recipe author disables a component because the target platform already
provides it, so re-enabling would install a conflicting second copy. To deploy
a component the recipe disables, edit the recipe/overlay instead.

### Overrides that cannot take effect are rejected

An override whose component will not appear in the generated bundle is
rejected with an error rather than silently discarded. This covers a
component that is absent because the recipe disabled it, because
`--set <component>:enabled=false` removed it, because the `bundlers=`
filter excluded it, or because the component name is neither one the recipe declares nor a
registered `valueOverrideKeys` alias of one (usually a typo — registered
aliases such as `gpuoperator` for `gpu-operator` remain valid):

```bash
# Rejected: the second --set can never take effect
aicr bundle --recipe recipe.yaml \
  --set nv-sentinel:enabled=false \
  --set nv-sentinel:labeler.assumeDriverInstalled=true \
  --output ./bundles
```

The two flags ask for contradictory things — remove the component, and
configure it — so the command fails instead of shipping a bundle with
one request quietly dropped. The rejection here is about the contradiction,
not about who owns the component: a profile-owned presence lock is a
separate rejection, described at the end of this section. Only a scalar `--set <component>:enabled=false`
is exempt on a declared component: it is the supported way to remove
one, and it is also accepted on a component the recipe already disables.
`enabled=true` on a component the `bundlers=` filter excludes is
rejected like any other ineffective override, and the `enabled` key is
never honored from `--set-json`/`--set-file` (present or absent — the
typed path would write a literal `enabled:` chart value instead of
toggling the component).

A component whose presence a configuration profile owns cannot be removed
at all — `enabled=false` on it is rejected regardless of the rule above.
NVSentinel is in that position on the AKS family, whose `gpuStack`
profile names it. On the GKE-COS and OKE families the `gpuStack` profile
references NVSentinel as optional, so `--set nv-sentinel:enabled=false`
removes it, for example when NVSentinel is deployed outside AICR. The
profile's `labeler.assumeDriverInstalled` value still applies whenever
the bundle includes NVSentinel. A recipe generated by an earlier release
still locks presence; regenerate it with `aicr recipe` first. See
[NVSentinel on provider-installed-driver platforms](/aicr/v1.3.0/user-guide/component-catalog#nvsentinel-on-provider-installed-driver-platforms).

The same rule applies to `--set-json`, `--set-file`, `--dynamic`, and
the REST API's equivalent parameters. For `--dynamic` no path is exempt,
`enabled` included: a dynamic path on an absent component exports
nothing (there is no `cluster-values.yaml` to defer it to), and a
dynamic path is never a removal idiom. Unknown component names were
already rejected by the `bundlers=` filter and by `--dynamic`
registry validation; this extends the same fail-closed rule to every
override source.

## Pin node scheduling

Steer system components and GPU workloads onto the right nodes with selector and
toleration flags (repeatable):

```bash
aicr bundle --recipe recipe.yaml \
  --system-node-selector nodeGroup=system \
  --system-node-toleration dedicated=system:NoSchedule \
  --accelerated-node-selector nvidia.com/gpu.present=true \
  --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \
  --output ./bundles
```

Most components treat the selectors as optional, but components that declare
`requireNodeSelector` in the registry (`nvsentinel-mongodb`, `slinky-slurm`,
`slurm-accounting-mariadb`) fail the bundle without them. See the selector
rows in the
[`aicr bundle` flag table](/aicr/v1.3.0/user-guide/cli-reference#aicr-bundle).

## Prepare DRA nodes when opting in to eviction coordination

DRA eviction coordination is **opt-in**. By default a bundle containing both
`gpu-operator` and `nvidia-dra-driver-gpu` adds no eviction node label, and the
DRA kubelet plugin runs on every accelerated node with no extra labeling. The
trade-off is that the plugin is not descheduled ahead of a GPU driver container
restart; `aicr bundle` warns about this where GPU Operator manages the driver,
and [DRA Driver Upgrade Eviction](/aicr/v1.3.0/user-guide/cli-reference#dra-driver-upgrade-eviction)
describes what can go wrong.

Generate with `--dra-eviction-node-label key=value` to opt in. The rest of this
section applies only then. The same applies to the corresponding `-ocp`
components.

Opting in also keeps `dra-node-labeler` in the bundle. It applies the
configured `key=value` to every node GFD labels `nvidia.com/gpu.present=true`
and never rewrites an existing value, so the node-pool labeling described below
is only needed when the labeler is not in the bundle: because you removed it
with `--set dra-node-labeler:enabled=false`, because a `bundlers` filter left
it out, or on OpenShift, where it is not yet wired (NVIDIA/aicr#2828). A
`bundlers` selection that names the labeler but omits the flag or one of its
prerequisites is rejected rather than rendered without it. Everything below that says "labeler disabled" applies to
those cases equally.

### Choosing whether to opt in

The label is how GPU Operator's Driver Manager finds the plugin: it deschedules
the plugin by rewriting the label's value and restores it afterwards, so the
plugin's `nodeSelector` has to match the label for the mechanism to work at all.
That is what makes it a placement requirement, and why an unlabeled node ends up
with no plugin rather than with an uncoordinated one.

| | Not opted in (default) | Opted in |
|---|---|---|
| Node labeling | none needed | applied by `dra-node-labeler` from `nvidia.com/gpu.present`; every GPU node in the node pool definition only if the labeler is disabled |
| Plugin placement | every accelerated node | only nodes carrying the label |
| Driver restart | plugin is not descheduled first | plugin is descheduled and restored |
| If a node is missed | n/a | that node silently runs without DRA |

Opt in when GPU Operator manages the driver (`driver.enabled=true`). With the
bundled `dra-node-labeler` the label follows GFD's `nvidia.com/gpu.present`, so
nodes added later by autoscaling or replacement are labeled as soon as GFD sees
them; the remaining gap is a GPU node GFD has not labeled and that does not
already carry the configured `key=value`, which runs no kubelet plugin until
one of the two appears. If you disable the labeler, opt in only when you can
guarantee the label is set at provisioning time for every GPU node; otherwise
the default is the safer choice: a plugin that always runs, with a documented
risk at driver restarts, beats a plugin that silently does not run on some nodes.

There is nothing to opt in to where the driver is provider-installed
(`driver.enabled=false` — AKS `azure-managed`, GKE COS, OKE). GPU Operator
deploys no driver pod and therefore no Driver Manager, so no restart can occur
under the plugin and no warning is emitted.

Also note the mechanism is best-effort under `k8s-driver-manager` v0.12: the
configured label is paused in the same batch as other GPU operands and no wait
covers the standalone DRA kubelet plugin, so ordering against DRA claim holders
and completion of plugin teardown are not guaranteed. See
[NVIDIA/k8s-driver-manager#250](https://github.com/NVIDIA/k8s-driver-manager/issues/250).

> **Opt-in requirement (labeler disabled only):** if you pass
> `--set dra-node-labeler:enabled=false`, label every GPU node that must run
> the DRA kubelet plugin *before* applying the bundle. Applying it first can
> reduce the DaemonSet to zero eligible nodes, interrupting ComputeDomain/IMEX
> and any whole-GPU resources advertised through DRA. With the labeler in the
> bundle there is nothing to pre-label; the plugin follows the labeler.

### Set the label at node-pool provisioning time (labeler disabled)

This subsection applies only when `dra-node-labeler` has been disabled. Put the
label in the **node pool definition** — an EKS managed nodegroup `labels`
entry, a Karpenter `NodePool` `spec.template.metadata.labels` entry, or the
equivalent for your provisioner — alongside the `nodeGroup=gpu-worker` label
you already set there.

A one-off `kubectl label node` is a repair, not a configuration. It does not
survive node replacement or recycling, cluster autoscaling adding GPU nodes, or
a nodegroup scaled from zero. Any GPU node added afterwards arrives unlabeled
and silently runs without the DRA kubelet plugin, leaving the cluster
**partially DRA-enabled**. This is harder to detect than uniform failure,
because it is intermittent and node-dependent.

Use `kubectl label` only to repair nodes that already exist, and fix the node
pool definition in the same change so replacements inherit it. Substitute the
same `key=value` pair you passed to `--dra-eviction-node-label` — the examples
below use the documented default:

```bash
NODE="my-gpu-node"  # replace with the GPU node to repair
kubectl label node "${NODE}" nvidia.com/dra-kubelet-plugin=true
kubectl get nodes -l nvidia.com/dra-kubelet-plugin=true
```

### The failure mode is silent

An unlabeled GPU node produces no error anywhere. `helm install`/`helm
upgrade` reports success and the bundle's `deploy.sh` exits 0. What you get
instead is:

- no DRA kubelet plugin on any unlabeled node, and no `ResourceSlices` from
  it — so no ComputeDomain/IMEX capability there
- the `nvidia-dra-driver-gpu-kubelet-plugin` DaemonSet at `DESIRED=0` **if no
  GPU node carries the label at all**

Partial coverage is the shape node replacement and autoscaling produce: labeled nodes work normally while the rest silently lack
DRA. A split cluster is harder to notice than uniform failure, because the
DaemonSet looks healthy and only some workloads misbehave.

Because the absence is not self-announcing, confirm the selector matches the
nodes you expect **before** applying the bundle, and check the DaemonSet
afterwards.

### This applies to existing clusters, not just fresh installs

The requirement is easy to read as a fresh-install prerequisite, but the
upgrade path is especially easy to miss. A cluster whose bundle was generated
before this selector existed has a working kubelet-plugin DaemonSet selecting
on `nodeGroup=gpu-worker` alone. Regenerating the bundle and running `helm
upgrade` adds the second selector, and working functionality **disappears** —
still with no error. Revisit the node labels whenever you regenerate a bundle
for an existing deployment, not only when building a new cluster.

`aicr bundle` emits a non-blocking warning describing this requirement whenever
both components are enabled and an eviction label is configured. The
complementary opt-out warning fires only where GPU Operator manages the driver
(`driver.enabled=true`). See
[Storage Class](/aicr/v1.3.0/user-guide/cli-reference#storage-class), where that warning is
described alongside the other cluster-state dependency reported the same way.

### Upgrading from a build that applied the label implicitly

For a window on unreleased `main`, the eviction label was applied
automatically rather than requested. Bundles generated in that window — via the
CLI, the REST API, config, or a Go caller — received the
`nvidia.com/dra-kubelet-plugin=true` selector and the matching Driver Manager
environment entry without asking for them. No tagged release contains that
behavior, so only clusters built from `main` in that interval are affected.

At this head the same inputs produce the opposite result: omitting
`--dra-eviction-node-label` removes both the selector and the Driver Manager
entry. Regenerating and upgrading such a cluster therefore *drops* eviction
coordination silently — the reverse of the direction described above, and the
case the preceding subsection does not cover.

Decide deliberately, and only two answers are defensible:

- **Keep coordination.** Confirm the nodes still carry the label
  (`kubectl get nodes -l nvidia.com/dra-kubelet-plugin=true`), confirm the count
  matches the GPU nodes you expect, then pass
  `--dra-eviction-node-label nvidia.com/dra-kubelet-plugin=true` explicitly on
  every subsequent generation. Verify node labels *before* upgrading: passing
  the flag against unlabeled nodes takes the DaemonSet to `DESIRED=0`.
- **Accept the opt-out.** Regenerate without the flag and accept the documented
  risks in [DRA Driver Upgrade
  Eviction](/aicr/v1.3.0/user-guide/cli-reference#dra-driver-upgrade-eviction). The node labels
  become inert and can be removed at leisure.

The opt-out warning `aicr bundle` prints bounds the blast radius at generation
time, but it is not upgrade guidance: it fires on every unlabeled generation and
cannot know the cluster previously had coordination.

### Custom label conventions and post-install checks

The flag both opts in and selects the convention: generate the bundle with
`--dra-eviction-node-label key=value` and apply that exact pair to the nodes.
AICR gives the full pair to the DRA node selector, but GPU Operator's Driver
Manager receives only the label key because its eviction contract matches and
temporarily removes the label by key.

After installation and after every GPU driver upgrade, monitor the kubelet
plugin DaemonSet until all desired pods are ready. This also catches a Driver
Manager rollout that did not restore the eviction label:

```bash
kubectl -n nvidia-dra-driver get daemonset \
  nvidia-dra-driver-gpu-kubelet-plugin
```

The integration is not rendered when either component is absent. A dynamic
declaration intersecting `kubeletPlugin.nodeSelector` or `driver.manager.env`
is rejected because moving either path to install-time configuration would let
the two halves drift independently. See
[DRA Driver Upgrade Eviction](/aicr/v1.3.0/user-guide/cli-reference#dra-driver-upgrade-eviction) for
configuration details and NVIDIA's
[GPU Operator DRA installation guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.3/dra-intro-install.html)
for the upstream contract.

## Produce an offline (vendored) bundle

`--vendor-charts` pulls upstream Helm chart bytes into the bundle at bundle
time, so the artifact needs no Helm chart registry egress at deploy time. Each
vendored chart is recorded in `provenance.yaml` with name, version, source URL,
and SHA256. Requires the `helm` binary on `PATH`.

```bash
aicr bundle --recipe recipe.yaml --vendor-charts --output ./bundles
```

> Trade-off: vendoring freezes the chart version. A vendored bundle will keep
> installing a frozen chart even if upstream later yanks it for a CVE — you lose
> the fail-loud signal you get when pulling charts live. Container-image pulls
> may still require network access. For full air-gapped operation, also mirror
> images; see [Air-Gap Mirror](/aicr/v1.3.0/user-guide/air-gapped-mirroring).

Recipe-side manifests of mixed components (AICR-authored manifests shipped
alongside a vendored upstream chart — for example the network-operator
`NicClusterPolicy` or the AKS `nvidia-peermem-reloader` DaemonSet) get the same
lifecycle under `--vendor-charts` as they do without it. The vendored primary
folder wraps only the upstream chart, and the manifests are emitted as a
separate `<component>-post` Helm release installed immediately after it:

```text
002-network-operator/          # wrapper chart + charts/<chart>-<ver>.tgz
003-network-operator-post/     # recipe-side manifests, tracked release
```

Because they are ordinary members of that release rather than Helm hook
resources, they are patched in place by `helm upgrade` (three-way merge),
removed by `helm uninstall`, and applied normally by Argo CD under
`syncPolicy.automated`. Bundle-layer `NNN-` folder ordering sequences the two
releases, so any `helm.sh/hook` annotation a recipe manifest declares is
stripped when the `-post` chart is written — the same treatment the
non-vendored path has always applied.

> Earlier releases injected these manifests into the vendored wrapper chart as
> `helm.sh/hook: post-install` resources, which Helm never re-applied on
> upgrade, left behind on uninstall, and Argo CD silently skipped as a PostSync
> hook. Those live objects are **not** members of the new `<component>-post`
> release, so a redeploy of a rebundled layout fails with Helm ownership
> conflicts (`exists and cannot be imported into the current release`) until
> each resource is adopted or removed.

Adoption is the default path — it is non-destructive and required before
`helm upgrade --install` of the `-post` chart. For each previously
hook-injected resource (name and kind from the new
`<NNN>-<component>-post/templates/` files, or from the old primary
`templates/` if you still have that bundle):

```bash
COMPONENT="my-component"  # replace with the component that owns the -post chart
NAMESPACE="my-namespace"  # replace with the component's release namespace
KIND="configmap"          # replace with the resource kind
NAME="my-resource"        # replace with the resource name
# Omit -n "${NAMESPACE}" from both commands for cluster-scoped kinds (CRD,
# ClusterRole, NicClusterPolicy, …); release-namespace keeps it either way.
kubectl label -n "${NAMESPACE}" "${KIND}/${NAME}" app.kubernetes.io/managed-by=Helm --overwrite
kubectl annotate -n "${NAMESPACE}" "${KIND}/${NAME}" \
  meta.helm.sh/release-name="${COMPONENT}-post" \
  meta.helm.sh/release-namespace="${NAMESPACE}" --overwrite
```

Prefer `kubectl delete` only for kinds where cascade is acceptable (for example
a ConfigMap or ClusterRole with no dependents). **Do not** `kubectl delete -f`
CRD or Namespace manifests from a migration cleanup — deleting a CRD
garbage-collects every CR of that type cluster-wide (Gateway API / Inference
Extension CRDs under `agentgateway-crds` are the concrete risk), and deleting a
Namespace removes everything inside it. Those vendored CRDs carry
`helm.sh/resource-policy: keep` and `argocd.argoproj.io/sync-options:
Delete=false,Prune=false`. Helm reads `keep` from the release's stored manifest
when it uninstalls, so once a bundle carrying them has been applied, uninstalling
or pruning the release or Application that owns them leaves them in place. An
explicit `kubectl delete` is covered by neither.

## Gate on component readiness

`--readiness-hooks` emits a standalone readiness-gate chart for each component
that ships a readiness test, run as a post-component Job so the deployer blocks
on component-specific signals (e.g. GPU Operator `ClusterPolicy` state) that
Helm and Argo CD cannot assess natively. Supported with `--deployer helm`,
`argocd`, `argocd-helm`, and `terraform`; off by default.

```bash
aicr bundle --recipe recipe.yaml --readiness-hooks --output ./bundles
```

The gate Job takes its node selector and tolerations from the component's own
system scheduling values at install time, so `--set` and `--system-node-*`
place it alongside the component, as does `--dynamic` under `helm` and
`argocd-helm`. Keyless tolerations are not applied to the gate. See
[Readiness Gates](/aicr/v1.3.0/user-guide/cli-reference#readiness-gates) for each deployer.

## Split a bundle by class

`--split` renders one bundle set partitioned by the registry `class` of each
component ([ADR-018](https://github.com/NVIDIA/aicr/blob/main/docs/design/018-class-partitioned-bundles.md)).
The classes are ordered `core` &lt; `ops`: `core` is what a plain GPU pod needs
to run correctly, with its fabric and install prerequisites; `ops` is
everything else, workload platforms included. The [component
catalog](/aicr/v1.3.0/user-guide/component-catalog#components) lists every component's class.
Two owners can deploy the two classes separately, for example NVIDIA
deploying `core/` on a tenant data plane and the partner deploying `ops/`.

```bash
aicr bundle --recipe recipe.yaml --split --output ./bundles
```

The complete recipe is validated once, exactly as without `--split`, and
only then is the partition rendered: one directory per class that has an
enabled component, under a set root that is the verification unit.

```text
bundles/
├── recipe.yaml        # the full resolved recipe
├── split.yaml         # audit: recipeDigest, class order, class of every component
├── bundle-info.yaml   # build record; release paths are class-prefixed
├── checksums.txt      # covers the whole set
├── core/              # deploy.sh, README.md, NNN-<component>/ ...
└── ops/               # deploy after core
```

**Deploy and tear down in class order.** Install `core`, then `ops`; tear
down `ops` before `core`. Each class directory is an ordinary Helm
per-component bundle with its own `deploy.sh`, which does not check that
the earlier classes are installed. A class's README names the namespaces
it shares with an earlier class and the CRD groups an earlier class
installs, as far as `deploy.sh`'s CRD table knows them; its teardown must
leave those in place, because the earliest class that uses them removes
them, and it is torn down last. A dependency on a component in an earlier class is
satisfied outside the class directory, the same way a disabled component's
edge is.

**Verify the set, not a class.** The root `checksums.txt` and, with
`--attest`, the attestation cover the whole set. Run `aicr verify` at the
set root and keep the set intact: a class directory copied out of it has no
provenance of its own. `split.yaml` is audit metadata, never a trust root,
and nothing reads it at deploy time. Its `recipeDigest` identifies the
resolved recipe and, with the class map, the partition; bundle-time
overrides land in the class directories, not in `recipe.yaml`, so two sets
can share a `recipeDigest` and differ in their payload, which
`checksums.txt` identifies.

**Publication is create-only and atomic.** `--output` must name a path that
does not exist yet, under a parent that does. The set renders and verifies
in a private sibling of the destination and is published with one atomic
move, so a failure leaves no partial set behind and an existing path is
never replaced.

**What is rejected.** `--split` fails before writing anything when:

- the deployer is not `helm` (`helmfile`, `argocd`, `argocd-helm`, `flux`,
  `fleet` and `terraform` are all rejected in this release), the output is
  an OCI reference, `--output` is not given, checksums are disabled, or a
  component filter (`bundlers`) is set;
- the recipe targets OCP (`service: ocp`);
- a component has no registry entry, such as a fully specified ref in a
  hand-authored recipe; register it, for example through a `--data` registry
  entry that declares its `class`;
- a dependency points from an earlier class to a later one, including the
  opt-in `nvsentinel-preflight`, `nvsentinel-preflight-runai` and
  `nvsentinel-slinky-drainer` mixins under the shipped classification;
- every component of a declared class is disabled, by the recipe or by
  `--set <component>:enabled=false`;
- two components would emit the same Helm release name in the same
  namespace, counting the injected `-pre`, `-post` and `-readiness`
  releases across the whole set.

Without `--split` nothing changes: one bundle, every enabled component.

## Deploy the bundle

For the default `helm` deployer and an ordinary bundle, verify the bundle
before installing it:

```bash
cd bundles && aicr verify . && chmod +x deploy.sh && ./deploy.sh
```

For a bundle set generated with `--split`, verify at the set root, then run
each class's own `deploy.sh` in the order `split.yaml` lists under
`classOrder`. A set contains a directory only for classes that have an
enabled component, so read the order from the file rather than assuming
both `core/` and `ops/` exist (see
[Split a bundle by class](#split-a-bundle-by-class)). The snippet needs
`yq` v4 (mikefarah) and stops with a non-zero exit when the class list
cannot be read or a class fails:

```bash
cd bundles && aicr verify . || exit 1
classes=$(yq eval '.classOrder[]' split.yaml) && [ -n "$classes" ] \
  || { echo "cannot read classOrder from split.yaml" >&2; exit 1; }
for class in $classes; do
  (cd "$class" && chmod +x deploy.sh && ./deploy.sh) || exit 1
done
```

For the `terraform` deployer, verify in the step that generates the bundle,
then let Terraform install it:

```bash
aicr verify ./bundles && terraform -chdir=bundles init && terraform -chdir=bundles apply
```

(A bundle generated with `--terraform-child-module` is applied from the
configuration that calls it instead.)

For GitOps deployers, commit/publish the manifests per your Argo CD or Flux
workflow. Verify the bundle directory before it enters the repository or
registry; a pull-based controller reconciles on its own schedule, so a pipeline
step gates what gets published rather than what the cluster applies. For the
per-deployer gates and which of them the cluster can enforce, see
[Gating Deployment on Verification](/aicr/v1.3.0/integrator-guide/supply-chain-verification#gating-deployment-on-verification).

After deploying, confirm the cluster matches the recipe with
[`aicr validate --recipe <bundle>/recipe.yaml`](/aicr/v1.3.0/user-guide/validation#which-recipe-to-validate):
the bundle's `recipe.yaml` records the component set it deployed.