Components
A component in AICR is a registry entry pointing to a Helm chart or
Kustomize source that recipes can pull. The catalog lives in
recipes/registry.yaml;
per-component default values live under
recipes/components/<name>/.
Overlays bind a component to a cluster shape; bundlers turn that
binding into a deployer-specific artifact.
Most components need no Go code. The declarative path is one
registry entry plus a values.yaml. The legacy
pkg/component/generic.go::ComponentConfig
is marked Deprecated — it is unused in production. The live schema
is pkg/recipe/components.go::ComponentConfig.
For the recipe data model — overlays, mixins, criteria, merge order — see /aicr/v1.1.2/contributor-guide/recipes-overlays-and-mixins. This page is the contributor view for adding or changing components.
Where Does My Change Go?
Helm vs Kustomize
A component declares either helm: or kustomize: — never
both. ComponentRegistry.Validate rejects the mixed shape at load.
Kustomize limitations to know up front:
--set <key>:<path>=<value>flows through Helm value rendering only; Kustomize components silently ignore overrides.nodeScheduling.system/acceleratedpaths target Helm values; they do not apply to Kustomize sources.- The bundler runs
kustomize buildat bundle time and wraps the output astemplates/manifest.yamlinside the standard local-format folder (see Local Format and Bundle Classification for the classification rule).
Adding a Helm Component
1. Add the registry entry to recipes/registry.yaml:
2. Create recipes/components/my-operator/values.yaml with the
chart defaults you want every recipe to start from. Keep this file
minimal and widely applicable — cluster-specific tweaks belong in
values-<context>.yaml siblings referenced from an overlay.
3. Optional blocks on the registry entry:
validations:— bundle-time misconfiguration warnings (/aicr/v1.1.2/contributor-guide/validators)healthCheck.assertFile:— chainsaw conformance assertions (/aicr/v1.1.2/contributor-guide/validators)upgrades.file:— path to aComponentUpgradestransition record (/aicr/v1.1.2/contributor-guide/recipes-overlays-and-mixins)manifestFiles:— manifest YAMLs (paths relative to the recipes data root) bundled with the component whenever a recipe references it and the componentRef declares none; shipped in the injected-postlocal chart after the main release. Ref-declared lists take precedence.storageClassPaths:— where--storage-classis injectedsharedStorageClassPaths:— where--shared-storage-classis injectedpodScheduling.workload.workloadSelectorPaths— for workload-pod placementgkeCriticalPriority,hasSelfRefCRDs,manifestsUseChartCRDs— narrow service-specific quirks (see godoc onComponentConfigfor when these apply)
4. Run make bom-docs and commit the regenerated
docs/user/container-images.md in the same PR. CLAUDE.md treats this
as a hard rule whenever you change registry.yaml, a component’s
values.yaml, or any chart version pin. See BOM regeneration.
5. Run make qualify — covers tests, lint, and the recipe-resolution
suite that parses every registry entry.
Adding a Kustomize Component
No recipes/components/<name>/values.yaml is required — Kustomize
reads its inputs from the upstream source. Reminder: no --set
overrides, and nodeScheduling paths do not apply.
Schema Reference
Authoritative definitions live in
pkg/recipe/components.go.
One-liner per field:
nodeScheduling.system vs accelerated
This is the field most contributors get wrong on first PR.
system— paths into chart values for workloads that must land on management / control-plane nodes (e.g., operators, controllers, webhooks). The bundler writes the--system-node-selectorand--system-node-tolerationvalues here.accelerated— paths into chart values for workloads that must land on GPU nodes (e.g., device-plugin DaemonSets, driver-validation, DCGM exporters). The bundler writes the--accelerated-node-selectorand--accelerated-node-tolerationvalues here.
Concrete example from nvidia-dra-driver-gpu, which routes both sides:
The controller (a Deployment) lands on management nodes; the kubelet-plugin DaemonSet lands on GPU nodes — each routed by its own selector/toleration pair.
A tolerationPaths entry may appear under both tiers for a DaemonSet that
must run on every node (nfd worker.tolerations and
topologyUpdater.tolerations, aws-ebs-csi-driver node.tolerations). The
bundler writes such a path once, with the keyed --system-node-toleration
entries followed by the --accelerated-node-toleration entries, deduplicated;
keyless system entries (the tolerate-all default) are left out so they never
widen a restricted accelerated list. With no accelerated entries at all, the
path receives the system list as given. The usual opt-out ([]) and append-mode (non-empty overlay list) rules still
apply. nodeSelectorPaths must not be shared: a selector cannot mean “both
node classes”.
gpu-operator is the exception worth knowing: its accelerated block is
tolerations-only. Its operand DaemonSets have no chart/CRD nodeSelector
field — the operator self-places via its GFD/NFD deploy labels — so only
tolerationPaths: [daemonsets.tolerations] is routable there (#2474).
Wrong column = workloads land on the wrong node class. A DaemonSet
placed under system will miss GPU nodes; an operator under
accelerated will refuse to schedule on a cluster with tainted GPU
nodes only.
requireNodeSelector. By default, a component with nodeSelectorPaths
declared silently skips injection when its CLI flag is omitted — many valid
clusters have no distinct node pools, so an unset selector is often
intentional. Set nodeScheduling.system.requireNodeSelector: true (or the
accelerated counterpart) only when landing on the wrong node class silently
breaks the component, e.g. a StatefulSet whose PVC pins it to a node’s zone
and strands it there on a later reschedule (see slinky-slurm and
slurm-accounting-mariadb in registry.yaml). An overlay can still opt a
component’s paths out of the requirement with an explicit empty
nodeSelector: {} override, same as it opts out of toleration injection.
--dynamic is not an equivalent escape hatch: it is rejected on a path that
equals, contains, or is contained by a required path, since deferring any of
those to install time is the same unpinned state the requirement exists to
reject.
requireNodeSelectorIfStorageClassSet. requireNodeSelector’s
conditional counterpart, for a chart whose zone-pinning PVC only exists once
a storage class is configured, not by default. Set
nodeScheduling.system.requireNodeSelectorIfStorageClassSet: true (or the
accelerated counterpart) when the component’s chart defaults to ephemeral
storage (e.g. emptyDir), but a non-empty value at one of its
storageClassPaths or sharedStorageClassPaths paths can switch it to a PVC
at bundle time; a bare bundle of such a component with no storage class
configured stays unaffected, and --system-node-selector/
--accelerated-node-selector is required only once that PVC is actually in
play. The two flags are mutually exclusive on the same scheduling group, and
requireNodeSelectorIfStorageClassSet requires at least one
storageClassPaths or sharedStorageClassPaths entry to condition on;
ComponentRegistry.Validate rejects a registry entry that violates either
rule. --dynamic is rejected the same way as for requireNodeSelector,
regardless of whether a storage class is configured yet, since one could be
added later without rebuilding the bundle. kube-prometheus-stack is the
example in registry.yaml. Its Prometheus StatefulSet uses emptyDir until
--storage-class (or --set / an overlay override) puts a non-empty value
at its storageClassPaths entry.
valueOverrideKeys
--set <key>:<path>=<value> matches via GetByOverrideKey:
- The component
nameis checked first. - Each entry in
valueOverrideKeysis then checked.
For gpu-operator with valueOverrideKeys: [gpuoperator], both
--set gpu-operator:driver.version=... and
--set gpuoperator:driver.version=... resolve to the same component.
Pick a key that is easier to type (no hyphens) and document it in the
displayName-adjacent comments if non-obvious. Override keys are
globally unique — ComponentRegistry.Validate rejects duplicates.
Exposing Scheduling Knobs Without New Flags
A recurring first-PR instinct is to add a new CLI flag for every placement knob a chart exposes — one flag for the controller, one for the workers, one for a sidecar, and so on. Don’t. AICR already models node placement as value paths, not flags, so the existing primitives cover arbitrary chart depth without growing the CLI surface.
Two separate mechanisms are at work — keep them distinct:
Path routing. nodeScheduling is registry metadata that maps chart
value paths onto the system / accelerated node classes (see
the section above). It selects
which chart paths receive a node selector / toleration; it does not
participate in value precedence. The bundler fans --system-node-selector
/ --accelerated-node-selector (and the matching *-toleration flags)
out to every declared path, so one flag covers N workloads.
Value precedence. Once a path is targeted, its value is resolved across layers, lowest precedence first:
- Component value defaults — ship sane defaults in
recipes/components/<name>/values.yaml, or per-recipe in an overlaycomponentRefs[].valuesFile/ inlineoverrides. This is where the common case (e.g. a managed cluster’s standard node labels) works out of the box with no flags at all. - Deploy-time overrides —
--set <key>:<path>=<value>reaches any value path for last-mile deviations;--dynamic <key>:<path>defers a path to install time (it is stripped fromvalues.yamlintocluster-values.yamlfor the operator to fill in).
Merge order is base → valuesFile → overlay overrides →
--set / --dynamic, so a bundle-time override always wins over a
recipe default. The node-class flags (--system-node-selector et al.)
write into their routed paths at bundle time, alongside --set.
The narrow exception is a bundler-owned derived integration value:
when AICR renders both sides of a cross-chart contract and allowing either
side to drift would produce an invalid bundle, the bundler may enforce that
contract after ordinary overrides. The DRA/GPU Operator integration is the
worked example: when opted in, AICR merges the configured DRA eviction label into
kubeletPlugin.nodeSelector and writes the same label key to GPU Operator’s
NODE_LABEL_FOR_GPU_POD_EVICTION. This exception must remain gated on both
components being enabled, must reject dynamic declarations that could move
either managed path to install-time configuration, and must be covered across
every deployer. Ordinary workload placement still uses registry paths and must
not grow a special flag.
A bundler-owned contract that imposes cluster state must be opt-in. The DRA
eviction contract also illustrates the limit of the exception. Its selector is
an exact-match nodeSelector, so rendering it unconditionally made a node label
a precondition for the kubelet plugin to run at all — and an unlabeled node is
then silently excluded, with DESIRED=0 only when no node carries the label,
and a successful helm upgrade either way. AICR does not
own node labels, cannot verify them at bundle time, and generation is offline,
so it cannot even warn accurately about what a given cluster carries.
The rule that follows: a derived integration value may enforce consistency
between two charts AICR renders, but when it also requires state outside those
charts, the enforcement is opt-in. Absent the opt-in, AICR writes neither side
and owns neither path — no injection, no override protection, no dynamic-path
rejection. The single configuration surface that carries the state (here
--dra-eviction-node-label and its config/API/Go equivalents) doubles as the
opt-in, so there is no second boolean and no ambiguous combination. See
DRA Driver Upgrade Eviction
for the user-facing contract and issue #2469 for the reasoning.
Deciding where a knob belongs:
“When omitted, inherit from the node class” needs no special logic — a
path listed under system / accelerated is the inheritance.
Caveat: the chart must actually render the path. A nodeSelector
path only takes effect if the upstream chart template renders it. Some
charts honor only affinity / tolerations and silently drop
nodeSelector, so a declared path becomes a no-op. Verify against the
chart’s templates before adding a path. If the chart does not support
it, either pin placement through the affinity path the chart does
render (as a values.yaml default) or omit the path and document the
limitation in a registry comment. The slinky-slurm-operator entry in
registry.yaml
is a worked example of the latter.
deploymentOrder
RecipeResult.DeploymentOrder is derived, not authored.
TopologicalSort in pkg/recipe/metadata.go orders components
by componentRefs[].dependencyRefs declared in the overlay. Components
with no dependency between them are emitted in alphabetical order
(Kahn’s algorithm over a sorted queue), not in the order they are
listed in componentRefs. Express
ordering by declaring dependencyRefs on the dependent component, not
by writing a separate deploymentOrder block.
Local Format and Bundle Classification
The bundler emits a uniform NNN-<component>/ layout via
pkg/bundler/deployer/localformat.
Classification (single source of truth in localformat.classify):
If both helm and kustomize fields are populated, Validate
rejects the registry entry — there is no precedence rule because the
shape is invalid. manifestFiles are added post-chart; preManifestFiles
ship at sync-wave N-1 (e.g., a Namespace with PSS labels the chart
pods depend on).
Deployers
AICR ships six output adapters in
pkg/bundler/deployer/:
helm, helmfile, argocd, argocd-helm, flux, terraform, fleet. Each calls
localformat.Write() and then layers its own orchestration files
(deploy.sh, helmfile.yaml, Argo Application CRs, Flux
HelmReleases, Terraform module calls, Fleet fleet.yaml files with a GitRepo or HelmOps). Components do not need to be deployer-aware —
the bundler renders per-deployer from one component definition.
Deployment ordering
One model, three projections. Every deployer derives ordering from
the same source: each component’s declared dependencyRefs. What
differs is how faithfully a deployer’s native mechanism can express
that dependency graph. For the concurrent deployers, components with no
path between them are independent and roll out concurrently, while a
real dependency gates (the dependent waits for its dependency to be
healthy). The NNN-<name>/ folder numbers reflect DeploymentOrder
(a topological serialization) for readability, and drive the two
deliberately linear paths — the helm deploy.sh and --serial mode —
which install strictly in that order regardless of tiers.
pkg/recipe exposes the graph two ways, and deployers pick whichever
fits their mechanism:
TopologicalSort→ a flatDeploymentOrder(one valid serialization).ComponentRefsTopologicalLevels→ dependency-depth tiers, where a tier holds exactly the components whose dependencies are all satisfied by earlier tiers (so a tier’s members are mutually independent).
flux — the exact DAG. Flux’s dependsOn is a native dependency
graph, so the flux deployer projects each component’s declared
dependencyRefs directly onto it: a HelmRelease’s dependsOn names
exactly its dependencies’ terminal releases, nothing more. This is the
most faithful and most parallel rendering — a component waits only for
what it actually needs — and it reads naturally to flux users because
dependsOn mirrors dependencyRefs one-for-one. See declaredDependsOn.
terraform — the exact DAG, on the module call. Terraform’s
depends_on is a native dependency graph, so the terraform deployer
projects the declared edges onto it the same way flux does, and the two
deployers emit the same edge set for the same recipe. It differs in
where the edge is attached: each component is one module call, and
the edge sits on the call, so it orders everything the module contains
rather than one resource inside it. A component’s -pre, -post and
-readiness folders become slots on that same module and chain inside
it, so depends_on = [module.gpu_operator] covers the component’s gate
without naming it — the claim the module boundary is there to make. The
slots are assigned by the same -pre/-post/-readiness suffix test
argocd’s waveForFolder uses. See buildComponents and phaseOf.
The readiness slot hardcodes wait rather than reading the bundle-wide
variable: an async component (see deployer.ComponentOverrideFor) may
skip waiting on its own workloads, never on the gate that asserts it came
up. The gate Job is a helm hook, so the release blocks on it without
wait_for_jobs, and its timeout is the gate’s own deadline plus the
buffer deploy.sh adds, not var.timeout.
argocd / argocd-helm — tiers as sync-wave bands. Argo CD’s
sync-wave is a single integer per Application. Applications sharing a
wave sync together, and Argo advances to a higher wave only once the
current one is healthy — ordered bands, a partial order rather than a
total one. Because a component gets one integer, it cannot express “wait
for A but not B” when A and B sit at the same depth, so the DAG is
approximated by tiers: wave = tier*4 + phase, where the
per-component phase orders -pre → primary → -post → -readiness.
The stride-4 band width keeps consecutive tiers disjoint, so a tier’s
readiness gate still blocks the next tier. The cost of the coarser
banding is mild over-constraint (a component waits for its whole prior
tier, not just its own dependencies). See waveForFolder.
helmfile — tiers as sequential sub-helmfiles. Emits one
level-N.yaml sub-helmfile per tier, processed in sequence so each
tier’s CRDs land in the cluster’s REST mapper before the next tier’s
plan is rendered (issue #914). Within a sub-helmfile, needs: edges
chain only a component’s own -pre → primary → -post releases;
independent components in the tier carry no edge, so helmfile applies
them concurrently. Cross-tier ordering is the sub-helmfile sequence, not
per-release needs: — the same tier approximation as argocd. See
buildHelmfile.
helm — deliberately serial. The deploy.sh installs one component
at a time in DeploymentOrder, trading parallelism for the simplest
possible shell. The readiness gates already sequence it correctly.
Disabling parallelism (--serial)
aicr bundle --serial forces every deployer to install components
strictly one at a time in DeploymentOrder, an escape hatch for
reproducing the pre-parallelism ordering or bisecting a misbehaving
rollout. It affects the five concurrent deployers: argocd and
argocd-helm fall back to a linear sync-wave per folder, flux chains each
HelmRelease dependsOn to the previous component instead of
projecting the DAG, helmfile chains every release to its predecessor
via needs: (one linear apply chain) instead of only within a
component, and terraform chains every module call via depends_on. The
helm deploy.sh is already serial, so the flag is a no-op for it. Off
by default.
See /aicr/v1.1.2/contributor-guide/architecture-overview for the deployer matrix.
BOM Regeneration
docs/user/container-images.md is rendered fresh from each Helm
chart’s actual templates by make bom-docs. Run it and commit the
regenerated file in the same PR whenever you:
- Add or remove a component
- Bump a chart version (in
registry.yaml, an overlay, or a mixin) - Change a
values.yamlin a way that affects which images render (image-repo override, subchart enable/disable, etc.)
The BOM’s version column and component set are gated at PR time
(TestCommittedBOMVersionsMatchRegistry plus the bom-freshness
merge-gate job), so a missed regen after a version or component-set
change fails CI. Not gated at PR time is rendered-image drift — a
chart pulling a new image with no pin change on our side; make bom-check (a full re-render comparison) is its opt-in blocking
check, and the weekly BOM-refresh workflow auto-detects it and opens a
PR. Still run make bom-docs locally on any chart-touching change.
Boundary: Components Are Metadata
A component entry describes what to deploy and where its values
land. Components do not carry apply, wait, uninstall, rollback,
or readiness-polling logic — those concerns belong to the deployer
that consumes the bundle. If you find yourself writing custom apply
code inside the bundler or under pkg/component/, you are on the
wrong side of the boundary — see
/aicr/v1.1.2/contributor-guide/architecture-overview “What AICR Is Not”.
See Also
- /aicr/v1.1.2/contributor-guide/recipes-overlays-and-mixins — overlays, mixins, criteria, the recipe data model
- /aicr/v1.1.2/contributor-guide/validators — bundle-time component validation checks
- /aicr/v1.1.2/contributor-guide/validators — chainsaw health checks and validator runner
- /aicr/v1.1.2/contributor-guide/architecture-overview — contributor index and architectural boundary
- integrator/recipe-development.md — end-user recipe authoring
- user/component-catalog.md — end-user component catalog
pkg/recipe/components.go—ComponentConfigsource of truthrecipes/registry.yaml— live component catalog