Recipes, Overlays, and Mixins

The recipe data layer is the rule-based engine that turns a Criteria query ({service, accelerator, intent, os, platform, nodes}) into a resolved RecipeResult — the merged spec, component refs, deployment order, and validation phases that aicr bundle consumes.

This page covers everything related to AICR recipes for contributors: the three layers that contribute data (registry, overlay, mixin), the on-disk schemas for each, the resolver’s merge algorithm, and the invariants the resolver enforces. End-user recipe authoring lives in recipe-development.md; this page is for contributors changing recipe content or extending the resolver in pkg/recipe.

Where does my change go? Most changes hit exactly one of three files. Skim Decision matrix before editing — picking the wrong layer leaks defaults across recipes or duplicates content across overlays.

Layered Model

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│   Registry   │  │    Mixin     │  │   Overlay    │
│ recipes/     │  │ recipes/     │  │ recipes/     │
│ registry.yaml│  │ mixins/*.yaml│  │ overlays/    │
│              │  │              │  │              │
│ Component    │  │ Composable   │  │ Criteria-    │
│ catalog +    │  │ fragment     │  │ matched      │
│ defaults     │  │ (constraints │  │ recipe with  │
│ (chart, ns,  │  │ + componentRefs)│ │ spec.base   │
│ scheduling)  │  │              │  │ inheritance  │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       └─────────────────┴─────────────────┘
                         │
                         ▼
              ┌────────────────────┐
              │ RecipeResult       │
              │ (merged, ordered,  │
              │  validated)        │
              └────────────────────┘

Resolution: the resolver loads the base spec (overlays/base.yaml) as the merge seed, then merges each matching overlay’s inheritance chain on top (base → … → leaf), then applies the leaf’s mixins, then finally injects registry defaults for any component field the chain left unset. Per-component values files (recipes/components/<name>/) are pulled in at bundle time, not at recipe resolution.

Decision Matrix

Rule of thumb: a change targeting all recipes goes in registry; a change targeting one cluster shape goes in an overlay; a change shared by ≥ 2 overlays as an opt-in fragment goes in a mixin.

Registry (recipes/registry.yaml)

The registry is the component catalog. Each entry declares a chart or kustomization the resolver can reference and supplies defaults the resolver injects into any ComponentRef that leaves the field unset.

Top-level schema (ComponentRegistry):

1
apiVersion: aicr.run/v1alpha2
2
kind: ComponentRegistry
3
components:
4
  - name: <component-id>
5
    ...

ComponentConfig fields (see pkg/recipe/components.go):

HelmConfig: defaultRepository, defaultChart, defaultVersion, defaultNamespace. KustomizeConfig: defaultSource, defaultPath, defaultTag. A component must have either helm or kustomize, not both.

pkg/component/generic.go carries a ComponentConfig marked Deprecated: — that is a separate, unused-in-production legacy type; the live ComponentConfig is the one in pkg/recipe/components.go.

Defaults flow into a ComponentRef only when the field is empty — see applyRegistryDefaults below. The registry default (helm.defaultVersion, or kustomize.defaultTag for Kustomize components) is the single source of truth for a component’s version; base, overlay, and mixin componentRefs alike carry no version pins except exemption-declared divergences — see Version pinning is single-source.

Overlay (recipes/overlays/)

An overlay is a RecipeMetadata document with a spec.criteria block that selects it for matching queries. Overlays live in recipes/overlays/ and inherit single-parent via spec.base.

1
kind: RecipeMetadata
2
apiVersion: aicr.run/v1alpha2
3
metadata:
4
  name: gb200-eks-ubuntu-training
5
spec:
6
  base: gb200-eks-training       # inheritance chain root → leaf
7
  mixins:                        # composed AFTER inheritance merge
8
    - os-ubuntu
9
  criteria:
10
    service: eks                 # query → overlay selection
11
    accelerator: gb200
12
    os: ubuntu
13
    intent: training
14
    # platform: kubeflow         # optional 6th dimension
15
  constraints:                   # OS/K8s/GPU/SystemD constraints
16
    - name: K8s.server.version
17
      value: ">= 1.34"
18
  componentRefs: []              # overrides on inherited components
19
  validation:                    # per-phase validation config
20
    readiness:   { ... }
21
    deployment:  { ... }
22
    performance: { ... }
23
    conformance: { ... }

Criteria fields (see pkg/recipe/criteria.go type Criteria):

--data overlays may contribute additional values via the criteria registry — Has(FieldX, ...) is consulted when a value misses the fast-path switch in Parse<X>. Adding a new value to a Go enum (e.g., a new accelerator) is multi-file work; audit CriteriaAccelerator* callers as listed in CLAUDE.md before merging.

Specificity. Each criteria carries a specificity score equal to the count of non-any, non-empty fields. The current Specificity() in criteria.go counts six fields: service, accelerator, intent, os, platform, nodes. Overlays are sorted by specificity ascending, so less-specific overlays merge first.

Matching is asymmetric. Recipe-side any is a wildcard (matches anything in the query); query-side any is not a wildcard (matches only recipe-side any). A generic query never resolves to a hardware-specific recipe. See MatchesCriteriaField in criteria.go.

Inheritance. spec.base walks a single-parent chain from leaf → … → base (the root spec, held separately on the metadata store). Cycles are detected at catalog load. Per-field merge: constraints merge by name (later wins on same name; new appended); componentRefs merge by name field-by-field; criteria are not inherited (each recipe declares its own).

Leaf. A leaf is the most specific overlay in a chain — the terminal node carrying fully-qualified criteria (every relevant dimension set, e.g. service + accelerator + os + intent + platform) that an end-user query actually resolves to. A leaf usually adds little of its own (often componentRefs: []); its job is to bind one inheritance chain plus its mixins under a single criteria fingerprint. “Base → … → leaf” throughout this page refers to walking from the root spec down to this node. Leaf is a role, not a distinct kind — every overlay is a RecipeMetadata; “leaf” just names the ones at the end of a chain.

Configuration Profile Contract

RecipeMetadata.Spec.Profile declares one overlay-scoped enum for qualified configuration ownership modes. A declaration requires recipe apiVersion aicr.run/v1alpha3; that version without a declaration, or a declaration on the legacy version, is rejected. Profile-version metadata and recipe artifacts are strictly decoded so an unknown field cannot silently disappear.

The core ProfileValue contract is closed to advertiser, constraints, and componentRefs{name,overrides}. It rejects valuesFile, component identity/deployment fields, root overrides.enabled, literal dotted keys, and nested empty maps. The advertiser field accepts exactly one non-empty value, external (validated against pkg/allocpolicy, the canonical append-only #1327 descriptor). A profile that owns advertisement — a declared advertiser: external, or explicit ownership of a non-synthetic policy-selector path (devicePlugin.enabled, DRA resources.gpus.enabled / gpuResourcesEnabledOverride) — triggers the recomputed policy closure: every enabled descriptor component’s selector paths plus its synthetic enabled join the effective lock set (RecipeResult.EffectiveLockSet()), which the bundle/mirror lock, the argocd-helm guard, and the hydrating gate all consume. The closure is recomputed at every artifact boundary and never persisted in ownedPaths. The hydrating gate additionally runs the shared tuple-coherence rules (external + devicePlugin.enabled=true and external

• DRA gpus.enabled=true both reject). Profiles that do not own advertisement (the AKS shape) trigger no closure and leave allocation-policy keys on the bundle-time WARN semantics. Value-bearing ownership is limited to Helm components; Kustomize components do not consume values overrides and support only a valueless reference for presence locking.

Resolution enforces these invariants:

1. Before snapshot filtering, collect declarations from every matching candidate chain. Deduplicate one declaring source reached through multiple chains; reject independently authored declarations.
2. Snapshot filtering may not remove the declaring source and fall back to an unprofiled composition.
3. Apply an explicit name=value selection or the required default after overlays and mixins. Every referenced component across all values must already be enabled in the surviving composition.
4. Every value has the same flattened override-path set. Apply the selected fragment at the highest recipe precedence and reject constraint-name collisions.
5. Evaluate selected profile constraints fail closed. A missing reading has a distinct invalid-request diagnostic; other evaluator failures propagate.
6. Stamp the result aicr.run/v1alpha3 and persist metadata.selectedProfile. Its sorted ownedPaths is the declaration-wide path union plus synthetic enabled for each referenced component.

The shared raw-artifact gate checks version/profile coupling and hydrates profile-owned values. Bundle and mirror compare final candidate state with the hydrated recipe before creating output. Exact, ancestor, or descendant dynamic paths reject unconditionally; static writes reject only when the effective three-state observation (present bytes, absent, or blocked) diverges. Argocd-helm emits the corresponding structural template-time guard. Other deployers have no supported install-time value surface.

Unprofiled compositions retain the legacy apiVersion and byte shape. Generation-side driver auto-detection skips a path owned by the selected profile. Evidence projection accepts profiled artifacts: the selected value joins the evidence path name and the corroboration tab as a lowercase -<name>-<value> segment (shared helper attestation.ProfileSegment), pointers record the selection, and the repo evidence gate recomputes each pointer’s digest with its recorded selection. The TestGrid coordinate is deliberately unsuffixed — its digest-bound build ID already partitions per value.

Mixin Composition

Inheritance is single-parent, which means cross-cutting concerns (OS constraints, platform components) would otherwise duplicate across every leaf. Mixins are composable fragments referenced via spec.mixins. They live in recipes/mixins/ and use kind RecipeMixin.

1
# recipes/mixins/os-ubuntu.yaml
2
kind: RecipeMixin
3
apiVersion: aicr.run/v1alpha2
4
metadata:
5
  name: os-ubuntu
6
spec:
7
  constraints:
8
    - name: OS.release.ID
9
      value: ubuntu
10
    - name: OS.release.VERSION_ID
11
      value: "24.04"
12
  componentRefs: []   # optional

Mixin files currently in the tree: os-ubuntu, os-talos, platform-inference, platform-kubeflow.

Mixin rules:

• A mixin carries only constraints and componentRefs. Setting criteria, base, mixins, or validation is rejected at load.
• Resolution order: base chain merged first, then mixins applied to the merged result. A leaf adopts a mixin by listing its file basename in spec.mixins.
• Mixin componentRefs are restricted to additive merges via mixinComponentRefSafeForMerge (see pkg/recipe/metadata_store.go). A mixin componentRef may only set name, namespace, manifestFiles, preManifestFiles. Setting any of chart, type, source, version, tag, path, valuesFile, overrides, patches, dependencyRefs, cleanup, expectedResources, healthCheckAsserts is rejected at compose time — those fields silently override the chain’s chosen chart, so the resolver names the offending field and refuses to merge (see ADR-005 “Silent constraint override” mitigation).
• When a snapshot evaluator is wired in, mixin constraints are evaluated against it after merging; failure invalidates the entire composed candidate. In plain query mode mixin constraints are merged but not evaluated.

Criteria Wildcard Overlays

Some overlays apply across an entire criteria dimension without being referenced via spec.base or spec.mixins. The resolver picks them up automatically because FindMatchingOverlays returns all maximal matches, not just the most specific one. Two wildcard patterns in the tree today: gb200-any.yaml (matches service: any) and monitoring-hpa.yaml (matches intent: any).

1
# recipes/overlays/gb200-any.yaml
2
spec:
3
  base: base
4
  criteria:
5
    service: any        # wildcard — matches eks, oke, gke, ...
6
    accelerator: gb200
7
  validation:
8
    deployment:
9
      constraints:
10
        - name: Deployment.gpu-operator.version
11
          value: ">= v25.10.0"

For a query {service: eks, accelerator: gb200, intent: training}, the resolver returns three independent maximal leaves — gb200-eks-training (matched by explicit criteria), gb200-any (matched by service: any), and monitoring-hpa (matched by intent: any). Each leaf’s inheritance chain is resolved separately and merged onto the base spec in specificity order.

Maximal-leaf filter. filterToMaximalLeaves (in metadata_store.go) drops any match that is a transitive spec.base ancestor of another match — ancestors re-enter the output via chain resolution, so keeping them as separate matches would double-count their contributions. Independent leaves on unrelated chains (wildcard + explicit) are kept; one is not an ancestor of the other.

When to use a wildcard overlay vs a mixin:

Precedence. Leaves merge in specificity-ascending order, so a service-specific leaf overrides the wildcard on same-named constraints. spec.validation.<phase> blocks merge per-field: checks and constraints union (nil = inherit, [] = clear, non-empty = union); nodeSelection and infrastructure are wholesale-replace. Don’t carry per-fabric values in a wildcard (NCCL bandwidth thresholds differ per service); reserve wildcards for content genuinely uniform across the wildcard dimension.

Merge Algorithm

The resolver lives in pkg/recipe/metadata_store.go. The merge proceeds in this temporal order:

base chain (root → leaf) → mixins → selected profile → registry defaults → CLI/API --set

The base inheritance chain is merged first (root → leaf, later ancestors override earlier ones on same-named entries). Mixins are applied after the chain, not before it: mergeMixins appends each referenced mixin’s constraints and componentRefs. A mixin constraint whose name already exists in the chain (or in another mixin) is rejected — constraints have no merge semantic, so a name collision is treated as an unambiguous conflict, not resolved by last-wins precedence. Registry defaults then fill any still-empty componentRef fields, and CLI/API --set overrides win last.

Implementation notes:

1. Seed. initBaseMergedSpec() clones s.Base (parsed from overlays/base.yaml) into the merge target. The base spec is held separately on the metadata store; it is not an overlay candidate in FindMatchingOverlays.
2. Chain merge. For each maximal leaf, the inheritance chain is walked root → leaf and mergedSpec.Merge(&recipe.Spec) is called for each. Same-named constraints/componentRefs override; new entries append.
3. Mixin merge. mergeMixins(mergedSpec) walks spec.mixins on the leaf, loads each from recipes/mixins/, and appends. mixinComponentRefSafeForMerge rejects mixin componentRefs that touch identity/sourcing fields.
4. Profile specialization. When the pre-filter composition found one declaration and that source survived filtering, apply the selected value after the chain and mixins. The fragment supersedes earlier assignments only on its closed override surface. Unprofiled resolution skips this step.
5. Registry defaults. applyRegistryDefaults(provider, refs) fills in chart/version/namespace/source/tag/path defaults for any ComponentRef field still empty after the chain merge. Failure to load the registry is propagated, not swallowed — partial refs would fail downstream far from the root cause.
6. Topological sort. TopologicalSort() orders components by dependencyRefs for the final DeploymentOrder. Cycles produce ErrCodeInvalidRequest. Components disabled via overrides.enabled: false (ComponentRef.IsEnabled()) are excluded from the ordering, and an edge pointing at a declared-but-disabled component is treated as satisfied (assumed provided externally) so it does not trigger a false cycle; an edge to an undeclared component still surfaces as ErrCodeInvalidRequest. TopologicalLevels() / ComponentRefsTopologicalLevels() apply the same filter.

Deep-copy semantics. deepMergeMap (metadata.go) recurses into nested map[string]any. Non-map values (scalars and []any) are deep-copied via serializer.DeepCopyAny so dst never aliases src’s slice values. This matters: copying []any by reference during overlay merge would let a downstream mutation (e.g., bundler appending a toleration) leak back into the cached source map and corrupt subsequent queries. The CLAUDE.md anti-patterns list calls this out — any new helper that touches overlay-derived maps must follow the same rule.

Criteria Coverage Post-Condition

Resolution enforces a post-condition (issue #1542): every criteria dimension the caller explicitly states — service, accelerator, intent, os, platform — must be honored by at least one overlay in the final applied set (appliedOverlays), or resolution fails with ErrCodeInvalidRequest instead of silently returning a recipe that disregards a stated value. verifyCriteriaCoverage (pkg/recipe/coverage.go) runs last in both build paths, against the final applied set:

• In BuildRecipeResult, after mergeOverlayChains and mergeMixins — ancestor-supplied coverage counts, but mixins carry no criteria and cannot contribute coverage.
• In BuildRecipeResultWithEvaluator, after per-overlay constraint exclusion and the mixin-constraint-failure rebuild — a dimension whose only coverage came from a constraint-excluded overlay is still uncovered. The error’s context carries excludedOverlays / constraintWarnings when constraint exclusion occurred during resolution, so the caller has that context even though the exclusion is not always what caused this specific dimension to go uncovered.

The error’s uncovered context entries (dimension, requestedValue, validCompletions) are computed from the maximal set of overlays that carry the requested value without conflicting with any other stated dimension — see completionTuplesFor / minimalTuples. nodes is deliberately excluded from coverageDimensions: no overlay gates on node count, so covering it would reject every --nodes query. It remains a matching dimension (present in Criteria.Matches) but carries no coverage guarantee — it is advisory.

Composition with the OS guard. requireOSIfNeeded (the joint service+accelerator OS gate) is a separate, pre-existing check and runs first, before the merge. It is not subsumed by the coverage post-condition: coverage is satisfied when service and accelerator are each honored by some overlay independently, while the OS guard demands one overlay carry both service and accelerator together before it will consider the OS-agnostic tier served. Both checks apply; a request can trip either one independently.

Evaluator error classification is fail-closed. During constraint evaluation on the snapshot-driven path, ErrCodeNotFound (the evaluator’s designed signal for “measurement absent from snapshot”) is the only error code that degrades gracefully into overlay exclusion. Every other error code returned by the evaluator propagates as a hard resolution failure (see isNotFoundEvalError in pkg/recipe/coverage.go, consumed at both call sites in metadata_store.go) — a malformed constraint expression or an evaluator bug must not be swallowed as a quiet exclusion. ConstraintEvaluatorFunc returns a plain error, so both fail-closed branches (evaluateOverlayConstraints, evaluateMixinConstraints) pass a non-NotFound error through aicrerrors.PropagateOrWrap(..., ErrCodeInternal, ...) before returning it — an evaluator that hasn’t adopted pkg/errors still surfaces a coded error instead of an uncoded 500 at the server layer.

The engine stays strict; the CLI’s snapshot path relaxes derived-only failures. Everything above describes pkg/recipe’s behavior, which never relaxes the post-condition — a coverage failure there is always terminal. The CLI’s --snapshot flow (pkg/cli/query.go) layers a caller-side retry on top: service, accelerator, and os can be derived from the snapshot fingerprint rather than stated by the user (intent and platform are always user-stated — the fingerprint never derives them). If a coverage error’s uncovered dimensions are all fingerprint-derived (none came from --config or a CLI flag), the CLI clears those dimensions to unstated and retries resolution once, logging a warning per relaxed dimension; if any uncovered dimension was user-stated, the error still propagates unchanged. This lets an overlay tree that is deliberately agnostic to a dimension (e.g. Kind’s OS-agnostic overlays) tolerate a snapshot that still reports a concrete value for it, without weakening the post-condition for anyone who asked for that dimension explicitly.

Determinism

Recipe output is reproducible: same inputs → same bytes. The data layer enforces this via two rules.

Use serializer.MarshalYAMLDeterministic for any output that feeds a digest, signature, OCI manifest, or fingerprint. yaml.v3 walks Go maps in randomized order, so two consecutive marshals of the same map[string]any produce different byte sequences. Plain yaml.Marshal is fine for human-readable scratch output but is a correctness bug anywhere a downstream consumer hashes the bytes.

Per-dimension ordered lists, not unordered maps. RecipeResult fields like appliedOverlays, componentRefs, deploymentOrder, and the per-dimension fingerprint diff are ordered slices, not maps, so iteration is deterministic.

Recipe Store Immutability

The metadata store is read-only after init. LoadMetadataStoreFor(dp) returns a sync.Once-cached *MetadataStore per DataProvider identity, so concurrent recipe builds against the same provider share the store without locks. Per-request mutations (chain resolution, constraint evaluation, registry defaulting) happen on clones, never on the cached spec.

Deferred registration. pendingRegistryEntry stages each overlay’s criteria for the per-provider criteria registry before registration. The actual Register(field, value, origin) calls only fire after every overlay parses cleanly, the base recipe is present, and dependency validation passes. Partial catalog loads never leak into the registry; a malformed overlay does not poison criteria validation for the next process.

Eviction. EvictCachedStore(provider) and EvictCachedRegistry(provider) drop a single provider’s cache entry without disturbing other providers. Use after rewriting a --data overlay on disk.

Observable RecipeResult Surfaces

RecipeResult (in pkg/recipe/metadata.go) is the resolver’s externally-visible product. Fields beyond ComponentRefs and DeploymentOrder that contributors should be aware of:

ComponentRef extras beyond the chart-identity fields:

Adding a Recipe

1. Decide registry vs overlay vs mixin (decision matrix).
2. Write the YAML in the correct directory. For an overlay, set spec.base to the most specific shared ancestor and let the chain carry shared constraints; only declare what differs.
3. Ship the chainsaw health check (registry entries only). Every new component in recipes/registry.yaml MUST declare healthCheck.assertFile pointing at recipes/checks/<name>/health-check.yaml, and that file MUST use only the read-only assert / error operation allowlist (no script, apply, wait, command, etc. — see pkg/chainsaw/allowlist.go). The contract is enforced at PR time by pkg/recipe.TestComponentRegistry_RequiresHealthCheck and pkg/chainsaw.TestValidateTestReadOnly_RegistryContent — both gate make qualify. See #1223 and the chainsaw health check section in /aicr/contributor-guide/validators for the assertion patterns currently in use (DaemonSet numberReady == desiredNumberScheduled, Deployment Available=True, CRD Established=True).
4. Run make bom-docs and commit docs/user/container-images.md if your change touches registry.yaml, a component’s values.yaml, or a chart version pin (see BOM regeneration).
5. Unit tests. make test runs the recipe-resolution suite — pkg/recipe/yaml_test.go (static catalog: parse, refs, enum values, inheritance depth, no cycles) and pkg/recipe/metadata_test.go (runtime merge, topological sort). Both gate make qualify. If your change adds a registry entry, a new overlay file, or a mixin, the static suite typically picks it up without new test code.
6. Integration validation. For a new chart pin, run make qualify and let the e2e pipeline render the bundle. KWOK simulated clusters (make kwok-e2e RECIPE=<name>) catch most resolution regressions without GPU hardware.

BOM Regeneration

docs/user/container-images.md is auto-generated from the actual rendered Helm templates of every chart referenced by the registry. It is regenerated by make bom-docs.

Run make bom-docs and commit the regenerated docs/user/container-images.md in the same PR whenever you:

• Add or remove a component in recipes/registry.yaml
• Bump a chart version — normally the registry default; overlay/mixin pins exist only as declared exemptions, and bumping one changes the BOM’s variants table
• Modify a component’s values.yaml in a way that changes which images render (image repo override, subchart enable/disable, etc.)

The regen can also surface drift from upstream chart updates — when a chart bumps an image inside its own templates without a registry pin change on our side. That drift will appear in the BOM diff whether you expected it or not.

Version freshness is gated; image drift is not

TestCommittedBOMVersionsMatchRegistry (tools/bom/freshness_test.go, run by make test → make qualify) checks the committed doc’s version column against the registry pins with no Helm rendering, so a chart-pin bump that forgets make bom-docs fails CI. The check treats the doc as an exact registry projection: it is bidirectional (a component added to the registry without a doc row, or a doc row left behind after a component is removed, both fail), rejects duplicate rows, and compares every row — pinned components by their effective type (Helm defaultVersion or Kustomize defaultTag) and unpinned components against the — sentinel, so a fabricated version on an otherwise unpinned row cannot slip through. Because a docs-only PR that edits the committed BOM skips the full tests job, the bom-freshness job in .github/workflows/merge-gate.yaml runs this same test whenever docs/user/container-images.md or recipes/registry.yaml change, so the gate holds for docs-only edits too.

TestCommittedBOMVariantsMatchRecipePins (same file, same jobs) gates the doc’s Version variants table the same way: it derives every explicit base/overlay/mixin Helm pin that differs from its registry default and compares bidirectionally — a new divergent pin without a variant row fails, and a stale row without a backing pin fails. Variant discovery reads only the recipe data (the pins are the source facts); it has no dependency on the version-pin guard’s exemption policy, which decides whether a divergence is allowed, not what deploys.

Neither gate catches upstream image drift — a chart bumping an image inside its own templates without a pin change on our side. Full make bom-check verifies that too by re-rendering, but it is opt-in only — not wired into make qualify, make lint, or the PR gate. So you still must run make bom-docs after a values change.

Version pinning is single-source

The BOM renders each chart at its registry defaultVersion, but at resolution the registry default is only a fallback: a componentRef that sets version (Helm) or tag (Kustomize) in base.yaml, an overlay, or a mixin overrides it. Declared divergent pins are represented truthfully by the BOM’s Version-variants table (#1611), so a default row and a declared variant can coexist. The residual fiction is the sole-consumer case: a divergence that leaves the registry default with zero consumers makes the default row advertise a version no recipe installs (issue #1424; #1418’s aws-efa bug).

The registry default (helm.defaultVersion / kustomize.defaultTag) is therefore the single source of truth for a component’s version, and base/overlay/mixin componentRefs carry no version pins apart from declared divergences (#1616): resolution falls back to the registry default, so a version bump edits registry.yaml in one place and no overlay needs touching. TestOverlayVersionPinsMatchRegistry (pkg/recipe/version_pin_guard_test.go, run by make test → make qualify) fails on a non-exempted pin whenever the component has a matching registry default: a pin that diverges from it is undeclared drift, and a pin that equals it is redundant — it doubles bump churn and shields the overlay from an external registry’s default override. (Refs outside the registry, or whose matching default is empty, are out of the guard’s scope; make bom-pinning-check separately requires every embedded chart-bearing Helm component to declare defaultVersion — manifest-only components have no chart version to pin.) If an overlay must legitimately run a different chart version (e.g. a platform validated against an older chart), pin it and add an entry to versionPinExemptions with a justification — a declared divergence is not a silent one. Only Helm version divergences are exemptable today; a Kustomize tag exemption is rejected until the BOM variants pipeline can represent it. Do not exempt a component whose only consumer diverges; that reinstates the exact fiction the guard prevents.

Common Pitfalls

• Skipping make bom-docs after a values change that alters rendered images. A stale version column now fails CI (TestCommittedBOMVersionsMatchRegistry), but image drift from a values change without a pin bump does not surface in qualify — the BOM goes stale silently. See Version freshness is gated; image drift is not.
• Mutating in place during merge. Overlay-derived map[string]any and []any must be deep-copied, not aliased. deepMergeMap does this for you; a bespoke helper that recurses into maps but copies []any by reference will alias and corrupt the cached source map.
• Plain yaml.Marshal on output that feeds a digest. Use serializer.MarshalYAMLDeterministic for any byte sequence a downstream consumer hashes (evidence predicate body, OCI manifest, signature input, fingerprint).
• Adding a new criteria value to the Go enum but missing call sites. A new accelerator, OS, intent, or platform value is enumerated in many files — the criteria registry, OpenAPI spec, every docs page that lists current values, issue templates, the Specificity() helper. Start from the Go type in criteria.go and follow the audit list in CLAUDE.md.
• Setting identity fields in a mixin componentRef when overriding an inherited component. A mixin may not override chart, version, valuesFile, etc. on a component the inheritance chain already carries — the resolver rejects with the offending field name. Move chart-changing logic to an overlay. (A mixin may still introduce a new component with these fields.)
• Pinning a component version/tag in a recipe ref. The registry default (helm.defaultVersion / kustomize.defaultTag) is the single source of truth and base/overlay/mixin refs carry no non-exempted version/tag pins; a bump edits registry.yaml only. TestOverlayVersionPinsMatchRegistry fails on a non-exempted pin with a matching registry default — divergent (undeclared drift; most dangerously, when the diverging overlay is the component’s sole consumer the BOM’s default row advertises a version no recipe installs) or default-equal (redundant) — see Version pinning is single-source.
• Assuming the cluster fingerprint is trustworthy. The fingerprint block persisted in aicr snapshot output is advisory; trust-bearing consumers recompute via fingerprint.FromMeasurements(...) before acting. See the collector docs and ADR-007 for details.
• Adding a collector subtype or closed-space key without updating the constraint path catalog. pkg/measurement/catalog.go enumerates every addressable path, and recipe loading rejects anything it cannot address (#1783). Each subtype declares two independent key spaces, because extraction reads them from different places: {Type}.{Subtype}.{Key} resolves against Subtype.Data (scalar), and {Type}.{Subtype}[<selector>].{Key} resolves against ItemEntry.Data then ItemEntry.Context (item). Register a new key in whichever space emits it — an item-only subtype needs its keys in the item space, or every selector path through it is rejected. An entry is required for a new subtype (unless the Type is open-subtype) and for a new key in a closed space; a new key in an open space needs nothing. A missing entry does not weaken a check — it makes a legitimate constraint path fail at load for whoever first writes it. Declare a key space open when it is not provably fixed (image names, sysctl paths, label keys). Note that ItemEntry.Context keys are addressable and belong in the item space; only Subtype.Context is never addressable, so do not list those.

See Also

• recipe-development.md — end-user recipe authoring guide
• /aicr/contributor-guide/components — adding a component to the registry
• /aicr/contributor-guide/validators — adding bundle-time component validation checks
• /aicr/contributor-guide/validators — adding a validator check or health check
• ADR-005 — overlay refactoring rationale (mixin composition, maximal-leaf resolver, wildcard overlays)
• ADR-007 — fingerprint, evidence bundle, verification
• pkg/recipe godoc — implementation
• api/aicr/v1/server.yaml — recipe API contract and criteria enums