> This page is for AI Cluster Runtime, version Latest · v1.0.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v1.0.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt > - v1.0.0: https://docs.nvidia.com/aicr/v1.0.0/llms.txt > - v0.22.0: https://docs.nvidia.com/aicr/v0.22.0/llms.txt > - v0.21.1: https://docs.nvidia.com/aicr/v0.21.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server. # Validator Development Guide AICR has **four** distinct validation surfaces. Picking the wrong one is the single most common source of wasted PRs. Read the table first, then jump to the matching section. The rest of this page is the contributor view for all four. | Surface | When it runs | Where it lives | Mechanism | |---------|-------------|----------------|-----------| | [**Constraint**](#constraints-declarative) (declarative) | `aicr validate` against a snapshot | Recipe overlay top-level `constraints:` or a `validation:` phase block | `pkg/constraints` evaluator (in-process) | | [**Container-per-validator check**](#container-per-validator-checks) | `aicr validate` against a live cluster | `validators//` + `recipes/validators/catalog.yaml` | One K8s Job per check | | [**Component validation**](#component-validations-bundle-time) (bundle-time) | `aicr bundle` | `pkg/bundler/validations/checks.go` + `registry.yaml` `validations:` | In-process Go `ValidationFunc` | | [**Chainsaw health check**](#chainsaw-health-checks) | Two surfaces with distinct runtimes: `make check-health` post-deploy locally (shells out to the `chainsaw` CLI installed on the developer's machine), AND `aicr validate --phase deployment` in-cluster (executes the Test format in-process via `pkg/chainsaw/inprocess.go` — no external binary in the deployment validator image) | `recipes/checks//health-check.yaml` | Chainsaw YAML (Test format on both surfaces; raw K8s YAML asserts use the chainsaw Go library inside `assertRawResources`) | Rule of thumb: declarative constraint against a snapshot value → surface 1. Active probe of a live cluster → surface 2 or 4. Pre-deployment sanity gate on the resolved recipe → surface 3. ## Constraints (declarative) A **constraint** is a declarative expression — `K8s.server.version >= 1.32.4` — declared in a recipe overlay's top-level `constraints:` or a `validation:` phase block and evaluated by `pkg/constraints` against a measurement from a snapshot. No code change is needed to add a constraint to an existing recipe; only to add a new **operator**. **Where they live in YAML:** ```yaml # recipes/overlays/.yaml spec: constraints: - name: K8s.server.version value: ">= 1.32.4" - name: OS.release.ID value: "ubuntu" validation: deployment: checks: [operator-health, expected-resources] performance: checks: [nccl-all-reduce-bw] constraints: - name: nccl-all-reduce-bw value: ">= 450" # GB/s ``` **Declared checks are resolved fail-closed.** Before the cluster is prepared or any Job runs, `Validator.preflightDeclaredChecks` (`pkg/validator/validator.go`) rejects the run with `ErrCodeInvalidRequest` if any declared check name does not resolve to exactly one catalog validator in its declared phase — a name matching nothing (typo or a check missing from the loaded, possibly `--data`, catalog), a name that exists only under a different phase (`nccl-all-reduce-bw` declared under `deployment`), or a name declared more than once in one phase's `checks` list. All offenders across every requested phase are aggregated into a single error. This closes the fail-open path where an all-unmatched phase silently filtered to zero tests → `skipped` → nonblocking, letting `aicr validate --fail-on-error` exit `0` on a recipe whose required gate the catalog cannot supply ([#2121](https://github.com/NVIDIA/aicr/issues/2121)). The gate runs in `--no-cluster` mode too, so typos are caught in offline recipe validation. There is no opt-out: a declared name that resolves nowhere is always an authoring error. A check that is *legitimately* not applicable at runtime reports its own `skip` sentinel from inside the container — it is still declared and still resolves to a catalog entry. **A caller may withhold a declared check, and is held to the same discipline.** `Validator.SkipChecks` (`pkg/validator/skip_checks.go`, reached from `--skip-check` / `spec.validate.execution.skipChecks`) narrows a run to the checks the CALLER can satisfy, which is a property of the run rather than of the recipe: a lane deploying a subset of the recipe, or running against simulated devices. `selectEntries` withholds each named check from its phase and records it on the phase's CTRF builder as `skipped`, so a withheld check is reported rather than dropped and the recipe-evidence bundle still accounts for it. It records the reason twice on purpose: as prose in `message`, and as the `skipCheckReasonCode` (`named-in-skip-checks`) under the allowlisted `extra.skipReason` key. Only the second survives the default bundle, whose minimal redaction policy blanks every `message`. A bundle carrying WHICH check was withheld but not WHY would be the same "reads as complete" defect the flag's guards exist to prevent. The CNCF evidence renderer does NOT: `pkg/evidence/cncf/renderer.go` drops every skipped entry before grouping (pinned by `TestRenderSkippedExcluded`), so a withheld requirement would leave no file and no index entry. `validateFlagCombinations` refuses `--skip-check` together with `--evidence-dir` for that reason, rather than emitting a submission that reads as complete. `preflightSkipChecks` runs beside `preflightDeclaredChecks`, on the same fail-closed terms and at the same point: a name matching no catalog validator is rejected, and so is a list that would remove every declared check from a requested phase (that phase would report `passed` while running nothing, since the skipped entries keep `Summary.Tests` above zero). A known name that no requested phase declares is inert rather than wrong, and warns. It is a skip list and not an allow list on purpose. The two differ only on a check nobody has considered yet: under a skip list a newly declared check runs, and a caller that cannot satisfy it goes red until someone decides; under an allow list it would be excluded in silence. Top-level `constraints` — and any declared under `validation.readiness.constraints` — are evaluated as a **pre-flight gate** before phase checks run; other phases' `constraints` are evaluated against each container check's reported metrics. Readiness placement matters for gates that must not participate in generation-time overlay filtering. (No shipped overlay currently declares one: the GKE device-plugin ownership check, issue #1755, briefly lived here before moving into the GKE `gpuStack` profile's per-value constraints, which are verified at snapshot-based generation (criteria-only generation has no snapshot evaluator and defers entirely to the pre-flight) AND re-evaluated by the same readiness pre-flight.) **Supported operators** (`pkg/constraints/expr/expr.go`, re-exported from `pkg/constraints`): | Operator | Use | Notes | |----------|-----|-------| | `>=`, `<=`, `>`, `<` | Version / numeric comparison | Always treated as a version comparison; parsed via `pkg/version` | | `==`, `!=` | Explicit equality / inequality | Version compare if the expected value looks like a version and both sides parse; otherwise string compare | | *(none)* | `OperatorExact` | Case-sensitive string equality — `value: "ubuntu"` | The parser is operator-prefix-longest-first so `>=` wins over `>`. Anything matching the version heuristic (starts with digit, contains a dot, optional `v` prefix) is parsed via `pkg/version`. Anything else falls back to string comparison. **Evaluation flow:** `ParseConstraintExpression(expr)` → `ParsedConstraint{Operator, Value, IsVersionComparison}` → `pc.Evaluate(actual)` returns `(bool, error)`. The evaluator returns an error (not `false`) when a value claimed to be a version fails to parse — callers in `pkg/validator/validator.go::checkReadiness` treat parse errors as `ErrCodeInvalidRequest`, fail-closed. **One name bypasses the scalar flow entirely:** the node-set form `NodeTopology.gpu-nodes.label` ([#1755](https://github.com/NVIDIA/aicr/issues/1755)) is dispatched by exact name in `constraints.Evaluate` *before* `ParseConstraintPath`, uses its own value grammar (`=` / `!`, validated with the Kubernetes label validators), and quantifies the predicate over the GPU-node set synthesized from `NodeTopology.label` readings instead of comparing a single reading. The label that defines the GPU-node universe is **service-specific** — GKE's `cloud.google.com/gke-accelerator` today ([#2359](https://github.com/NVIDIA/aicr/issues/2359)) — resolved from the snapshot's `k8s.node.provider` via `fingerprint.FromMeasurements`. Adding a new service means adding one entry to `gpuNodeUniverseLabels` in `pkg/constraints/gpu_nodes.go` and covering it in `gpu_nodes_test.go`; the label MUST be present on GPU nodes from pool creation, before the GPU Operator or NFD run (the constraint form is pre-deployment). See `pkg/constraints/gpu_nodes.go` for its fail-closed rules (truncation, empty universe, malformed or ambiguous encodings, unknown service, service with no declared universe label). **Adding a new operator:** 1. Add an `Operator` constant in `pkg/constraints/expr/expr.go`. 2. Insert it in the operator slice in `ParseConstraintExpression` — **longest prefix first** (e.g. `~=` before `~`). 3. Add a `case` arm in `(*ParsedConstraint).Evaluate`. Return an `errors.WrapWithContext(ErrCodeInvalidRequest, ...)` for malformed inputs; never fall back to string compare silently. 4. Extend the `TestParseConstraintExpression` / `TestParsedConstraint_Evaluate` table in `pkg/constraints/expr/expr_test.go`. Both happy path and parse-error path. 5. If the operator implies a numeric range or tolerance, the *interpretation* lives in the validator phase (e.g. `validators/performance` evaluates NCCL bandwidth with a 10% tolerance baked into the check, not the operator). ## Container-per-validator checks A **check** is a Go function that runs inside a Kubernetes Job spawned by `aicr validate` against a live cluster. One Job per check, isolated per run. Per-phase containers are built from `validators//main.go`; the catalog in `recipes/validators/catalog.yaml` is the authoritative list. **Three phases**, evaluated in this fixed order (`pkg/validator/phases.go`): **deployment → conformance → performance**. | Phase | Purpose | Example | |-------|---------|---------| | `deployment` | Components installed and healthy | GPU operator pods running | | `conformance` | Workload-specific requirements | DRA, gang scheduling, autoscaling | | `performance` | Cluster meets perf thresholds | NCCL bandwidth, AIPerf TTFT p99 | Performance runs **last** on purpose: its inference-perf benchmark saturates every GPU on the node and tears the DynamoGraphDeployment (and, in DRA wiring mode, its DRA ResourceClaims) down asynchronously. Running it before conformance starved conformance's GPU-needing checks (historically `dra-support`, whose 1-GPU test pod failed to schedule with "cannot allocate all claims" on single-node clusters; since #1620 that behavioral subtest runs only where full-GPU DRA is usable — the GPU allocation checks are capability-driven via the shared `validators/internal/allocmode` probe, with inference-perf's worker wiring mode-dispatched per chosen node — but the saturation-ordering rationale stands for every GPU-needing check). On MNNVL clusters `dra-support` also runs a behavioral IMEX channel subtest (#1649). The gate is the intersection of Ready, schedulable nodes carrying the `nvidia.com/gpu.clique` label and nodes with a usable `compute-domain.nvidia.com` ResourceSlice: no clique-labeled node means the subtest is not applicable (non-MNNVL nodes such as H100 still publish compute-domain slices, so slice presence alone cannot gate it); clique-labeled nodes with no usable compute-domain slice fail the check. The driver serves one ComputeDomain channel claim per node, so candidate nodes whose channel is already held by an allocated claim (a standing ComputeDomain such as Slinky Slurm's, or a running MNNVL workload) are excluded from the probe. When every candidate is occupied the check does not record not applicable (that would certify DRA without behavioral evidence on a ComputeDomain-only cluster); it instead verifies allocation and preparation from the holding claims: a claim that is allocated, reserved by a pod, and whose pod is Running on that node proves the driver allocated the channel and the kubelet prepared it. If no holder verifies, the check fails as inconclusive. When applicable the check creates a per-run ComputeDomain (`numNodes: 0`, `Single` allocation), waits for the driver-generated ResourceClaimTemplate, and runs a busybox probe pinned to the candidate nodes that consumes one channel from it. The verdict is the pod reaching Succeeded with exactly one channel device visible under `/dev/nvidia-caps-imex-channels`; the generated ResourceClaim's post-terminal state is recorded as best-effort evidence only, because the resource-claim controller releases and garbage-collects a completed pod's template-generated claim. It shares the per-run namespace, ownership token, and UID-guarded cleanup of the other GPU allocation probes. The GPU allocation checks follow an **Inspect / Verify / Select** separation (#1327): `allocmode.Detect` is the INSPECT step — it probes cluster facts (usable full-GPU DRA, usable device plugin, per-node device counts) without policy judgment; `allocmode.Verify` is the VERIFY step — it compares the recipe-configured `ValidationInput.GPUAllocationPolicy` (resolved from hydrated recipe values by `pkg/validator/v1.ResolveGPUAllocationPolicy`) against those facts and fails closed with `ErrCodeInvalidRequest` on mismatch; SELECT — the capability-preference dispatch inside each check — applies only when the policy is `unspecified` (standalone runs without recipe context). A configured policy forces the mechanism at secure-accelerator-access, dra-support's behavioral subtest, and inference-perf's worker wiring and node discovery. `secure-accelerator-access` also routes on the resolved recipe before any probe: when `slinky-slurm` resolves, it skips with the reason recorded. The Slinky NodeSet reserves every GPU on a node for its `slurmd` pod and Slurm GRES/cgroups isolate access, so the check's Kubernetes per-pod probe could either never schedule or attest the wrong access path (#2724). `slinky-slurm-gpu-access` verifies that path instead (#2756). From the login pod it runs an `srun --gpus=1` job, which must list exactly one GPU and open exactly one GPU minor device, then a job with no GPU request pinned with `--nodelist` to the same Slurm node, which must open none. The probe shell only reports facts (`KEY=value` lines: node, `nvidia-smi` UUIDs, the errno of a read-only open of each `/dev/nvidiaN`); `parseSlurmGPUProbe` and the two evaluate functions decide. It never lists `/dev/nvidia*` to infer isolation: the cgroup device controller leaves the nodes visible and refuses `open()` with EPERM. `PhaseAll` (the string `"all"`) is the CLI / recipe wildcard; `ParsePhaseSelection` collapses it to nil-meaning-everything. It is **exclusive** — combining `all` with any other phase is rejected. By default all phases run and produce results regardless of earlier failures — a performance threshold miss no longer silences conformance results. Pass `--fail-fast` (or set `spec.validate.execution.failFast: true` in config) to restore stop-on-first-failure behavior for cost-sensitive runs. `readiness` is also a field on `ValidationConfig` (see `pkg/recipe/validation.go`) and appears in overlay examples, but it is **not** a container-per-validator phase. Readiness runs as inline constraint evaluation in `pkg/validator/validator.go::checkReadiness` before any phase container is scheduled — see [Constraints](#constraints-declarative) above for how the evaluator works. ### Quick start Three steps to add a check to an existing validator container. **1. Implement** in `validators//my_check.go`: ```go func checkMyComponent(ctx *validators.Context) error { slog.Info("checking my-component") pods, err := ctx.Clientset.CoreV1().Pods("my-namespace").List( ctx.Ctx, metav1.ListOptions{LabelSelector: "app=my-component"}) if err != nil { return errors.Wrap(errors.ErrCodeInternal, "failed to list pods", err) } if len(pods.Items) == 0 { return errors.New(errors.ErrCodeNotFound, "no my-component pods found") } fmt.Printf("Found %d my-component pod(s)\n", len(pods.Items)) // → CTRF evidence return nil } ``` **2. Register** in `validators//main.go`: ```go validators.Run(map[string]validators.CheckFunc{ "my-component": checkMyComponent, }) ``` **3. Add a catalog entry** in `recipes/validators/catalog.yaml`: ```yaml - name: my-component phase: deployment description: "Verify my-component pods are running" image: ghcr.io/nvidia/aicr-validators/deployment:latest timeout: 2m args: ["my-component"] # must match the registered dispatch key ``` ### Container contract | Exit code | Meaning | CTRF | |-----------|---------|------| | `0` | passed | `passed` | | `1` | failed | `failed` | | `2` | skipped | `skipped` — return `validators.Skip(reason)` | | Channel | Captured as | |---------|-------------| | **stdout** | CTRF `stdout` (human-readable evidence) — use `fmt.Printf` | | **stderr** | Streamed live to the user — use `slog.*` | | `/dev/termination-log` | Failure reason (≤ 4096 bytes), written on `return error` | | **stdout sentinel lines** | Structured/side-channel data — see [Stdout sentinels](#stdout-sentinels) | ### Capability applicability contract A capability-gated conformance/performance check probes a live prerequisite (a Deployment, CRD, served API group, or non-empty list) before it runs, and turns that probe outcome into a verdict via `Capability.Require` (`validators/applicability.go`). A `Skip` (exit 2) is a claim that the capability is genuinely **inapplicable** — not a way to paper over a missing prerequisite or an infrastructure error. The dividing line is whether the resolved recipe *declares* the component that supplies the capability: `RecipeDeclares(ctx, component)` reports true only when that componentRef is present **and** enabled in `ctx.ValidationInput`. `Require(ctx, probeErr, present)` resolves the fate from the probe outcome. `present` is consulted only on a clean read (`probeErr == nil`) — e.g. a `List` that returned zero items or a discovery call that did not serve the expected group. Probe errors are classified by `classifyCapabilityProbeError`: | Probe outcome | recipe DECLARES | recipe does NOT declare | |---------------|-----------------|-------------------------| | clean read, present | `nil` (proceed) | `nil` (proceed) | | clean read, absent / empty | **FAIL** (`ErrCodeNotFound`) | **Skip** | | probe err: `NotFound` (incl. group-not-served) | **FAIL** (`ErrCodeNotFound`) | **Skip** | | probe err: `Forbidden` / `Unauthorized` (401/403) | **FAIL** (`ErrCodeUnauthorized`) — always | **FAIL** (`ErrCodeUnauthorized`) — always | | probe err: timeout / deadline | **FAIL** (`ErrCodeTimeout`) — always | **FAIL** (`ErrCodeTimeout`) — always | | probe err: transport (503 / conn reset / refused / HTTP2 lost) | **FAIL** (`ErrCodeUnavailable`) — always | **FAIL** (`ErrCodeUnavailable`) — always | | probe err: other / aggregated API discovery | **FAIL** (`ErrCodeInternal`) — always | **FAIL** (`ErrCodeInternal`) — always | **Infrastructure errors never Skip**, even when the recipe does not declare the component: a missing RBAC grant, an apiserver timeout, a dropped connection, or an aggregated discovery failure is not evidence that the capability is inapplicable — it is evidence the validator *could not tell*, which must block the gate. Only a clean `NotFound` / empty result on a **non-declared** capability is Skip-eligible. (A `Forbidden` on a declared dependency maps to `ErrCodeUnauthorized` rather than `ErrCodeInternal` precisely so operators read "the validator cannot see it — grant it RBAC," not a blanket internal error.) **Collection/list probes use `RequireList`.** When the probe is a `List` whose *empty result* (not an error) is what signals inapplicability — e.g. `detectPlatform` reading `Nodes().List` to classify the cloud — the caller passes the List error through `Capability.RequireList(err)` instead of `Require`. A List *error* is never Skip-eligible: even the rare `NotFound` shape on a collection endpoint is an apiserver/aggregation-layer anomaly, not the clean absence of a single object, so every List error blocks with a classified code and never a Skip. The empty-result inapplicability case is handled by the caller (`len(items) == 0`), keeping `Require`'s Skip path off the error branch entirely. **#1327 standalone boundary.** A standalone validator run carries no recipe context: `pkg/validator/v1.ToValidationInput` leaves `ComponentRefs` empty, so `RecipeDeclares` returns false (it is nil-safe on a nil `Context` or nil `ValidationInput`). On that path a clean-absent probe still `Skip`s, which preserves the capability-driven automatic selection that #1327 introduced. Fail-closed fires **only once the recipe actually declares** the dependency — declaration is what converts "inapplicable, so Skip" into "promised but missing, so fail." This closes the #2122 false-PASS: before this contract, a recipe that declared a capability whose prerequisite was absent — or whose probe hit an auth/timeout/ transport/discovery error — reported `passed` (via a Skip that read as non-blocking), letting a broken or unauthorized cluster clear conformance. ### Stdout sentinels A check runs inside a pod; the only channels that reach the orchestrator are the exit code, `/dev/termination-log`, and the captured stdout (pod logs). To move *more* than a pass/fail verdict across that boundary, checks emit **sentinel lines** — stdout lines with a reserved prefix that a specific parser recognizes and pulls out. Plain (non-prefixed) stdout is unaffected and flows into the CTRF `stdout` array as before. Three sentinels exist today, with deliberately different lifecycles: | Prefix | Emitted by | Parsed by | Payload | Kept in CTRF `stdout`? | Survives minimal redaction? | |--------|-----------|-----------|---------|------------------------|-----------------------------| | `RESULT:` + one space | any check (`fmt.Printf`) | `extractResultSummaries` (`pkg/validator/validator.go`) | free-form human text (throughput, bandwidth, TTFT…) | **yes** — line stays; trailing text is *also* echoed to the live CLI at INFO | **no** — dies with `stdout` under the default policy | | `##AICR-EXTRA##` + one space | `validators.EmitExtra` | `parseExtraSentinels` (`pkg/validator/job/result.go`) | one JSON object → `TestResult.Extra` (counts / enum codes) | **no** — stripped as transport, not evidence | **yes** — allowlisted keys are published (see below) | | `##AICR-PROVENANCE##` + one space | `validators.EmitRuntimeProvenance` | `parseExtraSentinels` (same pass) | one JSON object → `TestResult.RuntimeProvenance` (template digests + key paths) | **no** — stripped as transport | **yes** — bounded by `redact.boundRuntimeProvenance` (see "NCCL benchmark runtime provenance") | All are parsed with `strings.CutPrefix` and the parsers are pure, unit-tested functions. They are separate channels on purpose: `RESULT:` surfaces live metrics to a human watching a run, so it carries unbounded free-form text and is correctly redacted with the rest of `stdout`; `##AICR-EXTRA##` carries structured data that must *outlive* redaction, so it is low-cardinality, allowlisted, and stripped from the human evidence; `##AICR-PROVENANCE##` carries the one structured record that is neither a count nor a code — content digests plus template key paths — under its own bounding rule rather than the `Extra` allowlist. Do not route structured outcome data through `RESULT:` (it would not survive publication) or human prose through `##AICR-EXTRA##` (it would be dropped by the allowlist or leak identifiers). ### Structured evidence that survives redaction `stdout` and `message` are free-form log text: they can leak node names, IPs, DNS names, and secret/cert names, so the default **minimal** redaction policy (`pkg/evidence/redact`) strips them from a published, signed evidence bundle — only `--full` keeps them. A check that reports on `stdout` alone therefore shows as `passed` with no detail in the artifact a downstream consumer verifies by default: a 1-of-2-node pass is indistinguishable from 2-of-2, and a skip loses its reason. To carry outcome data that *does* survive minimal redaction, emit it into the CTRF `extra` map (`ctrf.TestResult.Extra`, mirroring the CTRF spec's `extra` object) via `validators.EmitExtra`: ```go // Emit failures are non-fatal — warn, but never flip the check's verdict on a // failed stdout write (checkNvidiaSMI wraps this as the emitExtraOrWarn helper): emit := func(extra map[string]string) { if err := validators.EmitExtra(extra); err != nil { slog.Warn("failed to emit validator extra", "error", err) } } // Success path — coverage disclosure (counts only, never node names/IPs): emit(map[string]string{"nodesValidated": "1", "nodesTotal": "2"}) // Skip path — reason enum code: emit(map[string]string{"skipReason": "no-gpu-nodes"}) ``` **Transport.** `EmitExtra` marshals the map to one JSON line prefixed with `ctrf.ExtraLinePrefix` (`##AICR-EXTRA##` followed by one space) on stdout — the only channel that crosses the pod boundary besides the exit code and termination log. `EmitRuntimeProvenance` uses the same transport under `ctrf.ProvenanceLinePrefix` (`##AICR-PROVENANCE##` + space) for the derived runtime's `RuntimeProvenance` record. The orchestrator (`pkg/validator/job.ExtractResult`) parses each sentinel line, keeps the **last valid non-empty** payload of each kind — the `Extra` map as `TestResult.Extra`, the provenance record (both digests present) as `TestResult.RuntimeProvenance` — and strips every sentinel line from the stored `stdout` (transport, not human evidence). A malformed line is non-fatal: it is logged and skipped without discarding an earlier valid payload — a garbled line never flips a pass to an error, nor clears a coverage line that preceded it. Keep the human `fmt.Printf` lines too; they still feed `--full` and live `aicr validate` output. **Low-cardinality + allowlist contract.** `extra` values MUST be counts or enum codes only (`"2"`, `"no-schedulable-gpu-nodes"`) — never node names, IPs, or hostnames. `pkg/evidence/redact`'s `ctrfExtraAllowlist` enforces this at the **publication boundary** (not just at emission, which raw prefixed stdout could bypass) with a fail-closed **key _and_ value** check: only the listed keys (`nodesValidated`, `nodesTotal`, `skipReason`, `runtimeSource`) survive, and each surviving value must pass its key's validator — a non-negative decimal count for the `nodes*` keys, and for `skipReason` a **closed set** of known codes (`ctrfSkipReasons`, currently `no-gpu-nodes`, `no-schedulable-gpu-nodes`, `nodes-busy`, and `named-in-skip-checks`, the one code a *caller* rather than a check mints, for `--skip-check`), and for `runtimeSource` the closed set `delivered-artifact` | `recipe-supplied-runtime` | `cluster-capability` (`ctrfRuntimeSources`). A closed set rather than a shape regex is deliberate: a kebab-case regex would still pass an arbitrary low-cardinality identifier like `customer-prod-cluster`. A value that is ill-shaped or unlisted (an IP under `nodesTotal`, a hostname or unminted code under `skipReason`) is dropped even though its key is allowed. Every other key is dropped too, and if nothing survives the map ships as absent (no empty `extra: {}`). Adding a new key (with its value validator) or a new `skipReason` code means adding it to that allowlist (and bumping `redact.PolicyVersion`) in the same change; there is no CTRF schema in `api/`, so the `pkg/validator/ctrf` godoc and this page are the contract. ### NCCL benchmark runtime provenance `nccl-all-reduce-bw*` results carry a `runtimeSource` code saying **which artifact the bandwidth number describes**. Pass/fail still comes from the bandwidth floor; provenance is a separate, closed-set label (#2297): | `runtimeSource` | Meaning | |---|---| | `delivered-artifact` | The benchmark runtime was **derived from the `ClusterTrainingRuntime` the recipe ships** (`torch-distributed-tcpxo`, read from the live API). The number attests to the delivered wiring. | | `recipe-supplied-runtime` | The recipe supplied the runtime itself via `nccl-benchmark-runtime(-ref)` (#1792) and owns its wiring. | | `cluster-capability` | The validator's embedded fixture: proves the fabric can reach the floor, says nothing about what the recipe ships. | The class is decided from the **recipe**, never from what happens to be installed: a leaf is `delivered-artifact` only when its enabled `kubeflow-trainer` componentRef both lists the `torch-distributed-tcpxo` runtime manifest and records a `tcpxoInterfaces` override (`validators/internal/gkenet.FabricRuntimeDelivered` — a mapping without the artifact describes nothing, and a listed manifest without a recorded mapping, which generation never produces, fails closed rather than downgrading to the fixture). Today that is `h100-gke-cos-training-kubeflow` alone — the other kubeflow leaves declare the Trainer but ship no fabric runtime, and are `cluster-capability`. A recipe-supplied runtime or an `nccl-benchmark-profile` combined with a delivered one is rejected; the benchmark cannot have two owners of its runtime or of its platform. For a delivered artifact the performance validator, **before any cluster mutation**, verifies recipe → deployed → cluster (the recipe's recorded mapping must equal the deployed runtime's exactly and in order; every selected network must exist, as a set comparison), then derives the benchmark runtime: the shipped `node` PodTemplateSpec is copied **wholesale — metadata and spec** — and only the paths in `benchmarkOwnedNodePaths` (worker `image`, `command`, `args`, `resources`, `terminationMessagePolicy`) are re-applied from the MPI skeleton (the skeleton's worker sets no `terminationMessagePolicy`, so that override clears a shipped value), with volumes and mounts merged additively. An override-path guard fails the run if the derived template differs from the shipped one anywhere else, and a baseline precondition covers every overridden path — the shipped worker must set no `command`/`args` (an entrypoint would hide fabric activation), must declare the NCCL fabric env, must request `nvidia.com/gpu` — and only that — under both `resources.limits` and `resources.requests`, with a quantity equal to the target nodes' per-node GPU count (checked at apply time, where that count is known, so a deployed runtime whose GPU request was removed or changed is failed rather than silently repaired by the skeleton's), and must set no `terminationMessagePolicy`; `image` is the one override with no precondition, since the benchmark binary lives only in the fixture image and the fabric plugin is mounted from the host. The measurement runs under the **shipped** environment. The derived worker's bootstrap exports the container's own `NCCL_*`/`CUDA_*`/`LD_LIBRARY_PATH` to the ssh sessions mpirun opens instead of re-sourcing the host `nccl-env-profile.sh` the capability fixture uses, and the launcher's `-x NCCL_*`/`-x CUDA_*`/`-x LD_LIBRARY_PATH=` tuning exports are stripped (`NCCL_DEBUG` is kept — log volume, not wiring). The object that is applied is derived at apply time from the skeleton rendered with the run's real template data, so placeholder types survive (`containers[].args` stay strings, which Trainer's structural CRD requires; `numProcPerNode` stays an integer). **Evidence carrier.** `runtimeSource` rides `Extra` and is emitted the moment the class is decided — before the delivered path's live verification — so a run that fails on a missing runtime or a mapping drift still records that it was a `delivered-artifact` measurement. The audit record — sha256 of the normalized shipped and applied worker templates, the paths at which they differ (benchmark overrides plus stamped scheduling), and the inventory of shipped paths inherited unchanged — is computed against the object as **stored** (read back after the create, so admission defaulting is part of the digest) and recorded only once the `TrainingRuntime` create succeeded, so a run that fails before or at application publishes no record; after that point it is published whether or not the measurement succeeds, twice: as a human-readable listing on stdout (`--full` only), and as the bounded `TestResult.RuntimeProvenance` carrier (`##AICR-PROVENANCE## ` sentinel → `pkg/validator/job`), which **survives minimal redaction**. The redaction policy for that carrier (`redact.boundRuntimeProvenance`, rule `ctrf.tests.runtimeProvenance.bound`): both digests must be lowercase sha256 hex or the record is dropped; paths are template *keys* only and must match the dotted key grammar; every **structural** segment must be a JSON field name reachable from core/v1 `PodTemplateSpec` (`redact.ctrfPodTemplateFields`, an exact set generated from `k8s.io/api` and pinned by `TestPodTemplateFieldsMatchAPI`) or the path is dropped, so the carrier can only describe the Kubernetes schema whether a non-schema key survived CRD pruning or the sentinel line were forged; every named-list selector (`containers[node]`, `volumes[x]`, `volumeMounts[x]`, `env[x]`) collapses to `[*]` unless it is an `env` selector naming a variable in the **exact** fabric set (`redact.ctrfFabricEnvNames`: the GPUDirect-TCPXO NCCL configuration the shipped runtime declares, plus `CUDA_VISIBLE_DEVICES` and `LD_LIBRARY_PATH`), which is kept because it is the evidence; a key under any **user-keyed map** of the PodTemplateSpec API, wherever it sits (`redact.ctrfFreeKeyMaps`: `labels`, `annotations`, `nodeSelector`, resource `limits`/`requests`, `overhead`, `matchLabels`, CSI `volumeAttributes`, flexVolume `options`, pod-certificate projection `userAnnotations`) collapses to the map unless the **whole key** is in the exact vendor set (`redact.ctrfVendorKeys`: `networking.gke.io/interfaces`, `networking.gke.io/default-interface`, `devices.gke.io/container.tcpxo-daemon`, `cloud.google.com/gke-accelerator`, `trainer.kubeflow.org/trainjob-ancestor-step`, `nvidia.com/gpu`, `nvidia.com/gpu.present`, `node.kubernetes.io/instance-type`). There is no prefix or domain rule anywhere in the policy: `NCCL_CUSTOMER_ACME_PROD`, `networking.gke.io/customer-prod` and a sidecar's `resources.limits.acme.internal/project-prod` are operator text and collapse like any other name. So `metadata.annotations.networking.gke.io/interfaces`, `spec.containers[*].env[NCCL_FASTRAK_IFNAME].value` and `spec.containers[*].resources.limits.nvidia.com/gpu` are kept while an operator's `spec.nodeSelector.my-org/pool` becomes `spec.nodeSelector`, `spec.initContainers[setup].resources.limits.acme.internal/x` becomes `spec.initContainers[*].resources.limits`, and `spec.volumes[customer-cache].secret.secretName` becomes `spec.volumes[*].secret.secretName`; lists are deduplicated, sorted and capped at 1024 entries. Adding a variable or key to the shipped runtime that the inventory should name means adding it to the corresponding set in the same change. No value — network name, node name, env value — ever appears. Separately, two deployment checks cover GKE GPUDirect-TCPXO networking: `gke-gpu-nic-networks` runs the census, readiness/binding, and runtime-wiring arms over the Network CRs; `gke-gpu-nic-topology` is gated only on the recipe declaring `gke-nccl-tcpxo` and reads node `nic-info` annotations to catch a gVNIC-displaced GPU NIC. A base `h100-gke-cos-training` recipe (TCPXO, no runtime) still runs the census + readiness + topology checks. **Mounted data:** `/data/snapshot/snapshot.yaml`, `/data/validation/validation.yaml` (override via `AICR_SNAPSHOT_PATH`, `AICR_VALIDATION_PATH`). **Environment** (set by the Job deployer; the catalog entry's `env` values are appended after these): | Variable | Purpose | |----------|---------| | `AICR_NAMESPACE` | Validation namespace (fallback) | | `AICR_CHECK_TIMEOUT` | Go-duration timeout for the check; honored by `ctx.Ctx`. Falls back to `defaults.CheckExecutionTimeout` if unset or malformed (logged WARN). | | `AICR_VALIDATOR_IMAGE_REGISTRY` | Override the image registry prefix (CLI passes through to inner workloads). | | `AICR_VALIDATOR_IMAGE_TAG` | Override the resolved tag when the binary's stamped commit has no published image (e.g. `edge` or `sha-`). See [Validator image tags](#validator-image-tags). Forwarded to inner workloads (including `aiperf-bench`). | | `AICR_NODE_SELECTOR` | Comma-separated `key=value`; read via `ctx.NodeSelector` | | `AICR_TOLERATIONS` | Comma-separated `key=value:effect`; read via `ctx.Tolerations` | **Forwarded from the orchestrator's shell.** These variables are read from the environment of the process running `aicr validate` and forwarded only to the listed checks (`buildEnv` in `pkg/validator/v1/job_plan_internal.go`). A catalog entry's `env` value for any of these names is dropped, so they cannot be set from the catalog or a `--data` overlay: | Variable | Forwarded to | Purpose | |----------|--------------|---------| | `HF_TOKEN` | `inference-perf` | Hugging Face token for model downloads; see [`inference-perf`](#inference-perf-model-concurrency-and-weights-cache). | | `AICR_INFERENCE_PERF_NO_CLEANUP` | `inference-perf` | When true (`strconv.ParseBool`), leaves the per-run namespace in place for inspection; forwarded as `1`. | | `AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY` | `inference-gateway` | When truthy, the check fails if the gateway's `LoadBalancer` Service is open to `0.0.0.0/0` — its `spec.loadBalancerSourceRanges` is empty or includes an any-source CIDR (`0.0.0.0/0` or `::/0`). Default (unset): the open exposure is recorded and warned but the check still passes. | | `AICR_NCCL_FABRIC` | `nccl-all-reduce-bw-net` | NET fabric selector: `efa` (default) or `roce`. | | `AICR_NCCL_RUNTIME_IMAGE` | `nccl-all-reduce-bw`, `nccl-all-reduce-bw-net`, `nccl-all-reduce-bw-nvls` | Overrides the NCCL launcher/worker image baked into the per-platform TrainingRuntime templates; a malformed reference fails closed in the pod. | **RBAC.** The engine creates a per-run ServiceAccount and ClusterRoleBinding named `aicr-validator-`. Per-run naming prevents concurrent runs from clobbering each other's RBAC. External tooling selects by label `app.kubernetes.io/name=aicr-validator`, not literal name. **Image-pull policy** is computed by `v1.ImagePullPolicy(image, imageTagOverride)` in `pkg/validator/v1/job_plan.go`: side-loaded (`ko.local/*`, `kind.local/*`) → `Never`; digest-pinned (`name@sha256:…`) → `IfNotPresent`; `AICR_VALIDATOR_IMAGE_TAG` set or `:latest` suffix → `Always`; otherwise → `IfNotPresent`. Both the outer validator Job and any inner workload Job share this helper so policy cannot drift. ### Validator image tags The catalog declares every validator image as `…:latest`; `catalog.ResolveImage` (`pkg/validator/catalog/catalog.go`) rewrites that tag at runtime so the validators match the `aicr` binary that launched them: 1. **Stamped build** — the binary's version + commit resolve the tag. `ResolveImage` checks the version first: a **release** build → that release's version tag (`:vX.Y.Z`, or `:vX.Y.Z-rc…` for a pre-release); otherwise a dev/`main` build → `:sha-`, the immutable per-commit image CI publishes for `main` pushes (only — see the caveat below the table). 2. **`AICR_VALIDATOR_IMAGE_TAG` set** — overrides step 1 for *all* catalog images uniformly, including the inner `aiperf-bench` runner the `performance` validator launches (so both must exist at that tag). What CI publishes: | Trigger | Tags built (`on-push.yaml` / `on-tag.yaml`) | |---------|----------------------------------------------| | Push to `main`, not docs-only | `:sha-` (immutable) **and** `:edge` (moving → latest validator-image build) | | Stable release `vX.Y.Z` | `:vX.Y.Z` **and** `:latest` | | Pre-release `vX.Y.Z-rc…` | `:vX.Y.Z-rc…` only — **not** `:latest` | `on-push.yaml` runs **only on `main`** and is skipped when a push touches *only* docs (`paths-ignore: **.md`, `docs/**`, `LICENSE`). So no `:sha-` is built — and `:edge` is not advanced — for a docs-only `main` commit, nor for any feature-branch / PR commit (the build job is gated to `refs/heads/main`). `:edge` therefore tracks the last `main` commit that ran the image build, *not necessarily* HEAD, and `sha-$(git rev-parse origin/main)` can 404 right after a docs-only merge. Confirm the tag exists (see below) and fall back to `:edge` or the last published SHA. **`:latest` is the last _stable_ release, never `main`.** The on-tag workflow first builds every validator architecture under one run-unique candidate tag, resolves one authoritative digest map, and scans both platforms and attests those exact digests. Only then may version aliases move; a stable release verifies all seven version aliases before it starts updating any `:latest` alias. A pre-release promotes its version alias but never `:latest`, so a validator change merged to `main` after the last stable release is absent from `:latest` until the next one. Running `AICR_VALIDATOR_IMAGE_TAG=latest` against a `main`-tracking recipe can therefore silently run *older* validator behavior — e.g. a `performance.constraints` pin such as `inference-model` / `inference-concurrency-per-gpu` / `nccl-benchmark-profile` is only honored by a validator new enough to read it; an older `:latest` validator ignores the pin and runs its compiled default, which can surface as a misleading result (for `nccl-benchmark-profile`, a silent skip) rather than a clear version error. The seven GHCR alias updates cannot be atomic across repositories. Promotion performs an all-images, read-only preflight before its first write and accepts each `:latest` only at that image's immediate-prior or current-candidate digest. It also rejects an out-of-order stable release when the same or a newer stable version is already public. If a later registry write fails, re-running the failed workflow jobs with the same candidate converges the remaining aliases without overwriting a conflicting version. Candidate tags are retained for that recovery and for audit; cleanup is intentionally deferred to a separate retention design. **To run the validator built on `main`** (e.g. testing a recipe whose pins are not yet in a release), point at `:edge` or a published `main` commit — *not* `:latest`: ```shell # Moving tag — latest main validator-image build: AICR_VALIDATOR_IMAGE_TAG=edge aicr validate -r recipe.yaml -s snapshot.yaml --phase performance # Immutable pin (reproducible) — use a published main commit, not blindly HEAD # (a docs-only HEAD has no image; verify with the registry check below): AICR_VALIDATOR_IMAGE_TAG=sha- aicr validate -r recipe.yaml -s snapshot.yaml ... ``` **Not when emitting evidence.** `aicr validate --emit-attestation` refuses a mutable validator tag and fails closed, because the attestation records validator images by tag alone — so the `:edge` form above is rejected there while the `:sha-` form is accepted. See [Validator image provenance](/aicr/contributor-guide/publishing-recipe-evidence#validator-image-provenance). Drop the override for an evidence run, or pass `--allow-mutable-validator-tags` if the evidence is disposable. A bare `go build` stamps `commit: unknown`, so step 1 can't resolve a `:sha-` tag and the override is required. `make build` stamps the commit — but CI publishes `:sha-` images **only for `main`** (the build job is gated to `refs/heads/main`), so auto-resolution works only when you build from a `main` commit whose image exists. Any feature-branch, fork, or PR build (pushed or not) stamps a SHA with **no** published image and still needs `AICR_VALIDATOR_IMAGE_TAG=edge` (or a published `main` SHA) — `:edge` is the closest tag to your branch. Find or trace the `main` tag against GitHub Container Registry (GHCR) — public read: ```shell REPO=nvidia/aicr-validators/performance SHA=$(git rev-parse origin/main) TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${REPO}:pull" | jq -r .token) # Does the image for this main commit exist? (200 = yes) curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" \ -H 'Accept: application/vnd.oci.image.index.v1+json' \ "https://ghcr.io/v2/${REPO}/manifests/sha-${SHA}" ``` To go the other way — which commit built a given image — read the OCI labels baked in by CI: `org.opencontainers.image.revision=` and `org.opencontainers.image.version=main-`. ### `validators.Context` API `LoadContext()` builds it from the container environment and returns the only struct a `CheckFunc` ever sees: ```go type Context struct { Ctx context.Context Cancel context.CancelFunc Clientset kubernetes.Interface RESTConfig *rest.Config DynamicClient dynamic.Interface Snapshot *snapshotter.Snapshot ValidationInput *v1.ValidationInput Namespace string NodeSelector map[string]string // nil = use defaults Tolerations []corev1.Toleration // nil = use defaults } ``` `ctx.Timeout(d)` returns a child context with a shorter deadline. `validators.Run(map)` is the container entry point; it dispatches by `os.Args[1]`, maps `Skip` → exit 2, errors → exit 1, nil → exit 0. **Scheduling overrides.** When creating inner workloads, check `ctx.NodeSelector` and `ctx.Tolerations` before applying hardcoded platform selectors. `nodeName` pinning (e.g. nvidia-smi, DRA isolation) bypasses the scheduler and should not apply `ctx.NodeSelector`. ### `PodLifecycle` helper For checks that deploy a single test pod (training NCCL, conformance DRA isolation, nvidia-smi probes), use `validators/helper/pod.go` rather than reimplementing watch/cleanup: ```go lc := &helper.PodLifecycle{Clientset: ctx.Clientset, Namespace: ctx.Namespace} pod, err := lc.CreatePodFromTemplate(ctx.Ctx, "testdata/probe.yaml.tmpl", subs) if err != nil { return errors.Wrap(...) } defer func() { _ = lc.CleanupPod(context.Background(), pod) }() // deferred cleanup uses fresh ctx if err := lc.WaitForPodSuccess(ctx.Ctx, pod, defaults.PodWaitTimeout); err != nil { logs, _ := lc.GetPodLogs(context.Background(), pod) return errors.WrapWithContext(errors.ErrCodeInternal, "probe failed", err, map[string]any{"logs": logs}) } ``` `WaitForPodSuccess`/`WaitForPodRunning` use the watch API (`pkg/k8s/pod`) — no polling, no sleep loops. The cleanup goroutine must use `context.Background()` because the parent is canceled on return; this is one of the two CLAUDE.md-sanctioned uses of `Background()`. ### Pre-flight gates are fail-closed `pkg/validator/validator.go::checkReadiness` evaluates the recipe's top-level `constraints` plus any `validation.readiness.constraints` *before* any phase runs. A parse error or a failing constraint returns `ErrCodeInvalidRequest` and aborts the entire run. **Do not** `slog.Warn; continue` on an evaluator error — that masquerades a broken validation YAML as a passing constraint, which is an explicit anti-pattern in CLAUDE.md. The `dependencyAffinity` pre-flight (validator catalog entries declaring a required dependency) follows the same rule. ### Node-scoped checks: cordoned GPU nodes A cordon (`Node.Spec.Unschedulable`) means "do not schedule new pods," not "exclude this node from GPU service" — it is commonly transient maintenance, not deliberate exclusion. A node-scoped check that enumerates only the schedulable cohort (e.g. via `helper.FindSchedulableGpuNodes`) can pass while silently validating fewer nodes than the cluster has, with nothing in the evidence indicating reduced coverage. That is the spuriously-passing-check failure mode CLAUDE.md's anti-pattern table calls out, applied to node enumeration instead of a single boolean. Checks that deploy a per-node test pod (`check-nvidia-smi`) must instead call `helper.FindGpuNodes`, which returns every GPU node tagged with its cordon state, and: 1. Validate only the schedulable subset. Note this is a **deliberate policy choice, not a technical necessity**: `check-nvidia-smi`'s probe pod is pinned via `nodeName` and tolerates all taints (`testdata/nvidia-smi-verify-pod.yaml`), so it bypasses the scheduler and *could* run on a cordoned node — `Unschedulable` is a scheduler-side predicate that kubelet admission of a directly-bound pod never consults. The check still excludes cordoned nodes because a cordon signals operator intent to keep workloads off the node during maintenance, even when a specific pod could technically slip past it. A future check with a different tradeoff should say so explicitly rather than assume cordons are unconditionally unschedulable. 2. Report cordoned nodes explicitly in the check's stdout evidence (`: skipped (cordoned)`), never omit them from the node count. 3. Print a coverage line (`RESULT: nodesValidated: /`) on every exit path (skip, failure, and success), not only the success path, so a pass *or* a failure on reduced scope is visible. `` is the count actually confirmed working, not the schedulable count: it is `0` on the all-cordoned and busy-skip paths (nothing was attempted yet) and the successful-node count on the failure path (a partial pass is not conflated with a full one). Exception: when the recipe declares `Deployment.gpu-driver.version`, those same all-cordoned and busy paths (and the no-GPU-nodes path) fail closed instead of Skip — a declared host-driver floor that cannot be measured must not PASS (#1995). Skip remains only when the floor constraint is absent. The `RESULT:` prefix is `pkg/validator/validator.go`'s `resultSummaryPrefix` convention: the validator runtime echoes the trailing text of any such stdout line into live CLI output via `slog.Info`, unconditionally — without it, the line is only visible after the run, inside the (possibly redacted) report. Caveat: the line still also lands in `TestResult.Stdout`/`.Message` in the CTRF report, which the default ("minimal") redaction policy (`pkg/evidence/redact`) strips from a signed evidence bundle. The `RESULT:` prefix makes the coverage figure visible during a live `aicr validate` run regardless of redaction, but it is not guaranteed to survive into the artifact a downstream consumer verifies by default. That is why the same counts are ALSO emitted through `validators.EmitExtra` (#1951) — see **Structured coverage survives redaction** below. `expected-resources`' `rdmaFabricProbeCoverage` is itself node-scoped and now follows this pattern too (#1952). It enumerates every GPU node via `helper.FindGpuNodes`, validates only the schedulable Mellanox RDMA-capable cohort for uniform allocatable fabric, but discloses each cordoned RDMA-capable node explicitly (`: skipped (cordoned)`), counts it in `nodesTotal`, and never narrows the printed total. Because the probe is re-run on every poll iteration (`verifyRDMAFabricReady`/`pollUntilStable`), the stdout enumeration/`RESULT:` line is printed **exactly once at the settled terminal outcome** (ready or fail-closed), never per tick. The structured Extra is emitted twice: an **eager floor** on the first observation that enumerates any RDMA-candidate node (with `nodesValidated=0` — nothing is certified mid-poll) and again at the terminal outcome. `parseExtraSentinels` keeps the last valid sentinel, so the terminal emit wins on a clean exit; the floor exists only so a cordoned-node narrowing still reaches the signed bundle on the rarer path where the process is SIGKILLed before the terminal emit runs (#1952). The broader no-margin kill race this once described (#2473) is now bounded generally by `ValidatorJobDeadlineHeadroom`: the Job's `activeDeadlineSeconds` adds that headroom on top of the catalog `timeout:`, so an exhausted poll has margin to unwind and reach the terminal emit before the Job's SIGKILL, and this floor remains defense-in-depth for a probe call that blocks past its own poll-budget cancellation. The gate stays fail-closed: "could not observe the fabric" reports `0` validated and never reads as ready. Unlike `check-nvidia-smi`, the RDMA gate never *skips* — it either certifies the cohort or fails closed — so it mints no `skipReason` enum. Its coverage rides the existing `nodesValidated`/`nodesTotal` allowlist keys unchanged (see below), so that change did not move the redaction `PolicyVersion`. Cluster-aggregate checks that assert on an operator's aggregate status (`gpu-operator-health`) remain unaffected — DaemonSet operands ignore cordons. **Structured coverage survives redaction.** The `RESULT:` stdout line is echoed to the live CLI but is stripped from a signed bundle by the default (`minimal`) redaction policy (`pkg/evidence/redact`), so both `check-nvidia-smi` and the RDMA gate ALSO emit the coverage through `validators.EmitExtra` as low-cardinality counts (`nodesValidated`/`nodesTotal`) or a closed-set `skipReason` code. Those keys are the only ones that clear the fail-closed `ctrfExtraAllowlist` (a value that structurally looks like a node name or IP is dropped even under an allowed key), so a signed bundle records reduced coverage — e.g. a cordoned RDMA node narrowing the fabric cohort — without shipping any operator-identifying text. Node names appear only in the redacted stdout enumeration, never in the Extra channel. See #1951/#1952. For *deliberate*, durable exclusion of a node from GPU service (as opposed to transient cordon-for-maintenance), use the GPU Operator's `nvidia.com/gpu.deploy.operands=false` node label instead of a cordon. It removes the node from allocatable `nvidia.com/gpu` entirely, so the node drops out of both node-scoped checks and ClusterPolicy aggregation consistently — a cordon does neither reliably. ### Performance benchmark tuning Performance checks ship validation *methodology* knobs as env vars on the catalog entry (overridable via `aicr validate ... --data`). Pass/fail thresholds live in the recipe overlay constraints; methodology lives with the validator. A value that fails to parse fails the check with `ErrCodeInvalidRequest` *before* any workload deploys — never silently fall back. Full list (defaults, semantics) is in the `validators/performance` package godoc. NCCL variants exposed today: `nccl-all-reduce-bw`, `nccl-all-reduce-bw-net`, `nccl-all-reduce-bw-nvls`. Inference: `inference-perf` (Dynamo + AIPerf). > **Constraint-name contract.** Each NCCL variant looks up a > constraint with the *exact* same name as the check. A recipe > running the `-net` or `-nvls` variant **must** declare a same-named > constraint; the variant will Skip if only the generic > `nccl-all-reduce-bw` constraint is present. #### NCCL: recipe-driven applicability (`nccl-benchmark-profile`) NCCL applicability defaults to the compiled `supportedNCCLCombinations` matrix (variant → service → accelerators), keyed to the recipe's `criteria`. The optional `nccl-benchmark-profile` performance constraint (bare `{accelerator}/{service}` value, e.g. `gb200/eks`) overrides that default so external `--data` recipes — new services, or embedded services extended to new accelerators — can run the embedded benchmarks ([#1703](https://github.com/NVIDIA/aicr/issues/1703)). Resolution is recipe-only (no env tier), and parsing fails closed: a malformed or unknown profile aborts the check with `ErrCodeInvalidRequest` instead of skipping. The profile keys template selection, service-specific fabric plumbing (EFA / GKE NIC discovery, worker scheduling defaults), and preflights; node identification (the GFD `gpu.product` filter) stays on the recipe's own `criteria.accelerator`. Profiles are restricted to pairs already present in the compiled matrix, so the `TestSupportedNCCLCombinationsHaveRuntimeTemplates` wiring guard (advertised tuple ⇒ parseable template) covers every profile a recipe can name. See `validators/performance/nccl_benchmark_profile.go`. #### NCCL: recipe-supplied runtime (`nccl-benchmark-runtime-ref`) A profile still requires an embedded template to borrow, so it cannot cover a private service+accelerator whose fabric matches none of the shipped templates ([#1792](https://github.com/NVIDIA/aicr/issues/1792)). The optional `nccl-benchmark-runtime-ref` performance constraint closes that gap: its value is a bare `{accelerator}/{service}` naming a Kubeflow `TrainingRuntime` the recipe ships in its `--data` tree at `validators/performance/testdata/{accelerator}/{service}/runtime.yaml` — the same layout the embedded templates use, so an external runtime is a drop-in for upstreaming. Resolution is split across the two-stage design: - **Orchestrator** (`pkg/validator/benchmark_runtime_ref.go`, `resolveBenchmarkRuntimeRef`, called at the top of `ValidatePhases` / `ValidatePhase`): reads the referenced file through the recipe `DataProvider` and lowers its content into the `nccl-benchmark-runtime` **carrier** constraint before the `/data/validation` ConfigMap is written. Fails closed (`ErrCodeInvalidRequest`) on a malformed/traversal ref, a missing/empty file, an absent `DataProvider`, or a ref set alongside an inline carrier. The two constraint names are defined once in `pkg/validator/v1` (`PerfConstraintNCCLBenchmarkRuntime*`) so the write side and the pod read side cannot drift. - **Pod** (`validators/performance/nccl_benchmark_runtime.go`): reads the `nccl-benchmark-runtime` carrier and renders it — via the same `${VAR}` substitution as the baked-in templates, through the shared `renderYAMLTemplate` core — in place of `testdata/{accelerator}/{service}`. `validateNcclAllReduceBw` bypasses the `ncclCombinationSupported` applicability gate entirely (applicability is granted by the recipe's explicit opt-in, keyed on its own criteria) and `applyNCCLResources` skips every service-specific step — EFA/TCPXO/RDMA discovery, the GB200-NVreg / GKE-TCPXO preflights, RoCE claim application, and NVLS/IMEX provisioning — because the supplied runtime owns its fabric wiring. Transport verification still runs (the `-net` / `-nvls` markers are transport-internal and fabric-agnostic), so a runtime paired with a named variant must still prove its transport rather than pass on bandwidth alone; the worker cohort is also sized against the runtime's own `nodeSelector` so `WorkerCount` matches placement. It fails closed on a carrier that is not a `trainer.kubeflow.org/v1alpha1` `TrainingRuntime` with a `node` replicatedJob. The runtime is applied only through `trainingRuntimeGVR` with a force-set name/namespace, so a recipe can supply a `TrainingRuntime` and nothing else — not an arbitrary resource kind. It is mutually exclusive with `nccl-benchmark-profile`. The one setup step it can opt into is IMEX. The run namespace is per-run (`ncclRunNamespace`), so an operator cannot pre-create a ComputeDomain or claim for the runtime. `customRuntimeManagesIMEX` inspects the pod-level `resourceClaims` when the plan is resolved, before any cluster mutation: - A reference to `ncclIMEXClaimTemplateName` (`nccl-all-reduce-imex`) sets `benchmarkRuntimePlan.managedIMEX`, and `applyNCCLResources` then provisions the ComputeDomain on any variant, as the NVLS path does. - Any other template, or any `resourceClaimName`, fails closed with `ErrCodeInvalidRequest`, since it could never resolve in the per-run namespace (#2569). #### `inference-perf`: model, concurrency, and weights cache The `inference-perf` check warms vLLM before measuring, so the one-time CUDA-graph/JIT compile cost is excluded from the reported throughput and p99 TTFT. Its knobs are read by the in-cluster validator from the `inference-perf` catalog entry's `env` (override per run with a catalog overlay in the `aicr validate --data ` directory). Unlike `HF_TOKEN` and `AICR_INFERENCE_PERF_NO_CLEANUP`, they are **not** forwarded from the orchestrator shell, so exporting the knobs below before `aicr validate` has no effect. The **model** and **per-GPU concurrency** can also be set per accelerator in the recipe overlay's `performance.constraints`, symmetric with the throughput / TTFT thresholds: ```yaml validation: performance: constraints: - name: inference-model value: Qwen/Qwen3-8B # HF model ID (bare value, no comparator) - name: inference-concurrency-per-gpu value: "256" # positive integer - name: inference-throughput value: ">= 50000" - name: inference-ttft-p99 value: "<= 2000" ``` Resolution precedence is **recipe constraint > catalog env knob > compiled default** (`Qwen/Qwen3-8B` at 256/GPU). A non-positive / non-integer `inference-concurrency-per-gpu` fails closed with `ErrCodeInvalidRequest`. The model-weights cache's StorageClass resolves with the same precedence via the `inference-model-cache-storage-class` constraint (see the `AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS` row below). | Variable | Default | Effect | |----------|---------|--------| | `AICR_INFERENCE_PERF_CONCURRENCY_PER_GPU` | `256` | Concurrent requests per GPU; total is this × free GPUs on the chosen node. Prefer the per-accelerator `inference-concurrency-per-gpu` recipe constraint over this global knob. | | `AICR_INFERENCE_PERF_MODEL` | `Qwen/Qwen3-8B` | Hugging Face model ID to benchmark. Override per accelerator via the `inference-model` recipe constraint. | | `AICR_INFERENCE_PERF_ROUTER_MODE` | `least-loaded` | Dynamo frontend routing strategy (`DYN_ROUTER_MODE`): one of `kv`, `round-robin`, `random`, `power-of-two`, `least-loaded`, `device-aware-weighted` (upstream's `direct` is excluded — it needs per-request worker IDs the benchmark never sends). Lets a characterization run A/B strategies without rebuilding the image. Consumed only when `inference-routing-mode` is `dynamo-router`; in `gateway-epp` mode the EPP selects endpoints and the value is unused. A **set** value is validated in either mode — an unknown value fails closed with `ErrCodeInvalidRequest` rather than being silently ignored (the routing mode is a recipe decision, so a typo must not pass under one recipe and abort under another). | | `AICR_INFERENCE_PERF_WORKLOAD_READY_TIMEOUT` | `10m` | Wait for the `DynamoGraphDeployment` to become ready (image pull + model load + worker health). Large models load slower — raise this **and** the catalog entry's `timeout` in tandem, or the parent deadline caps it. | | `AICR_INFERENCE_PERF_HEALTH_TIMEOUT` | `5m` | Wait for the endpoint to serve a real chat-completion *after* the workload reports Ready. Concurrent first-load from one RWO cache PVC can push first-serve past 5m; raise it (bounded by the catalog `timeout`). | | `AICR_INFERENCE_PERF_MODEL_CACHE_SIZE` | `100Gi` (on) | The PVC-backed model-weights cache is **on by default**. Set a different K8s quantity to resize, or a disable sentinel (`off`/`0`/`none`/`disabled`) to turn it off and download from HF directly. | | `AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS` | cluster default | StorageClass for the cache PVC. On a cluster with **no default SC and no value here**, the check **fails fast** with guidance rather than leaving the PVC `Pending` until timeout. AICR-deployed EKS gets a default `gp3` SC from `aws-ebs-csi-driver`, and GKE Standard typically ships with `standard-rwo`, but neither default is guaranteed to attach for every node machine family (A4X/GB200 nodes reject `pd-balanced` disks and need a Hyperdisk-backed class, see [GKE GB200 Storage Prerequisites](/aicr/integrator-guide/gke-gb-200-networking#storage-prerequisites)). Prefer the per-accelerator `inference-model-cache-storage-class` recipe constraint over this global knob. | | `AICR_INFERENCE_PERF_MODEL_CACHE_EXTRA_COMPATIBLE_TYPES` | unset | Comma-separated `parameters.type` values to add to the compiled machine-family/StorageClass compatibility allowlist (`storageCompatibilityRules` in `validators/performance/model_cache.go`), for example a new Hyperdisk type a cloud provider ships before AICR's allowlist is updated to include it. Without a matching entry here, the selected StorageClass (the `inference-model-cache-storage-class` constraint, `AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS`, or the cluster default when neither is set) fails closed with `ErrCodeInvalidRequest` if its type isn't allowlisted for the node's machine family, rather than binding a PVC that fails to attach at Job-run time. | | `AICR_INFERENCE_PERF_MODEL_CACHE_POPULATE_TIMEOUT` | `13m` | Wait for the one-time model-cache populate Job (cold image pull + first-ever Hugging Face download into the PVC). Separate from — and larger than — `AICR_INFERENCE_PERF_WORKLOAD_READY_TIMEOUT` because the populate Job pays a cold pull *and* a multi-GB download; provide the optional HF-token secret to remove anonymous-download throttling. Raise it (and the catalog `timeout`) for very large models. **Migration:** the cache-populate wait no longer honors `AICR_INFERENCE_PERF_WORKLOAD_READY_TIMEOUT` (which now bounds only the DynamoGraphDeployment readiness wait) — set this knob instead to widen the populate budget. | For gated models, or to lift Hugging Face rate limits on large downloads, set `HF_TOKEN` in the orchestrator environment: it is forwarded only to the `inference-perf` validator, which provisions an optional `aicr-hf-token` Secret the benchmark workers reference via `secretKeyRef`. A token raises *per-account* limits but does not bypass Hugging Face *per-IP* throttling — large models pulled by many workers benefit most from the shared cache. **Model-weights cache (PVC).** Many workers re-downloading a large model (and re-downloading on every crash-restart) repeatedly trips Hugging Face's **per-IP** throttle, so the cache is **on by default**: 1. The validator creates an `aicr-model-cache` **PVC** (ReadWriteOnce) in the per-run namespace. 2. A one-time **populate Job** — pinned to the same node the workers use (so the `WaitForFirstConsumer` RWO volume binds there) — downloads `config.model` into the PVC via `huggingface_hub` (using `HF_TOKEN` if present). The validator blocks on it before deploying. The populate container carries CPU/memory **requests** but no memory **limit** — a limit OOMKills large-model downloads via page cache on cgroup v2. 3. Workers mount the PVC **read-only** at `HF_HOME` with `HF_HUB_OFFLINE=1`, loading weights locally and never reaching HF (failing closed if the cache is incomplete). The PVC lives in the per-run namespace and is torn down on cleanup, so the cache is **intra-run** (one download shared by the run's N workers), not persisted across runs. Because it is RWO, all workers co-locate on one node — which the validator already enforces for a stable per-node baseline. Multi-node would require RWX storage (e.g. EFS); for at-scale serving, Dynamo's ModelExpress server is the alternative (see #1116). > **Throughput-gate scaling.** `buildInferenceConfig` sizes the workload to the > **free** GPUs on the chosen node, which on a shared node is fewer than the > full allocatable count. The `inference-throughput` gate is therefore scaled > by `freeGPUs / nodeGPUs` (throughput is ~linear in GPU count at fixed > per-GPU concurrency) so a healthy per-GPU result on a partially occupied node > is not failed against a full-node number. TTFT is a per-request latency and > is **not** scaled. #### Methodology: a baseline gate, and reading run-to-run fluctuation `inference-perf` is a **conformance baseline**, not a tuned peak-throughput benchmark — pass/fail answers *"is this deployment serving acceptably,"* not *"what is the maximum."* Read the numbers as a health floor, not a leaderboard. Design choices follow from that, and from what we measured debugging run-to-run TTFT fluctuation (see NVIDIA/aicr#1192): - **Throughput is the stable, discriminating signal; TTFT p99 is noisy at high concurrency.** Near the saturation knee the p99 curve is steep, so batching / scheduling timing produces large run-to-run swings on an otherwise healthy deployment. That is why the `inference-ttft-p99` constraint is a **generous ceiling** (catches gross stalls — real ones ran 9–45 s — while tolerating normal knee jitter), not a tight target. - **The verdict should reflect the deployment, not RNG.** The AIPerf workload is pinned for reproducibility — fixed random seed, fixed input/output token counts (stddev 0), a pinned prompt pool, and greedy decoding (`temperature: 0`). Input determinism stabilizes *throughput*; it does not remove system-side p99 jitter at the knee. - **Routing matters.** The inference-perf workload defaults to Dynamo's load-aware least-loaded router (`DYN_ROUTER_MODE=least-loaded`; override via the `AICR_INFERENCE_PERF_ROUTER_MODE` knob above), which balances by each worker's active in-flight load so a transiently-slow worker stops receiving its full share — mitigating the stochastic EKS H100 worker-stall / throughput degradation at the saturation knee (issue #1197). Frontend-to-worker requests use Dynamo's request plane (Dynamo 1.4+ defaults to TCP; AICR does not set `DYN_REQUEST_PLANE=nats`). Workers publish KV-cache events directly over ZMQ; the KV router consumes them end-to-end with no NATS relay. Least-loaded routing does not consume those events. The `inference-routing-mode` recipe input defaults to `dynamo-router`; set `gateway-epp` to validate the GAIE/EPP path through agentgateway with worker frontend sidecars in direct mode. The direct-mode sidecars honor EPP routing headers; they do not relay KV events. - **The AIPerf load generator co-locates with the GPU workers, but that is not resource contention.** It is CPU-only and the GPU node has ample CPU headroom (measured node CPU pressure ≈ 0 across runs); co-location does not starve the workers. Do not add worker CPU/memory requests to "fix" contention that the data does not show. - **Triaging an anomalous run:** the severe stalls we saw were **stochastic and often not reproducible** — re-run before concluding. Verify GPU health (clocks, ECC, throttle reasons, XID) to rule out hardware. And note `nvidia-smi` *utilization* is a duty-cycle metric (kernel-present time), **not** compute saturation — a worker can read 100% util while under-fed; cross-check **power draw and achieved throughput**, not utilization alone. - **A GPU driver restart needs a DRA plugin restart.** If you restart the GPU driver pod (`nvidia-driver-daemonset-*`) on a node — e.g. to clear suspected driver state between runs — also restart the NVIDIA DRA kubelet-plugin (`nvidia-dra-driver-gpu-kubelet-plugin-*`) on that node. Otherwise it serves stale CDI specs and every worker `ResourceClaim` fails with `FailedPrepareDynamicResources: … empty device edits`, leaving the decode workers stuck in `ContainerCreating` until the phase times out. - **The serve-readiness probe tolerates cold-start first-token latency.** A fresh worker's first inference captures CUDA graphs / JIT-warms kernels — measured at ~42 s on RTX PRO 6000. The readiness probe (`waitForEndpointReady`) therefore uses a generous **120 s** per-request timeout (`InferenceEndpointProbeTimeout`), not the generic 30 s `HTTPClientTimeout`; the latter cancelled the legitimate first request mid-warmup and failed healthy deployments with `timed out waiting for inference endpoint to serve requests` — the *same* outer symptom as the (fixed) #1192 discovery panic but a different root cause. AIPerf's own warmup absorbs steady-state once the probe passes. - **Inspecting a failed run.** `AICR_INFERENCE_PERF_NO_CLEANUP=1`, exported in the shell that runs `aicr validate` (it is forwarded to the pod, not read from the catalog), leaves the namespace, DGD, workers, frontend, and AIPerf Job in place after the run so a serve-wait / generate hang can be examined live (`kubectl logs` the frontend, ping `/v1/models` and `/v1/chat/completions`). Debug-only — delete the namespace manually afterward. ### Code walkthrough ```go // validators/deployment/operator_health.go func checkOperatorHealth(ctx *validators.Context) error { slog.Info("listing pods", "namespace", gpuOperatorNamespace) // → stderr pods, err := ctx.Clientset.CoreV1().Pods(gpuOperatorNamespace).List( ctx.Ctx, metav1.ListOptions{LabelSelector: gpuOperatorLabel}) if err != nil { return errors.Wrap(errors.ErrCodeInternal, "failed to list pods", err) } fmt.Printf("Found %d gpu-operator pod(s):\n", len(pods.Items)) // → CTRF evidence for _, p := range pods.Items { fmt.Printf(" %s: %s\n", p.Name, p.Status.Phase) } if runningCount == 0 { return errors.New(errors.ErrCodeInternal, "no pods in Running state") } return nil } ``` `slog.*` → stderr → streamed live. `fmt.Printf` → stdout → captured as CTRF evidence. `return nil` → 0, `return error` → 1, `return validators.Skip(reason)` → 2. ### Directory layout ```text validators/ ├── context.go # LoadContext, Context type ├── runner.go # Run() entry, exit-code mapping ├── helper/pod.go # PodLifecycle (watch, logs, cleanup) ├── deployment/ # phase image: deployment ├── performance/ # phase image: performance (+ aiperf-bench.Dockerfile) └── conformance/ # phase image: conformance ``` Each phase directory compiles to one container image; multiple checks share the binary, selected by `os.Args[1]`. ## Component validations (bundle-time) A **component validation** is an in-process Go function that runs during `aicr bundle` to catch component misconfigurations the recipe parser and Helm chart won't catch on their own — required flags unset, incompatible host-resource requests, missing dependency components. Runs **in-process**, no network, no Kubernetes. Anything requiring a real cluster belongs in a container-per-validator check or chainsaw health check, not here. ### Declaring a validation Add a `validations:` block to the component entry in `recipes/registry.yaml`: ```yaml components: - name: nodewright-customizations validations: - function: CheckWorkloadSelectorMissing severity: warning # warning (non-blocking) | error (blocking) conditions: intent: [training] # AND across keys, OR within a key message: "May cause nodewright to evict running training jobs." ``` | Field | Required | Notes | |-------|----------|-------| | `function` | yes | Must match a name registered in `pkg/bundler/validations/checks.go::init()` | | `severity` | yes | `error` stops the bundle; `warning` (the default for any unrecognized value) appends to the deployment notes; `info` logs only | | `conditions` | no | Keys are criteria fields from `pkg/recipe/criteria.go`. Empty = always runs | | `message` | no | Actionable detail appended to function output | Conditions are evaluated via `checkConditions(recipeResult, conditions)`. Keys = AND across, values within a key = OR. When a new accelerator, service, OS, intent, or platform is added to `pkg/recipe/criteria.go`, audit existing condition blocks per CLAUDE.md's enum-expansion rule. `RunValidations` (`pkg/bundler/validations/registry.go`) dispatches on the lowercased severity, and all three arms consume both return values of the check — a returned `error` is never silently dropped: | Severity | Warnings become | Errors become | |----------|-----------------|---------------| | `error` | blocking errors | blocking errors | | `warning` | deployment notes | blocking errors | | `info` | `slog.Info` only | `slog.Info` only | `info` is for an advisory that is true of a valid configuration — something the operator should weigh, not something they must fix — so it stays out of the deployment notes and surfaces only under `--debug`. Note the asymmetry: under `info` a returned error is logged rather than raised, so do not use `info` for a gate whose failure must be acted on. `warning` is the fallback for any unrecognized value, which keeps a typo'd severity non-silent. ### Shipping functions | Function | Checks | |----------|--------| | `CheckWorkloadSelectorMissing` | nodewright `--workload-selector` set when conditions match | | `CheckAcceleratedSelectorMissing` | nodewright `--accelerated-node-selector` set | | `CheckHostMofedWithoutNetworkOperator` | Host-mode MOFED component paired with `network-operator` | | `CheckWildcardAcceleratedToleration` | Accelerated-node tolerations carry no wildcard (keyless) entry — on AKS a wildcard deadlocks nodewright interrupt packages ([nodewright#296](https://github.com/NVIDIA/nodewright/issues/296)); wired at `severity: error`, skipped when the component is disabled via `--set` | | `CheckGB300HostKernelGranule` | Bare-metal GB300 (`service: generic`, `accelerator: gb300`) applies a tuned profile that sizes hugepages for a 64k-granule ARM64 host kernel. A 4k-granule host still boots — Linux rejects the `hugepagesz=512M` clause and drops its paired `hugepages=` count — so the node just runs without that pool. Advisory only (`severity: info`); skipped when the component is disabled or `tuningEnabled` resolves to false on the final effective values (recipe merge plus scalar `--set` and typed `--set-json`/`--set-file`, under the canonical name and its registry aliases). | | `CheckDriverOwnershipCoherence` | GPU driver-ownership coherence on the final effective values (recipe merge + `--set`/`--set-json`/`--set-file` under canonical names and registry aliases): a recipe whose snapshot observed no NVIDIA driver (`metadata.gpuDriverState: absent`) must not bundle with the preinstalled-driver assumption. When GPU Operator manages the driver, `nvidia-dra-driver-gpu.nvidiaDriverRoot` must equal `gpu-operator hostPaths.driverInstallDir`; with a preinstalled driver, the DRA root must avoid the unpopulated operator container root and may intentionally differ from `hostPaths.driverInstallDir` ([#1087](https://github.com/NVIDIA/aicr/issues/1087), [#1757](https://github.com/NVIDIA/aicr/issues/1757)). Wired at `severity: error`. | | `CheckMariaDBOperatorOwnershipCoherence` | MariaDB Operator installation safety for AICR-provided Slurm accounting: `metadata.mariaDBOperatorState` values `crs-detected` and `unknown` block bundling, `api-detected` or omitted evidence warns, and `absent` proceeds silently. Wired at `severity: warning` so warning results remain non-blocking while returned errors still fail the bundle. | | `CheckNPDNotDuplicatingProviderNPD` | `node-problem-detector` is bundled only on an allowlist of verified platforms — never where the provider already runs its own NPD (GKE, AKS), on OpenShift (no privileged SCC binding), or on Talos (restricted namespace) | | `CheckGKETCPXOInterfacesCoherence` | The final resolved kubeflow-trainer `tcpxoInterfaces` value matches the mapping the recipe records in `configuration.gke.tcpxoInterfaces` | | `CheckNVSentinelDriverLabelDetectable` | Where the GPU Operator does not own the driver and no driver pod exists (driver ships in the node image), NVSentinel sets `labeler.assumeDriverInstalled`, so its driver-label-gated DaemonSets do not come up half-rolled-out | | `CheckNVSentinelRuntimeClassCoherence` | NVSentinel metadata-collector `runtimeClassName` matches the RuntimeClass the GPU Operator creates from `operator.runtimeClass`, so its pods are not rejected at admission | | `CheckNVSentinelTracingEndpointRequired` | NVSentinel distributed tracing is not enabled without an OTLP collector endpoint | | `CheckNVSentinelNicHealthMonitorRequiresMetadataCollector` | `nic-health-monitor` is not enabled without `metadata-collector` (or a `nicInclusionRegexOverride`), which would otherwise discover zero devices | | `CheckNVSentinelPreflightDCGMReachable` | The preflight DCGM check's hostengine address resolves to a served Service (gpu-operator present, not relocated, and `dcgm.enabled`) | | `CheckNVSentinelPreflightGangSchedulerRequired` | Preflight gang coordination against KAI PodGroups is not enabled while `kai-scheduler` is disabled or absent | Registered in `pkg/bundler/validations/checks.go::init()`. ### ValidationFunc signature Fixed (`pkg/bundler/validations/interface.go`): ```go type ValidationFunc func( ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string, ) (warnings []string, errors []error) ``` - `componentName` is the registry name; resolve component refs via `recipeResult.ComponentRefs`. - `bundlerConfig` exposes CLI flags and merged values. - `conditions` is the YAML block, not the resolved criteria — use `checkConditions(recipeResult, conditions)` to gate. ### Adding a new function 1. Implement in `pkg/bundler/validations/checks.go` matching `ValidationFunc`. 2. Register: `registerCheck("CheckMyCondition", CheckMyCondition)` in `init()`. 3. Wire into a component's `validations:` block in `registry.yaml`. 4. Add a table-driven test in `checks_test.go` exercising every condition branch with synthetic `RecipeResult` and `bundlerConfig`. No cluster, no network. ### Common pitfalls - **Function name typo in YAML.** Fails closed — `RunValidations` raises `ErrCodeInvalidRequest` ("unknown validation function") rather than skipping the check. Add a test that calls `Get("...")` for every shipping check. - **Returning an error when you mean a warning.** Errors stop the bundle. If the user can ship through it, return a warning. - **Network or K8s calls.** Bundle must work offline. Push cluster probes to surface 2 or 4. ## Chainsaw health checks A **chainsaw health check** is a YAML test in `recipes/checks//health-check.yaml` that asserts a deployed component's state. Runs against a real cluster (typically a Kind cluster after `aicr bundle` + `helm install`) via the [Chainsaw](https://kyverno.github.io/chainsaw/) test runner. The same assertion file now powers TWO surfaces: 1. **`make check-health` / `make check-health-all`** — local Kind-cluster sanity invoked manually by chart authors. `check-health-all` sweeps only registry-linked components (`registry.yaml`'s `healthCheck.assertFile` entries); an opt-in-only check like `nvsentinel-observability` or `nvsentinel-preflight` runs via `check-health COMPONENT=`. 2. **`aicr validate --phase deployment`** — registry-declared content is loaded into `ComponentRef.HealthCheckAsserts` during recipe resolution (PR #1219) and executed by the deployment validator's chainsaw runner (PR #1220). Since #1236 the runner is **pure Go**: `pkg/chainsaw/inprocess.go` unmarshals the `chainsaw.kyverno.io/v1alpha1` Test, walks `spec.steps[].try[]`, and dispatches `assert` / `error` to kyverno-json's `checks.Check` engine against live cluster state. No external binary is shipped in the deployment validator image. CLI output is source-tagged `[chainsaw]` vs `[expectedResources]` so operators can disambiguate when both paths report on the same component. Only surface 1 still needs the `chainsaw` binary, and only on the developer's machine — that is why `.settings.yaml` keeps the `testing_tools.chainsaw` pin and `tools/update-chainsaw-checksums`. The **readiness gate** (`cmd/gate`, image `ghcr.io/nvidia/aicr-gate`, emitted by `aicr bundle --readiness-hooks`) is a third consumer of the same executor. It shelled out to an embedded `chainsaw` binary until PR #2038 removed it; that binary was the gate image's only source of HIGH CVEs and upstream ships no release that fixes them. It calls `chainsaw.Run` against a dynamic client built from the Job's read-only ServiceAccount, so `pkg/chainsaw` is the single assertion engine for validator and gate alike. The gate's `--namespace` flag survives as the default namespace for resource blocks that omit one (`defaultNamespaceFetcher` in `pkg/chainsawgate/runner`), preserving what `chainsaw --namespace` did. A Test that declares no `assert`/`error` operation is rejected rather than passing vacuously (#2040); a check that is intentionally a no-op — today the four `*-ocp-olm` components, whose readiness is enforced by the bundler's `--readiness-hooks` gate instead — must say so with the `aicr/no-op-check: "true"` annotation on the Test. Three sibling shapes are rejected for the same reason — each one would otherwise report a component healthy without having evaluated what it claims to (#2051): - **Empty content.** A blank, whitespace-only, or comments-and-`---`-only entry has no Test to carry the no-op annotation, and used to reach `assertRawResources` and pass on zero documents. - **A mixed stream.** Once any document is a chainsaw Test the whole stream routes to the in-process executor, and nothing else reads it, so a non-Test document alongside it was evaluated by no path at all. Only Test documents may share a stream; a trailing `---`, a comment-only document, and an explicit `null` are punctuation, not content. - **Both `assert` and `error` on one operation.** Chainsaw evaluates one action per operation and the executor reaches `assert` first, so the negative check silently never ran. Split them into separate `try` entries. **Indeterminate is not a verdict.** The gate reports `Unknown`, not `Fail`, when a component's state was never established — a discovery outage, an apiserver 5xx, a forbidden read. Those surface as `ErrCodeUnavailable`, which `pkg/chainsaw` never treats as terminal, so it retries for the whole budget and can only surface it once the budget runs out. Exit codes are unchanged (`Fail` and `Unknown` gate identically), but the distinction points the operator at a broken cluster instead of a broken component. A no-match from the RESTMapper is held to the same standard: it resolves to `ErrCodeNotFound` — the immediate-pass path for a negative `error:` assertion — only when the retry ran against discovery performed after the lookup began *and* the kind's own API group was fully enumerated. client-go drops `ErrGroupDiscoveryFailed` whenever partial results exist, so a kind in an unreachable group arrives as a bare `NoKindMatchError`; `resolveMapping` consults the same cached discovery client the mapper resolves through to tell the two apart. The scope is deliberately one group: real clusters routinely carry a broken aggregated APIService, and treating any partial failure as global would strand every negative assertion. **Registration.** A component opts in by declaring `healthCheck.assertFile` in `recipes/registry.yaml`: ```yaml components: - name: nfd healthCheck: assertFile: checks/nfd/health-check.yaml ``` The path is relative to `recipes/`. `make check-health COMPONENT=` invokes Chainsaw against `recipes/checks//health-check.yaml` (no-cluster flag has no effect here — chainsaw always needs a real cluster). **Assertion file** is plain Chainsaw: ```yaml apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: name: gpu-operator-health-check spec: timeouts: { assert: 5m } steps: - name: validate-deployment-exists try: - assert: resource: apiVersion: apps/v1 kind: Deployment metadata: { name: gpu-operator, namespace: gpu-operator } status: { (availableReplicas > `0`): true } ``` Use Chainsaw's `assert` (expected match) and `error` (unexpected match must not exist). Always include an existence guard before phase assertions so an empty namespace can't yield a vacuous pass. See the [Chainsaw assert reference](https://kyverno.github.io/chainsaw/latest/operations/assert/) for the full operator list. **Read-only allowlist.** Registry-declared assert files MUST use only `assert` and `error` operations. The deployment validator Job runs under a ServiceAccount bound to cluster-admin, so registry content is restricted at runtime to read-only Chainsaw operations (`pkg/chainsaw/allowlist.go`). Any other operation (`script`, `apply`, `create`, `delete`, `patch`, `update`, `wait`, `command`, `sleep`, `podLogs`, `events`, `describe`, `get`) is rejected with `ErrCodeInvalidRequest`, as is an operation that sets both `assert` and `error`. PR-time enforcement shipped in #1223: `pkg/chainsaw.TestValidateTestReadOnly_RegistryContent` walks every registry-declared assert file against this allowlist under `make qualify`, so violations are caught before they ever reach the validator. **Value-gate awareness (#1844).** A registry assert file is static — it cannot see the component's effective Helm values. That is a problem for a manifest whose whole resource is gated behind a value: e.g. `--set nodewrightcustomizations:tuningEnabled=false` on a single-package tuning manifest (`tuning-gke.yaml` / `tuning-generic.yaml`) suppresses the entire `tuning` Nodewright CR, so an unconditional `assert status.status: complete` would fail on a deliberately-untuned cluster. The deployment validator resolves this by rendering the component's manifests with the effective values (`recipe.GetComponentValues` → `manifest.Render`) and skipping the assert when the render yields no matching CR — see `nodewrightHealthCheckSuppressed` and `expectedNodewrightNames` in `validators/deployment/expected_resources.go`. The skip is **fail-closed**: whenever the values keep the CR, the render still lists it and the assert runs, so only an intentionally-absent CR is tolerated (a CR that *should* deploy but is missing on the cluster still fails). The same render drives the Go readiness check `verifyNodewrightReady`, so both surfaces agree on which CRs to expect. After the readiness probes finish, the `nodewright-customizations` assert is also skipped on a cluster whose operator predates the `NodeWright` kind the assert names. The signal is the recipe's own `nodewright-operator` pin (`resolveNodewrightGVR`). A pin below v0.18.0 (or no usable pin) with only `skyhook.nvidia.com` served takes the legacy path, where `verifyNodewrightReady` verifies each `Skyhook` by name, so a healthy legacy cluster passes without a static assert that can never match. A v0.18.0+ pin on a cluster that does not serve `nodewright.nvidia.com` fails closed — that is a broken operator install, and a stale legacy `Skyhook` must not stand in for the missing `NodeWright`. A discovery error also fails closed rather than skipping. Discovery counts a group as served only once it lists the `nodewrights` (or `skyhooks`) resource itself, and `verifyNodewrightReady` re-resolves that on every poll. A recipe that renders Nodewright CRs polls through CRD establishment and fails on timeout if neither group ever serves them. This discovery-based skip is scoped to `nodewright-customizations`. Other components' asserts are subject only to their own render-based suppression. The suppression must be expressed **in the recipe** — an overlay-declared component `overrides:` (how `tuningEnabled: false` ships as the AKS default) or an inline `overrides:` in the recipe file — because that is what the render sees. Two boundaries follow from *where* the deployment validator gets its inputs (it runs in-cluster inside a Job, reading the recipe from a mounted ConfigMap with **no `DataProvider`** — only the embedded data baked into the validator image): - **A bundle-time-only `--set` is not honored.** `aicr bundle --set nodewrightcustomizations:tuningEnabled=false` is applied in the bundler and never persisted to the recipe; `aicr validate` has no `--set` flag and reads the recipe as-is, so it renders *with* the CR and still expects it. To suppress the assert on a deliberately-untuned cluster, the value must live in the recipe (overlay default or inline `overrides:`), not a transient bundle flag. (`aicr recipe` likewise has no value `--set`, so the overlay/inline override is the only channel.) `Overrides` resolved from a `--data` overlay *are* honored, because recipe resolution runs CLI-side and bakes them into the serialized recipe before the Job receives it. The one bundle-time flag that *is* persisted is `--dra-eviction-node-label`: the bundler writes `enabled: true` onto the `dra-node-labeler` ref in the `recipe.yaml` at the bundle root, which is why post-deployment validation (the UAT readiness gate and conformance run, and the documented user workflows) reads that file rather than the original recipe (#2848). - **`--workload-gate` is the exception, honored via the cluster.** The bundler writes the taint into the nodewright-operator values (`controllerManager.manager.env.runtimeRequiredTaint`), which the chart renders as the `RUNTIME_REQUIRED_TAINT` env on the controller-manager Deployment. The Go readiness check finds that Deployment by its `app.kubernetes.io/component=manager,control-plane=controller-manager` labels, so neither `fullnameOverride` nor `nameOverride` matters, reads the env from it (`runtimeRequiredTaints`), and gates on exactly that taint plus the legacy `skyhook.nvidia.com=runtime-required:NoSchedule` the operator still removes during its deprecation window, so an arbitrary key, value, or effect passed at bundle time is what the validator waits to see cleared. When the Deployment or env is absent it falls back to the two chart defaults. A list error or more than one matching Deployment fails closed, and the same cardinality check fails the operator's health check. - **`--data`-external files referenced by path are not readable in the Job.** A component whose `manifestFiles` or base `valuesFile` exist only in an external `--data` directory cannot be read by the embedded-only validator (a pre-existing constraint for `manifestFiles`; the render's `valuesFile` read shares the same embedded scope). It fails **closed** — an error, never a false pass. In practice `nodewright-customizations` ships no values file and uses inline `overrides:`, so there is no exposure here today. **Chart-shape constants are pinned to the charts (#2629).** The Go readiness checks hardcode facts that upstream charts define, not AICR: - the `NodeWright` CR's group, version, and resource; - the chart's default runtime-required taint; - the operator controller-manager Deployment's selector labels; - the DRA driver's `-kubelet-plugin` DaemonSet suffix. `validators/deployment/testdata/chart_contracts.yaml` records what the pinned charts actually render. Under `make test`, `TestChartContractsMatchRenderInputs` fails when a pin in `recipes/registry.yaml`, or the values the base overlay resolves for one of those charts, changes without the file being re-rendered (leaf-overlay value overrides are not covered), and `TestValidatorConstantsMatchChartContracts` fails when a constant disagrees with the file. After such a change, regenerate it (needs helm and network): ```bash AICR_UPDATE_GOLDEN=1 go test ./validators/deployment/ -run '^TestChartContractsMatchPinnedCharts$' -count=1 ``` Then update any constant the second test reports. Without `AICR_UPDATE_GOLDEN`, the same test checks the committed file against the live charts; like the nvsentinel render tests, it is skipped under `-short`. Operator behavior the chart does not render, such as which of two served kinds carries status, is outside this check. **Running:** ```bash make check-health COMPONENT=gpu-operator # one component make check-health-all # every registry-linked component make validate-local RECIPE=recipe.yaml # full pipeline in Kind ``` ### Timeout budgeting During `aicr validate --phase deployment`, registry health checks in `recipes/checks//health-check.yaml` run in-process inside the `expected-resources` check (`pkg/chainsaw/inprocess.go`). A Test's `spec.timeouts.assert` is the **whole-Test budget** — one deadline shared across every step and retry. Slurm's [`health-check.yaml`](https://github.com/NVIDIA/aicr/blob/main/recipes/checks/slinky-slurm/health-check.yaml) uses `assert: 7m` so workload-readiness steps can converge before the pod-phase guard runs. The authored value can only **shorten** the caller's budget, never extend it: the effective deadline is `min(spec.timeouts.assert, caller budget)`. The caller budget is `defaults.ChainsawAssertTimeout` for the deployment validator and the gate's `--timeout` (`defaults.ReadinessGateExecTimeout`, 2m) for a readiness Job. This restores what the removed chainsaw-exec path enforced by wrapping the subprocess in an independent outer `context.WithTimeout`; running in-process, the authored value had been substituting for the caller budget outright, letting registry- or integrator-authored content overrun `--timeout` by any factor it chose. No in-tree check changes behavior — every authored value is at or below its caller's budget — but a readiness test copied from a validator check (`assert: 5m`) is now capped at the gate's 2m per evaluation rather than overrunning it. The `expected-resources` catalog timeout (8m in `recipes/validators/catalog.yaml`) is the **outer** envelope for the check's own budget (`AICR_CHECK_TIMEOUT`). It must exceed the longest in-tree `assert` value plus headroom for pre-chainsaw work, chainsaw teardown, and log flush (`defaults.JobEnvelopeMargin`). If assert runs too close to that catalog deadline, the check's own context can expire before chainsaw reports the failing step — operators see truncated output instead of a useful failure. The Job's `activeDeadlineSeconds` (this catalog timeout plus `defaults.ValidatorJobDeadlineHeadroom`) is a separate, later backstop and no longer the deadline this margin protects against. Raise the catalog `timeout` in tandem when you need a longer assert budget (`TestExpectedResourcesCatalogEnvelope` guards this). ## Constraint evaluation algorithm `pkg/constraints` is shared by surface 1, surface 2's recipe constraints, and the readiness pre-flight gate. The evaluation flow: 0. **Name dispatch.** `constraints.Evaluate` first matches the constraint name against the node-set form `NodeTopology.gpu-nodes.label` ([#1755](https://github.com/NVIDIA/aicr/issues/1755)), which has its own value grammar and evaluator and never reaches the steps below. Every other name proceeds through the scalar flow. 1. **Parse.** `ParseConstraintExpression(expr)` strips whitespace, finds the **longest** matching operator prefix (so `>=` wins over `>`), splits into `{Operator, Value}`. Empty value → `ErrCodeInvalidRequest`. 2. **Classify.** Operators other than `Exact`/`EQ`/`NE` are always version comparisons. `EQ`/`NE` are version comparisons iff the value passes `looksLikeVersion` (starts with digit, has a dot, optional `v` prefix). Everything else is string. 3. **Evaluate** against the snapshot measurement. Version compares route through `version.Version.Compare` (semver-aware). String compares are case-sensitive equality. 4. **Errors propagate, not bools.** A value declared as `>= 1.32.4` that fails to parse as a version returns `errors.WrapWithContext(ErrCodeInvalidRequest, "cannot parse actual version", err, ...)` — not `false`. The caller (validator pre-flight gate) must surface this as a failed constraint, not a passing one. This is the fail-closed invariant. Tolerance and range semantics (e.g. NCCL's 10% slack) live in the **check** that produces the measurement, not in the operator. The operator vocabulary stays minimal on purpose. ## Testing checklist Patterns common to all four surfaces. - **`--no-cluster` is mandatory** for any test that touches `pkg/validator` or `aicr validate` outside an explicit live-cluster fixture. `validator.New(validator.WithNoCluster(true))` for unit tests; the `--no-cluster` CLI flag for e2e and chainsaw. When `NoCluster` is true, RBAC and Jobs are skipped, all checks report `skipped - no-cluster mode`, but constraints still evaluate. - **Table-driven tests.** Required for multi-case logic per CLAUDE.md. See `pkg/constraints/evaluate_test.go` and `pkg/bundler/validations/checks_test.go` for the canonical shapes. - **Synthetic inputs.** Component validations take a hand-built `RecipeResult` and `bundlerConfig`. Container checks take a `validators.Context` with `fake.NewClientset(...)`. - **Chainsaw against Kind.** `make check-health COMPONENT=` runs against the local Kind cluster set up by `make dev-env`. KWOK cannot host chainsaw checks that need real workloads — see [/aicr/contributor-guide/testing](tests.md#kwok-matrix-testing) for what KWOK does and doesn't cover. - **CTRF output.** Container checks emit JSON via the runner. Assert on status/message in integration tests, not raw stdout. ## Common pitfalls - **`slog.Warn; continue` on a constraint or `ValidationFunc` parse error.** Masquerades broken YAML as passing. Fail closed — return `ErrCodeInvalidRequest`. (CLAUDE.md anti-pattern.) - **Function-name typo in `registry.yaml` `validations:` block.** Fails closed — `RunValidations` raises `ErrCodeInvalidRequest` ("unknown validation function"). Add a registry-lookup test for every shipping function. - **`yaml.Marshal` on `map[string]any` for output that feeds CTRF or a digest.** `yaml.v3` walks randomized Go map order. Use `serializer.MarshalYAMLDeterministic`. - **Container check that requires a real GPU node profile.** KWOK fakes labels and topology but not GPU runtime. Gate such checks behind a `nvidia.com/gpu` resource check that lets KWOK runs Skip cleanly. - **Network calls in a component validation.** Bundle must work offline. Push to a container check or chainsaw check instead. - **Re-pushing the same image tag during dev (`:dev`).** K8s default `IfNotPresent` keeps the stale image on previously-pulled nodes. Suffix per iteration (`:dev-v1`, `:dev-$(git rev-parse --short HEAD)`). ## See Also - [/aicr/contributor-guide/recipes-overlays-and-mixins](recipe.md) — recipe overlays and the `validation:` block - [/aicr/contributor-guide/testing](tests.md#kwok-matrix-testing) — recipe matrix tests without GPU hardware - [Validator Extension Guide](/aicr/integrator-guide/validator-extension) — external validators via `--data` - [CLAUDE.md](https://github.com/NVIDIA/aicr/blob/main/.claude/CLAUDE.md) — anti-patterns: fail-closed gates, `slog.Warn; continue`, watch-over-poll, `--no-cluster` - [Validator V2 ADR](https://github.com/NVIDIA/aicr/blob/main/docs/design/002-validatorv2-adr.md) — container-per-validator architecture decision - [Validator Catalog](https://github.com/NVIDIA/aicr/tree/main/recipes/validators) — authoritative `catalog.yaml`