Validating a Cluster
Task-oriented walkthrough for running aicr validate against a GPU cluster — from
capturing a snapshot through interpreting results. Covers both training and
inference workloads and all three validation phases (deployment, conformance,
performance).
For per-flag reference, see CLI reference: aicr validate. For the architectural view of how snapshot + recipe flow into the validator, see Data flow: Stage 3 Validate.
When to validate
Under a GitOps deployer (Argo CD, Flux, Fleet), start aicr validate --phase deployment only after the sync of every wave has completed, not while it is
still progressing. The phase treats a declared resource the cluster lacks as a
deployment that was not applied, not as one that is not ready yet: a Nodewright
CR the recipe declares (shipped by nodewright-customizations, which syncs in
a later wave than the operator) that is absent fails expected-resources
immediately instead of waiting for the sync to deliver it.
Readiness pre-flight constraints (K8s version, OS, kernel) run implicitly before
any phase. If pre-flight fails, no validator Jobs are deployed. A submission run
(--cncf-submission) adds two pre-flights of its own once the snapshot is
loaded or captured and before any validator Job: the cluster’s Kubernetes minor
must match a carried checklist exactly (1.34 and 1.38 are refused alike, since
a submission certifies one minor) unless
spec.validate.evidence.cncf.conformanceVersion names one, which the
submission README then records as selected explicitly; and the recipe’s
validation.conformance.checks must declare a complete proof path for every
applicable MUST of that checklist (secure-accelerator-access or the Slinky
Slurm alternative slinky-slurm-gpu-access; a conditional MUST declared under
spec.validate.evidence.cncf.notApplicable is not required). A recipe that
cannot establish a MUST is refused with INVALID_REQUEST naming the
requirements and the checks that would close each gap, since no run of it can
establish the submission. A plain run that includes the conformance phase logs
the same gaps as a warning and proceeds.
The workflow
- Snapshot — capture current cluster state (K8s / OS / GPU / topology) once.
- Recipe — generate the target configuration for your workload (training vs inference, platform, accelerator).
- Validate — run one or all phases against the snapshot and live cluster.
Which recipe to validate
Every bundle writes the recipe it was generated from to recipe.yaml at the
bundle root, after dropping the components it did not render. Bundle-time
--set values are not written back to it; the one bundle-time decision that is
persisted is the dra-node-labeler enablement under --dra-eviction-node-label.
That file is the effective component inventory of what was deployed, so
validation of a deployed cluster reads it. The examples below assume the bundle
was written with aicr bundle --output ./bundles, as in
Generating Bundles:
The original recipe is the input for the pre-deploy dry run (--no-cluster,
below): the bundle does not exist yet, and the dry run evaluates what the recipe
asks for, not what a bundle delivered. Validating the original recipe after
deployment still works, but it cannot see bundle-time decisions: a component the
bundler dropped is still declared (its health check is skipped when the
component’s own values leave a trace, and reports NOT_FOUND otherwise), and
a component the bundler enabled at bundle time is not checked.
Prerequisites
aicrCLI installed (see installation).kubectlconfigured for the target cluster (validator dispatches K8s Jobs; pre-flight only needs the snapshot).- Effectively cluster-admin for the identity running
aicr validate. Each run creates a ServiceAccount for its validator Jobs and binds it to the built-incluster-adminClusterRole through a per-runaicr-validator-<run-id>ClusterRoleBinding, and Kubernetes RBAC only lets you bind a role whose permissions you already hold. The run deletes the binding at cleanup;--no-cleanupleaves it active until you delete it yourself. AICR creates its ownaicr-validationnamespace on first run. To run without that grant, have an administrator provision the ServiceAccount instead; see Running under a pre-provisioned ServiceAccount. - AKS profiled recipes: the readiness pre-flight re-evaluates the recipe’s profile constraint (
K8s.aks-gpu-pools.gpu-driver), so the snapshot must carry that reading — capture it withaicr snapshot --aks-gpu-pools <az dump>, or pass the same flag toaicr validatewhen it captures live. A snapshot without the reading fails readiness closed (exit 2). - GKE recipes: the readiness pre-flight re-evaluates the recipe’s
gpuStackprofile constraint over the GPU-node set (nodes carryingcloud.google.com/gke-accelerator): the defaultgke-defaultvalue requires that no GPU node carries the opt-out labelgke-no-default-nvidia-gpu-device-plugin(GKE’s managed plugin stays thenvidia.com/gpuadvertiser), whilebundle-installeranddrarequire every GPU node to carrygke-no-default-nvidia-gpu-device-plugin=true(so the GPU Operator’s plugin — or, underdra, the NVIDIA DRA driver’s ResourceSlices — is the sole whole-GPU advertiser). The check fails closed (exit 2) on labels contradicting the selected value, mixed labels, malformed or ambiguous label readings, a snapshot with no identifiable GPU nodes, and when--max-nodes-per-entryactually truncated a participating label reading (a truncated node list cannot prove set membership — regenerate with no cap set, from the flag,AICR_MAX_NODES_PER_ENTRY, orspec.snapshot.execution.maxNodesPerEntry; a cap larger than the node count truncates nothing and validates normally).gke-defaultneeds no provider projection flag. Its constraint reads node labels the standard snapshot already carries.bundle-installeranddraadditionally re-evaluateK8s.gke-gpu-pools.gpu-driver-installation, corroborating that every GPU pool was actually created withgpu-driver-version=disabled. That reading requires a snapshot captured with--gke-gpu-pools <gcloud dump>(or passed toaicr validate --gke-gpu-pools), and without it is unavailable, so those values fail closed.draalso re-evaluates its ownK8s.server.version >= 1.35.0floor. See GKE GPU Setup for the full setup and the qualification matrix. - Run:ai recipes (
platform=runai): thegang-schedulingconformance check runs its test pods in a Run:ai Project namespace, so exportAICR_RUNAI_PROJECT_NAMESPACE=runai-<project>in the shell runningaicr validate; the check fails, rather than skips, without it. The Project needs an explicit CPU and memory quota and no CPU or memory limit, and a private probe image needs its pull Secret already in that namespace. See Run:ai cluster. NodeTopology.gpu-nodes.labelreadiness constraints outside GKE (including bare metal and--dataservices): GPU nodes are those the GPU Operator labelsnvidia.com/gpu.present=true, so capture the snapshot after it is running. A snapshot with no such node fails readiness closed (exit 2), and GPU nodes it has not labeled are not checked.
ai-service-metrics on your own metrics backend
By default the ai-service-metrics conformance check queries NKTP’s bundled
otel-lgtm. When NKTP exports to your own backend, or the recipe has no
nktp, name that backend’s Prometheus query API in the shell running
aicr validate:
- The value is a base URL; the check appends
/api/v1/query. It wins overotel-lgtmwhenever it is set. - The validator pod must reach the URL without credentials. A URL that
carries a user or password fails
aicr validatebefore any Job is created, when the run includes this check, because the URL is written into the check’s Job and recorded in its evidence. - The backend must hold what your monitoring system scrapes from a
ServiceMonitor in the check’s test namespace, including each target’s
upseries.
Without the variable, the check is skipped on such a cluster and the CNCF requirement is reported as not tested.
Training performance validation
Training performance runs an NCCL all-reduce benchmark — a Kubeflow TrainJob
that runs all_reduce_perf across GPU nodes and measures aggregate bus
bandwidth. Three check variants are available; the recipe picks the one (or
ones) that match the target fabric:
What the bandwidth number describes. Each nccl-all-reduce-bw* result
carries a runtimeSource label: delivered-artifact means the benchmark
runtime was derived from the ClusterTrainingRuntime the recipe ships, so the
number attests to the delivered wiring; recipe-supplied-runtime means the
recipe supplied the runtime itself; cluster-capability means the validator’s
own fixture was measured — proof the fabric can reach the floor, not proof of
what the recipe ships. Only h100-gke-cos-training-kubeflow produces
delivered-artifact today. The label is decided from the recipe, not from what
is installed on the cluster.
The applicability column is the default, derived from the recipe’s
criteria. A recipe whose criteria fall outside it can still run these
benchmarks explicitly — either by
borrowing an embedded benchmark profile
whose fabric matches its hardware, or, for a private service whose fabric
matches no embedded template, by
supplying its own benchmark runtime.
Overriding the NCCL workload image with AICR_NCCL_RUNTIME_IMAGE. Each
embedded nccl-all-reduce-bw / -net / -nvls template pins a specific
launcher/worker workload image — the CUDA/NCCL/MPI/SSH/transport runtime that
all_reduce_perf actually runs in, distinct per platform (e.g. GKE H100/TCPXO
ships a CUDA 12.9 image today). To qualify a different CUDA/NCCL combination —
for example CUDA 13 on GKE TCPXO with an R580-or-newer driver — set
AICR_NCCL_RUNTIME_IMAGE=<image ref> in the aicr validate environment. The
resolved image is rendered into every container that carries the workload
(the launcher’s SSH-setup init container and both the launcher’s and worker’s
main containers), so a run can never end up on a mixed image set; a
platform-specific sidecar unrelated to the NCCL workload itself (e.g. GKE’s
tcpxo-daemon transport daemon) is left untouched. A malformed image
reference fails the check immediately, before cluster discovery or NCCL
benchmark resources are created, rather than silently falling back to the
compiled default.
This is a different setting from aicr validate --image /
AICR_VALIDATOR_IMAGE_* (see Validator image
tags): those control the
validator’s own container image (the snapshot/orchestration binary),
never the inner NCCL workload. AICR_NCCL_RUNTIME_IMAGE only applies to the
three NCCL all-reduce checks, and only to the templates AICR owns — the
embedded per-platform template and a runtime derived from a delivered
artifact — it has
no effect when a recipe supplies its own benchmark
runtime, since that
runtime already owns its image end to end. For reproducible qualification
runs, prefer pinning by digest (name@sha256:...) over a mutable tag.
Choosing the self-installed Kubeflow Trainer release. When the cluster has
no Kubeflow Trainer and the recipe does not enable kubeflow-trainer, the NCCL
checks install a temporary Trainer from GitHub and remove it afterward. The
release defaults to the kubeflow-trainer pin in recipes/registry.yaml. When
the recipe has no kubeflow-trainer ref, set AICR_NCCL_TRAINER_VERSION in
the env block of each NCCL catalog entry that runs (for example through a
--data overlay) to choose a different release without rebuilding the
validator image. A recipe that disables kubeflow-trainer installs that ref’s
version and ignores AICR_NCCL_TRAINER_VERSION. aicr recipe fills in the
registry pin for a ref with no version, so set the ref’s version in the
overlay to choose a different release for such a recipe. The value is a
semantic version, with or without a leading v. The check fails before any
Trainer resource is applied when the value is malformed, when the tag
has no GitHub release, or when the release’s manifests lack what the installer
needs. That means TrainJob and TrainingRuntime CRDs serving the API version the
benchmark creates, the Trainer admission webhook configurations, and the
controller Deployment. A Trainer that is already installed is used as-is,
whatever its version, and the value is not checked.
GB200/EKS recipes (both training and inference intents) enable -net and
-nvls together rather than the auto-detect variant, because those nodes
expose two inter-node fabrics simultaneously and a single auto-detect test
would only exercise one of them.
GB200/OKE training recipes follow the same pattern and enable -net and
-nvls together, with the same >= 40 / >= 500 GB/s floors as GB200/EKS.
On OKE, -net exercises the NVL72 rack’s InfiniBand east-west fabric
(rdma0-3, advertised as nvidia.com/mlnxnics by the recipe’s
rdmaSharedDevicePlugin NicClusterPolicy) over NCCL’s built-in IB/verbs
transport — no EFA plumbing is involved. Both variants were validated
end to end on a BM.GPU.GB200.4 NVL72 rack.
The GB300/OKE training recipe enables -net and -nvls with >= 300 /
>= 700 GB/s floors. Its -net runs NCCL’s built-in IB/verbs transport over
RoCE: each worker pod requests the node’s four SR-IOV VFs
(nvidia.com/mlnxnics) and attaches them through the recipe’s rdma-net
network, and NCCL reads the node topology that nvidia-tuned writes to
/etc/nccl/topo.xml. On BM.GPU.GB300.4 nodes with NCCL 2.29.2, the 16 GiB bus
bandwidth measured 363 GB/s (NET) and 836 GB/s (NVLS) on 2 nodes, and 376 GB/s
and 909 GB/s on 4.
The generated recipe lists the selected variant(s) under
validation.performance.checks with a platform-tuned bandwidth constraint
(example: >= 300 GB/s for H100 + EFA; >= 40 GB/s NET and >= 500 GB/s
NVLS for GB200 + EFA, each sized for a 2-node pair).
Node-shape assumption. These bus-bandwidth floors are fixed absolute values calibrated on full, high-bandwidth nodes (8-GPU H100 NVLink/SXM with multi-NIC transport). They are not normalized for node fabric or GPU count, so a smaller or different-fabric H100 SKU (e.g. a single-GPU-per-node shape) can false-fail a healthy run. Making the NCCL gate fabric/transport-class aware is tracked in #1256.
Expected flow (~5–10 min per variant): readiness pre-flight → deploy
TrainingRuntime + TrainJob in a per-run namespace named
aicr-nccl-perf-<variant>-<run-id> → worker pods reach Running → run
all_reduce_perf → parse peak bus bandwidth → verify the intended transport
actually carried traffic (for -net / -nvls) → compare to recipe
constraint (10 % tolerance) → cleanup.
Each variant (-net, -nvls, default) gets its own namespace, deleted on
exit along with its TrainingRuntime/TrainJob. A namespace left behind by
an interrupted run is reclaimed by a same-run-ID retry only if no
Pending/Running/Unknown pod remains in it and its execution lock (a
Lease named aicr-nccl-run-lock in that namespace) has gone stale, 20
minutes past its last renewal. Otherwise the retry fails with a conflict
rather than risk two executions sharing one namespace; check
kubectl get pods -n <namespace> and the Lease’s age to find and clear
whichever is actually stuck. A standalone run (no run ID) is never retried
into its old namespace; that namespace is only pruned automatically, on a
later run once an hour has passed and both conditions above hold.
A passing CTRF entry:
Note: this guide does not yet list per-platform expected-bandwidth baselines (EKS + EFA, GKE + TCPXO, AKS, etc.). The recipe’s constraint value is the current pass/fail floor; measured values above that floor are treated as passing regardless of platform.
To run deployment validation first (recommended — verifies GPU Operator, DRA driver, and Kubeflow Trainer are installed and healthy before the benchmark):
Grace Blackwell NET preflight: GPUDirect RDMA prerequisites
Before running nccl-all-reduce-bw-net on GB200 (EKS, OKE or NVKE) or GB300 (EKS), a
preflight checks each GPU node for the driver-side prerequisite of GPUDirect
RDMA. Without it NCCL falls back to the Socket transport. The -net check
catches that on its own — it fails on a Using network Socket banner rather
than reporting a figure — so the preflight exists to fail fast, naming the
driver, instead of after a full benchmark run.
The preflight covers EFA on EKS and built-in IB/verbs on OKE and NVKE for GB200, and EFA on EKS for GB300. GB300 on OKE (RoCE) ships R595 in the node image and is not preflighted.
Before R595 — which includes 580.173.02, the version AICR pins — the
driver must be loaded with NVreg_GrdmaPciTopoCheckOverride=1. Without it the
driver refuses to let a PCIe-attached NIC (EFA on EKS, ConnectX IB on OKE and NVKE)
attach dma-buf handles to GPU memory, and the kernel logs:
Set the parameter according to who owns the driver:
Deleting the nvidia-driver DaemonSet pods does not apply the change, and
neither does editing only the ConfigMap. The reload decision is keyed off the
ClusterPolicy spec, so the spec itself has to change. Setting
kernelModuleConfig.name is such a change, but AICR’s GB200/GB300 overlays
already set it — where it is present, point it at a differently-named ConfigMap
so the spec actually differs. Confirm on a node afterwards:
On R595 and later the parameter no longer exists, and setting it has no
effect because the kernel silently ignores unknown module options. R595 replaced
it with a PCIe topology requirement the preflight cannot check, so validation
fails closed instead of assuming. On EKS p6e-gb200/gb300 that requirement is
known not to be satisfied — the measurement is recorded alongside the driver pin
in recipes/components/gpu-operator/values.yaml; on OKE it is unmeasured.
The remedy on R595+ is a driver at R580 — AICR ships 580.173.02 — pinned
through the ClusterPolicy where the GPU Operator owns the driver, or through the
node image where it does not.
Undetermined is a third outcome, reached without either verdict above: if
/proc/driver/nvidia/version or params cannot be read, or the version banner
does not parse, the preflight reports the state as undetermined rather than
assuming one. SELinux denying the read inside the container, or a
driver-container remount leaving the path empty, produces this. Changing the
driver version does not address it — read the file on a target node to see
whether it is unreadable or carries an unrecognised banner.
Driver not loaded is a fourth outcome, reached before any file is read: when
/proc/driver/nvidia does not exist on a target node, the kubelet refuses the
probe pod’s hostPath mount. If the preflight reads the kubelet’s
hostPath type check failed event and it is still the probe pod’s latest
event, it names that node as driver-absent instead of sending you to set the
flag. Otherwise (the events cannot be read, none matches, or a later event
for the pod has superseded it) it returns the original wait timeout. There
is no parameter to set until a driver is loaded. Finish the driver rollout (GPU
Operator nvidia-driver DaemonSet Ready, or a node image that ships the driver
on OKE oci-managed) and re-run. The kubelet leaves the pod in
ContainerCreating, so this surfaces after the probe’s wait (up to two
minutes), and it stops the check at the first such node rather than listing
every affected node.
Opting external recipes into a benchmark profile
The default applicability above is keyed to service + accelerator pairs the
validator ships templates for. A recipe whose criteria fall outside that set —
typically a criteria.service registered only through an external
--data directory, or an embedded service
extended to a new accelerator — would otherwise report the NCCL checks as
skipped even when its validation block declares them. Such a recipe opts
into an embedded benchmark with the nccl-benchmark-profile performance
constraint (a bare {accelerator}/{service} value, no comparator):
The profile selects the benchmark’s template and environment handling —
transport-class runtime template, EFA/TCPXO discovery, worker scheduling
defaults, and preflight checks — as if the cluster were the named pair, so
pick the profile whose fabric matches the hardware. GPU nodes are still
matched by the recipe’s own criteria.accelerator (via the GFD
nvidia.com/gpu.product label) when a matcher exists, but the profile’s
service keys the rest of node handling: an …/eks profile narrows workers
to a single node.kubernetes.io/instance-type, and the …/any profiles
(b200/any, gb200/any) require an explicit --node-selector identifying
the GPU nodes, exactly as service: any recipes do. When --node-selector
is passed it replaces the automatic filters rather than narrowing them.
Valid profiles are the pairs in the applicability table above: b200/any,
gb200/any, gb200/eks, gb200/gke, gb200/nvke, gb200/oke, gb300/eks, gb300/generic,
gb300/nvke, gb300/oke, h100/aks, h100/eks, h100/gke, h200/eks, vr200/nvke, vr200/rke2. A
malformed or unknown value fails the check rather than silently skipping
it. A valid profile that doesn’t implement a requested variant (e.g.
gb200/eks with the auto-detect nccl-all-reduce-bw check) skips just that
check with a message naming the profile. When a profile is set it wins over
criteria-derived applicability, and the pass/fail thresholds stay in the
same-named check constraints as always.
Supplying a benchmark runtime for a private service
A profile only helps when an embedded template already matches the
cluster’s hardware and fabric. A private service introduced entirely through
--data — a criteria.service (and possibly criteria.accelerator) the
validator ships no template for, with a fabric that reuses none of the
embedded ones — has no compiled pair to point a profile at. Such a recipe
supplies the benchmark itself: it ships a Kubeflow TrainingRuntime as a file
in its --data tree and references it with the nccl-benchmark-runtime-ref
performance constraint (a bare {accelerator}/{service} value). aicr validate
reads that file and gates the benchmark keyed on the recipe’s own criteria,
with no compiled applicability entry required.
Lay the runtime out exactly where the embedded templates live, so it is a drop-in for upstreaming later:
The supplied runtime owns its fabric wiring end to end: because the recipe
opted in explicitly, the validator bypasses the compiled applicability gate and
skips every service-specific setup step — EFA/TCPXO/RDMA NIC discovery, the
GB200-NVreg / GKE-TCPXO preflights, and NVLS/IMEX auto-provisioning (except as
described for IMEX below). It renders the runtime, sizes it (against the
runtime’s own nodeSelector when it pins one, so WorkerCount matches
placement), applies it alongside the shared TrainJob, and evaluates the
bandwidth threshold from the launcher logs.
Transport verification is not skipped: pairing the runtime with the -net
or -nvls check still asserts that transport actually carried traffic (the NCCL
markers are fabric-agnostic), so a named variant can’t pass on bandwidth alone.
What it must honor:
-
It must be a Kubeflow
TrainingRuntime(apiVersion: trainer.kubeflow.org/v1alpha1); any other kind/apiVersion is rejected. The validator force-sets its name and namespace, so a recipe can supply aTrainingRuntimeand nothing else — never an arbitrary resource. -
It must declare a
nodereplicatedJob (the worker cohort the sharedTrainJobsizes and the validator injects scheduling into and reads logs from), mirroring the embedded templates. -
The generic template variables
${NAMESPACE},${WORKER_COUNT},${GPU_COUNT_PER_NODE},${GPU_COUNT},${TEST_TYPE},${MIN_MESSAGE_SIZE}, and${MAX_MESSAGE_SIZE}are substituted; the service-specific ones (EFA/RDMA/GKE) are not, so the runtime must pin its own fabric resources. -
The runtime should pin its own worker
nodeSelector/tolerations, or pass--node-selectortoaicr validate, so workers land on the intended GPU nodes. -
IMEX access uses the validator-managed claim template. The benchmark runs in a per-run namespace that does not exist before the run, so nothing can be pre-created there for the runtime. To get an IMEX channel, reference the ResourceClaimTemplate
nccl-all-reduce-imexfrom the pod’sresourceClaims, and bind that claim in thenodecontainer’sresources.claims. Kubernetes exposes a pod-level claim only to containers that bind it, so the pod-level reference alone leaves the worker without the channel. The validator then creates the ComputeDomain that backs it, on any check variant, and waits for the template before starting the run. -
Any other pod-level
resourceClaimTemplateName, and anyresourceClaimName, is rejected: no such claim could exist in the per-run namespace.
nccl-benchmark-runtime-ref and nccl-benchmark-profile are mutually
exclusive — a recipe supplies its own runtime or borrows an embedded one,
never both. An absent or blank ref falls back to criteria/profile-derived
applicability. A malformed ref, or a file missing from --data, is rejected by
aicr validate before any validator Job is deployed (a resolution error on
stderr). A resolved file that is not a TrainingRuntime with a node
replicatedJob, or that references an unsupported claim, fails the check
itself (in the pod), before any benchmark resources are applied. Either way it fails
rather than silently skipping.
Sizing note. The worker cohort is sized against the runtime’s own
nodeSelectorwhen it pins one, but richer scheduling constraints (nodeAffinity, un-tolerated node taints) are the Kubernetes scheduler’s job, not the validator’s — a runtime that restricts placement below the sized cohort surfaces as a launcher timeout, not a false pass. Pass an explicit--node-selectorwhen you want the sized and scheduled cohorts to match exactly.
Upstreaming. Because the file already sits at
validators/performance/testdata/{accelerator}/{service}/runtime.yaml, adopting
it into AICR is a copy: move the file into the repo, add the accelerator/service
tuple to the compiled supportedNCCLCombinations matrix, and drop the
nccl-benchmark-runtime-ref constraint — the benchmark then runs from the
recipe’s plain criteria like any other embedded one.
Cluster Readiness Engine checks
An opt-in pair of Cluster Readiness Engine (CRE) checks runs against public CRE
(nvcre.nvidia.com, cluster-readiness-engine).
Neither check branches on service or accelerator; which combinations are
qualified is recorded as data. Shipped overlays keep the TrainJob
nccl-all-reduce-bw path, so nothing below runs unless you ask for it.
Enabling a CRE check takes two steps and needs both. Add the nvcre component
as described in
Enabling NVCRE — a bare component
reference without that section’s valuesFile and Trainer dependency hits the
failures documented there — and then declare the check with a same-named
constraint:
Both constraints are floors. The checks compare the measured value with >=
and refuse any other operator rather than evaluating it as a floor anyway, so a
constraint they cannot honor fails before any GPU work starts. The bandwidth
check applies the constraint the way nccl-all-reduce-bw does: it passes at 90%
of the value, so the same >= 300 means the same under either backend. The
goodput check passes at the value itself. Note the goodput units: the measurement is a ratio between 0 and 1, so a constraint written as a
percentage (>= 50) can never be satisfied.
The catalog entry each check drives, and the node footprint it runs on, come
from the qualification record rather than the check: for
nccl-cre-all-reduce-bw on eks x h100, communication/nccl-all-reduce on
two nodes, and for cre-training-goodput on eks x h100,
training/nemotron5-8b on two nodes. A combination that no entry records as
qualified skips with an unsupported-target reason.
The record also pins each entry’s workload image by digest, and the check
submits it as the category’s options.image, so the run pulls the bytes the
closure records rather than whatever the catalog’s tag points to. The pins
are the images the catalog selects on eks x h100, at the digests
recipes/components/nvcre/workload-images.yaml records for the CRE version
the registry pins, and the checks submit them whatever CRE version the cluster
runs: the EFA nccl-tests image for nccl-cre-all-reduce-bw, and
nvcr.io/nvidia/pytorch:25.08-py3 for cre-training-goodput. With the pin,
the NCCL launcher’s SSH-permissions init container also runs nccl-tests
rather than nvcr.io/nvidia/pytorch:26.01-py3, so the NCCL check no longer
pulls that image. A mirror for a disconnected cluster has to serve these two
images at the recorded digests.
Relation to the TrainJob check
On eks x h100, nccl-cre-all-reduce-bw
reads 1.65 GB/s (0.34%) higher than nccl-all-reduce-bw on the same two nodes:
six back-to-back pairs on NVCRE v0.8.2 gave 486.76 to 489.01 GB/s for the
TrainJob check and 489.63 to 490.03 GB/s for the CRE check, with a per-pair
offset of 0.92 to 3.14 GB/s. Both ran over Using network Libfabric; the CRE
check asserts that from its launcher log, and the TrainJob check’s default
template logs at NCCL_DEBUG=WARN so it cannot show it. That is the one
difference in what the two checks demand: the CRE check fails a run that fell
back to Socket, and the TrainJob check does not look. Both pass at 90% of the
constraint value, so a recipe can carry the same >= 300 for either backend
and both use the same floor. The two measured bandwidths differ by 0.92 to
3.14 GB/s per pair (1.65 GB/s on average), so a cluster that lands near the
floor can still pass one check and fail the other. The CRE check applied the
value without the 10% allowance before this was aligned; a CRE constraint that
had been written as 0.9 times the TrainJob value to compensate should be reset
to the TrainJob value. The offset was measured on one cluster, and no run
landed near the boundary, so it does not show that the two verdicts agree at
it.
The goodput figure depends on the step count, because startup time counts against it. The step count comes from the NVCRE catalog entry, so a figure belongs to the CRE version that produced it: 0.5723 on two nodes with the pinned v0.8.2 (462.05 TFLOPS per GPU), and 0.5835 on v0.6.0, both measured on a NodeWright-tuned cluster. Both are single measurements, so set the constraint as a floor below the value your cluster reaches, not at it. Run it on nodes that completed NodeWright tuning: on a cluster that had not, with the distro EFA driver, no hugepages, and an 8 MB memlock limit, a two-node run hung in NCCL initialization. Which of those differences causes it was not isolated.
Reclaiming an interrupted run. Both checks delete their Certification and
confirm its TrainJobs and workload pods are gone, but a validator pod killed
mid-run (eviction, OOM kill, its Job deleted under it) never gets to. Its
workloads keep holding the GPU nodes, and because Certification names carry a
random suffix, nothing in the name identifies them. Every Certification AICR
submits therefore carries
app.kubernetes.io/managed-by: aicr-validator, and the next CRE check in that
namespace tears down any it finds older than 45 minutes — longer than a live run
can take, so a concurrent validator’s Certification is never swept. To clean
up by hand instead:
Inference performance validation
Inference performance runs the inference-perf check — deploys a
DynamoGraphDeployment with a vLLM-served model (Qwen/Qwen3-8B by default,
overridable per accelerator — see below) plus an AIPerf benchmark Job, and
measures end-to-end output-token throughput and time-to-first-token (TTFT) p99.
Warm-up: AIPerf sends a wave of warm-up requests before the measured run,
so vLLM’s one-time CUDA-graph / JIT compilation (tens of seconds on a cold
worker) is excluded from the reported throughput and p99 TTFT — the numbers
reflect steady state, not cold start. Warm-up scales with concurrency and is
tunable via AICR_INFERENCE_PERF_WARMUP_PER_CONCURRENCY (see the
validator reference).
Determinism: the benchmark is driven reproducibly so the verdict reflects the
deployment, not run-to-run RNG — a fixed random seed, fixed input/output token
counts (stddev 0), a pinned synthetic-prompt pool, and greedy decoding
(temperature: 0). Note that throughput (not the latency tail) is the stable,
discriminating signal at high concurrency; TTFT p99 near the saturation knee can
still vary with batching/scheduling, which is why the TTFT constraint is a
generous ceiling rather than a tight target.
The generated recipe includes dynamo-platform in componentRefs and lists
inference-perf under validation.performance.checks with pass/fail
constraints plus benchmark inputs:
Node-shape assumption. The inference-throughput floor is a fixed
absolute full-node value calibrated on a full node (the shared >= 50000
gate was measured on 8-GPU H100; GB200 on a 4-GPU node). It is not
normalized for GPU count, and the evaluator only scales it down for partial
occupancy — not across node sizes — so a smaller H100 SKU (e.g. 1-/2-GPU
shapes such as p5.4xlarge, AKS NC80adis) can false-fail a healthy run.
inference-ttft-p99 is a per-request latency at fixed concurrency-per-GPU
and does not need GPU-count normalization. A normalized per-GPU throughput
floor is tracked in #1254.
inference-model and inference-concurrency-per-gpu are optional: omit them to
use the compiled defaults (Qwen3-8B at 256 concurrent requests per GPU), set them
per overlay to tune model and load for each accelerator, or override globally
with the AICR_INFERENCE_PERF_MODEL / AICR_INFERENCE_PERF_CONCURRENCY_PER_GPU
catalog knobs (recipe wins over catalog env wins over default).
inference-routing-mode selects the Dynamo Kubernetes routing path. The
default dynamo-router mode deploys a Dynamo frontend with load-aware
least-loaded routing (DYN_ROUTER_MODE=least-loaded), which balances by each
worker’s active in-flight load so a transiently-slow worker stops receiving its
full share (see issue #1197). Normal frontend-to-worker request/response traffic
uses Dynamo’s request plane (Dynamo 1.4+ defaults to TCP); AICR does not set
DYN_REQUEST_PLANE=nats. Workers publish KV-cache events directly over ZMQ
with no NATS relay, but least-loaded routing does not consume those events —
only DYN_ROUTER_MODE=kv does. Set it to gateway-epp
to exercise GAIE/EPP: the validator deploys an EPP component, worker frontend
sidecars in direct mode, and an HTTPRoute through the recipe’s inference
gateway (agentgateway by default; a recipe whose components include no
inference gateway fails the run before anything is deployed). The direct-mode sidecars honor EPP routing headers; they do not
relay KV events.
Model-weights cache and AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS. The benchmark downloads
the model once into a PVC and serves all workers from it (on by default;
avoids per-IP Hugging Face throttling). The cache PVC needs a StorageClass: it
uses the cluster’s default StorageClass unless you set one, with
precedence recipe constraint > catalog env > cluster default. Set it
per accelerator via the inference-model-cache-storage-class performance
constraint, or globally via
AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS=<name> (e.g. gp2/gp3 on EKS,
standard-rwo on GKE, except A4X/GB200 a4x-highgpu-4g nodes, which reject
standard-rwo’s pd-balanced disks and need a Hyperdisk-backed class
instead. See GKE GB200 Storage Prerequisites)
on the inference-perf catalog entry’s env (or via a
catalog overlay in the aicr validate --data <dir> directory). On a cluster
with no default StorageClass (common on EKS, since some clusters ship
only a non-default gp2) and neither set, the check fails fast in
seconds with guidance rather than hanging. Disable the cache instead with
AICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off. Unlike the recipe constraint, the
catalog env knob is not read from the shell environment of the process
running aicr validate (only HF_TOKEN is). AICR-deployed EKS clusters get a
default gp3 StorageClass from the aws-ebs-csi-driver component, so the
cache works there with no knob, provided the driver has AWS credentials — see
EBS CSI Driver Credentials.
Debugging a failed run with AICR_INFERENCE_PERF_NO_CLEANUP. By default the
validator deletes the per-run namespace (DGD, workers, frontend, AIPerf Job) on
both success and failure. To investigate a failure — e.g. a timed out waiting for inference endpoint to serve requests — set AICR_INFERENCE_PERF_NO_CLEANUP=1
and the validator leaves everything in place so you can kubectl logs the
frontend/workers and curl /v1/models and /v1/chat/completions live. Unlike the
other AICR_INFERENCE_PERF_* knobs, this one is read from the shell
environment of the process running aicr validate (forwarded to the
inference-perf pod, like HF_TOKEN), not from the catalog. Debug-only: you must
delete the aicr-inference-perf-<suffix> namespace manually afterward, or it
keeps GPU workers running.
Expected flow (~5–7 min on H100): readiness pre-flight → deploy a
DynamoGraphDeployment in a per-run namespace
aicr-inference-perf-<8-hex-suffix> → wait for state=successful (image pull
- model load) → endpoint readiness via a real
/v1/chat/completionsinference request (stricter than a/healthprobe, which returns 200 before the model can actually serve) → AIPerf benchmark Job parses throughput + TTFT p99 → compare to recipe constraints (10 % tolerance) → cleanup. Worker GPU wiring is configuration-selected and capability-verified for recipe-backed runs — the recipe’s resolved allocation policy picks the mechanism, and the inspection probe (the same one the conformance checks use) verifies the cluster serves it, failing closed on mismatch; only recipe-less standalone runs (unspecified) select by capability. The wiring supports exactly two GPU allocation configurations: (1) nodes publishing node-localgpu.nvidia.comResourceSlices (full-GPU DRA — the experimental opt-in; stock recipes default to device-plugin allocation since the #1327 flip), where workers bind a DRAResourceClaimTemplatesized from the validated per-node device count — kai-scheduler treats nodes bearing raw node-local GPU ResourceSlices as DRA-only and rejects scalar GPU requests, so the claim path is the only schedulable wiring there; and (2) ComputeDomain-only / no full-GPU slices (device-plugin nodes — the production default), where workers request GPUs vianvidia.com/gpulimits, which need nogpu.nvidia.comDeviceClass. Anything outside those two states fails fast with an actionable error instead of guessing: full-GPU ResourceSlices from a non-NVIDIA GPU driver, allocatedResourceClaims requesting a non-NVIDIA “gpu”-named DeviceClass,gpu.nvidia.comslices using non-node-local topologies (nodeSelector/allNodes/per-device node selection), pools published by node-local slices of multiple nodes, and allocatedResourceClaims from pools no node-local slice publishes (occupancy is attributed to nodes through the slices’spec.nodeName— the K8s API does not require pool names to be node names) are all rejected — these are outside the NVIDIA DRA driver’s supported configuration (#1327); generalized topology/driver support is tracked in #1652. When the recipe configures a GPU allocation policy (see Configured GPU allocation policy), the wiring is forced instead of capability-selected, and underdra-resource-claimcandidate GPU nodes are discovered from the probe’s validated DRA facts rather than scalarnvidia.com/gpuallocatable — so DRA-only nodes (device plugin disabled) are discoverable; full DRA-onlyaicr validateremains experimental (see the #1327 graduation checklist). The chosen node and wiring mode are recorded in the check’s evidence output.
All Dynamo Frontend and worker pods pin to a single GPU node via
kubernetes.io/hostname for a stable per-node baseline. On a shared cluster
where some GPUs on a candidate node are already in use by other workloads,
the validator picks the candidate with the most free GPUs within the
node’s own allocation ledger and sizes the benchmark to that count: DRA
ResourceClaim allocations subtract from DRA capacity, and device-plugin
nvidia.com/gpu occupancy from device-plugin capacity. A DRA-wired
candidate carrying any scalar nvidia.com/gpu workload is skipped
entirely — the device plugin’s physical device assignments are invisible to
the DRA allocator, so claims placed there could double-book a plugin-held
GPU. The check therefore does not need an explicit hostname override to
avoid saturated nodes. The inference-throughput
gate is a full-node baseline, so when the benchmark runs on fewer than the
node’s full GPU count the gate is scaled down by the same freeGPUs / nodeGPUs
fraction (throughput scales ~linearly at fixed concurrency-per-GPU) — a healthy
per-GPU result on a partially occupied node is not failed against a full-node
number. TTFT p99 is a per-request latency and is not scaled. Concurrent
aicr validate invocations are isolated from each other by the run-specific
suffix on both the namespace and the inner AIPerf Job name.
A passing CTRF entry (measured on EKS H100, 8 × H100 GPUs, Qwen/Qwen3-8B at 256 concurrency/GPU):
The RESULT: prefix on the first two lines is the contract documented in
pkg/validator/validator.go — any check that wants its summary lines echoed
to the CLI’s own output (not just the CTRF report) opts in by emitting that
prefix. The validator runtime strips the prefix when echoing; the full
prefixed line stays in stdout[].
To run deployment validation first (recommended — verifies GPU Operator, DRA driver, Dynamo operator, KAI scheduler, and supporting components are installed and healthy):
Skip scenarios
The inference validator has three explicit skip guards so it never runs where
it can’t succeed. Each produces a status: skipped CTRF entry with a specific
reason. Skipped checks are not failures: the validator container exits
with code 2 internally (mapped to CTRF skipped), but aicr validate itself
exits 0 for a skipped or passed phase — a skipped inference check never
drives a non-zero CLI exit on its own. (A phase that reports failed or other
exits 8 by default, and is informational only under --fail-on-error=false.)
Guards fire before any cluster mutation, so skips are cheap (typically < 10 s).
Configured GPU allocation policy
When you validate with a recipe, AICR resolves a whole-GPU allocation policy from the recipe’s fully hydrated component values and carries it to the validators (#1327):
The nvidia-dra-driver-gpu value resources.gpus.enabled is the switch:
true resolves dra-resource-claim; explicit false — or the component
absent or disabled — resolves device-plugin-extended-resource. On an
enabled DRA component the switch must be explicitly set: the upstream
chart’s declared default is true, so an absent value would diverge from
what Helm deploys. These configurations are rejected at resolution time with
an invalid-request error:
- an enabled
nvidia-dra-driver-gpucomponent withresources.gpus.enabledabsent (pin it explicitly in the recipe; stock recipes always do). gpus.enabled=truewithoutgpuResourcesEnabledOverride=true(the upstream chart install guard refuses it), and no whole-GPU advertiser remaining.gpus.enabledoff with the GPU operator component (gpu-operator, orgpu-operator-ocpon OpenShift recipes) absent, disabled, or carryingdevicePlugin.enabled=false.- A recipe enabling both
gpu-operatorandgpu-operator-ocp; two GPU operators collide at the operand level; enable exactly one (OpenShift recipes disable gpu-operator and carry gpu-operator-ocp).
Two further states are likewise rejected (they warned during the transition to the device-plugin production default and are errors since the flip):
- dual advertisement (both mechanisms enabled — exactly one whole-GPU advertiser is required)
- an inert
gpuResourcesEnabledOverride=truewithgpus.enabled=false(the waiver would disarm the upstream chart’s install-guard tripwire).
Stock recipes ship the production default: gpus.enabled=false,
gpuResourcesEnabledOverride=false, and devicePlugin.enabled=true; the
experimental DRA opt-in flips all three together in a recipe overlay. On
GKE-COS the gpuStack=dra profile value
(#2517) is the stock selection
of exactly that opt-in tuple — the one stock selection resolving
dra-resource-claim, still experimental pending the #1327 graduation
checklist (#3282); see
GKE GPU Setup.
Upgrading a cluster from the dual-advertised (pre-flip) configuration:
applying the flipped bundle does not drain existing workloads — a running
full-GPU gpu.nvidia.com claim pod keeps its prepared GPU while the
nvidia.com/gpu ledger, which cannot see that assignment, admits newly
converted scalar workloads onto the same device. Migrate in this order:
- Stop/delete all workloads holding full-GPU
gpu.nvidia.comResourceClaims (ComputeDomain/IMEX claims are unaffected and stay). - Confirm no allocated or reserved full-GPU claims remain, using the
driver-specific, fail-closed drain check below. On IMEX platforms,
allocated
compute-domain.nvidia.comclaims legitimately remain, so a plainkubectl get resourceclaims -Acan never come back empty — filter on the allocation driver instead, and treat ANY output (or any query failure) as unsafe to proceed. - Apply the flipped bundle (upgrade
nvidia-dra-driver-gpu). - Wait until the
gpu.nvidia.comResourceSlices and DeviceClass disappear (compute-domain.nvidia.comslices must remain). - Confirm scalar
nvidia.com/gpuallocatable is present on the GPU nodes. - Only then start the scalar (device-plugin) workloads.
The step-2 drain check — capture and test each stage separately (a piped
kubectl ... | jq ... masks a failed List as an empty, safe-looking
result), and proceed only when it exits 0:
While migrating, also delete any pending (unallocated) claims whose spec
references the gpu.nvidia.com DeviceClass: they hold no GPU (no
over-admission risk, so the check above rightly ignores them), but the flip
removes that DeviceClass, leaving such claims — and any pods referencing
them — stranded unschedulable forever.
Validators compare the configured policy against the inspected cluster state
and fail closed on mismatch — a cluster that cannot serve the configured
mechanism is a validation failure by design, never a silent fallback to the
other mechanism. Only recipe-less standalone runs (unspecified) keep the
capability-driven automatic selection.
Running all phases
Phases run sequentially. By default all phases run and produce results
regardless of earlier failures. Pass --fail-fast to stop after the first
phase that fails (e.g., to skip a 65-minute inference-perf run when deployment
already failed).
Skipping checks a run cannot satisfy
--phase selects whole phases. --skip-check (or
spec.validate.execution.skipChecks in a config file) works one level down,
withholding named checks from every phase that runs:
It is for a caller that cannot satisfy a check the recipe declares, typically a CI lane that deploys only part of the recipe or runs against simulated devices. It narrows the run, not the recipe: every other consumer of that recipe still gets the check, which is why this is not expressed as a recipe edit.
Three properties make it a scoping tool rather than a way to hide a failure:
- A skipped check is reported, not dropped, and its reason reaches the signed
bundle. It appears in the CTRF report as
skippedwith the reasonnamed in skipChecks, and as the structured codeextra.skipReason: named-in-skip-checks. The default (minimal) recipe-evidence bundle (--emit-attestation) does not carry the report verbatim: its redaction policy blanks every test’smessageandstdout, which is why the reason also rides the allowlistedextrachannel. So both the withheld check and why it was withheld travel with the attestation;--fullkeeps the prose message too. - The run fails closed on a list that would not do what it says. A name
matching no check in the catalog is rejected before any validation resource
is created, and so is a list that would leave a requested phase with nothing
to run (that phase would otherwise report
passedwhile running nothing, because the skipped entries keep its test count above zero). Stop requesting the phase instead. The recipe is loaded first, so acm://recipe is read from the cluster before the list is judged. - A check added later is not silenced. A skip list names what to withhold, so a new check in a recipe runs and can fail, which forces a decision rather than hiding one.
The CNCF conformance evidence path refuses the two flags together for a
different reason. pkg/evidence/cncf renders one markdown file per
requirement of the upstream checklist version selected from the cluster’s
Kubernetes minor (see Checklist Versions)
and aggregates each requirement’s status fail-closed from every check expected
to evidence it (see Requirement status).
The renderer records a skipped check as NOT TESTED with its reason code
(for example slurm-managed-gpus, when secure-accelerator-access defers to
the Slurm path and slinky-slurm-gpu-access did not run), so a skip is
reported rather than hidden, but a run that
withholds a check the recipe declares is narrower than the recipe by
construction and cannot evidence that recipe. --skip-check is for lanes
that cannot host a check at all; those are not submission lanes, so
--skip-check with --evidence-dir (and therefore with --cncf-submission;
the two flags require each other) is rejected up front:
The requirement aggregation, and the spec.validate.evidence.cncf.notApplicable
key it reads, apply to the evidence a submission run renders from the
validator run.
CNCF submission evidence
aicr validate --phase conformance is the only runner for CNCF AI
Conformance: the recipe declares which conformance checks run, and every run
executes all of them. --cncf-submission adds one thing to that run: after
the checks finish, the CTRF report is rendered into the submission directory
a vendor files as vX.Y/<product>/ in cncf/k8s-ai-conformance, under
--evidence-dir: PRODUCT.yaml (statuses from the run, metadata from
spec.validate.evidence.cncf.vendor), README.md (the index, with the
platform facts the notes cite), junit.xml and e2e.log (one testcase per
check, named by check) and one evidence page per requirement a non-skipped
check evidenced under evidence/, redacted in one pass and checked by the
pinned upstream validator as the last step. The flag never changes what a
check does or how it decides; for the same recipe the verdicts are identical
with and without it. The two flags require each other, --recipe is required
because the checks are selected from it, the vendor block is required in
--config, and --no-cluster and --skip-check are refused: submission
evidence comes from checks that ran on a live cluster against the whole
recipe. --evidence-dir must be absent or empty, with no parent path
segment of the form vX.Y/ (the upstream validator would read it as the
checklist version); the directory reaches it only when the run is complete,
and an incomplete run leaves its staged copy at <evidence-dir>.incomplete/
instead, which the next run refuses with exit 2 until it is removed. The
checklist version is the cluster’s Kubernetes minor and must be one this
build carries (v1.35 to v1.37); a cluster on another minor is refused with
exit 2 rather than certified against a neighboring checklist, unless
spec.validate.evidence.cncf.conformanceVersion names one deliberately (see
Submission directory
for the status, export and manual-attestation rules).
The config carries the vendor metadata and, for cluster_autoscaling, either
the conditional justification when the cluster has no autoscaler
(notApplicable, shown) or the documentation that attests a requirement the
run only observed (documented, below); see
evidence.cncf:
On a cluster whose autoscaler the cluster-autoscaling check observed but
did not drive, replace notApplicable with a documented entry; the run’s
OBSERVED result is then established by the documentation, which
PRODUCT.yaml cites instead of the testcase:
The --feature flag and spec.validate.evidence.cncf.features key scoped the
bash collector and went with it (#3602): the flag is gone, and a config that
still sets the key does not load (see
Deprecations).
The same run is available from the Go SDK as Client.CollectCNCFEvidence
(see Go library).
Exit status of a submission run
A submission run adds the requirement aggregation (see Requirement status) on top of the per-check rule every run applies:
- Flag and input refusals, and the readiness pre-flight, exit
2before any check runs, as for every validate run. - A check that
failedor crashed fails the run under--fail-on-error, as it does without the flag. Conformance checks with no requirement mapping (platform-health, the Slinky health checks) are invisible to the aggregation, so this rule is what catches them. - An applicable MUST requirement that no proof path establishes fails the run
with
INTERNALafter the evidence is written, so the directory still documents what was and was not shown. A path is established when every check on it is present andpassedas a demonstration; a check thatfailedon any path fails the requirement outright, and a check that wasskipped, crashed or is absent leaves only its own path incomplete. A check thatpassedbut marked its result observation-only (it recorded the cluster’s arrangement and drove no behavioral test; todaycluster-autoscalingon every path that passes) leaves its requirementOBSERVED, which is not established either: the exit message names it, e.g.cluster_autoscaling observed (cluster-autoscaling observed, not demonstrated). Only a cluster with no cluster autoscaler at all (fixed-size node pools, no Karpenter) may declare the requirement not applicable through aspec.validate.evidence.cncf.notApplicablejustification, which rendersNOT APPLICABLEand removes only that requirement from the verdict: the run still exits nonzero when another applicable MUST is not established or when a check itself failed (on EKS thecluster-autoscalingcheck fails when no autoscaler is confirmed, and the default--fail-on-errorstill applies). A platform whose autoscaler is present but not behaviorally demonstrated keepsOBSERVEDand is established only by aspec.validate.evidence.cncf.documentedentry naming public documentation, which PRODUCT.yaml then cites instead of the testcase. Where a requirement has alternative proofs (secure_accelerator_access:secure-accelerator-accesson ordinary recipes,slinky-slurm-gpu-accesson Slinky Slurm recipes) a passing alternative establishes it even when the primary check was skipped or did not run. A requirement declared not applicable is not counted. This verdict applies regardless of--fail-on-error=false. - The aggregation itself never fails the run on a SHOULD, whatever its
checks did; its page records the real status. A SHOULD-backed check that
failedstill trips the per-check rule above (pass--fail-on-error=falseto accept it), while the submission artifacts export its testcase as skipped so the directory stays submittable. - A write or redaction error, or the pinned upstream validator rejecting the
directory, fails the run with
INTERNALafter the recipe-evidence attestation (if any) was emitted; a validator that cannot start because the CNCF landscape is unreachable fails it withSERVICE_UNAVAILABLE. Flag and config refusals, including a missing or malformed vendor field and a non-empty--evidence-dir, exit2before any check runs.
A complete submission directory exists at --evidence-dir only when the run’s
requirement verdict is clean and the validator accepted it; otherwise the
staged copy at <evidence-dir>.incomplete/ documents what was and was not
shown.
Emitting recipe evidence
When a recipe PR targets hardware AICR maintainers cannot independently
re-run, the contributor needs to attach a signed evidence bundle so a
maintainer can verify the recipe offline. aicr validate produces the
bundle as a side effect when --emit-attestation is set; adding --push
signs it (cosign keyless via Sigstore) and uploads it to an OCI registry.
This is a different artifact from the CNCF-submission evidence above —
the two flag families produce independent outputs and may run from a
single aicr validate invocation.
The --push tag is just a human-readable label — the sha256: digest is
what pins the bundle, so tag choice never affects verification (the verifier
pulls by digest). Omit the tag, as above, and aicr derives a unique
per-recipe one, <recipe-slug>-<short-fingerprint> (e.g.
ghcr.io/<owner>/aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e), so
distinct attestations never collide on a shared tag. Pass an explicit tag to
override.
When the run’s workspace does not outlive it, for example in a Kubernetes Job
or an ephemeral CI step, add --image-refs <file> to write the pushed
bundle’s digest (the bundle.digest value in pointer.yaml) to a file you
choose, such as the container’s termination-message path
(/dev/termination-log by default). The file gets the same sha256:… line
aicr bundle --image-refs writes. Its path, and that it can be written, are
checked before validation starts. The kubelet bind-mounts the
termination-message file, so aicr writes it in place rather than renaming a
new file over it. The digest is written whenever the push succeeds, including a
run whose checks fail, since failed runs still publish their evidence. A run
that stops before the push leaves the file untouched, and an in-place write that
fails partway can leave it empty or partial. So start each run with an absent or
empty file and treat a complete sha256: digest in it as this run’s pushed
bundle; the exit status says whether the checks passed, not whether the file
was written.
After the command finishes:
The bundle is minimized by default. snapshot.yaml keeps only an
allowlisted set of fields (dropping node names, provider instance IDs, the
node label/taint set, OS tuning, loaded modules, and systemd config) and the
CTRF reports omit per-test stdout/message. Each test’s CTRF extra keeps only
allowlisted counts, measurements, thresholds, versions, allocation modes,
transport, fabric, and runtime-source codes, the nodeLocal and
versionSatisfied booleans, and skip-reason codes, never node names or IPs, and
is omitted when none remain. The signed predicate records the applied policy in
a redaction block, and the bundle self-verifies exactly like a full one. Pass
--full to publish the raw payloads instead.
Commit pointer.yaml to its per-source path
recipes/evidence/<recipe>/<src>/<digest>.yaml — the emit output prints
the exact copyTo path, and <src> is the slug derived from your signer
identity (see Artifact Verification).
The bundle itself lives in OCI. Then self-verify before opening the PR — the
same verifier runs against the committed pointer in the CI gate, so exit 0
locally means the gate will pass:
Flag reference: the evidence flags (--emit-attestation, --full,
--push, --no-sign, --bom, signing and registry options) are documented in
the aicr validate flag table.
By default, emission fails closed with INVALID_REQUEST when any validator
image is not immutably pinned (a moving tag such as :latest, or a non-AICR
registry without a digest), because the attestation names validators by tag.
The usual cause is a stale AICR_VALIDATOR_IMAGE_TAG; unset it.
--allow-mutable-validator-tags is the explicit opt-out: it emits anyway, so
use it only for disposable evidence you will not publish. For the recommended
producer workflow, which pushes an unsigned bundle from the cluster and signs it
in CI, see
Publishing Recipe Evidence.
Registry requirements: the registry must support the OCI 1.1 Referrers API (or its tag-schema fallback) so the Sigstore Bundle can be attached to the artifact. Known-good registries: GHCR, GitLab Container Registry, Harbor (≥ 2.8), AWS ECR, Google Artifact Registry, Azure Container Registry, JFrog Artifactory. Without referrer support the bundle pushes but the signature is not discoverable, and the verifier records signature-verify as “skipped (unsigned)” even on a signed bundle.
OIDC token resolution. --push resolves an identity token through
this precedence chain: --identity-token (or COSIGN_IDENTITY_TOKEN)
→ ambient GitHub Actions OIDC (ACTIONS_ID_TOKEN_REQUEST_URL
present) → --oidc-device-flow (or AICR_OIDC_DEVICE_FLOW=true) →
interactive browser. CI pipelines typically rely on the ambient
GitHub Actions path; local workstations get the browser flow. With
--signing-key <kms-uri> no token is resolved: the bundle is signed
with the KMS key instead, for private use. See
Key-signed evidence.
Local-only mode (no registry access). Omitting --push still
produces a complete bundle on disk — the verifier records the
signature step as “skipped (unsigned)” and the manifest-hash chain
becomes self-consistency only. Useful for catching accidental
corruption during development, but unsuitable for the CI gate, which
requires a signed bundle bound to a pointer.
For the full producer-and-consumer walkthrough — including OCI-only
verification, the tamper demo, and JSON output for CI gates — see
Recipe Evidence Demo.
For the bundle format and verifier semantics, see
ADR-007.
For the maintainer-side review checklist, see
Maintaining Recipe Contributions.
For the per-flag reference on aicr evidence verify, see
CLI reference.
Input modes
Snapshot and recipe can come from a file, an HTTPS URL, or a Kubernetes ConfigMap:
The ConfigMap form is useful when the snapshot is captured by an in-cluster agent — see agent deployment.
Dry-run mode
--no-cluster runs the validator against the snapshot alone, skipping all
Kubernetes API calls. Declarative constraints still evaluate; behavioral checks
report skipped - no-cluster mode (test mode).
Useful for CI pipelines that validate a recipe against a captured snapshot without needing cluster access.
Running under a pre-provisioned ServiceAccount
By default every live run creates a ServiceAccount for its validator Jobs, binds
it to cluster-admin through a per-run aicr-validator-<run-id>
ClusterRoleBinding, and deletes both at cleanup. That requires an identity
allowed to grant cluster-admin, which automation such as a CI runner or a
scheduled in-cluster Job often should not hold. It also leaves a privileged
binding behind if the process dies before cleanup, and gives admission or audit
policy keyed on a username no stable name to target.
--validator-service-account-name (or
spec.validate.agent.validatorServiceAccountName in
--config) moves that grant to an administrator. The
validator Jobs run as the named ServiceAccount, and the run creates and deletes
no ServiceAccount or ClusterRoleBinding and does not apply the namespace.
Before it creates anything, the run reads the ServiceAccount and asks the API
server, through a SubjectAccessReview, whether it may perform any verb on any
resource cluster-wide. It stops with exit 3 when the ServiceAccount does not
exist, with exit 2 when the ServiceAccount sets
automountServiceAccountToken: false (validator pods need its token), and with
exit 4 when it is not bound to cluster-admin or the caller may not ask. It
never falls back to per-run RBAC, because that fallback would create the
cluster-admin binding the flag exists to avoid.
An administrator provisions the namespace, the ServiceAccount, and its binding once:
The identity running aicr validate then needs the following, shown here bound
to a ci-runner ServiceAccount in a ci namespace. Everything is scoped to the
validation namespace except subjectaccessreviews, a read-only authorization
query that persists nothing:
Then validate against a captured snapshot:
--namespacemust name the ServiceAccount’s namespace. The default isaicr-validation.- The flag governs only the validator Jobs. Without
--snapshot,aicr validatefirst deploys the snapshot-capture agent, which creates or labels the namespace and creates its own run-scoped RBAC unless--service-account-namenames an existing agent ServiceAccount. See Using an existing ServiceAccount. Either way the agent needs permissions beyond the Role above, so a caller that holds only that Role supplies--snapshot. A missing or unbound validator ServiceAccount is reported before the agent is deployed. --no-cleanupkeeps the validator Jobs and ConfigMaps. The validator Jobs created no binding to leave behind; a live-capture agent’s run-scoped RBAC is kept as well.- Before deploying a validator that is placed next to another component’s
pods, the run lists pods in that component’s namespace (for example
monitoring) to warn when the placement selector matches nothing. Withoutlist podsthere, the run logs a lookup-failure warning and continues. - This changes who grants the validators’ access, and when. It does not make the caller less powerful: an identity that can create Jobs in the validation namespace can run any image as that ServiceAccount, and one that can write its ConfigMaps can change what the validators read. Keep the namespace dedicated to validation, and let only the intended caller create workloads or write ConfigMaps in it.
CI/CD integration
By default aicr validate exits non-zero when any phase reports failed or
other (set --fail-on-error=false to report those results and exit 0 anyway;
the readiness pre-flight still exits 2 regardless, and a
--cncf-submission run whose applicable MUST requirements are not all
established exits 8 regardless of --fail-on-error=false, see
Exit status of a submission run).
CTRF JSON is emitted to
stdout (or to --output <file>), so a pipeline can gate promotion on both the
exit code and the structured report:
Exit codes follow Unix conventions and are derived from the CLI’s structured
error codes (see pkg/errors/exitcode.go):
Important: two subtleties to be aware of when gating a pipeline on exit code:
- Both
failedandotherare blocking. A phase whose status isother— the check produced no usable verdict, e.g. a crash, an OOM, or a Job that failed for a non-deadline reason with its pod already gone — drives the same non-zero exit asfailed— an inconclusive check fails closed rather than passing silently.--fail-on-error=falsesuppresses both result-driven exits (the run reports the outcomes in the CTRF report and exits 0); it does not distinguishfailedfromother. A validator Job killed onactiveDeadlineSecondsreportsfailedrather thanother: its terminalFailed/DeadlineExceededcondition is a verdict, so the result names the deadline instead of the missing pod. The Job deadline is the catalog timeout plusValidatorJobDeadlineHeadroom(3m: 2m for container startup, 30s for the check’s deferred cleanup, 30s to flush its result), and the CLI waits for the Job to become terminal for that deadline plus the pod’s 60s SIGTERM window and a 5s margin (catalog timeout + 4m05s), measured from the Job’s observed start time (status.startTimewhen the apiserver has stamped it, otherwisecreationTimestamp). The CLI therefore never gives up on a run while the pod is still finishing its cleanup. When the Job controller deletes the pod at the deadline, the CLI reports what the check itself reported: the exit code and termination message of the pod in its deletion grace window, or of the last terminated pod it observed while waiting, is appended to the deadline message (check reported: exit code 3: terminated by SIGTERM …). The verdict staysfailed; if no terminated status was observed the generic deadline message stands.- Exit 5 is narrower than it sounds. A timeout inside a check’s own logic (DynamoGraphDeployment not ready, inference endpoint never healthy, AIPerf Job pod-wait deadline) surfaces as a failed phase, not as a structured
ErrCodeTimeout, so the CLI exits 8. So does a per-validator wait deadline:runPhasehands that error to the timeout handler and records the outcome as a check result rather than propagating it, which is the same path the wait-exhaustion exception in note 1 takes. The rule, rather than a list of sources: exit 5 applies whenever anErrCodeTimeoutreaches the top-level CLI. A timeout that is converted into a check result instead — whether raised inside a check or by a per-validator wait — surfaces as exit 8.
Interrupted runs. The first Ctrl-C or SIGTERM cancels validation and, with
cleanup enabled, deletes the in-flight validator Job using a fresh context.
Completed verdicts and the partial current phase are retained; unstarted checks
in that phase say not run: validation canceled and later phases do not start.
The interrupted phase is blocking (other unless already failed), and so is
its CTRF report. When no check entry already records the interruption, for
example because the signal landed between checks or between phases, the report
gains a validation-interrupted entry with status other, so summary.other
is non-zero. CTRF output is still attempted once at least one phase has
results, even if the cluster’s report ConfigMap cannot be written. A signal
that arrives before any phase has results, for example during cluster
preparation, writes no CTRF output. Partial output is written only for an
interruption. Any other error (for example a later phase’s pre-flight failure,
or a privileged RBAC revocation failure after every phase completed) writes no
CTRF output, as before.
Cleanup typically takes about 65s, and about 125s plus overhead in the worst
case. One shared 30s window covers result capture, Job deletion and report
persistence. The wait for the validator pod to terminate then covers the pod’s
full 60s SIGTERM grace from Job deletion (plus a 5s margin), so the validator’s
RBAC stays in place while a cooperative check cleans up. The existing 30s
per-run RBAC/data teardown window follows. The CLI records this 125s aggregate
deadline when cancellation is first observed, not when validation starts.
Partial output uses only the time remaining before that deadline (and at most
its existing 30s I/O cap), never a further cleanup window. If no time remains,
the in-memory results and original error are retained without starting output.
These limits are cooperative: an arbitrary file/stdout writer is not physically
interruptible just because it receives a context. A second Ctrl-C or SIGTERM
force-terminates the CLI and can leave resources behind. --no-cleanup
continues to leave them intentionally.
Exit 9 applies only when the signal arrives before the last phase completed.
A signal that arrives after that, for example during the final per-run RBAC/data
teardown or while evidence is rendered, signed or pushed, takes the normal
completion path on a canceled context. The run may then exit 0, 8 or another
code, depending on what that work returns.
If cancellation coincides with a privileged RBAC revocation failure, the
returned error is instead ErrCodeInternal (CLI exit 8), explicitly warning
that the cluster-admin binding may be orphaned. Both cancellation and cleanup
causes are preserved; check and revoke the potentially orphaned binding
manually. Partial CTRF output is still attempted within the same remaining
budget. Neither ordinary cancellation nor this combined failure renders, signs
or pushes conformance/recipe evidence.
When the validator pod itself receives SIGTERM (deadline, eviction or drain),
a cooperative check runs deferred cleanup, writes a SIGTERM interruption message
and exits 3, mapped to CTRF other rather than a completed failure verdict.
This container exit is distinct from the CLI’s exit 9 on operator cancellation.
It remains blocking (normally CLI exit 8), including when the check returns
nil after cancellation. The pod’s 60s grace (30s for cleanup, 30s to write the
termination message) is best effort: blocking checks, long cleanup, SIGKILL, OOM
or a crash may prevent a termination message, and a missing pod with a terminal
Job deadline condition still uses the Job-level failed classification
described above, carrying the check’s own message when one was observed. No
external orphan-resource sweeper is provided yet; namespaces a check leaves
behind after SIGKILL or a crash carry the aicr.run/gpu-test-run label.
Scripts that gate on validation outcome should treat any non-zero code as
failure rather than branching on specific values, and should additionally
check CTRF summary.failed and summary.other for a complete picture.
For informational-only runs (report results without failing the build):
--fail-on-error=false scopes to phase check results: a deployment,
performance, or conformance phase that reports failed or other is recorded
in the CTRF report and the run exits 0. It does not relax the readiness
pre-flight — if a readiness constraint
(K8s version, OS, kernel) is not met, the run still fails closed with exit 2
before any phase executes. In --no-cluster mode this is the dominant path,
since all phase checks report skipped and readiness is the only inline
constraint evaluation.
Troubleshooting
Readiness pre-flight fails
The CLI logs each readiness constraint comparison before any phase runs:
The run stops before deploying any Jobs and exits 2 (INVALID_REQUEST). This is
intentional fail-closed behavior: readiness gates whether the validators can run
at all, so it is terminal regardless of --fail-on-error (see above).
Fix: upgrade the cluster, or pick a recipe whose readiness constraints match the cluster’s actual versions.
Non-standard GPU labels or taints
Default GPU-node discovery looks for nodeGroup, node.kubernetes.io/instance-type, or GPU-related label substrings. If your cluster uses custom labels, override the scheduling of inner workloads with --node-selector and --toleration:
These flags affect the inner benchmark pods that run on GPU nodes (NCCL workers, Dynamo workers). The NCCL mpirun launcher follows the workers’ final node selector and tolerations, so it runs on a node that already holds the benchmark image; a recipe-supplied nccl-benchmark-runtime keeps its own launcher placement. The example above supplies --snapshot, so it does not launch the live snapshot agent. When --snapshot is omitted, the flags also configure that preliminary agent. With no toleration override, the agent tolerates all taints; an explicit spec.validate.agent.tolerations: [] clears that default. Neither flag affects the validator orchestrator Job itself.
For inference-perf specifically, --node-selector narrows the pool of candidate GPU nodes — the validator then picks the candidate with the most free GPUs (subtracting same-ledger occupancy only — DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity — and skipping DRA candidates that carry scalar nvidia.com/gpu workloads) and pins all Dynamo Frontend + worker pods to that node via kubernetes.io/hostname. The AIPerf benchmark runner pod is CPU-only, uses a tolerate-all / no-nodeSelector pod spec, and is unaffected by these flags.
A check reports skipped unexpectedly
Skips are always deliberate and always carry a reason, but the location of the reason in the CTRF entry depends on how the skip happened:
- Check-level skips (the CheckFunc ran and returned
validators.Skip(reason)— e.g., Guards A/B/C on inference,--no-clusterfrom inside a check): reason appears instdoutaslevel=INFO msg=SKIP reason="…". - Phase-level skips (the CheckFunc never ran — e.g., with
--fail-fast, a prior phase failed so subsequent phases synthesize skip entries; also--no-clusterfor checks that the runner marks skipped before dispatch): reason appears inmessage, notstdout.
Common reasons and their cause:
driver-runtime-management fails
The check (CNCF driver_runtime_management, SHOULD at v1.37 and MUST at v1.38)
runs a pod from a vanilla ubuntu:22.04 image that requests one GPU, and reads
the driver from inside it. The probe pod carries the
--image-pull-secret Secrets, copied into its per-run namespace (see the
image note below). Its artifacts (Driver runtime probe pod logs,
Advertised driver version, Driver runtime compatibility) name the failing
step:
The probe image (docker.io/library/ubuntu:22.04) pulls from Docker Hub by
default and, like the busybox probe image and the NCCL workload image (but
unlike the Dynamo runtime images inference-perf pulls), is not rewritten by
AICR_VALIDATOR_IMAGE_REGISTRY. In a mirrored environment set
AICR_DRIVER_RUNTIME_IMAGE=<image ref> in the aicr validate environment to
the full reference of your copy or of another vanilla OS image. It must ship no
accelerator driver or tools, and it must provide what the probe script runs on:
a POSIX /bin/sh, head and tr (timeout is used when present). A
distroless image cannot run the probe; mirroring the Ubuntu image is the
simplest choice. When the mirror needs credentials, --image-pull-secret
covers the probe too: every conformance check that creates a per-run namespace
copies the named Secrets from the validator namespace into it and attaches
them to its probe pods, and the copies are deleted with the namespace. A
Secret the flag names but the validator namespace lacks fails the check with
NotFound before any probe is created.
A probe pod whose RuntimeClass names a handler the node’s container runtime
does not serve stays Pending with ContainerCreating and no status change;
the checks read the kubelet’s FailedCreatePodSandBox event (the containerd
message no runtime for "..." is configured naming the handler, a missing
runtime handler, or a shim binary not found) and fail within seconds with
stuck: FailedCreatePodSandBox: ...
instead of waiting out the pod timeout. A sandbox failure from network setup
is transient and is not treated this way.
ai-service-metrics or pod-autoscaling cannot reach its metrics endpoint
On EKS clusters that split worker and system pods across separate security
groups (e.g. DGXC EKS with distinct customer/system ENI subnets), the
conformance check ai-service-metrics, which queries NKTP’s otel-lgtm, can
fail non-deterministically with:
pod-autoscaling, which dials Prometheus for its PodMonitor load wait when the
recipe declares kube-prometheus-stack, fails the same way at its preflight:
Both checks’ orchestrator Jobs carry the preferred dependencyAffinity
described below.
The validator orchestrator Job tolerates every taint and sets a preferred
dependencyAffinity toward the pod it dials (otel-lgtm for
ai-service-metrics, Prometheus for pod-autoscaling), so the scheduler
co-locates the two when possible. The preference is best-effort, so on fallback it
can still land on any worker node — including one whose ENI is in a security
group whose ingress to that pod’s SG is missing or asymmetric. On
such a fallback the outcome is not stable across re-runs: image-locality
scoring tends to keep the pod on whatever node won the first scheduling
decision, so a passing run on a fresh cluster does not prove the SG topology is
correct.
This is a cluster-side prerequisite, not an AICR bug per se — see
EKS Dynamo Networking Prerequisites
for the SG ingress rules required for tcp/9090, which both Prometheus and
otel-lgtm serve. The preferred
dependencyAffinity (#933,
resolved) makes a bad placement far less likely, but the 9090 SG rule remains
the reliable guarantee since the affinity is best-effort.
Workaround when SG changes are not available: re-run the check until the orchestrator lands on a node whose SG can reach that pod, then leave the image cached there so image-locality keeps subsequent runs on the same node. This is unreliable and should not be used as the steady-state validation strategy.
workload-sandboxing fails
The check (CNCF workload_sandboxing, SHOULD at v1.37 and MUST at v1.38)
looks for a sandboxed RuntimeClass, runs a CPU-only ubuntu:22.04 probe pod
on it and an unsandboxed control pod on the same node, and compares what the
two observe. No embedded recipe declares it yet: none ships a sandboxed
RuntimeClass, so until the #3615 runtime decision provides one and the
recipes declare the check, the workload_sandboxing evidence row renders
NOT TESTED (not run) at SHOULD. A cluster that already has gVisor or Kata
can run it today by adding workload-sandboxing to
spec.validation.conformance.checks in a recipe copy (and naming the class
with AICR_SANDBOX_RUNTIME_CLASS when detection does not match). Its artifacts (Sandboxed RuntimeClass, Sandboxed probe pod logs, Unsandboxed control pod logs, Workload sandboxing verdict) name the
failing step:
The probe image (docker.io/library/ubuntu:22.04) pulls from Docker Hub by
default and, like the busybox probe image and the NCCL workload image, is not
rewritten by AICR_VALIDATOR_IMAGE_REGISTRY. In a mirrored environment set
AICR_SANDBOX_PROBE_IMAGE=<image ref> in the aicr validate environment to
the full reference of your copy or of another vanilla OS image. It must
provide what the probe script runs on: a POSIX /bin/sh, uname, cat,
ls, wc, head, awk and tr. A distroless image cannot run the probe;
mirroring the Ubuntu image is the simplest choice. The two probe pods carry
the --image-pull-secret Secrets, copied into the per-run namespace as for
the other probe pods (see the driver-runtime-management note above), so a
mirror that needs credentials is served the same way as the validator image.
pod-autoscaling fails with “Prometheus did not load the PodMonitor”
When the recipe declares kube-prometheus-stack, the check creates its
PodMonitor in the Prometheus namespace, because that Prometheus loads monitors
only from the namespaces the bundle names and its own. Before starting its
metric budget it waits up to five minutes for Prometheus to list the
PodMonitor among its scrape pools (GET /api/v1/scrape_pools). That wait
exists because prometheus-operator reconciles monitors on its own schedule:
gaps of about three minutes between a monitor’s creation and Prometheus
loading it were observed on a healthy AKS cluster. Without
kube-prometheus-stack (the OpenShift overlay serves prometheus-adapter-ocp
from the platform monitoring stack) there is no Prometheus to ask: the
PodMonitor stays in the test namespace, the check records the wait as skipped,
and its baseline is bounded by the metric timeout alone.
The baseline then ends in one of three ways, each with its own message.
A failure naming the load, such as:
means Prometheus did not load the monitor within the wait. Two things have
to happen for it to: prometheus-operator must write the monitor into the
generated configuration, and Prometheus must reload that configuration. Check
both. For the first, look at the operator’s logs for the reconcile of that
namespace, and confirm the Prometheus resource’s podMonitorNamespaceSelector
names its own namespace. For the second, a rejected reload leaves Prometheus
running on its previous configuration with the pool absent even though the
operator did its part: check prometheus_config_last_reload_successful (0
after a failed reload) and the Prometheus and config-reloader container logs
for the reload error.
A failure reading “could not determine whether Prometheus loaded the PodMonitor … the last completed scrape pool listing failed” (or ”… no scrape pool listing completed”) means the scrape pool API stopped answering after the preflight, so no verdict on the monitor was reached; the underlying listing error (unreachable, HTTP status, malformed response) is attached. A listing the wait’s deadline interrupts does not change the verdict: “did not load” rests on the last listing that completed. Check the Prometheus pod and the network path from the validator pod to it, as in the section above.
A failure naming the collection instead (“Prometheus loaded the PodMonitor,
but the custom metrics pipeline does not serve the workload metric”) means the
configuration was loaded and the scrape or the adapter query did not produce a
sample: check the target under the active or dropped targets at
/api/v1/targets (a target relabeling dropped is listed as dropped, not
active), that the Prometheus series filter keeps
ai_conformance_hpa_queue_depth, and the prometheus-adapter rule.
ai-service-metrics fails with “did not report the stub’s target as up”
The check queries the backend NKTP exports to, not the recipe’s Prometheus.
It creates a ServiceMonitor in its test namespace and waits up to five minutes
for that backend to hold the stub target’s up series with value 1.
- No
upseries: NKTP did not discover the ServiceMonitor, or did not export the sample. NKTP needs no monitor label, so check the ServiceMonitor’s selector and port, the Target Allocators’ logs, and the collectors’ logs for failed scrapes or exports. upis 0: the target is discovered but its scrape fails. Check the network path from NKTP’s collectors to the stub pod.
A failure reading “discovered the ServiceMonitor but the workload metrics were
not collected” means up arrived but the stub’s own series did not within the
collection timeout: check the collectors’ logs as above.
The check is skipped, and the CNCF requirement reported as not tested, when
the recipe has no nktp or NKTP runs without otel-lgtm. To run it against
your own backend instead, see
ai-service-metrics on your own metrics backend.
Benchmark Job stuck or timed out
Each performance check has a Job-level activeDeadlineSeconds set by the catalog’s timeout: plus ValidatorJobDeadlineHeadroom (3m) on top. For inference-perf, the full pipeline (model-cache populate → workload ready → endpoint health → benchmark) can take up to ~50 min on cold-start clusters; the catalog budgets a 65 min timeout. If it still times out:
Common causes: image pull throttling, vLLM model load slowness, and every
eligible candidate GPU node being saturated by existing workloads. In the
saturated case the validator fails fast with a message like no eligible candidate GPU node has free GPUs (N matched; eligible candidates are saturated by existing workloads in the selected mechanism's ledger ...).
Occupancy is subtracted per allocation ledger (DRA allocations from DRA
capacity, device-plugin requests from device-plugin capacity), and nodes
excluded for other reasons — NotReady, probe-ineligible, kai-blocked raw
slices, or DRA-capable but carrying scalar nvidia.com/gpu workloads — are
not counted as saturated;
the fix is to free GPUs on one of the candidate nodes, or to pass
--node-selector kubernetes.io/hostname=<node> to target a specific node
you know is free. GPU-occupancy accounting fails closed rather than treating
GPUs as free: a pod list failure always fails the check, and so does any
error listing DRA resourceclaims (e.g. RBAC denied, timeout) — with one
exception: when the resource.k8s.io API is not served at all (NotFound),
which deterministically means zero DRA usage, the validator proceeds with
device-plugin pod requests alone.
Related
- CLI reference:
aicr validate— full flag reference and per-command examples - CLI reference:
aicr snapshot— snapshot capture options - CLI reference:
aicr recipe— recipe generation flags - Agent deployment — capture snapshots via an in-cluster Job
- Data flow: Stage 3 Validate — how the validator engine is built
- Validator Development Guide — add a new validator (contributor-facing)