Skip to navigation

Validating a Cluster

View as MarkdownOpen in Claude

Task-oriented walkthrough for running aicr validate against a GPU cluster — from capturing a snapshot through interpreting results. Covers both training and inference workloads and all three validation phases (deployment, conformance, performance).

For per-flag reference, see CLI reference: aicr validate. For the architectural view of how snapshot + recipe flow into the validator, see Data flow: Stage 3 Validate.

When to validate

PhaseWhat it answersTypical trigger
deploymentAre the components the recipe asks for actually installed and healthy?After ./deploy.sh finishes, before running any workload
conformanceDoes the cluster support workload-specific capabilities (DRA, gang scheduling, autoscaling, …)?Before opening the cluster to real workloads
performanceDoes the cluster hit expected bandwidth / throughput thresholds?After components are ready; before going to production

Under a GitOps deployer (Argo CD, Flux, Fleet), start aicr validate --phase deployment only after the sync of every wave has completed, not while it is still progressing. The phase treats a declared resource the cluster lacks as a deployment that was not applied, not as one that is not ready yet: a Nodewright CR the recipe declares (shipped by nodewright-customizations, which syncs in a later wave than the operator) that is absent fails expected-resources immediately instead of waiting for the sync to deliver it.

Readiness pre-flight constraints (K8s version, OS, kernel) run implicitly before any phase. If pre-flight fails, no validator Jobs are deployed. A submission run (--cncf-submission) adds two pre-flights of its own once the snapshot is loaded or captured and before any validator Job: the cluster’s Kubernetes minor must match a carried checklist exactly (1.34 and 1.38 are refused alike, since a submission certifies one minor) unless spec.validate.evidence.cncf.conformanceVersion names one, which the submission README then records as selected explicitly; and the recipe’s validation.conformance.checks must declare a complete proof path for every applicable MUST of that checklist (secure-accelerator-access or the Slinky Slurm alternative slinky-slurm-gpu-access; a conditional MUST declared under spec.validate.evidence.cncf.notApplicable is not required). A recipe that cannot establish a MUST is refused with INVALID_REQUEST naming the requirements and the checks that would close each gap, since no run of it can establish the submission. A plain run that includes the conformance phase logs the same gaps as a warning and proceeds.

The workflow

aicr snapshot ─┐
├─▶ aicr validate ─▶ CTRF report
aicr recipe ───┘ (passed / failed / skipped per check)
  1. Snapshot — capture current cluster state (K8s / OS / GPU / topology) once.
  2. Recipe — generate the target configuration for your workload (training vs inference, platform, accelerator).
  3. Validate — run one or all phases against the snapshot and live cluster.

Which recipe to validate

Every bundle writes the recipe it was generated from to recipe.yaml at the bundle root, after dropping the components it did not render. Bundle-time --set values are not written back to it; the one bundle-time decision that is persisted is the dra-node-labeler enablement under --dra-eviction-node-label. That file is the effective component inventory of what was deployed, so validation of a deployed cluster reads it. The examples below assume the bundle was written with aicr bundle --output ./bundles, as in Generating Bundles:

aicr validate --recipe ./bundles/recipe.yaml --phase deployment

The original recipe is the input for the pre-deploy dry run (--no-cluster, below): the bundle does not exist yet, and the dry run evaluates what the recipe asks for, not what a bundle delivered. Validating the original recipe after deployment still works, but it cannot see bundle-time decisions: a component the bundler dropped is still declared (its health check is skipped when the component’s own values leave a trace, and reports NOT_FOUND otherwise), and a component the bundler enabled at bundle time is not checked.

Prerequisites

  • aicr CLI installed (see installation).
  • kubectl configured for the target cluster (validator dispatches K8s Jobs; pre-flight only needs the snapshot).
  • Effectively cluster-admin for the identity running aicr validate. Each run creates a ServiceAccount for its validator Jobs and binds it to the built-in cluster-admin ClusterRole through a per-run aicr-validator-<run-id> ClusterRoleBinding, and Kubernetes RBAC only lets you bind a role whose permissions you already hold. The run deletes the binding at cleanup; --no-cleanup leaves it active until you delete it yourself. AICR creates its own aicr-validation namespace on first run. To run without that grant, have an administrator provision the ServiceAccount instead; see Running under a pre-provisioned ServiceAccount.
  • AKS profiled recipes: the readiness pre-flight re-evaluates the recipe’s profile constraint (K8s.aks-gpu-pools.gpu-driver), so the snapshot must carry that reading — capture it with aicr snapshot --aks-gpu-pools <az dump>, or pass the same flag to aicr validate when it captures live. A snapshot without the reading fails readiness closed (exit 2).
  • GKE recipes: the readiness pre-flight re-evaluates the recipe’s gpuStack profile constraint over the GPU-node set (nodes carrying cloud.google.com/gke-accelerator): the default gke-default value requires that no GPU node carries the opt-out label gke-no-default-nvidia-gpu-device-plugin (GKE’s managed plugin stays the nvidia.com/gpu advertiser), while bundle-installer and dra require every GPU node to carry gke-no-default-nvidia-gpu-device-plugin=true (so the GPU Operator’s plugin — or, under dra, the NVIDIA DRA driver’s ResourceSlices — is the sole whole-GPU advertiser). The check fails closed (exit 2) on labels contradicting the selected value, mixed labels, malformed or ambiguous label readings, a snapshot with no identifiable GPU nodes, and when --max-nodes-per-entry actually truncated a participating label reading (a truncated node list cannot prove set membership — regenerate with no cap set, from the flag, AICR_MAX_NODES_PER_ENTRY, or spec.snapshot.execution.maxNodesPerEntry; a cap larger than the node count truncates nothing and validates normally). gke-default needs no provider projection flag. Its constraint reads node labels the standard snapshot already carries. bundle-installer and dra additionally re-evaluate K8s.gke-gpu-pools.gpu-driver-installation, corroborating that every GPU pool was actually created with gpu-driver-version=disabled. That reading requires a snapshot captured with --gke-gpu-pools <gcloud dump> (or passed to aicr validate --gke-gpu-pools), and without it is unavailable, so those values fail closed. dra also re-evaluates its own K8s.server.version >= 1.35.0 floor. See GKE GPU Setup for the full setup and the qualification matrix.
  • Run:ai recipes (platform=runai): the gang-scheduling conformance check runs its test pods in a Run:ai Project namespace, so export AICR_RUNAI_PROJECT_NAMESPACE=runai-<project> in the shell running aicr validate; the check fails, rather than skips, without it. The Project needs an explicit CPU and memory quota and no CPU or memory limit, and a private probe image needs its pull Secret already in that namespace. See Run:ai cluster.
  • NodeTopology.gpu-nodes.label readiness constraints outside GKE (including bare metal and --data services): GPU nodes are those the GPU Operator labels nvidia.com/gpu.present=true, so capture the snapshot after it is running. A snapshot with no such node fails readiness closed (exit 2), and GPU nodes it has not labeled are not checked.

ai-service-metrics on your own metrics backend

By default the ai-service-metrics conformance check queries NKTP’s bundled otel-lgtm. When NKTP exports to your own backend, or the recipe has no nktp, name that backend’s Prometheus query API in the shell running aicr validate:

export AICR_SERVICE_METRICS_QUERY_URL="https://mimir.example.com/prometheus"
aicr validate --recipe recipe.yaml --phase conformance
  • The value is a base URL; the check appends /api/v1/query. It wins over otel-lgtm whenever it is set.
  • The validator pod must reach the URL without credentials. A URL that carries a user or password fails aicr validate before any Job is created, when the run includes this check, because the URL is written into the check’s Job and recorded in its evidence.
  • The backend must hold what your monitoring system scrapes from a ServiceMonitor in the check’s test namespace, including each target’s up series.

Without the variable, the check is skipped on such a cluster and the CNCF requirement is reported as not tested.

Training performance validation

Training performance runs an NCCL all-reduce benchmark — a Kubeflow TrainJob that runs all_reduce_perf across GPU nodes and measures aggregate bus bandwidth. Three check variants are available; the recipe picks the one (or ones) that match the target fabric:

What the bandwidth number describes. Each nccl-all-reduce-bw* result carries a runtimeSource label: delivered-artifact means the benchmark runtime was derived from the ClusterTrainingRuntime the recipe ships, so the number attests to the delivered wiring; recipe-supplied-runtime means the recipe supplied the runtime itself; cluster-capability means the validator’s own fixture was measured — proof the fabric can reach the floor, not proof of what the recipe ships. Only h100-gke-cos-training-kubeflow produces delivered-artifact today. The label is decided from the recipe, not from what is installed on the cluster.

CheckTransportDefault applicability (from recipe criteria)
nccl-all-reduce-bwAuto-detect (whatever NCCL picks)H100/H200 on EKS, H100 on GKE (GPUDirect TCPXO), H100 on AKS (ND-series InfiniBand — NCCL’s built-in IB/verbs transport over the rdma/hca_shared_devices_a shared device pool), and B200/GB200 on self-managed clusters (service=any). Preserves the pre-variant behavior.
nccl-all-reduce-bw-netNET (EFA on EKS; built-in IB/verbs on OKE and NVKE)GB200 + EKS, GB200 + OKE, GB200 + NVKE, GB300 + EKS, GB300 + OKE, and VR200 + NVKE. Asserts the intended NET fabric actually carried traffic — EFA on EKS; on OKE, the NVL72 InfiniBand east-west fabric (nvidia.com/mlnxnics shared HCAs) for GB200 and RoCE over SR-IOV VFs (nvidia.com/mlnxnics VFs attached through the rdma-net network) for GB300; on NVKE, the per-rail SR-IOV VFs (nvidia.com/sriov_resource_rail_<n>), over InfiniBand for GB200 and RoCE v2 for VR200 — catching silent fallback to Socket when GPUDirect RDMA is unavailable. A driver preflight gates the GB200 combinations and GB300 + EKS — see Grace Blackwell NET preflight.
nccl-all-reduce-bw-nvlsNVLS (MNNVL across an NVL72 IMEX domain)GB200 + EKS, GB200 + OKE, GB200 + NVKE, GB200 + GKE (A4X GPUDirect-RDMA/gIB carries the IMEX/NVLink fabric traffic; gIB is the transport driver, not the NCCL algorithm), GB300 (EKS, OKE, NVKE, generic), VR200 + RKE2, and VR200 + NVKE. Asserts the NVLS communicator actually initialized. Catches silent fallback to the NET fabric (EFA on EKS, InfiniBand on OKE GB200 and NVKE, RoCE on OKE GB300), or gIB/RDMA initialization failure (GKE), when the IMEX domain is misconfigured.

The applicability column is the default, derived from the recipe’s criteria. A recipe whose criteria fall outside it can still run these benchmarks explicitly — either by borrowing an embedded benchmark profile whose fabric matches its hardware, or, for a private service whose fabric matches no embedded template, by supplying its own benchmark runtime.

Overriding the NCCL workload image with AICR_NCCL_RUNTIME_IMAGE. Each embedded nccl-all-reduce-bw / -net / -nvls template pins a specific launcher/worker workload image — the CUDA/NCCL/MPI/SSH/transport runtime that all_reduce_perf actually runs in, distinct per platform (e.g. GKE H100/TCPXO ships a CUDA 12.9 image today). To qualify a different CUDA/NCCL combination — for example CUDA 13 on GKE TCPXO with an R580-or-newer driver — set AICR_NCCL_RUNTIME_IMAGE=<image ref> in the aicr validate environment. The resolved image is rendered into every container that carries the workload (the launcher’s SSH-setup init container and both the launcher’s and worker’s main containers), so a run can never end up on a mixed image set; a platform-specific sidecar unrelated to the NCCL workload itself (e.g. GKE’s tcpxo-daemon transport daemon) is left untouched. A malformed image reference fails the check immediately, before cluster discovery or NCCL benchmark resources are created, rather than silently falling back to the compiled default.

This is a different setting from aicr validate --image / AICR_VALIDATOR_IMAGE_* (see Validator image tags): those control the validator’s own container image (the snapshot/orchestration binary), never the inner NCCL workload. AICR_NCCL_RUNTIME_IMAGE only applies to the three NCCL all-reduce checks, and only to the templates AICR owns — the embedded per-platform template and a runtime derived from a delivered artifact — it has no effect when a recipe supplies its own benchmark runtime, since that runtime already owns its image end to end. For reproducible qualification runs, prefer pinning by digest (name@sha256:...) over a mutable tag.

Choosing the self-installed Kubeflow Trainer release. When the cluster has no Kubeflow Trainer and the recipe does not enable kubeflow-trainer, the NCCL checks install a temporary Trainer from GitHub and remove it afterward. The release defaults to the kubeflow-trainer pin in recipes/registry.yaml. When the recipe has no kubeflow-trainer ref, set AICR_NCCL_TRAINER_VERSION in the env block of each NCCL catalog entry that runs (for example through a --data overlay) to choose a different release without rebuilding the validator image. A recipe that disables kubeflow-trainer installs that ref’s version and ignores AICR_NCCL_TRAINER_VERSION. aicr recipe fills in the registry pin for a ref with no version, so set the ref’s version in the overlay to choose a different release for such a recipe. The value is a semantic version, with or without a leading v. The check fails before any Trainer resource is applied when the value is malformed, when the tag has no GitHub release, or when the release’s manifests lack what the installer needs. That means TrainJob and TrainingRuntime CRDs serving the API version the benchmark creates, the Trainer admission webhook configurations, and the controller Deployment. A Trainer that is already installed is used as-is, whatever its version, and the value is not checked.

GB200/EKS recipes (both training and inference intents) enable -net and -nvls together rather than the auto-detect variant, because those nodes expose two inter-node fabrics simultaneously and a single auto-detect test would only exercise one of them.

GB200/OKE training recipes follow the same pattern and enable -net and -nvls together, with the same >= 40 / >= 500 GB/s floors as GB200/EKS. On OKE, -net exercises the NVL72 rack’s InfiniBand east-west fabric (rdma0-3, advertised as nvidia.com/mlnxnics by the recipe’s rdmaSharedDevicePlugin NicClusterPolicy) over NCCL’s built-in IB/verbs transport — no EFA plumbing is involved. Both variants were validated end to end on a BM.GPU.GB200.4 NVL72 rack.

The GB300/OKE training recipe enables -net and -nvls with >= 300 / >= 700 GB/s floors. Its -net runs NCCL’s built-in IB/verbs transport over RoCE: each worker pod requests the node’s four SR-IOV VFs (nvidia.com/mlnxnics) and attaches them through the recipe’s rdma-net network, and NCCL reads the node topology that nvidia-tuned writes to /etc/nccl/topo.xml. On BM.GPU.GB300.4 nodes with NCCL 2.29.2, the 16 GiB bus bandwidth measured 363 GB/s (NET) and 836 GB/s (NVLS) on 2 nodes, and 376 GB/s and 909 GB/s on 4.

# Capture snapshot, generate training recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml
aicr recipe --service eks --accelerator h100 --os ubuntu \
--intent training --platform kubeflow \
--output recipe.yaml
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance

The generated recipe lists the selected variant(s) under validation.performance.checks with a platform-tuned bandwidth constraint (example: >= 300 GB/s for H100 + EFA; >= 40 GB/s NET and >= 500 GB/s NVLS for GB200 + EFA, each sized for a 2-node pair).

Node-shape assumption. These bus-bandwidth floors are fixed absolute values calibrated on full, high-bandwidth nodes (8-GPU H100 NVLink/SXM with multi-NIC transport). They are not normalized for node fabric or GPU count, so a smaller or different-fabric H100 SKU (e.g. a single-GPU-per-node shape) can false-fail a healthy run. Making the NCCL gate fabric/transport-class aware is tracked in #1256.

Expected flow (~5–10 min per variant): readiness pre-flight → deploy TrainingRuntime + TrainJob in a per-run namespace named aicr-nccl-perf-<variant>-<run-id> → worker pods reach Running → run all_reduce_perf → parse peak bus bandwidth → verify the intended transport actually carried traffic (for -net / -nvls) → compare to recipe constraint (10 % tolerance) → cleanup.

Each variant (-net, -nvls, default) gets its own namespace, deleted on exit along with its TrainingRuntime/TrainJob. A namespace left behind by an interrupted run is reclaimed by a same-run-ID retry only if no Pending/Running/Unknown pod remains in it and its execution lock (a Lease named aicr-nccl-run-lock in that namespace) has gone stale, 20 minutes past its last renewal. Otherwise the retry fails with a conflict rather than risk two executions sharing one namespace; check kubectl get pods -n <namespace> and the Lease’s age to find and clear whichever is actually stuck. A standalone run (no run ID) is never retried into its old namespace; that namespace is only pruned automatically, on a later run once an hour has passed and both conditions above hold.

A passing CTRF entry:

{
"name": "nccl-all-reduce-bw-net",
"status": "passed",
"suite": ["performance"],
"stdout": [
"NCCL All Reduce bandwidth (nccl-all-reduce-bw-net): <actual> GB/s",
"Constraint: >= <threshold> → true"
]
}

Note: this guide does not yet list per-platform expected-bandwidth baselines (EKS + EFA, GKE + TCPXO, AKS, etc.). The recipe’s constraint value is the current pass/fail floor; measured values above that floor are treated as passing regardless of platform.

To run deployment validation first (recommended — verifies GPU Operator, DRA driver, and Kubeflow Trainer are installed and healthy before the benchmark):

aicr validate --recipe ./bundles/recipe.yaml --snapshot snapshot.yaml --phase deployment

Grace Blackwell NET preflight: GPUDirect RDMA prerequisites

Before running nccl-all-reduce-bw-net on GB200 (EKS, OKE or NVKE) or GB300 (EKS), a preflight checks each GPU node for the driver-side prerequisite of GPUDirect RDMA. Without it NCCL falls back to the Socket transport. The -net check catches that on its own — it fails on a Using network Socket banner rather than reporting a figure — so the preflight exists to fail fast, naming the driver, instead of after a full benchmark run.

The preflight covers EFA on EKS and built-in IB/verbs on OKE and NVKE for GB200, and EFA on EKS for GB300. GB300 on OKE (RoCE) ships R595 in the node image and is not preflighted.

Before R595 — which includes 580.173.02, the version AICR pins — the driver must be loaded with NVreg_GrdmaPciTopoCheckOverride=1. Without it the driver refuses to let a PCIe-attached NIC (EFA on EKS, ConnectX IB on OKE and NVKE) attach dma-buf handles to GPU memory, and the kernel logs:

NVRM: dma-buf attach failed: topology not supported for mapping type FORCE_PCIE

Set the parameter according to who owns the driver:

Driver ownerHow to set it
GPU OperatorPoint ClusterPolicy spec.driver.kernelModuleConfig.name at a ConfigMap in gpu-operator containing nvidia.conf: options nvidia NVreg_GrdmaPciTopoCheckOverride=1
Node image (OKE default gpuStack=oci-managed)Set the module parameter in the image or node bootstrap (/etc/modprobe.d), then reboot the GPU nodes

Deleting the nvidia-driver DaemonSet pods does not apply the change, and neither does editing only the ConfigMap. The reload decision is keyed off the ClusterPolicy spec, so the spec itself has to change. Setting kernelModuleConfig.name is such a change, but AICR’s GB200/GB300 overlays already set it — where it is present, point it at a differently-named ConfigMap so the spec actually differs. Confirm on a node afterwards:

grep GrdmaPciTopoCheckOverride /proc/driver/nvidia/params

On R595 and later the parameter no longer exists, and setting it has no effect because the kernel silently ignores unknown module options. R595 replaced it with a PCIe topology requirement the preflight cannot check, so validation fails closed instead of assuming. On EKS p6e-gb200/gb300 that requirement is known not to be satisfied — the measurement is recorded alongside the driver pin in recipes/components/gpu-operator/values.yaml; on OKE it is unmeasured.

The remedy on R595+ is a driver at R580 — AICR ships 580.173.02 — pinned through the ClusterPolicy where the GPU Operator owns the driver, or through the node image where it does not.

Undetermined is a third outcome, reached without either verdict above: if /proc/driver/nvidia/version or params cannot be read, or the version banner does not parse, the preflight reports the state as undetermined rather than assuming one. SELinux denying the read inside the container, or a driver-container remount leaving the path empty, produces this. Changing the driver version does not address it — read the file on a target node to see whether it is unreadable or carries an unrecognised banner.

Driver not loaded is a fourth outcome, reached before any file is read: when /proc/driver/nvidia does not exist on a target node, the kubelet refuses the probe pod’s hostPath mount. If the preflight reads the kubelet’s hostPath type check failed event and it is still the probe pod’s latest event, it names that node as driver-absent instead of sending you to set the flag. Otherwise (the events cannot be read, none matches, or a later event for the pod has superseded it) it returns the original wait timeout. There is no parameter to set until a driver is loaded. Finish the driver rollout (GPU Operator nvidia-driver DaemonSet Ready, or a node image that ships the driver on OKE oci-managed) and re-run. The kubelet leaves the pod in ContainerCreating, so this surfaces after the probe’s wait (up to two minutes), and it stops the check at the first such node rather than listing every affected node.

Opting external recipes into a benchmark profile

The default applicability above is keyed to service + accelerator pairs the validator ships templates for. A recipe whose criteria fall outside that set — typically a criteria.service registered only through an external --data directory, or an embedded service extended to a new accelerator — would otherwise report the NCCL checks as skipped even when its validation block declares them. Such a recipe opts into an embedded benchmark with the nccl-benchmark-profile performance constraint (a bare {accelerator}/{service} value, no comparator):

validation:
performance:
checks:
- nccl-all-reduce-bw-net
- nccl-all-reduce-bw-nvls
constraints:
- name: nccl-benchmark-profile # run the GB200-on-EKS benchmarks
value: gb200/eks
- name: nccl-all-reduce-bw-net
value: ">= 40"
- name: nccl-all-reduce-bw-nvls
value: ">= 500"

The profile selects the benchmark’s template and environment handling — transport-class runtime template, EFA/TCPXO discovery, worker scheduling defaults, and preflight checks — as if the cluster were the named pair, so pick the profile whose fabric matches the hardware. GPU nodes are still matched by the recipe’s own criteria.accelerator (via the GFD nvidia.com/gpu.product label) when a matcher exists, but the profile’s service keys the rest of node handling: an …/eks profile narrows workers to a single node.kubernetes.io/instance-type, and the …/any profiles (b200/any, gb200/any) require an explicit --node-selector identifying the GPU nodes, exactly as service: any recipes do. When --node-selector is passed it replaces the automatic filters rather than narrowing them.

Valid profiles are the pairs in the applicability table above: b200/any, gb200/any, gb200/eks, gb200/gke, gb200/nvke, gb200/oke, gb300/eks, gb300/generic, gb300/nvke, gb300/oke, h100/aks, h100/eks, h100/gke, h200/eks, vr200/nvke, vr200/rke2. A malformed or unknown value fails the check rather than silently skipping it. A valid profile that doesn’t implement a requested variant (e.g. gb200/eks with the auto-detect nccl-all-reduce-bw check) skips just that check with a message naming the profile. When a profile is set it wins over criteria-derived applicability, and the pass/fail thresholds stay in the same-named check constraints as always.

Supplying a benchmark runtime for a private service

A profile only helps when an embedded template already matches the cluster’s hardware and fabric. A private service introduced entirely through --data — a criteria.service (and possibly criteria.accelerator) the validator ships no template for, with a fabric that reuses none of the embedded ones — has no compiled pair to point a profile at. Such a recipe supplies the benchmark itself: it ships a Kubeflow TrainingRuntime as a file in its --data tree and references it with the nccl-benchmark-runtime-ref performance constraint (a bare {accelerator}/{service} value). aicr validate reads that file and gates the benchmark keyed on the recipe’s own criteria, with no compiled applicability entry required.

Lay the runtime out exactly where the embedded templates live, so it is a drop-in for upstreaming later:

mydata/ # your --data directory
├── overlays/
│ └── mycloud-training.yaml # the recipe overlay (below)
└── validators/performance/testdata/
└── gb200/mycloud/runtime.yaml # the TrainingRuntime, real file
# overlays/mycloud-training.yaml
validation:
performance:
checks:
- nccl-all-reduce-bw
constraints:
- name: nccl-all-reduce-bw
value: ">= 450"
- name: nccl-benchmark-runtime-ref
value: gb200/mycloud # -> validators/performance/testdata/gb200/mycloud/runtime.yaml
# validators/performance/testdata/gb200/mycloud/runtime.yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: nccl-all-reduce-runtime
spec:
template:
spec:
replicatedJobs:
- name: node # required: the worker cohort
template:
spec:
template:
spec:
containers:
- name: node
image: <your NCCL benchmark image>
resources:
limits:
nvidia.com/gpu: "${GPU_COUNT_PER_NODE}"
# ...command, fabric resources, sidecars...
aicr validate -r recipe.yaml -s snapshot.yaml \
--data ./mydata --phase performance

The supplied runtime owns its fabric wiring end to end: because the recipe opted in explicitly, the validator bypasses the compiled applicability gate and skips every service-specific setup step — EFA/TCPXO/RDMA NIC discovery, the GB200-NVreg / GKE-TCPXO preflights, and NVLS/IMEX auto-provisioning (except as described for IMEX below). It renders the runtime, sizes it (against the runtime’s own nodeSelector when it pins one, so WorkerCount matches placement), applies it alongside the shared TrainJob, and evaluates the bandwidth threshold from the launcher logs. Transport verification is not skipped: pairing the runtime with the -net or -nvls check still asserts that transport actually carried traffic (the NCCL markers are fabric-agnostic), so a named variant can’t pass on bandwidth alone. What it must honor:

  • It must be a Kubeflow TrainingRuntime (apiVersion: trainer.kubeflow.org/v1alpha1); any other kind/apiVersion is rejected. The validator force-sets its name and namespace, so a recipe can supply a TrainingRuntime and nothing else — never an arbitrary resource.

  • It must declare a node replicatedJob (the worker cohort the shared TrainJob sizes and the validator injects scheduling into and reads logs from), mirroring the embedded templates.

  • The generic template variables ${NAMESPACE}, ${WORKER_COUNT}, ${GPU_COUNT_PER_NODE}, ${GPU_COUNT}, ${TEST_TYPE}, ${MIN_MESSAGE_SIZE}, and ${MAX_MESSAGE_SIZE} are substituted; the service-specific ones (EFA/RDMA/GKE) are not, so the runtime must pin its own fabric resources.

  • The runtime should pin its own worker nodeSelector/tolerations, or pass --node-selector to aicr validate, so workers land on the intended GPU nodes.

  • IMEX access uses the validator-managed claim template. The benchmark runs in a per-run namespace that does not exist before the run, so nothing can be pre-created there for the runtime. To get an IMEX channel, reference the ResourceClaimTemplate nccl-all-reduce-imex from the pod’s resourceClaims, and bind that claim in the node container’s resources.claims. Kubernetes exposes a pod-level claim only to containers that bind it, so the pod-level reference alone leaves the worker without the channel. The validator then creates the ComputeDomain that backs it, on any check variant, and waits for the template before starting the run.

    # the node replicatedJob's pod spec
    resourceClaims:
    - name: imex-channel
    resourceClaimTemplateName: nccl-all-reduce-imex
    containers:
    - name: node
    resources:
    claims:
    - name: imex-channel
    limits:
    nvidia.com/gpu: "$\{GPU_COUNT_PER_NODE\}"
  • Any other pod-level resourceClaimTemplateName, and any resourceClaimName, is rejected: no such claim could exist in the per-run namespace.

nccl-benchmark-runtime-ref and nccl-benchmark-profile are mutually exclusive — a recipe supplies its own runtime or borrows an embedded one, never both. An absent or blank ref falls back to criteria/profile-derived applicability. A malformed ref, or a file missing from --data, is rejected by aicr validate before any validator Job is deployed (a resolution error on stderr). A resolved file that is not a TrainingRuntime with a node replicatedJob, or that references an unsupported claim, fails the check itself (in the pod), before any benchmark resources are applied. Either way it fails rather than silently skipping.

Sizing note. The worker cohort is sized against the runtime’s own nodeSelector when it pins one, but richer scheduling constraints (nodeAffinity, un-tolerated node taints) are the Kubernetes scheduler’s job, not the validator’s — a runtime that restricts placement below the sized cohort surfaces as a launcher timeout, not a false pass. Pass an explicit --node-selector when you want the sized and scheduled cohorts to match exactly.

Upstreaming. Because the file already sits at validators/performance/testdata/{accelerator}/{service}/runtime.yaml, adopting it into AICR is a copy: move the file into the repo, add the accelerator/service tuple to the compiled supportedNCCLCombinations matrix, and drop the nccl-benchmark-runtime-ref constraint — the benchmark then runs from the recipe’s plain criteria like any other embedded one.

Cluster Readiness Engine checks

An opt-in pair of Cluster Readiness Engine (CRE) checks runs against public CRE (nvcre.nvidia.com, cluster-readiness-engine). Neither check branches on service or accelerator; which combinations are qualified is recorded as data. Shipped overlays keep the TrainJob nccl-all-reduce-bw path, so nothing below runs unless you ask for it.

Enabling a CRE check takes two steps and needs both. Add the nvcre component as described in Enabling NVCRE — a bare component reference without that section’s valuesFile and Trainer dependency hits the failures documented there — and then declare the check with a same-named constraint:

validation:
performance:
checks:
- nccl-cre-all-reduce-bw
- cre-training-goodput
constraints:
- name: nccl-cre-all-reduce-bw
value: '>= 300' # bus bandwidth in GB/s
- name: cre-training-goodput
value: '>= 0.5' # ratio between 0 and 1, not a percentage

Both constraints are floors. The checks compare the measured value with >= and refuse any other operator rather than evaluating it as a floor anyway, so a constraint they cannot honor fails before any GPU work starts. The bandwidth check applies the constraint the way nccl-all-reduce-bw does: it passes at 90% of the value, so the same >= 300 means the same under either backend. The goodput check passes at the value itself. Note the goodput units: the measurement is a ratio between 0 and 1, so a constraint written as a percentage (>= 50) can never be satisfied.

CheckWhat it measuresQualified combinations
nccl-cre-all-reduce-bwBus bandwidth from a CRE Certification BandwidthMeasurement; AICR still asserts the transport from launcher logseks x h100
cre-training-goodputRuntime goodput from a CRE Certification GoodputMeasurementeks x h100

The catalog entry each check drives, and the node footprint it runs on, come from the qualification record rather than the check: for nccl-cre-all-reduce-bw on eks x h100, communication/nccl-all-reduce on two nodes, and for cre-training-goodput on eks x h100, training/nemotron5-8b on two nodes. A combination that no entry records as qualified skips with an unsupported-target reason.

The record also pins each entry’s workload image by digest, and the check submits it as the category’s options.image, so the run pulls the bytes the closure records rather than whatever the catalog’s tag points to. The pins are the images the catalog selects on eks x h100, at the digests recipes/components/nvcre/workload-images.yaml records for the CRE version the registry pins, and the checks submit them whatever CRE version the cluster runs: the EFA nccl-tests image for nccl-cre-all-reduce-bw, and nvcr.io/nvidia/pytorch:25.08-py3 for cre-training-goodput. With the pin, the NCCL launcher’s SSH-permissions init container also runs nccl-tests rather than nvcr.io/nvidia/pytorch:26.01-py3, so the NCCL check no longer pulls that image. A mirror for a disconnected cluster has to serve these two images at the recorded digests.

Relation to the TrainJob check

On eks x h100, nccl-cre-all-reduce-bw reads 1.65 GB/s (0.34%) higher than nccl-all-reduce-bw on the same two nodes: six back-to-back pairs on NVCRE v0.8.2 gave 486.76 to 489.01 GB/s for the TrainJob check and 489.63 to 490.03 GB/s for the CRE check, with a per-pair offset of 0.92 to 3.14 GB/s. Both ran over Using network Libfabric; the CRE check asserts that from its launcher log, and the TrainJob check’s default template logs at NCCL_DEBUG=WARN so it cannot show it. That is the one difference in what the two checks demand: the CRE check fails a run that fell back to Socket, and the TrainJob check does not look. Both pass at 90% of the constraint value, so a recipe can carry the same >= 300 for either backend and both use the same floor. The two measured bandwidths differ by 0.92 to 3.14 GB/s per pair (1.65 GB/s on average), so a cluster that lands near the floor can still pass one check and fail the other. The CRE check applied the value without the 10% allowance before this was aligned; a CRE constraint that had been written as 0.9 times the TrainJob value to compensate should be reset to the TrainJob value. The offset was measured on one cluster, and no run landed near the boundary, so it does not show that the two verdicts agree at it.

The goodput figure depends on the step count, because startup time counts against it. The step count comes from the NVCRE catalog entry, so a figure belongs to the CRE version that produced it: 0.5723 on two nodes with the pinned v0.8.2 (462.05 TFLOPS per GPU), and 0.5835 on v0.6.0, both measured on a NodeWright-tuned cluster. Both are single measurements, so set the constraint as a floor below the value your cluster reaches, not at it. Run it on nodes that completed NodeWright tuning: on a cluster that had not, with the distro EFA driver, no hugepages, and an 8 MB memlock limit, a two-node run hung in NCCL initialization. Which of those differences causes it was not isolated.

Reclaiming an interrupted run. Both checks delete their Certification and confirm its TrainJobs and workload pods are gone, but a validator pod killed mid-run (eviction, OOM kill, its Job deleted under it) never gets to. Its workloads keep holding the GPU nodes, and because Certification names carry a random suffix, nothing in the name identifies them. Every Certification AICR submits therefore carries app.kubernetes.io/managed-by: aicr-validator, and the next CRE check in that namespace tears down any it finds older than 45 minutes — longer than a live run can take, so a concurrent validator’s Certification is never swept. To clean up by hand instead:

NAMESPACE="nvcre" # replace with the namespace you validated
CERTIFICATION="aicr-cre-nccl-1a2b3c4d" # replace with a name from the listing
kubectl get certifications.nvcre.nvidia.com -A \
-l app.kubernetes.io/managed-by=aicr-validator
kubectl delete certifications.nvcre.nvidia.com "${CERTIFICATION}" -n "${NAMESPACE}"
# Deleting the Certification is not sufficient on its own: CRE's controller
# drops its finalizer without waiting for the workloads.
kubectl get trainjobs,pods -n "${NAMESPACE}" -l nvcre.nvidia.com/job

Inference performance validation

Inference performance runs the inference-perf check — deploys a DynamoGraphDeployment with a vLLM-served model (Qwen/Qwen3-8B by default, overridable per accelerator — see below) plus an AIPerf benchmark Job, and measures end-to-end output-token throughput and time-to-first-token (TTFT) p99.

Warm-up: AIPerf sends a wave of warm-up requests before the measured run, so vLLM’s one-time CUDA-graph / JIT compilation (tens of seconds on a cold worker) is excluded from the reported throughput and p99 TTFT — the numbers reflect steady state, not cold start. Warm-up scales with concurrency and is tunable via AICR_INFERENCE_PERF_WARMUP_PER_CONCURRENCY (see the validator reference).

Determinism: the benchmark is driven reproducibly so the verdict reflects the deployment, not run-to-run RNG — a fixed random seed, fixed input/output token counts (stddev 0), a pinned synthetic-prompt pool, and greedy decoding (temperature: 0). Note that throughput (not the latency tail) is the stable, discriminating signal at high concurrency; TTFT p99 near the saturation knee can still vary with batching/scheduling, which is why the TTFT constraint is a generous ceiling rather than a tight target.

# Capture snapshot, generate inference recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml
aicr recipe --service eks --accelerator h100 --os ubuntu \
--intent inference --platform dynamo \
--output recipe.yaml
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance

The generated recipe includes dynamo-platform in componentRefs and lists inference-perf under validation.performance.checks with pass/fail constraints plus benchmark inputs:

validation:
performance:
checks: [inference-perf]
constraints:
# Pass/fail thresholds (10% tolerance applied by the evaluator). Values
# shown are the measured H100 gate at 8B/256; each overlay sets its own.
- name: inference-throughput # output tokens/sec
value: ">= 50000"
- name: inference-ttft-p99 # time-to-first-token p99 in ms
value: "<= 2000"
# Optional per-accelerator inputs (bare value, no comparator).
# Precedence: recipe > AICR_INFERENCE_PERF_* env > compiled default.
- name: inference-model # HF model ID; default Qwen/Qwen3-8B
value: Qwen/Qwen3-8B
- name: inference-concurrency-per-gpu # positive integer; default 256
value: "256"
- name: inference-routing-mode # dynamo-router or gateway-epp
value: dynamo-router

Node-shape assumption. The inference-throughput floor is a fixed absolute full-node value calibrated on a full node (the shared >= 50000 gate was measured on 8-GPU H100; GB200 on a 4-GPU node). It is not normalized for GPU count, and the evaluator only scales it down for partial occupancy — not across node sizes — so a smaller H100 SKU (e.g. 1-/2-GPU shapes such as p5.4xlarge, AKS NC80adis) can false-fail a healthy run. inference-ttft-p99 is a per-request latency at fixed concurrency-per-GPU and does not need GPU-count normalization. A normalized per-GPU throughput floor is tracked in #1254.

inference-model and inference-concurrency-per-gpu are optional: omit them to use the compiled defaults (Qwen3-8B at 256 concurrent requests per GPU), set them per overlay to tune model and load for each accelerator, or override globally with the AICR_INFERENCE_PERF_MODEL / AICR_INFERENCE_PERF_CONCURRENCY_PER_GPU catalog knobs (recipe wins over catalog env wins over default).

inference-routing-mode selects the Dynamo Kubernetes routing path. The default dynamo-router mode deploys a Dynamo frontend with load-aware least-loaded routing (DYN_ROUTER_MODE=least-loaded), which balances by each worker’s active in-flight load so a transiently-slow worker stops receiving its full share (see issue #1197). Normal frontend-to-worker request/response traffic uses Dynamo’s request plane (Dynamo 1.4+ defaults to TCP); AICR does not set DYN_REQUEST_PLANE=nats. Workers publish KV-cache events directly over ZMQ with no NATS relay, but least-loaded routing does not consume those events — only DYN_ROUTER_MODE=kv does. Set it to gateway-epp to exercise GAIE/EPP: the validator deploys an EPP component, worker frontend sidecars in direct mode, and an HTTPRoute through the recipe’s inference gateway (agentgateway by default; a recipe whose components include no inference gateway fails the run before anything is deployed). The direct-mode sidecars honor EPP routing headers; they do not relay KV events.

Model-weights cache and AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS. The benchmark downloads the model once into a PVC and serves all workers from it (on by default; avoids per-IP Hugging Face throttling). The cache PVC needs a StorageClass: it uses the cluster’s default StorageClass unless you set one, with precedence recipe constraint > catalog env > cluster default. Set it per accelerator via the inference-model-cache-storage-class performance constraint, or globally via AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS=<name> (e.g. gp2/gp3 on EKS, standard-rwo on GKE, except A4X/GB200 a4x-highgpu-4g nodes, which reject standard-rwo’s pd-balanced disks and need a Hyperdisk-backed class instead. See GKE GB200 Storage Prerequisites) on the inference-perf catalog entry’s env (or via a catalog overlay in the aicr validate --data <dir> directory). On a cluster with no default StorageClass (common on EKS, since some clusters ship only a non-default gp2) and neither set, the check fails fast in seconds with guidance rather than hanging. Disable the cache instead with AICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off. Unlike the recipe constraint, the catalog env knob is not read from the shell environment of the process running aicr validate (only HF_TOKEN is). AICR-deployed EKS clusters get a default gp3 StorageClass from the aws-ebs-csi-driver component, so the cache works there with no knob, provided the driver has AWS credentials — see EBS CSI Driver Credentials.

Debugging a failed run with AICR_INFERENCE_PERF_NO_CLEANUP. By default the validator deletes the per-run namespace (DGD, workers, frontend, AIPerf Job) on both success and failure. To investigate a failure — e.g. a timed out waiting for inference endpoint to serve requests — set AICR_INFERENCE_PERF_NO_CLEANUP=1 and the validator leaves everything in place so you can kubectl logs the frontend/workers and curl /v1/models and /v1/chat/completions live. Unlike the other AICR_INFERENCE_PERF_* knobs, this one is read from the shell environment of the process running aicr validate (forwarded to the inference-perf pod, like HF_TOKEN), not from the catalog. Debug-only: you must delete the aicr-inference-perf-<suffix> namespace manually afterward, or it keeps GPU workers running.

Expected flow (~5–7 min on H100): readiness pre-flight → deploy a DynamoGraphDeployment in a per-run namespace aicr-inference-perf-<8-hex-suffix> → wait for state=successful (image pull

  • model load) → endpoint readiness via a real /v1/chat/completions inference request (stricter than a /health probe, which returns 200 before the model can actually serve) → AIPerf benchmark Job parses throughput + TTFT p99 → compare to recipe constraints (10 % tolerance) → cleanup. Worker GPU wiring is configuration-selected and capability-verified for recipe-backed runs — the recipe’s resolved allocation policy picks the mechanism, and the inspection probe (the same one the conformance checks use) verifies the cluster serves it, failing closed on mismatch; only recipe-less standalone runs (unspecified) select by capability. The wiring supports exactly two GPU allocation configurations: (1) nodes publishing node-local gpu.nvidia.com ResourceSlices (full-GPU DRA — the experimental opt-in; stock recipes default to device-plugin allocation since the #1327 flip), where workers bind a DRA ResourceClaimTemplate sized from the validated per-node device count — kai-scheduler treats nodes bearing raw node-local GPU ResourceSlices as DRA-only and rejects scalar GPU requests, so the claim path is the only schedulable wiring there; and (2) ComputeDomain-only / no full-GPU slices (device-plugin nodes — the production default), where workers request GPUs via nvidia.com/gpu limits, which need no gpu.nvidia.com DeviceClass. Anything outside those two states fails fast with an actionable error instead of guessing: full-GPU ResourceSlices from a non-NVIDIA GPU driver, allocated ResourceClaims requesting a non-NVIDIA “gpu”-named DeviceClass, gpu.nvidia.com slices using non-node-local topologies (nodeSelector/allNodes/per-device node selection), pools published by node-local slices of multiple nodes, and allocated ResourceClaims from pools no node-local slice publishes (occupancy is attributed to nodes through the slices’ spec.nodeName — the K8s API does not require pool names to be node names) are all rejected — these are outside the NVIDIA DRA driver’s supported configuration (#1327); generalized topology/driver support is tracked in #1652. When the recipe configures a GPU allocation policy (see Configured GPU allocation policy), the wiring is forced instead of capability-selected, and under dra-resource-claim candidate GPU nodes are discovered from the probe’s validated DRA facts rather than scalar nvidia.com/gpu allocatable — so DRA-only nodes (device plugin disabled) are discoverable; full DRA-only aicr validate remains experimental (see the #1327 graduation checklist). The chosen node and wiring mode are recorded in the check’s evidence output.

All Dynamo Frontend and worker pods pin to a single GPU node via kubernetes.io/hostname for a stable per-node baseline. On a shared cluster where some GPUs on a candidate node are already in use by other workloads, the validator picks the candidate with the most free GPUs within the node’s own allocation ledger and sizes the benchmark to that count: DRA ResourceClaim allocations subtract from DRA capacity, and device-plugin nvidia.com/gpu occupancy from device-plugin capacity. A DRA-wired candidate carrying any scalar nvidia.com/gpu workload is skipped entirely — the device plugin’s physical device assignments are invisible to the DRA allocator, so claims placed there could double-book a plugin-held GPU. The check therefore does not need an explicit hostname override to avoid saturated nodes. The inference-throughput gate is a full-node baseline, so when the benchmark runs on fewer than the node’s full GPU count the gate is scaled down by the same freeGPUs / nodeGPUs fraction (throughput scales ~linearly at fixed concurrency-per-GPU) — a healthy per-GPU result on a partially occupied node is not failed against a full-node number. TTFT p99 is a per-request latency and is not scaled. Concurrent aicr validate invocations are isolated from each other by the run-specific suffix on both the namespace and the inner AIPerf Job name.

A passing CTRF entry (measured on EKS H100, 8 × H100 GPUs, Qwen/Qwen3-8B at 256 concurrency/GPU):

{
"name": "inference-perf",
"status": "passed",
"suite": ["performance"],
"stdout": [
"RESULT: Inference throughput: 108789.87 tokens/sec",
"RESULT: Inference TTFT p99: 687.50 ms",
"Throughput constraint: >= 50000 → PASS",
"TTFT p99 constraint: <= 2000 → PASS"
]
}

The RESULT: prefix on the first two lines is the contract documented in pkg/validator/validator.go — any check that wants its summary lines echoed to the CLI’s own output (not just the CTRF report) opts in by emitting that prefix. The validator runtime strips the prefix when echoing; the full prefixed line stays in stdout[].

To run deployment validation first (recommended — verifies GPU Operator, DRA driver, Dynamo operator, KAI scheduler, and supporting components are installed and healthy):

aicr validate --recipe ./bundles/recipe.yaml --snapshot snapshot.yaml --phase deployment

Skip scenarios

The inference validator has three explicit skip guards so it never runs where it can’t succeed. Each produces a status: skipped CTRF entry with a specific reason. Skipped checks are not failures: the validator container exits with code 2 internally (mapped to CTRF skipped), but aicr validate itself exits 0 for a skipped or passed phase — a skipped inference check never drives a non-zero CLI exit on its own. (A phase that reports failed or other exits 8 by default, and is informational only under --fail-on-error=false.)

GuardTriggerSkip message
ARecipe lists inference-perf in checks: but no matching inference-throughput / inference-ttft-p99 constraintsno inference-throughput or inference-ttft-p99 constraint in recipe
Binference-perf is selected but dynamo-platform is not in recipe componentRefsskipped - dynamo-platform not in recipe components
Cdynamo-platform is declared but the DynamoGraphDeployment CRD is not installed on the cluster (operator not deployed yet)skipped - DynamoGraphDeployment CRD not installed on cluster (dynamo-platform component declared but operator not deployed yet)

Guards fire before any cluster mutation, so skips are cheap (typically < 10 s).

Configured GPU allocation policy

When you validate with a recipe, AICR resolves a whole-GPU allocation policy from the recipe’s fully hydrated component values and carries it to the validators (#1327):

PolicyMeaning
device-plugin-extended-resourceWhole GPUs via the device plugin (nvidia.com/gpu requests)
dra-resource-claimWhole GPUs via DRA (gpu.nvidia.com ResourceClaims)
dra-extended-resourceReserved (KEP-5004 mapped extended resource) — not yet validated
unspecifiedNo recipe context (standalone validator runs): capability-driven automatic selection

The nvidia-dra-driver-gpu value resources.gpus.enabled is the switch: true resolves dra-resource-claim; explicit false — or the component absent or disabled — resolves device-plugin-extended-resource. On an enabled DRA component the switch must be explicitly set: the upstream chart’s declared default is true, so an absent value would diverge from what Helm deploys. These configurations are rejected at resolution time with an invalid-request error:

  • an enabled nvidia-dra-driver-gpu component with resources.gpus.enabled absent (pin it explicitly in the recipe; stock recipes always do).
  • gpus.enabled=true without gpuResourcesEnabledOverride=true (the upstream chart install guard refuses it), and no whole-GPU advertiser remaining.
  • gpus.enabled off with the GPU operator component (gpu-operator, or gpu-operator-ocp on OpenShift recipes) absent, disabled, or carrying devicePlugin.enabled=false.
  • A recipe enabling both gpu-operator and gpu-operator-ocp; two GPU operators collide at the operand level; enable exactly one (OpenShift recipes disable gpu-operator and carry gpu-operator-ocp).

Two further states are likewise rejected (they warned during the transition to the device-plugin production default and are errors since the flip):

  • dual advertisement (both mechanisms enabled — exactly one whole-GPU advertiser is required)
  • an inert gpuResourcesEnabledOverride=true with gpus.enabled=false (the waiver would disarm the upstream chart’s install-guard tripwire).

Stock recipes ship the production default: gpus.enabled=false, gpuResourcesEnabledOverride=false, and devicePlugin.enabled=true; the experimental DRA opt-in flips all three together in a recipe overlay. On GKE-COS the gpuStack=dra profile value (#2517) is the stock selection of exactly that opt-in tuple — the one stock selection resolving dra-resource-claim, still experimental pending the #1327 graduation checklist (#3282); see GKE GPU Setup.

Upgrading a cluster from the dual-advertised (pre-flip) configuration: applying the flipped bundle does not drain existing workloads — a running full-GPU gpu.nvidia.com claim pod keeps its prepared GPU while the nvidia.com/gpu ledger, which cannot see that assignment, admits newly converted scalar workloads onto the same device. Migrate in this order:

  1. Stop/delete all workloads holding full-GPU gpu.nvidia.com ResourceClaims (ComputeDomain/IMEX claims are unaffected and stay).
  2. Confirm no allocated or reserved full-GPU claims remain, using the driver-specific, fail-closed drain check below. On IMEX platforms, allocated compute-domain.nvidia.com claims legitimately remain, so a plain kubectl get resourceclaims -A can never come back empty — filter on the allocation driver instead, and treat ANY output (or any query failure) as unsafe to proceed.
  3. Apply the flipped bundle (upgrade nvidia-dra-driver-gpu).
  4. Wait until the gpu.nvidia.com ResourceSlices and DeviceClass disappear (compute-domain.nvidia.com slices must remain).
  5. Confirm scalar nvidia.com/gpu allocatable is present on the GPU nodes.
  6. Only then start the scalar (device-plugin) workloads.

The step-2 drain check — capture and test each stage separately (a piped kubectl ... | jq ... masks a failed List as an empty, safe-looking result), and proceed only when it exits 0:

claims="$(kubectl get resourceclaims -A -o json)" || { echo "claim query FAILED — do not proceed"; exit 1; }
unsafe="$(printf '%s' "$claims" | jq -r '
.items[]
| select([.status.allocation.devices.results[]?.driver] | index("gpu.nvidia.com"))
| "\(.metadata.namespace)/\(.metadata.name) reservedFor=\([.status.reservedFor[]?.name] | join(","))"')" \
|| { echo "claim filter FAILED — do not proceed"; exit 1; }
[ -z "$unsafe" ] || { printf 'unsafe full-GPU claims remain:\n%s\n' "$unsafe"; exit 1; }
echo "no full-GPU gpu.nvidia.com claims — safe to proceed"

While migrating, also delete any pending (unallocated) claims whose spec references the gpu.nvidia.com DeviceClass: they hold no GPU (no over-admission risk, so the check above rightly ignores them), but the flip removes that DeviceClass, leaving such claims — and any pods referencing them — stranded unschedulable forever.

Validators compare the configured policy against the inspected cluster state and fail closed on mismatch — a cluster that cannot serve the configured mechanism is a validation failure by design, never a silent fallback to the other mechanism. Only recipe-less standalone runs (unspecified) keep the capability-driven automatic selection.

Running all phases

aicr validate --recipe ./bundles/recipe.yaml --snapshot snapshot.yaml
# equivalent to: --phase deployment --phase conformance --phase performance

Phases run sequentially. By default all phases run and produce results regardless of earlier failures. Pass --fail-fast to stop after the first phase that fails (e.g., to skip a 65-minute inference-perf run when deployment already failed).

Skipping checks a run cannot satisfy

--phase selects whole phases. --skip-check (or spec.validate.execution.skipChecks in a config file) works one level down, withholding named checks from every phase that runs:

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
--phase conformance \
--skip-check gpu-operator-health --skip-check dra-support

It is for a caller that cannot satisfy a check the recipe declares, typically a CI lane that deploys only part of the recipe or runs against simulated devices. It narrows the run, not the recipe: every other consumer of that recipe still gets the check, which is why this is not expressed as a recipe edit.

Three properties make it a scoping tool rather than a way to hide a failure:

  • A skipped check is reported, not dropped, and its reason reaches the signed bundle. It appears in the CTRF report as skipped with the reason named in skipChecks, and as the structured code extra.skipReason: named-in-skip-checks. The default (minimal) recipe-evidence bundle (--emit-attestation) does not carry the report verbatim: its redaction policy blanks every test’s message and stdout, which is why the reason also rides the allowlisted extra channel. So both the withheld check and why it was withheld travel with the attestation; --full keeps the prose message too.
  • The run fails closed on a list that would not do what it says. A name matching no check in the catalog is rejected before any validation resource is created, and so is a list that would leave a requested phase with nothing to run (that phase would otherwise report passed while running nothing, because the skipped entries keep its test count above zero). Stop requesting the phase instead. The recipe is loaded first, so a cm:// recipe is read from the cluster before the list is judged.
  • A check added later is not silenced. A skip list names what to withhold, so a new check in a recipe runs and can fail, which forces a decision rather than hiding one.

The CNCF conformance evidence path refuses the two flags together for a different reason. pkg/evidence/cncf renders one markdown file per requirement of the upstream checklist version selected from the cluster’s Kubernetes minor (see Checklist Versions) and aggregates each requirement’s status fail-closed from every check expected to evidence it (see Requirement status). The renderer records a skipped check as NOT TESTED with its reason code (for example slurm-managed-gpus, when secure-accelerator-access defers to the Slurm path and slinky-slurm-gpu-access did not run), so a skip is reported rather than hidden, but a run that withholds a check the recipe declares is narrower than the recipe by construction and cannot evidence that recipe. --skip-check is for lanes that cannot host a check at all; those are not submission lanes, so --skip-check with --evidence-dir (and therefore with --cncf-submission; the two flags require each other) is rejected up front:

[INVALID_REQUEST] --skip-check cannot be combined with --evidence-dir: a
withheld check leaves its requirement NOT TESTED, so the run is narrower than
the recipe by construction and cannot produce conformance evidence for it

The requirement aggregation, and the spec.validate.evidence.cncf.notApplicable key it reads, apply to the evidence a submission run renders from the validator run.

CNCF submission evidence

aicr validate --phase conformance is the only runner for CNCF AI Conformance: the recipe declares which conformance checks run, and every run executes all of them. --cncf-submission adds one thing to that run: after the checks finish, the CTRF report is rendered into the submission directory a vendor files as vX.Y/<product>/ in cncf/k8s-ai-conformance, under --evidence-dir: PRODUCT.yaml (statuses from the run, metadata from spec.validate.evidence.cncf.vendor), README.md (the index, with the platform facts the notes cite), junit.xml and e2e.log (one testcase per check, named by check) and one evidence page per requirement a non-skipped check evidenced under evidence/, redacted in one pass and checked by the pinned upstream validator as the last step. The flag never changes what a check does or how it decides; for the same recipe the verdicts are identical with and without it. The two flags require each other, --recipe is required because the checks are selected from it, the vendor block is required in --config, and --no-cluster and --skip-check are refused: submission evidence comes from checks that ran on a live cluster against the whole recipe. --evidence-dir must be absent or empty, with no parent path segment of the form vX.Y/ (the upstream validator would read it as the checklist version); the directory reaches it only when the run is complete, and an incomplete run leaves its staged copy at <evidence-dir>.incomplete/ instead, which the next run refuses with exit 2 until it is removed. The checklist version is the cluster’s Kubernetes minor and must be one this build carries (v1.35 to v1.37); a cluster on another minor is refused with exit 2 rather than certified against a neighboring checklist, unless spec.validate.evidence.cncf.conformanceVersion names one deliberately (see Submission directory for the status, export and manual-attestation rules).

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
--config aicr-config.yaml \
--phase conformance \
--cncf-submission \
--evidence-dir ./submission

The config carries the vendor metadata and, for cluster_autoscaling, either the conditional justification when the cluster has no autoscaler (notApplicable, shown) or the documentation that attests a requirement the run only observed (documented, below); see evidence.cncf:

kind: AICRConfig
apiVersion: aicr.run/v1beta1
spec:
validate:
evidence:
cncf:
vendor:
name: Example Vendor # exactly as listed under CNCF Members in the landscape
platformName: Example Platform
platformVersion: 1.0.0
description: One sentence describing the offering.
contactEmail: maintainers@example.com
websiteUrl: https://example.com/platform
documentationUrl: https://example.com/platform/docs
productLogoUrl: https://example.com/platform/logo.svg
k8sConformanceUrl: https://github.com/cncf/k8s-conformance/tree/master/v1.37/example-platform
notApplicable:
cluster_autoscaling: "Fixed-size node pools; no cluster autoscaler or Karpenter is deployed."

On a cluster whose autoscaler the cluster-autoscaling check observed but did not drive, replace notApplicable with a documented entry; the run’s OBSERVED result is then established by the documentation, which PRODUCT.yaml cites instead of the testcase:

documented:
cluster_autoscaling:
evidence:
- https://example.com/platform/docs/cluster-autoscaler
notes: "The managed cluster autoscaler scales the GPU node pool on pending accelerator pods."

The --feature flag and spec.validate.evidence.cncf.features key scoped the bash collector and went with it (#3602): the flag is gone, and a config that still sets the key does not load (see Deprecations). The same run is available from the Go SDK as Client.CollectCNCFEvidence (see Go library).

Exit status of a submission run

A submission run adds the requirement aggregation (see Requirement status) on top of the per-check rule every run applies:

  • Flag and input refusals, and the readiness pre-flight, exit 2 before any check runs, as for every validate run.
  • A check that failed or crashed fails the run under --fail-on-error, as it does without the flag. Conformance checks with no requirement mapping (platform-health, the Slinky health checks) are invisible to the aggregation, so this rule is what catches them.
  • An applicable MUST requirement that no proof path establishes fails the run with INTERNAL after the evidence is written, so the directory still documents what was and was not shown. A path is established when every check on it is present and passed as a demonstration; a check that failed on any path fails the requirement outright, and a check that was skipped, crashed or is absent leaves only its own path incomplete. A check that passed but marked its result observation-only (it recorded the cluster’s arrangement and drove no behavioral test; today cluster-autoscaling on every path that passes) leaves its requirement OBSERVED, which is not established either: the exit message names it, e.g. cluster_autoscaling observed (cluster-autoscaling observed, not demonstrated). Only a cluster with no cluster autoscaler at all (fixed-size node pools, no Karpenter) may declare the requirement not applicable through a spec.validate.evidence.cncf.notApplicable justification, which renders NOT APPLICABLE and removes only that requirement from the verdict: the run still exits nonzero when another applicable MUST is not established or when a check itself failed (on EKS the cluster-autoscaling check fails when no autoscaler is confirmed, and the default --fail-on-error still applies). A platform whose autoscaler is present but not behaviorally demonstrated keeps OBSERVED and is established only by a spec.validate.evidence.cncf.documented entry naming public documentation, which PRODUCT.yaml then cites instead of the testcase. Where a requirement has alternative proofs (secure_accelerator_access: secure-accelerator-access on ordinary recipes, slinky-slurm-gpu-access on Slinky Slurm recipes) a passing alternative establishes it even when the primary check was skipped or did not run. A requirement declared not applicable is not counted. This verdict applies regardless of --fail-on-error=false.
  • The aggregation itself never fails the run on a SHOULD, whatever its checks did; its page records the real status. A SHOULD-backed check that failed still trips the per-check rule above (pass --fail-on-error=false to accept it), while the submission artifacts export its testcase as skipped so the directory stays submittable.
  • A write or redaction error, or the pinned upstream validator rejecting the directory, fails the run with INTERNAL after the recipe-evidence attestation (if any) was emitted; a validator that cannot start because the CNCF landscape is unreachable fails it with SERVICE_UNAVAILABLE. Flag and config refusals, including a missing or malformed vendor field and a non-empty --evidence-dir, exit 2 before any check runs.

A complete submission directory exists at --evidence-dir only when the run’s requirement verdict is clean and the validator accepted it; otherwise the staged copy at <evidence-dir>.incomplete/ documents what was and was not shown.

Emitting recipe evidence

When a recipe PR targets hardware AICR maintainers cannot independently re-run, the contributor needs to attach a signed evidence bundle so a maintainer can verify the recipe offline. aicr validate produces the bundle as a side effect when --emit-attestation is set; adding --push signs it (cosign keyless via Sigstore) and uploads it to an OCI registry. This is a different artifact from the CNCF-submission evidence above — the two flag families produce independent outputs and may run from a single aicr validate invocation.

OWNER="my-github-org" # replace with your GHCR owner
aicr validate \
--recipe recipe.yaml \
--snapshot snapshot.yaml \
--emit-attestation ./out \
--push "ghcr.io/${OWNER}/aicr-evidence"

The --push tag is just a human-readable label — the sha256: digest is what pins the bundle, so tag choice never affects verification (the verifier pulls by digest). Omit the tag, as above, and aicr derives a unique per-recipe one, <recipe-slug>-<short-fingerprint> (e.g. ghcr.io/<owner>/aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e), so distinct attestations never collide on a shared tag. Pass an explicit tag to override.

When the run’s workspace does not outlive it, for example in a Kubernetes Job or an ephemeral CI step, add --image-refs <file> to write the pushed bundle’s digest (the bundle.digest value in pointer.yaml) to a file you choose, such as the container’s termination-message path (/dev/termination-log by default). The file gets the same sha256:… line aicr bundle --image-refs writes. Its path, and that it can be written, are checked before validation starts. The kubelet bind-mounts the termination-message file, so aicr writes it in place rather than renaming a new file over it. The digest is written whenever the push succeeds, including a run whose checks fail, since failed runs still publish their evidence. A run that stops before the push leaves the file untouched, and an in-place write that fails partway can leave it empty or partial. So start each run with an absent or empty file and treat a complete sha256: digest in it as this run’s pushed bundle; the exit status says whether the checks passed, not whether the file was written.

After the command finishes:

./out
├── pointer.yaml # locator; copy into recipes/evidence/
└── summary-bundle/
├── recipe.yaml # canonical post-resolution recipe
├── snapshot.yaml # snapshot at validate-time (minimized by default)
├── bom.cdx.json # CycloneDX BOM (auto-generated from
│ # recipe + validator catalog when
│ # --bom is omitted)
├── ctrf/ # per-phase test results (per-test stdout/message omitted by default)
├── manifest.json # per-file sha256 inventory
├── statement.intoto.json # unsigned in-toto Statement
└── attestation.intoto.jsonl # signed (when --push is set)

The bundle is minimized by default. snapshot.yaml keeps only an allowlisted set of fields (dropping node names, provider instance IDs, the node label/taint set, OS tuning, loaded modules, and systemd config) and the CTRF reports omit per-test stdout/message. Each test’s CTRF extra keeps only allowlisted counts, measurements, thresholds, versions, allocation modes, transport, fabric, and runtime-source codes, the nodeLocal and versionSatisfied booleans, and skip-reason codes, never node names or IPs, and is omitted when none remain. The signed predicate records the applied policy in a redaction block, and the bundle self-verifies exactly like a full one. Pass --full to publish the raw payloads instead.

Commit pointer.yaml to its per-source path recipes/evidence/<recipe>/<src>/<digest>.yaml — the emit output prints the exact copyTo path, and <src> is the slug derived from your signer identity (see Artifact Verification). The bundle itself lives in OCI. Then self-verify before opening the PR — the same verifier runs against the committed pointer in the CI gate, so exit 0 locally means the gate will pass:

RECIPE="h100-eks-ubuntu-training" # replace with the copyTo recipe segment
SRC="0123456789abcdef0123456789abcdef" # replace with the copyTo signer segment
DIGEST="sha256-0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" # replace with the copyTo file name, without .yaml
aicr evidence verify "recipes/evidence/${RECIPE}/${SRC}/${DIGEST}.yaml"

Flag reference: the evidence flags (--emit-attestation, --full, --push, --no-sign, --bom, signing and registry options) are documented in the aicr validate flag table.

By default, emission fails closed with INVALID_REQUEST when any validator image is not immutably pinned (a moving tag such as :latest, or a non-AICR registry without a digest), because the attestation names validators by tag. The usual cause is a stale AICR_VALIDATOR_IMAGE_TAG; unset it. --allow-mutable-validator-tags is the explicit opt-out: it emits anyway, so use it only for disposable evidence you will not publish. For the recommended producer workflow, which pushes an unsigned bundle from the cluster and signs it in CI, see Publishing Recipe Evidence.

Registry requirements: the registry must support the OCI 1.1 Referrers API (or its tag-schema fallback) so the Sigstore Bundle can be attached to the artifact. Known-good registries: GHCR, GitLab Container Registry, Harbor (≥ 2.8), AWS ECR, Google Artifact Registry, Azure Container Registry, JFrog Artifactory. Without referrer support the bundle pushes but the signature is not discoverable, and the verifier records signature-verify as “skipped (unsigned)” even on a signed bundle.

OIDC token resolution. --push resolves an identity token through this precedence chain: --identity-token (or COSIGN_IDENTITY_TOKEN) → ambient GitHub Actions OIDC (ACTIONS_ID_TOKEN_REQUEST_URL present) → --oidc-device-flow (or AICR_OIDC_DEVICE_FLOW=true) → interactive browser. CI pipelines typically rely on the ambient GitHub Actions path; local workstations get the browser flow. With --signing-key <kms-uri> no token is resolved: the bundle is signed with the KMS key instead, for private use. See Key-signed evidence.

Local-only mode (no registry access). Omitting --push still produces a complete bundle on disk — the verifier records the signature step as “skipped (unsigned)” and the manifest-hash chain becomes self-consistency only. Useful for catching accidental corruption during development, but unsuitable for the CI gate, which requires a signed bundle bound to a pointer.

For the full producer-and-consumer walkthrough — including OCI-only verification, the tamper demo, and JSON output for CI gates — see Recipe Evidence Demo. For the bundle format and verifier semantics, see ADR-007. For the maintainer-side review checklist, see Maintaining Recipe Contributions. For the per-flag reference on aicr evidence verify, see CLI reference.

Input modes

Snapshot and recipe can come from a file, an HTTPS URL, or a Kubernetes ConfigMap:

# File (default)
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml
# HTTPS URL
aicr validate \
--recipe https://artifacts.example.com/recipes/h100-eks-inference.yaml \
--snapshot https://artifacts.example.com/snapshots/prod-cluster.yaml
# Kubernetes ConfigMap (for in-cluster operators)
aicr validate \
--recipe cm://gpu-operator/aicr-recipe \
--snapshot cm://gpu-operator/aicr-snapshot

The ConfigMap form is useful when the snapshot is captured by an in-cluster agent — see agent deployment.

Dry-run mode

--no-cluster runs the validator against the snapshot alone, skipping all Kubernetes API calls. Declarative constraints still evaluate; behavioral checks report skipped - no-cluster mode (test mode).

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --no-cluster

Useful for CI pipelines that validate a recipe against a captured snapshot without needing cluster access.

Running under a pre-provisioned ServiceAccount

By default every live run creates a ServiceAccount for its validator Jobs, binds it to cluster-admin through a per-run aicr-validator-<run-id> ClusterRoleBinding, and deletes both at cleanup. That requires an identity allowed to grant cluster-admin, which automation such as a CI runner or a scheduled in-cluster Job often should not hold. It also leaves a privileged binding behind if the process dies before cleanup, and gives admission or audit policy keyed on a username no stable name to target.

--validator-service-account-name (or spec.validate.agent.validatorServiceAccountName in --config) moves that grant to an administrator. The validator Jobs run as the named ServiceAccount, and the run creates and deletes no ServiceAccount or ClusterRoleBinding and does not apply the namespace. Before it creates anything, the run reads the ServiceAccount and asks the API server, through a SubjectAccessReview, whether it may perform any verb on any resource cluster-wide. It stops with exit 3 when the ServiceAccount does not exist, with exit 2 when the ServiceAccount sets automountServiceAccountToken: false (validator pods need its token), and with exit 4 when it is not bound to cluster-admin or the caller may not ask. It never falls back to per-run RBAC, because that fallback would create the cluster-admin binding the flag exists to avoid.

An administrator provisions the namespace, the ServiceAccount, and its binding once:

apiVersion: v1
kind: Namespace
metadata:
name: aicr-validation
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: aicr-validator
namespace: aicr-validation
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: aicr-validator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: aicr-validator
namespace: aicr-validation

The identity running aicr validate then needs the following, shown here bound to a ci-runner ServiceAccount in a ci namespace. Everything is scoped to the validation namespace except subjectaccessreviews, a read-only authorization query that persists nothing:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: aicr-validate-caller
namespace: aicr-validation
rules:
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [configmaps]
verbs: [create, get, update, patch, delete]
- apiGroups: [batch]
resources: [jobs]
verbs: [create, get, list, watch, patch, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [pods/log]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: aicr-validate-caller
namespace: aicr-validation
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: aicr-validate-caller
subjects:
- kind: ServiceAccount
name: ci-runner
namespace: ci
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: aicr-validate-caller
rules:
- apiGroups: [authorization.k8s.io]
resources: [subjectaccessreviews]
verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: aicr-validate-caller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: aicr-validate-caller
subjects:
- kind: ServiceAccount
name: ci-runner
namespace: ci

Then validate against a captured snapshot:

aicr validate \
--recipe recipe.yaml \
--snapshot snapshot.yaml \
--validator-service-account-name aicr-validator \
--output ctrf.json
  • --namespace must name the ServiceAccount’s namespace. The default is aicr-validation.
  • The flag governs only the validator Jobs. Without --snapshot, aicr validate first deploys the snapshot-capture agent, which creates or labels the namespace and creates its own run-scoped RBAC unless --service-account-name names an existing agent ServiceAccount. See Using an existing ServiceAccount. Either way the agent needs permissions beyond the Role above, so a caller that holds only that Role supplies --snapshot. A missing or unbound validator ServiceAccount is reported before the agent is deployed.
  • --no-cleanup keeps the validator Jobs and ConfigMaps. The validator Jobs created no binding to leave behind; a live-capture agent’s run-scoped RBAC is kept as well.
  • Before deploying a validator that is placed next to another component’s pods, the run lists pods in that component’s namespace (for example monitoring) to warn when the placement selector matches nothing. Without list pods there, the run logs a lookup-failure warning and continues.
  • This changes who grants the validators’ access, and when. It does not make the caller less powerful: an identity that can create Jobs in the validation namespace can run any image as that ServiceAccount, and one that can write its ConfigMaps can change what the validators read. Keep the namespace dedicated to validation, and let only the intended caller create workloads or write ConfigMaps in it.

CI/CD integration

By default aicr validate exits non-zero when any phase reports failed or other (set --fail-on-error=false to report those results and exit 0 anyway; the readiness pre-flight still exits 2 regardless, and a --cncf-submission run whose applicable MUST requirements are not all established exits 8 regardless of --fail-on-error=false, see Exit status of a submission run). CTRF JSON is emitted to stdout (or to --output <file>), so a pipeline can gate promotion on both the exit code and the structured report:

aicr validate \
--recipe recipe.yaml \
--snapshot cm://gpu-operator/aicr-snapshot \
--output ctrf.json

Exit codes follow Unix conventions and are derived from the CLI’s structured error codes (see pkg/errors/exitcode.go):

CodeMeaning
0All phases reported status passed or skipped; under --cncf-submission, additionally every applicable MUST requirement established (an all-skipped submission run exits 8), see Exit status of a submission run
2Invalid input or request (ErrCodeInvalidRequest) — bad CLI flag, malformed argument, a validator rejecting a recipe value (e.g., an inference constraint that uses the wrong comparator direction), or a readiness pre-flight constraint not met. The readiness gate always fails closed with this code, even under --fail-on-error=false
5A structured ErrCodeTimeout reached the top-level CLI — for example while loading the recipe or snapshot, waiting on the snapshot-agent Job, or signing evidence under --emit-attestation --push (which runs after the checks). A per-validator wait deadline does not reach this code: runPhase converts that error into a check result, so it surfaces as exit 8 (see note 2)
6A structured ErrCodeUnavailable reached the top-level CLI; under --cncf-submission, the pinned upstream validator could not start because the CNCF landscape was unreachable (Exit status of a submission run)
8A privileged RBAC revocation failed after operator cancellation (ErrCodeInternal; not suppressed by --fail-on-error=false), or one or more phases reported status failed (a check that returned a failure verdict — exit code 1 — or, in most cases, a validator Job killed on activeDeadlineSeconds — see the note below) or other (an indeterminate outcome — the check produced no usable verdict; a crash or OOM are the common examples, as is a validator Job that failed for a non-deadline reason with no inspectable pod), including per-check internal timeouts (e.g., DynamoGraphDeployment not ready within InferenceWorkloadReadyTimeout). Only the result-driven phase exits are suppressed by --fail-on-error=false. Under --cncf-submission also an applicable MUST requirement not established, regardless of --fail-on-error, and a submission write, redaction or upstream-validator failure (Exit status of a submission run).
9Operator interrupted the CLI with Ctrl-C or SIGTERM (ErrCodeCanceled) before the last validation phase completed, and privileged cleanup did not fail. Not suppressed by --fail-on-error=false; completed checks and the partial current phase are retained, and CTRF output is attempted once at least one phase has results. A signal that arrives after the last phase completed takes the normal completion path instead (see Interrupted runs below)

Important: two subtleties to be aware of when gating a pipeline on exit code:

  1. Both failed and other are blocking. A phase whose status is other — the check produced no usable verdict, e.g. a crash, an OOM, or a Job that failed for a non-deadline reason with its pod already gone — drives the same non-zero exit as failed — an inconclusive check fails closed rather than passing silently. --fail-on-error=false suppresses both result-driven exits (the run reports the outcomes in the CTRF report and exits 0); it does not distinguish failed from other. A validator Job killed on activeDeadlineSeconds reports failed rather than other: its terminal Failed/DeadlineExceeded condition is a verdict, so the result names the deadline instead of the missing pod. The Job deadline is the catalog timeout plus ValidatorJobDeadlineHeadroom (3m: 2m for container startup, 30s for the check’s deferred cleanup, 30s to flush its result), and the CLI waits for the Job to become terminal for that deadline plus the pod’s 60s SIGTERM window and a 5s margin (catalog timeout + 4m05s), measured from the Job’s observed start time (status.startTime when the apiserver has stamped it, otherwise creationTimestamp). The CLI therefore never gives up on a run while the pod is still finishing its cleanup. When the Job controller deletes the pod at the deadline, the CLI reports what the check itself reported: the exit code and termination message of the pod in its deletion grace window, or of the last terminated pod it observed while waiting, is appended to the deadline message (check reported: exit code 3: terminated by SIGTERM …). The verdict stays failed; if no terminated status was observed the generic deadline message stands.
  2. Exit 5 is narrower than it sounds. A timeout inside a check’s own logic (DynamoGraphDeployment not ready, inference endpoint never healthy, AIPerf Job pod-wait deadline) surfaces as a failed phase, not as a structured ErrCodeTimeout, so the CLI exits 8. So does a per-validator wait deadline: runPhase hands that error to the timeout handler and records the outcome as a check result rather than propagating it, which is the same path the wait-exhaustion exception in note 1 takes. The rule, rather than a list of sources: exit 5 applies whenever an ErrCodeTimeout reaches the top-level CLI. A timeout that is converted into a check result instead — whether raised inside a check or by a per-validator wait — surfaces as exit 8.

Interrupted runs. The first Ctrl-C or SIGTERM cancels validation and, with cleanup enabled, deletes the in-flight validator Job using a fresh context. Completed verdicts and the partial current phase are retained; unstarted checks in that phase say not run: validation canceled and later phases do not start. The interrupted phase is blocking (other unless already failed), and so is its CTRF report. When no check entry already records the interruption, for example because the signal landed between checks or between phases, the report gains a validation-interrupted entry with status other, so summary.other is non-zero. CTRF output is still attempted once at least one phase has results, even if the cluster’s report ConfigMap cannot be written. A signal that arrives before any phase has results, for example during cluster preparation, writes no CTRF output. Partial output is written only for an interruption. Any other error (for example a later phase’s pre-flight failure, or a privileged RBAC revocation failure after every phase completed) writes no CTRF output, as before. Cleanup typically takes about 65s, and about 125s plus overhead in the worst case. One shared 30s window covers result capture, Job deletion and report persistence. The wait for the validator pod to terminate then covers the pod’s full 60s SIGTERM grace from Job deletion (plus a 5s margin), so the validator’s RBAC stays in place while a cooperative check cleans up. The existing 30s per-run RBAC/data teardown window follows. The CLI records this 125s aggregate deadline when cancellation is first observed, not when validation starts. Partial output uses only the time remaining before that deadline (and at most its existing 30s I/O cap), never a further cleanup window. If no time remains, the in-memory results and original error are retained without starting output. These limits are cooperative: an arbitrary file/stdout writer is not physically interruptible just because it receives a context. A second Ctrl-C or SIGTERM force-terminates the CLI and can leave resources behind. --no-cleanup continues to leave them intentionally.

Exit 9 applies only when the signal arrives before the last phase completed. A signal that arrives after that, for example during the final per-run RBAC/data teardown or while evidence is rendered, signed or pushed, takes the normal completion path on a canceled context. The run may then exit 0, 8 or another code, depending on what that work returns.

If cancellation coincides with a privileged RBAC revocation failure, the returned error is instead ErrCodeInternal (CLI exit 8), explicitly warning that the cluster-admin binding may be orphaned. Both cancellation and cleanup causes are preserved; check and revoke the potentially orphaned binding manually. Partial CTRF output is still attempted within the same remaining budget. Neither ordinary cancellation nor this combined failure renders, signs or pushes conformance/recipe evidence.

When the validator pod itself receives SIGTERM (deadline, eviction or drain), a cooperative check runs deferred cleanup, writes a SIGTERM interruption message and exits 3, mapped to CTRF other rather than a completed failure verdict. This container exit is distinct from the CLI’s exit 9 on operator cancellation. It remains blocking (normally CLI exit 8), including when the check returns nil after cancellation. The pod’s 60s grace (30s for cleanup, 30s to write the termination message) is best effort: blocking checks, long cleanup, SIGKILL, OOM or a crash may prevent a termination message, and a missing pod with a terminal Job deadline condition still uses the Job-level failed classification described above, carrying the check’s own message when one was observed. No external orphan-resource sweeper is provided yet; namespaces a check leaves behind after SIGKILL or a crash carry the aicr.run/gpu-test-run label.

Scripts that gate on validation outcome should treat any non-zero code as failure rather than branching on specific values, and should additionally check CTRF summary.failed and summary.other for a complete picture.

For informational-only runs (report results without failing the build):

aicr validate ... --fail-on-error=false

--fail-on-error=false scopes to phase check results: a deployment, performance, or conformance phase that reports failed or other is recorded in the CTRF report and the run exits 0. It does not relax the readiness pre-flight — if a readiness constraint (K8s version, OS, kernel) is not met, the run still fails closed with exit 2 before any phase executes. In --no-cluster mode this is the dominant path, since all phase checks report skipped and readiness is the only inline constraint evaluation.

Troubleshooting

Readiness pre-flight fails

The CLI logs each readiness constraint comparison before any phase runs:

readiness constraint failed: name=K8s.server.version expected=">= 1.34" actual=v1.33.0-eks-abc

The run stops before deploying any Jobs and exits 2 (INVALID_REQUEST). This is intentional fail-closed behavior: readiness gates whether the validators can run at all, so it is terminal regardless of --fail-on-error (see above).

Fix: upgrade the cluster, or pick a recipe whose readiness constraints match the cluster’s actual versions.

Non-standard GPU labels or taints

Default GPU-node discovery looks for nodeGroup, node.kubernetes.io/instance-type, or GPU-related label substrings. If your cluster uses custom labels, override the scheduling of inner workloads with --node-selector and --toleration:

aicr validate \
--recipe recipe.yaml --snapshot snapshot.yaml --phase performance \
--node-selector my-org/gpu-pool=h100 \
--toleration dedicated=worker-workload:NoSchedule \
--toleration dedicated=worker-workload:NoExecute

These flags affect the inner benchmark pods that run on GPU nodes (NCCL workers, Dynamo workers). The NCCL mpirun launcher follows the workers’ final node selector and tolerations, so it runs on a node that already holds the benchmark image; a recipe-supplied nccl-benchmark-runtime keeps its own launcher placement. The example above supplies --snapshot, so it does not launch the live snapshot agent. When --snapshot is omitted, the flags also configure that preliminary agent. With no toleration override, the agent tolerates all taints; an explicit spec.validate.agent.tolerations: [] clears that default. Neither flag affects the validator orchestrator Job itself.

For inference-perf specifically, --node-selector narrows the pool of candidate GPU nodes — the validator then picks the candidate with the most free GPUs (subtracting same-ledger occupancy only — DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity — and skipping DRA candidates that carry scalar nvidia.com/gpu workloads) and pins all Dynamo Frontend + worker pods to that node via kubernetes.io/hostname. The AIPerf benchmark runner pod is CPU-only, uses a tolerate-all / no-nodeSelector pod spec, and is unaffected by these flags.

A check reports skipped unexpectedly

Skips are always deliberate and always carry a reason, but the location of the reason in the CTRF entry depends on how the skip happened:

  • Check-level skips (the CheckFunc ran and returned validators.Skip(reason) — e.g., Guards A/B/C on inference, --no-cluster from inside a check): reason appears in stdout as level=INFO msg=SKIP reason="…".
  • Phase-level skips (the CheckFunc never ran — e.g., with --fail-fast, a prior phase failed so subsequent phases synthesize skip entries; also --no-cluster for checks that the runner marks skipped before dispatch): reason appears in message, not stdout.

Common reasons and their cause:

Reason (excerpt)Where it appearsMeaningFix
no inference-throughput or inference-ttft-p99 constraint in recipestdoutCheck was invoked but recipe is missing the matching constraintsRe-generate the recipe or add the constraints
dynamo-platform not in recipe componentsstdoutInference check selected but dynamo-platform absent from componentRefsUse --platform dynamo when generating the recipe
DynamoGraphDeployment CRD not installedstdoutRecipe declares dynamo-platform but the operator is not deployedRun aicr bundle + ./deploy.sh first, or wait for bootstrap to complete
requires Service + Accelerator to be implementedstdoutThe recipe’s criteria are outside the NCCL benchmark’s default applicability (e.g. a service registered via --data)Add an nccl-benchmark-profile constraint — see Opting external recipes into a benchmark profile — or, if no embedded template fits, an nccl-benchmark-runtime-ref constraint — see Supplying a benchmark runtime for a private service
benchmark profile … does not implement the … NCCL variantstdoutThe declared nccl-benchmark-profile is valid but has no template for this check variantDrop the non-applicable check from validation.performance.checks, or pick a profile that implements it
failed to resolve nccl-benchmark-runtime-ref=… expected … in the --data treestderrThe referenced runtime file is missing from --data (or --data wasn’t passed)Place the TrainingRuntime at validators/performance/testdata/{accelerator}/{service}/runtime.yaml in your --data dir — see Supplying a benchmark runtime for a private service
nccl-benchmark-runtime … must be a … TrainingRuntime … must declare a "node" replicatedJobstdoutThe referenced file is not a Kubeflow trainer.kubeflow.org/v1alpha1 TrainingRuntime, or lacks the node replicatedJobFix the runtime file to be a valid TrainingRuntime with a node replicatedJob
nccl-benchmark-runtime and nccl-benchmark-profile are mutually exclusivestdoutThe recipe declares both escape hatches at onceKeep only one: borrow an embedded profile or supply your own runtime
skipped - no-cluster modemessage--no-cluster was passed — the runner short-circuits every phase before dispatching any JobRemove the flag to run behavioral checks
named in skipChecksmessage, and extra.skipReason as named-in-skip-checks (the channel that survives bundle redaction)The caller withheld this check with --skip-check or spec.validate.execution.skipChecksDrop the name from the skip list to run it again; see Skipping checks a run cannot satisfy
skipped due to previous phase failuremessage--fail-fast was set and an earlier phase failed, so subsequent phases were skippedFix the earlier phase first, or drop --fail-fast to run all phases regardless

driver-runtime-management fails

The check (CNCF driver_runtime_management, SHOULD at v1.37 and MUST at v1.38) runs a pod from a vanilla ubuntu:22.04 image that requests one GPU, and reads the driver from inside it. The probe pod carries the --image-pull-secret Secrets, copied into its per-run namespace (see the image note below). Its artifacts (Driver runtime probe pod logs, Advertised driver version, Driver runtime compatibility) name the failing step:

Verdict excerptMeaningFix
did not inject the accelerator driver toolsNo nvidia-smi appeared inside the vanilla container: the container runtime configuration (CDI or the nvidia runtime hook) is not injecting the driver stack for GPU-requesting podsCheck the GPU Operator toolkit DaemonSet and the node’s nvidia-container-runtime configuration; on AKS, the nvidia-setup tuning that writes it
failed to initialize or report a numeric versionnvidia-smi is present but NVML failed, typically Driver/library version mismatch after a driver upgrade without a node reboot or a user-space library that does not match the kernel moduleReconcile the driver on the node (driver DaemonSet restart or node reboot)
advertises driver version ... does not matchThe node label (nvidia.com/cuda.driver-version.full) or DRA driverVersion attribute says one version and the container sees anotherThe advertiser is stale (GPU Feature Discovery or the DRA kubelet plugin has not re-published since the driver changed); restart it or wait for the next publish
saw 0 accelerator device(s)The pod was granted a GPU but no /dev/nvidia[0-9]* appeared insideDevice-plugin or CDI device injection is broken on that node
WARNING: node ... advertises no accelerator driver version (artifact only)The platform advertises no version, so only the in-container probe decides; the check still passes when the probe succeedsExpected on a platform whose GPU stack publishes no GPU Feature Discovery label and no DRA driverVersion attribute

The probe image (docker.io/library/ubuntu:22.04) pulls from Docker Hub by default and, like the busybox probe image and the NCCL workload image (but unlike the Dynamo runtime images inference-perf pulls), is not rewritten by AICR_VALIDATOR_IMAGE_REGISTRY. In a mirrored environment set AICR_DRIVER_RUNTIME_IMAGE=<image ref> in the aicr validate environment to the full reference of your copy or of another vanilla OS image. It must ship no accelerator driver or tools, and it must provide what the probe script runs on: a POSIX /bin/sh, head and tr (timeout is used when present). A distroless image cannot run the probe; mirroring the Ubuntu image is the simplest choice. When the mirror needs credentials, --image-pull-secret covers the probe too: every conformance check that creates a per-run namespace copies the named Secrets from the validator namespace into it and attaches them to its probe pods, and the copies are deleted with the namespace. A Secret the flag names but the validator namespace lacks fails the check with NotFound before any probe is created.

A probe pod whose RuntimeClass names a handler the node’s container runtime does not serve stays Pending with ContainerCreating and no status change; the checks read the kubelet’s FailedCreatePodSandBox event (the containerd message no runtime for "..." is configured naming the handler, a missing runtime handler, or a shim binary not found) and fail within seconds with stuck: FailedCreatePodSandBox: ... instead of waiting out the pod timeout. A sandbox failure from network setup is transient and is not treated this way.

ai-service-metrics or pod-autoscaling cannot reach its metrics endpoint

On EKS clusters that split worker and system pods across separate security groups (e.g. DGXC EKS with distinct customer/system ENI subnets), the conformance check ai-service-metrics, which queries NKTP’s otel-lgtm, can fail non-deterministically with:

[SERVICE_UNAVAILABLE] metrics backend unreachable at http://lgtm.nktp.svc:9090 — verify network connectivity

pod-autoscaling, which dials Prometheus for its PodMonitor load wait when the recipe declares kube-prometheus-stack, fails the same way at its preflight:

[SERVICE_UNAVAILABLE] Prometheus at http://kube-prometheus-prometheus.monitoring.svc:9090 did not list its scrape pools (GET /api/v1/scrape_pools needs Prometheus >= v2.42 and network connectivity between the validator pod and the Prometheus service)

Both checks’ orchestrator Jobs carry the preferred dependencyAffinity described below.

The validator orchestrator Job tolerates every taint and sets a preferred dependencyAffinity toward the pod it dials (otel-lgtm for ai-service-metrics, Prometheus for pod-autoscaling), so the scheduler co-locates the two when possible. The preference is best-effort, so on fallback it can still land on any worker node — including one whose ENI is in a security group whose ingress to that pod’s SG is missing or asymmetric. On such a fallback the outcome is not stable across re-runs: image-locality scoring tends to keep the pod on whatever node won the first scheduling decision, so a passing run on a fresh cluster does not prove the SG topology is correct.

This is a cluster-side prerequisite, not an AICR bug per se — see EKS Dynamo Networking Prerequisites for the SG ingress rules required for tcp/9090, which both Prometheus and otel-lgtm serve. The preferred dependencyAffinity (#933, resolved) makes a bad placement far less likely, but the 9090 SG rule remains the reliable guarantee since the affinity is best-effort.

Workaround when SG changes are not available: re-run the check until the orchestrator lands on a node whose SG can reach that pod, then leave the image cached there so image-locality keeps subsequent runs on the same node. This is unreliable and should not be used as the steady-state validation strategy.

workload-sandboxing fails

The check (CNCF workload_sandboxing, SHOULD at v1.37 and MUST at v1.38) looks for a sandboxed RuntimeClass, runs a CPU-only ubuntu:22.04 probe pod on it and an unsandboxed control pod on the same node, and compares what the two observe. No embedded recipe declares it yet: none ships a sandboxed RuntimeClass, so until the #3615 runtime decision provides one and the recipes declare the check, the workload_sandboxing evidence row renders NOT TESTED (not run) at SHOULD. A cluster that already has gVisor or Kata can run it today by adding workload-sandboxing to spec.validation.conformance.checks in a recipe copy (and naming the class with AICR_SANDBOX_RUNTIME_CLASS when detection does not match). Its artifacts (Sandboxed RuntimeClass, Sandboxed probe pod logs, Unsandboxed control pod logs, Workload sandboxing verdict) name the failing step:

Verdict excerptMeaningFix
unmet prerequisite: no sandboxed RuntimeClassAuto-detection found no RuntimeClass whose name or handler contains gvisor, runsc, kata (the AKS kata-mshv-vm-isolation class matches), sandboxed, quark or krun. A sandboxed class may still exist under another name; the check never skips or records N/A for thisProvide a sandbox runtime (which one AICR recipes ship, and on which pool, is the pending decision in #3615), or set AICR_SANDBOX_RUNTIME_CLASS=<name> when the cluster has one under another name
AICR_SANDBOX_RUNTIME_CLASS names RuntimeClass "...", which does not exist on this clusterThe override names a class the cluster does not haveFix the name (kubectl get runtimeclass) or unset the variable to auto-detect
kernel identity: the sandboxed pod observes the same kernel identityEvery kernel identity field both pods reported (release, version, boot id) matched, or neither pod reported a comparable field. With matching fields the pods share the host kernel: the class’s handler maps to a runc-like runtimeRead the identity values in both probe log artifacts. If no field was comparable, find out why the probes could not read them; otherwise check that the containerd runtime configuration maps the handler to the sandbox binary
sandboxed probe pod stuck: FailedCreatePodSandBox: ... no runtime for "<handler>" is configured (or a missing runtime handler or shim binary)The pod was bound to a node whose container runtime does not serve the class’s handler; the check reads the kubelet’s FailedCreatePodSandBox event and fails within seconds instead of waiting out the pod timeoutCheck that the RuntimeClass scheduling.nodeSelector selects only nodes with the sandbox runtime installed, and that the handler name matches the runtime’s configuration on those nodes
sandboxed probe pod did not complete in time with status=ContainerCreatingThe pod was bound to a node but its sandbox never started for a reason the fast failure above does not recognise, or the node is cordoned or NotReadyRead the pod events in the per-run aicr-gpu-test-* namespace while the check runs
process isolation:, kernel and memory isolation:, filesystem isolation: or network isolation:The named probe saw host state inside the sandbox (a host daemon in /proc, a readable /dev/mem, a host path such as /var/lib/kubelet, a bridge such as cni0); the probe logs artifact shows what it sawThe sandbox shares that host namespace or mount; correct the runtime configuration
sandboxed probe pod stuck: or unsandboxed control pod stuck:The pod could not pull its image or be scheduled (Unschedulable); for the sandboxed pod, usually a RuntimeClass scheduling.nodeSelector that no node satisfiesInspect the pod events in the per-run aicr-gpu-test-* namespace while the check runs; for mirrors see the image note below

The probe image (docker.io/library/ubuntu:22.04) pulls from Docker Hub by default and, like the busybox probe image and the NCCL workload image, is not rewritten by AICR_VALIDATOR_IMAGE_REGISTRY. In a mirrored environment set AICR_SANDBOX_PROBE_IMAGE=<image ref> in the aicr validate environment to the full reference of your copy or of another vanilla OS image. It must provide what the probe script runs on: a POSIX /bin/sh, uname, cat, ls, wc, head, awk and tr. A distroless image cannot run the probe; mirroring the Ubuntu image is the simplest choice. The two probe pods carry the --image-pull-secret Secrets, copied into the per-run namespace as for the other probe pods (see the driver-runtime-management note above), so a mirror that needs credentials is served the same way as the validator image.

pod-autoscaling fails with “Prometheus did not load the PodMonitor”

When the recipe declares kube-prometheus-stack, the check creates its PodMonitor in the Prometheus namespace, because that Prometheus loads monitors only from the namespaces the bundle names and its own. Before starting its metric budget it waits up to five minutes for Prometheus to list the PodMonitor among its scrape pools (GET /api/v1/scrape_pools). That wait exists because prometheus-operator reconciles monitors on its own schedule: gaps of about three minutes between a monitor’s creation and Prometheus loading it were observed on a healthy AKS cluster. Without kube-prometheus-stack (the OpenShift overlay serves prometheus-adapter-ocp from the platform monitoring stack) there is no Prometheus to ask: the PodMonitor stays in the test namespace, the check records the wait as skipped, and its baseline is bounded by the metric timeout alone.

The baseline then ends in one of three ways, each with its own message.

A failure naming the load, such as:

[NOT_FOUND] HPA autoscaling test failed in phase baseline: [NOT_FOUND] Prometheus did not load the PodMonitor monitoring/hpa-workload-<id> within 5m0s: no scrape pool podMonitor/monitoring/hpa-workload-<id>/* among the 12 configured ...

means Prometheus did not load the monitor within the wait. Two things have to happen for it to: prometheus-operator must write the monitor into the generated configuration, and Prometheus must reload that configuration. Check both. For the first, look at the operator’s logs for the reconcile of that namespace, and confirm the Prometheus resource’s podMonitorNamespaceSelector names its own namespace. For the second, a rejected reload leaves Prometheus running on its previous configuration with the pool absent even though the operator did its part: check prometheus_config_last_reload_successful (0 after a failed reload) and the Prometheus and config-reloader container logs for the reload error.

A failure reading “could not determine whether Prometheus loaded the PodMonitor … the last completed scrape pool listing failed” (or ”… no scrape pool listing completed”) means the scrape pool API stopped answering after the preflight, so no verdict on the monitor was reached; the underlying listing error (unreachable, HTTP status, malformed response) is attached. A listing the wait’s deadline interrupts does not change the verdict: “did not load” rests on the last listing that completed. Check the Prometheus pod and the network path from the validator pod to it, as in the section above.

A failure naming the collection instead (“Prometheus loaded the PodMonitor, but the custom metrics pipeline does not serve the workload metric”) means the configuration was loaded and the scrape or the adapter query did not produce a sample: check the target under the active or dropped targets at /api/v1/targets (a target relabeling dropped is listed as dropped, not active), that the Prometheus series filter keeps ai_conformance_hpa_queue_depth, and the prometheus-adapter rule.

ai-service-metrics fails with “did not report the stub’s target as up”

The check queries the backend NKTP exports to, not the recipe’s Prometheus. It creates a ServiceMonitor in its test namespace and waits up to five minutes for that backend to hold the stub target’s up series with value 1.

  • No up series: NKTP did not discover the ServiceMonitor, or did not export the sample. NKTP needs no monitor label, so check the ServiceMonitor’s selector and port, the Target Allocators’ logs, and the collectors’ logs for failed scrapes or exports.
  • up is 0: the target is discovered but its scrape fails. Check the network path from NKTP’s collectors to the stub pod.

A failure reading “discovered the ServiceMonitor but the workload metrics were not collected” means up arrived but the stub’s own series did not within the collection timeout: check the collectors’ logs as above.

The check is skipped, and the CNCF requirement reported as not tested, when the recipe has no nktp or NKTP runs without otel-lgtm. To run it against your own backend instead, see ai-service-metrics on your own metrics backend.

Benchmark Job stuck or timed out

Each performance check has a Job-level activeDeadlineSeconds set by the catalog’s timeout: plus ValidatorJobDeadlineHeadroom (3m) on top. For inference-perf, the full pipeline (model-cache populate → workload ready → endpoint health → benchmark) can take up to ~50 min on cold-start clusters; the catalog budgets a 65 min timeout. If it still times out:

# validator orchestrator Job + AIPerf benchmark Job both live in aicr-validation.
# The orchestrator is named aicr-inference-perf-<hex> (random suffix per run);
# the AIPerf Job is named aicr-aiperf-<run-id-hash>.
kubectl -n aicr-validation get jobs | grep -E 'aicr-inference-perf-|aicr-aiperf-'
# tail each by full job name (label selectors require exact match)
ORCHESTRATOR_JOB="aicr-inference-perf-5d1e8a42" # replace with the aicr-inference-perf- Job listed above
AIPERF_JOB="aicr-aiperf-3f9a1c2b" # replace with the aicr-aiperf- Job listed above
kubectl -n aicr-validation logs -l "job-name=${ORCHESTRATOR_JOB}" --tail=200
kubectl -n aicr-validation logs -l "job-name=${AIPERF_JOB}" --tail=200
# the Dynamo workload (DynamoGraphDeployment, Frontend, worker pods)
# lives in a separate per-run namespace:
kubectl get ns | grep aicr-inference-perf-
WORKLOAD_NAMESPACE="aicr-inference-perf-3f9a1c2b" # replace with the namespace listed above
# (resourceclaimtemplates is populated in DRA wiring mode; empty in
# device-plugin mode — both are normal)
kubectl -n "${WORKLOAD_NAMESPACE}" get dynamographdeployments,pods,svc,resourceclaimtemplates

Common causes: image pull throttling, vLLM model load slowness, and every eligible candidate GPU node being saturated by existing workloads. In the saturated case the validator fails fast with a message like no eligible candidate GPU node has free GPUs (N matched; eligible candidates are saturated by existing workloads in the selected mechanism's ledger ...). Occupancy is subtracted per allocation ledger (DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity), and nodes excluded for other reasons — NotReady, probe-ineligible, kai-blocked raw slices, or DRA-capable but carrying scalar nvidia.com/gpu workloads — are not counted as saturated; the fix is to free GPUs on one of the candidate nodes, or to pass --node-selector kubernetes.io/hostname=<node> to target a specific node you know is free. GPU-occupancy accounting fails closed rather than treating GPUs as free: a pod list failure always fails the check, and so does any error listing DRA resourceclaims (e.g. RBAC denied, timeout) — with one exception: when the resource.k8s.io API is not served at all (NotFound), which deterministically means zero DRA usage, the validator proceeds with device-plugin pod requests alone.