> This page is for AI Cluster Runtime, version v0.21.1.
> For other versions, use one of these documentation indexes:
> - Latest · v1.0.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt
> - v1.0.0: https://docs.nvidia.com/aicr/v1.0.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/aicr/v0.22.0/llms.txt
> - v0.21.1: https://docs.nvidia.com/aicr/v0.21.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server.

# Validating a Cluster

Task-oriented walkthrough for running `aicr validate` against a GPU cluster — from
capturing a snapshot through interpreting results. Covers both training and
inference workloads and all three validation phases (deployment, conformance,
performance).

For per-flag reference, see [CLI reference: aicr validate](/aicr/v0.21.1/user-guide/cli-reference#aicr-validate).
For the architectural view of how snapshot + recipe flow into the validator, see
[Data flow: Stage 3 Validate](/aicr/v0.21.1/integrator-guide/data-flow#stage-3-validate-constraint-checking).

## When to validate

| Phase | What it answers | Typical trigger |
|-------|-----------------|-----------------|
| `deployment` | Are the components the recipe asks for actually installed and healthy? | After `./deploy.sh` finishes, before running any workload |
| `conformance` | Does the cluster support workload-specific capabilities (DRA, gang scheduling, autoscaling, ...)? | Before opening the cluster to real workloads |
| `performance` | Does the cluster hit expected bandwidth / throughput thresholds? | After components are ready; before going to production |

Readiness pre-flight constraints (K8s version, OS, kernel) run implicitly before
any phase. If pre-flight fails, no validator Jobs are deployed.

## The workflow

```text
  aicr snapshot ─┐
                 ├─▶ aicr validate ─▶ CTRF report
  aicr recipe ───┘                    (passed / failed / skipped per check)
```

1. **Snapshot** — capture current cluster state (K8s / OS / GPU / topology) once.
2. **Recipe** — generate the target configuration for your workload (training vs inference, platform, accelerator).
3. **Validate** — run one or all phases against the snapshot and live cluster.

## Prerequisites

- `aicr` CLI installed (see [installation](/aicr/v0.21.1/user-guide/installation)).
- `kubectl` configured for the target cluster (validator dispatches K8s Jobs; pre-flight only needs the snapshot).
- Cluster service account with RBAC to create Jobs, ConfigMaps, and read cluster state (AICR creates its own `aicr-validation` namespace on first run).
- **AKS profiled recipes**: the readiness pre-flight re-evaluates the recipe's profile constraint (`K8s.aks-gpu-pools.gpu-driver`), so the snapshot must carry that reading — capture it with `aicr snapshot --aks-gpu-pools <az dump>`, or pass the same flag to `aicr validate` when it captures live. A snapshot without the reading fails readiness closed (exit 2).
- **GKE recipes**: the readiness pre-flight re-evaluates the recipe's `gpuStack` profile constraint over the GPU-node set (nodes carrying `cloud.google.com/gke-accelerator`): the default `gke-default` value requires that **no** GPU node carries the opt-out label `gke-no-default-nvidia-gpu-device-plugin` (GKE's managed plugin stays the `nvidia.com/gpu` advertiser), while `bundle-installer` requires every GPU node to carry `gke-no-default-nvidia-gpu-device-plugin=true` (so the GPU Operator's plugin is the sole advertiser). The check fails closed (exit 2) on labels contradicting the selected value, mixed labels, malformed or ambiguous label readings, a snapshot with no identifiable GPU nodes, and when `--max-nodes-per-entry` actually truncated a participating label reading (a truncated node list cannot prove set membership — regenerate without the flag; a cap larger than the node count truncates nothing and validates normally). No provider projection flag is needed on GKE — the constraint reads node labels the standard snapshot already carries. See [GKE GPU Setup](/aicr/v0.21.1/integrator-guide/gke-gpu-setup#gpu-device-plugin-ownership) for the full setup and the qualification matrix.

## Training performance validation

Training performance runs an NCCL all-reduce benchmark — a Kubeflow `TrainJob`
that runs `all_reduce_perf` across GPU nodes and measures aggregate bus
bandwidth. Three check variants are available; the recipe picks the one (or
ones) that match the target fabric:

| Check | Transport | Default applicability (from recipe criteria) |
|---|---|---|
| `nccl-all-reduce-bw` | Auto-detect (whatever NCCL picks) | H100/H200 on EKS, H100 on GKE, H100 on AKS (ND-series InfiniBand — NCCL's built-in IB/verbs transport over the `rdma/hca_shared_devices_a` shared device pool), and B200/GB200 on self-managed clusters (`service=any`). Preserves the pre-variant behavior. |
| `nccl-all-reduce-bw-net` | NET (EFA on EKS by default; ConnectX RoCE via `AICR_NCCL_FABRIC=roce`; built-in IB/verbs on OKE) | GB200 + EKS, and GB200 + OKE. Asserts the intended NET fabric actually carried traffic — EFA on EKS, the NVL72 InfiniBand east-west fabric (`nvidia.com/mlnxnics` shared HCAs) on OKE — catching silent fallback to Socket when the NVIDIA driver is missing `NVreg_GrdmaPciTopoCheckOverride=1`. |
| `nccl-all-reduce-bw-nvls` | NVLS (MNNVL across an NVL72 IMEX domain) | GB200 (EKS, OKE); GB300 (generic); VR200 (RKE2). Asserts the NVLS communicator actually initialized — catches silent fallback to the NET fabric when the IMEX domain is misconfigured. |

The applicability column is the *default*, derived from the recipe's
`criteria`. A recipe whose criteria fall outside it can still run these
benchmarks explicitly — either by
[borrowing an embedded benchmark profile](#opting-external-recipes-into-a-benchmark-profile)
whose fabric matches its hardware, or, for a private service whose fabric
matches no embedded template, by
[supplying its own benchmark runtime](#supplying-a-benchmark-runtime-for-a-private-service).

On EKS, the `-net` check defaults to the AWS EFA fabric. On a ConnectX **RoCE** cluster
(e.g. DGXC GB300 `p6e-gb300r`), set `AICR_NCCL_FABRIC=roce` in the `aicr
validate` environment to run the NET test over NCCL's built-in IB/verbs
transport across `roce.networking.k8s.aws` DRA devices instead. The value is
scoped to the `-net` check only; unset (or `efa`) leaves every existing recipe
on the EFA path unchanged, and any other value is rejected. The RoCE runtime
image installs `openssh-server` at startup, so the GPU nodes need apt egress;
on an air-gapped cluster the RoCE NET test cannot bootstrap. This env override is
interim — snapshot-based fabric auto-detection (and removing the runtime
package install once a CUDA-13 image ships sshd) is tracked in
[NVIDIA/aicr#1413](https://github.com/NVIDIA/aicr/issues/1413).

GB200/EKS recipes (both `training` and `inference` intents) enable `-net` and
`-nvls` together rather than the auto-detect variant, because those nodes
expose two inter-node fabrics simultaneously and a single auto-detect test
would only exercise one of them.

GB200/OKE training recipes follow the same pattern and enable `-net` and
`-nvls` together, with the same `>= 40` / `>= 500` GB/s floors as GB200/EKS.
On OKE, `-net` exercises the NVL72 rack's InfiniBand east-west fabric
(`rdma0-3`, advertised as `nvidia.com/mlnxnics` by the recipe's
`rdmaSharedDevicePlugin` NicClusterPolicy) over NCCL's built-in IB/verbs
transport — no EFA or RoCE plumbing is involved. Both variants were validated
end to end on a `BM.GPU.GB200.4` NVL72 rack.

```bash
# Capture snapshot, generate training recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml

aicr recipe --service eks --accelerator h100 --os ubuntu \
            --intent training --platform kubeflow \
            --output recipe.yaml

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance
```

The generated recipe lists the selected variant(s) under
`validation.performance.checks` with a platform-tuned bandwidth constraint
(example: `>= 300 GB/s` for H100 + EFA; `>= 40 GB/s` NET and `>= 500 GB/s`
NVLS for GB200 + EFA, each sized for a 2-node pair).

**Node-shape assumption.** These bus-bandwidth floors are fixed absolute
values calibrated on full, high-bandwidth nodes (8-GPU H100 NVLink/SXM with
multi-NIC transport). They are *not* normalized for node fabric or GPU count,
so a smaller or different-fabric H100 SKU (e.g. a single-GPU-per-node shape)
can false-fail a healthy run. Making the NCCL gate fabric/transport-class
aware is tracked in [#1256](https://github.com/NVIDIA/aicr/issues/1256).

Expected flow (~5–10 min per variant): readiness pre-flight → deploy
`TrainingRuntime` + `TrainJob` in a per-run namespace named
`aicr-nccl-perf-<variant>-<run-id>` → worker pods reach `Running` → run
`all_reduce_perf` → parse peak bus bandwidth → verify the intended transport
actually carried traffic (for `-net` / `-nvls`) → compare to recipe
constraint (10 % tolerance) → cleanup.

Each variant (`-net`, `-nvls`, default) gets its own namespace, deleted on
exit along with its `TrainingRuntime`/`TrainJob`. A namespace left behind by
an interrupted run is reclaimed by a same-run-ID retry only if no
`Pending`/`Running`/`Unknown` pod remains in it and its execution lock (a
`Lease` named `aicr-nccl-run-lock` in that namespace) has gone stale, 20
minutes past its last renewal. Otherwise the retry fails with a conflict
rather than risk two executions sharing one namespace; check
`kubectl get pods -n <namespace>` and the Lease's age to find and clear
whichever is actually stuck. A standalone run (no run ID) is never retried
into its old namespace; that namespace is only pruned automatically, on a
later run once an hour has passed and both conditions above hold.

A passing CTRF entry:

```json
{
  "name": "nccl-all-reduce-bw-net",
  "status": "passed",
  "suite": ["performance"],
  "stdout": [
    "NCCL All Reduce bandwidth (nccl-all-reduce-bw-net): <actual> GB/s",
    "Constraint: >= <threshold> → true"
  ]
}
```

> **Note:** this guide does not yet list per-platform expected-bandwidth
> baselines (EKS + EFA, GKE + TCPXO, AKS, etc.). The recipe's constraint
> value is the current pass/fail floor; measured values above that floor
> are treated as passing regardless of platform.

To run deployment validation first (recommended — verifies GPU Operator, DRA
driver, and Kubeflow Trainer are installed and healthy before the benchmark):

```bash
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase deployment
```

### Opting external recipes into a benchmark profile

The default applicability above is keyed to service + accelerator pairs the
validator ships templates for. A recipe whose criteria fall outside that set —
typically a `criteria.service` registered only through an external
[`--data` directory](/aicr/v0.21.1/integrator-guide/data-extension), or an embedded service
extended to a new accelerator — would otherwise report the NCCL checks as
skipped even when its `validation` block declares them. Such a recipe opts
into an embedded benchmark with the `nccl-benchmark-profile` performance
constraint (a bare `{accelerator}/{service}` value, no comparator):

```yaml
validation:
  performance:
    checks:
      - nccl-all-reduce-bw-net
      - nccl-all-reduce-bw-nvls
    constraints:
      - name: nccl-benchmark-profile   # run the GB200-on-EKS benchmarks
        value: gb200/eks
      - name: nccl-all-reduce-bw-net
        value: ">= 40"
      - name: nccl-all-reduce-bw-nvls
        value: ">= 500"
```

The profile selects the benchmark's template and environment handling —
transport-class runtime template, EFA/TCPXO discovery, worker scheduling
defaults, and preflight checks — as if the cluster were the named pair, so
pick the profile whose fabric matches the hardware. GPU nodes are still
matched by the recipe's own `criteria.accelerator` (via the GFD
`nvidia.com/gpu.product` label) when a matcher exists, but the profile's
service keys the rest of node handling: an `…/eks` profile narrows workers
to a single `node.kubernetes.io/instance-type`, and the `…/any` profiles
(`b200/any`, `gb200/any`) require an explicit `--node-selector` identifying
the GPU nodes, exactly as `service: any` recipes do. When `--node-selector`
is passed it replaces the automatic filters rather than narrowing them.

Valid profiles are the pairs in the applicability table above: `b200/any`,
`gb200/any`, `gb200/eks`, `gb200/oke`, `h100/aks`, `h100/eks`, `h100/gke`,
`h200/eks`, `vr200/rke2`. A
malformed or unknown value **fails** the check rather than silently skipping
it. A valid profile that doesn't implement a requested variant (e.g.
`gb200/eks` with the auto-detect `nccl-all-reduce-bw` check) skips just that
check with a message naming the profile. When a profile is set it wins over
criteria-derived applicability, and the pass/fail thresholds stay in the
same-named check constraints as always.

### Supplying a benchmark runtime for a private service

A profile only helps when an **embedded** template already matches the
cluster's hardware and fabric. A private service introduced entirely through
`--data` — a `criteria.service` (and possibly `criteria.accelerator`) the
validator ships no template for, with a fabric that reuses none of the
embedded ones — has no compiled pair to point a profile at. Such a recipe
supplies the benchmark itself: it ships a Kubeflow `TrainingRuntime` as a file
in its `--data` tree and references it with the `nccl-benchmark-runtime-ref`
performance constraint (a bare `{accelerator}/{service}` value). `aicr validate`
reads that file and gates the benchmark keyed on the recipe's **own** criteria,
with no compiled applicability entry required.

Lay the runtime out **exactly where the embedded templates live**, so it is a
drop-in for upstreaming later:

```text
mydata/                                   # your --data directory
├── overlays/
│   └── mycloud-training.yaml             # the recipe overlay (below)
└── validators/performance/testdata/
    └── gb200/mycloud/runtime.yaml        # the TrainingRuntime, real file
```

```yaml
# overlays/mycloud-training.yaml
validation:
  performance:
    checks:
      - nccl-all-reduce-bw
    constraints:
      - name: nccl-all-reduce-bw
        value: ">= 450"
      - name: nccl-benchmark-runtime-ref
        value: gb200/mycloud   # -> validators/performance/testdata/gb200/mycloud/runtime.yaml
```

```yaml
# validators/performance/testdata/gb200/mycloud/runtime.yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
  name: nccl-all-reduce-runtime
spec:
  template:
    spec:
      replicatedJobs:
        - name: node          # required: the worker cohort
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: <your NCCL benchmark image>
                      resources:
                        limits:
                          nvidia.com/gpu: "${GPU_COUNT_PER_NODE}"
                      # ...command, fabric resources, sidecars...
```

```bash
aicr validate -r recipe.yaml -s snapshot.yaml \
              --data ./mydata --phase performance
```

The supplied runtime **owns its fabric wiring end to end**: because the recipe
opted in explicitly, the validator bypasses the compiled applicability gate and
skips every service-specific *setup* step — EFA/TCPXO/RDMA NIC discovery, the
GB200-NVreg / GKE-TCPXO preflights, and NVLS/IMEX auto-provisioning. It renders
the runtime, sizes it (against the runtime's own `nodeSelector` when it pins
one, so `WorkerCount` matches placement), applies it alongside the shared
`TrainJob`, and evaluates the bandwidth threshold from the launcher logs.
Transport verification is **not** skipped: pairing the runtime with the `-net`
or `-nvls` check still asserts that transport actually carried traffic (the NCCL
markers are fabric-agnostic), so a named variant can't pass on bandwidth alone.
What it must honor:

- It must be a Kubeflow `TrainingRuntime` (`apiVersion:
  trainer.kubeflow.org/v1alpha1`); any other kind/apiVersion is rejected. The
  validator force-sets its name and namespace, so a recipe can supply a
  `TrainingRuntime` and nothing else — never an arbitrary resource.
- It must declare a `node` replicatedJob (the worker cohort the shared
  `TrainJob` sizes and the validator injects scheduling into and reads logs
  from), mirroring the embedded templates.
- The generic template variables `${NAMESPACE}`, `${WORKER_COUNT}`,
  `${GPU_COUNT_PER_NODE}`, `${GPU_COUNT}`, `${TEST_TYPE}`,
  `${MIN_MESSAGE_SIZE}`, and `${MAX_MESSAGE_SIZE}` are substituted; the
  service-specific ones (EFA/RDMA/GKE) are not, so the runtime must pin its own
  fabric resources.
- The runtime should pin its own worker `nodeSelector`/`tolerations`, or pass
  `--node-selector` to `aicr validate`, so workers land on the intended GPU
  nodes.

`nccl-benchmark-runtime-ref` and `nccl-benchmark-profile` are mutually
exclusive — a recipe supplies its own runtime **or** borrows an embedded one,
never both. An absent or blank ref falls back to criteria/profile-derived
applicability. A malformed ref, or a file missing from `--data`, is rejected by
`aicr validate` **before any validator Job is deployed** (a resolution error on
stderr). A resolved file that is not a `TrainingRuntime` with a `node`
replicatedJob **fails the check itself** (in the pod). Either way it fails
rather than silently skipping.

> **Sizing note.** The worker cohort is sized against the runtime's own
> `nodeSelector` when it pins one, but richer scheduling constraints
> (`nodeAffinity`, un-tolerated node taints) are the Kubernetes scheduler's
> job, not the validator's — a runtime that restricts placement below the sized
> cohort surfaces as a launcher timeout, not a false pass. Pass an explicit
> `--node-selector` when you want the sized and scheduled cohorts to match
> exactly.

**Upstreaming.** Because the file already sits at
`validators/performance/testdata/{accelerator}/{service}/runtime.yaml`, adopting
it into AICR is a copy: move the file into the repo, add the accelerator/service
tuple to the compiled `supportedNCCLCombinations` matrix, and drop the
`nccl-benchmark-runtime-ref` constraint — the benchmark then runs from the
recipe's plain criteria like any other embedded one.

## Inference performance validation

Inference performance runs the `inference-perf` check — deploys a
`DynamoGraphDeployment` with a vLLM-served model (Qwen/Qwen3-8B by default,
overridable per accelerator — see below) plus an AIPerf benchmark Job, and
measures end-to-end output-token throughput and time-to-first-token (TTFT) p99.

**Warm-up:** AIPerf sends a wave of warm-up requests *before* the measured run,
so vLLM's one-time CUDA-graph / JIT compilation (tens of seconds on a cold
worker) is excluded from the reported throughput and p99 TTFT — the numbers
reflect steady state, not cold start. Warm-up scales with concurrency and is
tunable via `AICR_INFERENCE_PERF_WARMUP_PER_CONCURRENCY` (see the
[validator reference](/aicr/v0.21.1/contributor-guide/validators#performance-benchmark-tuning)).

**Determinism:** the benchmark is driven reproducibly so the verdict reflects the
deployment, not run-to-run RNG — a fixed random seed, fixed input/output token
counts (stddev 0), a pinned synthetic-prompt pool, and greedy decoding
(`temperature: 0`). Note that throughput (not the latency tail) is the stable,
discriminating signal at high concurrency; TTFT p99 near the saturation knee can
still vary with batching/scheduling, which is why the TTFT constraint is a
generous ceiling rather than a tight target.

```bash
# Capture snapshot, generate inference recipe, validate the performance phase.
aicr snapshot --output snapshot.yaml

aicr recipe --service eks --accelerator h100 --os ubuntu \
            --intent inference --platform dynamo \
            --output recipe.yaml

aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase performance
```

The generated recipe includes `dynamo-platform` in `componentRefs` and lists
`inference-perf` under `validation.performance.checks` with pass/fail
constraints plus benchmark inputs:

```yaml
validation:
  performance:
    checks: [inference-perf]
    constraints:
      # Pass/fail thresholds (10% tolerance applied by the evaluator). Values
      # shown are the measured H100 gate at 8B/256; each overlay sets its own.
      - name: inference-throughput   # output tokens/sec
        value: ">= 50000"
      - name: inference-ttft-p99     # time-to-first-token p99 in ms
        value: "<= 2000"
      # Optional per-accelerator inputs (bare value, no comparator).
      # Precedence: recipe > AICR_INFERENCE_PERF_* env > compiled default.
      - name: inference-model                # HF model ID; default Qwen/Qwen3-8B
        value: Qwen/Qwen3-8B
      - name: inference-concurrency-per-gpu  # positive integer; default 256
        value: "256"
      - name: inference-routing-mode         # dynamo-router or gateway-epp
        value: dynamo-router
```

**Node-shape assumption.** The `inference-throughput` floor is a fixed
absolute full-node value calibrated on a full node (the shared `>= 50000`
gate was measured on 8-GPU H100; GB200 on a 4-GPU node). It is *not*
normalized for GPU count, and the evaluator only scales it down for partial
occupancy — not across node sizes — so a smaller H100 SKU (e.g. 1-/2-GPU
shapes such as `p5.4xlarge`, AKS `NC80adis`) can false-fail a healthy run.
`inference-ttft-p99` is a per-request latency at fixed concurrency-per-GPU
and does not need GPU-count normalization. A normalized per-GPU throughput
floor is tracked in [#1254](https://github.com/NVIDIA/aicr/issues/1254).

`inference-model` and `inference-concurrency-per-gpu` are optional: omit them to
use the compiled defaults (Qwen3-8B at 256 concurrent requests per GPU), set them
per overlay to tune model and load for each accelerator, or override globally
with the `AICR_INFERENCE_PERF_MODEL` / `AICR_INFERENCE_PERF_CONCURRENCY_PER_GPU`
catalog knobs (recipe wins over catalog env wins over default).

`inference-routing-mode` selects the Dynamo Kubernetes routing path. The
default `dynamo-router` mode deploys a Dynamo frontend with load-aware
least-loaded routing (`DYN_ROUTER_MODE=least-loaded`), which balances by each
worker's active in-flight load so a transiently-slow worker stops receiving its
full share (see issue #1197). Normal frontend-to-worker request/response traffic
uses Dynamo's request plane (Dynamo 1.4+ defaults to TCP); AICR does not set
`DYN_REQUEST_PLANE=nats`. Workers publish KV-cache events directly over ZMQ
with no NATS relay, but least-loaded routing does not consume those events —
only `DYN_ROUTER_MODE=kv` does. Set it to `gateway-epp`
to exercise GAIE/EPP: the validator deploys an EPP component, worker frontend
sidecars in direct mode, and an HTTPRoute through the AICR-managed inference
gateway. The direct-mode sidecars honor EPP routing headers; they do not
relay KV events.

**Model-weights cache and `AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS`.** The benchmark downloads
the model **once** into a PVC and serves all workers from it (on by default;
avoids per-IP Hugging Face throttling). The cache PVC needs a StorageClass: it
uses the cluster's **default** StorageClass unless you set
`AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS`. On a cluster with **no default
StorageClass** (common on EKS — e.g. only a non-default `gp2`) and no value set,
the check **fails fast** in seconds with guidance rather than hanging; set
`AICR_INFERENCE_PERF_MODEL_CACHE_STORAGE_CLASS=<name>` (e.g. `gp2`/`gp3` on EKS,
`standard-rwo` on GKE) on the `inference-perf` catalog entry's `env` (or via a
catalog overlay in the `aicr validate --data <dir>` directory), or disable the cache with
`AICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off`. Like the other
`AICR_INFERENCE_PERF_*` knobs, this is a **catalog/`--data`** setting — it is
**not** read from the shell environment of the process running `aicr validate`
(only `HF_TOKEN` is). AICR-deployed EKS clusters get a default `gp3` StorageClass
from the `aws-ebs-csi-driver` component, so the cache works there with no knob.

**Debugging a failed run with `AICR_INFERENCE_PERF_NO_CLEANUP`.** By default the
validator deletes the per-run namespace (DGD, workers, frontend, AIPerf Job) on
both success and failure. To investigate a failure — e.g. a `timed out waiting
for inference endpoint to serve requests` — set `AICR_INFERENCE_PERF_NO_CLEANUP=1`
and the validator leaves everything in place so you can `kubectl logs` the
frontend/workers and curl `/v1/models` and `/v1/chat/completions` live. Unlike the
other `AICR_INFERENCE_PERF_*` knobs, this one is read from the **shell
environment** of the process running `aicr validate` (forwarded to the
inference-perf pod, like `HF_TOKEN`), not from the catalog. Debug-only: you must
delete the `aicr-inference-perf-<suffix>` namespace manually afterward, or it
keeps GPU workers running.

Expected flow (~5–7 min on H100): readiness pre-flight → deploy a
`DynamoGraphDeployment` in a per-run namespace
`aicr-inference-perf-<8-hex-suffix>` → wait for `state=successful` (image pull
+ model load) → endpoint readiness via a real `/v1/chat/completions` inference
request (stricter than a `/health` probe, which returns 200 before the model
can actually serve) → AIPerf benchmark Job parses throughput +
TTFT p99 → compare to recipe constraints (10 % tolerance) → cleanup. Worker
GPU wiring is **configuration-selected and capability-verified** for
recipe-backed runs — the recipe's resolved allocation policy picks the
mechanism, and the inspection probe (the same one the conformance checks
use) verifies the cluster serves it, failing closed on mismatch; only
recipe-less standalone runs (`unspecified`) select by capability. The wiring
supports exactly two GPU allocation configurations: (1) nodes
publishing **node-local `gpu.nvidia.com` ResourceSlices** (full-GPU DRA — the
experimental opt-in; stock recipes default to device-plugin allocation since
the #1327 flip), where workers bind a DRA
`ResourceClaimTemplate` sized from the validated per-node device count —
kai-scheduler treats nodes bearing raw node-local GPU ResourceSlices as
DRA-only and rejects scalar GPU requests, so the claim path is the only
schedulable wiring there;
and (2) **ComputeDomain-only / no full-GPU slices** (device-plugin nodes —
the production default), where workers request GPUs via `nvidia.com/gpu`
limits, which need no `gpu.nvidia.com` DeviceClass. Anything outside those two states fails fast
with an actionable error instead of guessing: full-GPU ResourceSlices from a
non-NVIDIA GPU driver, allocated `ResourceClaim`s requesting a non-NVIDIA
"gpu"-named DeviceClass, `gpu.nvidia.com` slices using non-node-local
topologies (`nodeSelector`/`allNodes`/per-device node selection), pools
published by node-local slices of multiple nodes, and allocated
`ResourceClaim`s from pools no node-local slice publishes (occupancy is
attributed to nodes through the slices' `spec.nodeName` — the K8s API does
not require pool names to be node names) are all rejected —
these are outside the NVIDIA DRA driver's supported configuration
([#1327](https://github.com/NVIDIA/aicr/issues/1327)); generalized
topology/driver support is tracked in
[#1652](https://github.com/NVIDIA/aicr/issues/1652). When the recipe
configures a GPU allocation policy (see
[Configured GPU allocation policy](#configured-gpu-allocation-policy)), the
wiring is forced instead of capability-selected, and under
`dra-resource-claim` candidate GPU nodes are discovered from the probe's
validated DRA facts rather than scalar `nvidia.com/gpu` allocatable — so
DRA-only nodes (device plugin disabled) are discoverable; full DRA-only
`aicr validate` remains experimental (see the #1327 graduation checklist).
The chosen node and wiring mode are recorded in the check's
evidence output.

All Dynamo Frontend and worker pods pin to a single GPU node via
`kubernetes.io/hostname` for a stable per-node baseline. On a shared cluster
where some GPUs on a candidate node are already in use by other workloads,
the validator picks the candidate with the most free GPUs *within the
node's own allocation ledger* and sizes the benchmark to that count: DRA
`ResourceClaim` allocations subtract from DRA capacity, and device-plugin
`nvidia.com/gpu` occupancy from device-plugin capacity. A DRA-wired
candidate carrying **any** scalar `nvidia.com/gpu` workload is skipped
entirely — the device plugin's physical device assignments are invisible to
the DRA allocator, so claims placed there could double-book a plugin-held
GPU. The check therefore does not need an explicit hostname override to
avoid saturated nodes. The `inference-throughput`
gate is a full-node baseline, so when the benchmark runs on fewer than the
node's full GPU count the gate is scaled down by the same `freeGPUs / nodeGPUs`
fraction (throughput scales ~linearly at fixed concurrency-per-GPU) — a healthy
per-GPU result on a partially occupied node is not failed against a full-node
number. TTFT p99 is a per-request latency and is not scaled. Concurrent
`aicr validate` invocations are isolated from each other by the run-specific
suffix on both the namespace and the inner AIPerf Job name.

A passing CTRF entry (measured on EKS H100, 8 × H100 GPUs, Qwen/Qwen3-8B at 256 concurrency/GPU):

```json
{
  "name": "inference-perf",
  "status": "passed",
  "suite": ["performance"],
  "stdout": [
    "RESULT: Inference throughput: 108789.87 tokens/sec",
    "RESULT: Inference TTFT p99: 687.50 ms",
    "Throughput constraint: >= 50000 → PASS",
    "TTFT p99 constraint: <= 2000 → PASS"
  ]
}
```

The `RESULT: ` prefix on the first two lines is the contract documented in
`pkg/validator/validator.go` — any check that wants its summary lines echoed
to the CLI's own output (not just the CTRF report) opts in by emitting that
prefix. The validator runtime strips the prefix when echoing; the full
prefixed line stays in `stdout[]`.

To run deployment validation first (recommended — verifies GPU Operator, DRA
driver, Dynamo operator, KAI scheduler, and supporting components are installed
and healthy):

```bash
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --phase deployment
```

### Skip scenarios

The inference validator has three explicit skip guards so it never runs where
it can't succeed. Each produces a `status: skipped` CTRF entry with a specific
reason. Skipped checks are **not** failures: the validator container exits
with code 2 internally (mapped to CTRF `skipped`), but `aicr validate` itself
exits **0** for a skipped or passed phase — a skipped inference check never
drives a non-zero CLI exit on its own. (A phase that reports `failed` or `other`
exits 8 by default, and is informational only under `--fail-on-error=false`.)

| Guard | Trigger | Skip message |
|-------|---------|--------------|
| **A** | Recipe lists `inference-perf` in `checks:` but no matching `inference-throughput` / `inference-ttft-p99` constraints | `no inference-throughput or inference-ttft-p99 constraint in recipe` |
| **B** | `inference-perf` is selected but `dynamo-platform` is not in recipe `componentRefs` | `skipped - dynamo-platform not in recipe components` |
| **C** | `dynamo-platform` is declared but the `DynamoGraphDeployment` CRD is not installed on the cluster (operator not deployed yet) | `skipped - DynamoGraphDeployment CRD not installed on cluster (dynamo-platform component declared but operator not deployed yet)` |

Guards fire before any cluster mutation, so skips are cheap (typically `< 10 s`).

## Configured GPU allocation policy

When you validate with a recipe, AICR resolves a whole-GPU **allocation
policy** from the recipe's fully hydrated component values and carries it to
the validators ([#1327](https://github.com/NVIDIA/aicr/issues/1327)):

| Policy | Meaning |
|--------|---------|
| `device-plugin-extended-resource` | Whole GPUs via the device plugin (`nvidia.com/gpu` requests) |
| `dra-resource-claim` | Whole GPUs via DRA (`gpu.nvidia.com` ResourceClaims) |
| `dra-extended-resource` | Reserved (KEP-5004 mapped extended resource) — not yet validated |
| `unspecified` | No recipe context (standalone validator runs): capability-driven automatic selection |

The `nvidia-dra-driver-gpu` value `resources.gpus.enabled` is the switch:
`true` resolves `dra-resource-claim`; explicit `false` — or the component
absent or disabled — resolves `device-plugin-extended-resource`. On an
**enabled** DRA component the switch must be explicitly set: the upstream
chart's declared default is `true`, so an absent value would diverge from
what Helm deploys. These configurations are rejected at resolution time with
an invalid-request error:

* an enabled `nvidia-dra-driver-gpu` component with
`resources.gpus.enabled` absent (pin it explicitly in the recipe; stock
recipes always do).
* `gpus.enabled=true` without `gpuResourcesEnabledOverride=true` (the upstream
chart install guard refuses it), and no whole-GPU advertiser remaining.
* `gpus.enabled` off with the GPU operator component (`gpu-operator`, or
`gpu-operator-ocp` on OpenShift recipes) absent, disabled, or carrying
`devicePlugin.enabled=false`.
* A recipe enabling both `gpu-operator` and `gpu-operator-ocp`; two GPU
operators collide at the operand level; enable exactly one (OpenShift recipes
disable gpu-operator and carry gpu-operator-ocp).

Two further states are likewise rejected (they warned during the transition to
the device-plugin production default and are errors since the flip):

* dual advertisement (both mechanisms enabled — exactly one whole-GPU advertiser
is required)
* an inert `gpuResourcesEnabledOverride=true` with `gpus.enabled=false` (the
waiver would disarm the upstream chart's install-guard tripwire).

Stock recipes ship the production default: `gpus.enabled=false`,
`gpuResourcesEnabledOverride=false`, and `devicePlugin.enabled=true`; the
experimental DRA opt-in flips all three together in a recipe overlay.

**Upgrading a cluster from the dual-advertised (pre-flip) configuration:**
applying the flipped bundle does not drain existing workloads — a running
full-GPU `gpu.nvidia.com` claim pod keeps its prepared GPU while the
`nvidia.com/gpu` ledger, which cannot see that assignment, admits newly
converted scalar workloads onto the same device. Migrate in this order:

1. Stop/delete all workloads holding full-GPU `gpu.nvidia.com`
   ResourceClaims (ComputeDomain/IMEX claims are unaffected and stay).
2. Confirm no allocated or reserved full-GPU claims remain, using the
   **driver-specific, fail-closed drain check below**. On IMEX platforms,
   allocated `compute-domain.nvidia.com` claims legitimately remain, so a
   plain `kubectl get resourceclaims -A` can never come back empty —
   filter on the allocation driver instead, and treat ANY output (or any
   query failure) as unsafe to proceed.
3. Apply the flipped bundle (upgrade `nvidia-dra-driver-gpu`).
4. Wait until the `gpu.nvidia.com` ResourceSlices and DeviceClass disappear
   (`compute-domain.nvidia.com` slices must remain).
5. Confirm scalar `nvidia.com/gpu` allocatable is present on the GPU nodes.
6. Only then start the scalar (device-plugin) workloads.

The step-2 drain check — capture and test each stage separately (a piped
`kubectl ... | jq ...` masks a failed List as an empty, safe-looking
result), and proceed only when it exits 0:

```shell
claims="$(kubectl get resourceclaims -A -o json)" || { echo "claim query FAILED — do not proceed"; exit 1; }
unsafe="$(printf '%s' "$claims" | jq -r '
  .items[]
  | select([.status.allocation.devices.results[]?.driver] | index("gpu.nvidia.com"))
  | "\(.metadata.namespace)/\(.metadata.name) reservedFor=\([.status.reservedFor[]?.name] | join(","))"')" \
  || { echo "claim filter FAILED — do not proceed"; exit 1; }
[ -z "$unsafe" ] || { printf 'unsafe full-GPU claims remain:\n%s\n' "$unsafe"; exit 1; }
echo "no full-GPU gpu.nvidia.com claims — safe to proceed"
```

While migrating, also delete any *pending* (unallocated) claims whose spec
references the `gpu.nvidia.com` DeviceClass: they hold no GPU (no
over-admission risk, so the check above rightly ignores them), but the flip
removes that DeviceClass, leaving such claims — and any pods referencing
them — stranded unschedulable forever.

Validators compare the configured policy against the inspected cluster state
and **fail closed on mismatch** — a cluster that cannot serve the configured
mechanism is a validation failure by design, never a silent fallback to the
other mechanism. Only recipe-less standalone runs (`unspecified`) keep the
capability-driven automatic selection.

## Running all phases

```bash
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml
# equivalent to: --phase deployment --phase conformance --phase performance
```

Phases run sequentially. By default all phases run and produce results
regardless of earlier failures. Pass `--fail-fast` to stop after the first
phase that fails (e.g., to skip a 65-minute inference-perf run when deployment
already failed).

## Scoping CNCF submission evidence to specific features

The `--feature` flag scopes which CNCF AI conformance features get behavioral
evidence collected. It only applies to the CNCF-submission evidence collector
and is rejected by the CLI unless `--cncf-submission` is also set (which in
turn requires `--evidence-dir`). It does **not** scope the regular
`--phase conformance` validator run — that one always evaluates every check
defined in the recipe.

```bash
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
  --phase conformance \
  --cncf-submission \
  --evidence-dir ./evidence \
  --feature dra-support --feature gang-scheduling
```

Empty `--feature` (the default) collects evidence for every feature.

Valid feature names (from `pkg/evidence/cncf/collector.go`):

| Name | What it checks |
|------|----------------|
| `dra-support` | Dynamic Resource Allocation driver and ResourceSlices |
| `gang-scheduling` | Gang-scheduler presence and PodGroup support |
| `secure-access` | Cluster authn/authz posture for AI workloads |
| `accelerator-metrics` | GPU metrics exporter and Prometheus scrape config |
| `ai-service-metrics` | Inference-service metrics via custom-metrics API |
| `inference-gateway` | Gateway API + Inference Extension installation; also records the gateway `LoadBalancer` network exposure (open `0.0.0.0/0` vs scoped source ranges). Fails on an open gateway only when `AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY=true`. |
| `robust-operator` | Operator readiness and leader-election posture |
| `pod-autoscaling` | HPA-driven pod autoscaling: external GPU metric + behavioral scale-up/down test (pod-scoped custom metrics collected best-effort — absent for DRA-allocated GPUs, not a failure) |
| `cluster-autoscaling` | Karpenter (preferred) or EKS managed node-group autoscaling fallback |

### Evidence verdicts and exit status

Each collected feature records exactly one verdict in its evidence file:

- **PASS** — the capability was exercised and behaved as expected.
- **SKIP** — an optional prerequisite is absent (e.g. no inference gateway,
  no supported operator, an unsupported cluster-autoscaling provider, or an
  operator that is installed but has no workload to reconcile). A SKIP is not
  a failure.
- **FAIL** — a present capability was unhealthy, a required query failed, or
  the evidence file carried no single valid verdict (fail closed).

A **FAIL** in any feature makes `--cncf-submission` exit non-zero, so a
submission that records a genuine failure no longer reports overall success.
Absent optional prerequisites remain successful SKIPs and do not fail the run.

## Emitting recipe evidence

When a recipe PR targets hardware AICR maintainers cannot independently
re-run, the contributor needs to attach a signed **evidence bundle** so a
maintainer can verify the recipe offline. `aicr validate` produces the
bundle as a side effect when `--emit-attestation` is set; adding `--push`
signs it (cosign keyless via Sigstore) and uploads it to an OCI registry.
This is a different artifact from the CNCF-submission evidence above —
the two flag families produce independent outputs and may run from a
single `aicr validate` invocation.

```bash
aicr validate \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<owner>/aicr-evidence
```

The `--push` tag is just a human-readable label — the `sha256:` digest is
what pins the bundle, so tag choice never affects verification (the verifier
pulls by digest). Omit the tag, as above, and aicr derives a unique
per-recipe one, `<recipe-slug>-<short-fingerprint>` (e.g.
`ghcr.io/<owner>/aicr-evidence:h100-eks-ubuntu-training-3f9a1c2b4d5e`), so
distinct attestations never collide on a shared tag. Pass an explicit tag to
override.

After the command finishes:

```text
./out
├── pointer.yaml                  # locator; copy into recipes/evidence/
└── summary-bundle/
    ├── recipe.yaml               # canonical post-resolution recipe
    ├── snapshot.yaml             # snapshot at validate-time (minimized by default)
    ├── bom.cdx.json              # CycloneDX BOM (auto-generated from
    │                             #   recipe + validator catalog when
    │                             #   --bom is omitted)
    ├── ctrf/                     # per-phase test results (per-test stdout/message omitted by default)
    ├── manifest.json             # per-file sha256 inventory
    ├── statement.intoto.json     # unsigned in-toto Statement
    └── attestation.intoto.jsonl  # signed (when --push is set)
```

The bundle is **minimized by default**: `snapshot.yaml` keeps only an
allowlisted set of fields (dropping node names, provider instance IDs, the
node label/taint set, OS tuning, loaded modules, and systemd config) and the
CTRF reports omit per-test stdout/message — while *preserving* a small
allowlisted set of structured, low-cardinality outcome fields (the CTRF `extra`
object: coverage counts and skip-reason codes, never node names or IPs) so a
signed bundle still distinguishes e.g. a reduced-node pass. The signed predicate
records the applied policy in a `redaction` block, and the bundle self-verifies
exactly like a full one. Pass `--full` to publish the raw payloads instead.

Commit `pointer.yaml` to its per-source path
`recipes/evidence/<recipe>/<src>/<digest>.yaml` — the emit output prints
the exact `copyTo` path, and `<src>` is the slug derived from your signer
identity (see [Artifact Verification](/aicr/v0.21.1/user-guide/artifact-verification#per-source-pointer-layout-and-the-signer-allowlist)).
The bundle itself lives in OCI. Then self-verify before opening the PR — the
same verifier runs against the committed pointer in the CI gate, so exit 0
locally means the gate will pass:

```bash
aicr evidence verify recipes/evidence/<recipe>/<src>/<digest>.yaml
```

**Flag reference:**

| Flag | What it does |
|------|--------------|
| `--emit-attestation <dir>` | Write the bundle to `<dir>`. Required to produce evidence. The bundle is minimized by default — see `--full`. |
| `--full` | Emit the full (unredacted) bundle. By default the snapshot is reduced to an allowlisted set of fields and per-test CTRF stdout/message are omitted, keeping node names, provider instance IDs, the node label/taint set, OS tuning, and raw container logs out of the published artifact. Minimal bundles record the policy in `predicate.redaction` and self-verify normally. |
| `--push <oci-ref>` | Sign via cosign keyless OIDC and push to the registry. The digest pins the bundle, so the tag is just a label; omit it and aicr derives a unique per-recipe tag (`<recipe-slug>-<short-fingerprint>`). Pass an explicit tag to override. Without `--push`, the bundle is unsigned (development/self-debug only). |
| `--bom <path>` | Embed an existing CycloneDX BOM instead of the auto-generated one. Pass `make bom` output for an exhaustive BOM that includes chart-default sub-images. |
| `--identity-token <token>` | Pre-fetched OIDC identity token, skipping the browser flow. Reads `COSIGN_IDENTITY_TOKEN`. |
| `--oidc-device-flow` | Use OAuth device-code flow instead of opening a browser. Reads `AICR_OIDC_DEVICE_FLOW`. |
| `--plain-http` | HTTP instead of HTTPS (local-registry tests only). |
| `--insecure-tls` | Skip TLS verification (self-signed registries). |

**Registry requirements:** the registry must support the OCI 1.1
Referrers API (or its tag-schema fallback) so the Sigstore Bundle can
be attached to the artifact. Known-good registries: GHCR, GitLab
Container Registry, Harbor (≥ 2.8), AWS ECR, Google Artifact Registry,
Azure Container Registry, JFrog Artifactory. Without referrer support
the bundle pushes but the signature is not discoverable, and the
verifier records signature-verify as "skipped (unsigned)" even on a
signed bundle.

**OIDC token resolution.** `--push` resolves an identity token through
this precedence chain: `--identity-token` (or `COSIGN_IDENTITY_TOKEN`)
→ ambient GitHub Actions OIDC (`ACTIONS_ID_TOKEN_REQUEST_URL`
present) → `--oidc-device-flow` (or `AICR_OIDC_DEVICE_FLOW=true`) →
interactive browser. CI pipelines typically rely on the ambient
GitHub Actions path; local workstations get the browser flow.

**Local-only mode (no registry access).** Omitting `--push` still
produces a complete bundle on disk — the verifier records the
signature step as "skipped (unsigned)" and the manifest-hash chain
becomes self-consistency only. Useful for catching accidental
corruption during development, but unsuitable for the CI gate, which
requires a signed bundle bound to a pointer.

For the full producer-and-consumer walkthrough — including OCI-only
verification, the tamper demo, and JSON output for CI gates — see
[Recipe Evidence Demo](https://github.com/NVIDIA/aicr/blob/main/demos/evidence.md).
For the bundle format and verifier semantics, see
[ADR-007](https://github.com/NVIDIA/aicr/blob/main/docs/design/007-recipe-evidence.md).
For the maintainer-side review checklist, see
[Maintaining Recipe Contributions](/aicr/v0.21.1/contributor-guide/maintaining-aicr).
For the per-flag reference on `aicr evidence verify`, see
[CLI reference](/aicr/v0.21.1/user-guide/cli-reference#aicr-evidence-verify).

## Input modes

Snapshot and recipe can come from a file, an HTTPS URL, or a Kubernetes ConfigMap:

```bash
# File (default)
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml

# HTTPS URL
aicr validate \
  --recipe https://artifacts.example.com/recipes/h100-eks-inference.yaml \
  --snapshot https://artifacts.example.com/snapshots/prod-cluster.yaml

# Kubernetes ConfigMap (for in-cluster operators)
aicr validate \
  --recipe cm://gpu-operator/aicr-recipe \
  --snapshot cm://gpu-operator/aicr-snapshot
```

The ConfigMap form is useful when the snapshot is captured by an in-cluster
agent — see [agent deployment](/aicr/v0.21.1/user-guide/agent-deployment).

## Dry-run mode

`--no-cluster` runs the validator against the snapshot alone, skipping all
Kubernetes API calls. Declarative constraints still evaluate; behavioral checks
report `skipped - no-cluster mode (test mode)`.

```bash
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml --no-cluster
```

Useful for CI pipelines that validate a recipe against a captured snapshot
without needing cluster access.

## CI/CD integration

By default `aicr validate` exits non-zero when any phase reports `failed` or
`other` (set `--fail-on-error=false` to report those results and exit 0 anyway;
the readiness pre-flight still exits 2 regardless). CTRF JSON is emitted to
stdout (or to `--output <file>`), so a pipeline can gate promotion on both the
exit code and the structured report:

```bash
aicr validate \
  --recipe recipe.yaml \
  --snapshot cm://gpu-operator/aicr-snapshot \
  --output ctrf.json
```

Exit codes follow Unix conventions and are derived from the CLI's structured
error codes (see [`pkg/errors/exitcode.go`](https://github.com/NVIDIA/aicr/blob/main/pkg/errors/exitcode.go)):

| Code | Meaning |
|------|---------|
| `0` | All phases reported status `passed` or `skipped` |
| `2` | Invalid input or request (`ErrCodeInvalidRequest`) — bad CLI flag, malformed argument, a validator rejecting a recipe value (e.g., an inference constraint that uses the wrong comparator direction), or a **readiness pre-flight** constraint not met. The readiness gate always fails closed with this code, even under `--fail-on-error=false` |
| `5` | A structured `ErrCodeTimeout` reached the top-level CLI — for example while loading the recipe or snapshot, waiting on the snapshot-agent Job, or signing evidence under `--emit-attestation --push` (which runs after the checks). A per-validator wait deadline does **not** reach this code: `runPhase` converts that error into a check result, so it surfaces as exit `8` (see note 2) |
| `8` | One or more phases reported status `failed` (a check that returned a failure verdict — exit code `1` — or, in most cases, a validator Job killed on `activeDeadlineSeconds` — see the note below) **or** `other` (an indeterminate outcome — the check produced no usable verdict; a crash or OOM are the common examples, as is a validator Job that failed for a non-deadline reason with no inspectable pod), including per-check internal timeouts (e.g., DynamoGraphDeployment not ready within `InferenceWorkloadReadyTimeout`). Suppressed by `--fail-on-error=false` |

> **Important:** two subtleties to be aware of when gating a pipeline on exit code:
>
> 1. Both `failed` and `other` are blocking. A phase whose status is `other` — the check produced no usable verdict, e.g. a crash, an OOM, or a Job that failed for a non-deadline reason with its pod already gone — drives the same non-zero exit as `failed` — an inconclusive check fails closed rather than passing silently. `--fail-on-error=false` suppresses **both** result-driven exits (the run reports the outcomes in the CTRF report and exits 0); it does not distinguish `failed` from `other`. A validator Job killed on `activeDeadlineSeconds` normally reports `failed` rather than `other`: its terminal `Failed/DeadlineExceeded` condition is a verdict, so the result names the deadline instead of the missing pod. One narrow exception remains — the CLI waits `activeDeadlineSeconds` plus a 30s buffer for the Job to become terminal, and that buffer equals the validator pod's termination grace period, so a validator that consumes its full grace can exhaust the wait before the condition is stamped. Such a run falls back to the CLI-wait path and still reports `other`.
> 2. Exit 5 is narrower than it sounds. A timeout **inside** a check's own logic (DynamoGraphDeployment not ready, inference endpoint never healthy, AIPerf Job pod-wait deadline) surfaces as a failed phase, not as a structured `ErrCodeTimeout`, so the CLI exits **8**. So does a per-validator wait deadline: `runPhase` hands that error to the timeout handler and records the outcome as a check result rather than propagating it, which is the same path the full-grace exception in note 1 takes. The rule, rather than a list of sources: exit 5 applies whenever an `ErrCodeTimeout` reaches the top-level CLI. A timeout that is converted into a check result instead — whether raised inside a check or by a per-validator wait — surfaces as exit 8.

Scripts that gate on validation outcome should treat **any non-zero code** as
failure rather than branching on specific values, and should additionally
check CTRF `summary.failed` and `summary.other` for a complete picture.

For informational-only runs (report results without failing the build):

```bash
aicr validate ... --fail-on-error=false
```

`--fail-on-error=false` scopes to **phase check results**: a deployment,
performance, or conformance phase that reports `failed` or `other` is recorded
in the CTRF report and the run exits 0. It does not relax the readiness
pre-flight — if a readiness constraint
(K8s version, OS, kernel) is not met, the run still fails closed with exit 2
before any phase executes. In `--no-cluster` mode this is the dominant path,
since all phase checks report `skipped` and readiness is the only inline
constraint evaluation.

## Troubleshooting

### Readiness pre-flight fails

The CLI logs each readiness constraint comparison before any phase runs:

```text
readiness constraint failed: name=K8s.server.version expected=">= 1.34" actual=v1.33.0-eks-abc
```

The run stops before deploying any Jobs and exits 2 (`INVALID_REQUEST`). This is
intentional fail-closed behavior: readiness gates whether the validators can run
at all, so it is terminal regardless of `--fail-on-error` (see above).

Fix: upgrade the cluster, or pick a recipe whose readiness constraints match
the cluster's actual versions.

### Non-standard GPU labels or taints

Default GPU-node discovery looks for `nodeGroup`, `node.kubernetes.io/instance-type`, or GPU-related label substrings. If your cluster uses custom labels, override the scheduling of inner workloads with `--node-selector` and `--toleration`:

```bash
aicr validate \
  --recipe recipe.yaml --snapshot snapshot.yaml --phase performance \
  --node-selector my-org/gpu-pool=h100 \
  --toleration dedicated=worker-workload:NoSchedule \
  --toleration dedicated=worker-workload:NoExecute
```

These flags affect the inner benchmark pods that run on GPU nodes (NCCL workers, Dynamo workers). The example above supplies `--snapshot`, so it does not launch the live snapshot agent. When `--snapshot` is omitted, the flags also configure that preliminary agent. With no toleration override, the agent tolerates all taints; an explicit `spec.validate.agent.tolerations: []` clears that default. Neither flag affects the validator orchestrator Job itself.

For `inference-perf` specifically, `--node-selector` narrows the pool of candidate GPU nodes — the validator then picks the candidate with the most free GPUs (subtracting same-ledger occupancy only — DRA allocations from DRA capacity, device-plugin requests from device-plugin capacity — and skipping DRA candidates that carry scalar `nvidia.com/gpu` workloads) and pins all Dynamo Frontend + worker pods to that node via `kubernetes.io/hostname`. The AIPerf benchmark runner pod is CPU-only, uses a tolerate-all / no-nodeSelector pod spec, and is unaffected by these flags.

### A check reports `skipped` unexpectedly

Skips are always deliberate and always carry a reason, but the location of the
reason in the CTRF entry depends on how the skip happened:

- **Check-level skips** (the CheckFunc ran and returned `validators.Skip(reason)` — e.g., Guards A/B/C on inference, `--no-cluster` from inside a check): reason appears in `stdout` as `level=INFO msg=SKIP reason="…"`.
- **Phase-level skips** (the CheckFunc never ran — e.g., with `--fail-fast`, a prior phase failed so subsequent phases synthesize skip entries; also `--no-cluster` for checks that the runner marks skipped before dispatch): reason appears in `message`, not `stdout`.

Common reasons and their cause:

| Reason (excerpt) | Where it appears | Meaning | Fix |
|------------------|------------------|---------|-----|
| `no inference-throughput or inference-ttft-p99 constraint in recipe` | `stdout` | Check was invoked but recipe is missing the matching constraints | Re-generate the recipe or add the constraints |
| `dynamo-platform not in recipe components` | `stdout` | Inference check selected but `dynamo-platform` absent from `componentRefs` | Use `--platform dynamo` when generating the recipe |
| `DynamoGraphDeployment CRD not installed` | `stdout` | Recipe declares `dynamo-platform` but the operator is not deployed | Run `aicr bundle` + `./deploy.sh` first, or wait for bootstrap to complete |
| `requires Service + Accelerator to be implemented` | `stdout` | The recipe's `criteria` are outside the NCCL benchmark's default applicability (e.g. a service registered via `--data`) | Add an `nccl-benchmark-profile` constraint — see [Opting external recipes into a benchmark profile](#opting-external-recipes-into-a-benchmark-profile) — or, if no embedded template fits, an `nccl-benchmark-runtime-ref` constraint — see [Supplying a benchmark runtime for a private service](#supplying-a-benchmark-runtime-for-a-private-service) |
| `benchmark profile … does not implement the … NCCL variant` | `stdout` | The declared `nccl-benchmark-profile` is valid but has no template for this check variant | Drop the non-applicable check from `validation.performance.checks`, or pick a profile that implements it |
| `failed to resolve nccl-benchmark-runtime-ref=… expected … in the --data tree` | `stderr` | The referenced runtime file is missing from `--data` (or `--data` wasn't passed) | Place the `TrainingRuntime` at `validators/performance/testdata/{accelerator}/{service}/runtime.yaml` in your `--data` dir — see [Supplying a benchmark runtime for a private service](#supplying-a-benchmark-runtime-for-a-private-service) |
| `nccl-benchmark-runtime … must be a … TrainingRuntime … must declare a "node" replicatedJob` | `stdout` | The referenced file is not a Kubeflow `trainer.kubeflow.org/v1alpha1` `TrainingRuntime`, or lacks the `node` replicatedJob | Fix the runtime file to be a valid `TrainingRuntime` with a `node` replicatedJob |
| `nccl-benchmark-runtime and nccl-benchmark-profile are mutually exclusive` | `stdout` | The recipe declares both escape hatches at once | Keep only one: borrow an embedded profile **or** supply your own runtime |
| `skipped - no-cluster mode` | `message` | `--no-cluster` was passed — the runner short-circuits every phase before dispatching any Job | Remove the flag to run behavioral checks |
| `skipped due to previous phase failure` | `message` | `--fail-fast` was set and an earlier phase failed, so subsequent phases were skipped | Fix the earlier phase first, or drop `--fail-fast` to run all phases regardless |

### `ai-service-metrics` fails with "Prometheus unreachable"

On EKS clusters that split worker and system pods across separate security
groups (e.g. DGXC EKS with distinct customer/system ENI subnets), the
conformance check `ai-service-metrics` can fail non-deterministically with:

```text
[SERVICE_UNAVAILABLE] Prometheus unreachable at http://kube-prometheus-prometheus.monitoring.svc:9090 — verify network connectivity
```

The validator orchestrator Job tolerates every taint and sets a *preferred*
`dependencyAffinity` toward Prometheus, so the scheduler co-locates it with the
Prometheus pod when possible. The preference is best-effort, so on fallback it
can still land on any worker node — including one whose ENI is in a security
group whose ingress to the Prometheus-hosting SG is missing or asymmetric. On
such a fallback the outcome is **not stable across re-runs**: image-locality
scoring tends to keep the pod on whatever node won the first scheduling
decision, so a passing run on a fresh cluster does not prove the SG topology is
correct.

This is a cluster-side prerequisite, not an AICR bug per se — see
[EKS Dynamo Networking Prerequisites](/aicr/v0.21.1/integrator-guide/eks-dynamo-networking#required-security-group-rules)
for the SG ingress rules required for Prometheus (`tcp/9090`). The preferred
`dependencyAffinity` ([#933](https://github.com/NVIDIA/aicr/issues/933),
resolved) makes a bad placement far less likely, but the `9090` SG rule remains
the reliable guarantee since the affinity is best-effort.

Workaround when SG changes are not available: re-run the check until the
orchestrator lands on a node whose SG can reach Prometheus, then leave the
image cached there so image-locality keeps subsequent runs on the same node.
This is unreliable and should not be used as the steady-state validation
strategy.

### Benchmark Job stuck or timed out

Each performance check has a Job-level `activeDeadlineSeconds` set by the catalog's `timeout:`. For `inference-perf`, the full pipeline (workload ready → endpoint health → benchmark) can take up to 30 min on cold-start clusters. If it still times out:

```bash
# validator orchestrator Job + AIPerf benchmark Job both live in aicr-validation.
# The orchestrator is named aicr-inference-perf-<hex> (random suffix per run);
# the AIPerf Job is named aicr-aiperf-<run-id-hash>.
kubectl -n aicr-validation get jobs | grep -E 'aicr-inference-perf-|aicr-aiperf-'

# tail each by full job name (label selectors require exact match)
kubectl -n aicr-validation logs -l job-name=aicr-inference-perf-<hash> --tail=200
kubectl -n aicr-validation logs -l job-name=aicr-aiperf-<run-id-hash>  --tail=200

# the Dynamo workload (DynamoGraphDeployment, Frontend, worker pods)
# lives in a separate per-run namespace:
kubectl get ns | grep aicr-inference-perf-
# (resourceclaimtemplates is populated in DRA wiring mode; empty in
# device-plugin mode — both are normal)
kubectl -n aicr-inference-perf-<suffix> get dynamographdeployments,pods,svc,resourceclaimtemplates
```

Common causes: image pull throttling, vLLM model load slowness, and every
eligible candidate GPU node being saturated by existing workloads. In the
saturated case the validator fails fast with a message like `no eligible
candidate GPU node has free GPUs (N matched; eligible candidates are
saturated by existing workloads in the selected mechanism's ledger ...)`.
Occupancy is subtracted per allocation ledger (DRA allocations from DRA
capacity, device-plugin requests from device-plugin capacity), and nodes
excluded for other reasons — NotReady, probe-ineligible, kai-blocked raw
slices, or DRA-capable but carrying scalar `nvidia.com/gpu` workloads — are
not counted as saturated;
the fix is to free GPUs on one of the candidate nodes, or to pass
`--node-selector kubernetes.io/hostname=<node>` to target a specific node
you know is free. GPU-occupancy accounting fails closed rather than treating
GPUs as free: a pod list failure always fails the check, and so does any
error listing DRA `resourceclaims` (e.g. RBAC denied, timeout) — with one
exception: when the `resource.k8s.io` API is not served at all (NotFound),
which deterministically means zero DRA usage, the validator proceeds with
device-plugin pod requests alone.

## Related

- [CLI reference: `aicr validate`](/aicr/v0.21.1/user-guide/cli-reference#aicr-validate) — full flag reference and per-command examples
- [CLI reference: `aicr snapshot`](/aicr/v0.21.1/user-guide/cli-reference#aicr-snapshot) — snapshot capture options
- [CLI reference: `aicr recipe`](/aicr/v0.21.1/user-guide/cli-reference#aicr-recipe) — recipe generation flags
- [Agent deployment](/aicr/v0.21.1/user-guide/agent-deployment) — capture snapshots via an in-cluster Job
- [Data flow: Stage 3 Validate](/aicr/v0.21.1/integrator-guide/data-flow#stage-3-validate-constraint-checking) — how the validator engine is built
- [Validator Development Guide](/aicr/v0.21.1/contributor-guide/validators) — add a new validator (contributor-facing)