> This page is for NVIDIA Cluster Readiness Engine, version v0.6.0. > For other versions, use one of these documentation indexes: > - Latest ยท v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt > - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server. # Platform Detection & Overrides > How the controller auto-detects cloud platform and GPU architecture, and how overrides are applied. ## Auto-detection The controller detects two dimensions at runtime: | Dimension | Source | |-----------|--------| | Cloud platform | `spec.providerID` on the Node object (e.g. `aws://...`, `gce://...`) | | GPU architecture | `nvidia.com/gpu.product` node label (e.g. `NVIDIA-GB200`, `NVIDIA-H100-80GB-HBM3`) | On a mixed-architecture target, the detected GPU architecture is the one reported by the most nodes, with ties resolved to the architecture whose earliest node sorts first by name. Nodes missing the `nvidia.com/gpu.product` label do not participate in that vote, so an unlabeled node never outvotes labeled ones; `unknown` is detected only when no target node carries the label. Every path uses the same rule: the Certification, Workflow, and WorkloadRun controllers as well as the `nvcrectl` render, cluster info, and workloadrun commands. ### ResourceSlice fallback (no device plugin/GFD) Platforms that claim GPUs via Dynamic Resource Allocation instead of a device plugin run no NVIDIA GPU Feature Discovery DaemonSet, so no node ever carries the `nvidia.com/gpu.product` label. Before the majority-architecture vote runs, node discovery lists `gpu.nvidia.com` `ResourceSlice` objects cluster-wide and reads each device's `productName` attribute, writing it as the `nvidia.com/gpu.product` label onto the corresponding node's in-memory copy (never persisted back to the API server). The driver reports the space-separated NVML name (`NVIDIA GB300`), so it is sanitized exactly as GPU Feature Discovery writes the label: characters outside `[A-Za-z0-9-_. ]` are dropped and each run of whitespace becomes one hyphen (`NVIDIA-GB300`); identical hardware therefore reports one product in a fleet that mixes both, provided GPU Feature Discovery is not time-slicing (it then appends `-SHARED` to its label, so the products differ while the architecture still matches). Every existing label-based consumer โ€” the architecture vote, catalog architecture defaults, NIC detection โ€” is unaffected: they only ever read the label, and simply see it populated from a different source. Devices of type `vfio` (GPU passthrough) are skipped, because they report a PCI-IDs name rather than the NVML product name. The fallback supplies only the product. GPU nodes must still carry `nvidia.com/gpu.present=true`, which the NVIDIA GPU Operator sets: node discovery drops every node without it before the fallback runs. On a cluster whose GPU nodes carry no NVIDIA labels at all, such as one running only the standalone DRA GPU driver, label the GPU nodes `nvidia.com/gpu.present=true` yourself; otherwise node discovery finds no GPU nodes. This lookup is skipped entirely when every node already carries the label (every non-DRA cluster), and one `ResourceSlice` `List` call, read directly from the API server rather than the controller cache, covers the whole target set. A missing `resourceslices` RBAC grant, or a driver that publishes no `productName` attribute, degrades to "leave those nodes unlabeled" rather than failing the reconcile โ€” `unknown` architecture is the same outcome an unlabeled node with a device plugin produces today. The fallback reads `resource.k8s.io/v1` ResourceSlices (Kubernetes 1.34+, the same floor the DRA GPU driver needs); on older servers the `List` fails and nodes stay unlabeled. Offline `nvcrectl certification render` and `nvcrectl workloadrun render` have no cluster to read `ResourceSlice`s from, so on a label-less platform, offline render requires `--gpu-arch` to resolve architecture at all. `--gpu-arch` cannot be combined with `--dry-run`, which discovers real nodes and runs the same fallback the controllers do. The live controller writes detection results to `status.orchestration.detectedPlatform` and `status.orchestration.detectedGPUArchitecture` on the Workflow. When using `nvcrectl workflow render` (client-side), these values are also written as annotations (`nvcrectl.nvidia.com/detected-platform`, `nvcrectl.nvidia.com/detected-gpu-architecture`) on the rendered manifest for offline inspection. ## Override matching Catalog entries define a base `WorkflowSpec`. Overrides are matched by platform + GPU architecture and applied in order. Two patch mechanisms are supported: ### `jobTemplate` (strategic merge patch) Replaces entire arrays. Use when you want to set all values for a field (e.g. the full `env` list): ```yaml jobTemplate: spec: workload: trainJob: trainer: env: - name: MY_VAR value: "value" ``` <Warning> Arrays in `jobTemplate` overrides are **replaced entirely**, not merged by name. If the base spec has env vars, they will be wiped unless you include them in the override. </Warning> ### `jobTemplatePatch` (RFC 6902 JSON Patch) Appends or modifies individual fields without replacing arrays. Use `op: add` with path ending in `/-` to append to an array: ```yaml jobTemplatePatch: - op: add path: /spec/workload/trainJob/trainer/env/- value: name: EXTRA_VAR value: "extra" ``` <Warning> A `jobTemplatePatch` that appends to an array must come **after** any `jobTemplate` override that sets that array, or the appended values will be wiped. </Warning> ## Architecture-specific resources Different GPU architectures and cloud platforms require different Kubernetes resources: | Architecture | Platform | Interconnect | Key resources | |-------------|---------|-------------|--------------| | GB200 | AWS | EFA | `hugepages-2Mi`, `vpc.amazonaws.com/efa`, EFA hostPath volume, ComputeDomain | | GB200 | Azure | InfiniBand | mlnxnics dep, topo ConfigMap, ComputeDomain | | GB300 | AWS | RoCE | `roce-channel` resource claim (DRA), no hugepages, no EFA | | GB300 | Azure | InfiniBand | mlnxnics dep, topo ConfigMap, ComputeDomain | | H100 | AWS | EFA | `vpc.amazonaws.com/efa: 32`, no hugepages | | H100 | Azure | InfiniBand | mlnxnics dep, topo ConfigMap | | RTX PRO 6000 Blackwell | GCP G4 | TCP over `eth0`, PCIe GPU peer-to-peer | Variable GPU count by G4 machine size; `nvidia.com/gpu=present:NoSchedule` toleration; no RDMA resource request | | GB200/GB300 | On-prem | InfiniBand | arm64/GPU taint tolerations, portable IB NCCL env (no HCA pinning), NIC resource auto-detected or set via `nicResourceName`, ComputeDomain | RTX PRO 6000 defaults to **eight GPUs per node**, including on GCP. The default is architecture-based, not machine-shape detection. On smaller G4 nodes, set `gpusPerNode` explicitly in the Certification or WorkloadRun to the node's allocatable GPU count (for example, `gpusPerNode: 4` for `g4-standard-192`). Without that setting, nodes advertising fewer than eight allocatable GPUs are excluded by the capacity filter; an all-four-GPU target fails rather than silently reducing the request. An explicit smaller request on an eight-GPU node tests only the requested GPUs and does not establish full-device coverage. The NCCL overlay selects `NCCL_P2P_LEVEL=SYS` for eight GPUs and `PHB` for two or four GPUs, following [Google's G4 guidance](https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines#g4_series). Offline rendering also uses the eight-GPU default and does not validate node capacity. Set `gpusPerNode: 4` explicitly when previewing the four-GPU configuration. The live controller applies the capacity filter. The GCP RTX PRO 6000 override sets `NCCL_NET_PLUGIN=none` in addition to `NCCL_IB_DISABLE=1`. The PyTorch image includes an external RDMA plugin that otherwise attempts to initialize even on this TCP-only network. The live controller tracks which overrides matched in `status.orchestration.appliedOverrides`. When using `nvcrectl workflow render`, the same information is also written to the `nvcrectl.nvidia.com/applied-overrides` annotation on the rendered manifest. ## On-prem clusters `onprem` is the detection fallback: a node with an empty `spec.providerID` (and no Forge hostname or NKE site-name label) resolves to the `onprem` platform. Nodes provisioned by bare-metal stacks such as BCM typically carry no providerID, so they land here without any configuration. For GB200/GB300 targets, the on-prem override contributes: - **Tolerations** for the `kubernetes.io/arch=arm64:NoSchedule` and `nvidia.com/gpu=present:NoSchedule` taints common on NVL72 deployments. Without them, workload pods never schedule on tainted arm64 nodes. - **A portable InfiniBand NCCL environment** without HCA pinning: NCCL auto-detects HCAs when `NCCL_IB_HCA` is unset, so the same override works across sites with different HCA layouts. - **A NIC resource request**, detected automatically or configured via `nicResourceName`. Resource names vary by RDMA device plugin (`rdma/ib`, `nvidia.com/mlnxnics`, and others are all in the wild), so when the field is unset the controller inspects the target nodes: if exactly one candidate resource (any `rdma/*` name, or the exact name `nvidia.com/mlnxnics`) is allocatable at the resolved `mlnxPerNode` count on every target node, that name is requested. Detection never guesses or over-commits: with zero candidates, with several, or when candidates exist but no node set can cover the requested count, no NIC resource is requested (pods still schedule, without an explicit NIC allocation) and a Normal `NICResourceDetection` event on the Certification or WorkloadRun explains what was found, naming the requested count when candidates fall below it. Set `nicResourceName` to override detection or to resolve an ambiguous fleet. The per-container count always comes from `mlnxPerNode`, detected or not. GB200/GB300 default `mlnxPerNode` to 8; sites running a shared-device plugin (one pooled resource per pod) should set `mlnxPerNode: 1`; with the default of 8, a pooled resource advertised as `rdma/ib: 1` is not detected, because the resulting request could never schedule. Offline `nvcrectl certification render` and `nvcrectl workloadrun render` have no cluster to inspect, so without `--dry-run` they render field-only: the NIC resource appears only when `nicResourceName` is set. With `--dry-run`, real nodes are discovered and detection runs exactly as in the controllers. ```yaml apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification metadata: name: onprem-gb300-cert spec: target: nodeSelector: nvidia.com/gpu.product: NVIDIA-GB300 nodesPerJob: 18 enableMNNVL: true # Extended resource name advertised by the site's RDMA device plugin. # Optional: when omitted, the controller auto-detects a single qualifying # rdma/* or nvidia.com/mlnxnics resource allocatable at the mlnxPerNode # count on every target node. # Set it to override detection or when several candidates are advertised. nicResourceName: rdma/ib mlnxPerNode: 1 # shared-device plugin: one pooled resource per pod categories: - domain: communication variant: nccl-all-reduce ``` <Warning> **Metal3 detection trap.** Nodes with a `metal3://` providerID resolve to the `mistral` platform, not `onprem`. An on-prem site provisioned by Cluster API + Metal3 therefore silently picks up the Mistral site-specific override, which pins HCA names (`NCCL_IB_HCA`, `UCX_NET_DEVICES`) and the `rdma/ib` resource name. To preview what a generic on-prem render looks like regardless of providerID, pass the platform explicitly: `nvcrectl certification render --platform onprem `. </Warning> Diagnostics (`dcgm-level4`) needs no on-prem override: it already tolerates all taints and runs intra-node, so it schedules on tainted arm64 nodes unchanged.