Platform Detection & Overrides

Auto-detection

The controller detects two dimensions at runtime:

On a mixed-architecture target, the detected GPU architecture is the one reported by the most nodes, with ties resolved to the architecture whose earliest node sorts first by name. Nodes missing the nvidia.com/gpu.product label do not participate in that vote, so an unlabeled node never outvotes labeled ones; unknown is detected only when no target node carries the label. Every path uses the same rule: the Certification, Workflow, and WorkloadRun controllers as well as the nvcrectl render, cluster info, and workloadrun commands.

ResourceSlice fallback (no device plugin/GFD)

Platforms that claim GPUs via Dynamic Resource Allocation instead of a device plugin, such as Nscale NKS, run no NVIDIA GPU Feature Discovery DaemonSet, so no node ever carries the nvidia.com/gpu.product label. Before the majority-architecture vote runs, node discovery lists gpu.nvidia.com ResourceSlice objects cluster-wide and reads each device’s productName attribute, writing it as the nvidia.com/gpu.product label onto the corresponding node’s in-memory copy (never persisted back to the API server). The driver reports the space-separated NVML name (NVIDIA GB300), so it is sanitized exactly as GPU Feature Discovery writes the label: characters outside [A-Za-z0-9-_. ] are dropped and each run of whitespace becomes one hyphen (NVIDIA-GB300); identical hardware therefore reports one product in a fleet that mixes both, provided GPU Feature Discovery is not time-slicing (it then appends -SHARED to its label, so the products differ while the architecture still matches). Every existing label-based consumer — the architecture vote, catalog architecture defaults, NIC detection — is unaffected: they only ever read the label, and simply see it populated from a different source. Devices of type vfio (GPU passthrough) are skipped, because they report a PCI-IDs name rather than the NVML product name.

The fallback supplies only the product. GPU nodes must still carry nvidia.com/gpu.present=true, which the NVIDIA GPU Operator sets: node discovery drops every node without it before the fallback runs. On a cluster whose GPU nodes carry no NVIDIA labels at all, such as one running only the standalone DRA GPU driver, label the GPU nodes nvidia.com/gpu.present=true yourself; otherwise node discovery finds no GPU nodes.

This lookup is skipped entirely when every node already carries the label (every non-DRA cluster), and one ResourceSlice List call, read directly from the API server rather than the controller cache, covers the whole target set. A missing resourceslices RBAC grant, or a driver that publishes no productName attribute, degrades to “leave those nodes unlabeled” rather than failing the reconcile — unknown architecture is the same outcome an unlabeled node with a device plugin produces today. The fallback reads resource.k8s.io/v1 ResourceSlices (Kubernetes 1.34+, the same floor the DRA GPU driver needs); on older servers the List fails and nodes stay unlabeled.

Offline nvcrectl certification render and nvcrectl workloadrun render have no cluster to read ResourceSlices from, so on a label-less platform like Nscale, offline render requires --gpu-arch to resolve architecture at all. --gpu-arch cannot be combined with --dry-run, which discovers real nodes and runs the same fallback the controllers do.

The live controller writes detection results to status.orchestration.detectedPlatform and status.orchestration.detectedGPUArchitecture on the Workflow. When using nvcrectl workflow render (client-side), these values are also written as annotations (nvcrectl.nvidia.com/detected-platform, nvcrectl.nvidia.com/detected-gpu-architecture) on the rendered manifest for offline inspection.

Override matching

Catalog entries define a base WorkflowSpec. Overrides are matched by platform + GPU architecture and applied in order. Two patch mechanisms are supported:

jobTemplate (strategic merge patch)

Replaces entire arrays. Use when you want to set all values for a field (e.g. the full env list):

jobTemplate:
  spec:
    workload:
      trainJob:
        trainer:
          env:
            - name: MY_VAR
              value: "value"

<Warning> Arrays in jobTemplate overrides are replaced entirely, not merged by name. If the base spec has env vars, they will be wiped unless you include them in the override. </Warning>

jobTemplatePatch (RFC 6902 JSON Patch)

Appends or modifies individual fields without replacing arrays. Use op: add with path ending in /- to append to an array:

jobTemplatePatch:
  - op: add
    path: /spec/workload/trainJob/trainer/env/-
    value:
      name: EXTRA_VAR
      value: "extra"

<Warning> A jobTemplatePatch that appends to an array must come after any jobTemplate override that sets that array, or the appended values will be wiped. </Warning>

Architecture-specific resources

Different GPU architectures and cloud platforms require different Kubernetes resources:

OCI GB200 is the one OCI architecture that requests no NIC by default. The shapes do not consistently expose nvidia.com/mlnxnics, and the sriov-net NetworkAttachmentDefinition that the annotation references is not present on a stock cluster, so a default request leaves every worker pod Pending on Insufficient nvidia.com/mlnxnics. Sites running the SR-IOV device plugin set mlnxPerNode to their own per-node count.

Setting mlnxPerNode: 0 is a supported opt-out on any platform whose templates request nvidia.com/mlnxnics (Azure, OCI, TogetherAI, Forge): at zero, the resource and the network attachment annotation are both omitted rather than requested as "0". On Nscale, which claims its NICs through DRA, zero likewise omits the rdma.nscale.com claim while keeping the GPU claim.

RTX PRO 6000 defaults to eight GPUs per node, including on GCP. The default is architecture-based, not machine-shape detection. On smaller G4 nodes, set gpusPerNode explicitly in the Certification or WorkloadRun to the node’s allocatable GPU count (for example, gpusPerNode: 4 for g4-standard-192). Without that setting, nodes advertising fewer than eight allocatable GPUs are excluded by the capacity filter; an all-four-GPU target fails rather than silently reducing the request. An explicit smaller request on an eight-GPU node tests only the requested GPUs and does not establish full-device coverage.

The NCCL overlay selects NCCL_P2P_LEVEL=SYS for eight GPUs and PHB for two or four GPUs, following Google’s G4 guidance. Offline rendering also uses the eight-GPU default and does not validate node capacity. Set gpusPerNode: 4 explicitly when previewing the four-GPU configuration. The live controller applies the capacity filter.

The GCP RTX PRO 6000 override sets NCCL_NET_PLUGIN=none in addition to NCCL_IB_DISABLE=1. The PyTorch image includes an external RDMA plugin that otherwise attempts to initialize even on this TCP-only network.

The live controller tracks which overrides matched in status.orchestration.appliedOverrides. When using nvcrectl workflow render, the same information is also written to the nvcrectl.nvidia.com/applied-overrides annotation on the rendered manifest.

Host RDMA device access

Requesting no NIC resource leaves the workload with no RDMA access at all. That is the right default where the site runs a device plugin and just needs to name it, but some clusters expose the Mellanox devices only as host device nodes under /dev/infiniband, with no SR-IOV policy and no plugin to request. OCI GB200 shapes are the common case.

For those, set rdmaDeviceAccess: hostPath on the Certification (or on one category’s options). It mounts the host’s /dev/infiniband into the workload container and runs that container privileged:

spec:
  rdmaDeviceAccess: hostPath

The mount and the privileged flag are one unit rather than two independent choices. Kubelet builds a container’s device cgroup allowlist from the resources it requests, so a hostPath mount on its own makes /dev/infiniband/uverbs0 visible while open() on it still fails with EPERM. No capability changes that, including IPC_LOCK, which the communication entries already grant.

The option applies on every platform and architecture, because how a cluster plumbs RDMA is a property of the cluster rather than of its cloud. Leaving it unset, or setting it to devicePlugin, renders exactly what it rendered before: the NIC resource request described above and nothing else. Workload pods will be rejected in namespaces that enforce a restricted or baseline Pod Security Standard, which forbid privileged containers. See ADR-088.

A privileged container is root-equivalent on the node. It runs without the runtime’s default seccomp filter and without AppArmor or SELinux confinement, so the pod-level seccompProfile: RuntimeDefault the training entries set no longer constrains the worker container even though the pod spec still shows it. The /dev/infiniband mount names the devices the workload uses; it is not what limits access. Treat the option as granting the workload root on the node, because that is what privileged: true does.

Some CRI-O runtimes reject a privileged container whose GPUs come from the NVIDIA device plugin, on the grounds that the device paths are already present on the host. The catalog records one such configuration: the OCI arm of the DCGM entry turns privileged off for exactly that reason. Every container this option patches requests nvidia.com/gpu, so if your runtime enforces that rule, the pods fail to start rather than running without the mount. Try one category before turning the option on for a whole certification.

The value is case-sensitive and is validated by nvcrectl as well as by the API server, so rdmaDeviceAccess: HostPath is an error rather than a render that silently omits the mount.

When a node has no RDMA devices

The volume is declared type: Directory, so kubelet refuses to start the pod on a node where /dev/infiniband does not exist rather than mounting an empty directory that NCCL would later report as “no devices found”. Kubelet retries the mount indefinitely, so the pod stays in ContainerCreating with a FailedMount event naming the path.

timeoutPerJob bounds that wait and is the only thing that does. The communication entries time the Job out after one hour by default; the training entries after 24 hours, a backstop chosen because a healthy training run can legitimately exceed an hour. In both cases the Workflow reports the failure. Set spec.timeoutPerJob (or categories[].options.timeoutPerJob) to tighten either bound; on a training category that is the difference between learning about a missing /dev/infiniband in minutes and in a day. Watch for a pod stuck in ContainerCreating with a FailedMount event, and try one category before enabling the option across a whole certification.

startupStallTimeoutSeconds, which the training entries do set, does not cover this case on any entry. Startup-stall detection is anchored on the GoodputMeasurement, and the GoodputMeasurement only starts sampling once worker-0 reaches Running, which a pod held in ContainerCreating never does.

On-prem clusters

onprem is the detection fallback: a node with an empty spec.providerID (and no Forge hostname or NKE site-name label) resolves to the onprem platform. Nodes provisioned by bare-metal stacks such as BCM typically carry no providerID, so they land here without any configuration.

For GB200/GB300 (NVL72) targets, the on-prem override contributes:

• Tolerations for the kubernetes.io/arch=arm64:NoSchedule and nvidia.com/gpu=present:NoSchedule taints common on NVL72 deployments, on the MPI launcher pod as well as the node pods. Without them, workload pods never schedule on tainted arm64 nodes.
• A portable InfiniBand NCCL environment without HCA pinning: NCCL auto-detects HCAs when NCCL_IB_HCA is unset, so the same override works across sites with different HCA layouts.
• A NIC resource request, detected automatically or configured via nicResourceName. Resource names vary by RDMA device plugin (rdma/ib, nvidia.com/mlnxnics, and others are all in the wild), so when the field is unset the controller inspects the target nodes: if exactly one candidate resource (any rdma/* name, or the exact name nvidia.com/mlnxnics) is allocatable at the resolved mlnxPerNode count on every target node, that name is requested. Detection never guesses or over-commits: with zero candidates, with several, or when candidates exist but no node set can cover the requested count, no NIC resource is requested (pods still schedule, without an explicit NIC allocation) and a Normal NICResourceDetection event on the Certification or WorkloadRun explains what was found, naming the requested count when candidates fall below it. Set nicResourceName to override detection or to resolve an ambiguous fleet. The per-container count always comes from mlnxPerNode, detected or not. GB200, GB300, B200, and B300 default mlnxPerNode to 8; sites running a shared-device plugin (one pooled resource per pod) should set mlnxPerNode: 1; with the default of 8, a pooled resource advertised as rdma/ib: 1 is not detected, because the resulting request could never schedule.

For x86 HGX B200/B300 targets, the on-prem override is deliberately smaller: a toleration for the nvidia.com/gpu taint with any value (operator: Exists, since generic on-prem fleets do not agree on present), applied to the MPI launcher as well as the node pods, and the same NIC resource request, detected or set via nicResourceName, with the count from mlnxPerNode. It sets no NCCL environment: an HGX chassis has no multi-node NVLink, so the NVL72 tuning does not apply, and fabric-specific tuning (RoCE GID index, HCA selection) waits on multi-node validation. Sites running a shared RDMA device plugin (one pooled resource such as rdma/rdma_shared_device_a advertised at a large count) should set mlnxPerNode: 1; the default of 8 schedules, but consumes eight quota units for the same device access. On a WorkloadRun, spec.resources and the NIC request merge by resource name and the NIC request’s count wins, so when both name the same resource, mlnxPerNode sets the count and spec.resources should carry only other resources. See ADR-094.

An architecture missing from the catalog’s GPU defaults table (gpu-defaults.yaml) has no node shape and falls back to four GPUs and no NICs unless gpusPerNode and mlnxPerNode are set. The controllers emit a Warning event, reason GPUArchitectureDefaults, on the Certification or WorkloadRun whenever the detected architecture is missing from the table, naming the fallback and the counts the run is actually sized at; every nvcrectl render path and nvcrectl cluster info print the same message to stderr. Set gpusPerNode and mlnxPerNode explicitly to correct the counts, and file an issue so the architecture can be added.

Offline nvcrectl certification render and nvcrectl workloadrun render have no cluster to inspect, so without --dry-run they render field-only: the NIC resource appears only when nicResourceName is set. With --dry-run, real nodes are discovered and detection runs exactly as in the controllers.

apiVersion: nvcre.nvidia.com/v1alpha1
kind: Certification
metadata:
  name: onprem-gb300-cert
spec:
  target:
    nodeSelector:
      nvidia.com/gpu.product: NVIDIA-GB300
  nodesPerJob: 18
  enableMNNVL: true
  # Extended resource name advertised by the site's RDMA device plugin.
  # Optional: when omitted, the controller auto-detects a single qualifying
  # rdma/* or nvidia.com/mlnxnics resource allocatable at the mlnxPerNode
  # count on every target node.
  # Set it to override detection or when several candidates are advertised.
  nicResourceName: rdma/ib
  mlnxPerNode: 1   # shared-device plugin: one pooled resource per pod
  categories:
    - domain: communication
      variant: nccl-all-reduce

<Warning> Metal3 detection trap. Nodes with a metal3:// providerID resolve to the mistral platform, not onprem. An on-prem site provisioned by Cluster API + Metal3 therefore silently picks up the Mistral site-specific override, which pins HCA names (NCCL_IB_HCA, UCX_NET_DEVICES) and the rdma/ib resource name. To preview what a generic on-prem render looks like regardless of providerID, pass the platform explicitly: nvcrectl certification render --platform onprem <cert.yaml>. An x86 HGX B200/B300 site provisioned the same way matches neither the Mistral block (gated on GB300) nor the HGX block (gated on onprem), so the controller applies no toleration and no NIC request there. </Warning>

Diagnostics (dcgm-level4) needs no on-prem override: it already tolerates all taints and runs intra-node, so it schedules on tainted arm64 nodes unchanged.

GCP H100 clusters

On GCP H100, the override runs NCCL over GPUDirect-TCPXO. It adds a tcpxo-daemon sidecar, the NCCL_FASTRAK_* environment, and a networking.gke.io/interfaces pod annotation that attaches the pod to the node’s eight GPU NIC networks as eth1 through eth8. GKE looks up each network in that annotation as a Network object when the pod is admitted, and rejects the pod if one is missing. The names are chosen by whoever provisions the cluster: AICR names them <deployment-id>-gpu-nic-0 through -7, and Google’s samples use vpc1 through vpc8.

So the controller reads the names from the target nodes rather than assuming them. GKE advertises an extended resource networking.gke.io.networks/<network> on every node attached to a network:

• Which networks. The controller uses the networks advertised on every target node, and only when there are exactly eight of them. It ignores the <network>.IP resources and the default network.
• Which order. When the nodes’ networking.gke.io/north-interfaces and networking.gke.io/nic-info annotations place each network on a host NIC, and all nodes agree, the pod’s eth1 carries the same network as the host’s eth1, and so on. Otherwise the names are sorted.
• When detection fails. Detection never guesses. With any count other than eight, the controller renders the default names gpu-nic0 through gpu-nic7 and emits a Warning GKENetworkDetection event on the Certification listing what it found. Causes include no multi-networking, a network missing from one node, a ninth network on every node, or an A3 High pool. GKE rejects the resulting pods unless the cluster has Network objects with those default names.

To see what the nodes advertise, list the networks and a GPU node’s allocatable resources:

kubectl get networks.networking.gke.io
kubectl get node <gpu-node> -o jsonpath='{.status.allocatable}'

Offline nvcrectl certification render has no nodes to inspect, so it renders the default names. With --dry-run, detection runs against the real nodes and prints the networks it found, or why it fell back, to stderr.

The TCPXO NCCL plugin comes from the node, not the image. GKE’s TCPXO installer puts it in /home/kubernetes/bin/nvidia, which the override mounts at /usr/local/nvidia. The plugin links a CUDA runtime that it does not bundle, so it loads the one in the workload image, and the two CUDA major versions must match. Google also pairs each plugin release with one tcpxo-daemon release. Both are properties of the installed plugin, not of GCP H100: Google qualifies plugin v1.0.16 and earlier against CUDA 12, and v1.0.17 against CUDA 13.2 (with GPU driver R595 and GKE 1.33.5-gke.1125000 or later).

Plugin version detection. The controller reads the plugin release from GKE’s nccl-tcpxo-installer pods in kube-system. The release is the tag of the init container image nccl-plugin-gpudirecttcpx-dev, read through an @sha256 digest if one is appended. The image is matched by that last path segment, so a mirrored registry is still recognized. Google allows upgrading the plugin one node pool at a time, so only the installers on the job’s target nodes count, and only Running ones. Google sometimes republishes a release as a rebuild with a build suffix (v1.0.13-1 a week after v1.0.13). A suffix does not change the release, so v1.0.17-1 gets the v1.0.17 row, and nodes caught mid-rollout between v1.0.17 and v1.0.17-1 agree on v1.0.17. The release then picks the images:

The images cover the NCCL test workers, launcher and launcher init container, and the training workers and their megatron-clone init container. The CUDA 13 images are an earlier CUDA 13 minor version than Google’s 13.2 qualification; they have the same libcudart.so.13 soname but have not yet been validated on a v1.0.17 cluster. The v1.0.15 and v1.0.16 rows are validated on an AICR A3 Mega cluster (GKE 1.35, driver 580.159.04): nccl-all-reduce reaches 337.72 GB/s (v1.0.15) and 337.48 GB/s (v1.0.16) bus bandwidth at 16 GB, and nemotron5-8b 425.6 and 422.9 TFLOPs per GPU.

Supported releases and fallbacks. v1.0.15 is the minimum supported release. Detection never guesses silently: unless every target node runs one mapped release, the controller renders the nearest safe row and emits a Warning TCPXOPluginDetection event on the Certification saying what it found and what it rendered.

• A release newer than the latest mapped one renders the latest row, since a newer plugin keeps at least the newest CUDA major. Those images are not validated against the newer release.
• A release between mapped ones that has no row of its own renders the newest row older than it, for the same reason, and is not validated either.
• A release older than v1.0.15 renders the v1.0.15 row, since every earlier release is also CUDA 12. Upgrade the installer if you can.
• No release at all (no running installer pod on a target node, nodes on different releases, an untagged or non-release tag, or a failed pod List) renders the v1.0.15 row: the release AICR-provisioned clusters pin, and the one validated on hardware. A cluster built from Google’s published manifest gets the latest plugin, which is CUDA 13, so if detection finds nothing there, set the image yourself.

To check the installed release:

kubectl -n kube-system get daemonset nccl-tcpxo-installer \
  -o jsonpath='{.spec.template.spec.initContainers[*].image}'

Offline nvcrectl certification render has no cluster to read, so it renders the v1.0.15 row. With --dry-run, detection runs against the real installer pods and prints the release and images it picked, or why it fell back, to stderr.

Overriding the image. Set image per category, under categories[].options.image, not as the certification-wide spec.image. The spec-level field is a default for every category, so it also replaces the image of categories that have nothing to do with TCPXO. diagnostics/dcgm-level4 runs dcgmi out of a DCGM image, and a PyTorch image has no dcgmi, so the spec-level form fixes NCCL and breaks diagnostics in the same certification:

categories:
  - domain: communication
    variant: nccl-all-reduce
    options:
      image: nvcr.io/nvidia/pytorch:26.01-py3   # CUDA 13 plugin
  - domain: diagnostics
    variant: dcgm-level4                        # keeps its own DCGM image

Whatever image you set, it must also ship what the entry runs: the *_perf_mpi NCCL test binaries, or the training stack. The tcpxo-daemon sidecar keeps its own image and is unaffected by image.

Symptom to fix. Both directions of the mismatch fail at plugin load, and the missing soname names the CUDA major the plugin wants, not the one the image has:

The TCPXO NCCL plugin also checks the NCCL environment against the a3plus_guest_config.textproto that GKE’s TCPXO installer puts on each node, and aborts the job on any value the file enforces. The NCCL_FASTRAK_* environment NVCRE sets matches the enforced values of current installers, for example NCCL_PROTO=Simple,LL128. It also leaves NCCL_ALGO unset, as the file recommends. If a job fails with NCCL WARN NCCL/NET (shim) mismatch enforced, compare the named variable with that file on the node: /home/kubernetes/bin/nvidia/lib64/a3plus_guest_config.textproto, which is mounted in the pod at /usr/local/nvidia/lib64/.

<Note> Only A3 Mega (a3-megagpu-8g) is supported. A3 High (a3-highgpu-8g) is also H100, but it uses GPUDirect-TCPX with four GPU NICs, which this override does not configure. Both report nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3, so the override cannot tell them apart. The eight-network rule is what flags an A3 High target. </Note>