Platform Detection & Overrides

Auto-detection

The controller detects two dimensions at runtime:

On a mixed-architecture target, the detected GPU architecture is the one reported by the most nodes, with ties resolved to the architecture whose earliest node sorts first by name. Nodes missing the nvidia.com/gpu.product label do not participate in that vote, so an unlabeled node never outvotes labeled ones; unknown is detected only when no target node carries the label. Every path uses the same rule: the Certification, Workflow, and WorkloadRun controllers as well as the nvcrectl render, cluster info, and workloadrun commands.

ResourceSlice fallback (no device plugin/GFD)

Platforms that claim GPUs via Dynamic Resource Allocation instead of a device plugin run no NVIDIA GPU Feature Discovery DaemonSet, so no node ever carries the nvidia.com/gpu.product label. Before the majority-architecture vote runs, node discovery lists gpu.nvidia.com ResourceSlice objects cluster-wide and reads each device’s productName attribute, writing it as the nvidia.com/gpu.product label onto the corresponding node’s in-memory copy (never persisted back to the API server). The driver reports the space-separated NVML name (NVIDIA GB300), so it is sanitized exactly as GPU Feature Discovery writes the label: characters outside [A-Za-z0-9-_. ] are dropped and each run of whitespace becomes one hyphen (NVIDIA-GB300); identical hardware therefore reports one product in a fleet that mixes both, provided GPU Feature Discovery is not time-slicing (it then appends -SHARED to its label, so the products differ while the architecture still matches). Every existing label-based consumer — the architecture vote, catalog architecture defaults, NIC detection — is unaffected: they only ever read the label, and simply see it populated from a different source. Devices of type vfio (GPU passthrough) are skipped, because they report a PCI-IDs name rather than the NVML product name.

The fallback supplies only the product. GPU nodes must still carry nvidia.com/gpu.present=true, which the NVIDIA GPU Operator sets: node discovery drops every node without it before the fallback runs. On a cluster whose GPU nodes carry no NVIDIA labels at all, such as one running only the standalone DRA GPU driver, label the GPU nodes nvidia.com/gpu.present=true yourself; otherwise node discovery finds no GPU nodes.

This lookup is skipped entirely when every node already carries the label (every non-DRA cluster), and one ResourceSlice List call, read directly from the API server rather than the controller cache, covers the whole target set. A missing resourceslices RBAC grant, or a driver that publishes no productName attribute, degrades to “leave those nodes unlabeled” rather than failing the reconcile — unknown architecture is the same outcome an unlabeled node with a device plugin produces today. The fallback reads resource.k8s.io/v1 ResourceSlices (Kubernetes 1.34+, the same floor the DRA GPU driver needs); on older servers the List fails and nodes stay unlabeled.

Offline nvcrectl certification render and nvcrectl workloadrun render have no cluster to read ResourceSlices from, so on a label-less platform, offline render requires --gpu-arch to resolve architecture at all. --gpu-arch cannot be combined with --dry-run, which discovers real nodes and runs the same fallback the controllers do.

The live controller writes detection results to status.orchestration.detectedPlatform and status.orchestration.detectedGPUArchitecture on the Workflow. When using nvcrectl workflow render (client-side), these values are also written as annotations (nvcrectl.nvidia.com/detected-platform, nvcrectl.nvidia.com/detected-gpu-architecture) on the rendered manifest for offline inspection.

Override matching

Catalog entries define a base WorkflowSpec. Overrides are matched by platform + GPU architecture and applied in order. Two patch mechanisms are supported:

jobTemplate (strategic merge patch)

Replaces entire arrays. Use when you want to set all values for a field (e.g. the full env list):

jobTemplate:
  spec:
    workload:
      trainJob:
        trainer:
          env:
            - name: MY_VAR
              value: "value"

<Warning> Arrays in jobTemplate overrides are replaced entirely, not merged by name. If the base spec has env vars, they will be wiped unless you include them in the override. </Warning>

jobTemplatePatch (RFC 6902 JSON Patch)

Appends or modifies individual fields without replacing arrays. Use op: add with path ending in /- to append to an array:

jobTemplatePatch:
  - op: add
    path: /spec/workload/trainJob/trainer/env/-
    value:
      name: EXTRA_VAR
      value: "extra"

<Warning> A jobTemplatePatch that appends to an array must come after any jobTemplate override that sets that array, or the appended values will be wiped. </Warning>

Architecture-specific resources

Different GPU architectures and cloud platforms require different Kubernetes resources:

RTX PRO 6000 defaults to eight GPUs per node, including on GCP. The default is architecture-based, not machine-shape detection. On smaller G4 nodes, set gpusPerNode explicitly in the Certification or WorkloadRun to the node’s allocatable GPU count (for example, gpusPerNode: 4 for g4-standard-192). Without that setting, nodes advertising fewer than eight allocatable GPUs are excluded by the capacity filter; an all-four-GPU target fails rather than silently reducing the request. An explicit smaller request on an eight-GPU node tests only the requested GPUs and does not establish full-device coverage.

The NCCL overlay selects NCCL_P2P_LEVEL=SYS for eight GPUs and PHB for two or four GPUs, following Google’s G4 guidance. Offline rendering also uses the eight-GPU default and does not validate node capacity. Set gpusPerNode: 4 explicitly when previewing the four-GPU configuration. The live controller applies the capacity filter.

The GCP RTX PRO 6000 override sets NCCL_NET_PLUGIN=none in addition to NCCL_IB_DISABLE=1. The PyTorch image includes an external RDMA plugin that otherwise attempts to initialize even on this TCP-only network.

The live controller tracks which overrides matched in status.orchestration.appliedOverrides. When using nvcrectl workflow render, the same information is also written to the nvcrectl.nvidia.com/applied-overrides annotation on the rendered manifest.

On-prem clusters

onprem is the detection fallback: a node with an empty spec.providerID (and no Forge hostname or NKE site-name label) resolves to the onprem platform. Nodes provisioned by bare-metal stacks such as BCM typically carry no providerID, so they land here without any configuration.

For GB200/GB300 targets, the on-prem override contributes:

• Tolerations for the kubernetes.io/arch=arm64:NoSchedule and nvidia.com/gpu=present:NoSchedule taints common on NVL72 deployments. Without them, workload pods never schedule on tainted arm64 nodes.
• A portable InfiniBand NCCL environment without HCA pinning: NCCL auto-detects HCAs when NCCL_IB_HCA is unset, so the same override works across sites with different HCA layouts.
• A NIC resource request, detected automatically or configured via nicResourceName. Resource names vary by RDMA device plugin (rdma/ib, nvidia.com/mlnxnics, and others are all in the wild), so when the field is unset the controller inspects the target nodes: if exactly one candidate resource (any rdma/* name, or the exact name nvidia.com/mlnxnics) is allocatable at the resolved mlnxPerNode count on every target node, that name is requested. Detection never guesses or over-commits: with zero candidates, with several, or when candidates exist but no node set can cover the requested count, no NIC resource is requested (pods still schedule, without an explicit NIC allocation) and a Normal NICResourceDetection event on the Certification or WorkloadRun explains what was found, naming the requested count when candidates fall below it. Set nicResourceName to override detection or to resolve an ambiguous fleet. The per-container count always comes from mlnxPerNode, detected or not. GB200/GB300 default mlnxPerNode to 8; sites running a shared-device plugin (one pooled resource per pod) should set mlnxPerNode: 1; with the default of 8, a pooled resource advertised as rdma/ib: 1 is not detected, because the resulting request could never schedule.

Offline nvcrectl certification render and nvcrectl workloadrun render have no cluster to inspect, so without --dry-run they render field-only: the NIC resource appears only when nicResourceName is set. With --dry-run, real nodes are discovered and detection runs exactly as in the controllers.

apiVersion: nvcre.nvidia.com/v1alpha1
kind: Certification
metadata:
  name: onprem-gb300-cert
spec:
  target:
    nodeSelector:
      nvidia.com/gpu.product: NVIDIA-GB300
  nodesPerJob: 18
  enableMNNVL: true
  # Extended resource name advertised by the site's RDMA device plugin.
  # Optional: when omitted, the controller auto-detects a single qualifying
  # rdma/* or nvidia.com/mlnxnics resource allocatable at the mlnxPerNode
  # count on every target node.
  # Set it to override detection or when several candidates are advertised.
  nicResourceName: rdma/ib
  mlnxPerNode: 1   # shared-device plugin: one pooled resource per pod
  categories:
    - domain: communication
      variant: nccl-all-reduce

<Warning> Metal3 detection trap. Nodes with a metal3:// providerID resolve to the mistral platform, not onprem. An on-prem site provisioned by Cluster API + Metal3 therefore silently picks up the Mistral site-specific override, which pins HCA names (NCCL_IB_HCA, UCX_NET_DEVICES) and the rdma/ib resource name. To preview what a generic on-prem render looks like regardless of providerID, pass the platform explicitly: nvcrectl certification render --platform onprem <cert.yaml>. </Warning>

Diagnostics (dcgm-level4) needs no on-prem override: it already tolerates all taints and runs intra-node, so it schedules on tainted arm64 nodes unchanged.