> This page is for AI Cluster Runtime, version Latest · v1.2.0-rc0 (default). > For other versions, use one of these documentation indexes: > - Latest · v1.2.0-rc0 (default): https://docs.nvidia.com/aicr/latest/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server. # NVKE GB200 Setup The `gb200-nvke-ubuntu-training` recipe targets NVKE (NVIDIA Kubernetes Engine, formerly NKE): bare-metal GB200 NVL trays provisioned by Forge/NICo, running Ubuntu 24.04 with NVIDIA's `-nvidia-64k` kernel. It is a Preview coordinate. `gb200-nvke-ubuntu-training-kubeflow` (`--platform kubeflow`) is the same leaf plus Kubeflow Trainer through the `platform-kubeflow` mixin. ## Cluster prerequisites - Kubernetes 1.34 or newer (DRA `resource.k8s.io/v1` for ComputeDomains). - Nodes carry `providerID: nico://`. `aicr snapshot` maps that scheme to `service: nvke`, so `aicr recipe --snapshot` selects this recipe without `--service`. - GPU nodes are untainted by the provisioner and run nothing GPU- or fabric-related: GPU Operator installs the driver, toolkit and GDRCopy. - Control-plane nodes carry GPU-node labels they do not earn (`nvidia.com/gpu.present=true`, `nvidia.com/gpu.deploy.driver=true`, `feature.node.kubernetes.io/pci-15b3.present=true` from the BlueField-3) while having no GPU. Anything that selects GPU nodes by those labels alone also selects the control plane; the recipe excludes control-plane nodes explicitly where that matters (`rdma-modules`), and `deploy.sh`'s DRA driver-migration gate waits its full timeout on them before continuing. - The primary CNI is OVN-Kubernetes on the BlueField-3, bound to the host's inbox `mlx5_core`. It is the only CNI on the image: no multus, no reference plugins beyond what OVN-Kubernetes needs. - Some NVKE builds hand every pod on the pod network one BlueField-3 VF: an `sdn-mutator` webhook adds a `nvidia.com/asap2_vf: 1` request and a node advertises 28, which caps non-hostNetwork pods at 28 per node, platform pods included (the recipe's own DaemonSets take eight slots per GPU node for GPU Operator plus nvsentinel, NFD, DRA and monitoring). The reference tenant has neither the webhook nor the resource. Where it does, keep a system node pool for the cluster-scoped charts or expect pods to stay Pending on `Insufficient nvidia.com/asap2_vf`. - East-west fabric: four ConnectX-7 InfiniBand rails per tray (`mlx5_0`, `mlx5_1`, `mlx5_3`, `mlx5_4`), active NDR, no IPoIB netdevs until the recipe loads `ib_ipoib`. `mlx5_2` is the BlueField-3 host representor. Firmware enables SR-IOV with `NUM_OF_VFS` 16 per PF (`sriov_totalvfs`), which is the count the recipe requests (see below). - InfiniBand traffic between hosts needs every tenant port, PF and VF, to be a full member of a common partition: a P_Key with the `0x8000` bit set, either `0xffff` (default partition) or a tenant P_Key assigned by the fabric (UFM). On the reference cluster every port, including allocated VFs, held only `0x7fff` (limited membership): links are up and LIDs assigned, but every RDMA connection between hosts fails with retry-exceeded, even same-tray loopback through the switch. Check the non-zero entries under `/sys/class/infiniband/mlx5_*/ports/1/pkeys/`. VFs are separate virtual ports with their own GUIDs, so the fabric must add the VF GUIDs to the partition too (UFM GUID mapping or ib-kubernetes); fixing the PFs alone is not enough. The fabric must provision partition membership for each VF GUID. This recipe does not select a non-default P_Key for IPoIB interfaces (the pinned `SriovIBNetwork` has no such field, and the ib-sriov CNI `pkey` only informs ib-kubernetes, which this recipe does not install); native RDMA applications can select their own P_Key index. Traffic between trays in the same NVLink domain uses multi-node NVLink and does not need this; it matters for jobs spanning NVLink domains and for IPoIB or RDMA storage. - `rdma_ucm`, `ib_umad` and `ib_ipoib` are not loaded by the NVKE image and `rdma-core` is not installed (the modules ship in `linux-modules-`); the recipe's `rdma-modules` component loads them (see below). A modules-load.d entry for all three in the image makes that component a no-op. Installing `rdma-core` instead also works: its udev rules load `/etc/rdma/modules/infiniband.conf` (`ib_ipoib`, `ib_umad`) and `rdma.conf` (`ib_iser`, `ib_uverbs`, `rdma_ucm`, `xprtrdma`, `svcrdma`), a superset of what the recipe loads. - The tenant image ships no multus: `/etc/cni/net.d` holds only `10-ovn-kubernetes.conf`, `/opt/cni/bin` has no `multus`, and no DaemonSet or package provides one (measured 2026-10-28). OVN-Kubernetes runs with `--enable-multi-network`, but that serves only its own NetworkAttachmentDefinitions; a pod that requests `sriov-network-rail-` on such a node gets the VF allocated and no IB interface, silently. The NicClusterPolicy therefore carries `secondaryNetwork.multus` and `cniPlugins`; the NVKE image contract is OVN-Kubernetes only, and multus is the recipe's to install. The operator renders the multus DaemonSet with `--skip-config-watch`, so the service-account token multus copies into `multus.kubeconfig` is not refreshed on rotation. Under default API-server settings the legacy-token extension keeps that copy valid for one year from issuance (`serviceaccount.ExpirationExtensionSeconds`); other settings may shorten it. Restart the multus pods before it expires to rewrite the file. The SR-IOV policies select only the ConnectX-7 root devices. ## Never install DOCA-OFED on these nodes The Network Operator in this recipe ships **no `ofedDriver`**. The host runs the inbox `mlx5_core` and OVN-Kubernetes rides on it. A DOCA driver container unloads `mlx5_core` to replace it, and the node's primary address disappears with it; a reboot restores the inbox driver and the operator applies the DOCA driver again. This was observed on NVKE GB200 and bare-metal GB300 clusters with recipes that carried an `ofedDriver` block. Do not add one back for NVKE, and do not combine this recipe with another Network Operator installation. The same posture applies to GPUDirect RDMA: it runs over DMA-BUF. The recipe sets no `driver.rdma` value (no `nvidia-peermem`, which has no build for the 64k-page Grace kernel and needs OFED symbols) and installs the `NVreg_GrdmaPciTopoCheckOverride=1` module parameter through the `kernel-module-params` ConfigMap, which the `nccl-all-reduce-bw-net` preflight requires. ## Containerd configuration The recipe sets `gpu-operator.toolkit.env.RUNTIME_CONFIG_SOURCE=file` so the NVIDIA toolkit reads `/etc/containerd/config.toml` directly. NVKE v0.22.0's qualified Ubuntu image ships containerd 2.3.0 with a version 3 configuration. The default command source reads `containerd config dump`, which migrates the effective configuration to version 4. A version 4 NVIDIA drop-in imported by the version 3 root prevents containerd from starting. Reading the file keeps the NVIDIA drop-in at the host's configuration version without changing the node image or migrating its root configuration. The same toolkit defect took down AKS nodes with a version 2 root (NVIDIA/aicr#1688). The toolkit also refuses `NVIDIA_VISIBLE_DEVICES` from unprivileged containers (`ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false`). GPUs reach a workload only through a `nvidia.com/gpu` request or a DRA claim; an unprivileged pod that selects the `nvidia` RuntimeClass and sets `NVIDIA_VISIBLE_DEVICES=all` sees none. Privileged management pods (the GPU Operator operands, NVSentinel's metadata-collector) keep the env-var path, because the toolkit always honors it for privileged containers. Unlike the AKS and OpenShift recipes, this one leaves `ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS` at its default of `false`: `DEVICE_LIST_STRATEGY` still emits the volume-mount device list beside CDI, but the toolkit ignores it while the flag is `false`, and CDI is what injects devices here. ## SR-IOV InfiniBand The Network Operator chart enables the SR-IOV Network Operator and the Maintenance Operator (external drainer, one node at a time). The NicClusterPolicy carries `nvIpam`, multus and the reference CNI plugins. Per rail, the `sriov-ib-networks` component (manifest-only, ordered after `network-operator`) supplies one nv-ipam `IPPool` (`10.255.<4n>.0/22`, outside the `192.168.0.0/16` pod network), one `SriovNetworkNodePolicy` (`linkType: IB`, `isRdma: true`, `deviceType: netdevice`, `numVfs: 16`, selected by the rail's PCI root device) and one `SriovIBNetwork` whose NetworkAttachmentDefinition lands in `default`. Nodes advertise `nvidia.com/sriov_resource_rail_0` to `_3`, 16 each. A pod takes one VF per rail and the matching attachments: ```yaml metadata: annotations: k8s.v1.cni.cncf.io/networks: default/sriov-network-rail-0,default/sriov-network-rail-1,default/sriov-network-rail-2,default/sriov-network-rail-3 spec: containers: - resources: limits: nvidia.com/sriov_resource_rail_0: 1 nvidia.com/sriov_resource_rail_1: 1 nvidia.com/sriov_resource_rail_2: 1 nvidia.com/sriov_resource_rail_3: 1 ``` This is the shape k8s-launch-kit generates for `--deployment-type sriov --fabric infiniband` on this topology, with the Maintenance Operator budget set to one node, with `numVfs` raised from l8k's 8 to the firmware's 16, with the IPPool subnets moved from l8k's `192.168.<4n>.0/22` to `10.255.<4n>.0/22` (outside the OVN-Kubernetes pod network), and with the SR-IOV config daemon's `mellanox` plugin disabled (`sriovOperatorConfig.disablePlugins`). That plugin rewrites firmware `NUM_OF_VFS` to exactly `numVfs` and then runs `mstfwreset -l 3`, which this tray's firmware rejects ("not supported in the current state of this device"); with it enabled, a policy that differed from the firmware count drained the node and then looped on the reset with the node cordoned. With the plugin off, firmware is a host prerequisite (SR-IOV enabled, `NUM_OF_VFS`, IB link type) and VF creation is a sysfs write with no drain or reboot, so `numVfs` must not exceed the firmware count. Check `mstconfig -d q NUM_OF_VFS` before changing `numVfs`. The policies and pools select nodes by `feature.node.kubernetes.io/rdma.available=true` rather than `pci-15b3.present`: the control plane carries that label (and `pci-0302_10de.present`) without any ConnectX-7, and when it was selected the operator placed a device-plugin pod there whose init container waited forever, which the `expected-resources` check flags. Enabling the SR-IOV subchart on a Network Operator release that was installed without it needs `charts/sriov-network-operator/crds` applied first: Helm installs a subchart's CRDs only on a fresh install, and validates the rendered `SriovOperatorConfig` before the chart's own CRD-upgrade hook runs. The IPPool CRD is created by the operator while it reconciles the NicClusterPolicy, not by the chart, so `sriov-ib-networks` can be applied before the CRD exists. `deploy.sh` retries the release until it does. With Argo CD or Terraform, build with `--readiness-hooks`: the `network-operator` readiness gate waits for the NicClusterPolicy to reach `ready`, which the operator reports only after its nv-ipam state, CRDs included, is applied, so the pools follow it. Helmfile cannot take `--readiness-hooks` and its bundle has no CRD wait, so install it in two stages. `kubectl wait` fails at once on an object that does not exist yet, so wait for the CRD to be created before waiting for it to be Established (`--for=create` needs kubectl 1.31 or later): ```bash cd bundle helmfile -l 'name!=sriov-ib-networks' apply kubectl wait --for=create crd/ippools.nv-ipam.nvidia.com --timeout=10m kubectl wait --for=condition=Established crd/ippools.nv-ipam.nvidia.com --timeout=2m helmfile -l name=sriov-ib-networks apply ``` Flux does not support readiness hooks either. If `sriov-ib-networks` exhausts its install retries on the missing CRD, wait for the CRD to be created and Established, then reset the release: ```bash kubectl wait --for=create crd/ippools.nv-ipam.nvidia.com --timeout=10m kubectl wait --for=condition=Established crd/ippools.nv-ipam.nvidia.com --timeout=2m flux reconcile helmrelease sriov-ib-networks -n flux-system --reset ``` The `sriov-ib-networks` health check asserts the CRD is Established, so `aicr validate` records that it landed. Measured on a reference tenant after a full teardown: the CRD appeared five seconds after the NicClusterPolicy and the policy was `ready` four seconds later; the gate held `sriov-ib-networks` until then. When tearing the fabric down, uninstall `sriov-ib-networks` and wait for every `SriovNetworkNodeState` to return to `Succeeded` before uninstalling `network-operator`: removing the policies makes the config daemon request a drain through the Maintenance Operator, and uninstalling the operators while that drain is in flight orphans the `NodeMaintenance`, which leaves the node cordoned with nothing left to release it. The Maintenance Operator's controller is what removes the `maintenance.nvidia.com/finalizer` from a `NodeMaintenance` and uncordons its node, so with the operator gone either reinstall `network-operator` and let it finish the cleanup, or recover by hand: ```bash kubectl get nodemaintenances.maintenance.nvidia.com -A kubectl delete nodemaintenance '' -n '' --wait=false # Only if the object stays pending on its finalizer: kubectl patch nodemaintenance '' -n '' --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl uncordon '' ``` A shared-HCA variant (`rdmaSharedDevicePlugin`, one pooled `nvidia.com/mlnxnics` resource over the PFs, no IPoIB) was qualified first on the same cluster and passed the deployment and conformance phases. It is not selectable from this recipe; it survives only in the history of [NVIDIA/aicr#3295](https://github.com/NVIDIA/aicr/pull/3295) before the SR-IOV switch. ## rdma-modules and node tainting `rdma-modules` is a manifest-only component carrying one `NodeWright` CR that loads `rdma_ucm`, `ib_umad` and `ib_ipoib` and writes `/etc/modules-load.d/aicr-rdma-ulp.conf` on every `pci-15b3.present` node that is not a control-plane node (the control plane also carries a BlueField-3; an auto-tainted node whose taints the package cannot tolerate would otherwise stay tainted). The package has no interrupt (no cordon, drain or reboot). `runtimeRequired: true` with `autoTaintNewNodes` lets nodewright taint GPU nodes until the modules are loaded, so the SR-IOV config daemon and GPU workloads never start against an empty `/dev/infiniband` or VFs without IPoIB netdevs. The CR selects `feature.node.kubernetes.io/pci-15b3.present=true` by default, the cohort the NicClusterPolicy pins its DaemonSets to. `--accelerated-node-selector` and `--accelerated-node-toleration` reach only this component: the NicClusterPolicy keeps its fixed affinity and GPU Operator selects nodes by its own labels, so a different selector must still cover every node the fabric DaemonSets land on, or those nodes never get the modules and are never gated. Disable the taint with `--set rdmamodules:autoTaintNewNodes=false`. On images that already load all three modules, disable the component in the recipe rather than at bundle time: set `overrides.enabled: false` on the `rdma-modules` componentRef in `recipe.yaml` (or in a `--data` overlay), which drops it from the bundle and from the deployment phase's expected resources alike. A bundle-time `--set rdmamodules:enabled=false` is not persisted, so `aicr validate` would still assert the CR. ## Generate and validate ```bash aicr snapshot --node-selector kubernetes.io/hostname='' --privileged \ --output snapshot.yaml aicr recipe --snapshot snapshot.yaml --intent training --output recipe.yaml aicr bundle --recipe recipe.yaml --readiness-hooks --output bundle ./bundle/deploy.sh aicr validate --recipe bundle/recipe.yaml --phase deployment --phase conformance ``` `aicr validate` runs the deployment and conformance phases against the live cluster, so the bundle must be deployed before it. Performance thresholds are copied from the GB200 OKE recipe. The `nccl-benchmark-profile` constraint names the `gb200/oke` runtime. `nccl-all-reduce-bw-nvls` runs with it and measured 839.5 GB/s on the reference tenant (2026-10-29), the NVKE NVLS baseline; the `>= 500` floor is kept until the result is recorded in qualification evidence. `nccl-all-reduce-bw-net` cannot schedule: that runtime's workers request `nvidia.com/mlnxnics`, which this leaf does not expose, so it waits for a `gb200/nvke` runtime requesting the per-rail VFs and attachments. ## Known gaps - No `gb200/nvke` NCCL runtime yet: `nccl-all-reduce-bw-net` borrows the `gb200/oke` runtime, whose workers request `nvidia.com/mlnxnics`, which this leaf does not expose, so that check cannot schedule. - The reference fabric has not passed RDMA traffic (limited P_Key membership, see prerequisites), so neither the SR-IOV nor the earlier shared-HCA shape has NCCL evidence. - `nccl-all-reduce-bw-nvls` passes on the reference tenant: 839.5 GB/s across both trays (8 GPUs, one multi-node NVLink domain), measured 2026-10-29. It needs DRA driver 0.5.1: the NVKE API server enables `OwnerReferencesPermissionEnforcement`, under which 0.5.0's daemon on the second tray cannot join the trays' shared NVLink clique (see [Upgrade Notes: `nvidia-dra-driver-gpu` 0.5.0 to 0.5.1](/aicr/user-guide/component-catalog#nvidia-dra-driver-gpu-050-to-051)). - No published evidence yet. - KWOK profiles model scheduling only; they cannot validate RDMA traffic.