NVKE VR200 Setup
The vr200-nvke-ubuntu-* recipes target NVKE (NVIDIA Kubernetes Engine,
formerly NKE): bare-metal Vera Rubin (VR200) NVL72 trays provisioned by
Forge/NICo, running Ubuntu 26.04 on an arm64 -nvidia-bos-64k kernel. Four
Preview coordinates:
vr200-nvke-ubuntu-trainingvr200-nvke-ubuntu-training-kubeflow— adds Kubeflow Trainer to the training leafvr200-nvke-ubuntu-inference— platform-neutral inference base, which resolves when--platformis omittedvr200-nvke-ubuntu-inference-dynamo— addsdynamo-platformandgroveto the inference leaf
All four declare gang-scheduling, a v1.36 MUST: the KAI Scheduler the check
exercises comes from the NVKE base, and a check the recipe does not name
renders as “not tested”, which never establishes the requirement.
Everything below applies to all four unless a section says otherwise. See The inference coordinate for what the inference leaf adds and leaves out, and The workload leaves for what the other two add.
It pairs the NVKE platform handling of gb200-nvke-ubuntu-training with the
VR200 node facts of vr200-rke2-ubuntu-training, and differs from both on the
driver: the NVKE node image pre-bakes none, so GPU Operator installs it.
Cluster prerequisites
- Kubernetes 1.34 or newer (DRA
resource.k8s.io/v1for ComputeDomains). Observed in practice: 1.36.3. - Ubuntu 26.04. The recipe constrains
OS.release.VERSION_IDto26.04. - The
nvidia-bos-64kkernel line. This is the hard prerequisite and the recipe cannot express it. See below. - Nodes carry
providerID: nico://<uuid>, soaicr snapshotresolvesservice: nvkewithout--service.
The nvidia-bos-64k kernel is mandatory and unconstrainable
On the generic nvidia-64k kernel no driver brings the GPUs up at all: PRI
reads return 0xbadf5720, FSP and GSP never hand off, and every GPU fails probe
with -EIO. Verified against driver 610.57.04, 615.71.09 and 615.78.08. On
nvidia-bos-64k the stock driver container works.
The recipe cannot gate on this. The constraint grammar has only >=, <=, >,
<, == and != with exact match — no substring or regex — and a version range
cannot separate the two series, because pkg/version.Compare drops every build
suffix except -gke.N. So 7.0.0-1018-nvidia-64k and
7.0.0-2022-nvidia-bos-64k both compare as 7.0.0. Check the kernel yourself
before deploying:
A kernel-flavour measurement, so the recipe can require this, is a follow-up.
The inference coordinate
vr200-nvke-ubuntu-inference is the same node facts at intent: inference. It
adds agentgateway-crds and agentgateway — the Gateway API surface — and
nothing that serves a model; actual serving is
vr200-nvke-ubuntu-inference-dynamo, described under The workload leaves.
It declares the gateway rather than expecting one on the cluster because AICR
gates ai_inference on the recipe: a leaf that ships no gateway records
SKIP (prerequisite absent) even when a healthy Gateway is Programmed, since
“a Gateway left over from another install is not evidence for this recipe”.
Three things it deliberately leaves out, each matching
vr200-rke2-ubuntu-inference:
- No performance block. An all-reduce floor is a training claim; the MNNVL measurement stays with the training coordinate.
- No AI operator.
robust_controller_check.gotakes its subject from the recipe —kubeflow-trainer, elsedynamo-platform, else skip — so the check skips for this leaf, which is why the conformance list does not schedule it.vr200-nvke-ubuntu-inference-dynamois the leaf that establishesrobust_controllerfor this coordinate. - No east-west NIC stack. Serving through the Gateway API needs no RoCE
rails, so neither
network-operatornorsriov-ib-networksis declared.
External address for the gateway
NVKE provisions no LoadBalancer controller, and the Service the controller
generates for the Gateway is type: LoadBalancer. A bare-metal LB
implementation (MetalLB, kube-vip) is what gives it an external address.
Without one the Service stays at EXTERNAL-IP <pending> and the gateway is
reachable only in-cluster or through its NodePort.
That is a reachability limit, not a conformance one. agentgateway v1.5.0
reports Programmed=True when no negative condition is present, regardless of
address allocation, and inference_gateway_check.go gates readiness on ready
endpoints rather than an external address — its LoadBalancer handling inspects
spec.loadBalancerSourceRanges for internet exposure and records
“No LoadBalancer Service” as an observation.
The Gateway is a separate bundle step. agentgateway renders as two
releases: the Helm chart, which installs the controller and the GatewayClass,
and agentgateway-post, which applies
recipes/components/agentgateway/manifests/inference-gateway.yaml and is what
creates agentgateway-system/inference-gateway. The chart alone creates no
Gateway, so an install or upgrade that runs only the helm upgrade leaves the
inference-gateway check with nothing to read. aicr validate --phase deployment catches the result: the agentgateway health check asserts that
agentgateway-system/inference-gateway exists with Accepted=True and
Programmed=True, so a cluster with no Gateway, or an unprogrammed one, fails
the deployment phase (#3714).
The check detects a missing or unprogrammed Gateway, not which deployment
created it; a healthy Gateway left by an earlier deployment satisfies it.
On an LB-less tenant, keep the step rather than skipping it, and generate the
bundle with --set agentgateway:serviceType=ClusterIP. The controller then
programs the Gateway onto a ClusterIP Service, the shape the NVKE GB200
cluster runs, and the conformance check records “No LoadBalancer Service”.
The knob is not what makes the health check pass: as noted above, agentgateway
v1.5.0 reports Programmed=True without an address. It avoids a Service that
sits at EXTERNAL-IP <pending> forever and a LoadBalancer-only field the
cluster cannot honor; with ClusterIP the rendered Service carries no
loadBalancerSourceRanges. Never edit the generated bundle to get there;
aicr verify checksums every file in it.
Evidence
Collected on a two-worker VR200 tenant cluster with no LoadBalancer present,
with ai_inference PASS and Programmed=True. The set is pending in
#3695 and is being recollected
against the current collector, so no result count is quoted here.
cluster_autoscaling skips on bare metal. #3695 also records two narrower
results — robust_controller was credited to an operator this recipe does not
declare, which the collector it ran under allowed and the current one does not,
and dra_support passed without its MNNVL behavioral subtest because no pod
held the ComputeDomain channel at collection time.
The workload leaves
vr200-nvke-ubuntu-training-kubeflow and vr200-nvke-ubuntu-inference-dynamo
exist so this coordinate can establish robust_controller, a v1.36 MUST. The
check takes its subject from the recipe — kubeflow-trainer, else
dynamo-platform, else skip — and the two plain leaves declare no AI operator,
so a submission from either stops short of the full MUST set. Each workload
leaf adds the operator and declares the check, following
gb200-nvke-ubuntu-training-kubeflow and
vr200-rke2-ubuntu-inference-dynamo.
-training-kubeflowtakes theplatform-kubeflowmixin, which brings Kubeflow Trainer and the torch-distributedClusterTrainingRuntimethe check reconciles aTrainJobagainst. It adds nocomponentRefsof its own.-inference-dynamoaddsdynamo-platformandgroveto the inference leaf — the serving platform the base deliberately leaves out.
Both establish the requirement. Measured on 2026-11-07 against a VR200 NVKE
tenant cluster deployed from the -inference-dynamo bundle, the Dynamo path
reports robust_controller: Implemented with no documented entry configured,
on operator pods, an Established CRD and a webhook that rejects an invalid
custom resource. Only the CR-reconciliation extra is observation-only there,
because a DynamoGraphDeployment cannot reach a terminal condition without a
model and GPU workers; the Kubeflow path adds a TrainJob reconciled to
Complete on top of the same base.
Neither carries a performance block. Nothing has been measured on VR200 under Dynamo, and a throughput or TTFT floor copied from another accelerator would be a claim about hardware these leaves have never run on.
What has been measured, and what has not
Measured. NCCL all_reduce_perf over MNNVL across two NVL72 trays, 8 GPUs,
1 GiB..16 GiB: 996 GB/s average, 1040 GB/s peak, zero mismatches, with NCCL
reporting one NVLS communicator spanning both nodes (nvlsRanks 8).
This is a single NVLink domain. The number says nothing about scaling beyond one domain, and it is two trays of an NVL72, not a full-rack figure.
The NIC RDMA east-west path is unqualified. The NICs are ConnectX-9, but on the trays this was
developed against every mlx5 port reports link_layer Ethernet with RoCE v2
GIDs and none reports InfiniBand, so the SR-IOV InfiniBand stack that
gb200-nvke-ubuntu-training ships does not apply. No RDMA transport number has
been measured on the Ethernet path. What the recipe declares instead is RoCE:
a SriovNetworkNodePolicy and a SriovNetwork per rail with the SR-IOV
operator on, deployed and attachable but carrying no traffic — see below.
Host NIC stack. The node image installs DOCA OVS and the DOCA-repo ibverbs
userspace, but the kernel driver is the distribution’s inbox mlx5_core, not
DOCA-OFED. Feature parity and qualified RDMA performance with DOCA-OFED are not
claimed. Measured on both workers of the reference cluster, identical on each:
- Kernel driver: inbox
mlx5_core, vermagic7.0.0-2022-nvidia-bos-64k.modinforeports noversion:field, which is how an in-tree module presents, andethtool -iechoes the kernel version rather than an OFED release. - Kernel:
linux-image-nvidia-bos-64k7.0.0-2022.23, Ubuntu 26.04.1 LTS. - RDMA userspace:
libibverbs1andibverbs-providersboth 2607.0.8-1. Neitherrdma-corenorlibrdmacmis installed. - NIC firmware: 82.48.4222 (PSID MT_0000001658) on the eight ConnectX-9 ports, 82.48.4238 (MT_0000001875) on the BlueField ports. This baseline predates the current stable line. NVIDIA’s published ConnectX-9 firmware runs 82.48.1000 (February 2026 GA), 82.49.1014 (June 2026) and 82.50.1002, so 82.50 is current and 82.48 is two branches behind it. The build number here (4222) sits far above the 82.48 GA build (1000), which is the shape of a pre-GA build for these trays rather than a stale GA, so confirm applicability before treating 82.50 as a drop-in upgrade. This recipe sets no firmware: there is no firmware template in the NicClusterPolicy, so the version is whatever the node image and tray ship.
ofed_infois absent, confirming no DOCA-OFED install.
That is the fixed baseline the unqualified label above is measured against.
A NicClusterPolicy is deployed, carrying nicConfigurationOperator, nvIpam and secondaryNetwork.
It stands up the nic-configuration-daemon that aicr snapshot --discover-network needs, plus the addressing and multus the RoCE rails
attach through. Still no ofedDriver: NVKE runs the host’s inbox mlx5_core.
Eight ConnectX-9 RoCE rails: deployed and attachable, but carrying no
traffic. The leaf ships an nv-ipam IPPool, a SriovNetworkNodePolicy and a
SriovNetwork per rail, and a nccl-all-reduce-bw-net check to exercise them.
On the reference cluster the operator created one VF per port on both workers
and advertised all eight as allocatable, nv-ipam assigned an address per rail,
multus attached net1..net8, and each VF came up with its own RDMA device
and a RoCE v2 GID at index 1. No packet has crossed between nodes: the spare
ConnectX-9 ports have no path between workers, because the BlueField DPU
bridges only the port carrying the node address, and bridging the rest is
DPU-side work outside the tenant cluster. So the >= 50 floor is a
placeholder, and the only collective path proven here is MNNVL inside one
NVLink domain.
These SriovNetworks set no VF attributes, deliberately. GB200’s set
linkState: enable; on this platform the PF driver refuses it, and trust and
spoofChk too, failing the attachment with operation not permitted. The PF
is owned by the BlueField DPU, so the host can create VFs but cannot
administer them. With all three omitted the attachment succeeds.
NCCL_IB_GID_INDEX must be set. The RoCE v2 GID sits at index 1 on these
ports; NCCL defaults to 0, which holds the v1 GID.
numVfs is 1, which is the ConnectX-9 firmware’s sriov_totalvfs on these
trays rather than a tuning choice — eight ports give eight VFs, so one worker
pod takes them all.
If the rails are ever removed, clear the multus CNI config too. Deleting a
NicClusterPolicy carrying secondaryNetwork removes multus’s DaemonSet and
RBAC but leaves /etc/cni/net.d/00-multus.conflist on every node, which sorts
ahead of OVN-Kubernetes and fails every subsequent pod creation cluster-wide
with Unauthorized. This was observed on this cluster.
It pins network-operator 26.7.1, not the registry default of 26.7.0. The
daemon in 26.7.0 has no firmware-discovery fallback, so a flint failure is
fatal — and flint cannot read ConnectX-9 here. Measured on a worker: 26.7.0
exits with failed to discover NIC devices, while 26.7.1 logs flint failed, falling back to sysfs and discovers every port. The fix is Mellanox
c27f7ae
(#499/#533), which is absent from the ghcr.io/mellanox v1.3.1 images the
other in-tree policies pin, so this leaf pins the nvcr.io/nvidia/mellanox
network-operator-v26.7.1 images instead. ConnectX-7 and ConnectX-8 platforms
never hit this, because flint works there.
Verified on the reference cluster: the daemon discovered all eight ConnectX-9
ports and wrote a NicDevice per port, each carrying firmware 82.48.4222,
PSID mt_0000001658, and its pci/netdev/rdma mapping.
Not verified end to end. The compiled-validator path for this coordinate is exercised by CI on merge, not on hardware here.
Withholding the network check
nccl-all-reduce-bw-net runs on this leaf: vr200/nvke is in the compiled
supportedNCCLCombinations[variantNET] matrix and the recipe declares the
check. On a tenant where the DPU bridges only the port carrying the node
address, the rails have no path between workers, so the check fails the
performance phase. Withhold it on those tenants:
Keep it withheld until the DPU bridges the spare ConnectX-9 ports.
Use the flag rather than dropping the constraint from the recipe. The flag records a limitation of one run; editing the recipe changes the check for everyone who consumes this leaf.
One consequence: --skip-check cannot be combined with --evidence-dir, and
so not with --cncf-submission, because a run that withholds a check the
recipe declares is narrower than the recipe and cannot evidence it. A run that
withholds this check therefore produces no conformance evidence for this
recipe. The ordinary CTRF report still records the check as skipped, and so
does a signed recipe attestation (--emit-attestation), which carries the
named-in-skip-checks reason code through bundle redaction.
Known gaps
- ConnectX-9 qualification and an SR-IOV/RDMA east-west fabric for multi-domain training.
- A kernel-flavour measurement so
nvidia-bos-64kcan be a constraint. - A stable VR200 product matcher; until then, mixed GPU pools need
--node-selector.