GKE A4X Max (GB300) Networking Prerequisites

For the GB300 GKE COS recipes (gb300-gke-cos-training, gb300-gke-cos-training-kubeflow, gb300-gke-cos-inference, and gb300-gke-cos-inference-dynamo, all on a4x-maxgpu-4g-metal bare-metal nodes), inter-node GPU traffic runs over GPUDirect RDMA on RoCE, and intra-rack GPU traffic runs over Multi-Node NVLink (MNNVL) through an IMEX domain provisioned as a DRA ComputeDomain.

A4X Max is not A4X. Both are Grace-Blackwell NVL72 racks, but the GKE networking path is different, and the two are not interchangeable:

The A4X column shows the multi-networking path used by Google’s A4X guide and AICR’s GB200 recipes; A4X also supports GKE managed DRANET. A4X Max has no such choice: multi-networking isn’t supported for the a4x-maxgpu-4g-metal machine type, so none of the Network/GKENetworkParamSet setup described in GKE GB200 Networking applies here. Follow this page instead.

These steps summarize the prerequisites AICR depends on. They are not a complete provisioning runbook — follow Google’s Create an AI-optimized GKE cluster that uses A4X Max guide for the full procedure, including cluster creation, the workload policy, and reservation affinity.

What AICR installs, and what it doesn’t

AICR’s GB300 GKE bundle installs the in-cluster GPU stack. Everything in the second column is a workload prerequisite that you provision: RDMA and MNNVL workloads cannot run without it, but aicr validate does not check most of it.

Because AICR already deploys nvidia-dra-driver-gpu, do not also run the ResourceQuota and helm install nvidia-dra-driver-gpu steps from Google’s guide on a cluster you intend to manage with AICR. Two installs of the same driver contend for the same CRDs, and the quota is scoped to the system-node-critical priority class, which AICR’s install doesn’t use.

Of the prerequisites in the second column, aicr validate needs only the GPU nodes and a control plane that meets the GKE version floor. It can pass without the node pool’s networking flags, asapd-lite, managed DRANET, or any workload ComputeDomain and ResourceClaimTemplate. The dra-support check verifies the NVIDIA DRA driver and counts only *.nvidia.com ResourceSlices, so it never sees the RDMA devices behind the mrdma.google.com DeviceClass. On nodes labeled nvidia.com/gpu.clique it also tests IMEX channel allocation, using a temporary ComputeDomain of its own. The GB300 GKE recipes run no performance check, so nothing exercises inter-node RDMA. Confirm the networking prerequisites with the commands in Verifying.

GKE version floor

All four GB300 GKE recipes carry this constraint:

constraints:
  - name: K8s.server.version
    value: ">= 1.34.3-gke.1318000 < 1.35 || >= 1.35.0-gke.2745000"

The two alternatives are the per-branch minimums published in the A4X Max requirements: on the 1.34 branch use 1.34.3-gke.1318000 or later, and on 1.35 or later use 1.35.0-gke.2745000 or later. A single >= 1.34.3-gke.1318000 would wrongly admit early 1.35 patches, so the floor is expressed as two alternatives rather than one range.

Google’s requirements say these versions help ensure that A4X Max uses GPU driver R580.95.05 (the A4X Max minimum) and Coherent Driver-based Memory Management (CDMM), both enabled by default, as well as GPUDirect RDMA and MNNVL. aicr validate fails readiness on an older control plane, naming this constraint.

CDMM is on by default on these versions and is incompatible with Multi-Instance GPU. Don’t plan MIG partitioning on A4X Max.

Infrastructure prerequisites

• GKE Dataplane V2, enabled at cluster creation. Managed DRANET and accelerator network profiles both require it, and it can’t be enabled on an existing cluster. Google’s cluster-create command also disables shielded nodes (--no-enable-shielded-nodes).
• Standard cluster, fixed-size node pool. Bare-metal nodes such as A4X Max are incompatible with Autopilot clusters and with autoscaling.
• Container-Optimized OS only. Ubuntu and Windows node images are not supported on a4x-maxgpu-4g-metal.
• Reservation-bound provisioning. Other provisioning models are not supported. Google recommends a specific, sub-block-targeted reservation, as in the command below.
• A HIGH_THROUGHPUT workload policy with accelerator-topology 1x72, so the node pool lands in one NVLink domain.
• Whole-node RDMA. A Pod must request all four GPUs and all of the node’s secondary NICs. RDMA cannot be shared between Pods on one node. AICR doesn’t add these requests for you: each workload must specify its own four-GPU limit and RDMA resource claim, including TrainJobs that use the bundled torch-distributed runtime.
• Hugepages pre-allocated on the node pool.

Creating the node pool

This command creates a pool for the default gpuStack=gke-default profile: GKE installs the driver (gpu-driver-version=latest), and GKE’s managed device plugin advertises nvidia.com/gpu. The networking-relevant flags are --accelerator-network-profile=auto and the two node labels:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=COMPUTE_REGION \
    --node-locations=COMPUTE_ZONE \
    --num-nodes=NODE_COUNT \
    --placement-policy=WORKLOAD_POLICY_NAME \
    --machine-type=a4x-maxgpu-4g-metal \
    --accelerator=type=nvidia-gb300,count=4,gpu-driver-version=latest \
    --system-config-from-file=node_custom.yaml \
    --accelerator-network-profile=auto \
    --node-labels=cloud.google.com/gke-networking-dra-driver=true,cloud.google.com/gke-dpv2-unified-cni=cni-migration \
    --reservation-affinity=specific \
    --reservation=RESERVATION_NAME/reservationBlocks/BLOCK_NAME/reservationSubBlocks/SUB_BLOCK_NAME

NODE_COUNT must be 18 or fewer; 18 gives the full 1x72 GPU topology in a single sub-block. node_custom.yaml pre-allocates hugepages:

linuxConfig:
  hugepageConfig:
    hugepage_size2m: 4096

--accelerator-network-profile=auto builds the accelerator network for you and labels the nodes gke.networks.io/accelerator-network-profile: auto. It requires the GKE service account in your project to have the compute.networkAdmin role. Google’s guide requires workloads to include the profile label in their nodeSelector; the Pod example below does.

cloud.google.com/gke-networking-dra-driver=true is what turns on GKE managed DRANET for the pool. Without it, no mrdma.google.com devices are published and every RDMA resource claim stays pending.

The gpuStack=bundle-installer profile, in which the bundle installs the driver and the GPU Operator’s device plugin advertises nvidia.com/gpu, needs a differently created pool: set gpu-driver-version=disabled in --accelerator, and append gke-no-default-nvidia-gpu-device-plugin=true to --node-labels. It also needs the pool list passed to aicr snapshot or aicr validate with --gke-gpu-pools. A pool created with the command above fails both of that profile’s constraints, so choose the profile before you create the pool. See GKE GPU Setup › GPU Device-Plugin Ownership.

Installing asapd-lite

The asapd-lite DaemonSet configures the MRDMA NICs. Without it, child interfaces aren’t created and DRANET filters out the RDMA NICs, so Pods get no RDMA.

The command pins the GoogleCloudPlatform/container-engine-accelerators commit validated with these recipes (asapd-lite v0.0.9). Google’s A4X Max guide applies the installer from master, which can be newer.

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/fb61c784c87e7de0cd68c18db14d62dea892ada5/asapd-lite-installer/asapd-lite-installer-a4x-max-bm-cos.yaml
kubectl get daemonset -n kube-system asapd-lite

READY should equal DESIRED, which counts every A4X Max node in the cluster, not only this pool. For one 18-node pool:

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
asapd-lite   18        18        18      18           18          <none>          5m

This DaemonSet is not part of the AICR bundle.

Verifying

kubectl get deviceclasses.resource.k8s.io mrdma.google.com
kubectl get resourceslices.resource.k8s.io \
    -o custom-columns=NODE:.spec.nodeName,DRIVER:.spec.driver

Expect the mrdma.google.com DeviceClass (GKE installs the networking DeviceClasses automatically on supported versions) and ResourceSlice objects publishing RDMA devices for each A4X Max node. On an AICR cluster the list also includes *.nvidia.com slices from AICR’s NVIDIA DRA driver, so a non-empty list doesn’t mean RDMA is available: use the DRIVER column to find the DRANET slices for each node. If an A4X Max node has none, or RDMA resource claims stay pending, recheck the prerequisites on this page:

• asapd-lite is Ready on the node.
• The node has the cloud.google.com/gke-networking-dra-driver=true label.
• The node pool was created with --accelerator-network-profile=auto.
• The cluster was created with GKE Dataplane V2.

AICR doesn’t test RDMA on these recipes. For an end-to-end check, run Google’s NCCL/gIB test: Run NCCL on GKE clusters that use A4X Max.

Workload wiring

AICR installs the NVIDIA DRA driver, but the ComputeDomain and the RDMA ResourceClaimTemplate are per-workload objects that the workload author creates. The ComputeDomain is required for MNNVL (IMEX channels), and the RDMA claim for GPUDirect RDMA.

A ComputeDomain provisions the IMEX channel for MNNVL:

apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: a4x-max-compute-domain
spec:
  numNodes: NUM_NODES
  channel:
    resourceClaimTemplate:
      name: a4x-max-compute-domain-channel

A ResourceClaimTemplate requests the node’s RDMA NICs through DRANET:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: all-mrdma
spec:
  spec:
    devices:
      requests:
        - name: req-mrdma
          exactly:
            deviceClassName: mrdma.google.com
            allocationMode: ExactCount
            count: 8

The Pod then claims both, requests all four GPUs, pins itself to arm64 nodes in the accelerator network profile pool, and mounts the host driver directory:

spec:
  nodeSelector:
    gke.networks.io/accelerator-network-profile: auto
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - arm64
  volumes:
    - name: library-dir-host
      hostPath:
        path: /home/kubernetes/bin/nvidia
  containers:
    - name: my-container
      volumeMounts:
        - name: library-dir-host
          mountPath: /usr/local/nvidia
      env:
        - name: LD_LIBRARY_PATH
          value: /usr/local/nvidia/lib64
      resources:
        limits:
          nvidia.com/gpu: 4
        claims:
          - name: compute-domain-channel
          - name: rdma
  resourceClaims:
    - name: compute-domain-channel
      resourceClaimTemplateName: a4x-max-compute-domain-channel
    - name: rdma
      resourceClaimTemplateName: all-mrdma

The arm64 affinity and the /home/kubernetes/bin/nvidia host mount are not optional: A4X Max nodes are Grace (ARM64), and on COS the NVIDIA GPU driver and its userspace libraries are host-managed rather than shipped in the container.

The RDMA userspace is the exception. Unlike A4X, Google’s A4X Max guide installs it in the container image rather than on the host: build your image with the doca-ofed-userspace package (arm64-sbsa build) and the latest NCCL (libnccl2, libnccl-dev). See Configure your workload manifest for RDMA and IMEX domain for the installation steps.

References

• Create an AI-optimized GKE cluster that uses A4X Max
• Allocate network resources by using GKE managed DRANET
• Configure automated networking for accelerator VMs
• NVIDIA DRA Driver for GPUs