Software Multiplane Spectrum-X Quick Start
You can automate the configuration of this use case with NVIDIA Kubernetes Launch Kit. For more details, see Configuration Assistance with Kubernetes Launch Kit.
This walkthrough deploys a Software Multiplane Spectrum-X cluster on
Kubernetes using ConnectX-8 SuperNICs (nicType: 1023). Each SuperNIC
is split into multiple PFs, each assigned to a separate plane, and the
software stack performs Software Plane Load Balancing (swplb) across
them. Used on B300 and GB300 platforms — set numberOfPlanes: 2
for Dual-Plane or numberOfPlanes: 4 for Quad-Plane (B300 only). The
configuration uses RA 2.2 with multiplaneMode: swplb. The example below
uses numberOfPlanes: 2. Replace TODO_* values with your
cluster-specific values before applying.
Add the NVIDIA NGC Helm repository:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
Install the Network Operator. Spectrum-X Operator and NIC Configuration Operator are deployed via the same chart and enabled later through NicClusterPolicy.
helm install network-operator nvidia/network-operator \
-n nvidia-network-operator \
--create-namespace \
--version 26.4.0 \
--set sriovNetworkOperator.enabled=true \
--wait
Verify the installation:
kubectl -n nvidia-network-operator get pods
Enable the NIC Configuration Operator, NV-IPAM, Spectrum-X Operator (with XPlane), and the secondary network components.
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
name: nic-cluster-policy
spec:
nicConfigurationOperator:
operator:
image: nic-configuration-operator
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
configurationDaemon:
image: nic-configuration-operator-daemon
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
nicFirmwareStorage:
create: true
pvcName: nic-fw-storage-pvc
storageClassName: nic-fw-storage-class
availableStorageSize: 1Gi
logLevel: info
nvIpam:
image: nvidia-k8s-ipam
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
enableWebhook: false
spectrumXOperator:
image: spectrum-x-operator
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
# xPlane is only used when multiplaneMode=hwplb (Hardware Multiplane).
# Including it here lets you flip multiplaneMode without re-applying NicClusterPolicy.
xPlane:
image: xplane
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
secondaryNetwork:
cniPlugins:
image: plugins
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
multus:
image: multus-cni
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
kubectl apply -f nicclusterpolicy.yaml
Map PCI addresses to rail/plane indices and define interface naming. Replace TODO_PCI_* with the PCI addresses of the Spectrum-X NICs on your nodes.
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
name: spectrum-x-interface-names
namespace: nvidia-network-operator
spec:
pfsPerNic: 2
rdmaDevicePrefix: "rdma_rail%rail_id%_plane%plane_id%"
netDevicePrefix: "net_rail%rail_id%_plane%plane_id%"
railPciAddresses:
- ["TODO_PCI_RAIL0_NIC0", "TODO_PCI_RAIL0_NIC1"]
- ["TODO_PCI_RAIL1_NIC0", "TODO_PCI_RAIL1_NIC1"]
kubectl apply -f nicinterfacenametemplate.yaml
Configure the ConnectX-8 SuperNICs for Spectrum-X RA 2.2 with swplb multiplane mode. For Quad-Plane (B300 only), set numberOfPlanes: 4.
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
name: spectrum-x-configuration
namespace: nvidia-network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/network-sriov.capable: "true"
nicSelector:
nicType: "1023" # ConnectX-8 SuperNIC (B300, GB300)
template:
numVfs: 1
linkType: Ethernet
spectrumXOptimized:
enabled: true
version: "RA2.2"
overlay: "none"
multiplaneMode: "swplb"
numberOfPlanes: 2
kubectl apply -f nicconfigurationtemplate.yaml
With swplb, each rail is split into multiple planes, and each plane requires its own CIDRPool. Create one pool per (rail, plane) combination. Replace TODO_* with subnets that match your cluster’s east-west topology.
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-0-plane-0
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL0_PLANE0_CIDR # e.g., 10.0.0.0/15
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL0_PLANE0_SUBNET # same as cidr
- dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-0-plane-1
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL0_PLANE1_CIDR
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL0_PLANE1_SUBNET
- dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-1-plane-0
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL1_PLANE0_CIDR
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL1_PLANE0_SUBNET
- dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-1-plane-1
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL1_PLANE1_CIDR
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL1_PLANE1_SUBNET
- dst: TODO_EAST_WEST_SUBNET
kubectl apply -f cidrpool.yaml
With swplb, railTopology has one entry per (rail, plane) combination. Each entry selects a single PF (netdev name from NicInterfaceNameTemplate) and references the matching per-plane CIDRPool.
apiVersion: spectrumx.nvidia.com/v1alpha2
kind: SpectrumXRailPoolConfig
metadata:
name: rails
namespace: nvidia-network-operator
spec:
draEnabled: true
networkNamespace: default
numVfs: 1
railTopology:
- name: rail0p0
nicSelector:
pfNames: ["net_rail0_plane0"]
cidrPoolRef: rail-0-plane-0
mtu: 9216
- name: rail0p1
nicSelector:
pfNames: ["net_rail0_plane1"]
cidrPoolRef: rail-0-plane-1
mtu: 9216
- name: rail1p0
nicSelector:
pfNames: ["net_rail1_plane0"]
cidrPoolRef: rail-1-plane-0
mtu: 9216
- name: rail1p1
nicSelector:
pfNames: ["net_rail1_plane1"]
cidrPoolRef: rail-1-plane-1
mtu: 9216
kubectl apply -f spectrumxrailpoolconfig.yaml
Request one VF per (rail, plane) combination. The network annotation lists each rail-plane and the resource request matches the corresponding DRA resource.
apiVersion: v1
kind: Pod
metadata:
name: spectrum-x-test
namespace: default
annotations:
k8s.v1.cni.cncf.io/networks: rail0p0,rail0p1,rail1p0,rail1p1
spec:
containers:
- name: spectrum-x-test
image: nvcr.io/nvidia/doca/doca:3.3.0-full-rt-host
command: ["/bin/bash", "-c", "sleepinfinity"]
securityContext:
capabilities:
add: ["IPC_LOCK", "NET_RAW"]
resources:
requests:
nvidia.com/rail_0_plane_0: "1"
nvidia.com/rail_0_plane_1: "1"
nvidia.com/rail_1_plane_0: "1"
nvidia.com/rail_1_plane_1: "1"
limits:
nvidia.com/rail_0_plane_0: "1"
nvidia.com/rail_0_plane_1: "1"
nvidia.com/rail_1_plane_0: "1"
nvidia.com/rail_1_plane_1: "1"
kubectl apply -f pod.yaml
kubectl -n default exec -it spectrum-x-test -- rdma link