Single-Plane Spectrum-X Quick Start
You can automate the configuration of this use case with NVIDIA Kubernetes Launch Kit. For more details, see Configuration Assistance with Kubernetes Launch Kit.
This walkthrough deploys a Single-Plane Spectrum-X cluster on Kubernetes:
one PF per rail, one CIDRPool per rail, one network per rail. Used on
HGX H100/H200/B200 platforms (BlueField-3 SuperNIC, nicType: a2dc) and
GB200 NVL72 platforms (ConnectX-7 NIC, nicType: 1021).
ConnectX-8 SuperNIC (nicType: 1023) also supports single-plane configuration —
useful if you want a single-plane setup on B300/GB300 hardware. The
configuration uses RA 2.2 with multiplaneMode: none and
numberOfPlanes: 1. The example below uses BlueField-3 SuperNIC; change
nicType for other NICs. Replace TODO_* values with your
cluster-specific values before applying.
Add the NVIDIA NGC Helm repository:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
Install the Network Operator. Spectrum-X Operator and NIC Configuration Operator are deployed via the same chart and enabled later through NicClusterPolicy.
helm install network-operator nvidia/network-operator \
-n nvidia-network-operator \
--create-namespace \
--version 26.4.0 \
--set sriovNetworkOperator.enabled=true \
--wait
Verify the installation:
kubectl -n nvidia-network-operator get pods
Enable the NIC Configuration Operator, NV-IPAM, Spectrum-X Operator (with XPlane), and the secondary network components.
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
name: nic-cluster-policy
spec:
nicConfigurationOperator:
operator:
image: nic-configuration-operator
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
configurationDaemon:
image: nic-configuration-operator-daemon
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
nicFirmwareStorage:
create: true
pvcName: nic-fw-storage-pvc
storageClassName: nic-fw-storage-class
availableStorageSize: 1Gi
logLevel: info
nvIpam:
image: nvidia-k8s-ipam
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
enableWebhook: false
spectrumXOperator:
image: spectrum-x-operator
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
# xPlane is only used when multiplaneMode=hwplb (Hardware Multiplane).
# Including it here lets you flip multiplaneMode without re-applying NicClusterPolicy.
xPlane:
image: xplane
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
secondaryNetwork:
cniPlugins:
image: plugins
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
multus:
image: multus-cni
repository: nvcr.io/nvidia/cloud-native
version: network-operator-v26.4.0
kubectl apply -f nicclusterpolicy.yaml
Map PCI addresses to rails and define interface naming. With single-plane configuration there is one PF per NIC, so pfsPerNic is 1 and %plane_id% is always 0. Replace TODO_PCI_* with the PCI addresses of the BlueField-3 SuperNICs on your nodes.
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
name: spectrum-x-interface-names
namespace: nvidia-network-operator
spec:
pfsPerNic: 1
rdmaDevicePrefix: "rdma_rail%rail_id%"
netDevicePrefix: "net_rail%rail_id%"
railPciAddresses:
- ["TODO_PCI_RAIL0_NIC0"]
- ["TODO_PCI_RAIL1_NIC0"]
kubectl apply -f nicinterfacenametemplate.yaml
Configure the NICs for Spectrum-X RA 2.2 in single-plane mode. Use nicType: a2dc for BlueField-3 SuperNIC (HGX H100/H200/B200), nicType: 1021 for ConnectX-7 NIC (GB200), or nicType: 1023 for ConnectX-8 SuperNIC.
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
name: spectrum-x-configuration
namespace: nvidia-network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/network-sriov.capable: "true"
nicSelector:
nicType: "a2dc" # BlueField-3 SuperNIC (HGX H100/H200/B200). Use "1021" for ConnectX-7 NIC (GB200) or "1023" for ConnectX-8 SuperNIC.
template:
numVfs: 1
linkType: Ethernet
spectrumXOptimized:
enabled: true
version: "RA2.2"
overlay: "none"
multiplaneMode: "none"
numberOfPlanes: 1
kubectl apply -f nicconfigurationtemplate.yaml
In single-plane mode each rail is a single network, so one CIDRPool per rail is sufficient. Replace TODO_* with subnets that match your cluster’s east-west topology.
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-0
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL0_CIDR # e.g., 10.0.0.0/15
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL0_SUBNET # same as cidr
- dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
name: rail-1
namespace: nvidia-network-operator
spec:
cidr: TODO_RAIL1_CIDR
gatewayIndex: 0
perNodeNetworkPrefix: 31
perNodeExclusions:
- startIndex: 1
endIndex: 1
routes:
- dst: TODO_RAIL1_SUBNET
- dst: TODO_EAST_WEST_SUBNET
kubectl apply -f cidrpool.yaml
One entry in railTopology per rail, selecting the single PF in that rail (matching NicInterfaceNameTemplate) and referencing the matching CIDRPool.
apiVersion: spectrumx.nvidia.com/v1alpha2
kind: SpectrumXRailPoolConfig
metadata:
name: rails
namespace: nvidia-network-operator
spec:
draEnabled: true
networkNamespace: default
numVfs: 1
railTopology:
- name: rail0
nicSelector:
pfNames: ["net_rail0"]
cidrPoolRef: rail-0
mtu: 9216
- name: rail1
nicSelector:
pfNames: ["net_rail1"]
cidrPoolRef: rail-1
mtu: 9216
kubectl apply -f spectrumxrailpoolconfig.yaml
Request one VF per rail. The network annotation references the rails created by SpectrumXRailPoolConfig.
apiVersion: v1
kind: Pod
metadata:
name: spectrum-x-test
namespace: default
annotations:
k8s.v1.cni.cncf.io/networks: rail0,rail1
spec:
containers:
- name: spectrum-x-test
image: nvcr.io/nvidia/doca/doca:3.3.0-full-rt-host
command: ["/bin/bash", "-c", "sleepinfinity"]
securityContext:
capabilities:
add: ["IPC_LOCK", "NET_RAW"]
resources:
requests:
nvidia.com/rail_0: "1"
nvidia.com/rail_1: "1"
limits:
nvidia.com/rail_0: "1"
nvidia.com/rail_1: "1"
kubectl apply -f pod.yaml
kubectl -n default exec -it spectrum-x-test -- rdma link