Spectrum-X Kubernetes Architecture and Components
A Spectrum-X Kubernetes deployment is a layered stack split across two distinct domains:
Host Layer — operating-system prerequisites, firmware settings, kernel drivers, and the host virtual switch. Owned by an automated provisioning stack (or by manual / scripted operator action) and out of Network Operator scope.
Kubernetes Layer — Kubernetes-native operators, CNIs, IPAM, and discovery components. Deployed and reconciled by NVIDIA Network Operator on top of a working Kubernetes cluster.
This page maps each component to its role, dependencies, and default-enabled state in Network Operator 26.7.0. For exact image versions, see the software components table in Platform Support. For end-to-end walkthroughs, see Spectrum-X Kubernetes Quick Start.
Spectrum-X K8s deployments are split across three phases. Host-side provisioning (Day 0) is owned outside Network Operator scope; Day 1 and Day 2 are reconciled by Network Operator.
Phase | Scope | Owner |
|---|---|---|
| Day 0 (pre-K8s) | Host OS bootstrap, PCIe / SR-IOV firmware, switch fabric, SuperNIC firmware floor, hugepages, RDMA exclusive mode, DOCA-Host / OVS-DOCA install. | Automated provisioning stack or manual / scripted. |
| Day 1 (cluster bring-up) | Driver, operator, and CNI deployment on top of a working Kubernetes cluster; Spectrum-X CRD reconciliation. | NVIDIA Network Operator. |
| Day 2 (workload) | Pod connectivity (secondary networks), rail / plane IP allocation, VF / representor lifecycle. | NVIDIA Network Operator (CNI + IPAM). |
Host Layer (Day 0 — outside Network Operator scope) — operating-system prerequisites, the DOCA-Host driver, and the OVS-DOCA virtual switch.
Kubernetes Layer (Day 1 / Day 2 — owned by Network Operator) — node discovery, the containerized driver, the operator stack, the CNI and data-plane components, and the Spectrum-X profile ConfigMap that feeds NIC configuration.
Each layer is broken out in the sections below.
The Host Layer is the foundation: operating-system prerequisites, firmware settings, kernel drivers, and the host virtual switch that exist on each node before Kubernetes is installed. These components are typically owned by an automated provisioning stack or by manual / scripted operator action, and are not managed by Network Operator.
OS prerequisites
Component | Role |
|---|---|
| SR-IOV enabled in BIOS/UEFI | Platform firmware setting that permits Virtual Function creation.
The NIC-side nvconfig (SRIOV_EN, SRIOV_NUM_OF_VFS, and the
multiplane parameters) is not set here — the NIC Configuration
Operator derives it from numVfs and the Spectrum-X profile. See
Spectrum-X NIC Configuration. |
| Hugepages | Memory backing for OVS-DOCA: 400 MB per SuperNIC from DOCA 3.4
onwards (1 GB on earlier DOCA), reserved on the kernel command
line. Hardware Multiplane needs a second reservation for DOCA
xPlane — at least 150 × 2 MB pages per SuperNIC, provisioned with
doca-hugepages config --app doca-xplane. |
RDMA namespace exclusive mode
(options ib_core netns_mode=0) |
Per-namespace RDMA device isolation, required to assign RDMA devices into pod network namespaces. |
Driver
Component | Role |
|---|---|
| DOCA-Host | Host-installed NVIDIA OFED kernel modules and DOCA tooling. Includes OVS-DOCA (see Virtual switch below). Default driver path for Spectrum-X RA 2.3 validated deployments. |
DOCA-Host (host-installed) and the containerized DOCA-OFED driver (deployed via Network Operator — see Driver (containerized)) are mutually exclusive. Choose one per node; do not deploy both. Spectrum-X RA 2.3 validated deployments typically use host-installed DOCA-Host.
Virtual switch
Component | Role |
|---|---|
| OVS-DOCA | Hardware-accelerated Open vSwitch, bundled with DOCA-Host. Provides the per-rail OVS bridges into which the Kubernetes Layer’s OVS-CNI plugs SR-IOV VFs at pod creation time. |
The Kubernetes Layer is what NVIDIA Network Operator deploys and reconciles on top of a working Kubernetes cluster. It includes Kubernetes-native operators, CNIs, IPAM, and discovery components. Configuration is driven through the Spectrum-X CRDs documented in Spectrum-X CRDs and API Reference.
Discovery
Component | Role | Default |
|---|---|---|
| Node Feature Discovery (NFD) | Labels nodes with PCI / RDMA / GPU features so SR-IOV Network Operator and the scheduler can target the right hosts. | On by default. |
| NIC Feature Discovery | NFD extension that exposes NIC-specific capabilities (link type, device IDs) as node labels. | Opt-in. |
Driver (containerized)
Component | Role | Default | Spectrum-X relevance |
|---|---|---|---|
DOCA-OFED driver container (doca-driver) |
Containerized NVIDIA DOCA-OFED kernel driver, loaded on the node by Network Operator. Mutually exclusive with host-installed DOCA-Host — choose one per node. On immutable operating systems such as Red Hat CoreOS, where host packages cannot be installed, this is the only way to deliver DOCA kernel drivers for ConnectX and BlueField hardware, and it keeps the driver on the same cloud-native lifecycle as the rest of the stack. | Opt-in via ofedDriver on NicClusterPolicy. |
Optional by default; required on immutable hosts. |
Operator
NVIDIA Network Operator is the only networking operator you install.
It is the entry point for all NVIDIA networking on Kubernetes — RDMA and
RoCE, InfiniBand, and Spectrum-X alike — and it owns the
NicClusterPolicy and NicNodePolicy APIs through which everything
else is configured. (GPU workloads additionally use NVIDIA GPU Operator,
which is installed separately.)
Everything in the table below is a sub-operator: packaged in the
Network Operator Helm chart, switched on through NicClusterPolicy,
and version-pinned and reconciled by Network Operator. None of them is
installed, upgraded, or configured on its own — Spectrum-X is enabled by
turning the right sub-operators on, not by adding another operator
alongside Network Operator.
Sub-operator | Role | Default | Spectrum-X relevance |
|---|---|---|---|
| SR-IOV Network Operator | Owns SriovNetworkNodePolicy and OVSNetwork; manages VF
creation and switchdev mode, and ships the components the
Spectrum-X data path uses — OVS-CNI, RDMA-CNI, the SR-IOV
Device Plugin, and the DRA driver. |
Off by default; on for Spectrum-X. | Essential. |
| NIC Configuration Operator | Owns NicConfigurationTemplate, NicFirmwareTemplate, and
NicInterfaceNameTemplate; applies firmware and rail-name
templates to each SuperNIC. Reads NIC tuning from the Spectrum-X
profile ConfigMap named by spectrumXOptimized.version — see
Spectrum-X NIC Configuration. |
Opt-in. | Recommended. |
| Spectrum-X Operator | Owns SpectrumXRailPoolConfig and deploys the
flow-controller DaemonSet that programs per-rail OVS flows on
each host (writing into the OVS-DOCA bridges from the Host Layer).
Essential for any Spectrum-X mode (single-plane, swplb,
hwplb). |
Opt-in via spectrumXOperator. |
Essential. |
| Maintenance Operator | Orchestrates node-maintenance windows (drain / cordon) for safe firmware updates and OFED driver upgrades. | Opt-in. | Recommended. Enabled in the Spectrum-X reference values
(maintenanceOperator.enabled: true). |
CNI / data-plane
The CNI / data-plane sub-layer is what actually plugs Virtual Functions into pods and routes traffic across rails and planes. Most components are deployed by the SR-IOV Network Operator sub-chart but are listed individually here because they play distinct roles.
Component | Role | Default | Spectrum-X relevance |
|---|---|---|---|
| Multus CNI | Meta-CNI that allows pods to attach to multiple networks (one per rail / plane). | Opt-in via secondaryNetwork.multus. |
Essential. |
| OVS-CNI | Plugs SR-IOV VFs into the rail’s OVS bridge and chains with NV-IPAM
for address allocation. The bridge itself is created by an operator
— per-PF by the SR-IOV Network Operator for swplb / none,
or br-xplane by the Spectrum-X Operator for hwplb — on top
of the OVS-DOCA switch from the Host
Layer. Shipped under the SR-IOV Network Operator sub-chart. |
With SR-IOV Op. | Essential. |
| SR-IOV Network Device Plugin | Advertises VFs to kubelet as schedulable nvidia.com/...
resources. |
With SR-IOV Op. | Essential. |
| RDMA-CNI | Moves RDMA devices into the pod network namespace (requires RDMA
exclusive mode on the host) and applies the RDMA QoS values the
Spectrum-X Operator sets on the generated OVSNetwork. Every
Spectrum-X rail chains it, so it is not optional in practice. |
With SR-IOV Op. | Essential. |
| NV-IPAM | Rail / plane-aware IP allocation for pods. Consumes CIDRPool
CRDs and assigns IPs to VFs on pod creation. Rails normally point
at a pool with cidrPoolRef; railTopology[].ipam is an
advanced alternative for supplying the NV-IPAM block by hand. |
Opt-in via nvIpam. |
Essential. |
| Spectrum-X flow-controller DaemonSet | Programs OVS flows on each host so packets are forwarded onto the correct rail / plane uplink. Not a separately versioned CNI; deployed by Spectrum-X Operator. | With Spectrum-X Op. | Essential. |
| DOCA xPlane | DaemonSet that manages local and remote plane failover inside OVS-DOCA: consumes route-availability updates from the Spectrum-X fabric, programs the OpenFlow rules that divert traffic away from a failed plane, and exposes gRPC telemetry. Deployed by the Spectrum-X Operator; its image tag follows the DOCA release, not the Network Operator version. Currently requires ConnectX-8 SuperNIC with firmware 40.48.1000 or later. See When DOCA xPlane is deployed. | With Spectrum-X Op. | Required for Hardware Multiplane. |
Optional and tech preview
These components are optional for a Spectrum-X Kubernetes deployment, or are still tech preview.
Component | Role | Status |
|---|---|---|
SR-IOV DRA driver (dra-driver-sriov) |
Dynamic Resource Allocation driver for SR-IOV. Enables fine-grained
VF claims via ResourceClaim and ResourceClaimTemplate
objects, and pairs a VF with the GPU behind the same PCIe root.
Requires Kubernetes 1.34 or later (resource.k8s.io/v1); Spectrum-X RA 2.3 requires
Kubernetes 1.35 or later — see Platform Support. See DRA SR-IOV Driver. |
Tech preview. |
| NicNodePolicy CRD | Per-node-group DOCA-OFED driver management — supersedes the
cluster-wide ofedDriver setting. Use it when one cluster mixes
node types with different NIC hardware, driver versions, or
configurations. Spectrum-X reference-architecture deployments are
homogeneous, so a single NicClusterPolicy is sufficient; check
the Spectrum-X documentation for your Reference Architecture before
mixing node types. See NicNodePolicy. |
GA. Optional for Spectrum-X. |
| DOCA Telemetry Service | Host telemetry exporter for OVS, RDMA, and SuperNIC statistics. | Opt-in. |
Beyond creating CRDs, the Spectrum-X Operator applies SR-IOV and host settings on your behalf. Do not configure these by hand — the operator reconciles them, and manual changes are overwritten.
Setting | What it does | Applied when |
|---|---|---|
flow_steering_mode=hmfs |
devlink parameter enabling hardware-managed flow steering on each
PF, written through the SriovNetworkNodePolicy the operator
generates. |
Every Spectrum-X rail, all multiplane modes. |
esw_multiport=true |
devlink parameter putting the PF e-switch into multiport mode, so a single e-switch spans the rail’s planes. | Hardware Multiplane only. |
eSwitch mode switchdev |
Required for VF representors and OVS offload. | Every Spectrum-X rail. |
| OVS bridge | For swplb and none, a per-PF bridge created by the SR-IOV
Network Operator (GroupingPolicy: perPF, datapath netdev,
uplink type doca) — this is what the manageSoftwareBridges
feature gate enables. For hwplb, br-xplane, created by the
Spectrum-X Operator. |
Every Spectrum-X rail. |
| RDMA QoS meta-plugin | The generated OVSNetwork chains the RDMA CNI with
rdmaQoS: {tos: 96, tc: 96} and names the Pod’s RDMA device
rdma_<rail topology name>. |
Every Spectrum-X rail. |
| VRF meta-plugin | Isolates each rail in its own VRF, named after the rail topology
entry. Inspect routes with ip -6 route show vrf <rail>. |
Rails whose cidrPoolRef resolves to an IPv6 CIDRPool. The
operator reads the address family from the referenced pool, so a
rail that supplies its NV-IPAM block inline through
railTopology[].ipam does not get a VRF. |
| OVS safe-start drop-in | Installs /var/lib/spectrum-x/xplane-ovs-safe-start.sh and an
ExecStartPre drop-in at
/etc/systemd/system/ovs-vswitchd.service.d/20-xplane-safe-start.conf.
On OVS start it removes any xPlane uplink whose PF has not yet
entered switchdev mode, so OVS never comes up with a
half-configured uplink. |
Hardware Multiplane only. |
The DOCA XPlane Service Guide documents the devlink and eSwitch steps
as manual devlink commands for bare-metal hosts. In a Kubernetes
deployment the Spectrum-X Operator performs them for you — follow this
page rather than the bare-metal procedure.
The following dependencies are enforced by Network Operator’s reconciliation loop and validated by CRD webhooks where applicable:
Spectrum-X Operator requires SR-IOV Network Operator (for OVS-CNI and the SR-IOV Device Plugin), NIC Configuration Operator (for firmware and rail-name templates), NV-IPAM (for rail / plane IP allocation), and Multus (for secondary network attachment). It will not function standalone.
SR-IOV Network Operator requires NFD labels on nodes and SR-IOV enabled in the host BIOS / firmware (Host Layer). It is off by default in
NicClusterPolicyand must be explicitly enabled for Spectrum-X.DOCA-OFED driver container (Kubernetes Layer) and host-installed DOCA-Host (Host Layer) are alternatives, not additive. Choose one per node; do not deploy both.
OVS-CNI (Kubernetes Layer) plugs VFs into OVS bridges provided by OVS-DOCA (Host Layer). OVS-DOCA must be present on each node before the Kubernetes Layer can program rail / plane flows.
When DOCA xPlane is deployed
The Spectrum-X Operator deploys the xPlane DaemonSet per rail topology entry that selects more than one PF — that is, when
SpectrumXRailPoolConfig.railTopology[].nicSelector.pfNames lists two or
more PFs. It does not read multiplaneMode: that field lives on
NicConfigurationTemplate, which a different operator owns.
Mode | Rail topology shape | DOCA xPlane |
|---|---|---|
hwplb |
One entry per rail, listing every plane’s PF | Deployed |
swplb |
One entry per rail-plane, a single PF each | Not deployed |
none |
One entry per rail, a single PF | Not deployed |
Before creating the br-xplane bridge, the operator waits for the
selected PFs to enter switchdev mode.
Image versions and supported platforms: see the software components table in Platform Support.
Spectrum-X CRDs and API reference: see Spectrum-X CRDs and API Reference.
NIC firmware and rail-name configuration: see Spectrum-X NIC Configuration.
End-to-end deployment walkthroughs: see Spectrum-X Kubernetes Quick Start.