NVIDIA Spectrum-X NIC Configuration

NVIDIA NIC Configuration Operator in Network Operator 26.7.0 applies Spectrum-X NIC tuning from a Spectrum-X profile — a versioned YAML document published for each Reference Architecture and supplied to the cluster as a labeled ConfigMap. The spectrumXOptimized.version field names that ConfigMap.

This replaces the fixed RA-version enum used in earlier releases. In Network Operator 26.4.0 and earlier, version accepted only a fixed set of built-in RA versions, and the tuning tables shipped inside the operator image. From 26.7.0, the operator carries no built-in profiles: every Spectrum-X deployment must apply a profile ConfigMap first, and NIC tuning can be revised without a new operator release.

For the Network Operator Spectrum-X RA support matrix and validated hardware, operating system, and Kubernetes combinations, see Platform Support.

Spectrum-X configuration applies to ConnectX-7 NIC, ConnectX-8 SuperNIC, ConnectX-9 SuperNIC, and BlueField-3 SuperNIC; Hardware Plane Load Balancing (hwplb) additionally requires ConnectX-8 SuperNIC or ConnectX-9 SuperNIC. Device IDs and the modes each NIC accepts are listed under NIC type constraints.

To install the operator and for more information on the CRDs, see NIC Firmware Configuration and Configuration Details.

A Spectrum-X profile holds the NIC tuning for one Reference Architecture: the non-volatile mlxConfig parameters applied per multiplane mode and device ID, the runtimeConfig parameters for RoCE, adaptive routing, congestion control and inter-packet gap, and the congestion-control settings docaCCVersion and useSoftwareCCAlgorithm.

Download the profile published for your Reference Architecture from the NVOnline portal, or contact your NVIDIA CPM. Apply it as-is — the published file already carries the network.nvidia.com/operator.nic-configuration.spectrum-x-profile label and the complete data.profile body:

            

            
kubectl apply -f spectrum-x-profile-configmap.yaml
        

The ConfigMap name is the value you reference from spectrumXOptimized.version. Profiles are keyed by name and not by namespace, so use unique profile names across every namespace the operator watches.

For the full profile ConfigMap format, the label rules, and a complete annotated example, see Configuration Details.

To enable the DOCA SPC-X CC algorithm on NIC devices, the DOCA SPC-X CC .deb package for ubuntu 22.04 is required. This configuration step will be removed in the future, once the DOCA SPC-X CC algorithm is publicly available. To access the package, contact your NVIDIA CPM. The package should be available in the cluster and then its URL should be provided in the packageUrlSource field of the SpectrumXOperator CR.

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  # should point to the URL of the DOCA SPC-X CC .deb package for Ubuntu 22.04
  docaSpcXCCUrlSource: "https://example.com/doca-spcx-cc_3.1.0105-1_amd64.deb"
        

Firmware Upgrade

If the firmware on the devices needs to be updated, extend the NicFirmwareSource CR with fields for ConnectX and BlueField firmware. Make sure to use the correct firmware for your devices.

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  # should point to the URL of the DOCA SPC-X CC .deb package for Ubuntu 22.04
  docaSpcXCCUrlSource: "https://example.com/doca-spcx-cc_3.1.0105-1_amd64.deb"
  # a list of firmware binaries zip archives from the Mellanox website, can point to any URL accessible from the cluster
  binUrlSources:
    - https://www.mellanox.com/downloads/firmware/fw-ConnectX8-rel-40_46_3048-900-9X85E-00NX-MC0_Ax-UEFI-14.39.14-FlexBoot-3.8.100.signed.bin.zip
  # a URL to the BlueField Bundle (BFB) file, can point to any URL accessible from the cluster
  bfbUrlSource:
    - https://example.com/bf-fwbundle-3.1.0-77_25.07-prod.bfb
        

Configure and apply the NicFirmwareTemplate CR:

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nicSelector:
    nicType: "a2dc" # BlueField-3 SuperNIC. Can also be "1021" for ConnectX-7, "1023" for ConnectX-8, or "1025" for ConnectX-9
    # partNumbers:
    # - "MCX713106AEHEA_QP1"
  template:
    nicFirmwareSourceRef: spectrum-x-configuration
    updatePolicy: Update
        

Reference the profile ConfigMap by name from spectrumXOptimized.version. Spectrum-X optimizations require linkType: Ethernet and numVfs: 1.

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: a2dc # BlueField-3 SuperNIC. Use "1023" for ConnectX-8 SuperNIC, or "1025" for ConnectX-9 SuperNIC (tech preview)
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "spectrum-x-ra2.3-profile" # name of the Spectrum-X profile ConfigMap
      overlay: "none" # For an L3 EVPN overlay, use "l3"
        

Multiplane configuration

Multiplane modes let a NIC drive multiple data planes on the same rail. Set multiplaneMode and numberOfPlanes alongside the profile reference.

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-multiplane-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "1023" # ConnectX-8 SuperNIC. Use "1025" for ConnectX-9 SuperNIC (tech preview)
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "spectrum-x-ra2.3-profile"
      overlay: "none"
      multiplaneMode: "hwplb" # Hardware Plane Load Balancing
      numberOfPlanes: 4
        

The profile must contain tuning for the multiplane mode and device ID you select. A profile that has no mlxConfig entry for the requested mode leaves those parameters at their device defaults.

Multiplane modes

The following multiplane modes are available:

NIC type constraints

Configure custom interface names

The NicInterfaceNameTemplate CRD allows you to define custom naming patterns for RDMA and network device interfaces on Spectrum-X NICs. This is useful in multiplane and multi-rail deployments where predictable interface naming is required.

The operator deploys udev rules to the host to rename network and RDMA interfaces according to the specified naming template.

The template uses the following placeholders for device name construction:

• %nic_id%: The index of the NIC in the flattened list of NICs

• %plane_id%: The index of the plane of the specific NIC

• %rail_id%: The index of the rail where the given NIC belongs to

            

            
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  # Number of PFs per NIC, used to calculate the number of planes per NIC
  pfsPerNic: 2
  # Template for RDMA device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  rdmaDevicePrefix: "rdma_%nic_id%_%plane_id%_%rail_id%"
  # Template for net device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  netDevicePrefix: "net_%nic_id%_%plane_id%_%rail_id%"
  # PCI address to rail mapping. First dimension is rail index, second is NIC PCI addresses in the rail
  railPciAddresses:
    - ["0000:1a:00.0", "0000:2a:00.0"]
    - ["0000:3a:00.0", "0000:4a:00.0"]
        

The railPciAddresses field defines the PCI address to rail mapping. The first dimension is the rail index and the second dimension is the list of PCI addresses of the NICs in that rail.

Generated udev rules

The operator generates udev rules based on the template and writes them to the host. The rules are written to two separate files.

Example generated udev rules for net devices (/etc/udev/rules.d/10-nic-net-interface-naming.rules):

            

            
# Auto-generated by nic-configuration-operator
# Do not edit manually

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"
        

Example generated udev rules for RDMA devices (/etc/udev/rules.d/10-nic-rdma-interface-naming.rules):

            

            
# Auto-generated by nic-configuration-operator
# Do not edit manually

ACTION=="add", KERNELS=="0000:1a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_0_0"
ACTION=="add", KERNELS=="0000:1a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_1_0"
ACTION=="add", KERNELS=="0000:3a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_0_1"
ACTION=="add", KERNELS=="0000:3a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_1_1"
        

The following validation rules are enforced by the API:

• Spectrum-X optimizations can only be enabled when linkType is Ethernet and numVfs is 1.

• Spectrum-X optimizations can only be enabled for ConnectX-7 NIC (nicType: 1021), ConnectX-8 SuperNIC (nicType: 1023), ConnectX-9 SuperNIC (nicType: 1025), or BlueField-3 SuperNIC (nicType: a2dc).

• When Spectrum-X optimizations are enabled, roceOptimized must not be enabled (RoCE settings are included in the Spectrum-X configuration).

• version is required and is not validated against existing ConfigMaps — any name is accepted.

• When multiplaneMode is none, numberOfPlanes must be 1.

• When multiplaneMode is not none, numberOfPlanes must not be 1.

• The hwplb multiplane mode can only be enabled for ConnectX-8 SuperNIC (nicType: 1023) or ConnectX-9 SuperNIC (nicType: 1025).

rawNvConfig may be combined with Spectrum-X optimizations: the raw parameters are merged as overrides on top of the parameters calculated from the Spectrum-X profile. Parameter names must not use index-range syntax such as NAME[0..3] — list each index explicitly.