Verify and Troubleshoot Spectrum-X

Work through the checks in the order below: each layer depends on the one before it, so the first failing check is where to start debugging. The examples assume the Network Operator is installed in the nvidia-network-operator namespace.

Profile ConfigMaps are watched and loaded by the NIC configuration daemon, which runs on every node — not by the operator Pod. It logs a line for each profile it picks up:

            

            
kubectl -n nvidia-network-operator logs \
  -l app.kubernetes.io/name=nic-configuration-daemon \
  | grep "Loaded Spectrum-X profile"
        

If nothing matches, confirm the ConfigMap carries the watch label:

            

            
kubectl get configmap -A \
  -l network.nvidia.com/operator.nic-configuration.spectrum-x-profile
        

The label value is ignored — only the key must be present. The ConfigMap name is what spectrumXOptimized.version must match, character for character.

Each SuperNIC appears as a NicDevice. Check the conditions:

            

            
kubectl -n nvidia-network-operator get nicdevices
kubectl -n nvidia-network-operator get nicdevice <name> -o jsonpath='{.status.conditions}' | jq
        

Reasons you may see on the ConfigUpdateInProgress condition:

SpectrumXRailPoolConfig reports an aggregate status and a per-node breakdown:

            

            
kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.syncStatus}'

kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.nodeStates}' | jq
        

syncStatus is one of Unknown, InProgress, Failed, or Succeeded. When it is not Succeeded, nodeStates[] identifies which nodes are affected and carries a message for each.

Confirm the operator generated the downstream resources — one per rail, or one per rail-plane in swplb:

            

            
kubectl -n nvidia-network-operator get ovsnetwork
kubectl -n nvidia-network-operator get sriovnetworknodepolicy
        

With hwplb the planes are hidden from the workload: a Pod sees one interface per rail whether every plane is healthy or only one is. A degraded plane is therefore invisible from inside the Pod, and DOCA xPlane is the only place to see it.

The xPlane DaemonSet runs as xplane-ds:

            

            
kubectl -n nvidia-network-operator get ds xplane-ds
kubectl -n nvidia-network-operator logs -l app=xplane
        

Query plane state with the bundled client:

            

            
kubectl -n nvidia-network-operator exec ds/xplane-ds -- \
  doca-xplane-client get-status

kubectl -n nvidia-network-operator exec ds/xplane-ds -- \
  doca-xplane-client get-planes-summary
        

get-status reports an overallState and counts local versus remote failures. When a plane is DEGRADED, narrow it down:

            

            
# Local failures — link down on this host
doca-xplane-client get-plane-failures-local --plane_id 0

# Remote failures — route withdrawn by the fabric
doca-xplane-client get-plane-failures-remote --plane_id 0

# How much traffic was diverted away from this plane
doca-xplane-client get-plane-traffic-summary --plane_id 0
        

Deploy the test Pod from your walkthrough and confirm one RDMA device per rail:

            

            
kubectl exec -it spectrum-x-test -- rdma link
        

On IPv6 rails the Spectrum-X Operator places each rail in its own VRF, named after the rail topology entry, so routes are not in the main table:

            

            
kubectl exec -it spectrum-x-test -- ip -6 route show vrf rail0
        

            

            
ip vrf exec rail0 ib_write_bw -R ...
        

• Configuration reference and validation rules: see Spectrum-X NIC Configuration.

• What the operator configures on your behalf: see Architecture and Components.

• CRD field reference: see Spectrum-X CRDs and API Reference.

• Log collection for support: see SOS Report.