eSwitch Troubleshooting

To set devlink to "Legacy" mode in BlueField, run:

            

            
# devlink dev eswitch set pci/0000:03:00.0 mode legacy
# devlink dev eswitch set pci/0000:03:00.1 mode legacy
        

Please verify that:

• No virtual functions are open. To verify if VFs are configured, run:

  Copy

  Copied!

              
  
              
  # /opt/mellanox/iproute2/sbin/rdma link | grep -i up
  link mlx5_0/2 state ACTIVE physical_state LINK_UP netdev pf0vf0
  link mlx5_1/2 state ACTIVE physical_state LINK_UP netdev pf1vf0
          

  If any VFs are configured, destroy them by running:

  Copy

  Copied!

              
  
              
  # echo 0 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
  # echo 0 > /sys/class/infiniband/mlx5_1/device/mlx5_num_vfs
          

• If any SFs are configured, delete them by running:

  Copy

  Copied!

              
  
              
  /sbin/mlnx-sf -a delete --sfindex <SF Index>
          

  Warning

  You may retrieve the <SF Index> of the currently installed SFs by running:

  Copy

  Copied!

              
  
              
  # mlnx-sf -a show
   
  SF Index: pci/0000:03:00.0/229408
    Parent PCI dev: 0000:03:00.0
    Representor netdev: en3f0pf0sf0
    Function HWADDR: 02:61:f6:21:32:8c
    Auxiliary device: mlx5_core.sf.2
      netdev: enp3s0f0s0
      RDMA dev: mlx5_2
   
  SF Index: pci/0000:03:00.1/294944
    Parent PCI dev: 0000:03:00.1
    Representor netdev: en3f1pf1sf0
    Function HWADDR: 02:30:13:6a:2d:2c
    Auxiliary device: mlx5_core.sf.3
      netdev: enp3s0f1s0
      RDMA dev: mlx5_3
          

  Pay attention to the SF Index values. For example:

  Copy

  Copied!

              
  
              
  /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.0/229408
  /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.1/294944
          

If the error "Error: mlx5_core: Can't change mode when flows are configured" is encountered while trying to configure legacy mode, please make sure that

1. Any configured SFs are deleted (see above for commands).

2. Shut down the links of all interfaces, delete any ip xfrm rules, delete any configured OVS flows, and stop openvswitch service. Run:

   Copy

   Copied!

               
   
               
   ip link set dev p0 down
   ip link set dev p1 down
   ip link set dev pf0hpf down
   ip link set dev pf1hpf down
   ip link set dev vxlan_sys_4789 down
    
   ip x s f ;
   ip x p f ;
    
   tc filter del dev p0 ingress
   tc filter del dev p1 ingress
   tc qdisc show dev p0 
   tc qdisc show dev p1 
   tc qdisc del dev p0 ingress 
   tc qdisc del dev p1 ingress 
   tc qdisc show dev p0 
   tc qdisc show dev p1
    
   systemctl stop openvswitch-switch
           

What this looks like:

            

            
# sudo /opt/mellanox/iproute2/sbin/rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev p1
        

• Check if you are working in legacy mode.

  Copy

  Copied!

              
  
              
  # devlink dev eswitch show pci/0000:03:00.<0|1>
          

  If the following line is printed, this means that you are working in legacy mode:

  Copy

  Copied!

              
  
              
  pci/0000:03:00.<0|1>: mode legacy inline-mode none encap enable
          

  Please configure the DPU to work in switchdev mode. Run:

  Copy

  Copied!

              
  
              
  devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev
          

• Check if you are working in separated mode:

  Copy

  Copied!

              
  
              
  # mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i cpu
  * INTERNAL_CPU_MODEL SEPERATED_HOST(0)
          

  Please configure the DPU to work in embedded mode. Run:

  Copy

  Copied!

              
  
              
  devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev