Unable to configure legacy mode

To set devlink to "Legacy" mode in BlueField, run:

# devlink dev eswitch set pci/0000:03:00.0 mode legacy
# devlink dev eswitch set pci/0000:03:00.1 mode legacy

Please verify that:

• No virtual functions are open. To verify if VFs are configured, run: 

  # /opt/mellanox/iproute2/sbin/rdma link | grep -i up
  link mlx5_0/2 state ACTIVE physical_state LINK_UP netdev pf0vf0
  link mlx5_1/2 state ACTIVE physical_state LINK_UP netdev pf1vf0

  If any VFs are configured, destroy them by running:

  # echo 0 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
  # echo 0 > /sys/class/infiniband/mlx5_1/device/mlx5_num_vfs

• If any SFs are configured, delete them by running:

  /sbin/mlnx-sf -a delete --sfindex <SF Index>

  You may retrieve the <SF Index> of the currently installed SFs by running:

  # mlnx-sf -a show
   
  SF Index: pci/0000:03:00.0/229408
    Parent PCI dev: 0000:03:00.0
    Representor netdev: en3f0pf0sf0
    Function HWADDR: 02:61:f6:21:32:8c
    Auxiliary device: mlx5_core.sf.2
      netdev: enp3s0f0s0
      RDMA dev: mlx5_2
   
  SF Index: pci/0000:03:00.1/294944
    Parent PCI dev: 0000:03:00.1
    Representor netdev: en3f1pf1sf0
    Function HWADDR: 02:30:13:6a:2d:2c
    Auxiliary device: mlx5_core.sf.3
      netdev: enp3s0f1s0
      RDMA dev: mlx5_3

  Pay attention to the SF Index values. For example: 

  /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.0/229408
  /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.1/294944
  

If the error "Error: mlx5_core: Can't change mode when flows are configured" is encountered while trying to configure legacy mode, please make sure that

1. Any configured SFs are deleted (see above for commands).

2. Shut down the links of all interfaces, delete any ip xfrm rules, delete any configured OVS flows, and stop openvswitch service. Run:

   ip link set dev p0 down
   ip link set dev p1 down
   ip link set dev pf0hpf down
   ip link set dev pf1hpf down
   ip link set dev vxlan_sys_4789 down
   
   ip x s f ;
   ip x p f ;
   
   tc filter del dev p0 ingress
   tc filter del dev p1 ingress
   tc qdisc show dev p0 
   tc qdisc show dev p1 
   tc qdisc del dev p0 ingress 
   tc qdisc del dev p1 ingress 
   tc qdisc show dev p0 
   tc qdisc show dev p1
   
   systemctl stop openvswitch-switch
   

Arm appears as two interfaces

What this looks like:

# sudo /opt/mellanox/iproute2/sbin/rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev p1

• Check if you are working in legacy mode.

  # devlink dev eswitch show pci/0000:03:00.<0|1>

  If the following line is printed, this means that you are working in legacy mode:

  pci/0000:03:00.<0|1>: mode legacy inline-mode none encap enable

  Please configure the DPU to work in switchdev mode. Run:

  devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev

• Check if you are working in separated mode:

  # mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i cpu
  * INTERNAL_CPU_MODEL SEPERATED_HOST(0)

  Please configure the DPU to work in embedded mode. Run:

  devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev