eSwitch Troubleshooting

1.0

To set devlink to "Legacy" mode in BlueField, run:

Copy
Copied!
            

# devlink dev eswitch set pci/0000:03:00.0 mode legacy # devlink dev eswitch set pci/0000:03:00.1 mode legacy

Please verify that:

  • No virtual functions are open. To verify if VFs are configured, run:

    Copy
    Copied!
                

    # /opt/mellanox/iproute2/sbin/rdma link | grep -i up link mlx5_0/2 state ACTIVE physical_state LINK_UP netdev pf0vf0 link mlx5_1/2 state ACTIVE physical_state LINK_UP netdev pf1vf0

    If any VFs are configured, destroy them by running:

    Copy
    Copied!
                

    # echo 0 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs # echo 0 > /sys/class/infiniband/mlx5_1/device/mlx5_num_vfs

  • If any SFs are configured, delete them by running:

    Copy
    Copied!
                

    /sbin/mlnx-sf -a delete --sfindex <SF Index>

    Warning

    You may retrieve the <SF Index> of the currently installed SFs by running:

    Copy
    Copied!
                

    # mlnx-sf -a show   SF Index: pci/0000:03:00.0/229408 Parent PCI dev: 0000:03:00.0 Representor netdev: en3f0pf0sf0 Function HWADDR: 02:61:f6:21:32:8c Auxiliary device: mlx5_core.sf.2 netdev: enp3s0f0s0 RDMA dev: mlx5_2   SF Index: pci/0000:03:00.1/294944 Parent PCI dev: 0000:03:00.1 Representor netdev: en3f1pf1sf0 Function HWADDR: 02:30:13:6a:2d:2c Auxiliary device: mlx5_core.sf.3 netdev: enp3s0f1s0 RDMA dev: mlx5_3

    Pay attention to the SF Index values. For example:

    Copy
    Copied!
                

    /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.0/229408 /sbin/mlnx-sf -a delete --sfindex pci/0000:03:00.1/294944

If the error "Error: mlx5_core: Can't change mode when flows are configured" is encountered while trying to configure legacy mode, please make sure that

  1. Any configured SFs are deleted (see above for commands).

  2. Shut down the links of all interfaces, delete any ip xfrm rules, delete any configured OVS flows, and stop openvswitch service. Run:

    Copy
    Copied!
                

    ip link set dev p0 down ip link set dev p1 down ip link set dev pf0hpf down ip link set dev pf1hpf down ip link set dev vxlan_sys_4789 down   ip x s f ; ip x p f ;   tc filter del dev p0 ingress tc filter del dev p1 ingress tc qdisc show dev p0 tc qdisc show dev p1 tc qdisc del dev p0 ingress tc qdisc del dev p1 ingress tc qdisc show dev p0 tc qdisc show dev p1   systemctl stop openvswitch-switch

What this looks like:

Copy
Copied!
            

# sudo /opt/mellanox/iproute2/sbin/rdma link link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0 link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev p1

  • Check if you are working in legacy mode.

    Copy
    Copied!
                

    # devlink dev eswitch show pci/0000:03:00.<0|1>

    If the following line is printed, this means that you are working in legacy mode:

    Copy
    Copied!
                

    pci/0000:03:00.<0|1>: mode legacy inline-mode none encap enable

    Please configure the DPU to work in switchdev mode. Run:

    Copy
    Copied!
                

    devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev

  • Check if you are working in separated mode:

    Copy
    Copied!
                

    # mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i cpu * INTERNAL_CPU_MODEL SEPERATED_HOST(0)

    Please configure the DPU to work in embedded mode. Run:

    Copy
    Copied!
                

    devlink dev eswitch set pci/0000:03:00.<0|1> mode switchdev

© Copyright 2023, NVIDIA. Last updated on Jan 3, 2024.