Configuring VF to be Trusted
To support hardware offloading and software steering on VFs, the trusted mode needs to be enabled.
The procedure below is using a ConnectX-5 adapter cards (pf0) with 2 VFs.
In Legacy SR-IOV mode.
Configure all the trusted VF using mlxreg to use software steering.
NoteFirmware version used must be >= xx.29.1016
For all trusted VFs (mt4121_pciconf0 is the MST device example):
$ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_name VHCA_TRUST_LEVEL --yes --set
"all_vhca=0x1,trust_level=0x1"For a specific VF:
$ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_id 0xc007 --reg_len 0x40 --indexes "0x0.0:32=<vhca id>" --yes --set "0x4.0:32=0x1"
Create 2 VFs on the PF pf0 when in Legacy SR-IOV mode.
$ echo 2 > /sys/class/net/pf0/device/mlx5_num_vfs
Verify the VFs are created.
$ lspci | grep Mellanox 82:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 82:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 82:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 82:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
Set the VFs to be trusted for the kernel by using one of the methods below.
Using sysfs file.
$ echo ON | tee /sys/class/net/pf0/device/sriov/0/trust $ echo ON | tee /sys/class/net/pf0/device/sriov/1/trust
Using “ip link” command.
$ ip link set p0 vf 0 trust on $ ip link set p0 vf 1 trust on
VFs can be attached to the VM.
In switchdev SR-IOV mode with bond (VF-LAG), for single port switchdev SR-IOV, the first two steps are not needed.
Probe the bond module, take pf0, pf1 and mode4 for example.
$ modprobe bonding miimon=100 mode=4 $ ip link add bond1 type bond miimon 100 mode 4 xmit_hash_policy layer3+4
Add PFs to the bond.
$ ip link set pf0 master bond1 $ ip link set pf1 master bond1
Configure all the trusted VF using mlxreg to use SW steering.
NoteFirmware version used must be >= xx.29.1016
For all trusted VFs:
$ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_name VHCA_TRUST_LEVEL --yes --set
"all_vhca=0x1,trust_level=0x1"For a specific VF:
$ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_id
0xc007--reg_len0x40--indexes"0x0.0:32=<vhca id>"--yes --set"0x4.0:32=0x1"
Create VFs on the PFs.
$ echo 2 > /sys/class/net/pf0/device/mlx5_num_vfs $ echo 2 > /sys/class/net/pf1/device/mlx5_num_vfs
Unbind all the VFs.
$ echo "0000:82:0Y.X" >> /sys/bus/pci/drivers/mlx5_core/unbind
[Optional] Set encap to none.
$ echo "none" > /sys/class/net/pf0/compat/devlink/encap $ echo "none" > /sys/class/net/pf1/compat/devlink/encap
Set switchdev mode.
$ echo "switchdev" > "/sys/class/net/pf0/compat/devlink/mode" $ echo "switchdev" > "/sys/class/net/pf1/compat/devlink/mode"
All VFs are trusted and can be probed into the VM. If they need to be used in the hypervisor, they need to be bound.
NoteFirmware version used must be >= xx.29.1016
For all trusted VFs.
$ mst start $ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_id 0xc007 --reg_len 0x40 --indexes "0x0.0:32=0x80000000" --yes --set "0x4.0:32=0x1"
All VFs are trusted and could be probed into VM. If they need to be used in hypervisor, they need to be bound.
$ echo "0000:82:0Y.X" >> /sys/bus/pci/drivers/mlx5_core/bind
Trusted SF's configuration for Switchdev mode.
Configure the following to create SFs
$ mlxconfig -d
0000:43:00.0s PF_BAR2_ENABLE=0PER_PF_NUM_SF=1PF_TOTAL_SF=252PF_SF_BAR_SIZE=10Power cycle the server.
Configure the device in switchdev mode.
$ devlink dev eswitch set pci/
0000:43:00.0mode switchdevConfigure it back to legacy mode.
$ devlink dev eswitch set pci/
0000:43:00.0mode legacyDisable encap on the PF FDB.
$ echo none > /sys/
class/net/pf0/compat/devlink/encapRestore the switchdev mode.
$ devlink dev eswitch set pci/
0000:43:00.0mode switchdevConfigure all SFs to be trusted.
$ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_name VHCA_TRUST_LEVEL --yes --set
"all_vhca=0x1,trust_level=0x1"Alias mlxdevm to simplify the commands.
$ alias mlxdevm=
"/opt/mellanox/iproute2/sbin/mlxdevm"Create two SFs for example.
$ mlxdevm port add pci/
0000:43:00.0flavour pcisf pfnum0sfnum101$ mlxdevm port add pci/0000:43:00.0flavour pcisf pfnum0sfnum102Set the MAC address for the new SFs.
$ mlxdevm port function set pci/
0000:43:00.0/32768hw_addr aa:bb:cc:09:83:04$ mlxdevm port function set pci/0000:43:00.0/32769hw_addr aa:bb:cc:09:83:04Activate the new SFs.
$ mlxdevm port function set pci/
0000:43:00.0/32768state active $ mlxdevm port function set pci/0000:43:00.0/32769state activeIf on a NVIDIA BlueField-2 DPU
$ echo mlx5_core.sf.
6> /sys/bus/auxiliary/drivers/mlx5_core.sf_cfg/unbind $ echo mlx5_core.sf.6> /sys/bus/auxiliary/drivers/mlx5_core.sf/bind
Trusted HPF/VF on the host for NVIDIA BlueField-2
Stop the VM if it is a VF and in use.
On the host side, stop the driver.
$ /etc/init.d/openibd stop
On the Arm side, configure the device in legacy mode.
# devlink dev eswitch set pci/
0000:03:00.0mode legacy # devlink dev eswitch set pci/0000:03:00.1mode legacyDisable encap.
# echo none > /sys/
class/net/p0/compat/devlink/encap # echo none > /sys/class/net/p1/compat/devlink/encapConfigure the device back to switchdev mode.
# devlink dev eswitch set pci/
0000:03:00.0mode switchdev # devlink dev eswitch set pci/0000:03:00.1mode switchdevConfigure the HPF to be trusted.
# mlxreg -d /dev/mst/mt41686_pciconf0 --reg_name VHCA_TRUST_LEVEL --yes --set
"all_vhca=0x1,trust_level=0x1"# mlxreg -d /dev/mst/mt41686_pciconf0.1--reg_name VHCA_TRUST_LEVEL --yes --set"all_vhca=0x1,trust_level=0x1"Restart the driver on the host side.
$ /etc/init.d/openibd start
Start the driver and VM if it is a VF.