In virtio-net controller, each VF gets the same number of MSIX and virtqueues (VQs) so that each data VQ has a MSIX assigned. This means that changing the number of MSIX updates the number of VQs.
By default, each VF is assigned with the same number of MSIX, the default number is determined by the minimum of NUM_VF_MSIX and VIRTIO_NET_EMULATION_NUM_MSIX.
Using dynamic VF MSIX, a VF can be assigned with more MSIX/queues than its default. MSIX hardware resources of all VF devices are managed by PF via a shared MSIX pool. The user can reduce the MSIX of one VF, thus releasing its MSIX resources to the shared pool. On the other hand, another VF can be assigned with more MSIX than its default to gain more performance.
In virtio-net controller, each VF gets the same number of MSIX and virtqueues (VQs) so that each data VQ has a MSIX assigned. This means that changing the number of MSIX updates the number of VQs.
By default, each VF is assigned with the same number of MSIX, the default number is determined by the minimum of NUM_VF_MSIX and VIRTIO_NET_EMULATION_NUM_MSIX.
Using dynamic VF MSIX, a VF can be assigned with more MSIX/queues than its default. MSIX hardware resources of all VF devices are managed by PF via a shared MSIX pool. The user can reduce the MSIX of one VF, thus releasing its MSIX resources to the shared pool. On the other hand, another VF can be assigned with more MSIX than its default to gain more performance.
The default number of MSIX for each VF is determined by minimum(NUM_VF_MSIX, VIRTIO_NET_EMULATION_NUM_MSIX).
For example, configure the default MSIX number for a VF to be 32:
[DPU]# mlxconfig -y -d 03:00.0 s VIRTIO_NET_EMULATION_NUM_MSIX=32 NUM_VF_MSIX=32
Power cycle the BlueField and host to have the mlxconfig taking effect.
The MSIX pool for VFs is managed by their PF. To check the share pool size, run the following command (using PF 0 as example):
[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i msix_num_pool_size
By default, the share pool size is empty (0), since all MSIX resources have already been allocated to VFs evenly. Upon reducing the MSIX of one or more VFs, the reduced MSIX is released back to the pool.
However, the number of MSIX can be assigned to a given VF is also bound by capability. To check the those caps, run the following command:
[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i max_msix_num
[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i min_msix_num
To check the currently assigned number of MSIX, run the following command:
[DPU]# virtnet query -p 0 -v 0 | grep num_msix
If num_msix is less than max_msix_num cap, more MSIX can be assigned to the VF.
To allocate more MSIX to one VF, there should be MSIX available from the pool. This is done by reducing the MSIX from another VF(s).
The following example shows the steps to reallocate MSIX from VF1 to VF0, assuming that each VF has 32 MSIX available as default:
Unbind both VF devices from host driver.
[host]# echo <vf0_bdf> > /sys/bus/pci/drivers/virtio-pci/unbind [host]# echo <vf1_bdf> > /sys/bus/pci/drivers/virtio-pci/unbind
Reduce the MSIX of VF1.
[DPU]# virtnet modify -p 0 -v 1 device -n 4
Check pool size of PF0.
[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i msix_num_pool_size
Confirm the reduced MSIX are added to the share pool.
Increase the MSIX of VF0.
[DPU]# virtnet modify -p 0 -v 0 device -n 48
Check the MSIX of VF0.
[DPU]# virtnet query -p 0 -v 0 | grep -i num_msix
Bind both VF devices to host driver.
[host]# echo <vf0_bdf> > /sys/bus/pci/drivers/virtio-pci/bind [host]# echo <vf1_bdf> > /sys/bus/pci/drivers/virtio-pci/bind
NoteThe number of MSIX must be a power of 2.
The MSIX resources of a VF can be reduced to 0. However, the VF would not be functional in this case.
[DPU]# virtnet modify -p 0 -v 1 device -n 0
To use such a VF, assign a valid MSIX number:
[DPU]# virtnet modify -p 0 -v 1 device -n 10
The minimum MSIX resource number that allows the VF to load the host driver is 4 if VIRTIO_NET_F_CTRL_VQ is negotiated, or 2 if VIRTIO_NET_F_CTRL_VQ is not negotiated.