NVIDIA BlueField Virtio-net v1.9.0
NVIDIA BlueField Virtio-net v1.9.0

VF Dynamic MSIX

In virtio-net controller, each VF gets the same number of MSIX and virtqueues (VQs) so that each data VQ has a MSIX assigned. This means that changing the number of MSIX updates the number of VQs.

By default, each VF is assigned with the same number of MSIX, the default number is determined by the minimum of NUM_VF_MSIX and VIRTIO_NET_EMULATION_NUM_MSIX.

Using dynamic VF MSIX, a VF can be assigned with more MSIX/queues than its default. MSIX hardware resources of all VF devices are managed by PF via a shared MSIX pool. The user can reduce the MSIX of one VF, thus releasing its MSIX resources to the shared pool. On the other hand, another VF can be assigned with more MSIX than its default to gain more performance.

image-2024-4-18_15-17-0-version-1-modificationdate-1715098419861-api-v2.png

In virtio-net controller, each VF gets the same number of MSIX and virtqueues (VQs) so that each data VQ has a MSIX assigned. This means that changing the number of MSIX updates the number of VQs.

By default, each VF is assigned with the same number of MSIX, the default number is determined by the minimum of NUM_VF_MSIX and VIRTIO_NET_EMULATION_NUM_MSIX.

Using dynamic VF MSIX, a VF can be assigned with more MSIX/queues than its default. MSIX hardware resources of all VF devices are managed by PF via a shared MSIX pool. The user can reduce the MSIX of one VF, thus releasing its MSIX resources to the shared pool. On the other hand, another VF can be assigned with more MSIX than its default to gain more performance.

image-2024-4-18_15-17-0-version-1-modificationdate-1718747072180-api-v2.png

The default number of MSIX for each VF is determined by minimum(NUM_VF_MSIX, VIRTIO_NET_EMULATION_NUM_MSIX).

For example, configure the default MSIX number for a VF to be 32:

Copy
Copied!
            

[DPU]# mlxconfig -y -d 03:00.0 s VIRTIO_NET_EMULATION_NUM_MSIX=32 NUM_VF_MSIX=32

Power cycle the BlueField and host to have the mlxconfig taking effect.

The MSIX pool for VFs is managed by their PF. To check the share pool size, run the following command (using PF 0 as example):

Copy
Copied!
            

[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i msix_num_pool_size

By default, the share pool size is empty (0), since all MSIX resources have already been allocated to VFs evenly. Upon reducing the MSIX of one or more VFs, the reduced MSIX is released back to the pool.

However, the number of MSIX can be assigned to a given VF is also bound by capability. To check the those caps, run the following command:

Copy
Copied!
            

[DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i max_msix_num [DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i min_msix_num

To check the currently assigned number of MSIX, run the following command:

Copy
Copied!
            

[DPU]# virtnet query -p 0 -v 0 | grep num_msix

If num_msix is less than max_msix_num cap, more MSIX can be assigned to the VF.

To allocate more MSIX to one VF, there should be MSIX available from the pool. This is done by reducing the MSIX from another VF(s).

The following example shows the steps to reallocate MSIX from VF1 to VF0, assuming that each VF has 32 MSIX available as default:

  1. Unbind both VF devices from host driver.

    Copy
    Copied!
                

    [host]# echo <vf0_bdf> > /sys/bus/pci/drivers/virtio-pci/unbind [host]# echo <vf1_bdf> > /sys/bus/pci/drivers/virtio-pci/unbind

  2. Reduce the MSIX of VF1.

    Copy
    Copied!
                

    [DPU]# virtnet modify -p 0 -v 1 device -n 4

  3. Check pool size of PF0.

    Copy
    Copied!
                

    [DPU]# virtnet list | grep -i '"pf_id": 0' -A 8 | grep -i msix_num_pool_size

    Confirm the reduced MSIX are added to the share pool.

  4. Increase the MSIX of VF0.

    Copy
    Copied!
                

    [DPU]# virtnet modify -p 0 -v 0 device -n 48

  5. Check the MSIX of VF0.

    Copy
    Copied!
                

    [DPU]# virtnet query -p 0 -v 0 | grep -i num_msix

  6. Bind both VF devices to host driver.

    Copy
    Copied!
                

    [host]# echo <vf0_bdf> > /sys/bus/pci/drivers/virtio-pci/bind [host]# echo <vf1_bdf> > /sys/bus/pci/drivers/virtio-pci/bind

    Note

    The number of MSIX must be a power of 2.

The MSIX resources of a VF can be reduced to 0. However, the VF would not be functional in this case.

Copy
Copied!
            

[DPU]# virtnet modify -p 0 -v 1 device -n 0

To use such a VF, assign a valid MSIX number:

Copy
Copied!
            

[DPU]# virtnet modify -p 0 -v 1 device -n 10

The minimum MSIX resource number that allows the VF to load the host driver is 4 if VIRTIO_NET_F_CTRL_VQ is negotiated, or 2 if VIRTIO_NET_F_CTRL_VQ is not negotiated.

© Copyright 2024, NVIDIA. Last updated on Jun 17, 2024.