Virtqueues (VQs) are the mechanism for bulk data transport on virtio devices. Each device can have zero or more VQs.
VQs can be in one of the following modes:
Split
Packed
This is currently the default VQ type.
Split VQ format was the only format supported by version 1.0 of virtio spec. In this mode, each VQ is separated into three parts:
Descriptor table – occupies the descriptor area
Available ring – occupies the driver area
Used ring – occupies the device area
Each of these parts is physically-contiguous in guest memory. Split VQ has a very simple design, but its sparse memory usage puts pressure on CPU cache utilization and requires several PCIe transactions for each descriptor.
Configuration
The following shows how the output of the virtnet list command appears only when split VQ mode is enabled:
"supported_virt_queue_types": {
"value": "0x1",
" 0": "SPLIT"
},
Packed VQ addresses the limitations of split VQ by merging the three rings in one location in virtual environment guest memory. This mode allows for fewer PCIe transactions and better CPU cache utilization per each descriptor access.
Configuration
Packed VQ mode can be enabled by defining packed_vq in the configuration file at the following path /opt/mellanox/mlnx_virtnet/virtnet.conf.
The following is an example of the packed_vq enabled in the configuration file:
{
"single_port": 1,
"packed_vq": 1,
"sf_pool_percent": 0,
"sf_pool_force_destroy": 0,
"vf": {
"mac_base": "CC:48:15:FF:00:00",
"vfs_per_pf": 126
}
}
The controller must be restarted after the configuration file is modified for the changes to take effect. Make sure to unload virtio-net/virtio-pcie drivers on the host and run:
[dpu]# systemctl restart virtio-net-controller.service
To check if the configuration has taken effect and controller supported packed VQ mode, run:
[dpu]# virtnet list
Check for PACKED in supported_virt_queue_types:
"supported_virt_queue_types": {
"value": "0x3",
" 0": "SPLIT",
" 1": "PACKED"
},
Virtio-net/virtio-pci drivers can be loaded at this point to create VQs in packed mode. Once the driver is loaded to verify that the device has packed VQ mode enabled, run the following command:
[dpu]# virtnet query -p <PFID> -v <VFID>
Check for VIRTNET_F_RING_PACKED in the driver features:
"driver_feature": {
"value": "0x8930012700e7182f",
" 0": "VIRTIO_NET_F_CSUM",
" 1": "VIRTIO_NET_F_GUEST_CSUM",
" 2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
" 3": "VIRTIO_NET_F_MTU",
" 5": "VIRTIO_NET_F_MAC",
" 11": "VIRTIO_NET_F_HOST_TSO4",
" 12": "VIRTIO_NET_F_HOST_TSO6",
" 16": "VIRTIO_NET_F_STATUS",
" 17": "VIRTIO_NET_F_CTRL_VQ",
" 18": "VIRTIO_NET_F_CTRL_RX",
" 21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
" 22": "VIRTIO_NET_F_MQ",
" 23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
" 32": "VIRTIO_F_VERSION_1",
" 33": "VIRTIO_F_IOMMU_PLATFORM",
" 34": "VIRTIO_F_RING_PACKED",
" 37": "VIRTIO_F_SR_IOV",
" 40": "VIRTIO_F_RING_RESET",
" 52": "VIRTIO_NET_F_VQ_NOTF_COAL",
" 53": "VIRTIO_NET_F_NOTF_COAL",
" 56": "VIRTIO_NET_F_HOST_USO",
" 59": "VIRTIO_NET_F_GUEST_HDRLEN",
" 63": "VIRTIO_NET_F_SPEED_DUPLEX"
},
If there are VFs mapped to multiple VMs then it is possible to have some devices create VQs in packed mode and some in split mode depending on the OS version and whether the driver has the feature supported.
Known Limitations
The following features are not currently supported when packed VQ is enabled:
Mergeable buffer
Jumbo MTU
UDP segmentation offload and RSS hash report