Virt Queue Types

NVIDIA BlueField Virtio-net v1.9.0

Virtqueues (VQs) are the mechanism for bulk data transport on virtio devices. Each device can have zero or more VQs.

VQs can be in one of the following modes:

  • Split

  • Packed

This is currently the default VQ type.

Split VQ format was the only format supported by version 1.0 of virtio spec. In this mode, each VQ is separated into three parts:

  • Descriptor table – occupies the descriptor area

  • Available ring – occupies the driver area

  • Used ring – occupies the device area

Each of these parts is physically-contiguous in guest memory. Split VQ has a very simple design, but its sparse memory usage puts pressure on CPU cache utilization and requires several PCIe transactions for each descriptor.

Configuration

The following shows how the output of the virtnet list command appears only when split VQ mode is enabled:

Copy
Copied!
            

"supported_virt_queue_types": { "value": "0x1", " 0": "SPLIT" },


Packed VQ addresses the limitations of split VQ by merging the three rings in one location in virtual environment guest memory. This mode allows for fewer PCIe transactions and better CPU cache utilization per each descriptor access.

Configuration

Packed VQ mode can be enabled by defining packed_vq in the configuration file at the following path /opt/mellanox/mlnx_virtnet/virtnet.conf.

The following is an example of the packed_vq enabled in the configuration file:

Copy
Copied!
            

{   "single_port": 1,   "packed_vq": 1,   "sf_pool_percent": 0,   "sf_pool_force_destroy": 0,    "vf": {     "mac_base": "CC:48:15:FF:00:00",     "vfs_per_pf": 126   } }

The controller must be restarted after the configuration file is modified for the changes to take effect. Make sure to unload virtio-net/virtio-pcie drivers on the host and run:

Copy
Copied!
            

[dpu]# systemctl restart virtio-net-controller.service

To check if the configuration has taken effect and controller supported packed VQ mode, run:

Copy
Copied!
            

[dpu]# virtnet list

Check for PACKED in supported_virt_queue_types:

Copy
Copied!
            

"supported_virt_queue_types": {                                                  "value": "0x3",                                                                "    0": "SPLIT",                                                              "    1": "PACKED"                                                            },   

Virtio-net/virtio-pci drivers can be loaded at this point to create VQs in packed mode. Once the driver is loaded to verify that the device has packed VQ mode enabled, run the following command:

Copy
Copied!
            

[dpu]# virtnet query -p <PFID> -v <VFID>

Check for VIRTNET_F_RING_PACKED in the driver features:

Copy
Copied!
            

"driver_feature": {                                                                                             "value": "0x8930012700e7182f",                                                                                "    0": "VIRTIO_NET_F_CSUM",                                                                                 "    1": "VIRTIO_NET_F_GUEST_CSUM",                                                                           "    2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",                                                                  "    3": "VIRTIO_NET_F_MTU",                                                                                  "    5": "VIRTIO_NET_F_MAC",                                                                                  "   11": "VIRTIO_NET_F_HOST_TSO4",                                                                            "   12": "VIRTIO_NET_F_HOST_TSO6",                                                                            "   16": "VIRTIO_NET_F_STATUS",                                                                               "   17": "VIRTIO_NET_F_CTRL_VQ",                                                                              "   18": "VIRTIO_NET_F_CTRL_RX",                                                                              "   21": "VIRTIO_NET_F_GUEST_ANNOUNCE",                                                                       "   22": "VIRTIO_NET_F_MQ",                                                                                   "   23": "VIRTIO_NET_F_CTRL_MAC_ADDR",                                                                        "   32": "VIRTIO_F_VERSION_1",                                                                                "   33": "VIRTIO_F_IOMMU_PLATFORM",                                                                           "   34": "VIRTIO_F_RING_PACKED",                                                                              "   37": "VIRTIO_F_SR_IOV",                                                                                   "   40": "VIRTIO_F_RING_RESET",                                                                               "   52": "VIRTIO_NET_F_VQ_NOTF_COAL",                                                                         "   53": "VIRTIO_NET_F_NOTF_COAL",                                                                            "   56": "VIRTIO_NET_F_HOST_USO",                                                                             "   59": "VIRTIO_NET_F_GUEST_HDRLEN",                                                                         "   63": "VIRTIO_NET_F_SPEED_DUPLEX"                                                                        },                            

If there are VFs mapped to multiple VMs then it is possible to have some devices create VQs in packed mode and some in split mode depending on the OS version and whether the driver has the feature supported.

Known Limitations

The following features are not currently supported when packed VQ is enabled:

  • Mergeable buffer

  • Jumbo MTU

  • UDP segmentation offload and RSS hash report

© Copyright 2024, NVIDIA. Last updated on May 7, 2024.