VirtIO Acceleration through Hardware vDPA
Hardware vDPA requires QEMU v2.12 (or with upstream 6.1.0) and DPDK v20.11 as minimal versions.
To install QEMU:
Clone the sources:
git clone https://git.qemu.org/git/qemu.git cd qemu git checkout v2.12
Build QEMU:
mkdir bin cd bin ../configure --target-list=x86_64-softmmu --enable-kvm make -j24
To install DPDK:
Clone the sources:
git clone git://dpdk.org/dpdk cd dpdk git checkout v20.11
Install dependencies (if needed):
yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel pandoc libibverbs libmlx5 libmnl-devel -y
Configure DPDK:
export RTE_SDK=$PWD make config T=x86_64-
native
-linuxapp-gcc cd build sed -i's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g'
.config sed -i's/\(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=\)n/\1y/g'
.configBuild DPDK:
make -j
Build the vDPA application:
cd $RTE_SDK/examples/vdpa/ make -j
To configure huge pages:
mkdir -p /hugepages
mount -t hugetlbfs hugetlbfs /hugepages
echo <more> > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
echo <more> > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
To configure a vDPA VirtIO interface in an existing VM's xml file (using libvirt):
Open the VM's configuration XML for editing:
virsh edit <domain name>
Perform the following:
Change the top line to:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
Assign a memory amount and use 1GB page size for huge pages (size must be the same as that used for the vDPA application), so that the memory configuration looks as follows.
<memory unit=
'KiB'
>4194304
</memory> <currentMemory unit='KiB'
>4194304
</currentMemory> <memoryBacking> <hugepages> <page size='1048576'
unit='KiB'
/> </hugepages> </memoryBacking>Assign an amount of CPUs for the VM CPU configuration, so that the vcpu and cputune configuration looks as follows:
<vcpu placement=
'static'
>5
</vcpu> <cputune> <vcpupin vcpu='0'
cpuset='14'
/> <vcpupin vcpu='1'
cpuset='16'
/> <vcpupin vcpu='2'
cpuset='18'
/> <vcpupin vcpu='3'
cpuset='20'
/> <vcpupin vcpu='4'
cpuset='22'
/> </cputune>Set the memory access for the CPUs to be shared, so that the cpu configuration looks as follows:
<cpu mode=
'custom'
match='exact'
check='partial'
> <model fallback='allow'
>Skylake-Server-IBRS</model> <numa> <cell id='0'
cpus='0-4'
memory='8388608'
unit='KiB'
memAccess='shared'
/> </numa> </cpu>Set the emulator in use to be the one built in step 2, so that the emulator configuration looks as follows:
<emulator><path to qemu executable></emulator>
Add a virtio interface using QEMU command line argument entries, so that the new interface snippet looks as follows:
<qemu:commandline> <qemu:arg value=
'-chardev'
/> <qemu:arg value='socket,id=charnet1,path=/tmp/sock-virtio0'
/> <qemu:arg value='-netdev'
/> <qemu:arg value='vhost-user,chardev=charnet1,queues=16,id=hostnet1'
/> <qemu:arg value='-device'
/> <qemu:arg value='virtio-net-pci,mq=on,vectors=6
,netdev=hostnet1,id=net1,mac=e4:11
:c6:d3:45
:f2,bus=pci.0
,addr=0x6
, page-per-vq=on,rx_queue_size=1024
,tx_queue_size=1024
'/> </qemu:commandline>WarningIn this snippet, the vhostuser socket file path, the amount of queues, the MAC and the PCIe slot of the virtio device can be configured.
Hardware vDPA supports switchdev mode only.
Create the ASAP2 environment:
Create the VFs.
Enter switchdev mode.
Set up OVS.
Run the vDPA application:
cd $RTE_SDK/examples/vdpa/build ./vdpa -w <VF PCI BDF>,
class
=vdpa --log-level=pmd,info -- -iCreate a vDPA port via the vDPA application CLI:
create /tmp/sock-virtio0 <PCI DEVICE BDF>
WarningThe vhostuser socket file path must be the one used when configuring the VM.
Start the VM:
virsh start <domain name>
For further information on the vDPA application, visit the Vdpa Sample Application DPDK documentation.