VirtIO Acceleration through Hardware vDPA

Hardware vDPA requires QEMU v2.12 (or with upstream 6.1.0) and DPDK v20.11 as minimal versions.

To install QEMU:

1. Clone the sources:

   Copy

   Copied!

               
   
               
   git clone https://git.qemu.org/git/qemu.git
   cd qemu
   git checkout v2.12
           

2. Build QEMU:

   Copy

   Copied!

               
   
               
   mkdir bin
   cd bin
   ../configure --target-list=x86_64-softmmu --enable-kvm
   make -j24
           

To install DPDK:

1. Clone the sources:

   Copy

   Copied!

               
   
               
   git clone git://dpdk.org/dpdk
   cd dpdk
   git checkout v20.11
           

2. Install dependencies (if needed):

   Copy

   Copied!

               
   
               
   yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel pandoc libibverbs libmlx5 libmnl-devel -y
           

3. Configure DPDK:

   Copy

   Copied!

               
   
               
   export RTE_SDK=$PWD
   make config T=x86_64-native-linuxapp-gcc
   cd build
   sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g' .config
   sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=\)n/\1y/g' .config
           

4. Build DPDK:

   Copy

   Copied!

               
   
               
   make -j
           

5. Build the vDPA application:

   Copy

   Copied!

               
   
               
   cd $RTE_SDK/examples/vdpa/
   make -j
           

To configure huge pages:

            

            
mkdir -p /hugepages
mount -t hugetlbfs hugetlbfs /hugepages
echo <more> > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
echo <more> > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
        

To configure a vDPA VirtIO interface in an existing VM's xml file (using libvirt):

1. Open the VM's configuration XML for editing:

   Copy

   Copied!

               
   
               
   virsh edit <domain name>
           

2. Perform the following:

   1. Change the top line to:

      Copy

      Copied!

                  
      
                  
      <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
              

   2. Assign a memory amount and use 1GB page size for huge pages (size must be the same as that used for the vDPA application), so that the memory configuration looks as follows.

      Copy

      Copied!

                  
      
                  
      <memory unit='KiB'>4194304</memory>
      <currentMemory unit='KiB'>4194304</currentMemory>
      <memoryBacking>
        <hugepages>
          <page size='1048576' unit='KiB'/>
        </hugepages>
      </memoryBacking>
              

   3. Assign an amount of CPUs for the VM CPU configuration, so that the vcpu and cputune configuration looks as follows:

      Copy

      Copied!

                  
      
                  
      <vcpu placement='static'>5</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='14'/>
        <vcpupin vcpu='1' cpuset='16'/>
        <vcpupin vcpu='2' cpuset='18'/>
        <vcpupin vcpu='3' cpuset='20'/>
        <vcpupin vcpu='4' cpuset='22'/>
      </cputune>
              

   4. Set the memory access for the CPUs to be shared, so that the cpu configuration looks as follows:

      Copy

      Copied!

                  
      
                  
      <cpu mode='custom' match='exact' check='partial'>
        <model fallback='allow'>Skylake-Server-IBRS</model>
        <numa>
          <cell id='0' cpus='0-4' memory='8388608' unit='KiB' memAccess='shared'/>
        </numa>
      </cpu>
              

   5. Set the emulator in use to be the one built in step 2, so that the emulator configuration looks as follows:

      Copy

      Copied!

                  
      
                  
      <emulator><path to qemu executable></emulator>
              

   6. Add a virtio interface using QEMU command line argument entries, so that the new interface snippet looks as follows:

      Copy

      Copied!

                  
      
                  
      <qemu:commandline>
        <qemu:arg value='-chardev'/>
        <qemu:arg value='socket,id=charnet1,path=/tmp/sock-virtio0'/>
        <qemu:arg value='-netdev'/>
        <qemu:arg value='vhost-user,chardev=charnet1,queues=16,id=hostnet1'/>
        <qemu:arg value='-device'/>
        <qemu:arg value='virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=e4:11:c6:d3:45:f2,bus=pci.0,addr=0x6,
         page-per-vq=on,rx_queue_size=1024,tx_queue_size=1024'/>
      </qemu:commandline>
              

      Warning

      In this snippet, the vhostuser socket file path, the amount of queues, the MAC and the PCIe slot of the virtio device can be configured.

1. Create the ASAP² environment:

   1. Create the VFs.

   2. Enter switchdev mode.

   3. Set up OVS.

2. Run the vDPA application:

   Copy

   Copied!

               
   
               
   cd $RTE_SDK/examples/vdpa/build
   ./vdpa -w <VF PCI BDF>,class=vdpa --log-level=pmd,info -- -i
           

3. Create a vDPA port via the vDPA application CLI:

   Copy

   Copied!

               
   
               
   create /tmp/sock-virtio0 <PCI DEVICE BDF>
           

   Warning

   The vhostuser socket file path must be the one used when configuring the VM.

4. Start the VM:

   Copy

   Copied!

               
   
               
   virsh start <domain name>
           

For further information on the vDPA application, visit the Vdpa Sample Application DPDK documentation.