DOCA Documentation v2.5.0 LTS
1.0

VirtIO Acceleration through Hardware vDPA

Hardware vDPA requires QEMU v2.12 (or with upstream 6.1.0) and DPDK v20.11 as minimal versions.

To install QEMU:

  1. Clone the sources:

    Copy
    Copied!
                

    git clone https://git.qemu.org/git/qemu.git cd qemu git checkout v2.12

  2. Build QEMU:

    Copy
    Copied!
                

    mkdir bin cd bin ../configure --target-list=x86_64-softmmu --enable-kvm make -j24

To install DPDK:

  1. Clone the sources:

    Copy
    Copied!
                

    git clone git://dpdk.org/dpdk cd dpdk git checkout v20.11

  2. Install dependencies (if needed):

    Copy
    Copied!
                

    yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel pandoc libibverbs libmlx5 libmnl-devel -y

  3. Configure DPDK:

    Copy
    Copied!
                

    export RTE_SDK=$PWD make config T=x86_64-native-linuxapp-gcc cd build sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g' .config sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=\)n/\1y/g' .config

  4. Build DPDK:

    Copy
    Copied!
                

    make -j

  5. Build the vDPA application:

    Copy
    Copied!
                

    cd $RTE_SDK/examples/vdpa/ make -j

To configure huge pages:

Copy
Copied!
            

mkdir -p /hugepages mount -t hugetlbfs hugetlbfs /hugepages echo <more> > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages echo <more> > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages

To configure a vDPA VirtIO interface in an existing VM's xml file (using libvirt):

  1. Open the VM's configuration XML for editing:

    Copy
    Copied!
                

    virsh edit <domain name>

  2. Perform the following:

    1. Change the top line to:

      Copy
      Copied!
                  

      <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

    2. Assign a memory amount and use 1GB page size for huge pages (size must be the same as that used for the vDPA application), so that the memory configuration looks as follows.

      Copy
      Copied!
                  

      <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> </memoryBacking>

    3. Assign an amount of CPUs for the VM CPU configuration, so that the vcpu and cputune configuration looks as follows:

      Copy
      Copied!
                  

      <vcpu placement='static'>5</vcpu> <cputune> <vcpupin vcpu='0' cpuset='14'/> <vcpupin vcpu='1' cpuset='16'/> <vcpupin vcpu='2' cpuset='18'/> <vcpupin vcpu='3' cpuset='20'/> <vcpupin vcpu='4' cpuset='22'/> </cputune>

    4. Set the memory access for the CPUs to be shared, so that the cpu configuration looks as follows:

      Copy
      Copied!
                  

      <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>Skylake-Server-IBRS</model> <numa> <cell id='0' cpus='0-4' memory='8388608' unit='KiB' memAccess='shared'/> </numa> </cpu>

    5. Set the emulator in use to be the one built in step 2, so that the emulator configuration looks as follows:

      Copy
      Copied!
                  

      <emulator><path to qemu executable></emulator>

    6. Add a virtio interface using QEMU command line argument entries, so that the new interface snippet looks as follows:

      Copy
      Copied!
                  

      <qemu:commandline> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=charnet1,path=/tmp/sock-virtio0'/> <qemu:arg value='-netdev'/> <qemu:arg value='vhost-user,chardev=charnet1,queues=16,id=hostnet1'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=e4:11:c6:d3:45:f2,bus=pci.0,addr=0x6, page-per-vq=on,rx_queue_size=1024,tx_queue_size=1024'/> </qemu:commandline>

      Warning

      In this snippet, the vhostuser socket file path, the amount of queues, the MAC and the PCIe slot of the virtio device can be configured.

Warning

Hardware vDPA supports switchdev mode only.

  1. Create the ASAP2 environment:

    1. Create the VFs.

    2. Enter switchdev mode.

    3. Set up OVS.

  2. Run the vDPA application:

    Copy
    Copied!
                

    cd $RTE_SDK/examples/vdpa/build ./vdpa -w <VF PCI BDF>,class=vdpa --log-level=pmd,info -- -i

  3. Create a vDPA port via the vDPA application CLI:

    Copy
    Copied!
                

    create /tmp/sock-virtio0 <PCI DEVICE BDF>

    Warning

    The vhostuser socket file path must be the one used when configuring the VM.

  4. Start the VM:

    Copy
    Copied!
                

    virsh start <domain name>

For further information on the vDPA application, visit the Vdpa Sample Application DPDK documentation.

© Copyright 2023, NVIDIA. Last updated on Feb 9, 2024.