Transitional Device

NVIDIA BlueField Virtio-net v1.9.0

A transitional device is a virtio device which supports drivers conforming to virtio specification 1.x and legacy drivers operating under virtio specification 0.95 (i.e., legacy mode) so that servers with old Linux kernels can still utilize virtio-based technology.

  1. Run the following command on the DPU:

    Copy
    Copied!
                

    [dpu]# mlxconfig -d 03:00.0 s VIRTIO_NET_EMULATION_PF_PCI_LAYOUT=1 VIRTIO_EMULATION_HOTPLUG_TRANS=1

  2. Add the following parameters to the Linux boot arguments on the guest OS (host OS or VM) side:

    Copy
    Copied!
                

    virtio_pci.force_legacy=1 intel_iommu=off

    Refer to the known limitations note box below.

  3. Cold reboot the host system.

  4. If virtio_pci is a kernel module rather than built-in from the guest OS, run the following command after both the host and BlueField OSes are up:

    Copy
    Copied!
                

    [host]# modprobe –rv virtio_pci [host]# modprobe –v virtio_pci force_legacy=1

  5. To create a transitional hotplug virtio-net device. Run the following command on the BlueField (with additional -l/--legacy):

    Copy
    Copied!
                

    [dpu]# virtnet hotplug -i mlx5_0 -f 0x0 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024 -l

  6. Referf to "Virtio-net Deployment" for the rest of configuration of hotplug devices.

Note

Known limitations:

  • AMD CPU is not supported.

  • Only kernel versions 3.10 and above are supported. intel_iommu=off is not required for kernel 5.1 and above.

  • An x86-64 system has only 64K I/O port space which is shared by all peripherals. The virtio transitional device uses I/O BAR. The hotplug device is under one PCIe bridge which is at the emulated PCIe switch downstream port. According to the PCIe specification, the granularity for the bridge I/O window is 4K bytes. If the system cannot satisfy the I/O resource demands by the emulated PCIe switch (depending on the port number of the PCIe switch), the I/O BAR allocation fails. One hot-plug device requires one emulated PCIe switch port. Each emulated PCIe switch port takes 4K bytes of I/O space if the transitional virtio device is supported. Use cat /proc/ioports to check how many I/O port resources are allocated for the host bridge which contains the NIC. The number of supported hotplug transitional virtio device equals: (allocated I/O port space – 4k) / 4k.

1. Referf to "Virtio-net Deployment" for the configuration of sriov device.

2. Add "lm_prov": "kernel" option into controller configuration file and restart the virtio net controller.

Copy
Copied!
            

[DPU]# cd /opt/mellanox/mlnx_virtnet [DPU]# cat virtnet.conf { ... "lm_prov": "kernel" ... }   [DPU]# systemctl restart virtio-net-controller.service

3. Create VF devices on host

Copy
Copied!
            

[HOST]# modprobe virtio_pci [HOST]# modprobe virtio_net [HOST]# echo <vf_num> > /sys/bus/pci/devices/<pf_bdf>/sriov_numvfs

4. Bind VF device with virtio_vfio_pci kernel driver

Copy
Copied!
            

[HOST]# echo <vf_bdf> > /sys/bus/pci/devices/<vf_bdf>/driver/unbind [HOST]# echo 0x1af4 0x1041 > /sys/bus/pci/drivers/virtio_vfio_pci/new_id [HOST]# modprobe -v virtio_vfio_pci [HOST]# lspci -s <vf_bdf> -vvv ... Kernel driver in use: virtio_vfio_pci ...

5. Add the following option into QEMU cmdline to passthrough VF device into VM

Copy
Copied!
            

... -device vfio-pci,host=<vf_bdf>,id=hostdev0,bus=pci.<#BUS_IN_VM>,addr=<#FUNC_IN_VM> \ ...

6. Load virtio net driver as legacy mode in the VM

Copy
Copied!
            

[VM]# modprobe -v virtio_pci force_legacy=1 [VM]# modprobe -v virtio_net [VM]# lspci -s <vf_bdf_in_vm> -n 00:0a.0 0200: 1af4:1000

7. Check the VF, which should wok as transitional device

Copy
Copied!
            

[DPU]# virtnet query -p <pf_id> -v <vf_id> | grep transitional "transitional": 1,

Note

Known limitations:

  • Host kernel version should be later than v6.9.

  • if using this feature, the vfe-vdpa-dpdk solutions can not be used anymore, including vfe-vdpa-dpdk live migraion solution.

  • Libvirt does not support virtio_vfio_pci kernel driver, need to use QEMU cmdline to start the VM.

© Copyright 2024, NVIDIA. Last updated on Jun 18, 2024.