Transitional Device
A transitional device is a virtio device which supports drivers conforming to virtio specification 1.x and legacy drivers operating under virtio specification 0.95 (i.e., legacy mode) so that servers with old Linux kernels can still utilize virtio-based technology.
Run the following command on the DPU:
[dpu]# mlxconfig -d 03:00.0 s VIRTIO_NET_EMULATION_PF_PCI_LAYOUT=1 VIRTIO_EMULATION_HOTPLUG_TRANS=1
Add the following parameters to the Linux boot arguments on the guest OS (host OS or VM) side:
virtio_pci.force_legacy=1 intel_iommu=off
Refer to the known limitations note box below.
Cold reboot the host system.
If virtio_pci is a kernel module rather than built-in from the guest OS, run the following command after both the host and BlueField OSes are up:
[host]# modprobe –rv virtio_pci [host]# modprobe –v virtio_pci force_legacy=1
To create a transitional hotplug virtio-net device. Run the following command on the BlueField (with additional -l/--legacy):
[dpu]# virtnet hotplug -i mlx5_0 -f 0x0 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024 -l
Referf to "Virtio-net Deployment" for the rest of configuration of hotplug devices.
Known limitations:
AMD CPU is not supported.
Only kernel versions 3.10 and above are supported. intel_iommu=off is not required for kernel 5.1 and above.
An x86-64 system has only 64K I/O port space which is shared by all peripherals. The virtio transitional device uses I/O BAR. The hotplug device is under one PCIe bridge which is at the emulated PCIe switch downstream port. According to the PCIe specification, the granularity for the bridge I/O window is 4K bytes. If the system cannot satisfy the I/O resource demands by the emulated PCIe switch (depending on the port number of the PCIe switch), the I/O BAR allocation fails. One hot-plug device requires one emulated PCIe switch port. Each emulated PCIe switch port takes 4K bytes of I/O space if the transitional virtio device is supported. Use cat /proc/ioports to check how many I/O port resources are allocated for the host bridge which contains the NIC. The number of supported hotplug transitional virtio device equals: (allocated I/O port space – 4k) / 4k.
1. Referf to "Virtio-net Deployment" for the configuration of sriov device.
2. Add "lm_prov": "kernel" option into controller configuration file and restart the virtio net controller.
[DPU]# cd /opt/mellanox/mlnx_virtnet
[DPU]# cat virtnet.conf
{
...
"lm_prov": "kernel"
...
}
[DPU]# systemctl restart virtio-net-controller.service
3. Create VF devices on host
[HOST]# modprobe virtio_pci
[HOST]# modprobe virtio_net
[HOST]# echo <vf_num> > /sys/bus/pci/devices/<pf_bdf>/sriov_numvfs
4. Bind VF device with virtio_vfio_pci kernel driver
[HOST]# echo <vf_bdf> > /sys/bus/pci/devices/<vf_bdf>/driver/unbind
[HOST]# echo 0x1af4 0x1041 > /sys/bus/pci/drivers/virtio_vfio_pci/new_id
[HOST]# modprobe -v virtio_vfio_pci
[HOST]# lspci -s <vf_bdf> -vvv
...
Kernel driver in use: virtio_vfio_pci
...
5. Add the following option into QEMU cmdline to passthrough VF device into VM
...
-device vfio-pci,host=<vf_bdf>,id=hostdev0,bus=pci.<#BUS_IN_VM>,addr=<#FUNC_IN_VM> \
...
6. Load virtio net driver as legacy mode in the VM
[VM]# modprobe -v virtio_pci force_legacy=1
[VM]# modprobe -v virtio_net
[VM]# lspci -s <vf_bdf_in_vm> -n
00:0a.0 0200: 1af4:1000
7. Check the VF, which should wok as transitional device
[DPU]# virtnet query -p <pf_id> -v <vf_id> | grep transitional
"transitional": 1,
Known limitations:
Host kernel version should be later than v6.9.
if using this feature, the vfe-vdpa-dpdk solutions can not be used anymore, including vfe-vdpa-dpdk live migraion solution.
Libvirt does not support virtio_vfio_pci kernel driver, need to use QEMU cmdline to start the VM.