This feature enables users to create VirtIO-net emulated PCIe devices in the system where the NVIDIA® BlueField®-2 DPU is connected. This is done by the virtio-net-controller software module present in the DPU. Virtio-net emulated devices allow users to hot plug up to 16 virtio-net PCIe PF Ethernet NIC devices or 504 virtio-net PCIe VF Ethernet NIC devices in the host system where the DPU is plugged in. 

DPU software also enables users to create virtio block PCIe PF and SR-IOV PCIe VF devices. This is covered in the NVIDIA BlueField SNAP and virtio-blk SNAP Documentation.

Anchor
VirtIO-net Controller VirtIO-net Controller

VirtIO-net Controller

Virtio-net-controller is a systemd service running on the DPU, with a user interface frontend to communicate with the background service. An SF representor is created for each virtio-net device created on the host. Virtio-net controller only uses an SF number ≥1000.  Refer to section "Scalable Functions" for more information. 

Anchor
SystemD Service SystemD Service

SystemD Service

Controller systemd service is enabled by default and runs automatically if VIRTIO_NET_EMULATION_ENABLE is true from mlxconfig.

1. To check controller service status, run:

   Code Block
   language text
   $ systemctl status virtio-net-controller.service

2. To reload the service, make sure to unload virtio-net/virtio-pcie drivers on host. Then run: 

   Code Block
   language text
   $ systemctl restart virtio-net-controller.service

3. To monitor log output of the controller service, run: 

   Code Block
   language text
   $ journalctl -u virtio-net-controller

The controller service has an optional configuration file which allows users to customize several parameters. The configuration file should be defined on the DPU at the following path /opt/mellanox/mlnx_virtnet/virtnet.conf.

This file is read every time the controller starts. Dynamic change of virtnet.conf is not supported. It is defined as a JSON format configuration file. The currently supported options are:

• ib_dev_p0 – RDMA device (e.g., mlx5_0) used to create SF on port 0. This port is the EMU manager when is_lag is 0. Default value is mlx5_0.
• ib_dev_p1 – RDMA device (e.g., mlx5_1) used to create SF on port 1. Default value is mlx5_1.
• ib_dev_lag – RDMA LAG device (e.g., mlx5_bond_0) used to create SF on LAG. Default value is mlx5_bond_0. This port is EMU manager when is_lag is 1. ib_dev_lag and ib_dev_p0/ib_dev_p1 cannot be configured simultaneously.
• ib_dev_for_static_pf – the RDMA device (e.g., mlx5_0) which the static VirtIO PF is created on
• is_lag – specifies whether or not LAG is used. Note that if LAG is used, make sure to use the correct IB dev for static PF.

• pf_mac – base MAC address for static PFs. MACs are automatically assigned with the following pattern: pf_mac→pf_0, pf_mac+1→pf_1, etc.

  Note
  Note that the controller does not validate the MAC address (other than its length). The user must ensure MAC is valid and unique.

• recovery – specifies whether recovery is enabled. If unspecified, recovery is enabled by default. To disable it, set recovery to 0.

• sf_pool_percent – determines the initial SF pool size as the percentage of PF_TOTAL_SF of mlxconfig. Valid range: [0, 100]. For instance, if the value is 5, it means an SF pool with 5% of PF_TOTAL_SF is created. 0 means no SF pool is reserved beforehand (default). 

  Note
  PF_TOTAL_SF is shared by all applications. User must ensure the percent request is guaranteed or else the controller will not be able to reserve the requested SFs resulting in failure.

• sf_pool_force_destroy – specifies whether to destroy the SF pool. When set to 1, the controller destroys the SF pool when stopped/restarted (and the SF pool is recreated if sf_pool_percent is not 0 when starting), otherwise it does not. Default value is 0.

For example, the definition below has all static PFs using mlx5_0 (port 0) as the data path device in a non-lag configuration:

{
  "ib_dev_p0": "mlx5_0",
  "ib_dev_p1": "mlx5_1",
  "ib_dev_for_static_pf": "mlx5_0",
  "is_lag": 0,
  "pf_mac": "00:11:22:33:44:55",
  "recovery": 1,
  "sf_pool_percent": 0,
  "sf_pool_force_destroy": 0
}

The following is an example for LAG configuration:

{
  "ib_dev_lag": "mlx5_bond_0",
  "ib_dev_for_static_pf": "mlx5_bond_0",
  "is_lag": 1,
  "pf_mac": "00:11:22:33:44:55",
  "recovery": 1,
  "sf_pool_percent": 0,
  "sf_pool_force_destroy": 0
}

Anchor
User Frontend User Frontend

User Frontend

To communicate with the service, a user frontend program (virtnet) is installed on the DPU. Run the following command to check its usage: 

# virtnet -h
usage: virtnet [-h] [-v] {hotplug,unplug,list,query,modify,log} ...

Nvidia virtio-net-controller command line interface v1.0.9
 
positional arguments:
  {hotplug,unplug,list,query,modify,log}
                        ** Use -h for sub-command usage
    hotplug             hotplug virtnet device
    unplug              unplug virtnet device
    list                list all virtnet devices
    query               query all or individual virtnet device(s)
    modify              modify virtnet device
    log                 set log level
 
optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

Note that each positional argument has its own help menu as well. For example: 

# virtnet log -h
usage: virtnet log [-h] -l {info,err,debug}
optional arguments:
  -h, --help            show this help message and exit
  -l {info,err,debug}, --level {info,err,debug}
                        log level: info/err/debug

To operate a particular device, either the VUID or device index can be used to locate the device. Both attributes can be fetched from command "virtnet list". For example, to modify the MAC of a specific VF, you may run either of the following commands:

# virtnet modify -p 0 –v 0 device -m 0C:C4:7A:FF:22:98

Or:

# virtnet modify -u <VUID-string> device -m 0C:C4:7A:FF:22:98

Anchor
Controller Recovery Controller Recovery

Controller Recovery

It is possible to recover the control and data planes if communications are interrupted so the original traffic can resume.

Recovery depends on the JSON files stored in /opt/mellanox/mlnx_virtnet/recovery where there is a file that corresponds to each device (either PF or VF). The following is an example of the data stored in these files:

{
  "port_ib_dev": "mlx5_0",
  "pf_id": 0,
  "function_type": "pf",
  "bdf_raw": 26624,
  "device_type": "hotplug",
  "mac": "0c:c4:7a:ff:22:93",
  "pf_num": 0,
  "sf_num": 2000,
  "mq": 1
}

These files should not be modified under normal circumstances. However, if necessary, advanced users may tune settings to meet their requirements. Users are responsible for the validity of the recovery files and should only perform this when the controller is not running.

Anchor
VirtIO-net PF Devices VirtIO-net PF Devices

VirtIO-net PF Devices

This section covers managing virtio-net PCIe PF devices using virtio-net controller.

Anchor
VirtIO-net PF Device Configuration VirtIO-net PF Device Configuration

VirtIO-net PF Device Configuration

1. Run the following command on the DPU:

   Code Block
   language text
   $ mlxconfig -d /dev/mst/mt41686_pciconf0 s INTERNAL_CPU_MODEL=1

2. Add the following kernel boot parameters to the Linux boot arguments:

   Code Block
   language text
   intel_iommu=on iommu=pt pci=realloc

3. Cold reboot the host system.

4. Apply the following configuration on the DPU:

   Code Block
   language text
   $ mst start $ mlxconfig -d /dev/mst/mt41686_pciconf0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 $ mlxconfig -d /dev/mst/mt41686_pciconf0 s \ PCI_SWITCH_EMULATION_ENABLE=1 \ PCI_SWITCH_EMULATION_NUM_PORT=16 \ VIRTIO_NET_EMULATION_ENABLE=1 \ VIRTIO_NET_EMULATION_NUM_VF=0 \ VIRTIO_NET_EMULATION_NUM_PF=0 \ VIRTIO_NET_EMULATION_NUM_MSIX=10 \ ECPF_ESWITCH_MANAGER=1 \ ECPF_PAGE_SUPPLIER=1 \ SRIOV_EN=0 \ PF_SF_BAR_SIZE=10 \ PF_TOTAL_SF=64 $ mlxconfig -d /dev/mst/mt41686_pciconf0.1 s \ PF_SF_BAR_SIZE=10 \ PF_TOTAL_SF=64

5. Cold reboot the host system a second time.

Creating Hotplug VirtIO-net PF Device

Anchor
Creating Hotplug VirtIO-net Device Creating Hotplug VirtIO-net Device

Virtio emulated network PCIe devices are created and destroyed using virtio-net-controller application console. When this application is terminated, all created Virtio-net emulated devices are hot unplugged.

1. Create a hotplug virtio-net device. Run:

   Code Block
   language text
   $ virtnet hotplug -i mlx5_0 -f 0x0 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024

   Note
   The maximum number of virtio-net queues is bound by the minimum of the following numbers: VIRTIO_NET_EMULATION_NUM_MSIX from the command mlxconfig -d <mst_dev> q max_virtq from the command virtnet list

   This creates one hotplug virtio-net device with MAC address 0C:C4:7A:FF:22:93, MTU 1500, and 3 virtio queues with a depth of 1024 entries. This device is uniquely identified by its index. This index is used to query and update device attributes. If the device is created successfully, an output appears similar to the following: 

   Code Block
   language text
   { "bdf": "85:00.0", "vuid": "VNETS1D0F0", "id": 3, "sf_rep_net_device": "en3f0pf0sf2000", "mac": "0C:C4:7A:FF:22:93" }

2. Add the representor port of the device to the OVS bridge and bring it up. Run: 

   Code Block
   language text
   $ ovs-vsctl add-port <bridge> en3f0pf0sf2000 $ ip link set dev en3f0pf0sf2000 up

   Once steps 1-3 are completed, virtio-net device should be available in the host system.

3. To query all the device configurations of virtio-net device that you created, run: 

   Code Block
   language text
   $ virtnet query –p 0

4. To list all the virtio-net devices, run: 

   Code Block
   language text
   $ virtnet list

5. To modify device attributes, for example, changing its MAC address, run: 

   Code Block
   language text
   $ virtnet modify -p 0 device -m 0C:C4:7A:FF:22:98

6. Once usage is complete, to hot-unplug a virtio-net device, run:

   Code Block
   language text
   $ virtnet unplug -p 0

Anchor
Virtio-net SR-IOV VF Devices Virtio-net SR-IOV VF Devices

Virtio-net SR-IOV VF Devices

This section covers managing virtio-net PCIe SR-IOV VF devices using virtio-net-controller.

Anchor
Creating Hotplug VirtIO-net VF Device Creating Hotplug VirtIO-net VF Device

Virtio-net SR-IOV VF Device Configuration

1. On the DPU, make sure virtio-net-controller service is enabled so that it starts automatically. Run:

   Code Block
   language text
   systemctl status virtio-net-controller.service

2. On the host, enable SR-IOV. Please refer to MLNX_OFED documentation under Features Overview and Configuration > Virtualization > Single Root IO Virtualization (SR-IOV) > Setting Up SR-IOV for instructions on how to do that. Make sure the parameters "intel_iommu=on iommu=pt pci=realloc" exist in grub.conf file.

3. It is recommended to add pci=assign-busses to the boot command line when creating more than 127 VFs. Without this option, the following errors might appear from host and the virtio driver will not probe these devices. 

   Code Block
   language text
   pci 0000:84:00.0: [1af4:1041] type 7f class 0xffffff pci 0000:84:00.0: unknown header type 7f, ignoring device

4. Run the following command on the DPU: 

   Code Block
   language text
   mst start && mlxconfig -d /dev/mst/mt41686_pciconf0 s INTERNAL_CPU_MODEL=1

5. Cold reboot the host system.

6. Apply the following configuration on the DPU in three steps to support up to 125 VFs per PF (500 VFs in total).

   1. Code Block
      language text
      $ mst start && mlxconfig -d /dev/mst/mt41686_pciconf0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1

   2. Code Block
      language text
      $ mlxconfig -d /dev/mst/mt41686_pciconf0 s \ PCI_SWITCH_EMULATION_ENABLE=0 \ PCI_SWITCH_EMULATION_NUM_PORT=0 \ VIRTIO_NET_EMULATION_ENABLE=1 \ VIRTIO_NET_EMULATION_NUM_VF=126 \ VIRTIO_NET_EMULATION_NUM_PF=4 \ VIRTIO_NET_EMULATION_NUM_MSIX=4 \ ECPF_ESWITCH_MANAGER=1 \ ECPF_PAGE_SUPPLIER=1 \ SRIOV_EN=1 \ PF_SF_BAR_SIZE=8 \ PF_TOTAL_SF=508 \ NUM_OF_VFS=0

   3. Code Block
      language text
      $ mlxconfig -d /dev/mst/mt41686_pciconf0.1 s PF_TOTAL_SF=1 PF_SF_BAR_SIZE=8

7. Cold reboot the host system.

Anchor
Creating Virtio-net SR-IOV VF Devices Creating Virtio-net SR-IOV VF Devices

Creating Virtio-net SR-IOV VF Devices

1. On the host, make sure the static virtio network device presents. Run:

   Code Block
   language text
   # lspci | grep -i virtio 85:00.3 Network controller: Red Hat, Inc. Virtio network device

2. On the host, make sure virtio_pci and virtio_net are loaded. Run:

   Code Block
   language text
   # lsmod | grep virtio

   The net device should be created:

   Code Block
   language text
   # ethtool -i p7p3 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:85:00.3 supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no

3. To create SR-IOV VF devices on the host, run: 

   Code Block
   language text
   # echo 2 > /sys/bus/pci/drivers/virtio-pci/0000\:85\:00.3/sriov_numvfs

   Note
   When the number of VFs created is high, SR-IOV enablement may take several minutes.

   2 VFs should be created from the host: 

   Code Block
   language text
   # lspci | grep -i virt 85:00.3 Network controller: Red Hat, Inc. Virtio network device 85:04.5 Network controller: Red Hat, Inc. Virtio network device 85:04.6 Network controller: Red Hat, Inc. Virtio network device

4. From the DPU virtio-net controller, run the following command to get VF information.

   Code Block
   language text
   # virnet list { "vf_id": 0, "parent_pf_id": 0, "function_type": "VF", "vuid": "VNETS0D0F2VF1", "bdf": "83:00.6", "sf_num": 3000, "sf_parent_device": "mlx5_0", "sf_rep_net_device": "en3f0pf0sf3000", "sf_rep_net_ifindex": 19, "sf_rdma_device": "mlx5_7", "sf_vhca_id": "0x192", "msix_config_vector": "0x0", "num_msix": 10, "max_queues": 4, "max_queues_size": 256, "net_mac": "5A:94:07:04:F6:1C", "net_mtu": 1500 },

   You may use the pci-bdf to match the PF/VF on the host to the information showing on DPU. 

   To query all the device configurations of the virtio-net device of that VF, run:

   Code Block
   language text
   $ virtnet query -p 0 -v 0

   Add the corresponding SF representor to the OVS bridge and bring it up. Run:

   Code Block
   language text
   # ovs-vsctl add-port <bridge> en3f0pf0sf1004 # ip link set dev en3f0pf0sf1004 up

   Now the VF is functional. 

   Note
   When port MTU (p0/p1 of the DPU) is changed after the controller is started, you must restart controller service. It is not recommended to use jumbo MTUs because that may lead to performance degradation.

5. To destroy SR-IOV VF devices on the host, run: 

   Code Block
   language text
   # echo 0 > /sys/bus/pci/drivers/virtio-pci/0000\:85\:00.3/sriov_numvfs

   Note
   When the command returns from the host OS, it does not necessarily mean the controller finished its operations. Look at controller log from the DPU and make sure you see a log like below before removing VirtIO kernel modules or recreate VFs.  Code Block language text # virtio-net-controller[3544]: [INFO] virtnet.c:617:virtnet_device_vfs_unload: PF(0): Unload (4) VFs finished

   Once VFs are destroyed, created SFs from the DPU side are not destroyed but are saved into the SF pool to be reused later.