Single Root IO Virtualization (SR-IOV)
SR-IOV InfiniBand is at beta level.
Single Root IO Virtualization (SR-IOV) is a technology that allows a physical PCIe device to present itself multiple times through the PCIe bus. This technology enables multiple virtual instances of the device with separate resources. Mellanox adapters are capable of exposing in ConnectX-4 onwards adapter cards up to 63/127 virtual instances called Virtual Functions (VFs) depending on the firmware capabilities. These Virtual Functions can then be provisioned separately. Each VF can be seen as an addition device connected to the Physical Function. It shares the same resources with the Physical Function.
SR-IOV is commonly used in conjunction with an SR-IOV enabled hypervisor to provide virtual machines direct hardware access to network resources, hence increasing its performance.
In this chapter we will demonstrate setup and configuration of SR-IOV in a ESXi environment using Mellanox ConnectX® adapter cards family.
To set up an SR-IOV environment, the following is required:
nmlx5_core Driver
A server/blade with an SR-IOV-capable motherboard BIOS
Mellanox ConnectX® Adapter Card family with SR-IOV capability
Hypervisor that supports SR-IOV such as: ESXi6.5
For further information, see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-CC021803-30EA-444D-BCBE-618E0D836B9F.html
Configuring SR-IOV for ConnectX-4 Onward
To configure SR-IOV for ConnectX-4 onward, perform the following steps:
Install the MLNX-NATIVE-ESX-<adapter card> driver for ESXi that supports SR-IOV.
Download the MFT package. Go to:
www.mellanox.com → Products → Software → InfiniBand/VPI Drivers → MFT (http://www.mellanox.com/page/management_tools)Install MFT.
# esxcli software vib install -v <MST Vib> # esxcli software vib install -v <MFT Vib>
Reboot system.
Start the mst driver.
# /opt/mellanox/bin/mst start
Check if SR-IOV is enabled in the firmware.
/opt/mellanox/bin/mlxconfig -d /dev/mst/mt4115_pciconf0 q Device #
1
: ---------- Device type: ConnectX4 PCI device: /dev/mst/mt4115_pciconf0 Configurations: Current SRIOV_EN1
NUM_OF_VFS8
FPP_EN1
If not, use mlxconfig to enable it.
Note: The example below shows how to enable SR-IOV and allow the creation of 16 VFs.
mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=
1
NUM_OF_VFS=16
Validate the configuration set.
# /opt/mellanox/bin/mlxconfig -e -d /dev/mst/mt4115_pciconf0 query | grep -e Config -e SRIOV_EN -e NUM_OF_VFS Configurations: Default Current Next Boot NUM_OF_VFS
0
0
0
SRIOV_EN False(0
) False(0
) False(0
)Set the number of Virtual Functions you need to create for the PF using the max_vfs module parameter.
Warningmax_vfs is a module parameter which expects an array on numbers where each number in the array indicates how many VFs should be created on that port.
The number of elements in this array is based on the “supported_num_ports” module parameter.
The example below shows the creation of 8 VFs on the first port and 7 on the second port.
esxcli system module parameters set -m nmlx5_core -p
"max_vfs=8,7"
Reboot the server.
WarningThe number of max_vf is set per port. See the nmlx5_core Module Parameters table in the introduction, for more information.