SR-IOV Configuration

To set up SR-IOV, do the following:

1. Install the mstflint tools.

   Copy

   Copied!

               
   
               
   # yum install mstflint
           

2. Check the device’s PCI.

   Copy

   Copied!

               
   
               
   # lspci | grep Mellanox
           

   Example:

   Copy

   Copied!

               
   
               
   00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
           

3. Check if SR-IOV is enabled in the firmware.

   Copy

   Copied!

               
   
               
   mstconfig -d <device pci> q
           

   Example:

   Copy

   Copied!

               
   
               
   # mstconfig -d 00:06.0 q
    
   Device #1:
    
   Device type:	ConnectX3Pro	
   PCI device:	00:06.0	
   Configurations:	                  Current
   SRIOV_EN	                      True(1)
   NUM_OF_VFS	                      8
   LINK_TYPE_P1	                  ETH(2)
   LINK_TYPE_P2	                  IB(1)
   LOG_BAR_SIZE	                  3
   BOOT_PKEY_P1	                  0
   BOOT_PKEY_P2	                  0
   BOOT_OPTION_ROM_EN_P1	          True(1)
   BOOT_VLAN_EN_P1	                  False(0)
   BOOT_RETRY_CNT_P1	              0
   LEGACY_BOOT_PROTOCOL_P1	          PXE(1)
   BOOT_VLAN_P1	                  1
   BOOT_OPTION_ROM_EN_P2	          True(1)
   BOOT_VLAN_EN_P2	                  False(0)
   BOOT_RETRY_CNT_P2	              0
   LEGACY_BOOT_PROTOCOL_P2	          PXE(1)
   BOOT_VLAN_P2	                  1
   IP_VER_P1	                      IPv4(0)
   IP_VER_P2	                      IPv4(0)
           

4. Enable SR-IOV:

   Copy

   Copied!

               
   
               
   mstconfig –d <device pci> s SRIOV_EN=<False|True>
           

5. Configure the needed number of VFs

   Copy

   Copied!

               
   
               
   mstconfig –d <device pci> s NUM_OF_VFS=<NUM>
           

   Warning

   This file will be generated only if IOMMU is set in the grub.conf file (by adding “intel_iommu=on” to /boot/grub/grub.conf file).

6. [mlx4 devices only] Create/Edit the file /etc/modprobe.d/mlx4.conf:

   Copy

   Copied!

               
   
               
   options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2 for IB/ETH],[ 1/2 for IB/ETH]
           

   Example:

   Copy

   Copied!

               
   
               
   options mlx4_core num_vfs=8 port_type_array=1,1
           

7. [mlx5 devices only] Write to the sysfs file the number of needed

   Copy

   Copied!

               
   
               
   echo [num_vfs] > sys/class/net/ib2/device/sriov_numvfs
           

   Example:

   Copy

   Copied!

               
   
               
   # echo 8 > /sys/class/net/ib2/device/sriov_numvfs
           

8. Reboot the driver.

9. Load the driver and verify that the VFs were created.

   Copy

   Copied!

               
   
               
   # lspci | grep mellanox
           

   Example:

   Copy

   Copied!

               
   
               
   00:06.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
   00:06.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
   00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
           

For further information, refer to section "Setting Up SR-IOV" in the MLNX_OFED User Manual.