SR-IOV Configuration

To set up SR-IOV, do the following:

1. Download mstflint

   Copy

   Copied!

               
   
               
   # zypper install mstflint
           

2. Check the device’s PCI

   Copy

   Copied!

               
   
               
   lspci | grep Mellanox
           

   Example:

   Copy

   Copied!

               
   
               
   24:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
           

3. Check if SR-IOV is enabled in the in the firmware.

   Copy

   Copied!

               
   
               
   mstconfig -d <device pci> q
           

   Example:

   Copy

   Copied!

               
   
               
   # mstconfig -d 00:06.0 q
   Device #1:
    
   Device type:      ConnectX3Pro     
   PCI device:       00:06.0    
    
   Configurations:            Current
   SRIOV_EN                   True(1)
   NUM_OF_VFS                 8
   LINK_TYPE_P1               ETH(2)
   LINK_TYPE_P2               IB(1)
   LOG_BAR_SIZE               3
   BOOT_PKEY_P1               0
   BOOT_PKEY_P2               0
   BOOT_OPTION_ROM_EN_P1      True(1)
   BOOT_VLAN_EN_P1            False(0)
   BOOT_RETRY_CNT_P1          0
   LEGACY_BOOT_PROTOCOL_P1    PXE(1)
   BOOT_VLAN_P1               1
   BOOT_OPTION_ROM_EN_P2      True(1)
   BOOT_VLAN_EN_P2            False(0)
   BOOT_RETRY_CNT_P2          0
   LEGACY_BOOT_PROTOCOL_P2    PXE(1)
   BOOT_VLAN_P2               1
   IP_VER_P1                  IPv4(0)
   IP_VER_P2                  IPv4(0)           
           

4. Enable SR-IOV:

   Copy

   Copied!

               
   
               
   mstconfig –d <device pci> s SRIOV_EN=<False|True>
           

5. Configure the needed number of VFs.

   Copy

   Copied!

               
   
               
   mstconfig –d <device pci> s NUM_OF_VFS=<NUM>
           

   Warning

   This file will be generated only if IOMMU is set in grub configuration. (by adding “intel_iommu=on” to /etc/default/grub).

6. [mlx4 devices only] Edit the file /etc/modprobe.d/mlx4.conf:

   Copy

   Copied!

               
   
               
   options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2
           

   Example:

   Copy

   Copied!

               
   
               
   options mlx4_core num_vfs=8 port_type_array=1,1
           

7. [mlx5 devices only] Write to the sysfs file the number of needed

   Copy

   Copied!

               
   
               
   echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs 
           

8. Reboot the

9. Load the driver and verify that the VFs were

   Copy

   Copied!

               
   
               
   lspci | grep mellanox  
           

   Example:

   Copy

   Copied!

               
   
               
   24:00.0 Ethernet controller: Mellanox Technologies MT28800 Family
   [ConnectX-5 Ex]
   24:00.1 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
   24:00.2 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
   24:00.3 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
   24:00.4 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
   24:00.5 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
           

For further information, refer to section "Setting Up SR-IOV" in the MLNX_OFED User Manual.