NVIDIA Docs Hub Homepage NVIDIA Networking Networking Software Red Hat Enterprise Linux (RHEL) 9.0 Driver Documentation SR-IOV Configuration

SR-IOV Configuration

To set up SR-IOV, do the following:

Install the mstflint tools.

Copy
Copied!

            
            # yum install mstflint

Check the device’s PCI.

Copy
Copied!

            
            # lspci | grep Mellanox

Example:

Copy
Copied!

            
            00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

Check if SR-IOV is enabled in the firmware.

Copy
Copied!

            
            mstconfig -d <device pci> q

Example:

Copy
Copied!

            
            # mstconfig -d 00:06.0 q
 
Device #1:
 
Device type:	ConnectX3Pro	
PCI device:	00:06.0	
Configurations:	                  Current
SRIOV_EN	                      True(1)
NUM_OF_VFS	                      8
LINK_TYPE_P1	                  ETH(2)
LINK_TYPE_P2	                  IB(1)
LOG_BAR_SIZE	                  3
BOOT_PKEY_P1	                  0
BOOT_PKEY_P2	                  0
BOOT_OPTION_ROM_EN_P1	          True(1)
BOOT_VLAN_EN_P1	                  False(0)
BOOT_RETRY_CNT_P1	              0
LEGACY_BOOT_PROTOCOL_P1	          PXE(1)
BOOT_VLAN_P1	                  1
BOOT_OPTION_ROM_EN_P2	          True(1)
BOOT_VLAN_EN_P2	                  False(0)
BOOT_RETRY_CNT_P2	              0
LEGACY_BOOT_PROTOCOL_P2	          PXE(1)
BOOT_VLAN_P2	                  1
IP_VER_P1	                      IPv4(0)
IP_VER_P2	                      IPv4(0)

Enable SR-IOV:

Copy
Copied!

            
            mstconfig –d <device pci> s SRIOV_EN=<False|True>

Configure the needed number of VFs
Copy

Copied!
```
            
            mstconfig –d <device pci> s NUM_OF_VFS=<NUM>
        
```
Warning

This file will be generated only if IOMMU is set in the grub.conf file (by adding “intel_iommu=on” to /boot/grub/grub.conf file).

[mlx4 devices only] Create/Edit the file /etc/modprobe.d/mlx4.conf:

Copy
Copied!

            
            options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2 for IB/ETH],[ 1/2 for IB/ETH]

Example:

Copy
Copied!

            
            options mlx4_core num_vfs=8 port_type_array=1,1

[mlx5 devices only] Write to the sysfs file the number of needed

Copy
Copied!

            
            echo [num_vfs] > sys/class/net/ib2/device/sriov_numvfs

Example:

Copy
Copied!

            
            # echo 8 > /sys/class/net/ib2/device/sriov_numvfs

Reboot the driver.

Load the driver and verify that the VFs were created.

Copy
Copied!

            
            # lspci | grep mellanox

Example:

Copy
Copied!

            
            00:06.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
00:06.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

For further information, refer to section "Setting Up SR-IOV" in the MLNX_OFED User Manual.