SR-IOV Configuration

Red Hat Enterprise Linux (RHEL) 8.7 Driver Documentation

To set up SR-IOV, do the following:

  1. Install the mstflint tools.

    Copy
    Copied!
                

    # yum install mstflint

  2. Check the device’s PCI.

    Copy
    Copied!
                

    # lspci | grep Mellanox

    Example:

    Copy
    Copied!
                

    00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

  3. Check if SR-IOV is enabled in the firmware.

    Copy
    Copied!
                

    mstconfig -d <device pci> q

    Example:

    Copy
    Copied!
                

    # mstconfig -d 00:06.0 q   Device #1:   Device type: ConnectX3Pro PCI device: 00:06.0 Configurations: Current SRIOV_EN True(1) NUM_OF_VFS 8 LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 IB(1) LOG_BAR_SIZE 3 BOOT_PKEY_P1 0 BOOT_PKEY_P2 0 BOOT_OPTION_ROM_EN_P1 True(1) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 True(1) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0)

  4. Enable SR-IOV:

    Copy
    Copied!
                

    mstconfig –d <device pci> s SRIOV_EN=<False|True>

  5. Configure the needed number of VFs

    Copy
    Copied!
                

    mstconfig –d <device pci> s NUM_OF_VFS=<NUM>

    Warning

    This file will be generated only if IOMMU is set in the grub.conf file (by adding “intel_iommu=on” to /boot/grub/grub.conf file).

  6. [mlx4 devices only] Create/Edit the file /etc/modprobe.d/mlx4.conf:

    Copy
    Copied!
                

    options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2 for IB/ETH],[ 1/2 for IB/ETH]

    Example:

    Copy
    Copied!
                

    options mlx4_core num_vfs=8 port_type_array=1,1

  7. [mlx5 devices only] Write to the sysfs file the number of needed

    Copy
    Copied!
                

    echo [num_vfs] > sys/class/net/ib2/device/sriov_numvfs

    Example:

    Copy
    Copied!
                

    # echo 8 > /sys/class/net/ib2/device/sriov_numvfs

  8. Reboot the driver.

  9. Load the driver and verify that the VFs were created.

    Copy
    Copied!
                

    # lspci | grep mellanox

    Example:

    Copy
    Copied!
                

    00:06.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] 00:06.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

For further information, refer to section "Setting Up SR-IOV" in the MLNX_OFED User Manual.

© Copyright 2023, NVIDIA. Last updated on Feb 4, 2024.