InfiniBand Network

Linux Kernel Upstream Release Notes v6.5

General supported capabilities:

Supported Capabilities

Unsupported Capabilities

  • VXLAN

  • NVGRE

  • Receive Side Coalescing (RSC)

  • VLAN

  • Multiple and non-default PKyes

  • Head of Queue Lifetime Limit

Partition Keys (PKeys) are used to partition IPoIB communication by mapping a non-default full-membership PKey to index 0, and mapping the default PKey to an index other than zero. Driver’s over-end-points communicate via the PKey is set in index 0. Their communication with the Subnet Agent is done via the default PKey that is not necessarily set in index 0. To enable such behavior, the PKey in index 0 must be in full state.

PKey is a four-digit hexadecimal number specifying the InfiniBand partition key. It can be specified by the user when a non-default PKey is used.

The default PKey (0x7fff) is inserted in block 0 index 0, by default. PKey’s valid values are 0x1 - 0x7fff.

System Requirements

Firmware version:

14/16.23.1020 and higher

Warning

The feature is firmware dependent. If an earlier firmware version is used, traffic may fail as the feature is unsupported and the following event will be displayed in the Event Viewer:

Event ID: 0x0034:

Event message: : Non-default PKey is not supported by FW.

PKey Membership Types

The following are the available PKey’s membership types:

  • Full (default): Members with full membership may communicate with all hosts (members) within the network/partition.

  • Limited/partial: Members with limited membership cannot communicate with other members with limited membership. However, they can communicate between every other combination of membership types (e.g., full + limited, limited + full).

Changing PKey membership

Setting a full membership

ib partition <partition name> member all type full

Setting a limited membership

ib partition <partition name> member all type limited

Changing PKey membership using UFM

ib partition management defmember <limited/full>

Changing the PKey Index

PKey index can be changed using one of the following methods:

  • Subnet Manager (SM) in the Switch

  1. Obtain the partition.conf file.
    - In the switch, the file is located at: vtmp/infiniband-default/var/opensm/partitions.conf
    - In the Linux host, the file is located at: /etc/opensm/partitions.conf

  2. Add ",indx0" for a non-default PKey. If if already exists the default PKey, remove it.
    For example: non-deafult=0x3,ipoib: ALL=full; --> non-deafult=0x3,indx0,ipoib: ALL=full;

  3. Load/save the partition.conf file.

  • UFM

    Add a new full membership PKey with indx0. The newly added PKey will replace the default PKey.

Creating, Deleting or Configuring PKey

PKey can be created, deleted or configured using one of the following methods:

  • Subnet Manager (SM) in the Switch

    Note: To perform any of the actions below, you need to access the switch’s configuration.

    • To create a new PKey, run:

      Copy
      Copied!
                  

      ib partition <partition name> pkey <pkey number>

  • To delete a PKey, run:

    Copy
    Copied!
                

    no ib partition <partition name>

  • To configure the PKey to be IPoIB, run:

    Copy
    Copied!
                

    ib partition <partition name> ipoib force

  • UFM

    PKey can be created, deleted or configured using UFM by adding an extension to the partitions.conf file that is generated by the UFM. The new extension can be added by editing the /opt/ufm/files/conf/partitions.conf.user_ext file according to the desired action (create/delete/configure). The content of this extension file is added to the partitions.conf file upon file synchronization done by the UFM on every logical model change. Synchronization can also be triggered manually by running the /opt/ufm/scripts/sync_partitions_conf.sh script. The script merges the /opt/ufm/files/conf/partitions.conf.user_ext file into the /opt/ufm/files/conf/opensm/partitions.conf file and starts the heavy sweep on the SM.

NIC Teaming allows you to group between one and 32 physical Ethernet/IPoIB network adapters into one or more software-based virtual network adapters. These virtual network adapters provide fast performance and fault tolerance in the event of a network adapter failure.
On Windows Server edition, there is a built-in module that supports teaming and VLAN Ethernet. For more information see here.

One of the existing limitations with Windows OS support is that it does not support NIC teaming solution for IPoIB devices.

Warning

A team can either have only IPoIB members or only Ethernet members.

To overcome these limitations, WinOF-2 driver provides a teaming solution. The supported modes are:

  • Active-Standby mode - In this mode, the user can pick the primary adapter responsible for sending traffic. In the event of a link failure, a failover happens and the standby adapter takes over. User can also tell us to not failback to primary in case of fail-over followed by primary adapter has link up. For this mode, set the failover mode in the mlx5MuxTool.

Please refer to content below on how to configure using custom teaming solution with RSS functionality.

Prerequisites

Warning

IPoIB teaming solution is supported ONLY when using the WinOF-2 mlx5mux driver.

  • Operating Systems: Windows 10 and above

  • Adapter Cards: All supported devices supported by WinOF-2 driver.

  • For using the Mlx5muxtool, users that do not install the full package (HKEY_LOCAL_MACHINE\SOFTWARE\Mellanox\MLNX_WinOF2\InstalledPath) must point this key (InstalledPath) to the location of the mux drivers as the tool searches for the mux drivers files in a folder called "mux" in the folder that define by this key(InstalledPath).

Configuring Command Line Based Teaming

1. Show the help menu. The following command prints out all supported modes and functionalities:

Copy
Copied!
            

mlx5muxtool.exe --help [TEAMING] To list all adapters including teams, use: mlx5muxtool showlist To create a team use: mlx5muxtool create team <Type> <Name> [NoFailBackToPrimary] [IPoIB] Type is one of the following: Aggregate | Failover For IPoIB team, only type 'Failover' is supported   To add adapter to the team use: mlx5muxtool attach team <TeamName> {<Adapter-GUID>} [primary] [SetTeamMacAddress] To remove an adapter from the team use: mlx5muxtool detach team <TeamName> {<Adapter-GUID>} To delete a team use: mlx5muxtool removeteam <TeamName> To query an existing team, use: mlx5muxtool queryteam <TeamName>

Example:

Copy
Copied!
            

mlx5muxtool create team Aggregate MyTeam mlx5muxtool attach team MyTeam {2E9C1992-98B5-43C3-97A0-9993AEAC7F80} mlx5muxtool attach team MyTeam {8D05C52B-BCD6-4FCE-8235-1E90BD334519}

2. Show all the adapter cards (including all created teams already).

Copy
Copied!
            

mlx5muxtool.exe showlist {90F5F52D-4384-4263-BD12-4588CA5CE80A} Mellanox ConnectX-5 Adapter #2 (IPoIB) {62B9661A-17C4-4AF3-AAA1-2B3337FD02E0} Mellanox ConnectX-5 Adapter (IPoIB) {136A1E6F-1168-48D4-B9CC-55EE563D427B} Mellanox ConnectX-6 Adapter (IPoIB) {87B55F92-D573-471B-882C-379773296A6D} Mellanox ConnectX-6 Adapter #2 (IPoIB)

3. Create an empty IPoIB team.

Copy
Copied!
            

mlx5muxtool.exe create team failover MyTeam IPoIB Adding team MyTeam Team created {FED1925F-F88F-4970-B4C3-38AA030874DF}

4. Attach members to the team.

Copy
Copied!
            

mlx5muxtool.exe attach team MyTeam {90F5F52D-4384-4263-BD12-4588CA5CE80A} primary Attaching adapter {90F5F52D-4384-4263-BD12-4588CA5CE80A} to team MyTeam

5. Query the team.

Copy
Copied!
            

mlx5muxtool.exe queryteam MyTeam   Found 1 team(s)   Name : MyTeam GUID : {FED1925F-F88F-4970-B4C3-38AA030874DF} PortType : IPoIB TeamType : Failover MemberCount : 2 Member[0] : {62B9661A-17C4-4AF3-AAA1-2B3337FD02E0} (SLOT 5 Port 2) Member[1] : {90F5F52D-4384-4263-BD12-4588CA5CE80A} (Primary) (SLOT 5 Port 1)

6. Detach members from the team.

Copy
Copied!
            

mlx5muxtool.exe detach team MyTeam {62B9661A-17C4-4AF3-AAA1-2B3337FD02E0} Dettaching adapter {62B9661A-17C4-4AF3-AAA1-2B3337FD02E0} from team MyTeam

7. Remove an entire team.

Copy
Copied!
            

mlx5muxtool.exe removeteam MyTeam Delete team MyTeam Deleting member {90F5F52D-4384-4263-BD12-4588CA5CE80A}


© Copyright 2023, NVIDIA. Last updated on Nov 1, 2023.