Multiple Interfaces over Non-Default PKeys Support

Operating Systems: Windows Server 2012, Windows Server 2012 R2 and Windows Server 2016.

OpenSM enables the configuration of partitions (PKeys) in an InfiniBand fabric. IPoIB supports the creation of multiple interfaces via the part_man tool. Each of those interfaces can be configured to use a different partition from the ones that were configured for OpenSM. This can allow partitioning of the IPoIB traffic between the different virtual IPoIB interfaces.

To create a new interface on a new PKey on a native Windows machine:

1. Configure OpenSM to recognize the partition you would like to add.
   For further details please refer to the section titled “Partitions” in Mellanox OFED for Linux User Manual.

2. Create a new interface using the part_man tool.
   For further details please refer to part_man - Virtual IPoIB Port Creation Utility.

3. Assign Port IPs to the new interfaces.
   For further details please refer to Assigning Port IP After Installation in the Ethernet Network section.

   Warning

   Make sure the OpenSM using the partitions configuration, and the new interfaces were configured to run over the same physical port.

To create a new interface on a new PKey on a Windows virtual machine over a Linux host:

On the Linux host:

1. Configure the OpenSM to recognize the partition you would like to add.
   For further details please refer to the section titled “Partitions” in Mellanox OFED for Linux User Manual.

2. Map the physical PKey table to the virtual PKey table used by the VM.
   For further details please refer to the section titled “Partitioning IPoIB Communication using PKeys” in Mellanox OFED for Linux User Manual.

On the Windows VM:

1. Create a new interface using the part_man tool.
   For further details please refer to part_man - Virtual IPoIB Port Creation Utility.

2. Assign Port IPs to the new interfaces.

   For further details please refer to Assigning Port IP After Installation in the Ethernet Network section.

   Warning

   Make sure the OpenSM using the partitions configuration, the physical-to-virtual PKey table mapping and the new interfaces were all configured over the same physical port.

To assign a non-default PKey to the physical IPoIB port on a Windows virtual machine over a Linux host:

On the Windows VM:

1. Disable the driver on the port or disable the bus driver with all the ports it carries through the device manger.

On the Linux host:

1. Configure the OpenSM to recognize the partition you would like to add.
   For further details please refer to the section titled “Partitions” in Mellanox OFED for Linux User Manual.

2. Map the physical PKey table to the virtual PKey table used by the VM in the following way:

   1. Map the physical Pkey index you would like to use for the physical port to index 0 in the virtual Pkey table.

   2. Map the physical PKey index of the default PKey (index 0) to any index (for example: index1) in the virtual PKey table.

For further details please refer to the section titled “Partitioning IPoIB Communication using PKeys” in Mellanox OFED for Linux User Manual.

On the Windows VM:

1. Enable the drivers which were disabled.

To change a configuration of an existing port:

1. Disable the driver on the port affected by the change you would like to make (or disable the bus driver with all the ports it carries) through the device manger in Windows OS.

2. If required, configure the OpenSM to recognize the partition you would like to add or change.
   For further details please refer to the section titled “Partitions” in Mellanox OFED for Linux User Manual.

3. If the change is on a VM over a Linux host, map the physical PKey table to the virtual PKey table as required.
   For further details please refer to the section titled “Partitioning IPoIB Communication using PKeys” in Mellanox OFED for Linux User Manual.

4. Enable the drivers you disabled in Windows OS.