InfiniBand Network
General supported capabilities:
Supported Capabilities | Unsupported Capabilities |
|
Partition Keys (PKeys) are used to partition IPoIB communication by mapping a non-default full-membership PKey to index 0, and mapping the default PKey to an index other than zero. Driver’s over-end-points communicate via the PKey is set in index 0. Their communication with the Subnet Agent is done via the default PKey that is not necessarily set in index 0. To enable such behavior, the PKey in index 0 must be in full state.
PKey is a four-digit hexadecimal number specifying the InfiniBand partition key. It can be specified by the user when a non-default PKey is used.
The default PKey (0x7fff) is inserted in block 0 index 0, by default. PKey’s valid values are 0x1 - 0x7fff.
System Requirements | |
Firmware version: | 14/16.23.1020 and higher |
The feature is firmware dependent. If an earlier firmware version is used, traffic may fail as the feature is unsupported and the following event will be displayed in the Event Viewer:
Event ID: 0x0034:
Event message: <Adapter name>: Non-default PKey is not supported by FW.
PKey Membership Types
The following are the available PKey’s membership types:
Full (default): Members with full membership may communicate with all hosts (members) within the network/partition.
Limited/partial: Members with limited membership cannot communicate with other members with limited membership. However, they can communicate between every other combination of membership types (e.g., full + limited, limited + full).
Changing PKey membership | |
Setting a full membership | ib partition <partition name> member all type full |
Setting a limited membership | ib partition <partition name> member all type limited |
Changing PKey membership using UFM | ib partition management defmember <limited/full> |
Changing the PKey Index
PKey index can be changed using one of the following methods:
Subnet Manager (SM) in the Switch
Obtain the partition.conf file.
- In the switch, the file is located at: vtmp/infiniband-default/var/opensm/partitions.conf
- In the Linux host, the file is located at: /etc/opensm/partitions.confAdd ",indx0" for a non-default PKey. If if already exists the default PKey, remove it.
For example: non-deafult=0x3,ipoib: ALL=full; --> non-deafult=0x3,indx0,ipoib: ALL=full;Load/save the partition.conf file.
UFM
Add a new full membership PKey with indx0. The newly added PKey will replace the default PKey.
Creating, Deleting or Configuring PKey
PKey can be created, deleted or configured using one of the following methods:
Subnet Manager (SM) in the Switch
Note: To perform any of the actions below, you need to access the switch’s configuration.To create a new PKey, run:
ib partition <partition name> pkey <pkey number>
To delete a PKey, run:
no ib partition <partition name>
To configure the PKey to be IPoIB, run:
ib partition <partition name> ipoib force
UFM
PKey can be created, deleted or configured using UFM by adding an extension to the partitions.conf file that is generated by the UFM. The new extension can be added by editing the /opt/ufm/files/conf/partitions.conf.user_ext file according to the desired action (create/delete/configure). The content of this extension file is added to the partitions.conf file upon file synchronization done by the UFM on every logical model change. Synchronization can also be triggered manually by running the /opt/ufm/scripts/sync_partitions_conf.sh script. The script merges the /opt/ufm/files/conf/partitions.conf.user_ext file into the /opt/ufm/files/conf/opensm/partitions.conf file and starts the heavy sweep on the SM.