APPENDIX—Skyway Multi-PKEY Configuration With Traffic Isolation
This apprendix describes a way to configure the InfiniBand cluster connected to Skyway to support traffic isolation per HCA/host.
The “ALL” option will configure all the HCA GUIDs in the fabric, including the vGUIDs for both the default and non-default PKEYs.
Example of OpenSM partitions.conf file
cat /etc/opensm/partitions.conf
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
The traffic per host needs to be isolated (i.e., all HCA GUIDs should not be members of the non-default partitions). Therefore, the actual required HCA GUIDs need to be added to the non-default partition and included the 8 Skyway GUIDs.
Example:
Test=0x1, ipoib, defmember=full : ALL_SWITCHES=full,SELF=full; 0x…..1, 0x……2, skyway guid1, skyway guid2, …., skyway guid8;
All Skyway GUIDs including the vGUIDs are needed:
Configure the partitions.conf file on the SM node.
cat /etc/opensm/partitions.conf Default=
0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;Run ibdiagnet on one of the InfiniBand host.
Get all Skyway GUIDs from the ibdiagnet2.vports_pkey file:
# cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep <skyway host name> | awk -F
"guid="'{print $2}'| paste -s -d","| sed -e's/,/, /g'> filenameLab example:
# cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep skyway | awk -F
"guid="'{print $2}'| paste -s -d","| sed -e's/,/, /g'> /tmp/skyway_guids.txtEdit this file to add the needed partitions.conf configuration and replace it with the old one.
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ipoib, defmember=full : ALL_SWITCHES, SELF, 0x58a2e10300470a66,0x58a2e103004709ca, 0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260
0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260