APPENDIX—Skyway Multi-PKEY Configuration With Traffic Isolation
Example of OpenSM partitions.conf file:
cat /etc/opensm/partitions.conf
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
The “ALL” will configure all the HCA guids in the fabric, including the vGUIDs for both the default and non-default PKEYs.
The customer needs to isolate the traffic per host. Meaning all HCA GUIDs should not members of the non-default partitions.
Therefore, they added the actual needed HCA GUIDs to the non-default partition and included the 8 skyway GUIDs.
Example:
Test=0x1, ipoib, defmember=full : ALL_SWITCHES=full,SELF=full; 0x…..1, 0x……2, skyway guid1, skyway guid2, …., skyway guid8;
What is needed is all Skyway GUIDs including the vGUIDs.
Current way to get them:
Configure the partitions.conf file on the SM node as below.
cat /etc/opensm/partitions.conf Default=
0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;Run ibdiagnet on one of the IB host.
Run the below to get all skyway GUIDs from the ibdiagnet2.vports_pkey file.
# cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep <skyway host name> | awk -F
"guid="'{print $2}'| paste -s -d","| sed -e's/,/, /g'> filenameLab example:
# cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep skyway | awk -F
"guid="'{print $2}'| paste -s -d","| sed -e's/,/, /g'> /tmp/skyway_guids.txtEdit this file to add the needed partitions.conf configuration and replace it with the old one.
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ipoib, defmember=full : ALL_SWITCHES, SELF, 0x58a2e10300470a66,0x58a2e103004709ca, 0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260
0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260