NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance v8.2.2302 LTS
NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance v8.2.2302 LTS

APPENDIX—Skyway Multi-PKEY Configuration With Traffic Isolation

Example of OpenSM partitions.conf file:

Copy
Copied!
            

cat /etc/opensm/partitions.conf  Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full; Test=0x1, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

The “ALL” will configure all the HCA guids in the fabric, including the vGUIDs for both the default and non-default PKEYs.

The customer needs to isolate the traffic per host. Meaning all HCA GUIDs should not members of the non-default partitions.

Therefore, they added the actual needed HCA GUIDs to the non-default partition and included the 8 skyway GUIDs.

Example:

Copy
Copied!
            

Test=0x1, ipoib, defmember=full :  ALL_SWITCHES=full,SELF=full; 0x…..1, 0x……2, skyway guid1, skyway guid2, …., skyway guid8;

What is needed is all Skyway GUIDs including the vGUIDs.

Current way to get them:

  1. Configure the partitions.conf file on the SM node as below.

    Copy
    Copied!
                

    cat /etc/opensm/partitions.conf  Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

  2. Run ibdiagnet on one of the IB host.

  3. Run the below to get all skyway GUIDs from the ibdiagnet2.vports_pkey file.

    Copy
    Copied!
                

    # cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep <skyway host name> | awk -F "guid=" '{print $2}' | paste -s -d "," | sed -e 's/,/, /g' > filename

    Lab example:

    Copy
    Copied!
                

    # cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep skyway | awk -F "guid=" '{print $2}' | paste -s -d "," | sed -e 's/,/, /g' > /tmp/skyway_guids.txt

  4. Edit this file to add the needed partitions.conf configuration and replace it with the old one.

Copy
Copied!
            

Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full; Test=0x1, ipoib, defmember=full : ipoib, defmember=full : ALL_SWITCHES, SELF, 0x58a2e10300470a66,0x58a2e103004709ca, 0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260

Copy
Copied!
            

0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260

© Copyright 2024, NVIDIA. Last updated on Aug 4, 2024.