NVIDIA Docs Hub Homepage NVIDIA Networking Networking Software Switch Software NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance v8.2.2400 LTS Configuring Partition Keys (PKEYs)

Configuring Partition Keys (PKEYs)

Configure partition keys on the Skyway appliance.

Copy
Copied!

            
            #### pkey_id = [1, 7ffe]  (hex)

Configuring partition keys on the subnet manager.

Copy
Copied!

            
            ib switch (config) # ib partition <partition name> pkey 0x<pkey_id>
ib switch (config) # ib partition  <partition name> ipoib
ib switch (config) # ib partition <partition name> member ALL type full

Example:

Copy
Copied!

            
            ib switch (config) # ib partition pkey_0x1 pkey 0x1
ib switch (config) # ib partition pkey_0x1 ipoib
ib switch (config) # ib partition pkey_0x1 member ALL type full

Configure PKEY on the InfiniBand host.

Copy
Copied!

            
            #### pkey_full = pkey_id (hex) + 8000 (hex)
echo "0x<pkey_full>" > /sys/class/net/ib<interface_number>/create_child                                  
ifconfig ib<interface_number>.< pkey_full> up
ifconfig  ib<interface_number>.< pkey_full>  <Ib host pkey port ip> netmask <netmask>
ip route add <eth subnet> via <Gw pkey port virtual ip>
ip route add <gw eth subnet> via <Gw pkey port virtual ip>

Example:

Copy
Copied!

            
            echo "0x8001" > /sys/class/net/ib1/create_child
ifconfig ib1.8001 up
ifconfig ib1.8001 111.222.62.11 netmask 255.255.255.0
ip route add 2.2.2.0/24 via 111.222.62.1
ip route add 192.168.1.0/24 via 111.222.62.1

Note

Note that this example shows how to configure PKEY that has an ID of 0x1 (0x8001 in hex === 8000+pkeyID). When configuring other PKEYs on the InfiniBand host, make sure to add 8000 to the PKEY ID.

Configure the Skyway appliance.

Copy
Copied!

            
            gateway (config) # interface ib port-channel 1 pkey 0x<pkey_id>
gateway (config interface ib port-channel 1 pkey 0x<pkey_id>) # exit
gateway (config) # interface ib port-channel 1 pkey 0x<pkey_id> ip address <Gw pkey port ip> <netmask>
gateway (config) # interface ib port-channel 1 pkey 0x<pkey_id> virtual ip address <Gw pkey port virtual ip> <netmask>
gateway (config) # configuration write

Example:

Copy
Copied!

            
            gateway (config) # interface ib port-channel 1 pkey 0x1
gateway (config interface ib port-channel 1 pkey 0x1) # exit
gateway (config) # interface ib port-channel 1 pkey 0x1 ip address 111.222.62.2 255.255.255.0
gateway (config) # interface ib port-channel 1 pkey 0x1 virtual ip address 111.222.62.1 255.255.255.0
gateway (config) # configuration write

Add routes on the Ethernet host.

Copy
Copied!

            
            ip route add <pkey subnet>/<netmask> via <eth router port ip>

Example:

Copy
Copied!

            
            ip route add 111.222.62.0/24 via <eth router port ip>

Add routes on the Ethernet router.

Copy
Copied!

            
            ip route <pkey subnet>/<netmask>  <gw eth pc ip>

Example:

Copy
Copied!

            
            ip route 111.222.62.0 /24 <gw eth pc>

Note

The configuration mentioned above is an example of when traffic from each HCA can tun on any PKEY. In case it is needed to isolate the traffic per host, all HCA GUIDs should not be members of the non-default partitions.

In case such a configuration is requried only for a subset of PKEYs in a cluster, the exact GUIDs (and VGIDs) need to be specified in the same configuration.

See the following apprendix, for an example.