IB Router

IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.

The forwarding between the InfiniBand subnets is performed using GRH (global route header) lookup.

IB router capabilities are supported only on SB7780 or SB7880 switch systems which come with the following default configuration:

• L3 capabilities enabled

• 2 SWIDs, with interface 1/1 mapped to infiniband-default and interface 1/2 mapped to infiniband-1

The IB router’s basic functionality includes:

• Removal of current L2 LRH (local routing header)

• Routing table lookup – using GID from GRH

• Building new LRH according to the destination and the routing table

The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.

For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0]}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.

Prerequisites

1. Check system capabilities to make sure IB L3 is supported. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show system capabilities
   IB: Supported, L2, L3, Adaptive Routing
   Max SM nodes: 2048
   IB Max licensed speed: EDR
           

   Note

   Please notice the second line in that output.

2. Configure system profile to multi-switch with 2 SWIDs. Run:

   Copy

   Copied!

               
   
               
   switch (config) # system profile ib num-of-swids 2 no-adaptive-routing ib-router
           

   Note

   Note that some of the interfaces may not be mapped to a SWID.

3. Verify system profile configuration. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show system profile
   Profile:             ib
   Number of SWIDs:     2
   Adaptive Routing:    no
   IB Routing:          yes
           

   Note

   Note the number of SWIDs configured and that IB Routing is set to “yes”.

Configuring IB Router

1. Map an interface to a SWID. Run:

   Copy

   Copied!

               
   
               
   switch (config) # interface ib 1/1 switchport access subnet infiniband-default force
   switch (config) # interface ib 1/2 switchport access subnet infiniband-1 force
           

2. Verify SWID configuration on the aforementioned interfaces. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show interfaces ib status
   Interface      Description      IB Subnet                 Speed                   Current line rate   Logical port state   Physical port state
   ---------      -----------      ---------                 ---------               -----------------   ------------------   -------------------
   IB1/1                           infiniband-default        -                       -                   Down                 Polling
   IB1/2                           infiniband-1              edr                     100.0 Gbps           Initialize           LinkUp
   IB1/3                           -                         -                       -                   -                    -
   ...
           

3. Configure and enable IB router. Run:

   Copy

   Copied!

               
   
               
   switch (config) # ib router
   switch (config) # no ib router shutdown
           

4. Enable IB subnet interface. Run:

   Copy

   Copied!

               
   
               
   switch (config) # no interface ib-subnet infiniband-default shutdown
   switch (config) # no interface ib-subnet infiniband-1 shutdown
           

5. Verify configuration. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show ib router
   Routing state: enabled
    
   IB subnet               Routing enabled
     infiniband-default      enabled
     infiniband-1            enabled
   switch (config) # show interfaces ib-subnet infiniband-default
   infiniband-default state:
     GUID                  : F4:52:14:03:00:6E:F2:8B
     Alias GID             : N/A
     LID                   : 10
     Subnet prefix         : FE:C0:00:00:00:00:00:08
     Physical state        : LinkUp
     Logical state         : Active
     L3 interface state    : Up
           

IP to GID Resolution

1. Go to the following Github: https://github.com/Mellanox/ip2gid

2. Clone the Git repository

3. Compile and run on each node in the fabric

4. Change the device MAC address of the IPoIB device to be based on the alias GID and not the GUID.
   For example, # echo fec0:0000:0000:0003:0014:0500:0000:0001 > /sys/class/net/ib0/set_mac
   where fe:c0:00:00:00:00:00:02:00:14:05:00:00:00:00:01 is the alias GID given by the SM to that node.

5. Add route using "ip route add" command to the relevant hosts.
   # ifconfig ib0 12.0.3.1/24 --> set ip for ib0
   # ip route add 12.0.1.0/24 via 12.3.0.250 --> adding route to hosts with 12.1.xxx.xxx IP
   # ip route add 12.0.2.0/24 via 12.3.0.250 --> adding route to hosts with 12.2.xxx.xxx IP

The IB router expects the subnet prefix to be constructed according to some very specific rules. By default, the command which enables IB routers validates the subnet prefix prior to allowing the change.

The commands which affect subnet prefix checking are as follows:

• ib sm <name> enable – starts SM on this node or any node in cluster

• ib sm subnet-prefix <subnet-prefix> – configures the subnet prefix

• ib sm rtr-aguid-enable <1 | 2> – enables support for alias GIDs as needed by IB routers

When any of these commands is run, while the other two have already been issued, the value of the subnet prefix is checked. If it is not valid, the current commit is rejected and the OpenSM state does not change.

To disable subnet prefix checking

1. Verify the status of subnet prefix override. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show ib sm subnet-prefix-override 
   enable
           

2. If enabled, disable subnet-prefix-override. Run:

   Copy

   Copied!

               
   
               
   switch (config) # ib sm subnet-prefix-override
           

3. Verify configuration. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show ib sm subnet-prefix-override 
   disable
           

ib router

switch (config) # ib router

ib router shutdown

switch (config) # no ib router shutdown

interface ib-subnet

switch (config) # interface ib-subnet infiniband-3

interface ib-subnet shutdown

switch (config) # no interface ib-subnet infiniband-3 shutdown

show ib router

switch (config) # show ib router
Routing state: enabled
IB Subnet                  Routing enabled
  infiniband-default       enabled
  infiniband-1             disabled
  infiniband-2             enabled
  infiniband-3             enabled

show interfaces ib-subnet

switch (config) # show interfaces ib-subnet infiniband-3
infiniband-3 state:
  GUID                  : F4:52:14:03:00:6E:F2:8B
  Alias GID             : N/A
  LID                   : 10
  Subnet prefix         : FE:C0:00:00:00:00:00:08
  Physical state        : LinkUp
  Logical state         : Active
  L3 interface state    : Up