IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.
The forwarding between the InfiniBand subnets is performed using GRH (global route header) lookup.
IB router capabilities are supported only on SB7780 or SB7880 switch systems which come with the following default configuration:
L3 capabilities enabled
2 SWIDs, with interface 1/1 mapped to infiniband-default and interface 1/2 mapped to infiniband-1
The IB router’s basic functionality includes:
Removal of current L2 LRH (local routing header)
Routing table lookup – using GID from GRH
Building new LRH according to the destination and the routing table
The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.
For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0]}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.
For more information on IB router architecture and functionality, please refer to the community post “IB Router Architecture and Functionality”.
IB router requires HCA configuration such as SM, partition key, MPI, GID translation, and more. To learn more about these configurations, please refer to the community post “HowTo Configure IB Routers”.
Prerequisites
Check system capabilities to make sure IB L3 is supported. Run:
switch(config) # show system capabilities IB: Supported, L2, L3, Adaptive Routing Max SM nodes:2048IB Max licensed speed: EDRWarningPlease notice the second line in that output.
Configure system profile to multi-switch with 2 SWIDs. Run:
switch(config) # system profile ib num-of-swids2no-adaptive-routing ib-routerWarningNote that some of the interfaces may not be mapped to a SWID.
Verify system profile configuration. Run:
switch(config) # show system profile Profile: ib Number of SWIDs:2Adaptive Routing: no IB Routing: yesWarningNote the number of SWIDs configured and that IB Routing is set to “yes”.
Configuring IB Router
Map an interface to a SWID. Run:
switch(config) #interfaceib1/1switchport access subnet infiniband-defaultforceswitch(config) #interfaceib1/2switchport access subnet infiniband-1forceVerify SWID configuration on the aforementioned interfaces. Run:
switch(config) # show interfaces ib status Interface Description IB Subnet Speed Current line rate Logical port state Physical port state --------- ----------- --------- --------- ----------------- ------------------ ------------------- IB1/1infiniband-default- - Down Polling IB1/2infiniband-1edr100.0Gbps Initialize LinkUp IB1/3- - - - - ...Configure and enable IB router. Run:
switch(config) # ib routerswitch(config) # no ib router shutdownEnable IB subnet interface. Run:
switch(config) # nointerfaceib-subnet infiniband-defaultshutdownswitch(config) # nointerfaceib-subnet infiniband-1shutdownVerify configuration. Run:
switch(config) # show ib router Routing state: enabled IB subnet Routing enabled infiniband-defaultenabled infiniband-1enabledswitch(config) # show interfaces ib-subnet infiniband-defaultinfiniband-defaultstate: GUID : F4:52:14:03:00:6E:F2:8B Alias GID : N/A LID :10Subnet prefix : FE:C0:00:00:00:00:00:08Physical state : LinkUp Logical state : Active L3interfacestate : Up
IP to GID Resolution
Go to the following Github: https://github.com/Mellanox/ip2gid
Clone the Git repository
Compile and run on each node in the fabric
Change the device MAC address of the IPoIB device to be based on the alias GID and not the GUID.
For example, # echo fec0:0000:0000:0003:0014:0500:0000:0001 > /sys/class/net/ib0/set_mac
where fe:c0:00:00:00:00:00:02:00:14:05:00:00:00:00:01 is the alias GID given by the SM to that node.Add route using "ip route add" command to the relevant hosts.
# ifconfig ib0 12.0.3.1/24 --> set ip for ib0
# ip route add 12.0.1.0/24 via 12.3.0.250 --> adding route to hosts with 12.1.xxx.xxx IP
# ip route add 12.0.2.0/24 via 12.3.0.250 --> adding route to hosts with 12.2.xxx.xxx IP
The IB router expects the subnet prefix to be constructed according to some very specific rules. By default, the command which enables IB routers validates the subnet prefix prior to allowing the change.
The commands which affect subnet prefix checking are as follows:
ib sm <name> enable – starts SM on this node or any node in cluster
ib sm subnet-prefix <subnet-prefix> – configures the subnet prefix
ib sm rtr-aguid-enable <1 | 2> – enables support for alias GIDs as needed by IB routers
When any of these commands is run, while the other two have already been issued, the value of the subnet prefix is checked. If it is not valid, the current commit is rejected and the OpenSM state does not change.
To disable subnet prefix checking
Verify the status of subnet prefix override. Run:
switch(config) # show ib sm subnet-prefix-override enableIf enabled, disable subnet-prefix-override. Run:
switch(config) # ib sm subnet-prefix-overrideVerify configuration. Run:
switch(config) # show ib sm subnet-prefix-override disable
ib router
ib router Enables the set of commands that allow control of IB router functionality. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.0500 | |
Example | switch (config) # ib router | |
Related Commands | system profile | |
Notes | ||
ib router shutdown
ib router shutdown Disables IB router. | ||
Syntax Description | N/A | |
Default | Disabled | |
Configuration Mode | config | |
History | 3.6.0500 | |
Example | switch (config) # no ib router shutdown | |
Related Commands | ||
Notes | This command does not clear IB router configuration | |
interface ib-subnet
interface ib-subnet <swid-name> Creates routing on IB router subnet. | ||
Syntax Description | swid-name | Name of the SWID: infiniband-default, infiniband-1...infiniband-5 |
Default | N/A | |
Configuration Mode | config | |
History | 3.6.0500 | |
Example | switch (config) # interface ib-subnet infiniband-3 | |
Related Commands | system profile | |
Notes | The maximum number of SWIDs depends on the number of SWIDs defined in the profile | |
interface ib-subnet shutdown
interface ib-subnet <swid-name> shutdown Disables routing on IB router subnet. | ||
Syntax Description | swid-name | Name of the SWID: infiniband-default, infiniband-1...infiniband-5 |
shutdown | Admin down on router interface | |
Default | Disabled | |
Configuration Mode | config | |
History | 3.6.0500 | |
Example | switch (config) # no interface ib-subnet infiniband-3 shutdown | |
Related Commands | ||
Notes | ||
show ib router
show ib router Displays current IB router functionality. | ||
Syntax Description | N/A | |
Default | N/A | |
Configuration Mode | Any command mode | |
History | 3.6.0500 | |
Example |
| |
Related Commands | ||
Notes | ||
show interfaces ib-subnet
show interfaces ib-subnet [<swid-name>] [brief] Displays statistics of one or all IB subnets with enabled IB routing. | ||
Syntax Description | swid-name | Name of the SWID: infiniband-default, infiniband-1...infiniband-5 |
brief | Displays output in a table format | |
Default | Disabled | |
Configuration Mode | config | |
History | 3.6.0500 | |
Example | switch (config) # show interfaces ib-subnet infiniband-3 | |
Related Commands | ||
Notes | ||