NVIDIA MLNX-OS User Manual v3.10.4100 (LTS)
v8.2.2200

IB Router

IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.

The forwarding between the InfiniBand subnets is performed using GRH (global route header) lookup.

IB router capabilities are supported only on SB7780 or SB7880 switch systems which come with the following default configuration:

  • L3 capabilities enabled

  • 2 SWIDs, with interface 1/1 mapped to infiniband-default and interface 1/2 mapped to infiniband-1

The IB router’s basic functionality includes:

  • Removal of current L2 LRH (local routing header)

  • Routing table lookup – using GID from GRH

  • Building new LRH according to the destination and the routing table

The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.

Site-Local_Unicast_GID_Format.png

For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0]}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.

image2021-2-1_12-2-50.png

Warning

For more information on IB router architecture and functionality, please refer to the community post “IB Router Architecture and Functionality”.

Warning

IB router requires HCA configuration such as SM, partition key, MPI, GID translation, and more. To learn more about these configurations, please refer to the community post “HowTo Configure IB Routers”.

Prerequisites

  1. Check system capabilities to make sure IB L3 is supported. Run:

    Copy
    Copied!
                

    switch (config) # show system capabilities IB: Supported, L2, L3, Adaptive Routing Max SM nodes: 2048 IB Max licensed speed: EDR

    Warning

    Please notice the second line in that output.

  2. Configure system profile to multi-switch with 2 SWIDs. Run:

    Copy
    Copied!
                

    switch (config) # system profile ib num-of-swids 2 no-adaptive-routing ib-router

    Warning

    Note that some of the interfaces may not be mapped to a SWID.

  3. Verify system profile configuration. Run:

    Copy
    Copied!
                

    switch (config) # show system profile Profile: ib Number of SWIDs: 2 Adaptive Routing: no IB Routing: yes

    Warning

    Note the number of SWIDs configured and that IB Routing is set to “yes”.

Configuring IB Router

  1. Map an interface to a SWID. Run:

    Copy
    Copied!
                

    switch (config) # interface ib 1/1 switchport access subnet infiniband-default force switch (config) # interface ib 1/2 switchport access subnet infiniband-1 force

  2. Verify SWID configuration on the aforementioned interfaces. Run:

    Copy
    Copied!
                

    switch (config) # show interfaces ib status Interface Description IB Subnet Speed Current line rate Logical port state Physical port state --------- ----------- --------- --------- ----------------- ------------------ ------------------- IB1/1 infiniband-default - - Down Polling IB1/2 infiniband-1 edr 100.0 Gbps Initialize LinkUp IB1/3 - - - - - ...

  3. Configure and enable IB router. Run:

    Copy
    Copied!
                

    switch (config) # ib router switch (config) # no ib router shutdown

  4. Enable IB subnet interface. Run:

    Copy
    Copied!
                

    switch (config) # no interface ib-subnet infiniband-default shutdown switch (config) # no interface ib-subnet infiniband-1 shutdown

  5. Verify configuration. Run:

    Copy
    Copied!
                

    switch (config) # show ib router Routing state: enabled   IB subnet Routing enabled infiniband-default enabled infiniband-1 enabled switch (config) # show interfaces ib-subnet infiniband-default infiniband-default state: GUID : F4:52:14:03:00:6E:F2:8B Alias GID : N/A LID : 10 Subnet prefix : FE:C0:00:00:00:00:00:08 Physical state : LinkUp Logical state : Active L3 interface state : Up

IP to GID Resolution

  1. Go to the following Github: https://github.com/Mellanox/ip2gid

  2. Clone the Git repository

  3. Compile and run on each node in the fabric

  4. Change the device MAC address of the IPoIB device to be based on the alias GID and not the GUID.
    For example, # echo fec0:0000:0000:0003:0014:0500:0000:0001 > /sys/class/net/ib0/set_mac
    where fe:c0:00:00:00:00:00:02:00:14:05:00:00:00:00:01 is the alias GID given by the SM to that node.

  5. Add route using "ip route add" command to the relevant hosts.
    # ifconfig ib0 12.0.3.1/24 --> set ip for ib0
    # ip route add 12.0.1.0/24 via 12.3.0.250 --> adding route to hosts with 12.1.xxx.xxx IP
    # ip route add 12.0.2.0/24 via 12.3.0.250 --> adding route to hosts with 12.2.xxx.xxx IP

The IB router expects the subnet prefix to be constructed according to some very specific rules. By default, the command which enables IB routers validates the subnet prefix prior to allowing the change.

The commands which affect subnet prefix checking are as follows:

  • ib sm <name> enable – starts SM on this node or any node in cluster

  • ib sm subnet-prefix <subnet-prefix> – configures the subnet prefix

  • ib sm rtr-aguid-enable <1 | 2> – enables support for alias GIDs as needed by IB routers

When any of these commands is run, while the other two have already been issued, the value of the subnet prefix is checked. If it is not valid, the current commit is rejected and the OpenSM state does not change.

To disable subnet prefix checking

  1. Verify the status of subnet prefix override. Run:

    Copy
    Copied!
                

    switch (config) # show ib sm subnet-prefix-override enable

  2. If enabled, disable subnet-prefix-override. Run:

    Copy
    Copied!
                

    switch (config) # ib sm subnet-prefix-override

  3. Verify configuration. Run:

    Copy
    Copied!
                

    switch (config) # show ib sm subnet-prefix-override disable

ib router

ib router
no ib router

Enables the set of commands that allow control of IB router functionality.
The no form of the command disables IB router commands and removes all related configurations.

Syntax Description

N/A

Default

N/A

Configuration Mode

config

History

3.6.0500

Example

switch (config) # ib router

Related Commands

system profile

Notes

ib router shutdown

ib router shutdown
no ib router shutdown

Disables IB router.
The no form of the command enables IB router.

Syntax Description

N/A

Default

Disabled

Configuration Mode

config

History

3.6.0500

Example

switch (config) # no ib router shutdown

Related Commands

Notes

This command does not clear IB router configuration

interface ib-subnet

interface ib-subnet <swid-name>
no interface ib-subnet <swid-name>

Creates routing on IB router subnet.
The no form of the command removes routing on router interface.

Syntax Description

swid-name

Name of the SWID: infiniband-default, infiniband-1...infiniband-5

Default

N/A

Configuration Mode

config

History

3.6.0500

Example

switch (config) # interface ib-subnet infiniband-3

Related Commands

system profile

Notes

The maximum number of SWIDs depends on the number of SWIDs defined in the profile

interface ib-subnet shutdown

interface ib-subnet <swid-name> shutdown
no interface ib-subnet <swid-name> shutdown

Disables routing on IB router subnet.
The no form of the command enables routing on router interface.

Syntax Description

swid-name

Name of the SWID: infiniband-default, infiniband-1...infiniband-5

shutdown

Admin down on router interface
Admin up on router interface with no form of command

Default

Disabled

Configuration Mode

config

History

3.6.0500

Example

switch (config) # no interface ib-subnet infiniband-3 shutdown

Related Commands

Notes

show ib router

show ib router

Displays current IB router functionality.

Syntax Description

N/A

Default

N/A

Configuration Mode

Any command mode

History

3.6.0500

Example

switch (config) # show ib router
Routing state: enabled

IB Subnet Routing enabled
infiniband-default enabled
infiniband-1 disabled
infiniband-2 enabled
infiniband-3 enabled

Related Commands

Notes

show interfaces ib-subnet

show interfaces ib-subnet [<swid-name>] [brief]

Displays statistics of one or all IB subnets with enabled IB routing.

Syntax Description

swid-name

Name of the SWID: infiniband-default, infiniband-1...infiniband-5

brief

Displays output in a table format

Default

Disabled

Configuration Mode

config

History

3.6.0500

Example

switch (config) # show interfaces ib-subnet infiniband-3
infiniband-3 state:
GUID : F4:52:14:03:00:6E:F2:8B
Alias GID : N/A
LID : 10
Subnet prefix : FE:C0:00:00:00:00:00:08
Physical state : LinkUp
Logical state : Active
L3 interface state : Up

Related Commands

Notes

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.