Appendix – IB Router
IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.
The forwarding between the IB subnets is performed using GRH lookup. The IB router’s basic functionality includes:
Removal of current L2 LRH (local routing header)
Routing table lookup – using GID from GRH
Building new LRH according to the destination according to the routing table
The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.
Site-Local Unicast GID Format
For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.
Host-to-Host IB Router Unicast Flow
The following scripts are supplied as part of UFM installation package.
set_num_of_subnets.sh
Arguments
/opt/ufm/scripts/ib_router/set_num_of_subnets.sh --hostname <hostname> --username <username> --password <password> --num-of-subnets <num-of-subnets>
Description – Configures system profile to InfiniBand allowing multiple switch IDs
Syntax Description
hostname
IB router hostname or IP address
username
IB router username
password
IB router user password
num-of-subnets
Specified number of subnets (AKA SWIDs) to be initialized by the system. Value range: 2-6
Example
/opt/ufm/scripts/ib_router/set_num_of_subnets.sh --hostname 10.6.204.12 --username admin --password admin --num-of-subnets 6
As a result of running this script, reboot is performed and all configuration is removed
add_interfaces_to_subnet.sh
Arguments
/opt/ufm/scripts/ib_router/add_interfaces_to_subnet.sh --hostname <hostname> --username <username> --password <password> --interface <interface | interface-range> --subnet <subnet>
Description
Maps an interface to a subnet and enables it
SyntaxDescription
hostname
IB router hostname or IP address
username
IB router username
password
IB router user password
interface | interface-range
Single IB interface or range of IB interfaces.
Single IB interface: 1/<interface>
Range of IB interfaces: 1/<interface>-1/<interface>
subnet
Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5
Example
/opt/ufm/scripts/ib_router/add_interfaces_to_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/1-1/6 --subnet infiniband-1
remove_interfaces_from_subnet.sh
Arguments
/opt/ufm/scripts/ib_router/remove_interfaces_from_subnet.sh --hostname <hostname> --username <username> --password <password> --interface <interface | interface-range>
Description
Un-maps an interface from a subnet after it has been disabled
Syntax Description
hostname
IB router hostname or IP address
username
IB router username
password
IB router user password
interface | interface-range
Single IB interface or range of IB interfaces.
Single IB interface: 1/<interface>
Range of IB interfaces: 1/<interface>-1/<interface>
Example
/opt/ufm/scripts/ib_router/remove_interfaces_from_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/6Example
add_subnet_to_router.sh
Arguments
/opt/ufm/scripts/ib_router/add_subnet_to_router.sh --hostname <hostname> --username <username> --password <password> --subnet <subnet>
Description
Creates routing on IB subnet interface and enables routing on that interface
Syntax Description
hostname
IB router hostname or IP address
username
IB router username
password
IB router user password
subnet
Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5
Example
/opt/ufm/scripts/ib_router/add_subnet_to_router.sh --hostname 10.6.204.12 --username admin --password admin --subnet infiniband-3Example
As a result of running this script, the set of commands that allow control of IB router functionality is being enabled
remove_subnet_from_router.sh
Arguments
/opt/ufm/scripts/ib_router/remove_subnet_from_router.sh --hostname <hostname> --username <username> --password <password> --subnet <subnet>
Description
Destroys routing on IB subnet interface after routing on that interface has been disabled
Syntax Description
hostname
IB router hostname or IP address
username
IB router username
password
IB router user password
subnet
Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5
Example
/opt/ufm/scripts/ib_router/remove_subnet_from_router.sh --hostname 10.6.204.12 --username admin --password admin --subnet infiniband-defaultExample
set_ufm_sm_router_support.sh
Arguments
/opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh [-c <subnet prefix>] [-r][-h]
Description
[-c <subnet prefix>]: Used for updating OpenSM configuration file with new subnet prefix and forces OpenSM to re-read configuration.
[-r]: Used for resetting OpenSM configuration to default value and canceling IB routing.
Syntax Description
-c
Configure new IB subnet prefix. Should be followed by new IB router subnet prefix value
-r
Reset to default
-h
Show help
Example
/opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh -c 0xfec0000000001234Examples
/opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh -r
Step 1: Configure multi-switch. Run:
/opt/ufm/scripts/set_num_of_subnets.sh --hostname 10.6.204.12 --username admin --password admin --num-of-subnets 6
Step 2: Map interface to a subnet. Run:
/opt/ufm/scripts/add_ports_to_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/1 --subnet infiniband-default
Step 3: Create routing on IB subnet interface. Run:
/opt/ufm/scripts/add_subnet_to_router.sh --hostname 10.6.204.12 -–username admin --password admin --subnet infiniband-default