Appendix – InfiniBand Router

NVIDIA UFM-SDN Appliance User Manual v4.9.0

IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.
The forwarding between the IB subnets is performed using GRH lookup. The IB router’s basic functionality includes:

  • Removal of current L2 LRH (local routing header)

  • Routing table lookup – using GID from GRH

  • Building new LRH according to the destination according to the routing table

The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.

Site-Local Unicast GID Format


For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.

Host-to-Host IB Router Unicast Flow


Please refer to UFM-SDN Appliance Command Reference Guide document for the IB router commands’ details

  1. Configure multi-switch. Run:


    ufmapl [ mgmt-sa ] (config) # ib router set-num-of-subnets --hostname --username admin --password admin --num-of-subnets 6

  2. Map interface to a subnet. Run:


    ufmapl [ mgmt-sa ] (config) # ib router add-interfaces-to-subnet --hostname --username admin --password admin --interface 1/1 --subnet infiniband-default

  3. Create routing on IB subnet interface. Run:


    ufmapl [ mgmt-sa ] (config) # ib router add-subnet-to-router --hostname -–username admin --password admin --subnet infiniband-default

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.