NVIDIA MLNX_OFED Documentation v5.8-3.0.7.0.101 for DGX H100 Systems
v5.8

IB Router

IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.
The forwarding between the IB subnets is performed using GRH lookup. The IB router's basic functionality includes:

  • Removal of current L2 LRH (local routing header)

  • Routing

  • table lookup – using GID from GRH

  • Building new LRH according to the destination according to the routing table

The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.

Local Unicast GID Format

image2019-3-4_12-5-46-version-1-modificationdate-1701071305323-api-v2.png

For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.

Host-to-Host IB Router Unicast Flow

worddava699c8dc9881157c6daec5e174e919b5-version-1-modificationdate-1701071305650-api-v2.png

© Copyright 2023, NVIDIA. Last updated on Nov 27, 2023.