IB Router

InfiniBand (IB) Router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation, fault resilience and connection to different topologies used by different subnets.

Prerequisites

1. Platform support: IB Router functionality is supported only on Q3400-RA and Q3401-RD systems.

2. Subnet Manager (SM): Ensure Subnet Managers are not running before configuring the switch as an IB router and assigning interfaces to the IB subnets (SWIDs). Complete all IB router configuration and SWID assignments before starting SMs.

3. Limitations:

   Note

   IPoIB: When IB router profile is configured, IPoIB interfaces are not supported and will remain down.

   Note

   FNM: External FNM ports are restricted to the default SWID and cannot be assigned to other subnets. Servers should not be connected to the external FNM ports on IB router switches.

NVOS Switch Configuration

1. Configure the system profile to enable IB routing with multiple SWIDs:

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv action change system profile ib-routing enabled num-of-swids 2
   The operation will reset the system configuration and initiate a reboot.
   Type [y] to confirm the system profile change and reboot.
   Type [N] to abort.
   Do you want to continue? [y/N] y
   Action executing ...
   System will be rebooted, configurations and system files might be deleted
   Action executing ...
           

   Note

   Profile change triggers a factory reset. All existing configurations will be reset to defaults and the system will reboot.

2. After the reboot, verify the system profile configuration:

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv show system profile
                            operational
   -----------------------  -----------
   adaptive-routing         enabled
   adaptive-routing-groups  1792
   breakout-mode            enabled
   ib-routing               enabled
   num-of-swids             2
           

   Note

   • Ensure that ib-routing is set to enabled and that num-of-swids matches your configured value.

   • After the profile change, all IB interfaces are assigned to the default SWID (infiniband-default). To confirm the IB interface assignment to the infiniband-default SWID, use the nv show interface command.

3. Assign each IB interface to the appropriate IB subnet (SWID) according to the topology port mappings.

   Use the following command format:

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv set interface <interface-id> link ib-subnet <swid-name>
           

   where <interface-id> is an interface name or range, and <swid-name> is the target SWID.

   Required workflow for new SWID mapping to take effect (Beta):

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv set interface <interface-id> link ib-subnet <swid-name>
   admin@nvos:~$ nv config apply -y
   admin@nvos:~$ nv config save
   admin@nvos:~$ nv action reboot system
           

   Example:

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv set interface sw3-4p1-2 link ib-subnet infiniband-1
   admin@nvos:~$ nv config apply -y
   Warning: IB interface configuration is changed. In order to apply configuration the port will be restarted.
   admin@nvos:~$ nv config save
   admin@nvos:~$ nv action reboot system
   Action executing ...
   Performing reboot cold...
   Action executing ...
   Disconnecting from NVOS, system is offline during reboot
           

   Note

   Due to beta-version limitation, SWID changes take effect only after reboot with saved configuration.

   Because nv show interface displays operational state only, it may continue to show the previous SWID until the system is rebooted.

   After saving the configuration and rebooting, nv show interface will reflect the new SWID.

   Warning

   Ensure Subnet Managers are not running before assigning interfaces to SWIDs. SWID assignments must be completed prior to SM configuration.

   Changing SWID assignments while Subnet Managers are running may cause inconsistent SM state requiring a reboot of the router.

4. Verify the SWID (IB subnet) assignment for IB interfaces:

   Copy

   Copied!

               
   
               
   admin@nvos:~$ nv show interface
   Interface  State  Speed  MTU    Type   IB Speed  IB Subnet           Logical State  Physical State
   ---------  -----  -----  -----  -----  --------  ------------------  -------------  --------------
   eth0       up     1G     1500   eth                                                                
   eth1       up     1G     1500   eth                                                                
   fnm1       up     800G   4096   fnm    xdr       infiniband-default  Active         LinkUp
   sw1p1      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
   sw1p2      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
   sw2p1      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
   sw2p2      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
   sw3p1      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
   sw3p2      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
   sw4p1      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
   sw4p2      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
   ...
           

Verify Routing Table

After configuring both the IB router and starting Subnet Managers, verify the routing table:

            

            
admin@nvos:~$ nv show ib router routing-table
IB Subnet           Subnet Prefix  Valid
------------------  -------------  -----
infiniband-default  1              true
infiniband-1        2              true
        

• IB Router Commands