Appendix – IB Router

NVIDIA UFM Enterprise User Manual v6.11.1

IB router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation and fault resilience between islands and IB subnets, and enabling connection to different topologies used by different subnets.

The forwarding between the IB subnets is performed using GRH lookup. The IB router’s basic functionality includes:

  • Removal of current L2 LRH (local routing header)

  • Routing table lookup – using GID from GRH

  • Building new LRH according to the destination according to the routing table

The DLID in the new LRH is built using simplified GID-to-LID mapping (where LID = 16 LSB bits of GID) thereby not requiring to send for ARP query/lookup.

Site-Local Unicast GID Format

image2019-6-17_9-34-53.png

For this to work, the SM allocates an alias GID for each host in the fabric where the alias GID = {subnet prefix[127:64], reserved[63:16], LID[15:0}. Hosts should use alias GIDs in order to transmit traffic to peers on remote subnets.

Host-to-Host IB Router Unicast Flow

image2019-6-17_9-35-19.png

The following scripts are supplied as part of UFM installation package.

set_num_of_subnets.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/set_num_of_subnets.sh --hostname <hostname> --username <username> --password <password> --num-of-subnets <num-of-subnets>

  • Description – Configures system profile to InfiniBand allowing multiple switch IDs

  • Syntax Description

    hostname

    IB router hostname or IP address

    username

    IB router username

    password

    IB router user password

    num-of-subnets

    Specified number of subnets (AKA SWIDs) to be initialized by the system. Value range: 2-6

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/set_num_of_subnets.sh --hostname 10.6.204.12 --username admin --password admin --num-of-subnets 6

Warning

As a result of running this script, reboot is performed and all configuration is removed


add_interfaces_to_subnet.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/add_interfaces_to_subnet.sh --hostname <hostname> --username <username> --password <password> --interface <interface | interface-range> --subnet <subnet>

  • Description

    Maps an interface to a subnet and enables it

  • SyntaxDescription

    hostname

    IB router hostname or IP address

    username

    IB router username

    password

    IB router user password

    interface | interface-range

    Single IB interface or range of IB interfaces.

    Single IB interface: 1/<interface>

    Range of IB interfaces: 1/<interface>-1/<interface>

    subnet

    Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/add_interfaces_to_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/1-1/6 --subnet infiniband-1

remove_interfaces_from_subnet.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/remove_interfaces_from_subnet.sh --hostname <hostname> --username <username> --password <password> --interface <interface | interface-range>

  • Description

    Un-maps an interface from a subnet after it has been disabled

  • Syntax Description

    hostname

    IB router hostname or IP address

    username

    IB router username

    password

    IB router user password

    interface | interface-range

    Single IB interface or range of IB interfaces.

    Single IB interface: 1/<interface>

    Range of IB interfaces: 1/<interface>-1/<interface>

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/remove_interfaces_from_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/6Example

add_subnet_to_router.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/add_subnet_to_router.sh --hostname <hostname> --username <username> --password <password> --subnet <subnet>

  • Description

    Creates routing on IB subnet interface and enables routing on that interface

  • Syntax Description

    hostname

    IB router hostname or IP address

    username

    IB router username

    password

    IB router user password

    subnet

    Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/add_subnet_to_router.sh --hostname 10.6.204.12 --username admin --password admin --subnet infiniband-3Example

Warning

As a result of running this script, the set of commands that allow control of IB router functionality is being enabled


remove_subnet_from_router.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/remove_subnet_from_router.sh --hostname <hostname> --username <username> --password <password> --subnet <subnet>

  • Description

    Destroys routing on IB subnet interface after routing on that interface has been disabled

  • Syntax Description

    hostname

    IB router hostname or IP address

    username

    IB router username

    password

    IB router user password

    subnet

    Name of IB subnet (AKA SWID): infiniband-default, infiniband-1…infiniband-5

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/remove_subnet_from_router.sh --hostname 10.6.204.12 --username admin --password admin --subnet infiniband-defaultExample

set_ufm_sm_router_support.sh

  • Arguments

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh [-c <subnet prefix>] [-r][-h]

  • Description

    [-c <subnet prefix>]: Used for updating OpenSM configuration file with new subnet prefix and forces OpenSM to re-read configuration.

    [-r]: Used for resetting OpenSM configuration to default value and canceling IB routing.

  • Syntax Description

    -c

    Configure new IB subnet prefix. Should be followed by new IB router subnet prefix value

    -r

    Reset to default

    -h

    Show help

  • Example

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh -c 0xfec0000000001234Examples

    Copy
    Copied!
                

    /opt/ufm/scripts/ib_router/set_ufm_sm_router_support.sh -r

Step 1: Configure multi-switch. Run:

Copy
Copied!
            

/opt/ufm/scripts/set_num_of_subnets.sh --hostname 10.6.204.12 --username admin --password admin --num-of-subnets 6

Step 2: Map interface to a subnet. Run:

Copy
Copied!
            

/opt/ufm/scripts/add_ports_to_subnet.sh --hostname 10.6.204.12 --username admin --password admin --interface 1/1 --subnet infiniband-default

Step 3: Create routing on IB subnet interface. Run:

Copy
Copied!
            

/opt/ufm/scripts/add_subnet_to_router.sh --hostname 10.6.204.12 -–username admin --password admin --subnet infiniband-default

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.