General Networkwide Deployment Guidelines

Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Onyx™operating system.

Warning

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Warning

The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.

image2022-3-22_15-19-42.png

Ethernet Guidelines

The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).
For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.

Warning

Make sure MTU on the Ethernet router connected to Skyway isat least 2 bytessmaller than the InfiniBand IPoIB MTU configuration (e.g. set InfiniBandMTU to 4092 and Ethernet MTU to 4090).

InfiniBand Guidelines

Warning

All InfiniBand ports must be connected to the same InfiniBand fabric.

Subnet Manager Configuration

Warning

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously.

If opensm runs on an InfiniBand switch, configure the following:

Copy
Copied!
            

switch (config) # ib sm virt enable switch (config) # ib sm virt-max-ports-in-process 0

If opensm runs on a host, add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf):

Copy
Copied!
            

# Virtualization support # 0: Ignore Virtualization - No virtualization support # 1: Disable Virtualization - Disable virtualization on all # Virtualization supporting ports # 2: Enable Virtualization - Enable (virtualization on all # Virtualization supporting ports virt_enabled 2     # Maximum number of ports to be processed simultaneously # by Virtualization Manager (0 - process all pending ports) virt_max_ports_in_process 0

Warning

The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".

For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2.

Warning

IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.

  1. On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example, 1.1.1.3).

    Copy
    Copied!
                

    # ifconfig ib0 1.1.1.2/24 # ip route add 0/0 via 1.1.1.3

    Warning

    The NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.

  2. Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example, 2.2.2.1).

    Copy
    Copied!
                

    gateway > enable gateway # configure terminal gateway (config) # interface ib port-channel 1 ip address 1.1.1.1/24 gateway (config) # interface ib port-channel 1 virtual ip address 1.1.1.3/24 gateway (config) # interface ethernet port-channel 1 ip address 2.2.2.2/24 gateway (config) # ip route 0.0.0.0/0 2.2.2.1

  3. Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example, 2.2.2.2).
    Below is an example using a NVIDIA-Onyx-based switch with port 1/1 connected to the gateway.
    In this example, ports 1-8 on the router (see line 7) are connected to the 8 Ethernet ports on Skyway.

    Copy
    Copied!
                

    eth_router > enable eth_router # configure terminal eth_router (config) # ip routing eth_router (config) # lacp eth_router (config) # interface port-channel 1 eth_router (config interface port-channel 1) # exit eth_router (config) # interface ethernet 1/1-1/8 channel-group 1 mode active eth_router (config) # vlan 2 eth_router (config vlan 2) # exit eth_router (config) # interface port-channel 1 switchport access vlan 2 eth_router (config) # interface vlan 2 ip address 2.2.2.1 /24 eth_router (config) # ip route 1.1.1.0 /24 2.2.2.2

    Warning

    Note that the above connection describes a connection between Skyway and a single Ethernet router with LAG. It is possible to also connect to two Ethernet routers in an MLAG configuration. For more information, see the following community post for MLAG configuration on NVIDIA Onyx-based switches: support.mellanox.com/s/article/how-to-configure-mlag-on-mellanox-switches.

    image2022-3-22_15-25-26.png

  4. To ensure proper deployment, ping between a host in the subnet 192.168.1.0/24 and the InfiniBand host with IP 1.1.1.2 should be successful.

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.