General Networkwide Deployment Guidelines

Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Onyx™ operating system.

Warning

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Warning

The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.

Warning

The Skyway solution mandates a dedicated subnet between the appliance and the layer-3 Ethernet switch/router.
This subnet is not visible from outside and can be configured to be ANY class-C subnet, including IPs which are not part of the customer’s IP pool.

Skyway appliance MUST be directly attached to L3 Ethernet switch(es)/router(es) (at least one) to complement the function of an IPoIB router. The role of divisions is as follows:
Skyway forwards IPoIB traffic to a single IP address/MAC on the Ethernet side; the directly-attached L3 Ethernet switch(es)/router(es) provide Ethernet-side load balancing functionality (auch as MLAG) and routing.

All Skyway devices with a single HA domain share the same virtual IP.

image2022-3-22_15-19-42.png

Ethernet Guidelines

The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).
For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.

Warning

Make sure MTU on the Ethernet router connected to Skyway is at least 2 bytes smaller than the InfiniBand IPoIB MTU configuration (e.g. set InfiniBand MTU to 4092 and Ethernet MTU to 4090).


InfiniBand Guidelines

Warning

All InfiniBand ports must be connected to the same InfiniBand fabric.

Subnet Manager Configuration

Warning

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously.

If opensm runs on an InfiniBand switch, configure the following:

Copy
Copied!
            

switch (config) # ib sm virt enable switch (config) # ib sm virt-max-ports-in-process 0

If opensm runs on a host, add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf):

Copy
Copied!
            

# Virtualization support # 0: Ignore Virtualization - No virtualization support # 1: Disable Virtualization - Disable virtualization on all # Virtualization supporting ports # 2: Enable Virtualization - Enable (virtualization on all # Virtualization supporting ports virt_enabled 2     # Maximum number of ports to be processed simultaneously # by Virtualization Manager (0 - process all pending ports) virt_max_ports_in_process 0

Warning

The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".

For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2.

Warning

IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.

  1. On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example, 1.1.1.3).

    Copy
    Copied!
                

    # ifconfig ib0 1.1.1.2/24 # ip route add 0/0 via 1.1.1.3

    Warning

    The NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.

  2. Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example, 2.2.2.1).

    Copy
    Copied!
                

    gateway > enable gateway # configure terminal gateway (config) # interface ib port-channel 1 ip address 1.1.1.1/24 gateway (config) # interface ib port-channel 1 virtual ip address 1.1.1.3/24 gateway (config) # interface ethernet port-channel 1 ip address 2.2.2.2/24 gateway (config) # ip route 0.0.0.0/0 2.2.2.1

  3. Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example, 2.2.2.2). Below is an example using a NVIDIA-Onyx-based switch with port 1/1 connected to the gateway. In this example, ports 1-8 on the router (see line 7) are connected to the 8 Ethernet ports on Skyway. null 0000018a-7c10-d123-a1fa-fed948790001 0000018a-7c10-d123-a1fa-fed948790002
  4. To ensure proper deployment, ping between a host in the subnet 192.168.1.0/24 and the InfiniBand host with IP 1.1.1.2 should be successful.

© Copyright 2023, NVIDIA. Last updated on Sep 9, 2023.