image image image image image

On This Page

Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Onyx™ operating system.

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.

The Skyway solution mandates a dedicated subnet between the appliance and the layer-3 Ethernet switch/router.
This subnet is not visible from outside and can be configured to be ANY class-C subnet, including IPs which are not part of the customer’s IP pool.

General Networkwide Guidelines

Ethernet Guidelines

The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).
For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.

Make sure MTU on the Ethernet router connected to Skyway is at least 2 bytes smaller than the InfiniBand IPoIB MTU configuration (e.g. set InfiniBand MTU to 4092 and Ethernet MTU to 4090).

InfiniBand Guidelines

All InfiniBand ports must be connected to the same InfiniBand fabric.

Subnet Manager Configuration

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously. 

If opensm runs on an InfiniBand switch, configure the following:

switch (config) # ib sm virt enable
switch (config) # ib sm virt-max-ports-in-process 0

If opensm runs on a host, add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf):

# Virtualization support
# 0: Ignore Virtualization - No virtualization support
# 1: Disable Virtualization - Disable virtualization on all
# Virtualization supporting ports
# 2: Enable Virtualization - Enable (virtualization on all
# Virtualization supporting ports
virt_enabled 2

# Maximum number of ports to be processed simultaneously
# by Virtualization Manager (0 - process all pending ports)
virt_max_ports_in_process 0

Configuring IP Addresses and Routes  

The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".

For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2. 

IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.

  1. On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example,

    # ifconfig ib0
    # ip route add 0/0 via

    The NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.

  2. Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example,

    gateway > enable
    gateway # configure terminal
    gateway (config) # interface ib port-channel 1 ip address
    gateway (config) # interface ib port-channel 1 virtual ip address
    gateway (config) # interface ethernet port-channel 1 ip address
    gateway (config) # ip route
  3. Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example,
    Below is an example using a NVIDIA-Onyx-based switch with port 1/1 connected to the gateway.
    In this example, ports 1-8 on the router (see line 7) are connected to the 8 Ethernet ports on Skyway.

    eth_router > enable
    eth_router # configure terminal
    eth_router (config) # ip routing
    eth_router (config) # lacp
    eth_router (config) # interface port-channel 1
    eth_router (config interface port-channel 1) # exit
    eth_router (config) # interface ethernet 1/1-1/8 channel-group 1 mode active
    eth_router (config) # vlan 2
    eth_router (config vlan 2) # exit
    eth_router (config) # interface port-channel 1 switchport access vlan 2
    eth_router (config) # interface vlan 2 ip address /24
    eth_router (config) # ip route /24

    Note that the above connection describes a connection between Skyway and a single Ethernet router with LAG. It is possible to also connect to two Ethernet routers in an MLAG configuration. For more information, see the following community post for MLAG configuration on NVIDIA Onyx-based switches:

  4. To ensure proper deployment, ping between a host in the subnet and the InfiniBand host with IP should be successful.