NVIDIA Skyway InfiniBand-to-Ethernet Gateway User Manual
NVIDIA Docs Hub Homepage  NVIDIA Networking  Networking Switches  NVIDIA Skyway InfiniBand-to-Ethernet Gateway User Manual  Networkwide Deployment Guidelines

On This Page

Networkwide Deployment Guidelines

Skyway GA400 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Cumulus operating system.

Note

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Note

Make sure that validation of network connectivity is be done strictly between Ethernet and InfiniBand host. Ping with test IPs belonging to the Skyway are mostly invalid and should not be used for validation.

Note

The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.

Note

The Skyway solution mandates a dedicated subnet between the appliance and the layer-3 Ethernet switch/router.

This subnet is not visible from outside and can be configured to be ANY class-C subnet, including IPs which are not part of the customer’s IP pool.

Skyway appliance MUST be directly attached to L3 Ethernet switch(es)/router(es) (at least one) to complement the function of an IPoIB router. The role of divisions is as follows:

Skyway forwards IPoIB traffic to a single IP address/MAC on the Ethernet side; the directly-attached L3 Ethernet switch(es)/router(es) provide Ethernet-side load balancing functionality (auch as MLAG) and routing.

All Skyway devices with a single HA domain share the same virtual IP.

image2022-3-22_15-19-42-version-1-modificationdate-1776167010210-api-v2.png

General Networkwide Guidelines

Ethernet Guidelines

The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).

For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.

Note

Make sure MTU on the Ethernet router connected to Skyway is at least 2 bytes smaller than the InfiniBand IPoIB MTU configuration (e.g., set InfiniBand MTU to 4092 and Ethernet MTU to 4090).


InfiniBand Guidelines

Note

All InfiniBand ports must be connected to the same InfiniBand fabric.

Subnet Manager Configuration

Note

Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.

Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously.

Add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf) on the host, where it is running:

Copy
Copied!
            

            
# Virtualization support
# 0: Ignore Virtualization - No virtualization support
# 1: Disable Virtualization - Disable virtualization on all
# Virtualization supporting ports
# 2: Enable Virtualization - Enable (virtualization on all
# Virtualization supporting ports
virt_enabled 2
 
 
# Maximum number of ports to be processed simultaneously
# by Virtualization Manager (0 - process all pending ports)
virt_max_ports_in_process 0

Configuring IP Addresses and Routes

Note

The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".

For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2.

Note

IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.

  1. On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example, 1.1.1.3).

    Copy
    Copied!
                
    
            
    # ifconfig ib0 1.1.1.2/24
# ip route add 0/0 via 1.1.1.3

    Note

    The NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.

  2. Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example, 2.2.2.1).

    Copy
    Copied!
                
    
            
    gateway > enable
gateway # configure terminal
gateway (config) # interface ib port-channel 1 ip address 1.1.1.1/24
gateway (config) # interface ib port-channel 1 virtual ip address 1.1.1.3/24
gateway (config) # interface ethernet port-channel 1 ip address 2.2.2.2/24
gateway (config) # ip route 0.0.0.0/0 2.2.2.1

  3. Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example, 2.2.2.2).

    Below is an example using a NVIDIA-Cumulus-based switch with ports 1/1 - 1/4 (each one has a breakout mode 2x) are connected to the 8 Ethernet ports on Skyway

    Copy
    Copied!
                
    
            
    eth_router # # split each port to two
eth_router # nv set interface swp1-4 link breakout 2x
eth_router # nv set interface swp1s0-1,swp2s0-1,swp3s0-1,swp4s0-1 link state up
eth_router # nv set bridge domain bridge71 untagged 71
eth_router # nv set bridge domain bridge71 vlan 71
eth_router # # create LAG and add its members
eth_router # nv set interface bond71 bond lacp-rate fast
eth_router # nv set interface bond71 bond member swp1s0-1,swp2s0-1,swp3s0-1,swp4s0-1
eth_router # nv set interface bond71 bridge domain bridge71
eth_router # nv set interface bond71 link mtu 2046
eth_router # nv set interface bond71 type bond
eth_router # # create VLAN interface and set its IP address
eth_router # nv set interface vlan71 base-interface bridge71
eth_router # nv set interface vlan71 ip address 2.2.2.1/24
eth_router # nv set interface vlan71 vlan 71
eth_router # # static route to IB subnet
eth_router # nv set vrf default router static 1.1.1.0/24 address-family ipv4-unicast
eth_router # nv set vrf default router static 1.1.1.0/24 via 2.2.2.1 type ipv4-address

    Note

    Note that the above connection describes a connection between Skyway and a single Ethernet router with LAG. It is possible to also connect to two Ethernet routers in an MLAG configuration. For more information, see the following community post for MLAG configuration on NVIDIA Onyx-based switches: http://https://enterprise-support.nvidia.com/s/article/how-to-configure-mlag-on-the-switches.

    image2022-3-22_15-25-26-version-1-modificationdate-1776167010547-api-v2.png

    Note

    Additional commands might be needed the ensure the MTU restrictions described in Ethernet Guidelines above.

  4. To ensure proper deployment, ping between a host in the subnet 192.168.1.0/24 and the InfiniBand host with IP 1.1.1.2 should be successful.

    Note

    Make sure that validation of network connectivity is be done strictly between Ethernet and InfiniBand host.

    Ping with test IPs belonging to the Skyway are mostly invalid and should not be used for validation.

    List of valid ping conditions:

    • Ping from InfiniBand server to Skyway InfiniBand VIP

    • Ping from InfiniBand server to Skyway Eth port channel interface IP

    • Ping from InfiniBand server to Eth router interface IP

    • Ping from Eth server to Skyway Eth port channel interface IP

    • Ping from Eth server to Eth router interface IP

    List of invalid ping conditions:

    • Ping from InfiniBand server to Skyway InfiniBand port channel interface IP will fail or be intermittent

    • Ping from Ethernet server to Skyway VIP should fail or be intermittent

    • Ping from Ethernet server to Skyway InfiniBand port channel interface IP should fail or be intermittent

    • Ping from Skyway Ethernet port-channel are unreliable: ping can fail (response sent to the other Skyway) or succeed (response sent to the same Skyway)

© Copyright 2026, NVIDIA. Last updated on Apr 14, 2026
content here