Introduction

This is the user guide for the NVIDIA Skyway InfiniBand-to-Ethernet gateway. This document contains the complete product overview, installation and initialization instructions, and product specifications.

NVIDIA Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet storage or other Ethernet-based communications to access the InfiniBand datacenter, and vice versa. The solution, leveraging ConnectX’s hardware-based forwarding of IP packets and standard IP-routing protocols, supports 200Gb/s HDR connectivity today, and is future-ready to support higher speeds.

NVIDIA Skyway Highlights

The NVIDIA Skyway system populates eight ConnectX-6 InfiniBand/VPI adapter cards, fans, and two PSUs in the system's rear panel, as shown in the below figure.

Network Interface Cards

NVIDIA Skyway is shipped populated with eight ConnectX-6 dual-port network interface cards (NICs) which enable the hardware-based forwarding of IP packets from the InfiniBand to Ethernet, and vice versa.

Power Supply Units

NVIDIA Skyway is equipped with two redundant, load-sharing power supply units at the rear side of the system. The PSUs are housed in a 2U canister containing the power supplies. Each PSU has an extraction handle, PSU status LED, and a power socket.

For power supply unit LEDs operations, please refer to the System Monitoring section.

The system enables hot-swapping which enables components to be exchanged while the system is online without affecting operational integrity.

Fans

Power Supply Fans

NVIDIA Skyway is equipped with one fan per power supply unit on the rear panel of the appliance.

Internal Fans

NVIDIA Skyway is equipped with six internal fans for cooling the CPU and expansion cards. Under normal operation, the cooling fans operate at a constant speed. If the system module fails or one of the temperature thresholds is exceeded, the cooling fans automatically raise their rotation speeds to draw more airflow.

Check the package contents list to see that all the parts have been sent. Check the parts for visible damage that may have occurred during shipping. Please note that the product must be placed on an antistatic surface.

Rail Kit Package Contents

Management Interfaces, PSUs, and Fans

For a full list of features, please refer to the system’s product brief at www.nvidia.com/en-us/networking. In the main menu, click on PRODUCTS → INFINIBAND → GATEWAY & ROUTERS SYSTEMS → select the desired product page.

InfiniBand-to-Ethernet Gateway Operational Description

NVIDIA Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet storage or other Ethernet-based communications to access the InfiniBand datacenter, and vice versa. The solution, leveraging ConnectX’s hardware-based forwarding of IP packets and standard IP-routing protocols, supports 200Gb/s HDR connectivity, today, and is future-ready to support higher speeds.

NVIDIA Skyway contains 8 ConnectX VPI dual-port adapter cards which enable the hardware-based forwarding of IP packets from InfiniBand to Ethernet, and vice versa. NVIDIA Skyway also includes the NVIDIA Gateway Operating System, MLNX-GW, which manages the appliance and handles the high availability and load balancing between the ConnectX cards and gateway appliances.

A single NVIDIA Skyway supports a maximum bandwidth of 1.6Tb/s, utilizing 16 ports, each of which reaches 100Gb/s traffic. In terms of connectivity, the InfiniBand ports can be connected to the InfiniBand network using HDR/HDR100 or EDR speeds, while the Ethernet ports can be connected to the Ethernet network using 200Gb/s or100Gb/s.

Load Balancing and High Availability Operational Description

On the Ethernet side, the load balancing and high availability functions are achieved by leveraging Ethernet LAG (Link Aggregation). LACP (Link Aggregation Control Protocol) is used to establish the LAG and to verify connectivity. On the InfiniBand side, these functions are achieved by assuring that different flows go through different ConnectX HCAs, so that, in case a HCA drops, another HCA will continue passing its flows.

At initialization, 256 gateway GIDs are spread evenly among all InfiniBand ports of the gateway appliances. When an InfiniBand node initiates a traffic flow through the gateway, it first sends a broadcast ARP request with the default gateway IP address to determine the gateway’s GID. All ConnectX cards receive the request, but only one sends the ARP response. The response is sent from the ConnectX cards that were assigned to handle the range of GIDs corresponding to the sending node’s IP address. When the originating node receives the gateway GID, it sends a path query to the subnet manager (SM) to determine the gateway LID and the communication flow is performed as usual.

The dynamic assignment of the 256 gateway GIDs is the basic element of the load balancing and high availability operations. For any change in gateway configurations (e.g., a cable is dropped, an Ethernet link is disabled, or an appliance is powered off), the gateway GIDs get reassigned by MLNX-GW to be handled by other ConnectX cards. From the end-node perspective, nothing is changed. The same GID and LID remain, even when handled by a different ConnectX HCA.

NVIDIA Skyway includes the NVIDIA Gateway operating system, MLNX-GW, which manages the appliance and handles the high availability and load balancing between the ConnectX cards and between gateway appliances. For a detailed description of MLNX-GW, see please see NVIDIA MLNX-GW User Manual for NVIDIA Skyway or contact your NVIDIA representative.

The list of certifications per system for different regions of the world (such as EMC, safety, and others) is located on the NVIDIA Netowrking website at http://www.mellanox.com/page/environmental_compliance.