General Networkwide Deployment Guidelines
Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Onyx™operating system.
Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.
The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.
Ethernet Guidelines
The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).
For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.
Make sure MTU on the Ethernet router connected to Skyway isat least 2 bytessmaller than the InfiniBand IPoIB MTU configuration (e.g. set InfiniBandMTU to 4092 and Ethernet MTU to 4090).
InfiniBand Guidelines
All InfiniBand ports must be connected to the same InfiniBand fabric.
Subnet Manager Configuration
Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.
Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously.
If opensm runs on an InfiniBand switch, configure the following:
switch (config) # ib sm virt enable
switch (config) # ib sm virt-max-ports-in-process 0
If opensm runs on a host, add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf):
# Virtualization support
# 0: Ignore Virtualization - No virtualization support
# 1: Disable Virtualization - Disable virtualization on all
# Virtualization supporting ports
# 2: Enable Virtualization - Enable (virtualization on all
# Virtualization supporting ports
virt_enabled 2
# Maximum number of ports to be processed simultaneously
# by Virtualization Manager (0 - process all pending ports)
virt_max_ports_in_process 0
The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".
For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2.
IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.
On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example, 1.1.1.3).
# ifconfig ib0
1.1.1.2/24# ip route add0/0via1.1.1.3WarningThe NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.
Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example, 2.2.2.1).
gateway > enable gateway # configure terminal gateway (config) #
interfaceib port-channel1ip address1.1.1.1/24gateway (config) #interfaceib port-channel1virtual ip address1.1.1.3/24gateway (config) #interfaceethernet port-channel1ip address2.2.2.2/24gateway (config) # ip route0.0.0.0/02.2.2.1Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example, 2.2.2.2).
Below is an example using a NVIDIA-Onyx-based switch with port 1/1 connected to the gateway.
In this example, ports 1-8 on the router (see line 7) are connected to the 8 Ethernet ports on Skyway.eth_router > enable eth_router # configure terminal eth_router (config) # ip routing eth_router (config) # lacp eth_router (config) #
interfaceport-channel1eth_router (configinterfaceport-channel1) # exit eth_router (config) #interfaceethernet1/1-1/8channel-group1mode active eth_router (config) # vlan2eth_router (config vlan2) # exit eth_router (config) #interfaceport-channel1switchport access vlan2eth_router (config) #interfacevlan2ip address2.2.2.1/24eth_router (config) # ip route1.1.1.0/242.2.2.2WarningNote that the above connection describes a connection between Skyway and a single Ethernet router with LAG. It is possible to also connect to two Ethernet routers in an MLAG configuration. For more information, see the following community post for MLAG configuration on NVIDIA Onyx-based switches: support.mellanox.com/s/article/how-to-configure-mlag-on-mellanox-switches.
To ensure proper deployment, ping between a host in the subnet 192.168.1.0/24 and the InfiniBand host with IP 1.1.1.2 should be successful.