General Networkwide Deployment Guidelines
Skyway GA100 is an appliance-based InfiniBand-to-Ethernet gateway, enabling Ethernet-based communications to access the InfiniBand datacenter and vice versa. The following section describes networkwide guidelines and provides a specific example when using a NVIDIA Ethernet switch running NVIDIA Onyx™ operating system.
Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.
The IPs and subnet masks described in the diagram below are provided as an example. Actual IP and subnet masks may differ in actual customer deployments.
The Skyway solution mandates a dedicated subnet between the appliance and the layer-3 Ethernet switch/router.
This subnet is not visible from outside and can be configured to be ANY class-C subnet, including IPs which are not part of the customer’s IP pool.
Skyway appliance MUST be directly attached to L3 Ethernet switch(es)/router(es) (at least one) to complement the function of an IPoIB router. The role of divisions is as follows:
Skyway forwards IPoIB traffic to a single IP address/MAC on the Ethernet side; the directly-attached L3 Ethernet switch(es)/router(es) provide Ethernet-side load balancing functionality (auch as MLAG) and routing.
All Skyway devices with a single HA domain share the same virtual IP.
Ethernet Guidelines
The connection between the Skyway and the Ethernet router requires configuring a LAG with active LACP on the Ethernet router (see step 3 in the "Configuring IP Addresses and Routes" below).
For increased resiliency, it is recommended to configure Ethernet routers in an MLAG configuration.
Make sure MTU on the Ethernet router connected to Skyway is at least 2 bytes smaller than the InfiniBand IPoIB MTU configuration (e.g. set InfiniBand MTU to 4092 and Ethernet MTU to 4090).
InfiniBand Guidelines
All InfiniBand ports must be connected to the same InfiniBand fabric.
Subnet Manager Configuration
Ensure the Subnet Manager is running in the InfiniBand cluster with IPoIB enabled.
Virtualization must be enabled by the Subnet Manager (SM). It is recommended to unlimit the maximum number of ports that are processed simultaneously.
If opensm runs on an InfiniBand switch, configure the following:
switch
(config) # ib sm virt enable
switch
(config) # ib sm virt-max-ports-in-process 0
If opensm runs on a host, add the following lines to the opensm.conf (by default at /etc/opensm/opensm.conf):
# Virtualization support
# 0
: Ignore Virtualization - No virtualization support
# 1
: Disable Virtualization - Disable virtualization on all
# Virtualization supporting ports
# 2
: Enable Virtualization - Enable (virtualization on all
# Virtualization supporting ports
virt_enabled 2
# Maximum number of ports to be processed simultaneously
# by Virtualization Manager (0
- process all pending ports)
virt_max_ports_in_process 0
The first port (port #1) of each HCA is an InfiniBand port and the second port (port #2) is an Ethernet port; therefore, the configuration of an InfiniBand "device/port" value should be "x/1" and the configuration of an Ethernet "device/port" value should be "x/2".
For example, for HCA #7, the configuration of the InfiniBand port is 7/1 and of the Ethernet port is 7/2.
IP addresses, subnet masks, port numbers, and interface names are used as an example and may vary according to the actual connectivity of the customer's deployment.
On the relevant InfiniBand nodes, configure an IP address on each InfiniBand port designated for the Skyway deployment (e.g., ib0). In addition, configure a default route with the Skyway IP as next hop (in this example, 1.1.1.3).
# ifconfig ib0
1.1
.1.2
/24
# ip route add0
/0
via1.1
.1.3
WarningThe NVIDIA Skyway IP which is configured as next hop should match the virtual IP of the InfiniBand port channel of the NVIDIA Skyway appliance.
Access and configure an IP address on the gateway's Ethernet and InfiniBand ports and configure a virtual IP address on the InfiniBand port. In addition, configure a route to the customer's Ethernet networks via the IP assigned on the Ethernet router's port (in this example, 2.2.2.1).
gateway > enable gateway # configure terminal gateway (config) #
interface
ib port-channel1
ip address1.1
.1.1
/24
gateway (config) #interface
ib port-channel1
virtual ip address1.1
.1.3
/24
gateway (config) #interface
ethernet port-channel1
ip address2.2
.2.2
/24
gateway (config) # ip route0.0
.0.0
/0
2.2
.2.1
Detect the ports on the Ethernet router that are connected to the gateway, assign the LAG to a VLAN, and configure the IP address on the VLAN interface. In addition, configure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel (in this example, 2.2.2.2).
Below is an example using a NVIDIA-Onyx-based switch with port 1/1 connected to the gateway.
In this example, ports 1-8 on the router (see line 7) are connected to the 8 Ethernet ports on Skyway.eth_router > enable eth_router # configure terminal eth_router (config) # ip routing eth_router (config) # lacp eth_router (config) #
interface
port-channel1
eth_router (configinterface
port-channel1
) # exit eth_router (config) #interface
ethernet1
/1
-1
/8
channel-group1
mode active eth_router (config) # vlan2
eth_router (config vlan2
) # exit eth_router (config) #interface
port-channel1
switchport access vlan2
eth_router (config) #interface
vlan2
ip address2.2
.2.1
/24
eth_router (config) # ip route1.1
.1.0
/24
2.2
.2.2
WarningNote that the above connection describes a connection between Skyway and a single Ethernet router with LAG. It is possible to also connect to two Ethernet routers in an MLAG configuration. For more information, see the following community post for MLAG configuration on NVIDIA Onyx-based switches: https://enterprise-support.nvidia.com/s/article/how-to-configure-mlag-on-mellanox-switches.
WarningAdditional commands might be needed the ensure the MTU restrictions described in Ethernet Guidelines above.
To ensure proper deployment, ping between a host in the subnet 192.168.1.0/24 and the InfiniBand host with IP 1.1.1.2 should be successful.