FDR SwitchX-2 based gateway (e.g. MSX6036G) is Proxy-ARP-based gateway. The Ethernet and InfiniBand networks are on the same IP subnet. NVIDIA Skyway (HDR/EDR gateway) is a Layer-3 (IP router) solution. The Ethernet and InfiniBand networks are not on the same IP subnet.
In order to migrate from MSX6036G-based gateway to NVIDIA Skyway, either the Ethernet or InfiniBand host needs to be moved to a different IP subnet. The following section describes the migration procedure, in case a customer wishes to handle this transition with minimum interruption to a working cluster.
Each operating system has its own configurations. In the examples below, Linux host Red Hat Enterprise Linux release 8.0 (Ootpa) and NVIDIA Onyx were used.
In the following section, migration instructions are provided. The process assumes that the FDR network works well and the Skyway-based connectivity is gradually added. The diagram below depicts this process. The lower part of the diagram (with the dotted lines) shows the FDR-gateway connectivity. Gradually bring up the Skyway-based connectivity (the upper part of the diagram) by following the "Migration Procedure" below. At the end of the process, the traffic will flow through NVIDIA Skyway. During this process, short interruption in traffic flow will occur. After the completion of the migration, all traffic will go through the Skyway.
1. Ethernet host to InfiniBand hosts with Iperf application.
2. Ethernet host to InfiniBand hosts with GPFS perf.
Network Devices
NVIDIA Skyway:
Step A: Connect NVIDIA Skyway to switches and configure IP addresses for NVIDIA Skyway InfiniBand and Ethernet interfaces (See "Starting the Command Line Interface (CLI)" in the Getting Started section)
Step B: Configure a static route on NVIDIA Skyway on the Ethernet network via the IP assigned on the Ethernet router's port (For more details, see "General Networkwide Deployment Guidelines" in the Getting Started section)
Ethernet Router:
Step C: Configure a router-port (via LAG or MLAG) on the Ethernet switch in the direction of NVIDIA Skyway.
Step D: C onfigure a route to the IPoIB network via the IP assigned to the gateway's Ethernet port-channel(for more details, see "General Networkwide Deployment Guidelines" in the Getting Started section)
Step E: Configure Interface VLAN to act as a “default gateway” (in the example, above, it is "21.144.0.1") for the Ethernet hosts (one example is that each port will be connected to the server and access VLAN X).
**The traffic will be interrupted at this point!
Ethernet Host:
Step F: Assign a new IP addresses and default gateway (in the example, above, the network was changed from "11.144.0.0/16" to "21.144.0.0/16") on the Ethernet hosts (can be done via DHCP or static).
**The traffic will resume after this point!
InfiniBand Host:
Step G: Assign default gateway on InfiniBand hosts (can be done via DHCP or static).
This example shows modification of Ethernet subnet. It is equally optional to change the InfiniBand subnet instread.
