Created on Jun 30, 2019
This Reference Deployment Guide (RDG) provides instructions on how to integrate Hardware VTEPs with VMware NSX-V in OVSDB Server High-Availability Mode on Cumulus Linux.
This guide assumes the following software and drivers are installed:
- VMware ESXi 6.7 Update 1
- vCenter 6.7 Update 1
- VMware NSX Data Center for vSphere version 6.4.4
- Cumulus Linux version 3.7.2. OVSDB server HA mode is in early access on this Cumulus Linux release.
- VMware NSX Data Center for vSphere Documentation
- Cumulus Linux OS
- Integrating Hardware VTEPs with VMware NSX-V
- OVSDB Server High Availability
- ConnectX®-5 Single/Dual-Port Adapter
- NVIDIA Inbox Driver
- How to Configure Network Bonding or Teaming in Ubuntu
NVIDIA Spectrum ® switches running Cumulus Linux OS supports integration with VMware NSX Data Center in both standalone and OVSDB server high availability.
For OVSDB server high availability mode (active-active data-plane), two switches must have MLAG configuration and act as an MLAG cluster.
The MLAG peers contain OVSDB server and vtepd configuration.
The OVSDB servers synchronize their databases and always maintain the replicated state unless any failover occurs.
Both of the vtepd components communicate with the active OVSDB server to read the configuration and then push it to the kernel.
Only the active OVSDB server communicates with the NSX controller unless a failover occurs and then the standby OVSDB server takes over automatically.
Although the Cumulus switches are configured as an MLAG pair, the NSX controller sees them as a single system (the NSX controller is not aware that multiple switches exist).
Hardware and Software Requirements
1. ESXi server platform with an adapter card based on ConnectX®-4/5 HCA devices.
2. 4 leaf and 2 spine switches based on NVIDIA Scale-Out SN2000 Ethernet Switch Series.
3. VMware vSphere 6.7 u1 cluster installed and configured.
4. VMware vCenter 6.7 u1 installed and configured.
5. VMware NSX Data Center for vSphere version 6.4.4 installed and configured.
6. Installer Privileges: The installation requires administrator privileges on the target machine.
In this guide, we will cover two use cases:
- Each Ubuntu physical server is able to connect the VM in the same VLAN-VNI on the other side.
- Each Ubuntu physical server is able to connect the VM in different VLAN-VNI on the other side.
For that purpose, NSX Edge will be implemented. It will allow the hosts from different subnets (VLANs/VNIs) to communicate.
Before you start, make sure you are familiar with VMware vSphere, vCenter, NSX Data Center and Cumulus Linux deployment and management procedures.
This guide does not contain step-by-step instructions for performing all of the required standard vSphere, vCenter and NSX-V installation and configuration tasks because they often depend on customer requirements.
In this guide, we are using the following hardware specification.
Before the OVSDB HA configuration, we need to make sure that both pairs of leaf switches are running an MLAG configuration. In this document, Leaf Switch 3 and Leaf Switch 4 (connected to the ESXi servers) are also running MLAG for HA.
Make sure both Ubuntu physical servers have a bond interface in active-active mode (lacp mode 4).
To configure MLAG, run the following commands.
Verify MLAG status on both MLAG pairs:
Verify MLAG status on both MLAG pairs:
On Leaf Switch 1 and Leaf Switch 2 there are loopback interfaces with MLAG anycast-ip (on top) for VXLAN tunnel usage.
On Leaf Switch 3 and Leaf Switch 4 there is IP-virtual address (VRR) of MLAG for VTEPs reachability on the ESXi hosts (VLAN 130 chosen for this purpose).
BGP Underlay Routing Configuration
Configure BGP underlay network reachability between all leaf and spine switches.
The connectivity should be between the MLAG anycast-ip to the VTEPs on the ESXi machines.
Verify the BGP state and underlay routing:
Connectivity from MLAG anycast-ip (Leaf Switches 1 and 2) to VLAN130 (Leaf Switches 3 and 4):
vSphere and NSX Configuration
Virtual Distributed Switch (VDS) Configuration
Make sure that you have a VDS with all necessary configuration in your environment Port Groups.
NSX Manager and Controllers Configuration
After underlay network reachability is verified, make sure that you have an NSX Manager and 3 NSX controllers.
All of them should be on the same subnet as the management network of the switches (in this case it’s 192.168.1.x/24).
ESXi Hosts Preparation
Make sure that all ESXi hosts are prepared and each one has VTEP with the IP addresses in 192.168.130.0/24 subnet - 192.168.130.51/52/53.
LACP Configuration on vSphere Side
As we saw earlier, all ESXi machines are connected in active-active (LACP) LAG to Leaf Switch 3 and Leaf Switch 4 (clag-1/2/3).
Create LACP LAG on the VDS, later all ESXi and virtual switches will use this LAG to connect to active-active mode to the physical switches.
For creating LACP LAG, go into Networking category in vSphere web client.
Select the VDS, and go to Configure tab.
In our lab we prepared a VDS.
Select LACP and press + to add a new LAG. In the pop-up window, type LAG’s Name, Number of ports, Mode and Load balancing mode.
Once the LAG created, run the Migrating network traffic to LAGs wizard to assign the relevant NICs to it. From this wizard it is possible to go over all the elements needed.
Click the 3th step – Manage Distributed Port Groups, select Teaming and failover and click Next.
Click Select distributed port groups:
Select the needed port groups (Mgmt DVS, storage and vMotion VDS) and click OK.
When all are selected, go to the next step by clicking Next.
Use the arrows to move lag1 to be the Active uplink and other Uplinks to be Unused.
Click Next and Finish.
Go to the wizard again and click the 2nd step Add and Manage Hosts.
Click the Manage host networking task and then Next.
Press on +Attached hosts.
Select the ESXi machines that need to connect to the LAG and click OK.
After hosts attached click Next.
Select Manage physical adapters, Migrate virtual machine networking checkboxes and click Next.
VMs can be migrated into the logical switches by selecting Migrate virtual machine networking. Otherwise, they can be added when the logical switch is created (covered later in the document).
Select the physical adapters that you wish to map into the LAG and click Assign uplink.
In the pop-up window, select the LAG port (lag1-0) for the 1st physical vmnic and click OK.
Add the 2nd vmnic to the 2nd LAG port exactly like the first one. Now both ports are assigned to the LAG.
Repeat the steps above to add all vmnics of the other ESXi machines to the LAG. Once all added, click Next.
On Analyze VM networking click Next.
Migrate the VMs into the logical switches by selecting the needed VM and clicking Assign port groups.
Select the logical switch (for VM01 – LogicalSwitch1 and for VM02 – LogicalSwitch2) and click OK.
When both VMs are attached to the relevant network (logical switch – VNI), click Next and then Finish.
OVSDB Server HA Configuration on Cumulus Linux and HW-VTEPs on ESXi Hypervisor
OVSDB Server Configuration on Cumulus Linux
Prior to configuring the HW-VTEP gateway, the logical switches, and ports that comprise the VXLAN, openvswitch-vtep service must be enabled and started.
Enable and start the openvswitch-vtep service on Leaf Switch 1 and Leaf Switch 2.
Run the OVSDB server configuration script on both the MLAG primary and secondary switches.
On the switch that will act as ACTIVE OVSDB server run the following script.
- db_ha active specifies that the OVSDB server on this switch is the active server
- db_ha_vip is any unused IP address in the peerlink subnet (4094 is typically used). This creates a /32 route that can be reached from either MLAG switch (169.254.1.3:9998 in the example below).
- db_ha_repl_sv specifies the IP address of the active OVSDB server (169.254.1.1:9999 in the example below). The standby OVSDB server uses this IP address to synchronize the database.
- controller_ip is the IP address of the NSX controller (192.168.1.221 in the example below)
- The ID for the VTEP (vtep1 in the example below)
- The datapath IP address of the VTEP – VXLAN anycast-ip (18.104.22.168 in the example below)
- The management IP address of the switch (192.168.1.7 in the example below). This interface is used for control traffic.
On the switch that will act as STANDBY OVSDB server, run exactly the same “vtep-bootstrap” command with the same options as on the ACTIVE OVSDB server but replace “db_ha active” with “db_ha standby”.
Once the service script is executed, certificate files will be created: hostname-cert.pem and hostname-privkey.pem.
Copy the certificate files from the active OVSDB server to the same location on the standby OVSDB server.
To complete the OVSDB server configuration, run the following commands on both active and standby OVSDB servers (both MLAG switches).
HW-VTEP Configuration on ESXi Hypervisor
Configure the switch as a VTEP Gateway (Hardware VTEP).
Go to vSphere Web Client into Networking and Security and click Service Definitions category.
Select Hardware Devices tab. To add the switch press +.
In the pop-up window, type HW-VTEP name and insert the Certificate created on the active OVSDB server.
Copy the certificate from the switch (make sure to copy from BEGIN CERTIFICATE to END CERTIFICATE).
Make sure to enable BFD service and click OK.
The active OVSDB server is now added as Hardware VTEP.
On ESXi Hypervisor, check the Connectivity and the BFD.
Connectivity will show UP and BFD ✓ if HW-VTEP added successfully.
On Leaf Switch 1 and Leaf Switch 2, verify that the OVSDB active and standby servers configured successfully.
The state of the active OVSDB server should be “active”.
The state of the standby OVSDB server should be “backup”.
The active OVSDB server is the only one that is connected to the NSX controllers. In case of failure, the standby OVSDB server will connect to the controllers.
Check active OVSDB server for connectivity to the NSX controllers.
Make sure that the BFD sessions between the VTEPs are UP and active.
Transport Zone and Logical Network Configuration
To configure the Transport Layer, go to vSphere Web Client into Networking and Security tab and click the Installation and Upgrade category.
Click on Logical Network Settings, configure VXLAN Port and Segment IDs (VNIs).
Click Edit on VXLAN Port, type 4789 and SAVE.
Click Edit on the Segment IDs (VNIs), type 5000-5999 and SAVE.
Go to Transport Zones tab and click +ADD.
Type transport zone name, select Unicast for VXLAN control plane handling by NSX Controller Cluster and select the ESXi Cluster. To finish click ADD.
Now the replication cluster needs to be configured, go back to Service Definitions category.
Go to Hardware Devices tab and Edit the Replication Cluster Nodes.
In the pop-up window, from the available objects, select the replication nodes (ESXi Hypervisors) and move them right using the arrow, then click OK.
These Hypervisors are now connected to the NSX controllers for BUM traffic replication between the VTEPs.
Layer 2 Traffic over VXLAN Tunnels (L2 Stretch)
For building connectivity between hosts, the logical layer must be configured. It requires the definition of logical switches (VXLAN instances) and the logical ports on them.
The configuration below is in reference to the following setup diagram.
Servers/VMs are connected as follows:
- VM01 (orange) and Ubuntu (orange) – VLAN1634 (VNI5000)
IP addresses: VM01 – 192.168.34.101/24, Ubuntu – 192.168.34.204/24
- VM02 (aqua) and Ubuntu (aqua) – VLAN1635 (VNI5001)
IP addresses: VM02 – 192.168.35.102/24, Ubuntu – 192.168.35.205/24
On Leaf1 and 2, Ubuntu orange is connected to MLAG port clag-1 and Ubuntu aqua to clag-2.
On Leaf3 and 4, ESXi machines connected to MLAG ports clag-1,clag-2.
Each VLAN will be mapped to the appropriate VNI. For that, two logical switches need to be created, one for VNI5000 and another for VNI5001.
Logical Switch Configuration
To configure the logical switches, go to the Logical Switches category and click +ADD.
In the pop-up window, type the logical switch’s name “LogicalSwitch1”, select transport zone, Unicast replication mode and enable IP Discovery. When done, click ADD.
New logical switch “LogicalSwitch1” is added successfully and the first Segment ID (VNI) is set to it (VNI 5000).
For the second VNI (5001) a new logical switch needs to be created—“LogicalSwitch2”. Once created, it will take the next Segment ID (VNI) automatically. To create it, follow exactly the same steps for “LogicalSwitch1” creation.
Now, both logical switches should be displayed.
Logical switchports need to be defined on the logical switches. This binds the VLAN-to-VNI for each switch port.
For that, go to the Logical Switches category, select the logical switch, click ACTIONS and select Manage Hardware Bindings.
Click +ADD to add a new port binding line, then Select the switch port/s to bind to the HW VTEP.
In the pop-up window, select the switch port (clag-1 connected to Ubuntu-blue) to bind and click OK.
Add a VLAN to the switch port binding (VLAN1634).
To bind the port connected to the second Ubuntu server (VLAN1635), perform the same steps as with LogicalSwitch1.
Select LogicalSwitch2, ACTIONS and Manage Hardware Bindings.
Click +ADD again to add a new line, then select the needed port (clag-2) and choose its VLAN (VLAN1635).
Now, there is a hardware port binding to each of the logical switches.
Connect the VMs located on the ESXi machines to the same logical switches, each VM in the appropriate logical switch. To add a VMs to a logical switch, select the switch and click ADD VM.
Select the VM (VM01) from Available Objects and click the arrow. Once it moved to the Selected Objects, click NEXT.
Select the vNIC of the VM and click NEXT and then FINISH.
Repeat the same steps to add VM02 the LogicalSwitch2. Once added, there will be VMs connected to the logical switches.
Add the logical switches to the LAG (lag1) that was created before. Go to Networking, choose the Logical Switch, click the Configure tab and edit to Policies settings.
In the pop-up window, go to Teaming and failover category and change the Failover order. Use the arrows to move the lag1 interface as Active uplink and the regular ports to Unused uplinks.
After all is configured, we get a VXLAN L2 stretch on VLAN1634 and VLAN1635.
Leaf switches 1 and 2 have VXLAN 5000 and 5001 interfaces in MLAG:
Check VLANs and VXLAN interfaces created on leaf1 and leaf2 by the NSX controller (should be identical).
Leaf switches 1 and 2 should have the MAC addresses of the remote VMs located on the ESXi machines. All of them pushed by the NSX controller from the appropriate VXLAN interfaces.
Check connectivity between Ubuntu servers to the VMs in the same VNI (L2 stretch):
Connectivity from Ubuntu-orange to VM01 (from 192.168.34.204 to 192.168.34.101).
Connectivity from Ubuntu-azure to VM02 (from 192.168.35.205 to 192.168.35.102).
Layer 3 Traffic Over VXLAN Tunnels (VXLAN Centralized Routing)
As we configured before, each Ubuntu server was able to connect the VM in the same VLAN-VNI on the other side.
If there is a need to communicate between the different VLANs, VXLAN routing must be enabled. For that purpose, NSX Edge will be implemented. It will allow hosts from different subnets (VLANs/VNIs) to communicate.
The routing is centralized, which means that the routing is done by a specific logical switch (NSX Edge) located on the ESX hypervisor which acts as the default gateway for all hosts in a particular subnet.
When Ubuntu from VLAN1634 needs to communicate with hosts in VLAN1635, the traffic will need to get the default gateway (NSX Edge) located on the ESX, then the routing decision will be made by it and the traffic will be routed to a different VNI.
NSX Edge Configuration
To create routing between the VLANs (VNIs), L3 VNI needs to be crated.
Add new a Logical Switch that will be used for L3 VNI. The creation process is the same as the previously added logical switches. Next VNI will be added to it automatically.
New logical switch created with VNI5002.
To create and configure NSX Edge. Go to NSX Edge category and click +.
In the pop-up window, select Edge Services Gateway, type Edge’s name, Hostname, check to Deploy NSX Edge checkbox and click Next (High-Availability can be checked if HA mode is desired).
Set NSX Edge CLI User Name, Password, and check the Enable SSH checkbox and click Next.
The password should match the following conditions
Configure desired Appliance Size and location (by clicking +).
Select the Cluster, Database and click OK.
Now NSX Edge resource pool created, click Next.
To configure NSX Edge interfaces by clicking +.
In the pop-up window, sent vNIC Name (NSX interface vNIC), Internal Type and connect to the Edge-HA logical switch by clicking Select and selecting Edge-HA.
Set interface Primary IP Address and Subnet Prefix Length (you can use a subnet of /30) and click OK.
After the NSX Edge interface created, click “Next”.
Skip the NSX Edge Default Gateway settings and continue to the next section by clicking Next.
To configure Firewall and HA. Set the firewall to Accept all, select the vNIC Edge-HA and configure Management IPs for NSX Edge HA (as mentioned, it’s optional to select HA).
Now, the NSX Edge is created, click Finish to deploy it.
NSX Edge is now created and deployed (in HA mode with two Edge VMs).
Now, the logical switches of VNI5000 and 5001 need to be added to the NSX Edge and VLANs default gateway addresses should be set.
Double click the created NSX Edge and go to the Interfaces section.
Select vnic1 and click EDIT.
In the pop-up window, type vNIC name (“VLAN1634 Gateway”), select Internal Type, connect to “LogicalSwitch1” (of VNI5000), set VLANs Default Gateway address by pressing +ADD. Once finished, click SAVE.
Now, VLAN1634 (VNI5000) has a default gateway for routing.
Select vNic 2 and add the LogicalSwitch2 for VLAN1635 (VNI5001) gateway. Follow the same steps as for vNIC 1.
When both Gateways are created, we have routing between the blue and the azure subnets (VNI5000 and VNI5001):
Connectivity from Ubuntu-blue to VM02 (from 192.168.34.204 to 192.168.35.102).
Connectivity from Ubuntu-azure to VM01 (from 192.168.35.205 to 192.168.34.101).
Leaf Switches 1 and 2 have now the Edges (HA) MAC addresses from both VNI5000 and 5001.
In normal situation Leaf Switch 1 is active OVSDB and Leaf Switch 2 is standby OVSDB server.
If the active OVSDB server fails (Leaf Switch 1), standby OVSDB server (Leaf Switch 2) will connect to the NSX controllers automatically and switch its role to ACTIVE OVSDB server (till the original active server is back again).
When Leaf Switch 1 fails, Leaf Switch 2 displays.
Once the active OVSDB server is up again (Leaf Switch 1), the standby server (Leaf Switch 2) will be disconnected from the NSX controllers and sync from the active OVSDB server again.
Leaf Switch 1 will be active again with the connection to the NSX Controllers (with BFD).
Leaf Switch 2 new state.