Ethernet Virtual Private Network - EVPN
VXLAN enables layer 2 segments to extend over an IP core (the underlay). The initial definition of VXLAN (RFC 7348) does not include any control plane and relied on a flood-and-learn approach for MAC address learning.
EVPN is a standards-based control plane for VXLAN defined in RFC 7432 and draft-ietf-bess-evpn-overlay that allows for building and deploying VXLANs at scale. It relies on multi-protocol BGP (MP-BGP) to exchange information and uses BGP-MPLS IP VPNs (RFC 4364). It enables not only bridging between end systems in the same layer 2 segment but also routing between different segments (subnets). There is also inherent support for multi-tenancy.
Cumulus Linux fully supports EVPN as the control plane for VXLAN, including for both intra-subnet bridging and inter-subnet routing, and provides these key features:
- VNI membership exchange between VTEPs using EVPN type-3 (Inclusive multicast Ethernet tag) routes.
- Host MAC and IP address exchange using EVPN type-2 (MAC and IP advertisement) routes.
- Host/VM mobility support (MAC and IP moves) through exchange of the MAC Mobility Extended community.
- Dual-attached hosts via VXLAN active-active mode. MLAG synchronizes MAC addresses between the peer switches.
- ARP/ND suppression is on VNIs by default, which enables VTEPs to suppress ARP flooding over VXLAN tunnels.
- Exchange of static MAC addresses through EVPN.
- Inter-subnet routing for both IPv4 and IPv6 hosts: Distributed symmetric routing between different subnets and centralized routing.
- Prefix-based routing using EVPN type-5 routes (EVPN IP prefix route).
- Layer 3 multi-tenancy.
- IPv6 tenant routing.
- ECMP for overlay networks on NVIDIA Spectrum-A1 ASICs. ECMP occurs in the overlay when there are multiple next hops.
- Head end replication is on by default. Cumulus Linux supports up to 128 VTEPs with head end replication.
Cumulus Linux supports the EVPN address family with both eBGP and iBGP peering. If you configure underlay routing with eBGP, you can use the same eBGP session to carry EVPN routes. In a typical 2-tier Clos network where the leafs are VTEPs, if you use eBGP sessions between the leafs and spines for underlay routing, the same sessions exchange EVPN routes. The spine switches act as route forwarders and do not install any forwarding state as they are not VTEPs. When the switch exchanges EVPN routes over iBGP peering, you can use OSPF as the IGP or resolve next hops using iBGP.
Cumulus Linux disables data plane MAC learning by default on VXLAN interfaces. Do not enable MAC learning on VXLAN interfaces: EVPN installs remote MACs.