VXLAN Hardware Stateless Offloads
VXLAN technology provides scalability and security challenges solutions. It requires extension of the traditional stateless offloads to avoid performance drop. ConnectX-3 Pro and ConnectX-4 family adapter card offer the following stateless offloads for a VXLAN packet, similar to the ones offered to non-encapsulated packets. VXLAN protocol encapsulates its packets using outer UDP header.
Available hardware stateless offloads:
Checksum generation (Inner IP and Inner TCP/UDP)
Checksum validation (Inner IP and Inner TCP/UDP). This will allow the use of GRO (in ConnectX-3 Pro card only) for inner TCP packets.
TSO support for inner TCP packets
RSS distribution according to inner packets attributes
Receive queue selection - inner frames may be steered to specific QPs
VXLAN Hardware Stateless Offloads requires the following prerequisites:
HCA and their minimum firmware required:
ConnectX-3 Pro - Firmware v2.32.5100
ConnectX-4 - Firmware v12.14.xxxx
ConnectX-4 Lx - Firmware v14.14.xxxx
Operating Systems:
RHEL7, Ubuntu 14.04 or upstream kernel 3.12.10 (or higher)
ConnectX-3 Pro Supported Features:
DMFS enabled
A0 static mode disabled
To enable the VXLAN offloads support load the mlx4_core driver with Device-Managed Flow- steering (DMFS) enabled. DMFS is the default steering mode.
To verify it is enabled by the adapter card:
Open the /etc/modprobe.d/mlnx.conf file.
Set the parameter debug_level to "1".
options mlx4_core debug_level=
1
Restart the driver.
Verify in the dmesg that the tunneling mode is: vxlan.
The net-device will advertise the tx-udp-tnl-segmentation flag shown when running "etht- hool -k $DEV | grep udp" only when VXLAN is configured in the OpenvSwitch (OVS) with the configured UDP port.
Example:
$ ethtool -k eth0 | grep udp_tnl
tx-udp_tnl-segmentation: on
As of firmware version 2.31.5050, VXLAN tunnel can be set on any desired UDP port. If using previous firmware versions, set the VXLAN tunnel over UDP port 4789.
To add the UDP port to /etc/modprobe.d/vxlan.conf:
options vxlan udp_port=<number decided above>
VXLAN offload is enabled by default for ConnectX-4 family devices running the minimum required firmware version and a kernel version that includes VXLAN support.
To confirm if the current setup supports VXLAN, run:
ethtool -k $DEV | grep udp_tnl
Example:
ethtool -k ens1f0 | grep udp_tnl
tx-udp_tnl-segmentation: on
ConnectX-4 family devices support configuring multiple UDP ports for VXLAN offload. Ports can be added to the device by configuring a VXLAN device from the OS command line using the "ip" command.
Note: If you configure multiple UDP ports for offload and exceed the total number of ports supported by hardware, then those additional ports will still function properly, but will not benefit from any of the stateless offloads.
Example:
ip link add vxlan0 type vxlan id 10
group 239.0
.0.10
ttl 10
dev ens1f0 dstport 4789
ip addr add 192.168
.4.7
/24
dev vxlan0
ip link set up vxlan0
Note: dstport' parameters are not supported in Ubuntu 14.4.
The VXLAN ports can be removed by deleting the VXLAN interfaces.
Example:
ip link delete vxlan0
To verify that the VXLAN ports are offloaded, use debugfs (if supported):
Mount debugfs.
mount -t debugfs nodev /sys/kernel/debug
List the offloaded ports.
ls /sys/kernel/debug/mlx5/$PCIDEV/VXLAN
Where $PCIDEV is the PCI device number of the relevant ConnectX-4 family device.
Example:ls /sys/kernel/debug/mlx5/
0000
:81
:00.0
/VXLAN4789
VXLAN tunneling adds 50 bytes (14-eth + 20-ip + 8-udp + 8-vxlan) to the VM Ethernet frame. Please verify that either the MTU of the NIC who sends the packets, e.g. the VM virtio-net NIC or the host side veth device or the uplink takes into account the tunneling overhead. Meaning, the MTU of the sending NIC has to be decremented by 50 bytes (e.g 1450 instead of 1500), or the uplink NIC MTU has to be incremented by 50 bytes (e.g 1550 instead of 1500)
From upstream 3.15-rc1 and onward, it is possible to use arbitrary UDP port for VXLAN. Note that this requires firmware version 2.31.2800 or higher. Additionally, you need to enable this kernel configuration option CONFIG_MLX4_EN_VXLAN=y (ConnectX-3 Pro only).
On upstream kernels 3.12/3.13 GRO with VXLAN is not supported