Q-in-Q Encapsulation per VF in Linux (VST)
This feature is supported on ConnectX-3 Pro and ConnectX-5 adapter cards only.
ConnectX-4 and ConnectX-4 Lx adapter cards support 802.1Q double-tagging (C-tag stack- ing on C-tag), refer to "802.1Q Double-Tagging" section.
This section describes the configuration of IEEE 802.1ad QinQ VLAN tag (S-VLAN) to the hypervisor per Virtual Function (VF). The Virtual Machine (VM) attached to the VF (via SR- IOV) can send traffic with or without C-VLAN. Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to the physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF.
The setup assumes there are two servers equipped with ConnectX-3 Pro/ConnectX-5 adapter cards.
Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag
Firmware version 2.36.5150 or higher must be installed for ConnectX-3 Pro HCAs
Firmware version 16.21.0458 or higher must be installed for ConnectX-5 HCAs
The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.
In order to configure SR-IOV in Ethernet mode for ConnectX-3 Pro adapter cards, please refer to "Configuring SR-IOV for ConnectX-3/ConnectX-3 Pro" section.
In order to configure SR-IOV in Ethernet mode for ConnectX-5 adapter cards, please refer to "Configuring SR-IOV for ConnectX-4/ConnectX-5 (Ethernet)" section. In the following configuration example, the VM is attached to VF0.
Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports.
Enable QinQ support in the hardware. Set the phv-bit flag using ethtool (on the hypervisor).
ethtool --set-priv-flags ens2 phv-bit on
Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:
By using sysfs only if the Kernel version used is v4.9 or older:
echo
'vlan 100 proto 802.1ad'> /sys/class/net/ens2/vf0/vlan_infoBy using the ip link command (available only when using the latest Kernel version):
# ip link set dev ens2 vf
0vlan100proto802.1adCheck the configuration using the ip link show command:
# ip link show ens2
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu1500qdisc mq state UP mode DEFAULT groupdefaultqlen1000link/ether 7c:fe:90:19:9e:21brd ff:ff:ff:ff:ff:ff vf0MAC00:00:00:00:00:00, vlan100, vlan protocol802.1ad , spoof checking off, link-state auto vf1MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf2MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf3MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf4MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto
Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):
# ip link set dev ens2 vf
0vlan100qos3proto802.1adCheck the configuration using the ip link show command:
# ip link show ens2
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu1500qdisc mq state UP mode DEFAULT groupdefaultqlen1000link/ether 7c:fe:90:19:9e:21brd ff:ff:ff:ff:ff:ff vf0MAC00:00:00:00:00:00, vlan100, qos3, vlan protocol802.1ad , spoof checking off, link-state auto vf1MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf2MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf3MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state auto vf4MAC00:00:00:00:00:00, vlan4095, spoof checking off, link-state autoRestart the driver in the VM attached to that VF.
(VM1)# /etc/init.d/openidb restart
Create a VLAN interface on the VM and add an IP address.
ip link add link ens5 ens5.
40type vlan protocol802.1q id40ip addr add42.134.135.7/16brd42.134.255.255dev ens5.40ip link set dev ens5.40upTo verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.
For further examples, refer to HowTo Configure QinQ Encapsulation per VF in Linux (VST) for ConnectX-3 Pro Community post.
Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:
By using sysfs:
echo
'100:0:802.1ad'> /sys/class/net/ens1f0/device/sriov/0/vlanBy using the ip link command (available only when using the latest Kernel version):
ip link set dev ens1f0 vf
0vlan100proto802.1adCheck the configuration using the ip link show command:
# ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500qdisc mq state UP mode DEFAULT qlen1000link/ether ec:0d:9a:44:37:84brd ff:ff:ff:ff:ff:ff vf0MAC00:00:00:00:00:00, vlan100, vlan protocol802.1ad, spoof checking off, link-state auto, trust off vf1MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf2MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf3MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf4MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off
Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):
ip link set dev ens1f0 vf
0vlan100qos3proto802.1adCheck the configuration using the ip link show command:
# ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500qdisc mq state UP mode DEFAULT qlen1000link/ether ec:0d:9a:44:37:84brd ff:ff:ff:ff:ff:ff vf0MAC00:00:00:00:00:00, vlan100, qos3, vlan protocol802.1ad, spoof checking off, link-state auto, trust off vf1MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf2MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf3MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf4MAC00:00:00:00:00:00, spoof checking off, link-state auto, trust offCreate a VLAN interface on the VM and add an IP address.
ip link add link ens5 ens5.
40type vlan protocol802.1q id40ip addr add42.134.135.7/16brd42.134.255.255dev ens5.40ip link set dev ens5.40upTo verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.