Q-in-Q Encapsulation per VF in Linux (VST)
This feature is supported on ConnectX-3 Pro and ConnectX-5 adapter cards only.
ConnectX-4 and ConnectX-4 Lx adapter cards support 802.1Q double-tagging (C-tag stack- ing on C-tag) - refer to " 802.1QDouble-Tagging "section.
This section describes the configuration of IEEE 802.1ad QinQ VLAN tag (S-VLAN) to the hypervisor per Virtual Function (VF). The Virtual Machine (VM) attached to the VF (via SR- IOV) can send traffic with or without C-VLAN. Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to the physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF.
The setup assumes there are two servers equipped with ConnectX-3 Pro/ConnectX-5 adapter cards.
Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag
Firmware version 2.36.5150 or higher must be installed for ConnectX-3 Pro HCAs
Firmware version 16.21.0458 or higher must be installed for ConnectX-5 HCAs
The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.
In order to configure SR-IOV in Ethernet mode for ConnectX-3 Pro adapter cards, please refer to " ConfiguringSR-IOVforConnectX-3/ConnectX-3Pro" section.
In order to configure SR-IOV in Ethernet mode for ConnectX-5 adapter cards, please refer to " ConfiguringSR-IOVforConnectX-4/ConnectX-5(Ethernet) "section. In the following configuration example, the VM is attached to VF0.
Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports.
Enable QinQ support in the hardware. Set the phv-bit flag using ethtool (on the hypervisor).
ethtool --set-priv-flags ens2 phv-bit on
Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:
By using sysfs only if the Kernel version used is v4.9 or older:
echo
'vlan 100 proto 802.1ad'
> /sys/class
/net/ens2/vf0/vlan_infoBy using the ip link command (available only when using the latest Kernel version):
# ip link set dev ens2 vf
0
vlan100
proto802
.1adCheck the configuration using the ip link show command:
# ip link show ens2
2
: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu1500
qdisc mq state UP mode DEFAULT groupdefault
qlen1000
link/ether 7c:fe:90
:19
:9e:21
brd ff:ff:ff:ff:ff:ff vf0
MAC00
:00
:00
:00
:00
:00
, vlan100
, vlan protocol802
.1ad , spoof checking off, link-state auto vf1
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf2
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf3
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf4
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto
Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):
# ip link set dev ens2 vf
0
vlan100
qos3
proto802
.1adCheck the configuration using the ip link show command:
# ip link show ens2
2
: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu1500
qdisc mq state UP mode DEFAULT groupdefault
qlen1000
link/ether 7c:fe:90
:19
:9e:21
brd ff:ff:ff:ff:ff:ff vf0
MAC00
:00
:00
:00
:00
:00
, vlan100
, qos3
, vlan protocol802
.1ad , spoof checking off, link-state auto vf1
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf2
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf3
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state auto vf4
MAC00
:00
:00
:00
:00
:00
, vlan4095
, spoof checking off, link-state autoRestart the driver in the VM attached to that VF.
(VM1)# /etc/init.d/openidb restart
Create a VLAN interface on the VM and add an IP address.
ip link add link ens5 ens5.
40
type vlan protocol802
.1q id40
ip addr add42.134
.135.7
/16
brd42.134
.255.255
dev ens5.40
ip link set dev ens5.40
upTo verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.
For further examples, refer to HowToConfigureQinQEncapsulationperVFinLinux(VST)for ConnectX-3 Pro Community post.
Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:
By using sysfs:
echo
'100:0:802.1ad'
> /sys/class
/net/ens1f0/device/sriov/0
/vlanBy using the ip link command (available only when using the latest Kernel version):
ip link set dev ens1f0 vf
0
vlan100
proto802
.1adCheck the configuration using the ip link show command:
# ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500
qdisc mq state UP mode DEFAULT qlen1000
link/ether ec:0d:9a:44
:37
:84
brd ff:ff:ff:ff:ff:ff vf0
MAC00
:00
:00
:00
:00
:00
, vlan100
, vlan protocol802
.1ad, spoof checking off, link-state auto, trust off vf1
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf2
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf3
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf4
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off
Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):
ip link set dev ens1f0 vf
0
vlan100
qos3
proto802
.1adCheck the configuration using the ip link show command:
# ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu
1500
qdisc mq state UP mode DEFAULT qlen1000
link/ether ec:0d:9a:44
:37
:84
brd ff:ff:ff:ff:ff:ff vf0
MAC00
:00
:00
:00
:00
:00
, vlan100
, qos3
, vlan protocol802
.1ad, spoof checking off, link-state auto, trust off vf1
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf2
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf3
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust off vf4
MAC00
:00
:00
:00
:00
:00
, spoof checking off, link-state auto, trust offCreate a VLAN interface on the VM and add an IP address.
ip link add link ens5 ens5.
40
type vlan protocol802
.1q id40
ip addr add42.134
.135.7
/16
brd42.134
.255.255
dev ens5.40
ip link set dev ens5.40
upTo verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.