This section describes the configuration of IEEE 802.1ad QinQ VLAN tag (S-VLAN) to the hypervisor per Virtual Function (VF). The Virtual Machine (VM) attached to the VF (via SR- IOV) can send traffic with or without C-VLAN. Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to the physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF. 

Setup

The setup assumes there are two servers equipped with ConnectX-3 Pro/ConnectX-5 adapter cards.

Prerequisites

• Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag
• Firmware version 2.36.5150 or higher must be installed for ConnectX-3 Pro HCAs
• Firmware version 16.21.0458 or higher must be installed for ConnectX-5 HCAs
• The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.
  • In order to configure SR-IOV in Ethernet mode for ConnectX-3 Pro adapter cards, please refer to "Configuring SR-IOV for ConnectX-3/ConnectX-3 Pro" section.
  • In order to configure SR-IOV in Ethernet mode for ConnectX-5 adapter cards, please refer to "Configuring SR-IOV for ConnectX-4/ConnectX-5 (Ethernet)" section. In the following configuration example, the VM is attached to VF0.
• Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports. 

Configuring Q-in-Q Encapsulation per Virtual Function for ConnectX-3 Pro

1. Enable QinQ support in the hardware. Set the phv-bit flag using ethtool (on the hypervisor). 

   ethtool --set-priv-flags ens2 phv-bit on

2. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN: 

   1. By using sysfs only if the Kernel version used is v4.9 or older: 

      echo 'vlan 100 proto 802.1ad' > /sys/class/net/ens2/vf0/vlan_info

   2. By using the ip link command (available only when using the latest Kernel version): 

       # ip link set dev ens2 vf 0 vlan 100 proto 802.1ad

      Check the configuration using the ip link show command: 

      # ip link show ens2
      2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group    default qlen 1000
         link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff
         vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad , spoof checking off,    link-state auto
         vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

3. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs): 

   # ip link set dev ens2 vf 0 vlan 100 qos 3 proto 802.1ad

   Check the configuration using the ip link show command:

   # ip link show ens2
   2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group    default qlen 1000
      link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad , spoof checking    off, link-state auto
      vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

4. Restart the driver in the VM attached to that VF. 

   (VM1)# /etc/init.d/openidb restart

5. Create a VLAN interface on the VM and add an IP address. 

   ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40
   ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40
   ip link set dev ens5.40 up

6. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.

For further examples, refer to HowTo Configure QinQ Encapsulation per VF in Linux (VST) for ConnectX-3 Pro Community post.

Configuring Q-in-Q Encapsulation per Virtual Function for ConnectX-5

1. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

   1. By using sysfs: 

      echo '100:0:802.1ad' > /sys/class/net/ens1f0/device/sriov/0/vlan

   2. By using the ip link command (available only when using the latest Kernel version): 

      ip link set dev ens1f0 vf 0 vlan 100 proto 802.1ad

      Check the configuration using the ip link show command: 

      # ip link show ens1f0
       ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
          link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
          vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
          vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off 

2. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs): 

   ip link set dev ens1f0 vf 0 vlan 100 qos 3 proto 802.1ad

   Check the configuration using the ip link show command: 

   # ip link show ens1f0
   ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
       link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
       vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
       vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off

3. Create a VLAN interface on the VM and add an IP address. 

   ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40
   ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40
   ip link set dev ens5.40 up

4. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.