Q-in-Q Encapsulation per VF in Linux

This section explains how to configure IEEE 802.1ad Q-in-Q VLAN tagging (S-VLAN) for a hypervisor on a per-VF basis. A VM connected to the VF via SR-IOV can transmit traffic with or without a C-VLAN tag.

When a VF is configured for VST Q-in-Q encapsulation, the adapter hardware automatically inserts an S-VLAN tag into outgoing packets from the VF to the physical port. For incoming packets, the hardware removes the S-VLAN tag before delivering the traffic to the VF.

The setup assumes there are two servers equipped with ConnectX-5/ConnectX-6 adapter cards.

• Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag

• Firmware version 16/20.21.0458 or higher must be installed for ConnectX-5/ConnectX-6 HCAs

• The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.

• Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports.

1. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

   1. By using sysfs:

      Copy

      Copied!

                  
      
                  
      echo '100:0:802.1ad' > /sys/class/net/ens1f0/device/sriov/0/vlan
              

   2. By using the ip link command (available only when using the latest Kernel version):

      Copy

      Copied!

                  
      
                  
      ip link set dev ens1f0 vf 0 vlan 100 proto 802.1ad
              

      Check the configuration using the ip link show command:

      Copy

      Copied!

                  
      
                  
      # ip link show ens1f0
       ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
          link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
          vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
          vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off 
              

2. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):

   Copy

   Copied!

               
   
               
   ip link set dev ens1f0 vf 0 vlan 100 qos 3 proto 802.1ad
           

   Check the configuration using the ip link show command:

   Copy

   Copied!

               
   
               
   # ip link show ens1f0
   ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
       link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
       vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
       vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
           

3. Create a VLAN interface on the VM and add an IP address.

   Copy

   Copied!

               
   
               
   ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40
   ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40
   ip link set dev ens5.40 up
           

4. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.