Q-in-Q Encapsulation per VF in Linux (VST)

This section describes the configuration of IEEE 802.1ad QinQ VLAN tag (S-VLAN) to the hypervisor per Virtual Function (VF). The Virtual Machine (VM) attached to the VF (via SR- IOV) can send traffic with or without C-VLAN. Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to the physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF.

The setup assumes there are two servers equipped with ConnectX-3 Pro/ConnectX-5 adapter cards.

• Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag

• Firmware version 2.36.5150 or higher must be installed for ConnectX-3 Pro HCAs

• Firmware version 16.21.0458 or higher must be installed for ConnectX-5 HCAs

• The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.

  • In order to configure SR-IOV in Ethernet mode for ConnectX-3 Pro adapter cards, please refer to " ConfiguringSR-IOVforConnectX-3/ConnectX-3Pro" section.

  • In order to configure SR-IOV in Ethernet mode for ConnectX-5 adapter cards, please refer to " ConfiguringSR-IOVforConnectX-4/ConnectX-5(Ethernet) "section. In the following configuration example, the VM is attached to VF0.

• Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports.

1. Enable QinQ support in the hardware. Set the phv-bit flag using ethtool (on the hypervisor).

   Copy

   Copied!

               
   
               
   ethtool --set-priv-flags ens2 phv-bit on
           

2. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

   1. By using sysfs only if the Kernel version used is v4.9 or older:

      Copy

      Copied!

                  
      
                  
      echo 'vlan 100 proto 802.1ad' > /sys/class/net/ens2/vf0/vlan_info
              

   2. By using the ip link command (available only when using the latest Kernel version):

      Copy

      Copied!

                  
      
                  
       # ip link set dev ens2 vf 0 vlan 100 proto 802.1ad
              

      Check the configuration using the ip link show command:

      Copy

      Copied!

                  
      
                  
      # ip link show ens2
      2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group    default qlen 1000
         link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff
         vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad , spoof checking off,    link-state auto
         vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
         vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
              

3. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):

   Copy

   Copied!

               
   
               
   # ip link set dev ens2 vf 0 vlan 100 qos 3 proto 802.1ad
           

   Check the configuration using the ip link show command:

   Copy

   Copied!

               
   
               
   # ip link show ens2
   2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group    default qlen 1000
      link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad , spoof checking    off, link-state auto
      vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
      vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
           

4. Restart the driver in the VM attached to that VF.

   Copy

   Copied!

               
   
               
   (VM1)# /etc/init.d/openidb restart
           

5. Create a VLAN interface on the VM and add an IP address.

   Copy

   Copied!

               
   
               
   ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40
   ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40
   ip link set dev ens5.40 up
           

6. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.

For further examples, refer to HowToConfigureQinQEncapsulationperVFinLinux(VST)for ConnectX-3 Pro Community post.

1. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

   1. By using sysfs:

      Copy

      Copied!

                  
      
                  
      echo '100:0:802.1ad' > /sys/class/net/ens1f0/device/sriov/0/vlan
              

   2. By using the ip link command (available only when using the latest Kernel version):

      Copy

      Copied!

                  
      
                  
      ip link set dev ens1f0 vf 0 vlan 100 proto 802.1ad
              

      Check the configuration using the ip link show command:

      Copy

      Copied!

                  
      
                  
      # ip link show ens1f0
       ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
          link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
          vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
          vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
          vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off 
              

2. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):

   Copy

   Copied!

               
   
               
   ip link set dev ens1f0 vf 0 vlan 100 qos 3 proto 802.1ad
           

   Check the configuration using the ip link show command:

   Copy

   Copied!

               
   
               
   # ip link show ens1f0
   ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
       link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff
       vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off
       vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
       vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off
           

3. Create a VLAN interface on the VM and add an IP address.

   Copy

   Copied!

               
   
               
   ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40
   ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40
   ip link set dev ens5.40 up
           

4. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.