NVIDIA MLNX_OFED Documentation v4.9-7.1.0.0 LTS
Linux Kernel Upstream Release Notes v5.17

Q-in-Q Encapsulation per VF in Linux (VST)

Warning

This feature is supported on ConnectX-3 Pro and ConnectX-5 adapter cards only.

Warning

ConnectX-4 and ConnectX-4 Lx adapter cards support 802.1Q double-tagging (C-tag stack- ing on C-tag), refer to "802.1Q Double-Tagging" section.

This section describes the configuration of IEEE 802.1ad QinQ VLAN tag (S-VLAN) to the hypervisor per Virtual Function (VF). The Virtual Machine (VM) attached to the VF (via SR- IOV) can send traffic with or without C-VLAN. Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to the physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF.

The setup assumes there are two servers equipped with ConnectX-3 Pro/ConnectX-5 adapter cards.

worddav336f9b6791fd85e08c8e6897697cd75b.png

  • Kernel must be of v3.10 or higher, or custom/inbox kernel must support vlan-stag

  • Firmware version 2.36.5150 or higher must be installed for ConnectX-3 Pro HCAs

  • Firmware version 16.21.0458 or higher must be installed for ConnectX-5 HCAs

  • The server should be enabled in SR-IOV and the VF should be attached to a VM on the hypervisor.

  • Network Considerations - the network switches may require increasing the MTU (to support 1522 MTU size) on the relevant switch ports.

  1. Enable QinQ support in the hardware. Set the phv-bit flag using ethtool (on the hypervisor).

    Copy
    Copied!
                

    ethtool --set-priv-flags ens2 phv-bit on

  2. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

    1. By using sysfs only if the Kernel version used is v4.9 or older:

      Copy
      Copied!
                  

      echo 'vlan 100 proto 802.1ad' > /sys/class/net/ens2/vf0/vlan_info

    2. By using the ip link command (available only when using the latest Kernel version):

      Copy
      Copied!
                  

      # ip link set dev ens2 vf 0 vlan 100 proto 802.1ad

      Check the configuration using the ip link show command:

      Copy
      Copied!
                  

      # ip link show ens2 2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad , spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

  3. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):

    Copy
    Copied!
                

    # ip link set dev ens2 vf 0 vlan 100 qos 3 proto 802.1ad

    Check the configuration using the ip link show command:

    Copy
    Copied!
                

    # ip link show ens2 2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad , spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

  4. Restart the driver in the VM attached to that VF.

    Copy
    Copied!
                

    (VM1)# /etc/init.d/openidb restart

  5. Create a VLAN interface on the VM and add an IP address.

    Copy
    Copied!
                

    ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40 ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40 ip link set dev ens5.40 up

  6. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.

For further examples, refer to HowTo Configure QinQ Encapsulation per VF in Linux (VST) for ConnectX-3 Pro Community post.

  1. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port per VF. There are two ways to add the S-VLAN:

    1. By using sysfs:

      Copy
      Copied!
                  

      echo '100:0:802.1ad' > /sys/class/net/ens1f0/device/sriov/0/vlan

    2. By using the ip link command (available only when using the latest Kernel version):

      Copy
      Copied!
                  

      ip link set dev ens1f0 vf 0 vlan 100 proto 802.1ad

      Check the configuration using the ip link show command:

      Copy
      Copied!
                  

      # ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off

  2. Optional: Add S-VLAN priority. Use the qos parameter in the ip link command (or sysfs):

    Copy
    Copied!
                

    ip link set dev ens1f0 vf 0 vlan 100 qos 3 proto 802.1ad

    Check the configuration using the ip link show command:

    Copy
    Copied!
                

    # ip link show ens1f0 ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether ec:0d:9a:44:37:84 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad, spoof checking off, link-state auto, trust off vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off

  3. Create a VLAN interface on the VM and add an IP address.

    Copy
    Copied!
                

    ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40 ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40 ip link set dev ens5.40 up

  4. To verify the setup, run ping between the two VMs and open Wireshark or tcpdump to capture the packet.

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.