Created on Aug 8, 2019

Introduction

This document provides the detailed steps to deploy Windows Server 2016 with SET technology over RoCEv2 lossless fabric.

Switch Embedded Teaming (SET) is an alternative NIC Teaming solution that you can use in environments that include Hyper-V and the Software Defined Networking (SDN) stack in Windows Server 2016. SET integrates some NIC Teaming functionality into the Hyper-V Virtual Switch. SET allows you to group between one and eight physical Ethernet network adapters into one or more software-based virtual network adapters. These virtual network adapters provide fast performance and fault tolerance in the event of a network adapter failure.

Table 1: Abbreviation

References

HowTo Configure Mellanox Spectrum Switch for Lossless RoCE

How To Configure MLAG on Mellanox Switches

HowTo Configure MAGP on Mellanox Switches

Solution Overview

Solution Logical Design

The following illustration shows an example configuration.

Deployment Guide

Network / Fabric Deployment and Configuration

Please use this document in order to configure LossLess RoCE for Mellanox Spectrum switches.

Please use this document in order to configure IPL for Mellanox switches.

You must configure MAGP for each VLAN ID which used in the Host vNIC adapters. Please use this document in order to configure MAGP on Mellanox switches.

Host Deployment and Configuration

Physical Host QoS configuration

1. Turn on Data Center Bridging.

Install-WindowsFeature Data-Center-Bridging

Success Restart Needed Exit Code      Feature Result                              

------- -------------- ---------      --------------                              

True    No             Success        {Data Center Bridging}  

2. Set the policies for SMB-Direct

New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

Name           : SMB

Owner          : Group Policy (Machine)

NetworkProfile : All

Precedence     : 127

JobObject      :

NetDirectPort  : 445

PriorityValue  : 3

3. Set policies for other traffic on the interface

New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

Name           : DEFAULT

Owner          : Group Policy (Machine)

NetworkProfile : All

Precedence     : 127

Template       : Default

JobObject      :

PriorityValue  : 0

4. Turn on Flow Control for SMB

          Enable-NetQosFlowControl -priority 3

          Get-NetQosFlowControl

Priority   Enabled    PolicySet        IfIndex IfAlias                 

--------   -------    ---------        ------- -------                 

0          False      Global                                           

1          False      Global                                           

2          False      Global                                           

3          True       Global                                           

4          False      Global                                           

5          False      Global                                           

6          False      Global                                           

7          False      Global       

5. Disable traffic FlowControl for classes other than 3

Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7

Get-NetQosFlowControl

Priority   Enabled    PolicySet        IfIndex IfAlias                 

--------   -------    ---------        ------- -------                 

0          False      Global                                           

1          False      Global                                           

2          False      Global                                           

3          True       Global                                           

4          False      Global                                           

5          False      Global                                           

6          False      Global                                           

7          False      Global

Create a vSwitch in SET mode and hNICs

1. Create a vSwitch in Switch Embedded Teaming mode

New-VMSwitch –Name "VSWSET" –NetAdapterName "pNIC1","pNIC2" -EnableEmbeddedTeaming $true -AllowManagementOS $true

Name    SwitchType NetAdapterInterfaceDescription

----    ---------- ------------------------------

VSWSET  External   Teamed-Interface   

2. List the Physical adapters in logical switch with SET

Get-VMSwitchTeam -Name VSWSET | FL

Name : VSWSET

Id : bff9d6c4-4259-4db2-8432-4870117f0da3

NetAdapterInterfaceDescription : {Mellanox ConnectX-4 VPI Adapter, Mellanox ConnectX-4 VPI Adapter #2}

TeamingMode : SwitchIndependent

LoadBalancingAlgorithm : Dynamic

3. To prevent auto-tagging the egress traffic with incorrect VLAN ID from both physical NIC please remove the ACCESS VLAN Setting and filtering ingress traffic which doesn’t match the ACCESS VLAN ID

Set-NetAdapterAdvancedProperty -Name "pNIC1" -RegistryKeyword VlanID -RegistryValue "0"

Set-NetAdapterAdvancedProperty -Name "pNIC2" -RegistryKeyword VlanID -RegistryValue "0"

4. Creating the Management NIC in order to use separate Host vNICs instances for RDMA and set VLANID 101

Rename-VMNetworkAdapter -ManagementOS -Name “VSWSET” -NewName “MGMT”

Set-VMNetworkAdapterVlan -VMNetworkAdapterName "MGMT" -VlanId "101" -Access -ManagementOS

5. Set Priority tagging on the Management Host vNIC

Set-VMNetworkAdapter -ManagementOS -Name "MGMT" -IeeePriorityTag on

6. Creating two Host vNIC for RDMA and set VLANID 102

          Add-VMNetworkAdapter –SwitchName "VSWSET" –Name SMB1 –ManagementOS

          Add-VMNetworkAdapter –SwitchName "VSWSET" –Name SMB2 –ManagementOS

          Get-VMNetworkAdapter –ManagementOS

Name InterfaceDescription                    ifIndex Status  MacAddress        LinkSpeed

---- -------------------- ------- ------  ----------        ---------

vEthernet (SMB1) Hyper-V Virtual Ethernet Adapter #1           6 Up      00-1D-D8-B7-1C-11  200 Gbps

vEthernet (SMB2) Hyper-V Virtual Ethernet Adapter #2          18 Up      00-1D-D8-B7-1C-0E  200 Gbps

vEthernet (MGMT) Hyper-V Virtual Ethernet Adapter             16 Up      00-1D-D8-B7-1C-01  200 Gbps

Set-VMNetworkAdapterVlan -VMNetworkAdapterName "SMB1" -VlanId "102" -Access -ManagementOS

Set-VMNetworkAdapterVlan -VMNetworkAdapterName "SMB2" -VlanId "102" -Access –ManagementOS

Get-VMNetworkAdapterVlan -ManagementOS

VMName VMNetworkAdapterName Mode     VlanList

------ -------------------- ----     --------

       SMB1                 Access   102

       MGMT                 Access   101

       SMB2                 Access   102

7. Because vNICs dedicated to storage are bound to a SET we need create an affinity between a vNIC and a pNIC ensures that the traffic from a given vNIC on the host (storage vNIC) uses a particular pNIC to send traffic so that it passes through the shorter path

Set-VMNetworkAdapterTeamMapping -ManagementOS -SwitchName VSWSET -VMNetworkAdapterName "SMB1" -PhysicalNetAdapterName "pNIC1"

Set-VMNetworkAdapterTeamMapping -ManagementOS -SwitchName VSWSET -VMNetworkAdapterName "SMB2" -PhysicalNetAdapterName "pNIC2"

8. Set Priority tagging (PCP mode) on the RDMA Host vNIC and enable RDMA on this vNICs

Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on

Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on

Enable-NetAdapterRdma -Name "vEthernet (SMB1)"

Enable-NetAdapterRdma -Name "vEthernet (SMB2)"

Performance Testing

Please assign an IP address to each RDMA enabled Host vNICs and provides testing with TEST-RDMA.PS1 PowerShell script. Please see below script execution examples:

1. With RDMA enabled (parameter -IsRoCE $true)

C:\TESTS\Test-RDMA.PS1 -IfIndex 6 -IsRoCE $true -RemoteIpAddress 192.168.2.2 -PathToDiskspd C:\TESTS\Diskspd-v2.0.17\amd64fre\

2. With RDMA disabled (parameter -IsRoCE $False)

C:\TESTS\Test-RDMA.PS1 -IfIndex 6 -IsRoCE $False -RemoteIpAddress 192.168.2.2 -PathToDiskspd C:\TESTS\Diskspd-v2.0.17\amd64fre\

Done.