Configuring RoCE

Configuring SwitchX® Based Switch System

To enable RoCE, the SwitchX should be configured as follows:

• Ports facing the host should be configured as access ports, and either use global pause or Port Control Protocol (PCP) for priority flow control
• Ports facing the network should be configured as trunk ports, and use Port Control Protocol (PCP) for priority flow control

For further information on how to configure SwitchX, please refer to the SwitchX User Manual.

Configuring Arista Switch

1. Set the ports that face the hosts as trunk.

   (config)# interface et10
   (config-if-Et10)# switchport mode trunk

2. Set VID allowed on trunk port to match the host VID.

   (config-if-Et10)# switchport trunk allowed vlan 100

3. Set the ports that face the network as trunk.

   (config)# interface et20
   (config-if-Et20)# switchport mode trunk

4. Assign the relevant ports to LAG.

   (config)# interface et10
   (config-if-Et10)# dcbx mode ieee
   (config-if-Et10)# speed forced 40gfull
   (config-if-Et10)# channel-group 11 mode active

5. Enable PFC on ports that face the network.

   (config)# interface et20
   (config-if-Et20)# load-interval 5
   (config-if-Et20)# speed forced 40gfull
   (config-if-Et20)# switchport trunk native vlan tag
   (config-if-Et20)# switchport trunk allowed vlan 11
   (config-if-Et20)# switchport mode trunk
   (config-if-Et20)# dcbx mode ieee
   (config-if-Et20)# priority-flow-control mode on
   (config-if-Et20)# priority-flow-control priority 3 no-drop

Using Global Pause (Flow Control)

To enable Global Pause on ports that face the hosts, perform the following:

(config)# interface et10
(config-if-Et10)# flowcontrol receive on
(config-if-Et10)# flowcontrol send on

Using Priority Flow Control (PFC)

To enable PFC on ports that face the hosts, perform the following:

(config)# interface et10
(config-if-Et10)# dcbx mode ieee
(config-if-Et10)# priority-flow-control mode on
(config-if-Et10)# priority-flow-control priority 3 no-drop

Configuring Router (PFC only)

The router uses L3's DSCP value to mark the egress traffic of L2 PCP. The required mapping, maps the three most significant bits of the DSCP into the PCP. This is the default behavior, and no additional configuration is required.

Copying Port Control Protocol (PCP) between Subnets

The captured PCP option from the Ethernet header of the incoming packet can be used to set the PCP bits on the outgoing Ethernet header.

Configuring the RoCE Mode

Configuring the RoCE mode requires the following:

• RoCE mode is configured per-driver and is enforced on all the devices in the system.

  The supported RoCE modes depend on the firmware installed. If the firmware does not support the needed mode, the fallback mode would be the maximum supported RoCE mode of the installed NIC.

• RoCE mode can be enabled and disabled via PowerShell.

To enable RoCEv1 using the PowerShell:
Open the PowerShell and run:

PS $ Set-MlnxDriverCoreSetting –RoceMode 1

To enable RoCEv2 using the PowerShell:
Open the PowerShell and run:

PS $ Set-MlnxDriverCoreSetting –RoceMode 2

To disable any version of RoCE using the PowerShell:
Open the PowerShell and run:

PS $ Set-MlnxDriverCoreSetting –RoceMode 0

To check the current version of RoCE using the PowerShell:
Open the PowerShell and run:

PS $ Get-MlnxDriverCoreSetting

Example output:

Caption         : DriverCoreSettingData 'mlx4_bus'
Description     : Mellanox Driver Option Settings
.
.
.
RoceMode        : 0

How to Configure Storage Space Direct over RDMA

Storage Spaces Direct uses industry-standard servers with local-attached drives to create highly available, highly scalable software-defined storage at a fraction of the cost of traditional SAN or NAS arrays.

To configure storage space direct over RDMA:

1. Deploy storage space direct.  For further information on how to do so, see: 
   https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-storage-spaces-direct.
2. Configure the host environment.
   

   1. Install DCB module, enable DCB in willing mode and remove all previous QoS configuration.

      Install-WindowsFeature "data-center-bridging"
      Remove-NetQosTrafficClass
      Remove-NetQosPolicy -Confirm:$False
      Set-NetAdapterQos -Enabled 1 *
      Set-NetQosDcbxSetting -Willing 0 -Confirm:$false

   2. Set the QoS rules, mapping SMB to priority 3 (RoCE).

      New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action3 -PolicyStore activestore -Confirm:$false
      New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action3 -Confirm:$false
      New-NetQosPolicy "DEFAULT" -PriorityValue8021Action 3 -PolicyStore activestore-Confirm:$false -Default
      New-NetQosPolicy "DEFAULT" -PriorityValue8021Action 3 -Confirm:$false -Default
      New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action0 -DSCPAction 0 -PolicyStore activestore -Confirm:$false
      New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action0 -DSCPAction 0 -Confirm:$false
      New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action0 -PolicyStore activestore -Confirm:$false
      New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action0 -Confirm:$false

   3. Enable PFC on priority 3.

      Disable-NetQosFlowControl 0,1,2,3,4,5,6,7

3. Configure RoCE v2 for ConnectX-3 Pro using Mellanox SwitchX switches is needed. 
   https://enterprise-support.nvidia.com/DOC-1444

For additional information, see: https://enterprise-support.nvidia.com/docs/DOC-2283

RoCE Default Configuration

Starting from v5.20, the default RoCE mode will be RoCE v2.
The RoCE mode will be set to RoCE v2 only upon the first installation of the package.
In any other case, the RoCE mode will not be changed.