Configuring RoCE

To enable RoCE, the SwitchX should be configured as follows:

  • Ports facing the host should be configured as access ports, and either use global pause or Port Control Protocol (PCP) for priority flow control

  • Ports facing the network should be configured as trunk ports, and use Port Control Protocol (PCP) for priority flow control

For further information on how to configure SwitchX, please refer to the SwitchX User Manual.

  1. Set the ports that face the hosts as trunk.

    Copy
    Copied!
                

    (config)# interface et10 (config-if-Et10)# switchport mode trunk

  2. Set VID allowed on trunk port to match the host VID.

    Copy
    Copied!
                

    (config-if-Et10)# switchport trunk allowed vlan 100

  3. Set the ports that face the network as trunk.

    Copy
    Copied!
                

    (config)# interface et20 (config-if-Et20)# switchport mode trunk

  4. Assign the relevant ports to LAG.

    Copy
    Copied!
                

    (config)# interface et10 (config-if-Et10)# dcbx mode ieee (config-if-Et10)# speed forced 40gfull (config-if-Et10)# channel-group 11 mode active

  5. Enable PFC on ports that face the network.

    Copy
    Copied!
                

    (config)# interface et20 (config-if-Et20)# load-interval 5 (config-if-Et20)# speed forced 40gfull (config-if-Et20)# switchport trunk native vlan tag (config-if-Et20)# switchport trunk allowed vlan 11 (config-if-Et20)# switchport mode trunk (config-if-Et20)# dcbx mode ieee (config-if-Et20)# priority-flow-control mode on (config-if-Et20)# priority-flow-control priority 3 no-drop

Using Global Pause (Flow Control)

To enable Global Pause on ports that face the hosts, perform the following:

Copy
Copied!
            

(config)# interface et10 (config-if-Et10)# flowcontrol receive on (config-if-Et10)# flowcontrol send on

Using Priority Flow Control (PFC)

To enable PFC on ports that face the hosts, perform the following:

Copy
Copied!
            

(config)# interface et10 (config-if-Et10)# dcbx mode ieee (config-if-Et10)# priority-flow-control mode on (config-if-Et10)# priority-flow-control priority 3 no-drop

The router uses L3's DSCP value to mark the egress traffic of L2 PCP. The required mapping, maps the three most significant bits of the DSCP into the PCP. This is the default behavior, and no additional configuration is required.

Copying Port Control Protocol (PCP) between Subnets

The captured PCP option from the Ethernet header of the incoming packet can be used to set the PCP bits on the outgoing Ethernet header.

Configuring the RoCE mode requires the following:

  • RoCE mode is configured per-driver and is enforced on all the devices in the system.

    Warning

    The supported RoCE modes depend on the firmware installed. If the firmware does not support the needed mode, the fallback mode would be the maximum supported RoCE mode of the installed NIC.

  • RoCE mode can be enabled and disabled via PowerShell.

To enable RoCEv1 using the PowerShell:
Open the PowerShell and run:

Copy
Copied!
            

PS $ Set-MlnxDriverCoreSetting –RoceMode 1

To enable RoCEv2 using the PowerShell:
Open the PowerShell and run:

Copy
Copied!
            

PS $ Set-MlnxDriverCoreSetting –RoceMode 2

To disable any version of RoCE using the PowerShell:
Open the PowerShell and run:

Copy
Copied!
            

PS $ Set-MlnxDriverCoreSetting –RoceMode 0

To check the current version of RoCE using the PowerShell:
Open the PowerShell and run:

Copy
Copied!
            

PS $ Get-MlnxDriverCoreSetting

Example output:

Copy
Copied!
            

Caption : DriverCoreSettingData 'mlx4_bus' Description : Mellanox Driver Option Settings . . . RoceMode : 0

Storage Spaces Direct uses industry-standard servers with local-attached drives to create highly available, highly scalable software-defined storage at a fraction of the cost of traditional SAN or NAS arrays.

To configure storage space direct over RDMA:

  1. Deploy storage space direct. For further information on how to do so, see:
    https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-storage-spaces-direct.

  2. Configure the host environment.

    1. Install DCB module, enable DCB in willing mode and remove all previous QoS configuration.

      Copy
      Copied!
                  

      Install-WindowsFeature "data-center-bridging" Remove-NetQosTrafficClass Remove-NetQosPolicy -Confirm:$False Set-NetAdapterQos -Enabled 1 * Set-NetQosDcbxSetting -Willing 0 -Confirm:$false

    2. Set the QoS rules, mapping SMB to priority 3 (RoCE).

      Copy
      Copied!
                  

      New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action3 -PolicyStore activestore -Confirm:$false New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action3 -Confirm:$false New-NetQosPolicy "DEFAULT" -PriorityValue8021Action 3 -PolicyStore activestore-Confirm:$false -Default New-NetQosPolicy "DEFAULT" -PriorityValue8021Action 3 -Confirm:$false -Default New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action0 -DSCPAction 0 -PolicyStore activestore -Confirm:$false New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action0 -DSCPAction 0 -Confirm:$false New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action0 -PolicyStore activestore -Confirm:$false New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action0 -Confirm:$false

    3. Enable PFC on priority 3.

      Copy
      Copied!
                  

      Disable-NetQosFlowControl 0,1,2,3,4,5,6,7

  3. Configure RoCE v2 for ConnectX-3 Pro using Mellanox SwitchX switches is needed.
    https://enterprise-support.nvidia.com/DOC-1444

For additional information, see: https://enterprise-support.nvidia.com/docs/DOC-2283

Starting from v5.20, the default RoCE mode will be RoCE v2.
The RoCE mode will be set to RoCE v2 only upon the first installation of the package.
In any other case, the RoCE mode will not be changed.

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.