Created on Jan 12,2022 by Boris Kovalev

Scope

This document describes the configuration process of VMware vSAN Express Storage Architecture (ESA) over NVIDIA® RDMA over Converged Ethernet (RoCE) in VMware vSphere 8.0 and NVIDIA® end-to-end 100 Gb/s Ethernet solution.

Abbreviations and Acronyms

Introduction

Hybrid cloud has become the dominant architecture for enterprises seeking to extend their compute capabilities by using public clouds while maintaining on-premises clusters that are fully interoperable with their cloud service providers.

To meet demands, provide services and allocate resources efficiently, enterprise IT teams have deployed hyperconverged architectures that use the same servers for compute and storage. These architectures include three core technologies: software-defined-compute (or server virtualization), software-defined-networking and software-defined storage (SDS). Taken together, these three enable a software-defined data center. Also these hyperconverged architectures widely adopt high-performance Ethernet for server-to-server and server-to-storage communication.

vSAN is VMware’s enterprise storage solution for SDS that supports hyperconverged infrastructure systems and is fully integrated with VMware vSphere as a distributed layer of software within the ESXi hypervisor. vSAN eliminates the need for external shared storage and simplifies storage configuration through storage policy-based management. Through virtual machine storage policies, users can define storage requirements and capabilities.

vSAN aggregates local, direct-attached storage devices to create and share a single storage pool across all hosts in the hyperconverged cluster, utilizing faster flash SSD for cache and inexpensive HDD to maximize capacity.

RDMA (Remote Direct Memory Access) is an innovative networking technology that boosts data communication performance and efficiency. RDMA makes data transfers more efficient and enables fast data move­ment between servers and storage without using the OS or burdening the server’s CPU. Throughput is increased, latency reduced and the CPU is freed to run applications.

RDMA over Converged Ethernet (RoCE) is a network protocol that allows RDMA over an Ethernet network. There are two RoCE versions, RoCE v1 and RoCE v2. RoCE v1 is an Ethernet link layer protocol and hence allows communication between any two hosts in the same Ethernet broadcast domain. RoCE v2 is an internet layer protocol which means that RoCE v2 packets can be routed. Although the RoCE protocol benefits from the characteristics of a converged Ethernet network, the protocol can also be used on a traditional or non-converged Ethernet network.

RDMA technology is already widely used for efficient data transfer in large cloud deployments, such as High Performance Computing (HPC), Artificial Intelligence (AI) and Machine Learning (ML), Non-Volatile Memory express over Fabrics (NVME-oF) and Internet Small Computer System Interface over RDMA (iSER)-based storage, NFSoRDMA, mission-critical SQL databases such as Oracle’s RAC (Exadata), IBM DB2 pureScale, Microsoft SQL solutions, Teradata and more.

For the last several years, VMware has been adding RDMA support to ESXi, including PVRDMA (paravirtual RDMA) to accelerate data transfers between virtual servers and iSER and NVMe-oF for remote storage acceleration.

VMware’s vSAN over RDMA is fully qualified and available as of the ESXi 7.0 U3d release, making it ready for deployments.

The announcement of VMware vSAN Express Storage Architecture™ or ESA for short, represents a massive step forward in the capabilities of the solution.This is an optional, alternative storage architecture (OSA) to the vSAN original storage architecture also found in vSAN 8.

The best way to think of the vSAN Express Storage Architecture is as a new way to process and store data.  It is an optional, alternative architecture in vSAN that is designed to achieve all-new levels of efficiency, scalability, and performance.  The ESA is optimized to exploit the full potential of the very latest in hardware, and unlocks new capabilities.

This document provides instructions on how to configure vSAN ESA over RDMA Datastores located on local NVMe disks in VMware vSphere 8.0 over NVIDIA end-to-end 100 Gb/s Ethernet solution.

HCI Bench v2.8.0 HCIBench will be used for benchmarks to show performance improvements between vSAN over RDMA and vSAN over TCP protocols by using the same hardware.

References

• What is RDMA?
• RDMA over Converged Ethernet (RoCE)
• ConnectX® Ethernet Driver for VMware® ESXi Server

• NVIDIA Cumulus

• VMware vSAN 7 Design Guide
• vSAN Network Design
• HCIBench | VMware Flings

Solution Architecture

Key Components and Technologies

• vSAN over RDMA Support for VMware
  vSAN over RDMA provides increased performance for vSAN. 
  Each vSAN host must have a vSAN certified RDMA-capable NIC, as listed in the vSAN section of the VMware Compatibility Guide. 
  All hosts in the cluster must support RDMA. If any host loses RDMA support, the entire vSAN cluster switches to TCP.
  vSAN over RDMA supports NIC failover, but does not support LACP or IP-hash-based NIC teaming.

• NVIDIA Cumulus Linux 
  NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.

Logical Design

Software Stack Components

This guide assumes that the following software and drivers are installed:

• VMware ESXi 8.0, build 20513097
• VMware vCenter 8.0, build 20519528 
• Distributed Switch 8.0.0
• NVIDIA® ConnectX® Driver for VMware ESXi Server v4.23.0.36
• NVIDIA® ConnectX®-6DX FW version 22.34.1002
• Network Operational System (NOS): NVIDIA Cumulus^™ v5.1
• HCIBench version 2.8.0

Bill of Materials

The following hardware setup is utilized in the vSphere environment that is described in this guide.

Management Cluster:

Workload Cluster:

Deployment and Configuration

Wiring

This document covers a highly available VMware vSphere cluster deployment.

Management Cluster:

Workload Cluster:

Network

Prerequisites

• Switch OS
  NVIDIA Cumulus 5.1
• Network adapter
  2x ConnectX-6 Dx or LX. Use only the same model network adapters from the same vendor on each end of the connection.
• Management Network 
  DHCP and DNS services are required. The components installation and configuration are not covered in this guide.

vSphere Switches Design

Management Cluster Host 

Workload Cluster Host 

Hosts Network Configuration

This table provides details on the ESXi server, switches and storage system, including their names and their network configuration.

The one distributed port group (DPG per cluster - vSAN-VLAN1630-DPG, vSAN-VLAN30-DPG) is required to support Active/Passive vSAN connectivity.

SL MGMT Cluster

SL WL01 Cluster

Network Switch Configuration

Port channel and VLAN configuration

Run the following commands on both Leaf switches in the Management Cluster to configure port channel and VLAN. 

nv set interface bond1 type bond
nv set interface bond1 bond member swp21-22
nv set interface bond1 bridge domain br_default
nv set bridge domain br_default vlan 130
nv set bridge domain br_default vlan 100
nv set interface bond1 bridge domain br_default vlan all 130
nv set interface bond1 bridge domain br_default untagged 100
nv set interface bond1 bridge domain br_default vlan all add 120
nv config apply
nv config save

Run the following commands on both Leaf switches in the Workload Cluster to configure the port channel and VLAN. 

nv set interface bond1 type bond
nv set interface bond1 bond member swp31-32
nv set interface bond1 bridge domain br_default
nv set bridge domain br_default vlan 215
nv set bridge domain br_default vlan 1620
nv set bridge domain br_default vlan 1630
nv set interface bond1 bridge domain br_default untagged 215
nv set interface bond1 bridge domain br_default vlan all 1620
nv set interface bond1 bridge domain br_default vlan all 1630
nv config apply
nv config save

Enable RDMA over Converged Ethernet Lossless (with PFC and ETS)

RoCE transport is utilized to accelerate vSAN networking. To get the highest possible results, the network is configured to be lossless.

Run the following commands on all Leaf switches to configure lossless networks for NVIDIA Cumulus. 

nv set qos roce
nv config apply
nv config save

To Check RoCE configuration, run the following command:

$sudo nv show qos roce
 
                    operational  applied   description
------------------  -----------  --------  ------------------------------------------------------
enable                           on        Turn the feature 'on' or 'off'.  The default is 'off'.
mode                lossless     lossless  Roce Mode
cable-length        100          100       Cable Length(in meters) for Roce Lossless Config
congestion-control
  congestion-mode   ECN                    Congestion config mode
  enabled-tc        0,3                    Congestion config enabled Traffic Class
  max-threshold     1.43 MB                Congestion config max-threshold
  min-threshold     146.48 KB              Congestion config min-threshold
pfc
  pfc-priority      3                      switch-prio on which PFC is enabled
  rx-enabled        enabled                PFC Rx Enabled status
  tx-enabled        enabled                PFC Tx Enabled status
trust
  trust-mode        pcp,dscp               Trust Setting on the port for packet classification
 
 
RoCE PCP/DSCP->SP mapping configurations
===========================================
        pcp  dscp                     switch-prio
    --  ---  -----------------------  -----------
    0   0    0,1,2,3,4,5,6,7          0
    1   1    8,9,10,11,12,13,14,15    1
    2   2    16,17,18,19,20,21,22,23  2
    3   3    24,25,26,27,28,29,30,31  3
    4   4    32,33,34,35,36,37,38,39  4
    5   5    40,41,42,43,44,45,46,47  5
    6   6    48,49,50,51,52,53,54,55  6
    7   7    56,57,58,59,60,61,62,63  7
 
 
RoCE SP->TC mapping and ETS configurations
=============================================
        switch-prio  traffic-class  scheduler-weight
    --  -----------  -------------  ----------------
    0   0            0              DWRR-50%
    1   1            0              DWRR-50%
    2   2            0              DWRR-50%
    3   3            3              DWRR-50%
    4   4            0              DWRR-50%
    5   5            0              DWRR-50%
    6   6            6              strict-priority
    7   7            0              DWRR-50%
 
 
RoCE pool config
===================
        name                   mode     size   switch-priorities  traffic-class
    --  ---------------------  -------  -----  -----------------  -------------
    0   lossy-default-ingress  Dynamic  50.0%  0,1,2,4,5,6,7      -
    1   roce-reserved-ingress  Dynamic  50.0%  3                  -
    2   lossy-default-egress   Dynamic  50.0%  -                  0,6
    3   roce-reserved-egress   Dynamic  inf    -                  3
 
 
Exception List
=================
        description

vSAN Cluster Creation

VMware vSAN Requirements

There are many considerations to be made from a requirements perspective when provisioning a VMware vSAN software-defined storage solution.

Please refer to the official VMware vSAN 8 Design Guide, chapter 5 "Requirements for Enabling vSAN".

vSAN with RDMA Requirements

vSAN version 8.0 and the versions above support RDMA communication. To use it:

• Each vSAN host must have a vSAN certified RDMA-capable NIC, as listed in the vSAN section of the VMware Compatibility Guide
• Only the same model network adapters from the same vendor can be used on each end of the connection
• All hosts in the cluster must support RDMA. If any host loses RDMA support, the entire vSAN cluster switches to TCP

Preparing vSphere Cluster for vSAN

Prerequisites

• Physical server configuration
  All ESXi servers must have the same PCIe placement for the NIC and expose the same interface name.
• vSphere cluster with minimum 3 VMware vSphere ESXi 8.0 hosts
• vCenter 8.0
• Installer privileges: The installation requires administrator privileges on the target machine
• Connection to ESXi host management interface
• High speed network connectivity
• Verify that NTP is configured in your environment and that it works properly.
  

To create a vSAN cluster, create a vSphere host cluster and enable vSAN on the cluster.

To enable the exchange of data in the vSAN cluster, you must provide a VMkernel network adapter for vSAN traffic on each ESXi host.

Firstly, make sure to create a vSphere Distributed Switch (vDS) with a distributed port group on a vSphere cluster with one Active and one Standby uplink.

Creating a Distributed Switch for vSAN Traffic

To create a new vDS:

1. Launch the vSphere Web Client and connect to a vCenter Server instance.
            

2. On the vSphere Web Client home screen, select the vCenter object from the list on the left.
   Hover over the Distributed Switches from the Inventory Lists area, then click on New Distributed Switch (see image below) to launch the New vDS creation wizard:
   
   
   

3. Provide a name for the new distributed switch, and select the location within the vCenter inventory where you would like to store the new vDS (a data center object or a folder).
   Click NEXT.
   
   
   
4. Select the version of the vDS to create. 
   Click NEXT.
   
   
   
5. Specify the Netwrok Offloads compatibility as None, and number of uplink ports as 2. Uncheck the Create a default port group box, and enter a name for that group.
   Click NEXT.
   
   
   
6. Click Finish.
   
   
   
7. Set the MTU for the newly created distributed switch.
   Right-click the new distributed switch in the list of objects, and select Settings → Edit Settings... from the Actions menu.
   
   
8. In the Storage-DSwitch-Edit Settings dialog box, set the MTU to 9000, Discovery protocol to Link Layer Discovery Protocol and Operation to Both.
   Click OK.
   
   

Adding Hosts to vDS

To add an ESXi host to an existing vDS:

1. Launch the vSphere Web Client, and connect to a vCenter Server instance.
   
   
2. Navigate to the list of Hosts in the SL MGMT cluster and select ESXi host.
   
   
3. Select Configure → Networking → Physical adapters.
   
   
4. Check the network ports that you are going to use. In this case, vmnic2 and vmnic3 are used.
            
   
5. Navigate to the list of distributed switches.
   
   

6. Right-click on the new distributed switch in the list of objects, and select Add and Manage Hosts from the Actions menu.
   
   

7. Select the Add hosts button and click NEXT.
   
   From the list of the new hosts, check the boxes with the names of each ESXi host you would like to add to the VDS.
   Click NEXT.
   
   
   
8. In the next Manage physical adapters menu, click on Adapters on all hosts and configure the adapters (in this case - vmnic2 and vmnic3) in an ESXi host as Uplink 1 and Uplink 2 for the VDS.
   
   
   
9. In the next Manage VMkernel adapters and Migrate VM networking menus, click NEXT to continue.
   
   
   
10. Click FINISH.
    
    

11. Repeat the Distributed Switch for vSAN Traffic steps for Workload cluster.

Creating a Distributed Port Group for Storage Traffic

This section lists the steps required to create two distributed port groups with one Active and one Standby uplinks.

1. Add VMkernel Adapters for Distributed Port Groups by right-clicking on Distributed switch, and select Distributed Port Group>New Distributed Port Group.
   
   
   
2. On the New Distributed Port Group dialog box, enter a Name such as <vSAN-VLAN1630-DPG>, and click NEXT.
   
   
   
3. In the VLAN type field, select VLAN, and set the VLAN ID to your VLAN (in this example - 1630). Check the Customize default policies configuration checkbox, and click NEXT.
   
   
   
4. On the Security dialog box, click NEXT.
   
   
   
5. On the Traffic shaping dialog box, click NEXT.
   
   
   

6. NIC Teaming with RDMA

   RDMA for vSAN supports the following teaming policies for virtual switches.
   • Route-based on originating virtual port
   • Route-based on source MAC hash
   • Use explicit failover order
   
   

7. In the Teaming and failover dialog box, select Uplink 1 as active uplink, and set Uplink 2 to be the standby uplink. Click NEXT.

   
   
   
8. In the Monitoring dialog box, set NetFlow to Disabled, and click NEXT.
   
   
   
9. In the Miscellaneous dialog box, set Block All Ports to No, and click NEXT.
   
   
   
10. In the Ready to complete dialog box, review all the changes, and click FINISH.
    

Repeat Creating a Distributed Port Group for vMotion Traffic.

In the VLAN type as VLAN field, set the VLAN ID to your VLAN (in this example - 1620).

In the Teaming and failover dialog box, select Uplink 2 as active uplink and set Uplink 1 to be the standby uplink.

Adding a VMkernel Network for vSAN

To add VMkernel adapters for distributed port groups, follow the steps below.

1. Right click on distributed port group and select Add VMkernel Adapters.
   
   
   
2. Select Attached Hosts, and click NEXT.
   
   
   
3. Select vSAN (in this example) in Available services, and click NEXT.
   
   
   
4. Enter the Network Settings and Gateway details, and click NEXT.
   
   
   
5. Click FINISH.
   
   

vSAN network is enabled for the host.

Repeat the Adding a VMkernel Network steps for the vMotion, and any other required port groups.

Once the ESXi Cluster Networking configuration is complete, it can be verified under the Distributed Switch>Configure>Topology tab.

Manually Enabling vSAN

Use the HTML5-based vSphere Client to configure your vSAN cluster.

To enable and configure vSAN, follow the steps below.

1. Navigate to an existing host cluster. In this example - SL-WL01-DS01 Cluster.
   
   
2. Select Configure → vSAN → Services. Select Standard vSAN cluster, and click Configure.
   
   

3. Select the vSAN ESA, and click Next.
   
   

4. Configure the data management features, including Data-At-Rest encryption, Data-In-Transit encryption and RDMA support. Click NEXT.
   
   
5. Claim disks for the vSAN cluster, and click NEXT.
   Each host requires at least one flash device in the cache tier, and one or more devices in the capacity tier.
   
   
   
6. Create a fault domain, in case you need it, and click NEXT.
   
   
7. Review the configuration, and click FINISH.
   
   
   

Configure a vSAN Datastore Storage Policy

To configure a Storage Policy, follow the steps below.

1. Navigate to your vSAN Datastore and select General under Configure.
2. Click on Edit Default Storage Policy.
   
   
3. Select vSAN ESA Default Plicy - RAID5 and click OK.
   
   

Configure License Settings for a vSAN Cluster

You must assign a license to a vSAN cluster before its evaluation period or its currently assigned license expire.

Prerequisites

• To view and manage the vSAN licenses, you must have the Global.Licenses privilege enabled on the vCenter Server system.

To assign a license to a vSAN cluster, follow the steps below.

1. Navigate to your vSAN cluster.
   
2. Right click on the cluster and select Assign vSAN Cluster License... from the Actions menu.
   
   
   
3. Select an existing license, and click OK.
   
   
   
4. Validate the Assigned License by:
   1. Navigating to your vSAN cluster.
   2. Clicking the Configure tab.
   3. Selecting vSAN Cluster under Licensing.
      
      
      
5. Once you enable vSAN, a single datastore is created.
   You can review the Skyline Health, Physical Disks, Resyncing Objects, Capacity and Performance.
   
   
   In addition, you can run Proactive Tests of the vSAN datastore.
   
   

Done!

Appendix

Test the Environment

Hardware and Software Components

Host under test:

• Server, Dell PowerEdge R750 vSAN Ready Node, with 2 x Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz 16 cores, 384GB of RAM.
• NVIDIA ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AN-CDA), with the driver 4.23.0.36-8vmw.800.1.0.20513097 and FW 22.34.10.02 versions.
• 4xDell Ent NVMe P5600 MU U.2 1.6TB (vSAN ESA Default Plicy - RAID5)
• VMware ESXi™ 8.0 GA

Network:
• NVIDIA Spectrum® SN3700 Open Ethernet Switch
   NVIDIA® Cumulus® Linux v5.1 Network OS
• NVIDIA MCP1650-H001E30 Passive Copper Cable InfiniBand HDR up to 200Gb/s QSFP56 LSZH 1m Black Pulltab 30AWG

Virtual Machine and Benchmark Configuration

We used HCI Bench v2.8.0 FIO benchmark workloads to measure performance with the following parameter configurations:

Performance Results

The HCI Bench used Random Read and Random Write IO patterns with various IO sizes from 4 KB to 1024 KB.

We compare the IOPS, Throughput and CPU usage between vSAN over RDMA and TCP on the cluster.

Storage policy - vSAN ESA Default Plicy - RAID5.

The benchmark runs had the virtual disks placed on:

Essential Storage Architecture:

• vSAN data: 4xDell Ent NVMe P5600 MU U.2 1.6TB

Disk Group 1 for Original Storage Architecture(OSA):

• vSAN cache: Dell Ent NVMe P5600 MU U.2 1.6TB
• vSAN data: 3xDell Ent NVMe P5600 MU U.2 1.6TB

Conclusion

The benchmark results in this performance study show consistent supremacy of vSAN ESA over RDMA compared to either vSAN ESA over TCP or vSAN OSA: 

• vSAN ESA over RDMA achives up to 86% more IOPs than ESA TCP and up to 165% more IOPs than OSA over RDMA.
  
• vSAN ESA over RDMA achives up to 86% higher throughput than ESA TCP and up to 165% throughput more than OSA over RDMA.
  
• vSAN ESA over RDMA achives up to 74% Read and up to 11.6% Write lower latency than ESA TCP and up to 79% Read and up to 11.6% Write lower latency than OSA over RDMA.
  
• vSAN ESA over RDMA achives up to 33.6% lower physical CPU usage than ESA TCP. 
  

Note: In our environment, we use read intensive disks. this can explained the difference between read and write results.

In addition, the benchmark results in this performance study show consistent supremacy of vSAN ESA over RDMA.

As a result, vSAN ESA over RDMA allows running more VMs on the same hardware with more performance and lower latency.

Running vSAN ESA over RDMA, which offloads the CPU from performing the data communication tasks, generates a significant performance boost critical in the new era of accelerated computing associated with massive data transfers.

It is expected that vSAN ESA over RDMA will eventually replace vSAN ESA over TCP and vSAN OSA over RDMA and become the leading transport technology in vSphere-enabled data centers.

Authors