Page History

Versions Compared

Old Version 47

changes.mady.by.user Boris Kovalev

Saved on Aug 18, 2019

compared with

New Version Current

changes.mady.by.user Boris Kovalev

Saved on Nov 19, 2022

Key

This line was added.
This line was removed.
Formatting was changed.

Created on Aug 8, 2019

Introduction

This document describes how to enable PVRDMA in VMware vSphere 6instructs how to enable PVRDMA in VMware vSphere 6.5/6.7 with Mellanox with NVIDIA ConnectX network cardcards.

This guide assumes that the following software and drivers are have been pre-installed:

VMware ESXi 6.7 Update 2, build 13006603
vCenter 6.7 Update 2, build 13007421
Distributed Switch 6.6.0
ConnectX® Ethernet Driver for VMware® ESXi Server 4.17.13.1-1vmw.670.2.48.13006603
CentOS 7.6

References

Components Overview

vSphere Distributed Switch

A vSphere Distributed Switch provides centralized management and monitoring of the networking configuration of all hosts that are associated with the switch. You must set up a distributed switch on a vCenter Server server system, and its settings will be propagated to all hosts that are associated with the switch.

Paravirtual RDMA (PVRDMA)

Direct Memory Access (DMA) is an ability of a device - A device's capability to access the host memory directly, without the intervention of the CPU.

Remote Direct Memory Access (RDMA) is the ability of accessing - The ability to accessing memory (read, write) memory on a remote machine without interrupting the processing of the CPU(s) processes on that the system.

RDMA Advantages:

Zero-copy - Allows applications can to perform data transfers without the involvement of involving the network software stack. Data is sent and received directly to the buffers without being copied between the network layers.
Kernel bypass - Allows applications can to perform data transfers directly from the user-space without the kernel's involvement.
No CPU involvementoffload - Allows applications can to access a remote memory without consuming any CPU time in on the remote server. The remote memory server will be read without any intervention from the remote process (or processor). Moreover, the caches cache of the remote CPU will not be filled with the accessed memory content.

PVRDMA Architecture

Image Removed

Virtual Machine

Expose dual function virtual PCIe device
- A network interface – vmxnet3
- An RDMA provider – PVRDMA

RDMA provider plugs in to the OpenFabrics Enterprise Distribution (OFED) stack
- In kernel and user-space

Full support for the Verbs RDMA API

Live vMotion, Snapshots and HA

PVRDMA backend

- - - - Creates virtual RDMA resources for VM
        Guests operate on these virtual resources

ESXi

Leverage native RDMA and Core drivers

Physical HCA services all VMs

Accelerating VM Data

Image Removed

VM memory address translations registered with HCA – Buffer registration.

Application issues a request (Work Request) to read/write from a particular guest address, size.

PVRDMA backend intercepts these requests issues requests to mapped hardware resources.

HCA performs DMAs to/from application memory without any SW involvement.

- - - - Enables direct zero-copy data transfers in HW

Solution Overview

Equipment

Image Removed

Solution Logical Design

Image Removed

Bill of Materials

Image Removed

Solution Physical Network Wiring

Image Removed

Network Configuration

The below table provides details of ESXi server names and network configuration:

ESXi

Server

Image Added

Accelerating VM Data

Image Added

Solution Overview

Setup

Image Added

Solution Logical Design

Image Added

Bill of Materials

Image Added

Solution Physical Network Wiring

Image Added

Configuration

Network Configuration

The below table provides the ESXi server names and details on their network configuration:

ESXi Server	Server Name	IP and NICs
ESXi Server	Server Name	High-Speed Ethernet Network	Management Network 192.168.1.0
ESXi-01	sl01w01esx21	none	eno0: From DHCP (reserved)
ESXi-02	sl01w01esx22	none	eno0: From DHCP (reserved)
ESXi-03	sl01w01esx23	none	eno0: From DHCP (reserved)
ESXi-04	sl01w01esx24	none	eno0: From DHCP (reserved)

The below table provides the VM names and details on their network configuration:

VM	Server Name	IP and NICs
High-Speed Ethernet Network	Management Network 192.168.1.0

Server

Name

IP and NICs

High-Speed Ethernet Network

Management Network

192.168.1.0

ESXi

VM-01

sl01w01esx21none

pvrdma-vm01

192.168.11.51

eno0: From DHCP (reserved)

ESXinone

VM-02

sl01w01esx22

pvrdma-vm02

192.168.11.52

eno0: From DHCP (reserved)

ESXinone

VM-03

sl01w01esx23

pvrdma-vm03

192.168.11.53

eno0: From DHCP (reserved)

ESXi

VM-04

sl01w01esx24none

pvrdma-vm04

192.168.11.54

eno0: From DHCP (reserved)

The below table provides details of VMs names and network configuration:

Server

Name

IP and NICsHigh-Speed Ethernet Network

Management Network

192.168.1.0

VM-01pvrdma-vm01192.168.11.51eno0: From DHCP (reserved)VM-02pvrdma-vm02192.168.11.52eno0: From DHCP (reserved)VM-03pvrdma-vm03192.168.11.53eno0: From DHCP (reserved)VM-04pvrdma-vm04192.168.11.54eno0: From DHCP (reserved)

ESXi Host Configuration

Check host configurations

1. EnableSSH Access to ESXi server.

2. Log into ESXi vSphere Command-Line Interface with root permissions.

3. Verify that the host is equipped with Mellanox adapter.

Code Block

ESXi Host Configuration

Check the host configurations:

Enable SSH Access to ESXi server.
Log into the ESXi vSphere Command-Line Interface with root permissions.

Verify that the host is equipped with an NVIDIA adapter card:

Code Block

language	text
theme	FadeToGrey
title	ESXi Console

~ lspci | grep Mellanox

0000:02:00.0 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic2]
0000:02:00.1 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic3]

Info
Note: in this case, the NVIDIA card is using vmnic2 and vmnic3.

Verify that the logical RDMA devices are currently registered on the system:
Code Block
language text
theme FadeToGrey
title ESXi Console

~ lspci

~ esxcli

rdma

grep

device

Mellanox 0000:02:00.0 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic2] 0000:02:00.1 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic3]

Info
Note: in this case, Mellanox card is using vmnic2 and vmnic3.

4. Verify the logical RDMA devices currently registered on the system.

Code Block

language	text
theme	FadeToGrey
title	ESXi Console

~ esxcli rdma device list Name Driver State MTU Speed Paired Uplink Description ------- ---------- ------

list
Name 	 Driver 	 State 	 MTU    Speed 	   Paired Uplink 	Description
-------	 ----------	 ------	 ----   --------

-------

------ -------

----------------------------

-------


vmrdma0	 nmlx5_rdma  Active	 1024	100 Gbps   vmnic2			MT28800 Family [ConnectX-5 MT28831]
vmrdma1	 nmlx5_rdma  Down	 1024	0		   vmnic3			MT28800 Family [ConnectX-5 MT28831]

Deployment

GuidePrerequisites: vSphere

Before starting the deployment process, a vSphere Distributed Switch (vDS) must be created:

Creating a vDS

Perform the following steps to To create a new vDS:

Launch the vSphere Web Client, and connect to a vCenter Server instance.

2. select

On the vSphere Web Client home screen, select the vCenter object from the list on the left.

Then,

Hover over the Distributed Switches from the Inventory Lists area

.3. On the right side of the vSphere Web Client, click the Create a

, and click on the New Distributed Switch icon (

it looks like a switch with a green plus mark in the corner).
This launches Supply

see below image):
Image Added
This will launch the New vDS creation wizard.

Image Removed

Provide a name for the new distributed switch, and select the location within the vCenter inventory

(a datacenter object or a folder)

where you would like to store the new vDS

(a datacenter object or a folder). Click Next.
Image Modified

5. Select the version of the vDS you would like to create.

Select the version of the vDS you would like to create:
Image Modified

6. Specify

Set the number of uplink ports

as 2 and create

to 2. Tick the Create a default port

group with the name.

group box, and give a name to that group:
Image Modified

Click Next

and then Finish

to Finish.

Adding

hosts

Hosts to the vDS

Perform the following steps to To add an ESXi host to an existing vDS:
1.
Launch the vSphere Web Client, and connect to a vCenter Server instance.
2.
Navigate to the list of distributed switches.
3. Select
Choose the new distributed switch
in
from the list of objects on the right, and select Add and Manage Hosts from the Actions menu
.
:

Image Modified
4.
Select the Add
Hosts radio button
hosts button, and click Next
.
:

Image Modified
5.
Click
the green plus icon
on the New hosts option (a green plus icon) to add an ESXi host.
Image Added
This opens the Select New Host dialog box.
Image Removed
6.
From the list of new hosts
to add, place a check mark next to the name of each ESXi host you would like
, tick the boxes with the names of the ESXi hosts you wish to add to the vDS.Image Added
Click
OK when
OK when you are done, and then click
Next to
Next to continue.
Image Removed
7. The
In the next screen, make sure both the Manage physical adapters and Manage VMkernel adapters
options
checkboxes are
selected
ticked. Click Next to continue.
Image Modified
8. Configure vmnic2 in
Configure vmnic2 in each ESXi host as an Uplink 1 for vDS
.
:
Image Modified
9.
Create and attach the vmkernel adapter vmk2 to
vDS port group
the sl01-w01-vds02-
pvrdma. Click
pvrdma vDS port group. Click the green plus icon
and Select an existing network.
Click OK to continue.
Image Removed
10. Click Next to continue.
Image Removed
11. Supply
, and select one of the existing networks. Click OK.Image Added
Image Added
Click Next.
Provide an IPv4 address and Subnet mask for the vmk2 vmkernel adapter
vmk2.
Click Next to continue.
:
Image Modified
12.
Click Next
to continue.
until the wizard is finished:
Image Modified
13. Click Next to continue.
Image Removed
14.
Image Added

Click Finish
.
:
Image Modified

Configure an ESXi Host for PVRDMA

To use PVRDMA in vSphere 6.5/6.7, your environment must meet meet several configuration requirements.

To configure an ESXi host for PVRDMA, perform follow the following below steps, will described in bottom.

Tag a VMkernel Adapter for PVRDMA

Select To tag a VMkernel adapter, select it and enable it for PVRDMA communication using by performing the following steps:

In the vSphere Web Client, navigate to the host.
On the Configure tab, expand System.Click the System subheading and click Advanced System Settings.
Locate Net.PVRDMAvmknic and click Edit.
Enter the value of the VMkernel adapter that you want to use, and click OK.

In our lab environment, we entered vmk2.

Image Removed

(Optional). You can use ESXI CLI to Tag

Image Added
Image Added
In this example, vmk2 was used.

Note
Optional: To tag a vmknic created on the DVS

Note

Optional:

To tag a vmknic created on the DVS

which VRDMA should use

and used by the VRDMA for TCP channel

by run

, you can use ESXI CLI by running the following command line:

Code Block

language	text
theme	FadeToGrey
title	ESXi Console

esxcli system settings advanced set -o /Net/PVRDMAVmknic -s

vmk1

vmk2

Enable

the

Firewall Rule for PVRDMA

Enable To enable the firewall rule for PVRDMA in the security profile of the ESXi host using the following procedure:

In the vSphere Web Client, navigate to the host.
On In the Configure tab, expand the System subheading.
In the Go to Security Profile → Firewall(6.7) or Firewall(6.5) section , click and click Edit.
Scroll to the pvrdma rule and select tick the check relevant box next to it.:
Click OK to finish.

Note

(

Optional

You can

use

use ESXI CLI

to Enable

to enable the pvrdma firewall rule (or disable the firewall

) with the following command line:

Code Block

language	text
theme	FadeToGrey
title	ESXi Console

esxcli network firewall ruleset set -e true -r pvrdma

Assign

PVRDMA Adapter to a Virtual Machine

To enable a virtual machine to exchange data using RDMA, you must associate the VM with a PVRDMA network adapter. The steps are as followsTo do so:

Locate the VM in the vSphere Web Client.
Select a data center, folder, cluster, resource pool , or a host, and click on the VMs tab.
Click Virtual Machines and double-click the VMfrom the list.
Power off the VM.
In the Configure tab of the VM, expand the Settings subheading, and select VM Hardware.
Click Edit, and select the Virtual Hardware tab in the dialog box displaying the settings.
Image Added
At the bottom of the window next to New device, select select Network, and click click Add.
Image Added
Expand the New Network section, and connect the VM to a distributed port group.
For Adapter Type, select PVRDMA.
Expand the Memory section, select Reserve all guest memory (All locked).
Image Removed
Click OK to close the dialog window.
Power on the virtual machine.

Configure Guest OS for PVRDMA

Warning
This step assumes a procedure to assign a PVRDMA Adapter to a Virtual Machine with CentOS 7.2 or later.

To configure a Guest OS for PVRDMA, you need to install a PVRDMA driver. Installation process depends on ESXi version, VM tools and guest OS version.

Related Documents

Content by Label
showLabels false
showSpace false
sort creation
cql label in ("roce","vmware")

Guest OS: CentOS 7.3 and later
VM hardware version 14
ESXi v6.7
Create VM with VM Compatibility version 14 and install CentOS 7.3 or later.
Add PVRDMA adapter over DVS portgroup from vCenter.
Install InfiniBand packages and reload pvrdma driver:
Code Block
language text
theme FadeToGrey
title VM Console
yum groupinstall "Infiniband Support" –y rmmod vmw_pvrdma modprobe vmw_pvrdma ibv_devinfo
Guest OS: CentOS 7.2
VM hardware version 13
ESXi v6.5
Create VM  with VM Compatibility version 13 and install CentOS 7.2.
Add PVRDMA adapter over DVS portgroup from vCenter.
Install InfiniBand drivers:
Code Block
language text
theme FadeToGrey
title VM Console
yum groupinstall "Infiniband Support" –y
Install pvrdma driver:
Code Block
language text
theme FadeToGrey
title VM Console
tar xf vrdma_ib_devel.tar cd vrdma_ib_devel/ make cp pvrdma.ko /lib/modules/3.10.0-327.el7.x86_64/extra/ depmod –a modprobe pvrdma
Install vrdma lib:
Code Block
language text
theme FadeToGrey
title VM Console
cd /tmp/ tar xf libvrdma_devel.tar cd libvrdma_devel/ ./autogen.sh ./configure --libdir=/lib64 make make install cp pvrdma.driver /etc/libibverbs.d/ rmmod pvrdma modprobe pvrdma ibv_devinfo
port group.
For Adapter Type, select PVRDMA.
Image Added
Expand the Memory section, tick the box next to Reserve all guest memory (All locked).
Image Added
Click OK to close the dialog window.
Power on the virtual machine.
Configure Guest OS for PVRDMA
Warning
A prerequisite of this step is assigning a PVRDMA adapter to a Virtual Machine with CentOS 7.2 or later, and Ubuntu 18.04.
To configure a Guest OS for PVRDMA, install a PVRDMA driver. The installation process depends on the ESXi version, VM tools and Guest OS version:
Guest OS: CentOS 7.3 and later
VM hardware version 14
ESXi v6.7 Guest OS: CentOS 7.2
VM hardware version 13
ESXi v6.5

Create
VM  with
a VM with VM Compatibility version 14, and
install CentOS
install CentOS version 7.
3 or
3 or later.
Add the PVRDMA adapter over a DVS portgroup from the vCenter.
Install the InfiniBand packages, and reload the pvrdma driver with the following command line:
Code Block
language text
theme FadeToGrey
title VM Console
yum groupinstall "Infiniband Support" –y rmmod vmw_pvrdma modprobe vmw_pvrdma ibv_devinfo

Create
VM  with
a VM with VM Compatibility version 13, and install CentOS 7.2.
Add the PVRDMA adapter over a DVS portgroup from
vCenter.
Install InfiniBand drivers
the vCenter.
Install the InfiniBand drivers with the following command line:
Code Block
language text
theme FadeToGrey
title VM Console
yum groupinstall "Infiniband Support" –y
Install the pvrdma driver:
Code Block
language text
theme FadeToGrey
title VM Console
yum groupinstall "Infiniband Support" –y
Install pvrdma driver
tar xf vrdma_ib_devel.tar cd vrdma_ib_devel/ make cp pvrdma.ko /lib/modules/3.10.0-327.el7.x86_64/extra/ depmod –a modprobe pvrdma
Install the vrdma lib:
Code Block
language text
theme FadeToGrey
title VM Console
cd /tmp/ tar xf
vrdma_ib
libvrdma_devel.tar cd
vrdma_ib
libvrdma_devel/
make

cp pvrdma
.
ko /lib/modules/3.10.0-327.el7.x86_64/extra/ depmod –a modprobe pvrdmaInstall vrdma lib
/autogen.sh ./configure --libdir=/lib64 make make install cp pvrdma.driver /etc/libibverbs.d/ rmmod pvrdma modprobe pvrdma ibv_devinfo

For Guest OS: Ubuntu 18.04, VM hardware version 14, ESXi v6.7 , the vmw_pvrdma driver should already be included in 18.04. The user level libraries can be installed using:
Code Block
language text
theme FadeToGrey
title VM Console
cd /tmp/ tar xf libvrdma_devel.tar cd libvrdma_devel/ ./autogen.sh ./configure --libdir=/lib64 make make install cp pvrdma.driver /etc/libibverbs.d/ rmmod pvrdma modprobe pvrdma ibv_devinfo
Info
You can upgrade VM Compatibility to v.
apt-get install rdma-core reboot
Info
In case the VM compatibility version does not match the above, upgrade the VM compatibility to version 13 in ESXi 6.5, and to version 14 in Esxi 6.7 if not upgraded.
Deployment Verification
To test the communication using PVRDMA we will use a Perftest, Perftest is used. This is a collection of tests written over uverbs intended for use as a performance micro-benchmark.
The tests may be used for
HW
hardware or
SW
software tuning, as well as for functional testing.
To install and run
benchmark go by the steps are as follows
the benchmark:
Install Perftest:
Code Block
language text
theme FadeToGrey
title VM Console
yum install git git clone https://github.com/linux-rdma/perftest.git cd perftest/ yum install autotools-dev automake yum install libtool yum install libibverbs-devel ./autogen.sh ./configure make -j 8
Check the network interface name:
Code Block
language text
theme FadeToGrey
title VM Console
ifconfig ... ens224f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
Add Static IP configuration to the network interface. Modify /etc/sysconfig/network-scripts/ifcfg-ens224f0
Code Block
language text
theme FadeToGrey
title Add to file
HWADDR=00:50:56:aa:65:92 DNS1=192.168.1.21 DOMAIN=vwd.clx BOOTPROTO="static" NAME="ens224f0" DEVICE="ens224f0" ONBOOT="yes" USERCTL=no IPADDR=192.168.11.51 NETMASK=255.255.255.0 PEERDNS=no IPV6INIT=no IPV6_AUTOCONF=no ping 192.168.11.51
Repeat steps 1-3
to
for the second
to
VM.

On the first VM ("Server"), run the following:
Code Block
language text
theme FadeToGrey
title VM01 "Server" Console
systemctl disable firewall systemctl stop firewalld systemctl disable firewalld firewall-cmd --state ./ib_write_bw -x 0 -d vmw_pvrdma0 --report_gbits
On the second VM ("Client"), run the following:
Code Block
language text
theme FadeToGrey
title VM02 "Client" Console
./ib_write_bw -x 0 -F 192.168.11.51 -d vmw_pvrdma0 --report_gbits ************************************ * Waiting for client to connect... * ************************************ -------------------------------------------------------------------------------- ------- RDMA_Write BW Test Dual-port : OFF Device : vmw_pvrdma0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF CQ Moderation : 100 Mtu : 1024[B] Link type : Ethernet GID index : 0 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet -------------------------------------------------------------------------------- ------- local address: LID 0000 QPN 0x0004 PSN 0xfb9486 RKey 0x000005 VAddr 0x007f68c62 a1000 GID: 254:128:00:00:00:00:00:00:02:80:86:255:254:170:101:146 remote address: LID 0000 QPN 0x0002 PSN 0xe72165 RKey 0x000003 VAddr 0x007f2ab4 361000 GID: 254:128:00:00:00:00:00:00:02:80:86:255:254:170:58:174 -------------------------------------------------------------------------------- ------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 90.56 90.39 0.172405 -------------------------------------------------------------------------------- -------

Done!
Authors
Include Page
SA:Boris Kovalev
SA:Boris Kovalev
Related Documents
Content by Label
showLabels false
showSpace false
sort creation
cql label in ("roce","vmware")

Content

Space Tools

Page History

Versions Compared

Old Version 47

New Version Current

Key

Introduction

References

Components Overview

vSphere Distributed Switch

Paravirtual RDMA (PVRDMA)

PVRDMA Architecture

Accelerating VM Data

Solution Overview

Equipment

Solution Logical Design

Bill of Materials

Image Removed

Solution Physical Network Wiring

Network Configuration

Image Added

Accelerating VM Data

Solution Overview

Setup

Solution Logical Design

Bill of Materials

Solution Physical Network Wiring

Configuration

Network Configuration

ESXi Host Configuration

Check host configurations

ESXi Host Configuration

Check the host configurations:

Deployment

Before starting the deployment process, a vSphere Distributed Switch (vDS) must be created:

Creating a vDS

Adding

Hosts to the vDS

Configure an ESXi Host for PVRDMA

Tag a VMkernel Adapter for PVRDMA

Image Removed

Enable

Firewall Rule for PVRDMA

Configure Guest OS for PVRDMA

Configure Guest OS for PVRDMA

Deployment Verification

Authors