Created on Apr 6, 2020
Introduction
This post describes th e procedure of how to configure NVIDIA network device into and from DirectPath I/O and Dynamic DirectPath I/O passthrough modes on VMware ESXi 7.0.
Assignable Hardware is a framework that allows Dynamic DirectPath I/O to use vSphere HA and DRS for initial VM placement.
When powering on a VM with a NVIDIA vGPU profile, DRS will choose an ESXi host to place that VM.
Assignable hardware requires hardware version 17 of the VM.
DRS load balancing of Dynamic DirectPath I/O devices is not available yet. So only for the initial placement of the VM.
References
DirectPath I/O and Dynamic DirectPath I/O passthrough modes
Configuring passthrough NVIDIA network device
Before configuring a device for PCI Passthrough, ensure that platform and device meets the PCI Passthrough requirements, see VMware vSphere VMDirectPath I/O: Requirements for Platforms and Devices (2142307).
To configure passthrough devices on an ESXi host:
1. Select an ESXi host from the Inventory of VMware vSphere Client.
2. Enter Maintenance Mode the ESXi host.
3. In the Navigator click Configure, Hardware and PCI Devices. The passthrough Configuration page lists all available pass-through devices.
4. Click CONFIGURE PASSTHROUGH.
5. Select the devices and add Hardware Label ("ConnectX-5").
6. Click OK.
The PCI slot of the device is 13:00.1.
7. Verify Passthrough mode for the device is configured:
a. Select an ESX\ESXi host from the Inventory of VMware vSphere Client.
b. In the Navigator click Configure, Hardware, PCI Devices and Passthrough-enabled devices. The Passthrough-enabled devices page lists all enabled pass-through devices.
c. Verify:
8. Exit Maintenance Mode the ESXi host.
Configure a PCI device on a virtual machine:
1. From the Navigator in vSphere Client, right-click the virtual machine and click Edit Settings.
2. Select the Virtual Hardware tab.
3. Click Add other device.
Choose the PCI Device.
Select DirectPath IO or Dynamic DirectPath IO.
From the SELECT HARDWARE drop-down menu, select PCI device.
Click OK.
Note: When the device is assigned, the virtual machine will have a memory reservation for the full configured memory size automatically.
Done!
Check Distributed Resource Scheduler (DRS) initial placement of workload Dynamic DirectPath I/O
Before checking Distributed Resource Scheduler (DRS) with for initial placement of workload , ensure that vSphere DRS is enabled:
1. Select an Cluster host from the Inventory of VMware vSphere Client.
2. Ensure that vSphere DRS is Turned ON.
3. Put Automation level to Manual. And click OK.
We put Automation level to Manual only to see placement recommendation. After you can put DRS back to your company police level.
Distributed Resource Scheduler (DRS) initial placement of workload Dynamic DirectPath I/O
You need to use same PCI Passthrough-enabled devices (device PCI ID) on all ESXi servers for DRS initial placement of workload Dynamic DirectPath I/O.
In our case it's 13:00.1
1. From the Navigator in vSphere Client, right-click the powered off virtual machine, click Power and Power On.
2. You will see Power On Recommendations.
3. Verify that you have the network adapter in the VM guest OS.
4. Power Off the VM.
5. Enter to Maintenance Mode the ESXi server.
6. Power On the VM.
7. You will see new Power On Recommendations.
Done!
Remove a PCI device from a virtual machine
1. From the Navigator in vSphere Client, right-click the virtual machine and click Edit Settings.
2. Select the Virtual Hardware tab.
3. Click X on the PCI device that you want to remove. And click OK.
Reverting NVIDIA network device back to non-passthrough mode
To force the change to non-passthrough mode, complete these steps:
1. Select an ESXi host from the Inventory of VMware vSphere Client.
2. Enter Maintenance Mode the ESXi host.
3. In the Navigator click Configure, Hardware and PCI Devices. The passthrough Configuration page lists all available pass-through devices.
4. Click CONFIGURE PASSTHROUGH.
5. Deselect the device you want to change to non-passthrough mode.
5. Click OK.
9. Exit Maintenance Mode the ESXi host.
Done!
Authors
|
|
Boris Kovalev Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the Mellanox Documents website. |