HowTo Change Port Type of NVIDIA ConnectX VPI Adapter on VMware ESXi 6.x and above - Latest Release

HowTo Configure NVIDIA Network Device in VMDirectPath IO Passthrough Mode on VMware ESXi 6.x

Created on Jun 9, 2019

Introduction

This post describes th e procedure of how to configure NVIDIA network device into and from VMDirectPath I/O passthrough mode on VMware ESXi 6.x.

References

About VMDirectPath I/O passthrough devices

The supported maximums for passthrough devices per VMs are:

  • ESXi 6.5: 16 Devices

  • ESXi 6.7: 16 Devices

1.png

Configuring passthrough NVIDIA network device

Before configuring a device for PCI Passthrough, ensure that platform and device meets the PCI Passthrough requirements, see VMware vSphere VMDirectPath I/O: Requirements for Platforms and Devices (2142307).

To configure passthrough devices on an ESX/ESXi host:

1. Select an ESXi host from the Inventory of VMware vSphere Client.

2. Enter Maintenance Mode the ESXi host.

3. In the Navigator click Manage, Hardware and PCI Devices. The passthrough Configuration page lists all available pass-through devices.

4. Select the devices and click Toggle passthrough.

2.png

5. Reboot host for the change to take effect. After rebooting, the devices are marked with a green icon and are enabled.

Note

Note: The configuration changes are saved in the /etc/vmware/esx.conf file.

The PCI slot number where the device was connected is 02:00.1. It is recorded as:

Copy
Copied!
            

/device/00000:002:00.1/owner = = "passthru"

6. To verify in:

UI:

a. Select an ESX\ESXi host from the Inventory of VMware vSphere Client.

b. In the Navigator click Manage, Hardware and PCI Devices. The passthrough Configuration page lists all available pass-through devices.

c. Verify:

3.png

In CLI:

a. Enable SSH Access to ESXi server.

b. Log into ESXi vSphere Command-Line Interface with root permissions.

c. View the /etc/vmware/esx.conf file by changing the owner of the PCI device from passthru to vmkernel.

ESXi Console

Copy
Copied!
            

# vi /etc/vmware/esx.conf

Find /device/<PCI_ID>/owner = "passthru".

d. Check that:

ESXi Console

Copy
Copied!
            

/device/00000:002:00.1/owner = = "passthru"

7. Reboot the ESXi host.

8. Exit Maintenance Mode the ESXi host.

To configure a PCI device on a virtual machine:

1. From the Navigator in vSphere Client, right-click the virtual machine and click Edit Settings.
2. Select the Virtual Hardware tab.
3. Click Add other device.

4.png

Choose the PCI Device.

5.png

Click Save.

Note

Note: When the device is assigned, the virtual machine must have a memory reservation for the full configured memory size.

6.png

Configuring non-passthrough NVIDIA network device

To force the change to non-passthrough mode, complete these steps:

  1. Enable SSH Access to ESXi server.

  2. Log into ESXi vSphere Command-Line Interface with root permissions.

  3. Enter Maintenance Mode the ESXi host.

  4. Edit the /etc/vmware/esx.conf file by changing the owner of the PCI device from passthru to vmkernel.

    ESXi Console

    Copy
    Copied!
                

    # vi /etc/vmware/esx.conf

  5. Find /device/<PCI_ID>/owner = "passthru". Write:

    ESXi Console

    Copy
    Copied!
                

    /passthru

  6. Change from:
    /device/00000:002:00.1/owner = = "passthru"
    Change to:
    /device/00000:002:00.1/owner = = "vmkernel"

7. Reboot the ESXi host.

8. Verify.

ESXi Console

Copy
Copied!
            

# lspci | grep Mellanox   0000:02:00.0 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic2] 0000:02:00.1 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic3]

7.png

9. Exit Maintenance Mode the ESXi host.

Done!

Authors

image2020-11-17_6-50-16.png

Boris Kovalev

Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the Mellanox Documents website.

Last updated on Sep 12, 2023.