Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Created on Jun 4, 2019

Updated on Sep 13, 2021

Introduction

This is a How To that shows how to install and test the NVIDIA ConnectX-5/ConnectX-6 NATIVE ESXi Driver for VMware vSphere ESXi 6.57/67.7 0 on single host, and perform basic initial configuration steps to enable the driver by using ESXi cli interface.

References

Hardware and Software Requirements

1. A server platform with an adapter card based on one of the following NVIDIA ConnectX®-4/5 PCI Express Adapter Cards (Ethernet, VPI) devices.

2. Installer Privileges: The installation requires administrator privileges on the target machine.

3. Device ID: For the latest list of device IDs, please visit NVIDIA website.

4. Supported NICs / Firmware: Recommended firmware versions you can find here.

Driver Installation (CLI)

1. Enable SSH Access to ESXi server.

2. Log into ESXi vSphere Command-Line Interface with root permissions.

3. Verify that the host is equipped with NVIDIA ConnectX (Mellanox) adapter.


Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ lspci | grep Mellanox

0000:0239:00.0 NetworkEthernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic2ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AC-CDA) [vmnic0]
0000:0239:00.1 NetworkEthernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic3ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AC-CDA) [vmnic1]
Info
Note: in this case, the Mellanox card is using vmnic2vmnic0 and vmnic3vmnic1.

4. Verify the driver's version was installed.

ESXi 67.50

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli software vib list | grep nmlx
nmlx5-core 4.19.16.13.510-10EM1vmw-650.0.0-8277601 VMW VMwareCertified 2018-04-24 
nmlx5-rdma 4.19.16.13.510-10EM1vmw-650.0.0-8277601 VMW VMwareCertified 2018-04-24

ESXi 6.7

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli software vib list | grep nmlx 

...

nmlx5-core 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-25
nmlx5-rdma 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-25


5. Download a latest Mellanox native ESXi drivers from here.

6. Unzip the binary image (.zip file).

7. Use SCP or any other file transfer method to copy the driver to the required ESXi host.

8. Enter Place a Host in Maintenance Mode the ESXi host.

9. Install the driver.

esxcli software vib install –d <path>/<bundle_file>

ESXi 67.50

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli software vib install -d /tmp/MLNX-NATIVE-ESX-ConnectX-4-5Mellanox-nmlx5_4.1621.1371.5-10EM-650101-1OEM.702.0.0.459867317630552.zip

ESXi 6.7

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli software vib install -d /tmp/MLNX-NATIVE-ESX-ConnectX-4-5Mellanox-nmlx5_4.17.1371.81-10EM-1OEM.670.0.0.8169922.zip


10. Reboot the server.

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ reboot

11. Verify the driver modules was installed and loaded successfully.

Code Block
languagetext
themeFadeToGrey
titleESXi Console
esxcli software vib list | grep nmlx
nmlx5-core 4.17.1371.81-1OEM.670.0.0.8169922 MEL VMwareCertified 2018-06-07
nmlx5-rdma 4.17.1371.81-1OEM.670.0.0.8169922 MEL VMwareCertified 2018-06-07

...
Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli system module list | grep nmlx5

nmlx5_core true true
nmlx5_rdma true true

12. Check physical network interface status.

Code Block
languagetext
themeFadeToGrey
titleESXi Console
~ esxcli network nic list


Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ---------- ------------ ----------- ------ ------ ----------------- ---- ----------------------------------------------------
...
vmnic2 0000:02:00.0 nmlx5_core Up Up 100000 Full ec:0d:9a:8c:c7:96 1500 Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
vmnic3 0000:02:00.1 nmlx5_core Up Down 0 Half ec:0d:9a:8c:c7:97 1500 Mellanox Technologies MT28800 Family [ConnectX-5 Ex]


13. Exit Maintenance Mode the ESXi host.


Done!

Related Documents

Content by Label
showLabelsfalse
showSpacefalse
sortcreation
cqllabel in ("esxi","connectx")