HowTo Change Port Type of NVIDIA ConnectX VPI Adapter on VMware ESXi 6.x and above

Created on Jun 4, 2019

Updated on Sep 13, 2021

Introduction

This How To describes how to manage the port type (InfiniBand or Ethernet) of NVIDIA ConnectX® InfiniBand/VPI Cards in VMware ESXi 6.x and above.

References

• InfiniBand Cards - Overview

• NVIDIA Mellanox OFED InfiniBand Driver for VMware® ESXi Server

• NVIDIA Firmware Tools (MFT) Product Page

• Using ESXi Shell in ESXi 5.x, 6.x and 7.x (2004746)

Hardware and Software Requirements

1. A server platform with an adapter card based on one of the following NVIDIA Technologies’ HCA devices:

• ConnectX®-5 Single/Dual-Port Adapter supporting 100Gb/s with VPI

• ConnectX®-6 Dx

2. Installer Privileges: The installation requires administrator privileges on the target machine.

3. Device ID: For the latest list of device IDs, please visit NVIDIA NETWORKING website.

Port Type Management

NVIDIA ConnectX adapter card may be equipped with one or two ports can be individually configured to work as InfiniBand or Ethernet ports.

The port type depends on the card type. In case of a VPI card, the default type is IB. If you wish to change the port type use the mlxconfig script, is included in NVIDIA Firmware Tools(MFT).

For further information on how to install MFT please use following link - How-to: Install NVIDIA Firmware Tools (MFT) on VMware ESXi 6.7/7.0.

The protocol types are:

• Port Type 1 = IB

• Port Type 2 = Ethernet

To print the current status of Mellanox devices:

1. Enable SSH Access to ESXi server.

2. Log into ESXi vSphere Command-Line Interface with root permissions.

3. Run the following command:

            

            
/opt/mellanox/bin/mst status
MST devices:
 
------------
 
mt4125_pciconf7
        

To use a VPI card as an Ethernet only type card:

1. Enter Maintenance Mode the ESXi host.

2. And run:

            

            
~ /opt/mellanox/bin/mlxconfig -d /dev/mt4125_pciconf7 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
        

3. Restart a server.

            

            
~ reboot
        

4. Exit from Maintenance Mode the ESXi host.

For more information on how to set the port type in ConnectX®-5/ConnectX®-6 , please refer to the MFT User Manual ( NVIDIA Firmware Tools (MFT) Product Page ).

Done!

Authors

Boris Kovalev Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the Mellanox Documents website.