Page History

...

This section describes how to install and test the Mellanox the NVIDIA OFED for Linux package on a single server with a Mellanox a NVIDIA ConnectX-5 adapter card installed.

...

...

Downloading NVIDIA OFED

1. Verify that the system has a Mellanox a NVIDIA network adapter installed.
   The following example shows a system with an installed Mellanox installed NVIDIA adapter card:

   Code Block
   language text
   # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014

2. Download the ISO image to your host.
   The image’s name has the format MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.iso. 
   You can download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox the NVIDIA web site at http://www.mellanoxnvidia.com > Products > Software > InfiniBand/VPI Drivers > Linux SW/Drivers > Download..

   1. Scroll down to the Download wizard, and click the Download tab.
   2. Choose your relevant package depending on your host operating system.
   3. Click the desired ISO/tgz package.
   4. To obtain the download link, accept the End User License Agreement (EULA).

3. Use the md5sum utility to confirm the file integrity of your ISO image. Run the following command and compare the result to the value provided on the download page.

   Code Block
   language text
   md5sum MLNX_OFED_LINUX-<ver>-<OS label>.iso

...

Installing NVIDIA OFED

Installation Script

The installation script, mlnxofedinstall, performs the following:

...

The installation script removes all previously installed Mellanox installed NVIDIA OFED packages and re-installs from scratch. You will be prompted to acknowledge the deletion of the old packages.

• If you need to install Mellanox install NVIDIA OFED on an entire (homogeneous) cluster, a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS. To install on all the cluster nodes, use cluster-aware tools (such as pdsh).

• If your kernel version does not match with any of the offered pre-built RPMs, you can add your kernel version by using the “mlnx_add_kernel_support.sh” script located inside the MLNX_OFED package.

  Note
  On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_kernel_support.sh script. The regular installation can be performed and weak updates mechanism will create symbolic links to the MLNX_OFED kernel modules.

  The “mlnx_add_kernel_support.sh” script can be executed directly from the mlnxofedinstall script. For further information, please see '--add-kernel-support' option below.

  Note
  On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_kernel_support.sh" is irrelevant on Ubuntu and Debian distributions.

  Example
  The following command will create a MLNX_OFED_LINUX ISO image for RedHat 6.3 under the /tmp directory.

  Code Block
  language text
  # ./MLNX_OFED_LINUX-x.x-x-rhel6.3-x86_64/mlnx_add_kernel_support.sh -m /tmp/MLNX_OFED_LINUX-x.x-x-rhel6.3-x86_64/ --make-tgz Note: This program will create MLNX_OFED_LINUX TGZ for rhel6.3 under /tmp directory. All Mellanox, OEM, OFED, or Distribution IB packages will be removed. Do you want to continue?[y/N]:y See log file /tmp/mlnx_ofed_iso.21642.log Building OFED RPMs. Please wait... Removing OFED RPMs... Created /tmp/MLNX_OFED_LINUX-x.x-x-rhel6.3-x86_64-ext.tgz

...

For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be running on one of the fabric nodes. At this point, Mellanox NVIDIA OFED for Linux has already installed the OpenSM Subnet Manager on your machine.
For the list of installation options, run:

...

After installation completion, information about the Mellanox the NVIDIA OFED installation, such as prefix, kernel version, and installation parameters can be retrieved by running the command /etc/infiniband/info.

Most of the Mellanox the NVIDIA OFED components can be configured or reconfigured after the installation, by modifying the relevant configuration files. See the relevant chapters in this manual for details.

...

   The firmware for this device is not distributed inside Mellanox driver: 0000:01:00.0 (PSID: IBM2150110033)
   To obtain firmware for this device, please contact your HW vendor.

...

Driver Load Upon System Boot

Upon system boot, the Mellanox the NVIDIA drivers will be loaded automatically.

To prevent automatic load of the Mellanox the NVIDIA drivers upon system boot:

...

Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox the NVIDIA OFED package. The script is part of the ofed-scripts RPM.

...

1. Log into the installation machine as root.

2. Mount the ISO image on your machine and copy its content to a shared location in your network.

   Code Block
   # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt

3. Download and install Mellanox Technologies install NVIDIA GPG-KEY:
   The key can be downloaded via the following link: http://www.mellanoxnvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox

   Code Block
   language text
   # wget http://www.mellanoxnvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox --2014-04-20 13:52:30-- http://www.mellanoxnvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox Resolving www.mellanoxnvidia.com... 72.3.194.0 Connecting to www.mellanoxnvidia.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox? 100%[=================================================>] 1,354 --.-K/s in 0s 2014-04-20 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox? saved [1354/1354]

4. Install the key.

   Code Block
   language text
   # sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mellanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]:

5. Check that the key was successfully imported.

   Code Block
   # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' | grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies <support@mellanox.com>)

6. Create a yum repository configuration file called "/etc/yum.repos.d/mlnx_ofed.repo" with the following content:

   Code Block
   [mlnx_ofed] name=MLNX_OFED Repository baseurl=file:///<path to extracted MLNX_OFED package>/RPMS enabled=1 gpgkey=file:///<path to the downloaded key RPM-GPG-KEY-Mellanox> gpgcheck=1

7. Check that the repository was successfully added.

   Code Block
   language text
   # yum repolist Loaded plugins: product-id, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status mlnx_ofed MLNX_OFED Repository 108 rpmforge RHEL 6Server - RPMforge.net - dag 4,597 repolist: 8,351

...

Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox the NVIDIA OFED package. The script is part of the ofed-scripts RPM.

...

1. Log into the installation machine as root.
2. Extract the MLNX_OFED pacakge package on a shared location in your network.
   You can download it from http://www.mellanoxnvidia.com > Products > Software> InfiniBand/VPI Drivers.

3. Create an apt-get repository configuration file called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content:

   Code Block
   # deb file:/<path to extracted MLNX_OFED package>/DEBS ./

4. Download and install Mellanox Technologies install NVIDIA GPG-KEY.

   Code Block
   # wget -qO - http://www.mellanoxnvidia.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -

5. Check that the key was successfully imported.

   Code Block
   # apt-key list pub 1024D/A9E4B643 2013-08-11 uid Mellanox Technologies <support@mellanox.com> sub 1024g/09FCC269 2013-08-11

6. Update the apt-get cache.

   Code Block
   # sudo apt-get update

...

1. View the available package groups by invoking:

   Code Block

   <pre># apt-cache search mlnx-ofed- mlnx-ofed-vma-eth - MLNX_OFED vma-eth installer package (with DKMS support) mlnx-ofed-hpc - MLNX_OFED hpc installer package (with DKMS support) mlnx-ofed-vma-vpi - MLNX_OFED vma-vpi installer package (with DKMS support) mlnx-ofed-basic - MLNX_OFED basic installer package (with DKMS support) mlnx-ofed-vma - MLNX_OFED vma installer package (with DKMS support) mlnx-ofed-all - MLNX_OFED all installer package (with DKMS support)

   Where:

   Code Block
   language cpp
   mlnx-ofed-all MLNX_OFED all installer package. mlnx-ofed-basic MLNX_OFED basic installer package. mlnx-ofed-vma MLNX_OFED vma installer package. mlnx-ofed-hpc MLNX_OFED HPC installer package. mlnx-ofed-vma-eth MLNX_OFED vma-eth installer package. mlnx-ofed-vma-vpi MLNX_OFED vma-vpi installer package.

2. Install the desired group.

   Code Block
   # apt-get install '<group name>'

   Example:

   Code Block
   # apt-get install mlnx-ofed-all

   Note
   Installing MLNX_OFED using the “apt-get” tool does not automatically update the firmware. To update the firmware to the version included in MLNX_OFED package, run: # apt-get install mlnx-fw-updater Or, update the firmware to the latest version available on Mellanox Technologies’ Web site the NVIDIA as described in Updating Adapter Firmware.

...

Use the script /usr/sbin/ofed_uninstall.sh to uninstall the Mellanox the NVIDIA OFED package. The script is part of the ofed-scripts package.

...

To update the device online on the machine from the Mellanox the NVIDIA site, use the following command line:

...

To update the device manually, please refer to the OEM Firmware Download page at http://www.mellanoxnvidia.com/page/firmware_table_dell?mtag=oem_firmware_download.

In case that you ran the mlnxofedinstall script with the ‘--without-fw-update’ option or you are using an OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case that you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web the NVIDIA web site (http://www.mellanoxnvidia.com > Support > Firmware Download).

1. Get the device’s PSID.

   Code Block
   mlxfwmanager_pci | grep PSID PSID: MT_1210110019

2. Download the firmware BIN file from the Mellanox the NVIDIA website or the OEM website.
   
   

3. Burn the firmware.

   Code Block
   mlxfwmanager_pci -i <fw_file.bin>

4. Reboot your machine after the firmware burning is completed.

...

All kernel modules included in MLNX_OFED for RHEL7 and SLES12 are signed with x.509 key to support loading the modules when Secure Boot is enabled.

Enrolling

...

NVIDIA's x.509 Public Key on Your Systems

In order to support loading MLNX_OFED drivers when an OS supporting Secure Boot boots on a UEFI-based system with Secure Boot enabled, the Mellanox the NVIDIA x.509 public key should be added to the UEFI Secure Boot key database and loaded onto the system key ring by the kernel.
Follow these steps below to add the MellanoxNVIDIA's x.509 public key to your system:

1. Download the x.509 public key.

   Code Block
   # wget http://www.mellanoxnvidia.com/downloads/ofed/mlnx_signing_key_pub.der

2. Add the public key to the MOK list using the mokutil utility.

   Code Block
   # mokutil --import mlnx_signing_key_pub.der

3. Reboot the system.

...

Depending on the application of the user's system, it may be necessary to modify the default configuration of network adapters based on the ConnectX® adapters. In case that tuning is required, please refer to the Performance Tuning Guide for Mellanox NVIDIA Network Adapters at https://supportcommunity.mellanoxnvidia.com/docs/DOC-2489.