NVIDIA MLNX_EN Documentation v23.10

Introduction

This manual is intended for system administrators responsible for the installation, configuration, management and maintenance of the software and hardware of Ethernet adapter cards. It is also intended for application developers.

This document provides information about MLNX_EN Linux driver, and instructions on how to install the driver on ConnectX network adapter solutions supporting the following uplinks to servers:

Uplink/NICs

Driver Name

Uplink Speed

BlueField-2

mlx5

  • InfiniBand: SDR, FDR, EDR, HDR

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE2, 100GbE2

BlueField

  • InfiniBand: SDR, QDR, FDR, FDR10, EDR

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE, 100GbE

ConnectX-6 Dx

  • Ethernet: 10GbE, 25GbE, 40GbE, 50GbE2, 100GbE2, 200GbE2

ConnectX-6 Lx

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE2

ConnectX-6

  • InfiniBand: SDR, FDR, EDR, HDR

  • Ethernet: 10GbE, 25GbE, 40GbE, 50GbE2, 100GbE2, 200GbE2

ConnectX-5/ConnectX-5 Ex

  • InfiniBand: SDR, QDR, FDR, FDR10, EDR

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE, 100GbE

ConnectX-4 Lx

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE

ConnectX-4

  • InfiniBand: SDR, QDR, FDR, FDR10, EDR

  • Ethernet: 1GbE, 10GbE, 25GbE, 40GbE, 50GbE, 56GbE1, 100GbE

  1. 56GbE is an NVIDIA proprietary link speed and can be achieved while connecting an NVIDIA adapter card to
    NVIDIA SX10XX switch series or when connecting an NVIDIA adapter card to another NVIDIA adapter card.

  2. Supports both NRZ and PAM4 modes.

MLNX_EN driver release exposes the following capabilities:

  • Single/Dual port

  • Multiple Rx and Tx queues

  • Rx steering mode: Receive Core Affinity (RCA)

  • MSI-X or INTx

  • Adaptive interrupt moderation

  • HW Tx/Rx checksum calculation

  • Large Send Offload (i.e., TCP Segmentation Offload)

  • Large Receive Offload

  • Multi-core NAPI support

  • VLAN Tx/Rx acceleration (HW VLAN stripping/insertion)

  • Ethtool support

  • Net device statistics

  • SR-IOV support

  • Flow steering

  • Ethernet Time Stamping

Package Images

MLNX_EN is provided as an ISO image or as a tarball per Linux distribution and CPU architecture that includes source code and binary RPMs, firmware and utilities. The ISO image contains an installation script (called install) that performs the necessary steps to accomplish the following:

  • Discover the currently installed kernel

  • Uninstall any previously installed MLNX_OFED/MLNX_EN packages

  • Install the MLNX_EN binary RPMs (if they are available for the current kernel)

  • Identify the currently installed HCAs and perform the required firmware updates

Software Components

MLNX_EN contains the following software components:

Components

Description

mlx5 driver

mlx5 is the low level driver implementation for the ConnectX-4 adapters. ConnectX-4 operates as a VPI adapter.

mlx5_core

Acts as a library of common functions (e.g. initializing the device after reset) required by the ConnectX-4 adapter cards.

mlx4 driver

mlx4 is the low level driver implementation for the ConnectX adapters. The ConnectX can operate as an InfiniBand adapter and as an Ethernet NIC.

To accommodate the two flavors, the driver is split into modules: mlx4_core, mlx4_en, and mlx4_ib.

Note: mlx4_ib is not part of this package.

mstflint

An application to burn a firmware binary image.

Software modules

Source code for all software modules (for use under conditions mentioned in the modules' LICENSE files)

Firmware

The ISO image includes the following firmware item:

  • Firmware images (.bin format wrapped in the mlxfwmanager tool) for ConnectX-4 and and above network adapters

Directory Structure

The tarball image of MLNX_EN contains the following files and directories:

  • install—the MLNX_EN installation script

  • uninstall.sh—the MLNX_EN un-installation script

  • RPMS/—directory of binary RPMs for a specific CPU architecture

  • src/—directory of the OFED source tarball

  • mlnx_add_kernel_support.sh—a script required to rebuild MLNX_EN for customized kernel version on supported Linux distribution

mlx5_core Module Parameters

The mlx5_core module supports a single parameter used to select the profile which defines the number of resources supported.

prof_sel

The parameter name for selecting the profile. The supported values for profiles are:

  • 0—for medium resources, medium performance

  • 1—for low resources

  • 2—for high performance (int) (default)

guids

charp

node_guid

guids configuration. This module parameter will be obsolete!

debug_mask

debug_mask: 1 = dump cmd data, 2 = dump cmd exec time, 3 = both. Default=0 (uint)

probe_vf

probe VFs or not, 0 = not probe, 1 = probe. Default = 1 (bool)

num_of_groups

Controls the number of large groups in the FDB flow table.

Default=4; Range=1-1024


The following parameters, supported in mlx4 driver only, can be changed using the Devlink user interface:

Parameter

Description

Parameter Type

internal_error_reset

Enables resetting the device on internal errors

Generic

max_macs

Max number of MACs per ETH port

Generic

region_snapshot_enable

Enables capturing region snapshots

Generic

enable_64b_cqe_eqe

Enables 64 byte CQEs/EQEs when supported by FW

Driver-specific

enable_4k_uar

Enables using 4K UAR

Driver-specific

© Copyright 2023, NVIDIA. Last updated on Dec 27, 2023.