NVIDIA ConnectX-6 Lx PCIe HHHL Ethernet Adapter Cards User Manual

Introduction

This is the User Manual for Ethernet adapter cards based on the ConnectX®-6 Lx integrated circuit device.

Providing up to two ports of 25GbE or a single-port of 50GbE connectivity, and PCIe Gen 3.0/4.0 x8 host connectivity, ConnectX-6 Lx is a member of NVIDIA’s world-class, award-winning, ConnectX family of network adapters. Continuing NVIDIA’s consistent innovation in networking, ConnectX-6 Lx provides agility and efficiency at every scale. ConnectX-6 Lx delivers cutting-edge 25GbE performance and security for uncompromising data centers.

ConnectX-6 Lx SmartNICs are available in several form factors, including low-profile PCIe and OCP 3.0 cards with SFP28 connectors for 10/25GbE applications, or QSFP28 for 50GbE applications. Low-profile PCIe cards are available with tall and short brackets.

Please refer to Feature and Benefits for more details.

NVIDIA ConnectX-6 Lx PCIe HHHL Cards – 25GbE Cards

MCX631102AS-ADAT

Part Number

MCX631102AN-ADAT

MCX631102AE-ADAT

MCX631102AC-ADAT

MCX631102AS-ADAT

Form Factor/Dimensions

Size: 3.79in. x 2.71in (96.30mm x 68.90mm)

Data Transmission Rate

Ethernet: 1/10/25 Gb/s

Network Connector Type

Dual-port SFP28

PCIe Interface

PCIe Gen 4.0 SERDES @ 16.0GT/s x8

Crypto (IPsec Offload)

-

-

Secure Boot

-

-

Adapter IC Part Number

MT28942A0-YCCF-AE

MT28942A0-YCCF-AE

MT28942A0-YCCF-AE

MT28942A0-YCCF-AE

RoHS

RoHS Compliant

NVIDIA ConnectX-6 Lx PCIe HHHL Cards– 50GbE Cards

Part Number

MCX631105AN-GDAT

MCX631105AE-GDAT

MCX631105AC-GDAT

Form Factor/Dimensions

Size: 3.79in. x 2.71in (96.30mm x 68.90mm)

Data Transmission Rate

Ethernet: 1/10/25/50 Gb/s

Network Connector Type

Single-port QSFP28

PCIe Interface

PCIe Gen 4.0 SERDES @ 8.0GT/s x8

Crypto (IPsec Offload)

-

Secure Boot

-

-

Adapter IC Part Number

MT28942A0-YCCF-AE

MT28942A0-YCCF-AE

MT28942A0-YCCF-AE

RoHS

RoHS Compliant

For more detailed information, see Specifications.

Warning

This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.

Feature

Description

PCI Express (PCIe)

PCIe Gen 4.0 SERDES@16.0GT/s through x8 Edge Connector

Up to 50Gb/s Ethernet

NVIDIA adapters comply with the following IEEE 802.3 standards:

50GbE / 25GbE / 10GbE / 1GbE

– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3ap based auto-negotiation and KR startup
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)

Memory

  • FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus (Note: A ddress 0x58 is reserved.)

  • SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by Macronix)

Overlay Networks

In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. NVIDIA ConnectX-6 Lx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

RDMA over Converged Ethernet (RoCE)

NVIDIA ConnectX-6 Lx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities, as well as NVIDIA ConnectX-6 Lx, advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

NVIDIA PeerDirect ®

NVIDIA PeerDirect ® communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. NVIDIA ConnectX-6 Lx advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU Offload

Adapter functionality enables reduced CPU overhead leaving more CPU resources available for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation

Quality of Service (QoS)

Support for port-based Quality of Service enabling various application requirements for latency and SLA.

Hardware-based I/O Virtualization

NVIDIA ConnectX-6 Lx provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server.

Storage Acceleration

A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage

• RDMA for high-performance storage access
• NVMe over Fabric offloads for target machine

SR-IOV

NVIDIA ConnectX-6 Lx SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.

High-Performance Accelerations

• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing

Connection tracking

ConnecX-6 Lx ASAP² connection-tracking hardware offload accelerates L4 firewall performance.

IPSec Offload (in selected OPNs)

ConnectX-6 Lx adapters offer a range of advanced built-in capabilities that bring infrastructure security down to every endpoint with unprecedented performance and scalability. ConnectX-6 Lx offers IPsec inline encryption/decryption acceleration to offload the CPU and run the entire data path encryption or decryption in the NIC hardware while maintaining line rate.

Secure Boot (Hardware Root Of Trust)

ConnectX-6 Lx delivers supply chain protection with hardware Root-of-Trust (RoT) for Secure Boot as well as Secure Firmware Update using RSA cryptography and cloning protection,

via a device-unique key, to guarantee firmware authenticity

  • RHEL/CentOS

  • Windows

  • FreeBSD

  • VMware

  • OpenFabrics Enterprise Distribution (OFED)

  • OpenFabrics Windows Distribution (WinOF-2)

  • Interoperable with 1/10/25/50 Gb/s Ethernet switches

  • Passive copper cable with ESD protection

  • Powered connectors for optical and active cable support

NVIDIA ConnectX-6 Lx technology maintains support for manageability through a BMC. NVIDIA ConnectX-6 Lx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

© Copyright 2023, NVIDIA. Last updated on Aug 31, 2023.