NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual
NVIDIA ConnectX-6 InfiniBand_Ethernet Adapter Cards for OCP Spec 3.0 User Manual

Introduction

This is the User Guide for Ethernet adapter cards based on the ConnectX®-6 InfiniBand/Ethernet integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and High-Performance Computing environments.

The following provides the ordering part number, port speed, number of ports, and PCI Express speed.

Important Note:

ConnectX-6 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA specifications and setups as defined in document XXX, as the OCP spec 3.0 available at that time did not contain any S&V definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is in the midst of retesting these cards to comply with OCP spec 3.0.

Model

Cards for OCP Spec 3.0

Part Number

MCX653435A-HDAI

MCX653436A-HDAI

MCX653435A-HDAE(a)

MCX653436A-HDAB

MCX653435A-EDAI

Host Management

Network Connector Type

Single-port QSFP56

Dual-port QSFP56

Single-port QSFP56

Dual-port QSFP56

Single-port QSFP56

Data Rate

InfiniBand

SDR/DDR/QDR/FDR/EDR/HDR100/HDR

SDR/DDR/QDR/FDR/EDR/HDR100

Ethernet

10/25/40/50/100/200 Gb/s

10/25/40/50/100 Gb/s

PCI Express Connectors

PCIe Gen 3.0 / 4.0 x16

SERDES @ 8.0GT/s / 16.0GT/s

Dimensions

Small Form Factor (SFF) OCP 3.0

4.52 in. x 2.99 in (115.00mm x 76.00mm)

Retention Mechanism

Internal Lock

Ejector Latch

Pull Tab (Thumbscrew)

Internal Lock

RoHS

RoHS Compliant

IC Part Number

MT28908A0-XCCF-HVM

MT28908A0-XCCF-HVM

MT28908A0-XCCF-HVM

MT28908A0-XCCF-HVM

MT28908A0-XCCF-HVM

Device ID

4123 for Physical Function (PF)

4124 for Virtual Function (VF)

a. MCX653435A-HDAE - The Gen 4.0 card has been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested when servers with Gen 4.0 support become available.

For more detailed information see Specifications.

Warning

This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.

Feature

Description

PCI Express (PCIe)

Uses PCIe Gen 3.0 (8GT/s) or Gen 4.0 (16GT/s) through x16 edge connector.

200Gb/s InfiniBand/Ethernet Adapter

ConnectX-6 offers the highest throughput InfiniBand/Ethernet adapter, supporting HDR 200Gb/s InfiniBand and 200Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate seamlessly over any converged network leveraging a consolidated software stack.

InfiniBand Architecture Specification v1.3 compliant

ConnectX-6 delivers low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. ConnectX-6 is InfiniBand Architecture Specification v1.3 compliant.

InfiniBand HDR

A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 53.125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 200Gb/s.

InfiniBand HDR100

A standard InfiniBand data rate, where each lane of a 2X port runs a bit rate of 53.125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/s.

Up to 200 Gigabit Ethernet

NVIDIA adapters comply with the following IEEE 802.3 standards:

• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet

• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes

• IEEE 802.3ba 40 Gigabit Ethernet

• IEEE 802.3by 25 Gigabit Ethernet

• IEEE 802.3ae 10 Gigabit Ethernet

• IEEE 802.3ap based auto-negotiation and KR startup

• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)

• IEEE 802.3ad, 802.1AX Link Aggregation

• IEEE 802.1Q, 802.1P VLAN tags and priority

• IEEE 802.1Qau (QCN)

• Congestion Notification

• IEEE 802.1Qaz (ETS)

• IEEE 802.1Qbb (PFC)

• IEEE 802.1Qbg

• IEEE 1588v2

• Jumbo frame support (9.6KB)

InfiniBand EDR

A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 25.78125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/s.

Memory Components

  • EEPROM - The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. (Note: A ddress 0x58 is reserved)

  • SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by Macronix)

Overlay Networks

In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

RDMA and RDMA over Converged Ethernet (RoCE)

ConnectX-6, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-6 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

NVIDIA PeerDirect™

PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-6 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU Offload

Adapter functionality enables reduced CPU overhead leaving more CPU resources available for computation tasks.

Open vSwitch (OVS) offload using ASAP2(TM)

• Flexible match-action flow tables

• Tunneling encapsulation/decapsulation

Quality of Service (QoS)

Support for port-based Quality of Service enabling various application requirements for latency and SLA.

Hardware-based I/O Virtualization

ConnectX-6 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server.

Storage Acceleration

A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage RDMA for high-performance storage access.

• NVMe over Fabric offloads for the target machine

SR-IOV

ConnectX-6 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.

NC-SI

The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over SMBus and MCTP over PCIe - Baseboard Management Controller interface.

High-Performance Accelerations

• Tag Matching and Rendezvous Offloads

• Adaptive Routing on Reliable Transport

• Burst Buffer Offloads for Background Checkpointing

Wake-on-LAN (WoL)

The adapter supported Wake-on-LAN (WoL), a computer networking standard that allows an adapter to be turned on or awakened by a network message.

In STBY mode, only port0 is available.

Reset-on-LAN (RoL)

Supported

  • RHEL/CentOS

  • Windows

  • FreeBSD

  • VMware

  • OpenFabrics Enterprise Distribution (OFED)

  • OpenFabrics Windows Distribution (WinOF-2)

  • Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches

  • Passive copper cable with ESD protection

  • Powered connectors for optical and active cable support

© Copyright 2023, NVIDIA. Last updated on Nov 13, 2023.