NVIDIA ConnectX-5 Ethernet Adapter Cards for OCP Spec 3.0 User Manual
NVIDIA ConnectX-5 Ethernet Adapter Cards for OCP Spec 3.0 User Manual

Introduction

This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and High-Performance Computing environments.

Important Notes:

  1. ConnectX-5 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA specifications and setup, as the OCP spec 3.0 available at that time did not contain any S&V definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is in the midst of retesting these cards to comply with OCP spec 3.0.

  2. In some of the OCP 3.0 cards, the insertion force that is required to install the card into a particular PCI connector may exceed the maximum insertion force that is allowed by the connector’s spec. NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion process.

The following tables provide the ordering part number, port speed, number of ports, and PCI Express speed per ConnectX-5 and ConnectX-5 Ex OPNs.

ConnectX-5 Ex Adapter Cards for OCP 3.0

Part Number

MCX566M-GDAI

MCX565M-CDAI

MCX565M-CDAB

MCX566A-CDAI

MCX566A-CDAB

Ethernet Data Rate

Up to 50Gb/s

Up to 100Gb/s

Up to 100Gb/s

Up to 100Gb/s

Up to 100Gb/s

Network Connector Type

x2 QSFP28 ports

x1 QSFP28 port

x1 QSFP28 port

x2 QSFP28 ports

x2 QSFP28 ports

PCI Express Connectors

PCIe Gen 3.0 / 4.0 x16 (a), (b)
SERDES @ 8.0GT/s / 16.0GT/s

Bracket

Internal-Lock Bracket

Internal-Lock Bracket

Pull-tab Bracket

Internal-Lock Bracket

Pull-tab bracket

Host Management

Multi-Host/ Socket Direct

NA

NA

IC Part Number

MT28804A0-FCCF-CEM

MT28804A0-FCCF-CEM

MT28804A0-FCCF-CEM

MT28808A0-FCCF-CE

MT28808A0-FCCF-CE

Device ID

4121 for Physical Function (PF)
4122 for Virtual Function (VF)

Form Factor Dimensions

Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)

RoHS

RoHS Compliant

ConnectX-5 Adapter Cards for OCP 3.0

Part Number

MCX562A-ACAI

MCX562A-ACAB

MCX566A-CCAI

MCX565A-CCAB

Ethernet Data Rate

Up to 25Gb/s

Up to 25Gb/s

Up to 100Gb/s

Up to 100Gb/s

Network Connector Type

x2 SFP28 ports

x2 SFP28 ports

x2 QSFP28 ports

x1 QSFP28 ports

PCI Express Connectors

PCIe Gen 3.0 x16
SERDES @ 8.0GT/s

Bracket

Internal-Lock Bracket

Pull-tab Bracket

Internal-Lock Bracket

Pull-tab Bracket

Host Management

IC Part Number

MT27808A0-FCCF-AE

MT27808A0-FCCF-AE

MT27808A0-FCCF-CE

MT27808A0-FCCF-CE

Device ID

4119 for Physical Function (PF)
4120 for Virtual Function (VF)

Form Factor Dimensions

Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)

RoHS

RoHS Compliant

a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead), and can support 200Gb/s when both network ports of the card run at 100Gb/s.
b. The Gen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested when servers with Gen 4.0 support become available.

For more detailed information see Specifications.

Warning

This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.

Feature

Description

PCI Express (PCIe)

Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge connector. Gen 1.1 and 2.0 compatible.

Up to 100 Gigabit Ethernet

NVIDIA adapters comply with the following IEEE 802.3 standards:

• 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)

Memory

  • EEPROM - The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus (Note: Address 0x58 is reserved.)

  • SPI - includes 128Mb SPI Flash device (by WINBOND-NUVOTON).

Overlay Networks

In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

RDMA and RDMA over Converged Ethernet (RoCE)

ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

NVIDIA PeerDirect™

PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-5 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation

Quality of Service (QoS)

Support for port-based Quality of Service enabling various application requirements for latency and SLA.

Hardware-based I/O Virtualization

ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server.

Storage Acceleration

A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for target machine

SR-IOV

ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.

NC-SI over RMII

The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be connected to a BMC.

High-Performance Accelerations

• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing

Host Management Technology

NVIDIA’s host management technology for standard and multi-host platforms optimizes board management and power, performance and firmware update management via NC-SI, MCTP over SMBus and MCTP over PCIe, as well as PLDM for Monitor and Control DSP0248 and PLDM for Firmware Update DSP0267.

Multi-Host Technology

NVIDIA Multi-Host™ technology, when enabled, allows multiple hosts to be connected into a single adapter by separating the PCIe interface into multiple and independent interfaces. By Using NVIDIA Multi Host™, ConnectX-5 lowers the total cost of ownership (TCO) in the data center by reducing CAPEX (cables, NICs, and switch port expenses), and by reducing OPEX by cutting down on switch port management and overall power usage. With NVIDIA Multi-Host™ technology powered by a shared buffer architecture, connection-tracking offloads, and RoCE enhancements, ConnectX-5 offers an extremely flexible solution for today’s demanding data center and cloud application.

NVIDIA Socket Direct™

NVIDIA Socket Direct technology brings improved performance to multi-socket servers by enabling direct access from each CPU in a multi-socket server to the network through its dedicated PCIe interface. With this type of configuration, each CPU connects directly to the network; this enables the interconnect to bypass a QPI (UPI) and the other CPU, optimizing performance and improving latency. CPU utilization improves as each CPU handles only its own traffic, and not the traffic from the other CPU. NVIDIA’s OCP 3.0 cards include native support for socket direct technology for multi-socket servers and can support up to 4 CPU sockets.

Wake-on-LAN (WoL)

Supported

Reset-on-Lan (RoL)

Supported

  • RHEL/CentOS

  • Windows

  • FreeBSD

  • VMware

  • OpenFabrics Enterprise Distribution (OFED)

  • OpenFabrics Windows Distribution (WinOF-2)

  • Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches

  • Passive copper cable with ESD protection

  • Powered connectors for optical and active cable support

© Copyright 2023, NVIDIA. Last updated on May 22, 2023.