Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 integrated circuit device for OCP Spec 2.0. These adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and High-Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
ConnectX-5 Ex Ethernet Adapter Cards
Model | ConnectX-5 Ex Cards for OCP Spec 2.0 | |
Part Number | MCX546A-BCAN | MCX546A-CDAN |
Ethernet Data Rate | 10/25/40 Gb/s | 10/25/40/50/100 Gb/s |
Host Management | - | ✓ |
Network Connector Type | Dual-port QSFP28 | Dual-port QSFP28 |
OCP Spec 2.0 Type | ||
PCI Express Connectors | PCIe Gen 3.0 x16 (a), (b) | PCIe Gen 3.0 / 4.0 x16 (a), (b) |
Dimensions | 3.07 in. x 4.33 in (78.00mm x 110.05mm) | |
RoHS | RoHS Compliant | |
IC Part Number | MT28808A0-FCCF-EV | |
Device ID | 4121 for Physical Function (PF) | |
a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead), and can support 200Gb/s when both network ports of the card run at 100Gb/s.
b. The Gen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested when servers with Gen 4.0 support become available.
ConnectX-5 Ethernet Adapter Cards
Model | ConnectX-5 Cards for OCP Spec 2.0 | ||||||
Part Number | MCX542A-ACAN | MCX542B-ACAN(a) | MCX542B-ACUN | MCX545B-GCUN | MCX545A-CCAN | MCX545A-CCUN | MCX545B-CCUN |
Network Connector Type | Dual-port SFP28 | Dual-port SFP28 | Dual-port SFP28 | Single-port QSFP28 | Single-port QSFP28 | Single-port QSFP28 | Single-port QSFP28 |
OCP Spec 2.0 Type | |||||||
Ethernet Data Rate | 10/25 Gb/s | 10/25/40/50 Gb/s | 10/25/40/50/100 Gb/s | 10/25/40/50/100 Gb/s | 10/25/40/50/100 Gb/s | ||
PCI Express Connectors | PCIe Gen 3.0 x16 | PCIe Gen 3.0 x8 | PCIe Gen 3.0 x8 | PCIe Gen 3.0 x16 | PCIe Gen 3.0 x16 | PCIe Gen 3.0 x16 | PCIe Gen 3.0 x16 |
Halogen Free | ✓ | ✓ | ✓ | - | - | - | - |
Host Management | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
UEFI | - | - | ✓ | ✓ | - | ✓ | ✓ |
Dimensions | 3.07 in. x 4.33 in (78.00mm x 110.05mm) | 2.67 in. x 4.33 in (68.00mm x 110.05mm) | 2.67 in. x 4.33 in (68.00mm x 110.05mm) | 2.67 in. x 4.33 in (68.00mm x 110.05mm) | 3.07 in. x 4.33 in (78.00mm x 110.05mm) | 3.07 in. x 4.33 in (78.00mm x 110.05mm) | 3.07 in. x 4.33 in (78.00mm x 110.05mm) |
RoHS | RoHS Compliant | ||||||
IC Part Number | MT28808A0-FCCF-EV | ||||||
Device ID | 4119 for Physical Function (PF) | ||||||
a. NVIDIA recommends populating MCX542B-ACAN in a standard PCIe x8 OCP connector which exposes PCIe lanes in a straight manner.
In case the OCP slot exposes PCIe lanes in a reversed manner, MCX542B-ACAN supports automatic lane reversal with FW image from April 2019 release and above.
For more detailed information see Specifications.
OCP Spec 2.0 Type 1 Stacking Height - Single-port Card
This section applies to MCX545B-CCUN and MCX545B-GCUN.
The single port 100Gb/s Ethernet adapter card complies with OCP Spec 2.0 Type 1 with 8mm stacking height.
OCP Spec 2.0 Type 1 Stacking Height - Dual-port Card
This section applies to MCX542A-ACAN and MCX542B-ACAN.
The dual port 10/25Gb/s Ethernet adapter card comply with OCP Spec 2.0 Type 1 with 8mm stacking height.
OCP Spec 2.0 Type 2 Stacking Height - Single-port Card
This section applies to MCX545A-CCAN and MCX545A-CCUN.
The single-port 100Gb/s adapter card follows OCP Spec 2.0 Type 2 with 12mm stacking height.
OCP Spec 2.0 Type 2 Stacking Height - Dual-port Card
This section applies to MCX546A-CDAN and MCX546A-BCAN.
The dual-port 100Gb/s adapter card follows OCP Spec 2.0 Type 2 with 12mm stacking height.
This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.
Feature | Description |
PCI Express (PCIe) | Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge connector. Gen 1.1 and 2.0 compatible. |
Up to 25 Gigabit Ethernet | NVIDIA adapters comply with the following IEEE 802.3 standards: • 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE |
Memory |
|
Overlay Networks | In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. |
RDMA and RDMA over Converged Ethernet (RoCE) | ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks. |
NVIDIA PeerDirect™ | PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-5 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes. |
CPU Offload | Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks. |
Quality of Service (QoS) | Support for port-based Quality of Service enabling various application requirements for latency and SLA. |
Hardware-based I/O Virtualization | ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. |
Storage Acceleration | A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage RDMA for high-performance storage access. |
SR-IOV | ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server. |
NC-SI | The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over SMBus and MCTP over PCIe - Baseboard Management Controller interface. |
High-Performance Accelerations | • Tag Matching and Rendezvous Offloads |
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support