Introduction
This is the User Guide for InfiniBand/Ethernet adapter cards based on the ConnectX®-5 integrated circuit device. NVIDIA offers an alternate ConnectX-5 Socket Direct™ card for servers without x16 PCIe slots. The adapter’s 16-lane PCIe bus is split into two 8-lane buses, with one bus accessible through a PCIe x8 edge connector and the other bus through an x8 parallel connector to an Auxiliary PCIe Connection Card. The two cards should be installed into two adjacent PCIe x8 slots and connected using a dedicated harness.
Since the two PCIe cards are installed in two PCIe slots, each card gets a unique PCI Bus and Device number. Each of the PCIe x8 busses sees two network ports; in effect, the two 100Gb/s physical ports of the ConnectX-5 Socket Direct Adapter are viewed as four netdevices by the system, with each netdevice providing a maximum of 50Gb/s data transfer rate. In other words, 100Gb/s on a (physical) adapter port (Port 1 or Port 2) will be split on the PCIe side to 50Gb/s on each of the two PCIe cards (the Adapter Card and the Auxiliary PCIe Connection Cards, respectively, in the above picture).
ConnectX-5 InfiniBand/Ethernet Socket Direct Adapter Cards
Model | ConnectX-5 InfiniBand/Ethernet Adapter Cards | |
Part Number | MCX556M-ECAT-S25 | MCX556M-ECAT-S35A |
Dimensions | Low Profile Adapter Card Size: 6.6 in. x 2.71 in. (167.65mm x 68.90mm) | |
Slim-Line SAS Cable: 9.8 in. x 1.02 in. (250cm x 25.95mm) | Slim-Line SAS Cable: 9.8 in. x 1.02 in. (350cm x 25.95mm) | |
Network Connector Type | 2x QSFP | |
Data Rate | InfiniBand: SDR/DDR/QDR/FDR/EDR | |
PCI Express Connectors | PCIe Gen 3.0 x8; SerDes @ 8.0GT/s | |
RoHS | RoHS Compliant | |
Adapter IC Part Number | MT28808A0-FCCF-EVM | |
Device ID (decimal) | 4119 for Physical Function (PF) and 4120 for Virtual Function (VF) | |
Full performance of the adapter card (100Gb/s) is possible only when installing both the adapter card and the auxiliary PCIe connection card in the dual PCIe x8 connectors.
Please refer to the performance optimization document for further guidance: https://community.nvidia.com/s/article/understanding-pcie-configuration-for-maximum-performance
For more detailed information see Specifications.
This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability.
Feature | Description |
PCI Express (PCIe) | Uses PCIe Gen 3.0 (8GT/s) through two x8 edge connectors. Gen 1.1 and 2.0 compatible. |
100Gb/s InfiniBand/Ethernet Adapter | ConnectX-5 offers the highest throughput Ethernet adapter, supporting EDR 100Gb/s InfiniBand and 100Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate seamlessly over any converged network leveraging a consolidated software stack. |
InfiniBand Architecture Specification v1.3 compliant | ConnectX-5 delivers low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. ConnectX-5 is InfiniBand Architecture Specification v1.3 compliant. |
EDR InfiniBand | A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 25.78125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/s. |
Up to 100 Gigabit Ethernet | NVIDIA adapters comply with the following IEEE 802.3 standards:
|
Memory |
|
Overlay Networks | In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. |
RDMA and RDMA over Converged Ethernet (RoCE) | ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks. |
NVIDIA PeerDirect™ | PeerDirect™ communication provides high efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-5 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes. |
CPU Offload | Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks. |
Open VSwitch (OVS) offload using ASAP2 | • Flexible match-action flow tables |
Quality of Service (QoS) | Support for port-based Quality of Service enabling various application requirements for latency and SLA. |
Hardware-based I/O Virtualization | ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. |
Storage Acceleration | A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage InfiniBand RDMA for high-performance storage access.
|
SR-IOV | ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server. |
High-Performance Accelerations |
|
Supported in OpenFabrics Enterprise Distribution (OFED).
Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Socket Direct technology maintains support for manageability through a BMC. The Socket Direct PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.