Introduction
The NVIDIA ConnectX-7 family of network adapters supports both the InfiniBand and Ethernet protocols. It enables a wide range of smart, scalable, and feature-rich networking solutions that address traditional enterprise needs up to the world’s most demanding AI, scientific computing, and hyperscale cloud data center workloads.
ConnectX-7 network adapters are offered in two form factors and various flavors : stand-up PCIe and Open Compute Project (OCP) Spec 3.0 cards. This user manual covers the PCIe stand-up cards, for the OCP 3.0 cards, please refer to NVIDIA ConnectX-7 Cards for OCP Spec 3.0 User Manual.
Make sure to use a PCIe slot capable of supplying the required power and airflow to the ConnectX-7, as stated in the Specifications chapter.
PCIe x16 Stand-up Adapter Cards
ConnectX-7 HCAs are available in various configurations; Single-port 400Gb/s or 200Gb/s, with octal small form-factor pluggable (OSFP) connectors or Dual-port 100 or 200Gb/s with quad small form-factor pluggable (QSFP112) connectors on PCIe standup half-height, half-length (HHHL) form factor, with options for NVIDIA Socket Direct. Also available, Dual-port 50/25 GbE with quad small form-factor pluggable (SFP56) connectors on PCIe standup full-height, half-length (FHHL) form factor, with timing capabilities.
ConnectX-7 cards can either support both InfiniBand and Ethernet, or Ethernet only, as described in the below table. The inclusive list of OPNs is available here.
ConnectX-7 adapter cards with OSFP form factor only support RHS (Riding Heat Sink) cage.
Supported Protocols |
Port Type |
Supported Speed |
Ethernet Only Card |
Dual-port QSFP112 |
|
Quad-port SFP56 |
|
|
InfiniBand and Ethernet Cards |
Single-port OSFP |
|
Socket Direct Ready Cards
The Socket Direct technology offers improved performance to dual-socket servers by enabling direct access from each CPU in a dual-socket server to the network through its dedicated PCIe interface.
NVIDIA offers ConnectX-7 Socket Direct adapter cards, which enable 400Gb/s or 200Gb/s connectivity, and also for servers with PCIe Gen 4.0 capability. The adapter’s 32-lane PCIe bus is split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16 Auxiliary PCIe Connection card. The two cards should be installed into two PCIe x16 slots and connected using two Cabline SA-II Plus harnesses.
To use this card in the Socket-Direct configuration, please order the additional PCIe Auxiliary Card kit according to the desired harness length. Cards that support socket direct can function as separate x16 PCIe cards.
Socket Direct cards can support both InfiniBand and Ethernet, or InfiniBand only, as described below.
Supported Protocols |
Port Type |
Supported Speed |
InfiniBand Only |
Single-port OSFP |
|
InfiniBand and Ethernet |
Dual-port QSFP112 |
|
Single-port QSFP112 |
|
For more information on the passive PCIe Auxiliary kit, please refer to PCIe Auxiliary Card Kit.
Item |
Description |
PCI Express slot |
In PCIe x16 Configuration PCIe Gen 5.0 (32GT/s) through x16 edge connector. In Socket Direct Configuration (2x PCIe x16)
|
System Power Supply |
Refer to Specifications |
Operating System |
|
Connectivity |
|
Category |
Qty |
Item |
Cards |
1 |
ConnectX-7 adapter card |
Accessories |
1 |
Adapter card short bracket |
1 |
Adapter card tall bracket (shipped assembled on the card) |
Make sure to use a PCIe slot capable of supplying the required power and airflow to the ConnectX-7 cards as stated in the Specifications chapter.
This section describes hardware features and capabilities. Please refer to the relevant driver and firmware release notes for feature availability.
Feature | Descriptiom | |||||||||||||||||||||||||||||
PCI Express (PCIe) |
| |||||||||||||||||||||||||||||
InfiniBand Architecture Specification v1.5 compliant |
InfiniBand Network Protocols and Rates:
| |||||||||||||||||||||||||||||
Up to 400 Gigabit Ethernet |
400GbE / 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE
| |||||||||||||||||||||||||||||
Memory Components |
| |||||||||||||||||||||||||||||
Overlay Networks |
In order to better scale their networks, datacenter operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-7 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol. | |||||||||||||||||||||||||||||
Quality of Service (QoS) |
Support for port-based Quality of Service enabling various application requirements for latency and SLA. | |||||||||||||||||||||||||||||
Hardware-based I/O Virtualization |
ConnectX-7 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server. | |||||||||||||||||||||||||||||
Storage Acceleration |
| |||||||||||||||||||||||||||||
SR-IOV |
ConnectX-7 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server. | |||||||||||||||||||||||||||||
High-Performance Accelerations |
| |||||||||||||||||||||||||||||
RDMA Message Rate |
330-370 million messages per second. | |||||||||||||||||||||||||||||
Secure Boot |
The secure boot process assures booting of authentic firmware/software that is intended to run on ConnectX-7. This is achieved using cryptographic primitives using asymmetric cryptography. ConnectX-7 supports several cryptographic functions in its HW Root-of-Trust (RoT) that has its key stored in on-chip FUSES. | |||||||||||||||||||||||||||||
Secure Firmware Update |
For further information, refer to the MFT User Manual. | |||||||||||||||||||||||||||||
Advanced storage capabilities |
Block-level encryption and checksum offloads. | |||||||||||||||||||||||||||||
Host Management |
| |||||||||||||||||||||||||||||
Accurate timing |
ConnectX-7 PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with a PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal, and the timestamp will be sent to the SW. | |||||||||||||||||||||||||||||
RDMA and RDMA over Converged Ethernet (RoCE) |
ConnectX-7, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over InfiniBand and Ethernet networks. Leveraging datacenter bridging (DCB) capabilities as well as ConnectX-7 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks. | |||||||||||||||||||||||||||||
NVIDIA PeerDirect™ |
PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-7 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes. | |||||||||||||||||||||||||||||
CPU Offload |
| |||||||||||||||||||||||||||||
PPS In/Out SMAs |
NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that allows ConnectX-7 to achieve sub 20u Sec accuracy and also offers many timing-related functions such as time-triggered scheduling or time-based SND accelerations (time-based ASAP²). Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN) compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and boundary clock. ConnectX-7 PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal, and the timestamp will be sent to the SW. The SyncE cards also includes an improved holdover to meet ITU-T G.8273.2 class C. |