NVIDIA ConnectX-7 Adapter Cards User Manual

Introduction

The NVIDIA ConnectX-7 family of network adapters supports both the InfiniBand and Ethernet protocols. It enables a wide range of smart, scalable, and feature-rich networking solutions that address traditional enterprise needs up to the world’s most demanding AI, scientific computing, and hyperscale cloud data center workloads.

ConnectX-7 network adapters are offered in two form factors and various flavors : stand-up PCIe and Open Compute Project (OCP) Spec 3.0 cards. This user manual covers the PCIe stand-up cards, for the OCP 3.0 cards, please refer to NVIDIA ConnectX-7 Cards for OCP Spec 3.0 User Manual.

Note

Make sure to use a PCIe slot capable of supplying the required power and airflow to the ConnectX-7, as stated in the Specifications chapter.

PCIe x16 Stand-up Adapter Cards

ConnectX-7 HCAs are available in various configurations; Single-port 400Gb/s or 200Gb/s, with octal small form-factor pluggable (OSFP) connectors or Dual-port 100 or 200Gb/s with quad small form-factor pluggable (QSFP112) connectors on PCIe standup half-height, half-length (HHHL) form factor, with options for NVIDIA Socket Direct. Also available, Dual-port 50/25 GbE with quad small form-factor pluggable (SFP56) connectors on PCIe standup full-height, half-length (FHHL) form factor, with timing capabilities.

ConnectX-7 cards can either support both InfiniBand and Ethernet, or Ethernet only, as described in the below table. The inclusive list of OPNs is available here.

Note

ConnectX-7 adapter cards with OSFP form factor only support RHS (Riding Heat Sink) cage.

Supported Protocols

Port Type

Supported Speed

Ethernet Only Card

Dual-port QSFP112

  • 100GbE

Quad-port SFP56

  • 50/25GbE

InfiniBand and Ethernet Cards

Single-port OSFP

  • NDR 400Gb/s and 400GbE

  • NDR200 200Gb/s and 200GbE


Socket Direct Ready Cards

The Socket Direct technology offers improved performance to dual-socket servers by enabling direct access from each CPU in a dual-socket server to the network through its dedicated PCIe interface.

NVIDIA offers ConnectX-7 Socket Direct adapter cards, which enable 400Gb/s or 200Gb/s connectivity, and also for servers with PCIe Gen 4.0 capability. The adapter’s 32-lane PCIe bus is split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16 Auxiliary PCIe Connection card. The two cards should be installed into two PCIe x16 slots and connected using two Cabline SA-II Plus harnesses.

To use this card in the Socket-Direct configuration, please order the additional PCIe Auxiliary Card kit according to the desired harness length. Cards that support socket direct can function as separate x16 PCIe cards.

Socket Direct cards can support both InfiniBand and Ethernet, or InfiniBand only, as described below.

Supported Protocols

Port Type

Supported Speed

InfiniBand Only

Single-port OSFP

  • NDR 400Gb/s

  • NDR200 200Gb/s

InfiniBand and Ethernet

Dual-port QSFP112

  • NDR200 200Gb/s and 200GbE

Single-port QSFP112

  • NDR 400Gb/s and 400GbE

For more information on the passive PCIe Auxiliary kit, please refer to PCIe Auxiliary Card Kit.

Item

Description

PCI Express slot

In PCIe x16 Configuration

PCIe Gen 5.0 (32GT/s) through x16 edge connector.

In Socket Direct Configuration (2x PCIe x16)

  • PCIe Gen 4.0/5.0 SERDES @16/32GT/s through edge connector

  • PCIe Gen 4.0 SERDES @16GT/s through PCIe Auxiliary Connection Card

System Power Supply

Refer to Specifications

Operating System

  • In-box drivers for major operating systems:

    • Linux: RHEL, Ubuntu

    • Windows

  • Virtualization and containers

    • VMware ESXi (SR-IOV)

    • Kubernetes

  • OpenFabrics Enterprise Distribution (OFED)

  • OpenFabrics Windows Distribution (WinOF-2)

Connectivity

  • Interoperable with 1/10/25/40/50/100/200/400 Gb/s Ethernet switches and SDR/DDR/EDR/HDR100/HDR/NDR200/NDR InfiniBand switches

  • Passive copper cable with ESD protection

  • Powered connectors for optical and active cable support

Category

Qty

Item

Cards

1

ConnectX-7 adapter card

Accessories

1

Adapter card short bracket

1

Adapter card tall bracket (shipped assembled on the card)

Note

Make sure to use a PCIe slot capable of supplying the required power and airflow to the ConnectX-7 cards as stated in the Specifications chapter.

Note

This section describes hardware features and capabilities. Please refer to the relevant driver and firmware release notes for feature availability.

PCI Express (PCIe)

According to the OPN you have purchased, the card uses the following PCIe express interfaces:

  • PCIe x16 configurations:

    PCIe Gen 4.0/5.0 (16GT/s / 32GT/s) through x16 edge connector.

  • 2x PCIe x16 configurations (Socket-Direct):

    PCIe Gen 4.0/5.0 ( SERDES @ 16GT/s / 32GT/s) through x16 edge connector

    PCIe Gen 4.0 SERDES @ 16GT/s through PCIe Auxiliary Connection Card

InfiniBand Architecture Specification v1.5 compliant

ConnectX-7 delivers low latency, high bandwidth, and computing efficiency for high-performance computing (HPC), artificial intelligence (AI), and hyperscale cloud data center applications. ConnectX-7 is InfiniBand Architecture Specification v1.5 compliant.

InfiniBand Network Protocols and Rates:

Protocol

Standard

Rate (Gb/s)

Comments

4x Port

(4 Lanes)

2x Ports

(2 Lanes)

NDR/NDR200

IBTA Vol2 1.5

425

212.5

PAM4 256b/257b encoding and RS-FEC

HDR/HDR100

IBTA Vol2 1.4

212.5

106.25

PAM4 256b/257b encoding and RS-FEC

EDR

IBTA Vol2 1.3.1

103.125

51.5625

NRZ 64b/66b encoding

FDR

IBTA Vol2 1.2

56.25

N/A

NRZ 64b/66b encoding

Up to 400 Gigabit Ethernet

ConnectX-7 adapter cards comply with the following IEEE 802.3 standards:

400GbE / 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE

Protocol

MAC Rate

IEEE802.3ck

100/200/400Gb/s Gigabit Ethernet

(Include ETC enhancement)

IEEE802.3cd

IEEE802.3bs

IEEE802.3cm

IEEE802.3cn

IEEE802.3cu

50/100/200/400Gb/s Gigabit Ethernet

(Include ETC enhancement)

IEEE 802.3bj

IEEE 802.3bm

100 Gigabit Ethernet

IEEE 802.3by

Ethernet Technology Consortium

25/50 Gigabit Ethernet

IEEE 802.3ba

40 Gigabit Ethernet

IEEE 802.3ae

10 Gigabit Ethernet

IEEE 802.3cb

2.5/5 Gigabit Ethernet

(For 2.5: support only 2.5 x1000BASE-X)

IEEE 802.3ap

Based on auto-negotiation and KR startup

IEEE 802.3ad

IEEE 802.1AX

Link Aggregation

IEEE 802.1Q

IEEE 802.1P VLAN tags and priority

IEEE 802.1Qau (QCN)

Congestion Notification

IEEE 802.1Qaz (ETS)

EEE 802.1Qbb (PFC)

IEEE 802.1Qbg

IEEE 1588v2

IEEE 802.1AE (MACSec)

Jumbo frame support (9.6KB)

Memory Components

  • SPI - includes 256Mbit SPI Quad Flash device.

  • FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is 128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. (Note: A ddress 0x58 is reserved.)

Overlay Networks

In order to better scale their networks, datacenter operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-7 effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.

Quality of Service (QoS)

Support for port-based Quality of Service enabling various application requirements for latency and SLA.

Hardware-based I/O Virtualization

ConnectX-7 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server.

Storage Acceleration

A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can leverage:

  • RDMA for high-performance storage access

  • NVMe over Fabric offloads for the target machine

  • NVMe over TCP acceleration

SR-IOV

ConnectX-7 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.

High-Performance Accelerations

  • Collective operations offloads

  • Vector collective operations offloads

  • MPI tag matching

  • MPI_Alltoall offloads

  • Rendezvous protocol offload

RDMA Message Rate

330-370 million messages per second.

Secure Boot

The secure boot process assures booting of authentic firmware/software that is intended to run on ConnectX-7. This is achieved using cryptographic primitives using asymmetric cryptography. ConnectX-7 supports several cryptographic functions in its HW Root-of-Trust (RoT) that has its key stored in on-chip FUSES.

Secure Firmware Update

The Secure firmware update feature enables a device to verify digital signatures of new firmware binaries to ensure that only officially approved versions can be installed from the host, the network, or a Board Management Controller (BMC). The firmware of devices with “secure firmware update” functionality (secure FW), restricts access to specific commands and registers that can be used to modify the firmware binary image on the flash, as well as commands that can jeopardize security in general.

For further information, refer to the MFT User Manual.

Advanced storage capabilities

Block-level encryption and checksum offloads.

Host Management

ConnectX-7 technology maintains support for host manageability through a BMC. ConnectX-7 PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter card. For configuring the adapter for the specific manageability solution in use by the server, please contact NVIDIA Support.

  • Protocols: PLDM, NCSI

  • Transport layer – RBT, MCTP over SMBus and MCTP over PCIe

  • Physical layer: SMBus 2.0 / I2C interface for device control and configuration, PCIe

  • PLDM for Monitor and Control DSP0248

  • PLDM for Firmware Update DSP026

  • IEEE 1149.6

  • Secured FW update

  • FW Recovery

  • NIC reset

  • Monitoring and control

  • Network port settings

  • Boot setting

Accurate timing

NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding PTP profiles. ConnectX-7 incorporates an integrated Hardware Clock (PHC) that allows ConnectX-7 to achieve sub 20u Sec accuracy and also offers many timing-related functions such as time-triggered scheduling or time-based SND accelerations (time-based ASAP²). Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN) compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and boundary clock.

ConnectX-7 PTP solution allows you to run any PTP stack on your host.

With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with a PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal, and the timestamp will be sent to the SW.

RDMA and RDMA over Converged Ethernet (RoCE)

ConnectX-7, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance over InfiniBand and Ethernet networks. Leveraging datacenter bridging (DCB) capabilities as well as ConnectX-7 advanced congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

NVIDIA PeerDirect™

PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-7 advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU Offload

Adapter functionality enables reduced CPU overhead allowing more available CPU for computation tasks.

  • Flexible match-action flow tables

  • Open VSwitch (OVS) offload using ASAP2®

  • Tunneling encapsulation/decapsulation

PPS In/Out SMAs

Applies to MCX713114TC-GEAT only:

NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that allows ConnectX-7 to achieve sub 20u Sec accuracy and also offers many timing-related functions such as time-triggered scheduling or time-based SND accelerations (time-based ASAP²). Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN) compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and boundary clock.

ConnectX-7 PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal, and the timestamp will be sent to the SW. The SyncE cards also includes an improved holdover to meet ITU-T G.8273.2 class C.

© Copyright 2024, NVIDIA. Last updated on Oct 22, 2023.