You can download a PDF version of the full guide here.
Introduction
Guide to Configuring 100G-PAM4 LinkX Interconnects for Quantum-2 InfiniBand and Spectrum-4 Ethernet OSFP-based Switches with ConnectX-7 Adapters and BlueField-3 DPUs based on twin-port OSFP, OSFP, and QSFP112 Cables and Transceivers
NVIDIA® LinkX® 400Gb/s Cable and Transceiver User Guides provides detailed information, figures, and ordering part numbers to assist in configuring cables and transceivers for use with network switches, BlueField® DPUs, and ConnectX® network adapters for both Ethernet and InfiniBand protocols. This document covers only cables and transceivers based on 100G-PAM4 modulation and a few specific parts for backwards compatibility linking to 50G-PAM4 and 25G-NRZ devices. Other user guides focus on 50G-PAM4 and 25G-NRZ cables and transceivers, switches, and network adapter interconnects.
Interconnect Documents
This guide is designed to be used in conjunction with additional documentation located in docs.nvidia.com/networking > Interconnect. This site is where the following LinkX cables and transceivers documents are provided.
400Gb/s Rate
The 400Gb/s rate cables and transceivers are used for both InfiniBand NDR Quantum-2 QM9700 and 400GbE SN5600 Spectrum™-4 Ethernet OSFP-based switches. The 400Gb/s rate consists of 4-channels of 100G-PAM4 signaling and based on the octal small form-factor plug (OSFP).
- For switches: two 400Gb/s ports are included in one 2x400G OSFP air-cooled switch cage in the switches and called a twin-port finned top OSFP. Since there are two 400G engines in the twin-port OSFP transceivers, this has an 800Gb/s aggregate electrical data rate and may also be referred to as 2x400G optical.
- For PCIe network adapters: ConnectX-7 network adapters and BlueField-3 Data Processing Units (DPUs) use 4-channel, 400Gb/s single-port flat top OSFP or QSFP112 cages with finned heat sinks on top of the board cages. The twin-port OSFP devices cannot be used with adapters and DPUs as the finned top is considerably taller than the single port cage.
- LinkX cables and transceivers tie them all together with copper cables and optical transceivers.
The 400Gb/s Spectrum-3 SN4000 series Ethernet switches are constructed from 8-channels of 50G-PAM4 signal modulation and use the quad small form-factor plug with double density (QSFP-DD) and are covered in another document. QSFP-DD is not used with InfiniBand.
Parts are Described by Use in the Switch
The DAC, ACC, and AOC cable part number descriptors are based on the connector used in the switches, as the devices used in network adapters may consist of multiple connector types such as the single-port OSFP, QSFP112, QSFP56, etc. These plugs or “form-factors” are used to contain optical transceivers and copper cables to form the switching networks linking CPU/GPU compute engines with storage subsystems and to other system clusters in the network. For example, an 800G twin-port OSFP-to-4x 200G QSFP112 splitter copper cable is listed in the parts lists as 800Gb/s twin-port OSFP, not 200Gb/s QSFP112.
Cable Descriptions Denoted by the Switch-side Connector
