The twin-port transceiver is a key innovation for expanding the 400G NDR InfiniBand Quantum-2 and 400GbE Spectrum-4 Ethernet (400G IB/EN) switch capabilities. The line rate is 400Gb/s for both 400GbE Ethernet and NDR InfiniBand based on the 100G-PAM4 modulation. The switches use Twin port OSFP cages supporting two transceiver engines in a single OSFP form-factor plug creating 800Gb/s electrical to the switch and 2x400G optics using two MPO/-12/APC optical connectors. Both switches use the same Twin port, 2x400G OSFP plugs for transceivers, copper DACs and ACCs, and are only used in Quantum-2 and Spectrum-4 OSFP air-cooled switches.
Twin-port devices enable several unique configurations to connect switches, adapters and DPUs. Switches maintain separate protocols. The MMA4Z00-NS transceiver includes both InfiniBand and Ethernet protocols which is activated depending on the switch protocol the transceiver used.
The twin-ports enable several unique configurations to connect switches to switches, ConnectX-7 adapters, and BlueField-3 DPUs.
The Twin port OSFP uses two, 4-channel MPO-12/APC optical connectors with two 4-channel fiber cables. These can link to a single port 400G OSFP or QSFP112 transceivers used in ConnectX-7 adapters and/or BlueField-3 DPUs.
The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112. Both ConnectX-7 and BlueField-3 devices can be used with Twin port OSFP transceivers at the same time with their respective form-factor type.
Twin-port multimode OSFP transceivers remain at 15 Watts for all configurations linking OSFP switches, OSFP and QSFP112 adapters, and DPUs simultaneously.
Linking Twin-port transceivers with 1:2 fiber splitter cables to 400G transceivers automatically creates 200Gb/s transceivers by activating only 2-channels and automatically reduces power consumption in the 400G transceiver from 8 to 5.5 Watts.
Twin-Port transceivers require ordering two fibers at specific lengths required
Each fiber cable can each be different lengths. Both fibers need to be same type: both straight or both splitters, and not mixed.
Both fibers should be the approximately same length to avoid inducing different latency delays in the fibers (4.5ns/meter).
Straight and splitter fibers cannot be used at the same time in Twin port OSFP transceivers.
NVIDIA supplies multimode, crossover, straight fiber cables up to 100-meters straight and 50-meters for splitters that enable linking transceivers directly together.
All combinations of Twin port 2x400G OSFP, 400G single port OSFP/QSFP112, ConnectX-7 and BlueField-3 contains both InfiniBand and Ethernet protocols which is activated upon inserting into an InfiniBand or Ethernet switch. This enables one set of cables, transceivers, adapters and DPUs to have multiple uses in a network – especially in DGX systems where low-latency InfiniBand switching may be used for GPU-to-GPU networking and Ethernet switching systems for storage and cluster communication.
The use cases include:
Switch-to-switch at 800Gb/s or to two switches at 400Gb/s
Switch-to-two combinations of 400G ConnectX-7 BlueField-3
Switch-to-four combinations of 200G ConnectX-7 and/or BlueField-3
Switch linked to the DGX H100 “Viking” CPU chassis to Cedar-7 complex
Switch-to-switch at 800Gb/s or to two switches at 400Gb/s
Two OSFP-based switches can be linked together at 800G (2x400G) using two Twin port OSFP transceivers and two straight, multimode fiber cables (MFP7E10-Nxxx) up to 50-meters. Also, the two fiber cables can be routed to two different switches forming two 400Gb/s links. The extra Twin port OSFP ports can be then routed to more switches.
Switch-to-two combinations of 400G ConnectX-7 and/or BlueField-3
A Twin port OSFP transceiver using two, straight fiber cables can support up to two adapter and/or DPU combinations. Each of the two, 4-channel fiber cables (MFP7E10-N0xx) can link to a 400G transceiver up to 50-meters in either OSFP (MMA4Z00-NS400) or QSFP112 (MMA1Z00-NS400) form-factor. The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors and power consumption is 8 Watts.
- Only ConnectX-7/OSFPs supports single port OSFP.
- QSFP112 form-factor is used in ConnectX-7/QSFP112s and/or BlueField-3/QSFP112 DPUs.Any combination of ConnectX-7 and BlueField-3 using OSFP or QSFP112 can be used at the same time on Twin port OSFP transceivers.
Switch-to-four combinations of 200G ConnectX-7 and/or BlueField-3
A Twin port OSFP transceiver using two, 1:2 fiber splitter cables can support up to four adapter and/or DPU combinations. Each of the two, 4-channel 1:2 fiber splitter cables (MFP7E20-N0xx) can link to a 400G transceiver up to 50-meters in either OSFP (MMA4Z00-NS400) or QSFP112 (MMA1Z00-NS400) form-factor. The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors. The two-fiber channel ends only activate two of the lanes in the 400G transceiver creating a 200G device and automatically reduces the power consumption of only the 400G transceivers from 8 Watts to 5.5 Watts. Twin port OSFP power consumption remains at 15 Watts.
- Only ConnectX-7/OSFPs can accept single port OSFP.
- QSFP112 form-factor is used in ConnectX-7/QSFP112s and/or BlueField-3/QSFP112 DPUs.
- Any combination of ConnectX-7 types and BlueField-3 can be used at the same time on Twin port OSFP transceivers.
Switch-to-DGX H100 GPU Systems
The DGX-H100 contains eight “Hopper” H100 GPUs in the top chassis section, two CPUs, storage, and InfiniBand and/or Ethernet networking in the bottom server section. This contains eight 400Gb/s ConnectX-7 ICs mounted on two mezzanine boards called “Cedar-7” cards for GPU-to-GPU InfiniBand or Ethernet networking. The cards I/Os are routed internally to four 800G Twin-port OSFP cages with internal riding heat sinks on top of the cages mounted on the front panel. This requires the use of flat-top transceivers, ACCs, and DACs in the DGX H100. The 400G IB/EN switches require finned-top 2x400G transceivers for additional cooling due to the reduced air flow inlets in the switches.
The Cedar-7-to-Switch links can be either single mode or multimode optics or ACC active copper cables and in InfiniBand or Ethernet.
Each Twin-port 2x400G transceiver provides two 400G ConnectX-7 links from the DGX to the Quantum-2 or Spectrum-4 switch. This reduces the ConnectX-7 card redundancy, complexity, and the number of transceivers compared to the DGX A100, which uses 8 separate HCAs and 8 transceivers or AOCs and two additional ConnectX-6s for InfiniBand or Ethernet Storage.
Additionally, for traditional networking to storage, clusters, and management, the DGX-H100 also supports up to four ConnectX-7 and/or two BlueField-3 DPUs in InfiniBand and/or Ethernet for storage I/O, and additional networking using 400G or 200G with OSFP or QSFP112 devices. These PCIe card slots are located on both sides of the OSFP GPU cages and use separate cables and/or transceivers.
