The Twin-port transceiver is a key innovation for expanding the 400G NDR InfiniBand Quantum-2 and 400GbE Spectrum-4 Ethernet (400G IB/EN) switch capabilities. The line rate is 400Gb/s for both 400GbE Ethernet and NDR InfiniBand based on the 100G-PAM4 modulation. The switches use Twin port OSFP cages supporting two transceiver engines in a single OSFP form-factor plug creating 800Gb/s electrical to the switch and 2x400G optics using two MPO/-12/APC optical connectors.
Both switches and use the same Twin port, 2x400G OSFP plugs for transceivers, copper DACs and ACCs and are only used in Quantum-2 and Spectrum-4 OSFP air-cooled switches.
Twin-port devices enable several unique configurations to connect switches together and to adapters and DPUs. Switches maintain separate protocols. The MMS4X00-NS transceiver includes both InfiniBand and Ethernet protocols which is activated depending on the switch protocol the transceiver is inserted into.
The Twin-ports enable several unique configurations to connect switches to switches, ConnectX-7 adapters and BlueField-3 DPUs.
The Twin port OSFP uses two, 4-channel MPO-12/APC optical connectors with two 4-channel fiber cables. These can link to single port 400G OSFP or QSFP112 form-factor transceivers used in ConnectX-7 adapters and/or BlueField-3 DPUs.
The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors. Both ConnectX-7 and BlueField-3 devices can be used with Twin port OSFP transceivers at the same time with their respective form-factor type.
Twin-port single mode OSFP transceivers remain at 17 Watts for all configurations linking OSFP switches, OSFP and QSFP112 adapters, and DPUs simultaneously.
Linking Twin-port transceivers with 1:2 fiber splitter cables to 400G transceivers automatically creates 200Gb/s transceivers by activating only 2-channels and automatically reduces power consumption in the 400G transceiver from 9 to 6.5 Watts.
Twin-Port transceivers require ordering two fibers at specific lengths required
Each fiber cable can each be different lengths. Both fibers need to be same type: both straight or both splitters and not mixed.
Both fibers should be the approximately same length to avoid inducing different latency delays in the fibers (4.5ns/meter).
Straight and splitter fibers cannot be used at the same time in Twin port OSFP transceivers.
The combinations of Twin port 2x400G OSFP, 400G single port OSFP/QSFP112, ConnectX-7 and BlueField-3 all contains both InfiniBand and Ethernet protocols which is activated upon inserting into an InfiniBand or Ethernet switch. This enables one set of cables, transceivers, adapters and DPUs to have multiple uses in a network – especially in DGX systems where low-latency InfiniBand switching may be used for GPU-to-GPU networking and Ethernet switching systems for storage and cluster communication.
The use cases include:
Switch-to-switch at 800Gb/s or to two switches at 400Gb/s
Switch-to-two combinations of 400G ConnectX-7 and/or BlueField-3
Switch-to-four combinations of 200G ConnectX-7 and/or BlueField-3
Switch linked to the DGX H100 “Viking” CPU chassis to Cedar-7 complex
Switch-to-switch at 800Gb/s or to two switches at 400Gb/s
This configuration enables transferring 800Gb/s between two switches using one, 17-Watt twin-port, finned-top DR8 transceivers in each switch and two straight fibers cables (MFP7E30-Nxxx) up to 100-meters. A 500-meter version (-NM) is also available for longer switch-to-switch reaches. The two 400G ports can also be routed to separate switches at 400Gb/s each.
Switch-to-two combinations of 400G ConnectX-7 and/or BlueField-3
A Twin port OSFP transceiver using two, straight fiber cables can support up to two adapter and/or DPU combinations. Each of the two, 4-channel fiber cables (MFP7E30) can link to a 400G transceiver up to 100-meters in either OSFP (MMS4X00-NS400) or QSFP112 (MMS1X00-NS400) form-factor. The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors and power consumption is 9 WattsA Twin port OSFP transceiver using two, straight fiber cables can support up to two adapter and/or DPU combinations. Each of the two, 4-channel fiber cables (MFP7E30) can link to a 400G transceiver up to 100-meters in either OSFP (MMS4X00-NS400) or QSFP112 (MMS1X00-NS400) form-factor. The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors and power consumption is 9 Watts.
- Only ConnectX-7/OSFPs supports single port OSFP.
- QSFP112 form-factor is used in ConnectX-7/QSFP112s and/or BlueField-3/QSFP112 DPUs.Any combination of ConnectX-7 and BlueField-3 using OSFP or QSFP112 can be used at the same time on Twin port OSFP transceivers.
Switch-to-four combinations of 200G ConnectX-7 and/or BlueField-3
A Twin port OSFP transceiver using two, 1:2 fiber splitter cables can support up to four adapter and/or DPU combinations. Each of the two, 4-channel 1:2 fiber splitter cables (MFP7E40-N0xx) can link to a 400G transceiver up to 50-meters in either OSFP (MMS4X00-NS400) or QSFP112 (MMS1X00-NS400) form-factor. The electronics, optics and optical connectors are the same for both single port OSFP and QSFP112 form-factors.
The 2-fiber split end (2x100G-PAM4) configuration automatically creates 200G NDR200 transceivers out of the 400G transceiver and reduces its power consumption from 9 to 6.5 Watts. Twin port OSFP power consumption remains at 17 Watts.
- Only ConnectX-7/OSFPs can accept single port OSFP.
- QSFP112 form-factor is used in ConnectX-7/QSFP112s and/or BlueField-3/QSFP112 DPUs.
- The straight fiber max reach is 100-meters, but splitters max reach is 50-meters.
Switch-to-DGX H100 GPU Systems
The DGX-H100 contains eight “Hopper” H100 GPUs in the top chassis section, two CPUs, storage, and InfiniBand and/or Ethernet networking in the bottom server section. This contains eight 400Gb/s ConnectX-7 ICs mounted on two mezzanine boards called “Cedar-7” cards for GPU-to-GPU InfiniBand or Ethernet networking. The cards I/Os are routed internally to four 800G Twin-port OSFP cages with internal riding heat sinks on top of the cages mounted on the front panel. This requires the use of flat-top transceivers, ACCs, and DACs in the DGX H100. The 400G IB/EN switches require finned-top 2x400G transceivers for additional cooling due to the reduced air flow inlets in the switches.
The Cedar-7-to-Switch links can be either single mode or multimode optics or ACC active copper cables and in InfiniBand or Ethernet.
Each Twin-port 2x400G transceiver provides two 400G ConnectX-7 links from the DGX to the Quantum-2 or Spectrum-4 switch. This reduces the ConnectX-7 card redundancy, complexity, and the number of transceivers compared to the DGX A100, which uses 8 separate HCAs and 8 transceivers or AOCs and two additional ConnectX-6s for InfiniBand or Ethernet Storage.
Additionally, for traditional networking to storage, clusters, and management, the DGX-H100 also supports up to four ConnectX-7 and/or two BlueField-3 DPUs in InfiniBand and/or Ethernet for storage I/O, and additional networking using 400G or 200G with OSFP or QSFP112 devices. These PCIe card slots are located on both sides of the OSFP GPU cages and use separate cables and/or transceivers.