Connecting NDR Components Together

This section discusses the options for interconnecting NDR switches and endpoints.

NDR Copper Connectivity

The NDR copper cable family is shown in Figure 18. It includes both DAC and ACC. DAC requires no port power but has a limited reach; ACC draws port power to amplify the signal—approximately 1.5 Watts per connector– and extend the reach. Copper cables are mainly used within a rack, or between adjacent racks.

Figure 18. Copper connectivity

Switch-to-switch cable assemblies have a finned OSFP connector at each end with a pair of copper cables between them. The two cables are logically independent InfiniBand links.

Cable assemblies to connect a switch to NDR adapters have a finned OSFP connector at the switch end and two copper cables, each ending in a flat OSFP connector. In the case of DPUs, where space is at a premium, the two copper cables end in QSFP112 connectors.

When the adapters or DPUs have ND200 ports (200 Gbps), each OSFP connector at the switch end splits into dual copper cables that use two data lanes instead of four. Four adapters or four DPUs can be connected. For DPUs, the cables end in QSFP112 connectors, otherwise they end in flat OSFP connectors.

Copper cables to connect NDR switches to HDR and EDR are discussed in NDR Connections to HDR and EDR.

NDR Switch-to-Switch Fiber Connectivity

Three options to interconnect switches using MPO/APC fiber are shown in Figure 19.

Figure 19. Switch-to-switch fiber

For single-mode fiber up to 30m, pairs of ‘NL’ OSFP transceivers can be used. Up to 100m can be achieved using ‘NS’ transceivers, which cost more and draw more power than ‘NS’ models. An ‘NL’ transceiver port can be connected to an ‘NS’ port but the maximum distance will be 30m.

Multi-mode fiber and multi-mode OSFP transceivers can connect switches up to 50m apart. This is the least expensive fiber option and requires the least port power.

Fibers in a pair can have different lengths. Cables that are not MPO/APC, such as those used for HDR and EDR, will not work. Multi-mode and single-mode transceivers will not interoperate.

NDR Switch-to-Endpoint Fiber Connectivity

There are multiple fiber options to connect switches to adapters and DPUs. Available configurations differ between single-mode and multi-mode. Single-mode is available for adapter connections reaching up to 30m, but not DPUs, as shown in Figure 20.

Figure 20. Single-mode fiber to adapters

NDR adapters are connected using OSFP finned twin ‘NL’ transceivers on the switch end, up to 30m of single-mode MPO/APC fiber, and OSFP flat single-mode transceivers at the adapter ports.

For NDR200 adapters, twin two-way single-mode splitters are used with 200 Gbps OSFP flat single-mode transceivers at the adapter. Each of the two splitters can be a different length. As usual, the physical fiber split occurs near the shared MPO connector.

Multi-mode fiber offers up to 50m connections to adapters and to DPUs. See Figure 21.

Figure 21. Multi-mode fiber to adapters and DPUs

Multi-mode options for NDR and NDR200 adapters are like single-mode. Transceiver form factors at the switch and at the adapter ports are the same, using multi-mode versions. The maximum distance is 50m and less port power is needed.

Multi-mode also offers DPU connectivity, where the DPU transceiver form factor is QSFP112. NDR and NDR200 dual-splitter connections can be up to 50m long.

Each of the two splitters can be a different length. The multi-mode splitter family is the same for DPU and adapter links.

NDR Connections to HDR and EDR

Cables for interoperability with HDR and EDR are shown in Figure 22. DAC cables are used for distances up to 2m and AOC cables provide up to 30m. These cables can be used to connect an NDR fabric to previous generation switches and adapters and can be mixed on the same NDR switch with NDR copper and optical cables.

Figure 22. NDR to HDR and EDR

The upper portion of Figure 22 shows solutions for HDR interoperability. All options use an OSFP twin connector at the switch end and the appropriate variant of QSFP for the other legs. The physical split is located close to the OSFP.

The lower portion of the Figure covers interoperability with EDR. EDR cannot be split so the options are fewer. Recall that 100 Gbps EDR and 100 Gbps HDR100 are not the same: EDR is four lanes of 25 Gbps and HDR100 is two lanes of 50 Gbps.

Note that there are no ACC cables, and no non-split cables. There is no easy way to use fiber to connect NDR to HDR or EDR because previous InfiniBand optics and fiber cables do not support MPO/APC.