Setting the InfiniBand Cluster Topology
InfiniBand fabric components can be connected using different topologies and it should be decided before building the cluster.
Fat-Tree is NVIDIA's recommended topology, AI factory should based on rail optimized.
Rail-optimized design means a GPU node with multiple interfaces will put each GPU “rail” (IB network interface) onto a different first level (LEAF) switch for cluster Interconnect. This allows multiple nodes to utilize their internal NVSwitch path to talk across a NIC that is just one switch hop away (instead of having to cross multiple switches, incurring additional latency).
The diagram below shows an example of cluster topology for AI factory (based on rail optimized):
3 Tier Topology
2 Tier Topology
To choose the best cluster planning that fits the cluster needs, please contact NVIDIA Support.
After selecting the InfiniBand interconnect principles, create a PTP excel file to describe the cluster connectivity and to generate the Topology file.
It is imperative to verify that the cluster has been connected according to the cluster planning to ensure cluster maintenance.
Fabric Network Management Port (FNM Port) - XDR only
The FNM port is a dedicated OSFP InfiniBand in-band management port. The port enables UFM platform to connect to the IB fabric without using the switch data ports.
Q3400-RA FNM port with its dedicated LED indicator
Q3400-RA – two ETH management ports and a dedicated in-band FNM port
