InfiniBand fabric components can be connected using different topologies and it should be decided before building the cluster.
Fat-Tree is NVIDIA's recommended topology, AI factory should based on rail optimized.
Rail-optimized design means a GPU node with multiple interfaces will put each GPU “rail” (IB network interface) onto a different first level (LEAF) switch for cluster Interconnect. This allows multiple nodes to utilize their internal NVSwitch path to talk across a NIC that is just one switch hop away (instead of having to cross multiple switches, incurring additional latency).
The diagram below shows an example of cluster topology for AI factory (based on rail optimized):
3 Tier Topology
2 Tier Topology
To choose the best cluster planning that fits the cluster needs, please contact NVIDIA Support.
After selecting the InfiniBand interconnect principles, create a PTP excel file to describe the cluster connectivity and to generate the Topology file.
It is imperative to verify that the cluster has been connected according to the cluster planning to ensure cluster maintenance.