GDS requires specific hardware topology to function. Verify the PCI topology to ensure GPUs and NICs are under the same switch. One way to accomplish this is through lstopo:
sudo apt install hwloc -y
lstopo --output-format png > lstopo.png
Example of compatible topology from a DGX-A100. The GPUs and the NICs are under the same switch:
Example of incompatible topology - the GPU (PCI 17:00.0) is directly attached to the CPU:
