Considerations

As with any well-planned network, those employing NVIDIA InfiniBand or Ethernet solutions should begin with a logical network design and a physical network design. This guide, being aimed at cable deployment, will speak to elements of the physical network design. It should be noted however that since logical design influences physical it should be completed first, or in tandem.

The logical network design must account for: servers, Ethernet switches, InfiniBand switches, storage, and any other cluster components. The physical network design will detail how these components are laid out and should include: floorplans, rack elevations, and infrastructure details. And most importantly, from the perspective of this guide, the physical network design will detail cable information including physical cable routes, lengths, and spares.

Developing a physical network design involves a great many considerations and following the recommendations and guidelines in the following sections will ensure a smooth and maintainable cable deployment (Figure 32).

Figure 32. Cleanly deployed cables

_images/cable-deployment-planning-01.png

Physical Cable Routes

Whether a switch exists at the ToR, intermediate/aggregate, or core/director tier in the fabric, cables destined for the same switch at the same rack position must be split in a 50/50 manner. That is, half the cables will be routed along the left side of the rack and half the cables will be routed along the right and then to their destination port. This division allows the great volume of the many InfiniBand cables to be balanced equally and thus reduces the total number of cables on any one side of the rack. Additionally, it allows for greater serviceability of all equipment within the rack ensuring no cable passes the midpoint of the rack impeding the insertion or removal of equipment.

If cables are run out the bottom and top of the rack, run them in quarters. This will reduce the congestion on the sides of the components where the cables run.

Selecting Cables

Generally fiber cables are employed for inter-rack runs due to their ability to traverse greater distances, smaller diameter, and better performance. However, when price is a determining factor, intra-rack cables will most often be passive copper cables as they are less costly and ideally suited to shorter runs. Active copper cables offer increased length compared to passive copper though not as long as fiber, and their price falls between the two. Active copper cables then are suited to both inter-rack and intra-rack runs.

Cable lengths should be chosen to minimize slack both inside the rack and in the cable management. Depending on equipment proximity, space, or cost constraints it may be advisable to mix copper passive, copper active, and fiber cables. While planning cable lengths consult the physical network design, pay particular heed to the floorplans and rack elevations, these documents will assist in selecting the proper lengths.

Note

  • Cables do impart some minor latency therefore it is best to keep all cluster components close to each other.

  • Active copper cables are usually thinner than passive copper cables and will fit easier within the rack.

  • Fiber cables are more expensive but take less room and weigh less than copper.

  • Use both passive and active Copper cables to minimize cost.

  • Use both Copper and Fiber cables if possible, to minimize cost.

Failure rates for InfiniBand cables are very low. However, like any hardware component, they do occur. Plan for one extra cable per 200 end points. Spares must be of varying lengths matching the lengths of the cables within its proximity. Alternatively, if spares are run, be sure that the cable has enough slack to reach additional ports.

Note

NVIDIA offers a unique option with cables, color-coded pull tabs. These tabs can assist the installer in identifying cables during the implementation.

Selecting Racks

For large multilayer fabrics built using high-density/director switches, it is best practice to house the switch chassis in its own rack. The rack should also meet the following guidelines:

  • Source extra wide racks.

  • Have 4-6 inches (10-15 cm) between the vertical supports and the sides of the rack. This is important when using copper cables due to the greater individual cable diameter.

  • Do not remove front doors, rear doors, or side panels.

  • Leave around 4U at the top and bottom of the rack. This will facilitate manipulating cables at the time of deployment.

Selecting Cable Management

Cable management comes in different sizes; review manufactures specifications and source large enough systems to account for the great number of cables and diameters. Where possible oversize cable management such that cable slack can be contained outside the rack.