Congestion Assessment on InfiniBand Links


The xmit_wait increment behavior on NVIDIA hardware may vary depending on the generation of the devices.

As per the InfiniBand specification, the xmit_wait counter represents the number of ticks in which the selected port had data to transmit, but no data was sent during the entire tick. This can happen due to insufficient credits or a lack of arbitration.

A tick is defined as a multiple of the time required to transfer one byte on an IBA lane, which is also known as the symbol time. For instance, in case of links operating at IBA SDR (Single Data Rate), the symbol time is 4 nsec regardless of the link width. Implementers can use a variety of multipliers of the basic tick interval.

The tick encoding for the portsamplecontrol attribute is:


0x00 = 1x symbol time (4 nanoseconds for SDR)   0x01 = 2x symbol time   ...   0xFF = 256x symbol time

To confirm the tick setting of a device, users can employ the perfquery tool, which displays the portsamplecontrol parameters for the desired port.

E.g., Querying device with LID 32 / port 5:


$ perfquery -c 32 5   # PortSamplesControl: Lid 32 port 5   OpCode:..........................0xff   PortSelect:......................0   Tick:............................0x07

The table below illustrates how the xmit_wait counter increments on different device hardware, and provides instructions for converting the port's xmit_wait counter to BW loss. This will enable users to evaluate the congestion on the port.


HW Behavior


Congestion Calculation

Quantum, Quantum-2,Quantum-3

Increments xmit_wait every 8 bytes – regardless of the port width.

HW counts 8B per port hence port will report:
tick == 7

Lost bytes per port = Counter*(tick+1)

SwitchIB-2, Switch-IB, Switch-X

Increments xmit_wait every 32 bytes per lane

HW counts 32B per lane hence port will report:
tick == 31

Lost bytes per port = Counter*(tick+1)*#lanes

To calculate the lost bandwidth rate in Gb/s per port, one can use the following formula:


© Copyright 2023, NVIDIA. Last updated on May 23, 2023.