From ConnectX-4 Lx and later, Send Queues (SQs) may be individually rate-limited, allowing granular rate control over specific software-defined flows. A rate-limited flow is allowed to transmit a few packets (burst) before its transmission rate is evaluated, and the next packet is scheduled for transmission accordingly.
The ConnectX family of products supports ultrafine rate limiting, enforcing an inter-burst gap of few nanoseconds and an average inter-burst gap in the picoseconds.
In each software flow, the packet pacing capability also allows determining the maximum burst size.
The rate limit assigned to an SQ/QP is indicated by the packet_pacing_rate_limit_index field in SQ context/QP context.
An SQ is associated with a rate limit at creation time using the command CREATE_SQ and may be modified at any stage with the command MODIFY_SQ.
A QP is associated with a rate limit at creation time using the command CREATE_QP and may be modified at any stage with the command MODIFY_SQ.
SQs/QPs must not be assigned with an invalid entry in the rate limit table.
Packet pacing is a credit-based token allocation and those tokens are allocated based on an internal clock. With 5T, the internal clock can be the PHC, which means that the rate remains accurate over a long time without drifting.