BatchedNMS#
Overview#
BatchedNMS performs greedy, class-aware non-maximum suppression (NMS): from a list of candidate detections, it
retains the highest-scoring box of each object and removes the overlapping boxes of the same class. It is the last step
of an object detector.
In the YOLOv5 PVA+GPU pipeline, VPU1 runs BatchedNMS on the candidate
list that three YoloV5Decode calls produce.
Algorithm Description#
The input boxes are four rows of left, top, right and bottom coordinates, with parallel arrays of scores and class identifiers, and a count of valid entries. Each class is processed separately:
Sort the candidates of the class by descending score.
Keep the highest-scoring candidate that is not suppressed.
Suppress every remaining candidate whose intersection-over-union (IoU) with the kept box is strictly greater than the threshold.
Repeat steps 2 and 3 until no candidates are left.
The classes are visited in the order in which they first appear in the input, and the output limit applies across all classes. The output is therefore sorted by score within each class, but it is not a global top-k across classes.
Implementation Details#
NMS visits the candidates of a class repeatedly, so the implementation keeps the complete valid input in VMEM. It then sorts each class with a library sort primitive and evaluates the IoU of one kept box against all candidates of the class as vector operations.
Dataflow Configuration#
The number of valid candidates is known only at run time. The operator therefore reads the count first and transfers only the valid data:
A SequenceDataFlow reads the S32 count, and the VPU clamps it to the configured input capacity.
The VPU sets a second SequenceDataFlow to transfer exactly that many elements of the four F32 box rows, the F16 scores and the S16 classes.
After suppression, the VPU sets the output SequenceDataFlow to transfer exactly the surviving detections: four F32 box rows, F32 scores, S32 classes and the S32 count.
This order prevents reads beyond the valid input and requires only one payload transfer. The output rows use the row pitches of the caller’s tensors.
Buffer Allocation#
Superbank C holds the four F32 input box rows.
Superbank A holds the F16 scores, a combined sort-output and suppression-mask buffer, one sort buffer, a six-row output buffer and the sort configuration.
Superbank B holds the S16 classes, the per-class score and box buffers, the second sort buffer, the count, the parameters and the dataflow handlers.
The output buffer has six rows of F32 size: four box coordinates, scores and classes. All output transfers can then use one common capacity without writing beyond the allocation.
Kernel Implementation#
Class collection. A first pass builds a 300-entry map of the classes present. For each class, a vector compare and
vcollate_idx_bitsgather up to 4,096 matching scores and boxes into contiguous per-class buffers.Sorting. The
SortPayloadVpuprimitive of the Accelerated Primitive Library sorts key-payload pairs in ascending order. The kernel converts each positive F16 score bit pattern into a negated integer key and carries the candidate index as payload, which produces a descending score order. The sort size is rounded up to a supported power of two from 512 to 8,192, with zero padding after the valid scores.Suppression. The kernel computes all box areas and clears the suppression mask. For each kept box, vector operations compute the intersection and union with all candidates of the class, and OR the result of
intersection > union * thresholdinto the mask. The multiplication avoids a division and keeps the strict IoU comparison.Output. Kept boxes are appended until all classes are processed or the output capacity is reached. The input and output counts are clamped, so an invalid count cannot address memory outside the tensors.
Limitations#
The configured input and output capacities are at most 8,192 and 1,000 boxes.
At most the first 4,096 boxes of one class take part in suppression.
Valid class identifiers are 0 to 299; the operator ignores other identifiers.
Scores must be finite and positive. The order of equal scores is unspecified.
The box rows can have row padding; the one-dimensional tensors are tightly packed.
Only the output entries below the returned count are written.
Performance#
Execution Time is the average time required to execute the operator on a single VPU core.
Note that each PVA contains two VPU cores, which can operate in parallel to process two streams simultaneously, or reduce execution time by approximately half by splitting the workload between the two cores.
Total Power represents the average total power consumed by the module when the operator is executed concurrently on both VPU cores.
Idle power is approximately 7.6 W on Jetson AGX Orin and 11.8 W on Jetson AGX Thor when the PVA is not processing data.
For detailed information on interpreting the performance table below and understanding the benchmarking setup, see Performance Benchmark.
Input Boxes |
IoU Threshold |
Output Boxes |
Execution Time |
Submit Latency |
Total Power |
|---|---|---|---|---|---|
128 |
0.45 |
32 |
0.214ms |
0.003ms |
8.207W |
1024 |
0.45 |
100 |
0.426ms |
0.004ms |
8.107W |
5120 |
0.45 |
1000 |
1.149ms |
0.004ms |
8.006W |
5120 |
0.65 |
1000 |
1.150ms |
0.004ms |
8.006W |
8192 |
0.45 |
1000 |
1.638ms |
0.004ms |
8.006W |
Input Boxes |
IoU Threshold |
Output Boxes |
Execution Time |
Submit Latency |
Total Power |
|---|---|---|---|---|---|
128 |
0.45 |
32 |
0.195ms |
0.003ms |
12.147W |
1024 |
0.45 |
100 |
0.439ms |
0.003ms |
12.147W |
5120 |
0.45 |
1000 |
1.235ms |
0.003ms |
11.752W |
5120 |
0.65 |
1000 |
1.235ms |
0.003ms |
11.752W |
8192 |
0.45 |
1000 |
1.781ms |
0.003ms |
11.752W |