YoloV5Decode#

Overview#

YoloV5Decode converts one YOLOv5 detection head into candidate detections: boxes, scores, class identifiers and a count. YOLOv5 has three heads, one per scale, so a detector calls the operator three times. The three calls can append to the same output tensors through a shared count, which produces one candidate list for non-maximum suppression without a concatenation step.

YoloV5Decode is a custom operator for YOLOv5 detection heads, so it supports only the head format of the YOLOv5 models trained on COCO: exactly 80 classes, three anchors per grid cell and the YOLOv5 box formula. The anchor sizes and strides (given as a tensor), the grid size, the confidence threshold, the image size for clipping, the head layout (CHW or HWC) and the output capacity are parameters. A detector with another number of classes or another box formula, such as an anchor-free YOLO model, requires a different decode.

In the YOLOv5 PVA+GPU pipeline, VPU1 runs the three decodes followed by BatchedNMS, and reads each head directly from the TensorRT output buffer.

Algorithm Description#

Each head holds, for each of three anchors at each grid cell \((g_x, g_y)\), the box offsets \(t_x, t_y, t_w, t_h\), an objectness logit and 80 class logits. With the anchor size \((a_w, a_h)\) and the head stride \(s\), the decoded box and the score of class \(c\) are:

\[\begin{split}c_x &= (2\sigma(t_x) - 0.5 + g_x)\,s, & c_y &= (2\sigma(t_y) - 0.5 + g_y)\,s,\\ w &= (2\sigma(t_w))^2\,a_w, & h &= (2\sigma(t_h))^2\,a_h,\\ score_c &= \sigma(obj)\,\sigma(cls_c).\end{split}\]

Every class whose score is strictly greater than the confidence threshold produces one detection, so one anchor can produce several detections. The box corners are clipped to the network input. The results are in the order of the anchors, then of the classes.

Implementation Details#

Most anchors in a frame contain no object, and each remaining anchor requires the same computation for 80 classes. The implementation therefore rejects empty anchors with one scalar test, and processes the classes of the remaining anchors as vectors.

Dataflow Configuration#

The grid is split into tiles of 49 cells, that is 147 anchors. 49 is \(16 \cdot 3 + 1\), which provides the transposing CHW load a conflict-free VMEM line pitch.

  • A double-buffered RasterDataFlow brings the prediction tiles into VMEM. An HWC head arrives as [49, 255] and is already anchor-major. A CHW head arrives as [255, 49].

  • A double-buffered SequenceDataFlow brings the matching [147, 5] anchor tile.

  • A SequenceDataFlow reads the incoming count, which is the position where this call starts to append.

  • SequenceDataFlows write the boxes, scores, classes and the new count. The VPU sets their sizes to the number of new detections, so only valid data is written.

Buffer Allocation#

  • Superbank A holds the four F32 output box rows and the F16 score and S16 class compaction buffers.

  • Superbank B holds the anchor tiles, the CHW-to-anchor-major transpose buffer, the count, the parameters and the dataflow handlers.

  • Superbank C holds the double-buffered prediction tiles.

Kernel Implementation#

  • Layout conversion in the load. For a CHW head, vfloat_load_transp converts each tile to the anchor-major layout that an HWC head already has, so one decode kernel serves both layouts.

  • Early rejection. A class score cannot exceed \(\sigma(obj)\). The host converts the threshold to a logit once, and the kernel skips every anchor whose objectness logit is not above it, without computing a sigmoid.

  • Classes as vectors. The 80 class logits of a surviving anchor are five full 16-lane vectors. The sigmoid uses the fexp2 and frcp intrinsics.

  • Compaction. The score comparison produces a lane mask, and vcollate_idx_bits with vpermute packs the surviving classes. Because the compaction stores F16 scores, the kernel repeats the comparison in F32, so the rule score > threshold holds exactly at F16 rounding boundaries.

  • Append. The count tensor is both input and output. Each call starts at the current count and adds its new detections, so three calls in stream order fill one candidate list. The caller sets the count to zero before the first call.

Limitations#

  • Exactly three anchors per cell and 80 classes.

  • The prediction and anchor tensors are tightly packed. The F32 box output can have row padding; the score, class and count outputs are packed.

  • At most 5,120 detections are stored. The incoming count is clamped to this capacity, and a call stops appending when the capacity is reached.

Performance#

Execution Time is the average time required to execute the operator on a single VPU core. Note that each PVA contains two VPU cores, which can operate in parallel to process two streams simultaneously, or reduce execution time by approximately half by splitting the workload between the two cores.

Total Power represents the average total power consumed by the module when the operator is executed concurrently on both VPU cores. Idle power is approximately 7.6 W on Jetson AGX Orin and 11.8 W on Jetson AGX Thor when the PVA is not processing data.

For detailed information on interpreting the performance table below and understanding the benchmarking setup, see Performance Benchmark.

Grid Size

Input Layout

Confidence Threshold

Decoded Boxes

Execution Time

Submit Latency

Total Power

84x84

CHW

0.25

1

1.147ms

0.004ms

10.213W

84x84

CHW

0.25

64

1.168ms

0.004ms

10.112W

84x84

CHW

0.25

512

1.323ms

0.004ms

10.112W

84x84

CHW

0.25

4096

2.552ms

0.005ms

9.81W

84x84

CHW

0.001

1

1.148ms

0.004ms

10.112W

84x84

HWC

0.25

1

0.714ms

0.004ms

10.213W

42x42

CHW

0.25

1

0.308ms

0.004ms

10.012W

28x28

CHW

0.25

1

0.152ms

0.004ms

9.911W

21x21

CHW

0.25

1

0.098ms

0.003ms

9.81W

7x7

CHW

0.25

1

0.037ms

0.003ms

9.103W

Grid Size

Input Layout

Confidence Threshold

Decoded Boxes

Execution Time

Submit Latency

Total Power

84x84

CHW

0.25

1

1.289ms

0.003ms

13.752W

84x84

CHW

0.25

64

1.314ms

0.003ms

13.752W

84x84

CHW

0.25

512

1.488ms

0.003ms

13.752W

84x84

CHW

0.25

4096

2.880ms

0.003ms

13.752W

84x84

CHW

0.001

1

1.289ms

0.003ms

13.752W

84x84

HWC

0.25

1

0.802ms

0.003ms

13.854W

42x42

CHW

0.25

1

0.343ms

0.003ms

13.449W

28x28

CHW

0.25

1

0.169ms

0.003ms

13.854W

21x21

CHW

0.25

1

0.107ms

0.003ms

13.55W

7x7

CHW

0.25

1

0.038ms

0.003ms

13.045W