YOLOv5 GPU-Only and PVA+GPU Pipelines#
Overview#
YOLOv5 is a single-stage object detector: a single pass of the neural network locates the objects in an image and returns, for each object, a bounding box, a class and a confidence score [1] [2]. The network evaluates the image on three grids, with cell sizes of 8, 16 and 32 pixels. At each cell, three anchors each predict a box, an objectness score and a score for each of the 80 COCO classes. These pipelines use YOLOv5s, the small model, with a 672×672 input.
Like most detection networks, YOLOv5 requires processing before and after the network:
Preprocessing. The network takes a planar F32 tensor of a fixed size, with values in [0, 1]. A camera image is usually interleaved 8-bit RGB of any size. Preprocessing resizes the image without changing its aspect ratio, pads it to a square (letterboxing), converts it to a planar F32 tensor and normalizes the 8-bit values to [0, 1].
Postprocessing. The network does not output a list of objects. It outputs raw predictions, 85 values for each of its 27,783 anchors. Postprocessing decodes them into boxes and scores, discards the boxes below a confidence threshold, and removes duplicate boxes of the same object with non-maximum suppression (NMS).
These are classic computer-vision tasks: data movement, element-wise arithmetic and sorting. On a GPU, they consume time that would otherwise be available to neural networks. The purpose of this demo is to show how the PVA performs these tasks while the GPU runs the network: PVA operators are submitted to CUDA streams, share buffers with TensorRT and execute concurrently with inference. The demo runs the same detector in two pipeline configurations, so that the difference can be measured (Figure 1):
GPU-only. Preprocessing, inference, decode and NMS all run on the GPU. Decode and NMS are part of the TensorRT engine.
PVA+GPU. The GPU runs only the network. VPU0 prepares the network input, and VPU1 converts the network output into detections. The three engines process three different frames concurrently.
On Jetson AGX Orin, the PVA+GPU pipeline has 1.27x the throughput of the GPU-only pipeline at MAXN (394 instead of 309 frames per second) and 1.32x at 30 W and 15 W. The GPU time per frame decreases from 3.23 ms to 2.47 ms, which leaves about 23 % of the GPU time available to other networks. The cost is a higher latency, because each frame passes through three pipeline cycles.
Figure 1: The two pipelines. In the PVA+GPU pipeline, VPU0, the GPU and VPU1 each process a different frame, and TensorRT and PVA share their input and output buffers without copies.#
Input Data#
Both pipelines use the same YOLOv5s weights. The GPU-only model adds decode and the TensorRT EfficientNMS_TRT
plugin to the network graph; the PVA+GPU model ends at the three detection heads.
The input is an 8-bit RGB image. The packaged 1280×720 traffic image is resized to 672×378 and padded at the top and bottom to the 672×672 network input.
Usage#
Each pipeline is a separate application. Both take an ONNX model followed by a TensorRT engine cache path:
./yolov5_gpu --image traffic.png yolov5s_672_nms.onnx yolov5s_672_nms.engine
./yolov5_pva_gpu --check-pva-driver
./yolov5_pva_gpu --image traffic.png \
yolov5s_trimmed_reshape_tranpose.onnx yolov5s_trimmed_reshape_tranpose.engine
--check-pva-driver reports whether the platform supports PVA operators on CUDA streams. --benchmark prints the
cycle time and the time of each stage. The READMEs in public/pipelines/yolov5 describe the other options, such as
INT8 precision and COCO evaluation.
Pipeline Stages#
Stage |
GPU-only |
PVA+GPU |
|---|---|---|
Preprocessing: resize, letterbox, HWC to CHW, scale to [0, 1] |
CUDA kernels |
VPU0: ImageResize, then NhwcToNchwLetterboxNormalize |
Inference: YOLOv5s backbone and three detection heads |
TensorRT on the GPU |
TensorRT on the GPU |
Decode: detection heads to scored candidate boxes |
TensorRT graph |
VPU1: YoloV5Decode, once per head |
NMS: class-aware suppression of overlapping boxes |
|
VPU1: BatchedNMS |
The three YoloV5Decode calls build one candidate list, so a single BatchedNMS call processes all three heads.
PVA and GPU Interoperability#
Three mechanisms allow the PVA and the GPU to operate as a single pipeline. They apply to any combination of PVA operators and CUDA or TensorRT work.
CUDA stream submission. Every PVA Solutions operator can be submitted to a CUDA stream.
CupvaCudaStreamSetAffinity binds a CUDA stream to one VPU. PVA work on that stream then
follows normal CUDA stream order, and cudaEventRecord and cudaStreamSynchronize apply to it as they do to GPU
work. One cycle of the PVA+GPU pipeline is:
cudaStream_t preStream, gpuStream, postStream; // created with cudaStreamNonBlocking
CupvaCudaStreamSetAffinity(preStream, CUPVA_VPU0);
CupvaCudaStreamSetAffinity(postStream, CUPVA_VPU1);
// Frame N on VPU0, slot a
pvaImageResizeSubmit(resizeOp, preStream, image[a], resized[a]);
pvaNhwcToNchwLetterboxNormalizeSubmit(letterboxOp, preStream, resized[a], networkInput[a]);
// Frame N-1 on the GPU, slot b: TensorRT reads networkInput[b] and writes heads[b]
context->enqueueV3(gpuStream);
// Frame N-2 on VPU1, slot c: three heads into one candidate list, then NMS
for (int head = 0; head < 3; ++head)
pvaYoloV5DecodeSubmit(decodeOp[head], postStream, heads[c][head], anchors[head],
boxes[c], scores[c], classes[c], count[c]);
pvaBatchedNMSSubmit(nmsOp, postStream, boxes[c], scores[c], classes[c], count[c],
detBoxes[c], detScores[c], detClasses[c], detCount[c]);
// Wait for all three streams before the slots advance
cudaStreamSynchronize(postStream);
cudaStreamSynchronize(preStream);
cudaStreamSynchronize(gpuStream);
Zero-copy buffer sharing. The application allocates the TensorRT input with cudaMallocHost and the
TensorRT outputs with cudaMalloc, and imports them into PVA with CupvaCudaMemImport. VPU0 writes the network
input that TensorRT reads, and VPU1 reads the detection heads that TensorRT writes, without a copy.
Three-stage software pipelining. The pipeline keeps three sets of buffers, or slots, and uses slot N mod 3 for frame N (Figure 2). In each cycle, VPU0, the GPU and VPU1 each process a different frame in its own slot, so no two engines access the same buffer. After the host submits the three stages, it waits for all three streams. The cycle time is therefore the time of the slowest stage plus the submission and synchronization overhead.
Figure 2: Triple buffering. In each cycle, VPU0, the GPU and VPU1 process three different frames in three different slots.#
CUDA-stream submission requires PVA SDK 2.7.0 or later and JetPack 7.0 or DriveOS 7.0 or later (PVA driver 2007 or
later). With --dla, the PVA+GPU pipeline runs inference on a DLA instead of the GPU; this configuration is not
measured below.
Performance#
The measurements use Jetson AGX Orin, FP16 engines, the packaged 1280×720 traffic image, a confidence threshold of
0.25, an NMS IoU threshold of 0.45 and no CUDA Graphs. The GPU and PVA clocks are locked at the maximum of each power
mode; the power-mode names are nvpmodel limits, not measured power. Each value is the mean of two runs of 5,000
frames after a warm-up. Image loading is not measured.
Results#
The PVA+GPU pipeline is 1.27x to 1.32x faster than the GPU-only pipeline at every power mode. The time per frame is 1 / FPS; for the PVA+GPU pipeline, it is one pipeline cycle.
Power mode |
GPU-only |
PVA+GPU |
Speedup |
|---|---|---|---|
MAXN |
3.23 ms (309 FPS) |
2.54 ms (394 FPS) |
1.27x |
30 W |
8.01 ms (125 FPS) |
6.07 ms (165 FPS) |
1.32x |
15 W |
16.11 ms (62 FPS) |
12.16 ms (82 FPS) |
1.32x |
The next table lists the time of each PVA+GPU stage at MAXN, measured on its own stream:
Stage |
Engine |
Time at MAXN |
|---|---|---|
Preprocessing |
VPU0 |
0.66 ms |
Inference |
GPU |
2.47 ms |
Postprocessing |
VPU1 |
1.50 ms |
The stages process different frames concurrently, so their sum is not the cycle time.
Analysis#
Figure 3: Time per frame at MAXN, to scale. In the PVA+GPU pipeline, the three stages run concurrently on three different frames, and preprocessing and postprocessing finish before inference.#
Throughput. The PVA+GPU cycle (2.54 ms) is within 3 % of the inference time (2.47 ms), so the throughput is limited by the network alone (Figure 3). Preprocessing and postprocessing are hidden behind inference. VPU0 and VPU1 are occupied for 26 % and 59 % of each cycle, which leaves capacity for additional work.
GPU time and latency.
At MAXN |
GPU-only |
PVA+GPU |
|---|---|---|
GPU time per frame |
3.23 ms |
2.47 ms |
Latency per frame |
3.2 ms |
about 7.6 ms |
The PVA+GPU pipeline requires 0.76 ms less GPU time per frame, so at the same frame rate about 23 % of the GPU time remains available to other networks. In exchange, each frame passes through three cycles, which more than doubles its latency.
Conditions for a performance gain. The PVA+GPU cycle time is approximately the longer of the inference time and the slowest PVA stage, plus a small submission and synchronization overhead. Two conclusions follow:
Throughput gain. The throughput increases when the GPU-only pipeline spends a significant share of its time on preprocessing and postprocessing, and the slowest PVA stage is shorter than inference. On Orin, postprocessing takes 1.50 ms, compared with 2.47 ms for inference.
GPU offload. GPU time is freed in all cases, which benefits systems in which the GPU also runs other networks.
The balance between the engines differs between platforms. On Thor, for example, the GPU spends much less time on decode and NMS, so the throughput gain can shrink or reverse, while the PVA+GPU pipeline still frees GPU time. Both pipelines should therefore be evaluated on the target platform.
References#
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.
Ultralytics, “YOLOv5,” ultralytics/yolov5.