YOLOv5 GPU-Only and PVA+GPU Pipelines#

Overview#

YOLOv5 is a single-stage object detector: a single pass of the neural network locates the objects in an image and returns, for each object, a bounding box, a class and a confidence score [1] [2]. The network evaluates the image on three grids, with cell sizes of 8, 16 and 32 pixels. At each cell, three anchors each predict a box, an objectness score and a score for each of the 80 COCO classes. These pipelines use YOLOv5s, the small model, with a 672×672 input.

Like most detection networks, YOLOv5 requires processing before and after the network:

  • Preprocessing. The network takes a planar F32 tensor of a fixed size, with values in [0, 1]. A camera image is usually interleaved 8-bit RGB of any size. Preprocessing resizes the image without changing its aspect ratio, pads it to a square (letterboxing), converts it to a planar F32 tensor and normalizes the 8-bit values to [0, 1].

  • Postprocessing. The network does not output a list of objects. It outputs raw predictions, 85 values for each of its 27,783 anchors. Postprocessing decodes them into boxes and scores, discards the boxes below a confidence threshold, and removes duplicate boxes of the same object with non-maximum suppression (NMS).

These are classic computer-vision tasks: data movement, element-wise arithmetic and sorting. On a GPU, they consume time that would otherwise be available to neural networks. The purpose of this demo is to show how the PVA performs these tasks while the GPU runs the network: PVA operators are submitted to CUDA streams, share buffers with TensorRT and execute concurrently with inference. The demo runs the same detector in two pipeline configurations, so that the difference can be measured (Figure 1):

  • GPU-only. Preprocessing, inference, decode and NMS all run on the GPU. Decode and NMS are part of the TensorRT engine.

  • PVA+GPU. The GPU runs only the network. VPU0 prepares the network input, and VPU1 converts the network output into detections. The three engines process three different frames concurrently.

On Jetson AGX Orin, the PVA+GPU pipeline has 1.27x the throughput of the GPU-only pipeline at MAXN (394 instead of 309 frames per second) and 1.32x at 30 W and 15 W. The GPU time per frame decreases from 3.23 ms to 2.47 ms, which leaves about 23 % of the GPU time available to other networks. The cost is a higher latency, because each frame passes through three pipeline cycles.

Dataflow of the GPU-only and PVA+GPU YOLOv5 pipelines.

Figure 1: The two pipelines. In the PVA+GPU pipeline, VPU0, the GPU and VPU1 each process a different frame, and TensorRT and PVA share their input and output buffers without copies.#

Input Data#

Both pipelines use the same YOLOv5s weights. The GPU-only model adds decode and the TensorRT EfficientNMS_TRT plugin to the network graph; the PVA+GPU model ends at the three detection heads.

The input is an 8-bit RGB image. The packaged 1280×720 traffic image is resized to 672×378 and padded at the top and bottom to the 672×672 network input.

Usage#

Each pipeline is a separate application. Both take an ONNX model followed by a TensorRT engine cache path:

./yolov5_gpu --image traffic.png yolov5s_672_nms.onnx yolov5s_672_nms.engine

./yolov5_pva_gpu --check-pva-driver
./yolov5_pva_gpu --image traffic.png \
    yolov5s_trimmed_reshape_tranpose.onnx yolov5s_trimmed_reshape_tranpose.engine

--check-pva-driver reports whether the platform supports PVA operators on CUDA streams. --benchmark prints the cycle time and the time of each stage. The READMEs in public/pipelines/yolov5 describe the other options, such as INT8 precision and COCO evaluation.

Pipeline Stages#

Stage

GPU-only

PVA+GPU

Preprocessing: resize, letterbox, HWC to CHW, scale to [0, 1]

CUDA kernels

VPU0: ImageResize, then NhwcToNchwLetterboxNormalize

Inference: YOLOv5s backbone and three detection heads

TensorRT on the GPU

TensorRT on the GPU

Decode: detection heads to scored candidate boxes

TensorRT graph

VPU1: YoloV5Decode, once per head

NMS: class-aware suppression of overlapping boxes

EfficientNMS_TRT plugin

VPU1: BatchedNMS

The three YoloV5Decode calls build one candidate list, so a single BatchedNMS call processes all three heads.

PVA and GPU Interoperability#

Three mechanisms allow the PVA and the GPU to operate as a single pipeline. They apply to any combination of PVA operators and CUDA or TensorRT work.

CUDA stream submission. Every PVA Solutions operator can be submitted to a CUDA stream. CupvaCudaStreamSetAffinity binds a CUDA stream to one VPU. PVA work on that stream then follows normal CUDA stream order, and cudaEventRecord and cudaStreamSynchronize apply to it as they do to GPU work. One cycle of the PVA+GPU pipeline is:

cudaStream_t preStream, gpuStream, postStream;   // created with cudaStreamNonBlocking
CupvaCudaStreamSetAffinity(preStream, CUPVA_VPU0);
CupvaCudaStreamSetAffinity(postStream, CUPVA_VPU1);

// Frame N on VPU0, slot a
pvaImageResizeSubmit(resizeOp, preStream, image[a], resized[a]);
pvaNhwcToNchwLetterboxNormalizeSubmit(letterboxOp, preStream, resized[a], networkInput[a]);

// Frame N-1 on the GPU, slot b: TensorRT reads networkInput[b] and writes heads[b]
context->enqueueV3(gpuStream);

// Frame N-2 on VPU1, slot c: three heads into one candidate list, then NMS
for (int head = 0; head < 3; ++head)
    pvaYoloV5DecodeSubmit(decodeOp[head], postStream, heads[c][head], anchors[head],
                          boxes[c], scores[c], classes[c], count[c]);
pvaBatchedNMSSubmit(nmsOp, postStream, boxes[c], scores[c], classes[c], count[c],
                    detBoxes[c], detScores[c], detClasses[c], detCount[c]);

// Wait for all three streams before the slots advance
cudaStreamSynchronize(postStream);
cudaStreamSynchronize(preStream);
cudaStreamSynchronize(gpuStream);

Zero-copy buffer sharing. The application allocates the TensorRT input with cudaMallocHost and the TensorRT outputs with cudaMalloc, and imports them into PVA with CupvaCudaMemImport. VPU0 writes the network input that TensorRT reads, and VPU1 reads the detection heads that TensorRT writes, without a copy.

Three-stage software pipelining. The pipeline keeps three sets of buffers, or slots, and uses slot N mod 3 for frame N (Figure 2). In each cycle, VPU0, the GPU and VPU1 each process a different frame in its own slot, so no two engines access the same buffer. After the host submits the three stages, it waits for all three streams. The cycle time is therefore the time of the slowest stage plus the submission and synchronization overhead.

Timeline of three frames in flight on VPU0, the GPU and VPU1.

Figure 2: Triple buffering. In each cycle, VPU0, the GPU and VPU1 process three different frames in three different slots.#

CUDA-stream submission requires PVA SDK 2.7.0 or later and JetPack 7.0 or DriveOS 7.0 or later (PVA driver 2007 or later). With --dla, the PVA+GPU pipeline runs inference on a DLA instead of the GPU; this configuration is not measured below.

Performance#

The measurements use Jetson AGX Orin, FP16 engines, the packaged 1280×720 traffic image, a confidence threshold of 0.25, an NMS IoU threshold of 0.45 and no CUDA Graphs. The GPU and PVA clocks are locked at the maximum of each power mode; the power-mode names are nvpmodel limits, not measured power. Each value is the mean of two runs of 5,000 frames after a warm-up. Image loading is not measured.

Results#

The PVA+GPU pipeline is 1.27x to 1.32x faster than the GPU-only pipeline at every power mode. The time per frame is 1 / FPS; for the PVA+GPU pipeline, it is one pipeline cycle.

Power mode

GPU-only

PVA+GPU

Speedup

MAXN

3.23 ms (309 FPS)

2.54 ms (394 FPS)

1.27x

30 W

8.01 ms (125 FPS)

6.07 ms (165 FPS)

1.32x

15 W

16.11 ms (62 FPS)

12.16 ms (82 FPS)

1.32x

The next table lists the time of each PVA+GPU stage at MAXN, measured on its own stream:

Stage

Engine

Time at MAXN

Preprocessing

VPU0

0.66 ms

Inference

GPU

2.47 ms

Postprocessing

VPU1

1.50 ms

The stages process different frames concurrently, so their sum is not the cycle time.

Analysis#

Stage times of the GPU-only and PVA+GPU pipelines at MAXN, drawn to scale.

Figure 3: Time per frame at MAXN, to scale. In the PVA+GPU pipeline, the three stages run concurrently on three different frames, and preprocessing and postprocessing finish before inference.#

Throughput. The PVA+GPU cycle (2.54 ms) is within 3 % of the inference time (2.47 ms), so the throughput is limited by the network alone (Figure 3). Preprocessing and postprocessing are hidden behind inference. VPU0 and VPU1 are occupied for 26 % and 59 % of each cycle, which leaves capacity for additional work.

GPU time and latency.

At MAXN

GPU-only

PVA+GPU

GPU time per frame

3.23 ms

2.47 ms

Latency per frame

3.2 ms

about 7.6 ms

The PVA+GPU pipeline requires 0.76 ms less GPU time per frame, so at the same frame rate about 23 % of the GPU time remains available to other networks. In exchange, each frame passes through three cycles, which more than doubles its latency.

Conditions for a performance gain. The PVA+GPU cycle time is approximately the longer of the inference time and the slowest PVA stage, plus a small submission and synchronization overhead. Two conclusions follow:

  • Throughput gain. The throughput increases when the GPU-only pipeline spends a significant share of its time on preprocessing and postprocessing, and the slowest PVA stage is shorter than inference. On Orin, postprocessing takes 1.50 ms, compared with 2.47 ms for inference.

  • GPU offload. GPU time is freed in all cases, which benefits systems in which the GPU also runs other networks.

The balance between the engines differs between platforms. On Thor, for example, the GPU spends much less time on decode and NMS, so the throughput gain can shrink or reverse, while the PVA+GPU pipeline still frees GPU time. Both pipelines should therefore be evaluated on the target platform.

References#

  1. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.

  1. Ultralytics, “YOLOv5,” ultralytics/yolov5.