NhwcToNchwLetterboxNormalize#
Overview#
NhwcToNchwLetterboxNormalize prepares a resized image as the input of a neural network. In one pass, it converts an
interleaved U8 RGB image to planar F32, scales every value, and places the image at the center of a larger output
(letterboxing). Resizing is a separate step, so the caller chooses the interpolation method.
In the YOLOv5 PVA+GPU pipeline, VPU0 runs ImageResize and then this operator, which writes the 3×672×672 network input directly into the TensorRT input buffer.
Algorithm Description#
For an input \(I[H_i, W_i, 3]\) and an output \(O[3, H_o, W_o]\), the image starts at the offset
and the image region of the output is
An odd extra row or column of padding is placed at the bottom or the right. The operator does not swap channels and does not subtract a mean or divide by a standard deviation.
Implementation Details#
The operator consists of data movement and one multiplication per value. The implementation therefore lets the DMA place each tile at its final position, and does the deinterleaving and the conversion with transposing loads and stores, so that the kernel accesses each value only once.
Dataflow Configuration#
The image is processed in fixed 64×64-pixel tiles with four RasterDataFlows:
One input RasterDataFlow copies only the resized image from DRAM into a double-buffered U8 HWC tile in VMEM. It uses
TRANS_MODE_1for transposing byte loads.Three output RasterDataFlows copy the R, G and B tiles to their F32 output planes. They share one handler and use
TRANS_MODE_1for transposing float stores.Each output starts at the image region in the center of its plane and uses the full output width as its line pitch. The kernel therefore writes no padding.
The right and bottom tiles use their actual size, so the image size does not have to be a multiple of 64.
Buffer Allocation#
The buffers are spread over the three VMEM superbanks, so that input loads and output stores can overlap:
Superbank A holds the double-buffered input tile, the input dataflow handler and the operator parameters.
Superbank B holds the double-buffered output tiles for channels 0 and 1.
Superbank C holds the double-buffered output tile for channel 2 and the shared output dataflow handler.
Kernel Implementation#
The kernel processes 16 rows at a time. Three consecutive transposing U8 loads each read one channel value from each
row, which deinterleaves the RGB input. The values are zero-extended, converted to F32, multiplied by scale, and
written with transposing stores into the three output tiles. Layout conversion and scaling therefore happen without an
intermediate tensor.
Padding Policy#
The PVA writes only the image region, so the caller selects how the padding is maintained:
PVA_NHWC_TO_NCHW_PADDING_PREFILLEDThe caller fills the padding of each output buffer once. No fill is needed for each submission. This is the preferred mode for a pipeline that owns its buffers, such as the YOLOv5 pipeline.
PVA_NHWC_TO_NCHW_PADDING_RESTORE_ON_SUBMITThe operator restores the padding before each PVA image write. On a CUDA stream, the host fill is queued with
cudaLaunchHostFunc. A cuPVA stream has no host callback, so the operator waits for the earlier work on the stream before it fills the host mapping. Each submission is then independent of the earlier contents of the buffer, but on a cuPVA stream the submission is synchronous with the earlier work.
If the padding is only above and below the image, it is filled as two contiguous bands. If the padding is also at the sides, each plane is filled completely, and the PVA write then overwrites the image region.
Limitations#
The input is a rank-3 U8 HWC tensor with exactly three channels. The pixels are packed; padding after each input row is allowed.
The output is a tightly packed rank-3 F32 CHW tensor with three planes.
The input width and height must not exceed the output dimensions.
scaleandpadValuemust be finite.The operator processes one image; there is no batch dimension.
Performance#
Execution Time is the average time required to execute the operator on a single VPU core.
Note that each PVA contains two VPU cores, which can operate in parallel to process two streams simultaneously, or reduce execution time by approximately half by splitting the workload between the two cores.
Total Power represents the average total power consumed by the module when the operator is executed concurrently on both VPU cores.
Idle power is approximately 7.6 W on Jetson AGX Orin and 11.8 W on Jetson AGX Thor when the PVA is not processing data.
For detailed information on interpreting the performance table below and understanding the benchmarking setup, see Performance Benchmark.
Input Size |
Output Size |
Execution Time |
Submit Latency |
Total Power |
|---|---|---|---|---|
672x378 |
672x672 |
0.214ms |
0.107ms |
10.611W |
672x504 |
672x672 |
0.281ms |
0.070ms |
10.812W |
672x672 |
672x672 |
0.369ms |
0.005ms |
11.513W |
378x672 |
672x672 |
0.314ms |
0.198ms |
11.714W |
640x360 |
640x640 |
0.192ms |
0.095ms |
10.51W |
Input Size |
Output Size |
Execution Time |
Submit Latency |
Total Power |
|---|---|---|---|---|
672x378 |
672x672 |
0.182ms |
0.080ms |
15.461W |
672x504 |
672x672 |
0.235ms |
0.051ms |
15.36W |
672x672 |
672x672 |
0.304ms |
0.003ms |
15.866W |
378x672 |
672x672 |
0.179ms |
0.149ms |
14.127W |
640x360 |
640x640 |
0.166ms |
0.071ms |
14.551W |