ConvertImageFormat#

Overview#

The ConvertImageFormat operator converts an input image with one format into an output image with another format.

Input (NV12 planes)

Output (RGB8)

../../_images/in5.png
../../_images/out5.png

ConvertImageFormat Parameters: NV12 to RGB8. U and V are contrast-stretched for display.

Algorithm description#

The algorithm is implemented as a pixel-wise conversion function that reads the pixels from the input image, applies a conversion-dependent series of transformations and writes the results to the output image in the same position.

  • The YUV to YUV and RGBA to RGBA image format conversions are handled as channel order swizzling.

  • The equations for YUV with studio range to RGB image format conversion are listed as follows:

\[\begin{split}& R = 1.164 \times (Y - 16) + 1.596 \times (V - 128) \\ & G = 1.164 \times (Y - 16) - 0.392 \times (U - 128) - 0.813 \times (V - 128) \\ & B = 1.164 \times (Y - 16) + 2.017 \times (U - 128)\end{split}\]
  • The equations for RGB to YUV with studio range image format conversion are listed as follows:

\[\begin{split}& Y = \ \ \ 0.257 \times R + 0.504 \times G + 0.098 \times B + 16 \\ & U = -0.148 \times R - 0.291 \times G + 0.439 \times B + 128 \\ & V = \ \ \ 0.439 \times R - 0.368 \times G - 0.071 \times B + 128\end{split}\]

Implementation Details#

Parameters#

  • Input image with one image format

  • Output image with another image format

Dataflow Configuration#

The RasterDataflow (RDF) is used to split the whole input/output image into tiles and transfer them between DRAM and VMEM one by one.

  • The number of input/output RDF equals to the number of plane for each input/output image format and at most 4 planes are supported.

  • The input/output tile size will be adjusted to make sure that the number of iterations for each input/output RDF are the same.

  • Since transpose load/store operations are used in the VPU functions to implement a simple way to load/store each channel in the image format separately, transposition mode should be set to TRANS_MODE_1 for each RDF to apply the tile line pitch which can meet the requirement of transpose load/store operations used in the VPU functions.

VMEM Buffer Allocation#

8 VMEM buffers are allocated,

  • 4 buffers are allocated for 1st~4th plane of input image

  • 4 buffers are allocated for 1st~4th plane of output image

VPU function implementation#

Fixed-point coefficients and vdotp4x2_bbh()/vdotp4_bbh() instructions are used to optimize the performance of the image format conversion functions.
  • Since the computation part of image format conversion function is composed of a series of MAC operators, vdotp4x2_bbh()/vdotp4_bbh() instruction can provide 4x/2x MAC throughput comparing to dvmaddbh().

  • Since there are 3 channels involved in the image format conversion function between RGB and YUV, one vdotp4x2_bbh() instruction is used to handle 2 of them and one vdotp4_bbh() instruction is used to handle the 3rd channel.

The truncation and clamping the 8-bit unsigned integer output data in the range [0, 255] is handled by setting the rounding/truncation option and saturation option in the agen configurations which can lead to zero performance overhead for these operations.

There are 4 functions implemented for the image format conversion,

  1. For YUV channel order swizzling, convert_image_format_yuv2yuv_knl_exec() is called.

    • If input image format is NV12, convert_image_format_sampling_knl_exec() should be called first for input UV plane upsampling.

    • If output image format is NV12, convert_image_format_sampling_knl_exec() should be called last for output UV plane downsampling.

    • The usage of convert_image_format_sampling_knl_exec() as upsampling or downsampling can be controlled by the agen configurations.

  2. For RGBA channel order swizzling, convert_image_format_rgba2rgba_knl_exec() is called.

    • If output image format has alpha channel but input image format doesn’t, the alpha channel’s value will be set as the maximum representable of its type, e.g., 255 for 8-bit unsigned integer.

  3. For YUV to RGB conversion, convert_image_format_yuv2rgb_knl_exec() is called.

    • If input image format is NV12, convert_image_format_sampling_knl_exec() should be called first for input UV plane upsampling.

  4. For RGB to YUV conversion, convert_image_format_rgb2yuv_knl_exec() is called.

    • If output image format is NV12, convert_image_format_sampling_knl_exec() should be called last for output UV plane downsampling.

Performance#

Execution Time is the average time required to execute the operator on a single VPU core. Note that each PVA contains two VPU cores, which can operate in parallel to process two streams simultaneously, or reduce execution time by approximately half by splitting the workload between the two cores.

Total Power represents the average total power consumed by the module when the operator is executed concurrently on both VPU cores. Idle power is approximately 7.6 W on Jetson AGX Orin and 11.8 W on Jetson AGX Thor when the PVA is not processing data.

For detailed information on interpreting the performance table below and understanding the benchmarking setup, see Performance Benchmark.

Image Size

Input Format

Output Format

Execution Time

Submit Latency

Total Power

1920x1080

BGR8

NV12

0.706ms

0.006ms

13.822W

1920x1080

BGR8

YUYV

0.633ms

0.008ms

14.124W

1920x1080

BGR8p

NV12

0.699ms

0.007ms

13.721W

1920x1080

BGR8p

YUYV

0.651ms

0.009ms

13.923W

1920x1080

NV12

BGR8

0.480ms

0.007ms

15.725W

1920x1080

NV12

BGR8p

0.510ms

0.007ms

15.02W

1920x1080

NV12_ER

RGB8

0.482ms

0.006ms

15.328W

1920x1080

NV12_ER

RGB8p

0.510ms

0.008ms

14.624W

1920x1080

NV12_ER

RGBA8

0.574ms

0.006ms

14.222W

1920x1080

RGB8

BGRA8

0.550ms

0.007ms

14.73W

1920x1080

RGB8

NV12_ER

0.706ms

0.006ms

13.822W

1920x1080

RGB8p

NV12_ER

0.701ms

0.006ms

13.323W

1920x1080

YUV8p

NV12

0.363ms

0.007ms

15.734W

1920x1080

YUV8p

YUYV

0.314ms

0.009ms

16.649W

1920x1080

YUYV

BGR8

0.442ms

0.006ms

15.426W

1920x1080

YUYV

BGR8p

0.450ms

0.008ms

15.825W

1920x1080

YUYV

NV12

0.266ms

0.006ms

15.93W

Image Size

Input Format

Output Format

Execution Time

Submit Latency

Total Power

1920x1080

BGR8

NV12

0.782ms

0.003ms

15.563W

1920x1080

BGR8

YUYV

0.702ms

0.003ms

17.377W

1920x1080

BGR8p

NV12

0.771ms

0.003ms

15.259W

1920x1080

BGR8p

YUYV

0.720ms

0.003ms

15.755W

1920x1080

NV12

BGR8

0.526ms

0.003ms

17.468W

1920x1080

NV12

BGR8p

0.559ms

0.003ms

18.279W

1920x1080

NV12_ER

RGB8

0.526ms

0.003ms

17.468W

1920x1080

NV12_ER

RGB8p

0.559ms

0.003ms

18.279W

1920x1080

NV12_ER

RGBA8

0.591ms

0.004ms

17.366W

1920x1080

RGB8

BGRA8

0.367ms

0.004ms

20.454W

1920x1080

RGB8

NV12_ER

0.782ms

0.003ms

15.563W

1920x1080

RGB8p

NV12_ER

0.774ms

0.003ms

15.259W

1920x1080

YUV8p

NV12

0.395ms

0.003ms

17.354W

1920x1080

YUV8p

YUYV

0.341ms

0.003ms

20.796W

1920x1080

YUYV

BGR8

0.458ms

0.003ms

18.166W

1920x1080

YUYV

BGR8p

0.491ms

0.004ms

18.267W

1920x1080

YUYV

NV12

0.287ms

0.003ms

17.569W