Work with Image Frames on GPU or CPU Buffers#

Effect filters accept image buffers as NvCVImage objects. The image buffers can be CPU or GPU buffers, but for performance reasons, the effect filters require GPU buffers. The SDK provides functions for converting an image representation to NvCVImage and transferring images between CPU and GPU buffers.

For more information about NvCVImage, refer to the NvCVImage API Guide. This section provides a synopsis of the most frequently used functions with the Video Effects SDK.

Converting Image Representations to NvCVImage Objects#

The Video Effects SDK provides functions for converting OpenCV images and other image representations to NvCVImage objects. Each function places a wrapper around an existing buffer. The wrapper prevents the buffer from being freed when the destructor of the wrapper is called.

Converting OpenCV Images to NvCVImage Objects#

Note

Use the wrapper functions that the SDK provides specifically for RGB OpenCV images.

To create an NvCVImage object wrapper for an OpenCV image, use the NVWrapperForCVMat() function.
To create an OpenCV image wrapper for an NvCVImage object, use the CVWrapperForNvCVImage() function.

Converting Image Frames on GPU or CPU Buffers to NvCVImage Objects#

Call the NvCVImage_Init() function to place a wrapper around an existing buffer (srcPixelBuffer).

Converting Decoded Frames from the NvDecoder to NvCVImage Objects#

Call the NvCVImage_Transfer() function to convert the decoded frame that is provided by the NvDecoder from the decoded pixel format to the format that is required by a feature of the Video Effects SDK. The following example shows a decoded frame that was converted from NV12 to the BGRA pixel format.

Note

The preceding sample assumes the typical colorspace specification for HD content. SD typically uses NVCV_601. There are eight possible combinations, and you should use the one that matches your video as described in the video header or proceed by trial and error:

Here is some additional information: - If the colors are incorrect, swap 709 and 601. - If they are washed out or blown out, swap VIDEO and FULL. - If the colors are shifted horizontally, swap INTSTITIAL and COSITED.

Converting an NvCVImage Object to a Buffer Encodable by NvEncoder#

To convert the NvCVImage object to the pixel format that is used during encoding via NvEncoder, if needed, call the NvCVImage_Transfer() function. The following example shows a frame that is encoded in the BGRA pixel format.

Allocating an NvCVImage Object Buffer#

You can allocate the buffer for an NvCVImage object by using the NvCVImage allocation constructor or image functions. In both options, the buffer is automatically freed by the destructor when the images go out of scope.

Using the NvCVImage Allocation Constructor to Allocate a Buffer#

The NvCVImage allocation constructor creates an object to which memory has been allocated and that has been initialized. Refer to Allocation Constructor for more information.

The final three optional parameters of the allocation constructor determine the properties of the resulting NvCVImage object:

The pixel organization determines whether blue, green, and red are in separate planes or interleaved.
The memory type determines whether the buffer resides on the GPU or the CPU.
The byte alignment determines the gap between consecutive scanlines.

The following examples show how to use the final three optional parameters of the allocation constructor to determine the properties of the NvCVImage object.

This example creates an object without setting the final three optional parameters of the allocation constructor. In this object, the blue, green, and red components interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default alignment.
This example creates an object with identical pixel organization, memory type, and byte alignment to the previous example by setting the final three optional parameters explicitly. As in the previous example, the blue, green, and red components are interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default, that is, optimized for maximum performance.
This example creates an object in which the blue, green, and red components are in separate planes, the buffer resides on the GPU, and the byte alignment ensures that no gap exists between one scanline and the next scanline.

Using Image Functions to Allocate a Buffer#

By declaring an empty image, you can defer buffer allocation.

Declare an empty NvCVImage object.
Allocate or reallocate the buffer for the image.
- To allocate the buffer, call the NvCVImage_Alloc() function. Allocate a buffer this way when the image is part of a state structure, where you will not know the size of the image until later.
- To reallocate a buffer, call NvCVImage_Realloc(). This function checks for an allocated buffer and reshapes the buffer if it is big enough before freeing the buffer and calling NvCVImage_Alloc().

Transferring Images Between CPU and GPU Buffers#

If the memory types of the input and output image buffers are different, an application can transfer images between CPU and GPU buffers.

Transferring Input Images from a CPU Buffer to a GPU Buffer#

To transfer an image from the CPU to a GPU buffer with conversion, given the following code:

NvCVImage srcCpuImg(width, height, NVCV_RGB, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU, 1);
NvCVImage dstGpuImg(width, height, NVCV_BGR, NVCV_F32, NVCV_PLANAR, NVCV_GPU, 1);

Create an NvCVImage object to use as a staging GPU buffer in one of the following ways:
- To avoid allocating memory in a video pipeline, create a GPU buffer during the initialization phase, with the same dimensions and format as the CPU image.
- To simplify your application program code, you can declare an empty staging buffer during the initialization phase:
```
NvCVImage stageImg;
```
An appropriately sized buffer will be allocated or reallocated as needed (if needed).
Call the NvCVImage_Transfer() function to copy the source CPU buffer contents into the final GPU buffer via the staging GPU buffer.

The same staging buffer can be reused in multiple NvCVImage_Transfer() calls in different contexts regardless of the image sizes and can avoid buffer allocations if it is persistent.

Transferring Output Images from a GPU Buffer to a CPU Buffer#

To transfer an image from the GPU to a CPU buffer with conversion, given the following code:

NvCVImage srcGpuImg(width, height, NVCV_BGR, NVCV_F32, NVCV_PLANAR, NVCV_GPU, 1);

NvCVImage dstCpuImg(width, height, NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU, 1);

Create an NvCVImage object to use as a staging GPU buffer in one of the following ways:
- To avoid allocating memory in a video pipeline, create a GPU buffer during the initialization phase with the same dimensions and format as the CPU image.
- To simplify your application program code, you can declare an empty staging buffer during the initialization phase:
```
NvCVImage stageImg;
```
An appropriately sized buffer is allocated or reallocated if needed. For more information about NvCVImage, refer to NvCVImage API Guide.

Call the NvCVImage_Transfer() function to copy the GPU buffer contents into the destination CPU buffer via the staging GPU buffer:

// Retrieve the image from the GPU to CPU, perhaps with conversion.

NvCVImage_Transfer(&srcGpuImg, &dstCpuImg, 1.0f, stream, &stageImg);

The same staging buffer can be used repeatedly without reallocations in NvCVImage_Transfer() if it is persistent.