Working with Image Frames on GPU or CPU Buffers#

Effect filters accept image buffers as NvCVImage objects. The image buffers can be CPU or GPU buffers, but for performance reasons, the effect filters require GPU buffers. The AR SDK provides functions for converting an image representation to an NvCVImage object and transferring images between CPU and GPU buffers.

For more information about NvCVImage, refer to the NvCVImage API Guide. This section provides a synopsis of the functions that are most frequently used with the AR SDK.

Converting Image Representations to NvCVImage Objects#

The SDK provides functions for converting OpenCV images and other image representations to NvCVImage objects. Each function places a wrapper around an existing buffer. The wrapper prevents the buffer from being freed when the destructor of the wrapper is called.

Converting OpenCV Images to NvCVImage Objects#

You can use the wrapper functions that the SDK provides specifically for RGB OpenCV images.

Note

The AR SDK provides wrapper functions only for RGB images. No wrapper functions are provided for YUV images.

  • To create an NvCVImage object wrapper for an OpenCV image, use the NVWrapperForCVMat() function.

    //Allocate source and destination OpenCV images
cv::Mat srcCVImg( );
cv::Mat dstCVImg(...);

// Declare source and destination NvCVImage objects
NvCVImage srcCPUImg;
NvCVImage dstCPUImg;

NVWrapperForCVMat(&srcCVImg, &srcCPUImg);
NVWrapperForCVMat(&dstCVImg, &dstCPUImg);

  • To create an OpenCV image wrapper for an NvCVImage object, use the CVWrapperForNvCVImage() function.

// Allocate source and destination NvCVImage objects
NvCVImage srcCPUImg(...);
NvCVImage dstCPUImg(...);

//Declare source and destination OpenCV images
cv::Mat srcCVImg;
cv::Mat dstCVImg;

CVWrapperForNvCVImage(&srcCPUImg, &srcCVImg);
CVWrapperForNvCVImage(&dstCPUImg, &dstCVImg);

Converting Other Image Representations to NvCVImage Objects#

To convert other image representations, call the NvCVImage_Init() function to place a wrapper around an existing buffer (srcPixelBuffer).

NvCVImage src_gpu;

vfxErr = NvCVImage_Init(&src_gpu, 640, 480, 1920, srcPixelBuffer,
NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_GPU);

NvCVImage src_cpu;

vfxErr = NvCVImage_Init(&src_cpu, 640, 480, 1920, srcPixelBuffer,
NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU);

Converting Decoded Frames from the NvDecoder to NvCVImage Objects#

To convert decoded frames from the NVDecoder to NvCVImage objects, call the NvCVImage_Transfer() function to convert the decoded frame that is provided by the NvDecoder from the decoded pixel format to the format that is required by a feature of the AR SDK.

The following sample shows a decoded frame that was converted from the NV12 frame format to the BGRA pixel format.

NvCVImage decoded_frame, BGRA_frame, stagingBuffer;

NvDecoder dec;

//Initialize decoder

//Assuming dec.GetOutputFormat() == cudaVideoSurfaceFormat_NV12

//Initialize memory for decoded frame
NvCVImage_Init(&decoded_frame, dec.GetWidth(), dec.GetHeight(),
               dec.GetDeviceFramePitch(), NULL, NVCV_YUV420, NVCV_U8, NVCV_NV12,
               NVCV_GPU, 1);

decoded_frame.colorSpace = NVCV_709 | NVCV_VIDEO_RANGE |
                           NVCV_CHROMA_COSITED;

//Allocate memory for BGRA frame, and set alpha opaque
NvCVImage_Alloc(&BGRA_frame, dec.GetWidth(), dec.GetHeight(), NVCV_BGRA,
                NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1);

cudaMemset(BGRA_frame.pixels, -1, BGRA_frame.pitch *
           BGRA_frame.height);

decoded_frame.pixels = (void*)dec.GetFrame();

//Convert from decoded frame format(NV12) to desired format(BGRA)
NvCVImage_Transfer(&decoded_frame, &BGRA_frame, 1.f, stream, &stagingBuffer);

Note

The preceding sample assumes the typical colorspace specification for HD content. SD typically uses NVCV_601. There are eight possible combinations, and you should use the one that matches your video as described in the video header or proceed by trial and error.

Here is some additional information:

  • If the colors are incorrect, swap 709<->601.

  • If they are washed out or blown out, swap VIDEO<->FULL.

  • If the colors are shifted horizontally, swap INTSTITIAL<->COSITED.

Converting an NvCVImage Object to a Buffer Encodable by NvEncoder#

To convert the NvCVImage object to the pixel format that is used during encoding via NvEncoder, if necessary, call the NvCVImage_Transfer() function.

The following sample shows a frame that is encoded in the BGRA pixel format.

//BGRA frame is 4-channel, u8 buffer residing on the GPU
NvCVImage BGRA_frame;
NvCVImage_Alloc(&BGRA_frame, dec.GetWidth(), dec.GetHeight(),
                NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1);

//Initialize encoder with a BGRA output pixel format
using NvEncCudaPtr = std::unique_ptr<NvEncoderCuda,
                     std::function<void(NvEncoderCuda*)>>;

NvEncCudaPtr pEnc(new NvEncoderCuda(cuContext, dec.GetWidth(),
                  dec.GetHeight(), NV_ENC_BUFFER_FORMAT_ARGB));

pEnc->CreateEncoder(&initializeParams);

// your code here

std::vector<std::vector<uint8_t>> vPacket;

//Get the address of the next input frame from the encoder
const NvEncInputFrame* encoderInputFrame = pEnc->GetNextInputFrame();

//Copy the pixel data from BGRA_frame into the input frame address obtained above
NvEncoderCuda::CopyToDeviceFrame(cuContext,
                                 BGRA_frame.pixels,
                                 BGRA_frame.pitch,
                                 (CUdeviceptr)encoderInputFrame->inputPtr,
                                 encoderInputFrame->pitch,
                                 pEnc->GetEncodeWidth(),
                                 pEnc->GetEncodeHeight(),
                                 CU_MEMORYTYPE_DEVICE,
                                 encoderInputFrame->bufferFormat,
                                 encoderInputFrame->chromaOffsets,
                                 encoderInputFrame->numChromaPlanes);

pEnc->EncodeFrame(vPacket);

Allocating an NvCVImage Object Buffer#

You can allocate the buffer for an NvCVImage object by using the NvCVImage allocation constructor or image functions. In both options, the buffer is automatically freed by the destructor when the images go out of scope.

Using the NvCVImage Allocation Constructor to Allocate a Buffer#

The NvCVImage allocation constructor creates an object to which memory has been allocated and that has been initialized. Refer to TODO: Allocation Constructor for more information.

The final three optional parameters of the allocation constructor determine the properties of the resulting NvCVImage object:

  • The pixel organization determines whether blue, green, and red are in separate planes or interleaved.

  • The memory type determines whether the buffer resides on the GPU or the CPU.

  • The byte alignment determines the gap between consecutive scanlines.

The following examples show how to use the final three optional parameters of the allocation constructor to determine the properties of the NvCVImage object.

This example creates an object without setting the final three optional parameters of the allocation constructor. In this object, the blue, green, and red components interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default alignment.

NvCVImage cpuSrc(
    srcWidth,
    srcHeight,
    NVCV_BGR,
    NVCV_U8
);

This example creates an object with identical pixel organization, memory type, and byte alignment to the previous example by setting the final three optional parameters explicitly. As in the previous example, the blue, green, and red components are interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default—that is, optimized for maximum performance.

NvCVImage src(
    srcWidth,
    srcHeight,
    NVCV_BGR,
    NVCV_U8,
    NVCV_INTERLEAVED,
    NVCV_CPU,
    0
);

This example creates an object in which the blue, green, and red components are in separate planes, the buffer resides on the GPU, and the byte alignment ensures that no gap exists between one scanline and the next scanline.

NvCVImage gpuSrc(
    srcWidth,
    srcHeight,
    NVCV_BGR,
    NVCV_U8,
    NVCV_PLANAR,
    NVCV_GPU,
    1
);

Using Image Functions to Allocate a Buffer#

By declaring an empty image, you can defer buffer allocation.

  1. Declare an empty NvCVImage object.

    NvCVImage xfr;

  2. Allocate or reallocate the buffer for the image.

  • To allocate the buffer, call the NvCVImage_Alloc() function.

    Allocate a buffer this way when the image is part of a state structure, where you do not know the size of the image until later.

  • To reallocate a buffer, call NvCVImage_Realloc().

    This function checks for an allocated buffer and reshapes the buffer if it is big enough before freeing the buffer and calling NvCVImage_Alloc().

Transferring Images Between CPU and GPU Buffers#

If the memory types of the input and output image buffers are different, an application can transfer images between CPU and GPU buffers.

Transferring Input Images from a CPU Buffer to a GPU Buffer#

  1. Create an NvCVImage object to use as a staging GPU buffer that has the same dimensions and format as the source CPU buffer.

    NvCVImage srcGpuPlanar(inWidth, inHeight, NVCV_BGR, NVCV_F32,
                       NVCV_PLANAR, NVCV_GPU, 1);

  2. Create a staging buffer in one of the following ways:

    • To avoid allocating memory in a video pipeline, create a GPU buffer that has the same dimensions and format as required for input to the video effect filter.

      NvCVImage srcGpuStaging(inWidth, inHeight, srcCPUImg.pixelFormat,
                        srcCPUImg.componentType, srcCPUImg.planar, NVCV_GPU);

    • To simplify your application program code, declare an empty staging buffer.

      NvCVImage srcGpuStaging;

      An appropriate buffer will be allocated or reallocated as needed.

  3. Call the NvCVImage_Transfer() function to copy the source CPU buffer contents into the final GPU buffer via the staging GPU buffer.

    //Read the image into srcCPUImg

NvCVImage_Transfer(&srcCPUImg, &srcGPUPlanar, 1.0f, stream,
                   &srcGPUStaging);

Transferring Output Images from a GPU Buffer to a CPU Buffer#

  1. Create an NvCVImage object to use as a staging GPU buffer that has the same dimensions and format as the destination CPU buffer.

    NvCVImage dstGpuPlanar(outWidth, outHeight, NVCV_BGR, NVCV_F32,
                       NVCV_PLANAR, NVCV_GPU, 1);

  2. Create a staging buffer in one of the following ways:

    • To avoid allocating memory in a video pipeline, create a GPU buffer that has the same dimensions and format as the output of the video effect filter.

      NvCVImage dstGpuStaging(outWidth, outHeight, dstCPUImg.pixelFormat,
                        dstCPUImg.componentType, dstCPUImg.planar, NVCV_GPU);

    • To simplify your application program code, declare an empty staging buffer.

      NvCVImage dstGpuStaging;

      An appropriately sized buffer will be allocated as needed.

  3. Call the NvCVImage_Transfer() function to copy the GPU buffer contents into the destination CPU buffer via the staging GPU buffer.

    //Retrieve the image from the GPU to CPU, perhaps with conversion.
NvCVImage_Transfer(&dstGpuPlanar, &dstCPUImg, 1.0f, stream,
                   &dstGpuStaging);