Images#

AI filters in the NVIDIA® AR and Video Effects SDKs accept image buffers as NvCVImage objects. The image buffers can be CPU or GPU buffers, but for performance reasons, the AI filters require GPU buffers. The SDKs provide the functions to convert an image representation to NvCVImage and transfer images between CPU and GPU buffers.

Convert Image Representations to NvCVImage Objects#

The AR and Video Effects SDKs provide functions to convert OpenCV images and other image representations to NvCVImage objects. Each function places a wrapper around an existing buffer. The wrapper prevents the buffer from being freed when the destructor of the wrapper is called.

Convert OpenCV Images to NvCVImage Objects#

You can use the wrapper functions that the SDKs provide specifically for RGB OpenCV images.

Note

The AR and Video Effects SDKs provide wrapper functions only for RGB images. No wrapper functions are provided for YUV images.

To create an NvCVImage object wrapper for an OpenCV image, use the NVWrapperForCVMat() function.

//Allocate source and destination OpenCV images
cv::Mat srcCVImg(...);
cv::Mat dstCVImg(...);

// Declare source and destination NvCVImage objects
NvCVImage srcCPUImg;
NvCVImage dstCPUImg;

NVWrapperForCVMat(&srcCVImg, &srcCPUImg);
NVWrapperForCVMat(&dstCVImg, &dstCPUImg);

To create an OpenCV image wrapper for an NvCVImage object, use the CVWrapperForNvCVImage() function.

// Allocate source and destination NvCVImage objects
NvCVImage srcCPUImg(...);
NvCVImage dstCPUImg(...);

//Declare source and destination OpenCV images
cv::Mat srcCVImg;
cv::Mat dstCVImg;

CVWrapperForNvCVImage (&srcCPUImg, &srcCVImg);
CVWrapperForNvCVImage (&dstCPUImg, &dstCVImg);

Convert Image Frames on GPU or CPU Buffers to NvCVImage Objects#

Call the NvCVImage_Init() function to place a wrapper around an existing buffer (srcPixelBuffer).

NvCVImage src_gpu;
vfxErr = NvCVImage_Init(&src_gpu, 640, 480, 1920, srcPixelBuffer, NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_GPU);

NvCVImage src_cpu;
vfxErr = NvCVImage_Init(&src_cpu, 640, 480, 1920, srcPixelBuffer, NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU);

Convert Decoded Frames from NvDecoder to NvCVImage Objects#

Call the NvCVImage_Transfer() function to convert the decoded frame that is provided by the NvDecoder from the decoded pixel format to the format that is required by a feature of the SDKs. The following example converts a decoded frame from the NV12 format to the BGRA pixel format.

NvCVImage decoded_frame, BGRA_frame, stagingBuffer;
NvDecoder dec;

//Initialize decoder...
//Assuming dec.GetOutputFormat() == cudaVideoSurfaceFormat_NV12

//Initialize memory for decoded frame
NvCVImage_Init(&decoded_frame, dec.GetWidth(), dec.GetHeight(), dec.GetDeviceFramePitch(), NULL, NVCV_YUV420, NVCV_U8, NVCV_NV12, NVCV_GPU, 1);
decoded_frame.colorSpace = NVCV_709 | NVCV_VIDEO_RANGE | NVCV_CHROMA_COSITED;

//Allocate memory for BGRA frame
NvCVImage_Alloc(&BGRA_frame, dec.GetWidth(), dec.GetHeight(), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1);

decoded_frame.pixels = (void*)dec.GetFrame();

//Convert from decoded frame format(NV12) to desired format(BGRA)
NvCVImage_Transfer(&decoded_frame, &BGRA_frame, 1.f, stream, & stagingBuffer);

Note

The preceding example assumes the typical colorspace specification for HD content. SD typically uses NVCV_601. There are 8 possible combinations, and you should use the one that matches your video as described in the video header or proceed by trial and error:

  • If the colors are incorrect, swap 709 and 601.

  • If the colors are washed out or blown out, swap VIDEO and FULL.

  • If the colors are shifted horizontally, swap INTSTITIAL and COSITED.

Convert an NvCVImage Object to a Buffer for Encoding by NvEncoder#

To convert the NvCVImage object to the pixel format that is used during encoding via NvEncoder, you can call the NvCVImage_Transfer() function. The following example encodes a frame in the BGRA pixel format.

//BGRA frame is 4-channel, u8 buffer residing on the GPU
NvCVImage BGRA_frame;
NvCVImage_Alloc(&BGRA_frame, dec.GetWidth(), dec.GetHeight(), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1);

//Initialize encoder with a BGRA output pixel format
using NvEncCudaPtr = std::unique_ptr<NvEncoderCuda, std::function<void(NvEncoderCuda*)>>;
NvEncCudaPtr pEnc(new NvEncoderCuda(cuContext, dec.GetWidth(), dec.GetHeight(), NV_ENC_BUFFER_FORMAT_ARGB));
pEnc->CreateEncoder(&initializeParams);
//...

std::vector<std::vector<uint8_t>> vPacket;
//Get the address of the next input frame from the encoder
const NvEncInputFrame* encoderInputFrame = pEnc->GetNextInputFrame();

//Copy the pixel data from BGRA_frame into the input frame address obtained above
NvEncoderCuda::CopyToDeviceFrame(cuContext,
               BGRA_frame.pixels,
               BGRA_frame.pitch,
               (CUdeviceptr)encoderInputFrame->inputPtr,
               encoderInputFrame->pitch,
               pEnc->GetEncodeWidth(),
               pEnc->GetEncodeHeight(),
               CU_MEMORYTYPE_DEVICE,
               encoderInputFrame->bufferFormat,
               encoderInputFrame->chromaOffsets,
               encoderInputFrame->numChromaPlanes);
pEnc->EncodeFrame(vPacket);

Allocate an NvCVImage Object Buffer#

You can allocate the buffer for an NvCVImage object by using the NvCVImage allocation constructor or image functions. In both options, the buffer is automatically freed by the destructor when the images go out of scope.

Use the NvCVImage Allocation Constructor to Allocate a Buffer#

The NvCVImage allocation constructor creates an object to which memory has been allocated and that has been initialized. For more information, see NvCVImage Allocation Constructor.

The final three optional parameters of the allocation constructor determine the properties of the resulting NvCVImage object:

  • The pixel organization determines whether blue, green, and red are in separate planes or interleaved.

  • The memory type determines whether the buffer resides on the GPU or the CPU.

  • The byte alignment determines the gap between consecutive scanlines.

The following examples show how to use the final three optional parameters of the allocation constructor to determine the properties of the NvCVImage object.

  • This example creates an object without setting the final three optional parameters of the allocation constructor. In this object, the blue, green, and red components interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default alignment.

    NvCVImage cpuSrc(
  srcWidth,
  srcHeight,
  NVCV_BGR,
  NVCV_U8
);

  • This example creates an object with pixel organization, memory type, and byte alignment that are identical to the previous example by setting the final three optional parameters explicitly. As in the previous example, the blue, green, and red components are interleaved in each pixel, the buffer resides on the CPU, and the byte alignment is the default; that is, optimized for maximum performance.

    NvCVImage src(
  srcWidth,
  srcHeight,
  NVCV_BGR,
  NVCV_U8,
  NVCV_INTERLEAVED,
  NVCV_CPU,
  0
);

  • This example creates an object in which the blue, green, and red components are in separate planes, the buffer resides on the GPU, and the byte alignment ensures that no gap exists between one scanline and the next scanline.

    NvCVImage gpuSrc(
  srcWidth,
  srcHeight,
  NVCV_BGR,
  NVCV_U8,
  NVCV_PLANAR,
  NVCV_GPU,
  1
);

Use Image Functions to Allocate a Buffer#

By declaring an empty image, you can defer buffer allocation.

  1. Declare an empty NvCVImage object.

    NvCVImage xfr;

  2. Allocate or reallocate the buffer for the image.

    • To allocate the buffer, call the NvCVImage_Alloc() function.

      Allocate a buffer this way when the image is part of a state structure, where you won’t know the size of the image until later.

    • To reallocate the buffer, call the NvCVImage_Realloc() function.

      This function checks for an allocated buffer and reshapes the buffer if it is big enough. The function then frees the buffer and calls NvCVImage_Alloc().

Transfer Images Between CPU and GPU Buffers#

If the memory types of the input and output image buffers are different, an application can transfer images between CPU and GPU buffers.

Transfer Input Images from a CPU Buffer to a GPU Buffer#

To transfer an image from the CPU to a GPU buffer with conversion, given the following code:

NvCVImage srcCpuImg(width, height, NVCV_RGB, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU, 1);
NvCVImage dstGpuImg(width, height, NVCV_BGR, NVCV_F32, NVCV_PLANAR, NVCV_GPU, 1);

  1. Create an NvCVImage object to use as a staging GPU buffer in one of the following ways:

    • To avoid allocating memory in a video pipeline, create a GPU buffer during the initialization phase, with the same dimensions and format as the CPU image.

      NvCVImage stageImg(srcCpuImg.width, srcCpuImg.height, srcCpuImg.pixelFormat, srcCpuImg.componentType, srcCpuImg.planar, NVCV_GPU);

    • To simplify your application program code, you can declare an empty staging buffer during the initialization phase.

      NvCVImage stageImg;

      An appropriately sized buffer is allocated or reallocated as needed.

  2. Call the NvCVImage_Transfer() function to copy the contents of the source CPU buffer via the staging GPU buffer into the final GPU buffer.

    // Transfer the image from the CPU to the GPU, perhaps with conversion.
NvCVImage_Transfer(&srcCpuImg, &dstGpuImg, 1.0f, stream, &stageImg);

The same staging buffer can be reused in multiple NvCVImage_Transfer calls in various contexts regardless of the image sizes and can avoid buffer allocations if it is persistent.

Transfer Output Images from a GPU Buffer to a CPU Buffer#

To transfer an image from the GPU to a CPU buffer with conversion, given the following code:

NvCVImage srcGpuImg(width, height, NVCV_BGR, NVCV_F32, NVCV_PLANAR, NVCV_GPU, 1);
NvCVImage dstCpuImg(width, height, NVCV_BGR, NVCV_U8, NVCV_INTERLEAVED, NVCV_CPU, 1);

  1. Create an NvCVImage object to use as a staging GPU buffer in one of the following ways:

    • To avoid allocating memory in a video pipeline, create a GPU buffer during the initialization phase with the same dimensions and format as the CPU image.

      NvCVImage stageImg(dstCpuImg.width, dstCpuImg.height, dstCPUImg.pixelFormat, dstCPUImg.componentType, dstCPUImg.planar, NVCV_GPU);

    • To simplify your application program code, you can declare an empty staging buffer during the initialization phase.

      NvCVImage stageImg;

      An appropriately sized buffer is allocated or reallocated as needed.

  2. Call the NvCVImage_Transfer() function to copy the contents of the source GPU buffer via the staging GPU buffer into the destination CPU buffer.

    // Retrieve the image from the GPU to CPU, perhaps with conversion.
NvCVImage_Transfer(&srcGpuImg, &dstCpuImg, 1.0f, stream, &stageImg);

    The same staging buffer can be used repeatedly without reallocations in NvCVImage_Transfer if it is persistent.