Wrap Raw Pointers

Overview

The Wrap Raw Pointers sample demonstrates how to wrap externally allocated memory buffers into VPI images for integration with existing pipelines. This is particularly useful when integrating VPI with other frameworks or when working with pre-allocated memory from hardware accelerators.

The sample shows three different memory types that can be wrapped:

• CUDA pointers: Device memory allocated via cudaMalloc
• CPU pointers: Host memory allocated via malloc
• NvBufSurface: NVIDIA buffer surfaces used in multimedia pipelines (Jetson platform)

Wrapping external memory can avoid unnecessary memory copies when the selected backend can access the wrapped allocation directly. The sample applies a vertical image flip operation to demonstrate processing on the wrapped buffers.

Instructions

The command line parameters are:

<input_image> <pointer_type>

where

• input_image: input image file name; accepts png, jpeg, and other common formats.
• pointer_type: type of pointer to wrap; one of cuda_ptr, cpu_ptr, or nvbufsurface.

Note

The output will be in grayscale as this sample uses single-channel Y8 format.

The nvbufsurface option is only available on Jetson platforms with NvBufSurface support.

Here are some examples:

• C++ (CUDA pointer)

  ./vpi_sample_21_wrap_raw_ptr ../assets/kodim08.png cuda_ptr

• C++ (CPU pointer)

  ./vpi_sample_21_wrap_raw_ptr ../assets/kodim08.png cpu_ptr

• C++ (NvBufSurface - Jetson only)

  ./vpi_sample_21_wrap_raw_ptr ../assets/kodim08.png nvbufsurface

The output is saved as output.png in the current directory.

Features

• Interop Wrapping: Wrap existing memory buffers and process them with a backend that can access that memory type
• Multiple Memory Types: Support for CUDA device memory, CPU host memory, and NvBufSurface
• Backend Selection: Automatically selects the appropriate VPI backend based on pointer type:
  • CUDA backend for CUDA pointers
  • CPU backend for CPU pointers
  • VIC backend for NvBufSurface (Jetson)
• OpenCV Integration: Demonstrates loading images with OpenCV and transferring to wrapped VPI buffers
• Image Processing: Applies vertical flip operation on wrapped buffers

Internal Copies

The wrapper itself does not copy the external allocation. Copies can still happen later if an algorithm backend needs a memory representation that cannot be shared with the wrapped allocation.

In particular, a CUDA pitch-linear pointer wrapped as a VPIImage is a good match for CUDA backend work. If that same image is then consumed by PVA, VIC, OFA, or another Tegra engine, VPI may create an internal backend-compatible allocation and copy the data before or after the operation. Similarly, wrapping an NvBufSurface may require a copy when VPI needs an image backed by NvSciBuf for another backend.

To avoid these copies in CUDA/VPI interop pipelines, prefer the reverse direction: allocate the image with vpiImageCreate, run VPI operations on that VPI-owned image, and lock out the CUDA view with vpiImageLockData using VPI_IMAGE_BUFFER_CUDA_PITCH_LINEAR when CUDA code needs to access it. Keep the returned VPIImageBufferPitchLinear data only while the image is locked. For multimedia or NvSci pipelines, apply the same rule: create in VPI first, then lock or extract the interop handle that the external code needs.

Workflow

1. Load input image using OpenCV (grayscale)
2. Allocate memory buffer based on specified pointer type
3. Copy image data to the allocated buffer
4. Create VPI image wrapper around the external buffer
5. Submit image flip operation to VPI stream
6. Synchronize and copy result back to OpenCV image
7. Save output image

Results

The sample produces a vertically flipped grayscale version of the input image saved as output.png.

Source Code

For convenience, here's the code that is also installed in the samples directory.

#include <opencv2/core/version.hpp>
#if CV_MAJOR_VERSION >= 3
#    include <opencv2/imgcodecs.hpp>
#else
#    include <opencv2/contrib/contrib.hpp> // for colormap
#    include <opencv2/highgui/highgui.hpp>
#endif
 
#include <cuda_runtime.h>
#include <nvbufsurface.h>
#include <vpi/Context.h>
#include <vpi/Image.h>
#include <vpi/Stream.h>
#include <vpi/algo/ImageFlip.h>
 
#include <algorithm>
#include <cctype>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <sstream>
#include <stdexcept>
#include <string>
#include <thread>
 
#define CHECK_STATUS(STMT)                                                                  \
    do                                                                                      \
    {                                                                                       \
        VPIStatus status = (STMT);                                                          \
        if (status != VPI_SUCCESS)                                                          \
        {                                                                                   \
            char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH];                                     \
            vpiGetLastStatusMessage(buffer, sizeof(buffer));                                \
            std::ostringstream ss;                                                          \
            ss << "line " << __LINE__ << " " << vpiStatusGetName(status) << ": " << buffer; \
            throw std::runtime_error(ss.str());                                             \
        }                                                                                   \
    } while (0);
 
namespace {
 
VPIImageData GetGenericPLData(int imgWidth, int imgHeight, VPIImageBufferType bufferType, VPIByte *pBase)
{
    VPIImageBufferPitchLinear imgData = {};
    imgData.format                    = VPI_IMAGE_FORMAT_Y8;
    imgData.numPlanes                 = 1;
    imgData.planes[0].width           = imgWidth;
    imgData.planes[0].height          = imgHeight;
    imgData.planes[0].pitchBytes      = sizeof(uint8_t) * imgWidth;
    imgData.planes[0].pixelType       = VPI_PIXEL_TYPE_INVALID;
    imgData.planes[0].offsetBytes     = 0;
    imgData.planes[0].pBase           = pBase;
    VPIImageData vpiImgData           = {};
    vpiImgData.bufferType             = bufferType;
    vpiImgData.buffer.pitch           = imgData;
    return vpiImgData;
}
 
} // namespace
 
int main(int argc, char *argv[])
{
    // =============================
    // Parse command line parameters
    if (argc != 3)
    {
        throw std::runtime_error(std::string("Usage: ") + argv[0] + " " +
                                 "<input_image> <cuda_ptr|cpu_ptr|nvbufsurface>");
    }
    std::string imgName   = argv[1];
    std::string ptrToWrap = argv[2];
    VPIBackend backend;
 
    if (ptrToWrap == "cuda_ptr")
    {
        backend = VPI_BACKEND_CUDA;
    }
    else if (ptrToWrap == "cpu_ptr")
    {
        backend = VPI_BACKEND_CPU;
    }
    else if (ptrToWrap == "nvbufsurface")
    {
        backend = VPI_BACKEND_VIC;
    }
    else
    {
        throw std::runtime_error("Pointer to wrap must be one of cuda_ptr, cpu_ptr or nvbufsurface");
    }
 
    VPIContext context = nullptr;
    CHECK_STATUS(vpiContextCreate(backend | VPI_BACKEND_CPU, &context));
    CHECK_STATUS(vpiContextSetCurrent(context));
 
    // =============================
    // Read input image
    cv::Mat image = cv::imread(imgName, cv::IMREAD_GRAYSCALE);
    if (image.empty())
    {
        throw std::runtime_error("Failed to read input image: " + imgName);
    }
    uint32_t imgWidth  = image.cols;
    uint32_t imgHeight = image.rows;
 
    // =============================
    // Allocate pointers to be wrapped and create wrapped images
    VPIImage bufIn = nullptr, bufOut = nullptr;
    VPIImageData dataIn = {}, dataOut = {};
    uint8_t *cudaIn = nullptr, *cudaOut = nullptr;
    NvBufSurface *surfIn = nullptr, *surfOut = nullptr;
    uint8_t *cpuIn = nullptr, *cpuOut = nullptr;
    VPIImageWrapperParams wrapParams = {};
    wrapParams.colorSpec             = VPI_COLOR_SPEC_DEFAULT;
    if (ptrToWrap == "nvbufsurface")
    {
        // =============================
        // Allocate NvBufSurface
        NvBufSurfaceCreateParams nv_buf_params{};
        nv_buf_params.width       = imgWidth;
        nv_buf_params.height      = imgHeight;
        nv_buf_params.memType     = NVBUF_MEM_SURFACE_ARRAY;
        nv_buf_params.layout      = NVBUF_LAYOUT_PITCH;
        nv_buf_params.colorFormat = NVBUF_COLOR_FORMAT_GRAY8;
        if (0 != NvBufSurfaceCreate(&surfIn, 1, &nv_buf_params))
        {
            throw std::runtime_error("NvBufSurfaceCreate failed");
        }
        if (0 != NvBufSurfaceCreate(&surfOut, 1, &nv_buf_params))
        {
            throw std::runtime_error("NvBufSurfaceCreate failed");
        }
 
        // =============================
        // Copy over data from OpenCV image to NvBufSurface
        if (0 != Raw2NvBufSurface(image.data, 0, 0, imgWidth, imgHeight, surfIn))
        {
            throw std::runtime_error("Copying to NvBufSurface failed");
        }
 
        // =============================
        // Wrap NvBufSurfaces directly in VPIImage
        dataIn.bufferType = VPI_IMAGE_BUFFER_NVBUFFER;
        dataIn.buffer.fd  = surfIn->surfaceList[0].bufferDesc;
        CHECK_STATUS(vpiImageCreateWrapper(&dataIn, &wrapParams, backend | VPI_BACKEND_CPU, &bufIn));
 
        dataOut.bufferType = VPI_IMAGE_BUFFER_NVBUFFER;
        dataOut.buffer.fd  = surfOut->surfaceList[0].bufferDesc;
        CHECK_STATUS(vpiImageCreateWrapper(&dataOut, &wrapParams, backend | VPI_BACKEND_CPU, &bufOut));
    }
    else if (ptrToWrap == "cuda_ptr")
    {
        // =============================
        // Allocate CUDA pitch
        if (0 != cudaMalloc(&cudaIn, imgWidth * imgHeight * sizeof(uint8_t)))
        {
            throw std::runtime_error("Allocating CUDA pitch failed");
        }
        if (0 != cudaMalloc(&cudaOut, imgWidth * imgHeight * sizeof(uint8_t)))
        {
            throw std::runtime_error("Allocating CUDA pitch failed");
        }
 
        // =============================
        // Copy over data from OpenCV image to input CUDA pitch
        if (0 != cudaMemcpy2D(cudaIn, imgWidth, image.data, imgWidth, imgWidth, imgHeight, cudaMemcpyHostToDevice))
        {
            throw std::runtime_error("Copying to CUDA pitch failed");
        }
 
        // =============================
        // Wrap CUDA pitches directly in VPIImage
        dataIn  = GetGenericPLData(imgWidth, imgHeight, VPI_IMAGE_BUFFER_CUDA_PITCH_LINEAR, cudaIn);
        dataOut = GetGenericPLData(imgWidth, imgHeight, VPI_IMAGE_BUFFER_CUDA_PITCH_LINEAR, cudaOut);
        CHECK_STATUS(vpiImageCreateWrapper(&dataIn, &wrapParams, backend | VPI_BACKEND_CPU, &bufIn));
        CHECK_STATUS(vpiImageCreateWrapper(&dataOut, &wrapParams, backend | VPI_BACKEND_CPU, &bufOut));
    }
    else
    {
        // =============================
        // Allocate CPU pitch
        cpuIn  = static_cast<uint8_t *>(malloc(imgWidth * imgHeight * sizeof(uint8_t)));
        cpuOut = static_cast<uint8_t *>(malloc(imgWidth * imgHeight * sizeof(uint8_t)));
 
        // =============================
        // Copy over data from OpenCV image to input CPU pitch
        memcpy(cpuIn, image.data, imgWidth * imgHeight);
 
        // =============================
        // Wrap CUDA pitches directly in VPIImage
        dataIn  = GetGenericPLData(imgWidth, imgHeight, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, cpuIn);
        dataOut = GetGenericPLData(imgWidth, imgHeight, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, cpuOut);
        CHECK_STATUS(vpiImageCreateWrapper(&dataIn, &wrapParams, backend | VPI_BACKEND_CPU, &bufIn));
        CHECK_STATUS(vpiImageCreateWrapper(&dataOut, &wrapParams, backend | VPI_BACKEND_CPU, &bufOut));
    }
 
    // =============================
    // Create VPI Stream
    VPIStream stream;
    CHECK_STATUS(vpiStreamCreate(0, &stream));
 
    // =============================
    // Submit image flip
    CHECK_STATUS(vpiSubmitImageFlip(stream, backend, bufIn, bufOut, VPI_FLIP_VERT));
 
    // =============================
    // Sync stream
    CHECK_STATUS(vpiStreamSync(stream));
 
    // =============================
    // Copy back to OpenCV Image and save output. Zero copying between VPI and wrapped type!
    if (ptrToWrap == "nvbufsurface")
    {
        if (0 != NvBufSurface2Raw(surfOut, 0, 0, imgWidth, imgHeight, image.data))
        {
            throw std::runtime_error("Copying from NvBufSurface failed");
        }
    }
    else if (ptrToWrap == "cuda_ptr")
    {
        if (0 != cudaMemcpy2D(image.data, imgWidth, cudaOut, imgWidth, imgWidth, imgHeight, cudaMemcpyDeviceToHost))
        {
            throw std::runtime_error("Copying to CUDA pitch failed");
        }
    }
    else
    {
        memcpy(image.data, cpuOut, imgWidth * imgHeight);
    }
    cv::imwrite("output.png", image);
 
    // =============================
    // Cleanup
    vpiStreamDestroy(stream);
    vpiImageDestroy(bufIn); // note that destroying a wrapper image will not free the underlying memory
    vpiImageDestroy(bufOut);
    if (ptrToWrap == "nvbufsurface")
    {
        NvBufSurfaceDestroy(surfIn);
        NvBufSurfaceDestroy(surfOut);
    }
    else if (ptrToWrap == "cuda_ptr")
    {
        cudaFree(cudaIn);
        cudaFree(cudaOut);
    }
    else
    {
        free(cpuIn);
        free(cpuOut);
    }
    vpiContextDestroy(context);
 
    if (ptrToWrap == "cuda_ptr")
    {
        // The Jetson CUDA/EGL driver stack can abort in process-global finalizers
        // after this CUDA-runtime sample has already released its resources.
        image.release();
        std::cout.flush();
        std::cerr.flush();
        std::_Exit(EXIT_SUCCESS);
    }
}