VPI - Vision Programming Interface

2.0 Release

Porting from VPI-1.2 to VPI-2.0

There were some incompatible API changes between VPI-1.2 and VPI-2.0. Existing code being ported to v2.0 needs to be updated accordingly. These changes are listed below, together with instructions on how update the code.

Image wrapping

Previously there were different functions to wrap existing images, one for each type of buffer, e.g CUDA images, host images, NvBuffer, EGLImage, etc. These are now merged in only one function, vpiImageCreateWrapper. This gives VPI an easier path for supporting more formats in the future, all with a similar interface.

VPIImageData was updated to handle all these types using the tagged union idiom. Each supported image type is represented by VPIImageBufferType enum given to VPIImageData::bufferType. Then the corresponding VPIImageBuffer attribute must be filled.

For example, in VPI-1.2, one would wrap an existing CUDA pitch-linear image as follows:

VPIImageData data;
data.format = VPI_IMAGE_FORMAT_Y8_ER;
data.numPlanes = 1;
data.planes[0].data = image_ptr;
data.planes[0].width = image_width;
data.planes[0].height = image_height;
data.planes[0].pixelType = VPI_PIXEL_TYPE_DEFAULT;
VPIImage img;
vpiImageCreateCUDAMemWrapper(&data, 0, &img);

Now with VPI-2.0, the same is accomplished with the following code:

data.buffer.pitch.planes[0].data = image_ptr;
data.buffer.pitch.planes[0].width = image_width;
data.buffer.pitch.planes[0].height = image_height;
vpiImageCreateWrapper(&data, NULL, 0, &img);
#define VPI_IMAGE_FORMAT_Y8_ER
Single plane with one pitch-linear 8-bit unsigned integer channel with full-range luma (grayscale) in...
Definition: ImageFormat.h:151
#define VPI_PIXEL_TYPE_DEFAULT
Used to signal that the pixel type must be inferred from image format.
Definition: PixelType.h:83
VPIImageBuffer buffer
Stores the image contents.
Definition: Image.h:237
VPIImagePlanePitchLinear planes[VPI_MAX_PLANE_COUNT]
Data of all image planes in pitch-linear layout.
Definition: Image.h:160
VPIImageBufferPitchLinear pitch
Image stored in pitch-linear layout.
Definition: Image.h:206
void * data
Pointer to the first row of this plane.
Definition: Image.h:141
VPIImageFormat format
Image format.
Definition: Image.h:152
VPIPixelType pixelType
Type of each pixel within this plane.
Definition: Image.h:115
VPIImageBufferType bufferType
Type of image buffer.
Definition: Image.h:234
int32_t height
Height of this plane in pixels.
Definition: Image.h:123
int32_t width
Width of this plane in pixels.
Definition: Image.h:119
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageCreateWrapper(const VPIImageData *data, const VPIImageWrapperParams *params, uint64_t flags, VPIImage *img)
Create an image object by wrapping an existing memory block.
@ VPI_IMAGE_BUFFER_CUDA_PITCH_LINEAR
CUDA-accessible with planes in pitch-linear memory layout.
Definition: Image.h:175
Stores information about image characteristics and content.
Definition: Image.h:230

Similarly to images, previously EGLImages were wrapped into an VPIImage using the following code:

VPIWrapEGLImageParams params;
vpiInitWrapEGLImageParams(&params);
params.colorSpec = VPI_COLOR_SPEC_BT709_ER;
VPIImage img;
vpiImageCreateEGLImageWrapper(&eglImage, &params, 0, &img);

Now, the same is done with the following code:

data.buffer.egl = egl_image;
vpiImageCreateWrapper(&data, &params, 0, &img);
@ VPI_COLOR_SPEC_BT709_ER
Color spec defining ITU-R BT.709 standard, full range.
Definition: ColorSpec.h:184
EGLImageKHR egl
Image stored as an EGLImageKHR.
Definition: Image.h:218
VPIColorSpec colorSpec
Color spec to override the one defined by the VPIImageData wrapper.
Definition: Image.h:294
VPIStatus vpiInitImageWrapperParams(VPIImageWrapperParams *params)
Initialize VPIImageWrapperParams with default values.
@ VPI_IMAGE_BUFFER_EGLIMAGE
EGLImage.
Definition: Image.h:185
Parameters for customizing image wrapping.
Definition: Image.h:290

Array wrapping

Similary to images, array wrapping API is updated in VPI-2.0. VPIArrayData was changed to use the tagged union idiom so that it supports different kinds of arrays representation. These kinds correspond to different VPIArrayBufferType enums, and VPIArrayData::bufferType is set to the desired kind, then the corresponding VPIArrayData::buffer is filled up.

For example, in VPI-1.2 wrapping existing arrays stored in CUDA memory was done as follows:

VPIArrayData data;
data.capacity = array_capacity;
data.data = array_ptr;
data.sizePointer = array_size_ptr;
data.strideBytes = array_elem_stride;
data.type = VPI_ARRAY_TYPE_KEYPOINT;
VPIArray array;
vpiArrayCreateCUDAMemWrapper(&data, 0, &array);

With VPI-2.0, the same is done as follows:

data.buffer.aos.capacity = array_capacity;
data.buffer.aos.data = array_ptr;
data.buffer.aos.sizePointer = array_size_ptr;
data.buffer.aos.strideBytes = array_elem_stride;
VPIArray array;
vpiArrayCreateWrapper(&data, 0, &array);
VPIStatus vpiArrayCreateWrapper(const VPIArrayData *data, uint64_t flags, VPIArray *array)
Create an array object by wrapping an existing host memory block.
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:232
@ VPI_ARRAY_TYPE_KEYPOINT
VPIKeypoint element.
Definition: ArrayType.h:76
@ VPI_ARRAY_BUFFER_CUDA_AOS
CUDA-accessible array-of-structures.
Definition: Array.h:149
Stores information about array characteristics and contents.
Definition: Array.h:168

Image and Pyramid locking

Previously in VPI-1.2, image and pyramid locking were done by vpiImageLock and vpiPyramidLock respectively, and it could return the image/pyramid contents only as host-accessible pitch-linear memory.

This was changed in VPI-2.0. The new vpiImageLockData and vpiPyramidLockData can return both host- and cuda-accessible pitch-linear memory. The desired memory type is given as parameter of type VPIImageBufferType. The returned buffer pointers are stored in the given VPIImageData (or VPIPyramidData), as before, but now the changes to this struct described above are taken into account.

Additionally, when direct access to the memory buffer is needed, only image or pyramid locking, with VPI-1.2 you'd pass NULL as pointer to the VPIImageData / VPIPyramidData parameter. Now with VPI-2.0 vpiImageLock and vpiPyramidLock are used for this purpose.

For example, with VPI-1.2 locking an image for host access was done like this:

VPIImageData data;
vpiImageLock(img, VPI_LOCK_READ, &data);
int image_width = data.planes[0].width;
int image_height = data.planes[0].height;
void *img_ptr = data.planes[0].data;
/* Do something with img_ptr */
vpiImageUnlock(img);

With VPI-1.2, the same is done as follows:

int image_width = data.buffer.pitch.planes[0].width;
int image_height = data.buffer.pitch.planes[0].height;
void *img_ptr = data.buffer.pitch.planes[0].data;
/* Do something with img_ptr */
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:435

Pyramid locking works similarly.

Array locking

Similarly to images, array locking also changed. To get direct access to the memory buffer, vpiArrayLockData is used, passing the desired buffer type. vpiArrayLock is used only for array locking, buffer isn't returned.

With VPI-1.2, locking an array for host access was done like this:

VPIArrayData data;
vpiArrayLock(img, VPI_LOCK_READ, &data);
int size = *data.sizePointer;
void *arr_ptr = data.data;
/* Do something with arr_ptr */
vpiArrayUnlock(img);

With VPI-2.0, the same is accomplished with:

int size = *data.buffer.aos.sizePointer;
void *arr_ptr = data.buffer.aos.data;
/* Do something with arr_ptr */
VPIStatus vpiArrayUnlock(VPIArray array)
Releases the lock on array object.
VPIStatus vpiArrayLockData(VPIArray array, VPILockMode mode, VPIArrayBufferType bufType, VPIArrayData *data)
Acquires the lock on an array object and returns the array contents.
@ VPI_ARRAY_BUFFER_HOST_AOS
Host-accessible array-of-structures.
Definition: Array.h:146

Image and Array Invalidation

In VPI-1.2, when the contents of the wrapped images and arrays were changed directly, outside VPI, user needed to call vpiImageInvalidate and vpiArrayInvalidate, respectively.

With VPI-2.0, these functions were removed and the images/arrays must be locked for writing prior updates are made. It's not necessary to return the buffer, as the wrapped buffers can be accessed directly.

For example, in VPI-1.2, say img is a VPIImage that wraps the existing wrapped_image buffer. To update this buffer outside VPI and make sure that VPI knows about these changes, one would do:

wrapped_image[0] = 123;
vpiImageInvalidate(img);

With VPI-2.0, image locking is needed. The actual invalidation of buffers controlled internally by VPI is done when the image is unlocked, as shown below:

wrapped_image_buffer[0] = 123;
VPIStatus vpiImageLock(VPIImage img, VPILockMode mode)
Acquires the lock on an image object.
@ VPI_LOCK_READ_WRITE
Lock memory for reading and writing.
Definition: Types.h:449

This works similarly with arrays.

Perspective Warp

Perspective warp now doesn't need a payload.

VPI-1.2 code that uses this algorithm looked like this:

VPIPayload warp;
vpiCreatePerspectiveWarp(VPI_BACKEND_CUDA, &warp);
vpiSubmitPerspectiveWarp(stream, VPI_BACKEND_CUDA, warp, input, xform, output, VPI_INTERP_LINEAR, VPI_BORDER_ZERO, 0);

With VPI-2.0 this gets simplified, and only submission is needed, as shown below:

VPIStatus vpiSubmitPerspectiveWarp(VPIStream stream, uint64_t backend, VPIImage input, const VPIPerspectiveTransform xform, VPIImage output, const VPIWarpGrid *grid, VPIInterpolationType interp, VPIBorderExtension border, uint64_t flags)
Submits a Perspective Warp operation to the stream.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BORDER_ZERO
All pixels outside the image are considered to be zero.
Definition: Types.h:278
@ VPI_INTERP_LINEAR
Linear interpolation.
Definition: Interpolation.h:93

Erode / Dilate

In VPI-1.2, the erode and dilate submission functions expected user to pass VPI_BORDER_INVALID when the algorithm shouldn't consider pixels outside the image boundary, as shown below:

vpiSubmitErode(stream, VPI_BACKEND_CPU, input, output, NULL, 3, 3, VPI_BORDER_INVALID);
vpiSubmitDilate(stream, VPI_BACKEND_CPU, input, output, NULL, 3, 3, VPI_BORDER_INVALID);

For VPI-2.0, such code must be changed to:

vpiSubmitErode(stream, VPI_BACKEND_CPU, input, output, NULL, 3, 3, VPI_BORDER_LIMITED);
vpiSubmitDilate(stream, VPI_BACKEND_CPU, input, output, NULL, 3, 3, VPI_BORDER_LIMITED);
VPIStatus vpiSubmitDilate(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const int8_t *kernelData, int32_t kernelWidth, int32_t kernelHeight, VPIBorderExtension border)
Runs a 2D dilate over an image.
VPIStatus vpiSubmitErode(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const int8_t *kernelData, int32_t kernelWidth, int32_t kernelHeight, VPIBorderExtension border)
Runs a 2D erode over an image.
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:92
@ VPI_BORDER_LIMITED
Consider image as limited to not access outside pixels.
Definition: Types.h:282