The NvMedia VPI component provides a set of APIs to run certain Computer Vision (CV) algorithms on the NVIDIA® Tegra® PVA (Programmable Vision Accelerator).

The following diagram illustrates the sequence of tasks that applications should follow while running CV algorithms when using NvMedia VPI.

The following sections describe of sequence of tasks the above flow diagram.

Allocate, configure, and set up all resources before attempting any other VPI calls.

For example, NvMediaVPICreateProcessStereoPairDescriptor.

Run the CV algorithm. For information about the algorithms, see Algorithm APIs in this topic.

Free all resources.

This section describes the algorithms you can call during runtime to perform CV processing.

The Box Filter algorithm blurs an image using windowWidth×windowHeight normalized averaging kernel.

Blur kernel is defined by

Filter anchor point is center of the kernel.

Currently, the only supported window size is:

For information on the supported layouts and formats, see:

The Harris Corner Detector algorithm detects Harris corners on an input image [4].

The standard Harris corner detection process is applied first. After that, a non-max suppression pruning process is applied, during which a preference is given to tracked keypoints.

The user may mask image coordinates for corner detection and tracking.

Gradient filter size is 3x3, 5x5 or 7x7.

This process performs non-max suppression on all detected corners, which removes multiple or spurious responses.

This process splits the input image into a 2D cell grid. It selects a single corner with the highest response score inside the cell. If several corners within the cell have the same response score, it selects the bottom-right corner.

For complete description of the following functions and structs, see Vision Programming Interface in the API Reference for this PDK.

For information on the supported layouts and formats, see:

The Convert Motion Vectors algorithm applies a noise reduction filter and up-samples the smoothed motion vector image. It consumes a noisy motion vector image.

The motion vector surface format type is interleaved signed 16 bits for X and Y.

For complete description of the following functions and structs, see Vision Programming Interface in the API Reference for this PDK.

This algorithm currently has the following limitations and constraints:

For information on the supported layouts and formats, see:

Given a pair of rectified images from a stereo camera, the Stereo Preprocess algorithm uses high-quality dense stereo matching to produce an output image of the same resolution as the input.

You may use the outputType in the task descriptor to use this algorithm as a standalone left and right disparity computation. Alternatively, you may compute hints for the NvEnc engine as part of a more complex pipeline that may also include the Stereo Postprocess stage.

For the standalone disparity computation, the algorithm produces a left image-based and a right image-based disparity map on output0 and output1, respectively. It provides disparity values as fixed-point integrals with 5 fraction bits:

For the hint generation computation, output0 consists of the hints map and output1 consists of a confidence map. That confidence map must be provided as an input to the Stereo Postprocess stage.

For this mode, user defined parameters control the confidence map following this process:

You must also set the NVENC hints density by setting the numberOfHintsPerMB parameter to a value between 1 and 8. The higher the number of hints, the more accuracy the NvEnc engine produces.

The output image contents depend on the user parameter of the NvMediaVPIOFSTOutputType enum type. Supported values are:

The element NvMediaVPIStereoPreprocessParams::confidenceThresh filters out disparity coordinate hints that have poor confidence.

For complete descriptions of the following functions and structs, see Vision Programming Interface in the API Reference.

For information on the supported layouts and formats, see:

The Stereo Postprocess algorithm applies a noise reduction filter, using the confidence map generated in the Stereo Preprocess algorithm used on the NVENC engine output disparity map. You may choose to up-sample the pruned disparity image.

The filtering operation is robust to outliers.

For complete descriptions of the following functions and structs, see Vision Programming Interface in the API Reference.

For information on supported layouts and formats, see:

The FAST Corners Detector algorithm computes the FAST corner response and reports the strongest keypoint in each cell of a regularly spaced 2D grid. The algorithm is based on the approaches described in [1] and [2], with the modifications described below.

To classify whether the candidate point p is actually a corner, the FAST corner detector uses pixels on the Bresenham circle around a candidate point. If N contiguous pixels are brighter than the candidate point by at least a threshold value t (an input parameter strength) or darker by at least t, then the candidate point is considered to be a corner. If non-maximum suppression flag is turned on, the algorithm:

In FAST corner detection, a candidate pixel p is detected as a corner or not based on its value and the pixel values on its radius-3 Bresenham circle (16 pixels), given the following conditions:

If either of the two conditions is true, the candidate pixel p is detected as a corner.

In this implementation, the minimum number of contiguous pixels N is set to 9 (FAST9).

The value of the intensity difference threshold strengthThresh of type uint32_t must be within:

Where the range is defined with unsigned uint16.

If the nonmax_suppression input parameter is true, the algorithm computes the corner strengths (responses) for the detected corners. Otherwise, the corner strength is undefined. The corner strength (response) Cp function is defined as the largest threshold t for which the pixel p remains a corner.

If the nonmax_suppression parameter is true, the algorithm filters the detected corners by a non-maxima suppression step. The corner with coordinates (x,y) is kept if and only if the following conditions are satisfied:

See also:

For complete description of the following functions and structs, see Vision Programming Interface in the API Reference for this PDK.

For information on the supported layouts and formats, see:

The Lucas-Kanade Feature Tracker algorithm computes the optic-flow of input keypoints between two image pyramids. For more information, see [3].

The function inputs are old image pyramid, an array of keypoints detected in the bottom level of the old image pyramid and a new image pyramid. It produces a new array of keypoints in the bottom level of the new image pyramid. Each keypoint in the new keypoints array has a status indicating whether the tracking failed (i.e. tracking process completed successfully IFF status is 0).

For complete description of the following functions, see Vision Programming Interface in the API Reference for this PDK.

The Lucas-Kanade Feature Tracker has the following limitations:

For information on the supported layouts and formats, see:

The Lucas-Kanade method finds the affine motion vector V for each key point in the old image array, using the following equation:

Where Ix and Iy are computed using the Scharr gradients on the old image:

Where I is defined as the adjacent pixels to the point p(x,y) under consideration. With a given window size of M , I has M^2 points and the pixel p(x,y) is centered in the window. It is obtained by a simple difference between the same pixels in both images.

In practice, to get an accurate solution, it is necessary to iterate multiple times on this scheme (in a Newton-Raphson fashion) until:

Each iteration, It is updated to be the difference between the old image and the pixel shifted with the estimated V in the new image. At each iteration, the function needs to check if the tracking point was lost. The criteria for a lost tracking are

Currently, clients must update the output array of keypoints before applying it as the input to the next frame. They can do that by zeroing out the whole output array and then running Harris Corner Detector to re-initialize the keypoints. For more information, see Harris Corner Detector in this topic.

The Kanade-Lucas-Tomasi (KLT) Tracker algorithm estimates the 2D translation and scale changes of an image template between original template coordinates and a given reference image using the Inverse Compositional algorithm. For more information, see [5].

Inputs are an array of template bounding boxes, a translation and scale changes predictions array and a reference image. Additionally, a template image input is used to update template patches (see details below). Outputs are the translation and scale changes estimations array from the input bounding box coordinates to the reference image coordinates and the template bounding box coordinates array in the reference image.

Refer to alternative bounding box / transformation supported varients for further details on update and retrieval functions for the bounding boxes and motion arrays.

Refer to the parameters definitions for further details on configuration and tuning.

For a complete description of the following functions, see Vision Programming Interface in the API Reference for this PDK.

The Kanade Lucas Tomasi Object Template Tracker has the following limitations:

For information on the supported layouts and formats, see:

Each template bounding box defines a template image patch stored internally with the function descriptor. Those template patches are tracked in reference images based on predicted translation and scale changes. An estimation translation and scale changes from the original bounding box coordinates to reference image coordinates are computed. Each such estimation includes a tracking validity flag (tracking success or failure) and whether a template update is required, based on user-defined threshold parameters.

In many use cases the user may want to capture an object template patch in frame t and then estimate the translation and scale changes of that template patch in some other reference frame t+k.

To support this use case, the user indicates for each template bounding box whether to use a previously cropped and stored image patch associated with the bounding box or to crop and store a new image patch from the input template image. The cropped image patches are stored with the task descriptor.

Assume object obj1 detected in framet using some object detection functionality. The obj1 bounding box in framet coordinates is denoted as bbox1t.

To estimate the translation and scale changes of obj1 in framet+1 of the input bbox1t, the update flag (updateTemplate in inputBoxList) is set (see bounding box and 2D transform detailed definitions below).

The input template image (templateImage) is framet, and the reference image (referenceImage) is framet+1.

The KLT function crops the bbox1t image patch from template image, stores the cropped patch with the function descriptor, and computes the image patch estimated translation and scale changes in the reference image. Assume obj1 translation and scale changes estimation status (trackingStatus flag in outputBoxList) between framet and framet+1 was valid and the function indicates no template update is required (updateTemplate flag in outputBoxList). Further assume a second object obj2 was also detected in framet+1 with a bounding box bbox2t+1.

For the next frame, framet+2, the template image is framet+1 and the reference image is framet+2. The input box list now consists of bbox1t with an unset template update flag and bbox2t+1 with a set template update flag. For obj1 the tracker uses the already stored image patch, thus effectively estimating cumulative translation and scale changes between framet and framet+2 For obj2, first bbox2t+1 is cropped and stored. Then the translation and scale changes are estimated for that image patch between framet+1 and framet+2.

Assume that for both obj1 and obj2 tracking status is valid. Further assume that while the obj2 template update is unset, the obj1 template update indicates that a new template is needed. (See the detailed parameters definition below for an explanation of what might trigger a template update recommendation.) One possible approach to the template update for obj1 is to use the latest successfully tracked bounding box. For obj1 that would be the tracked bounding box provided in framet+2 coordinates from the output bounding box list (outputBoxList), denoted as bbox1t+2.

From the next frame, the template image is framet+2 and the reference image is framet+3. The input box list consists of bbox1t+2 and bbox2t+1 for obj1 and obj2 respectively. The bbox1t+2 template update flag is set, triggering the tracker to crop the bbox1t+2 image patch from the template image (from framet+2) and replace the existing image patch (from framet).

The KLT functionality supports two flavors of bounding boxes: (1) an axis-aligned bounding box (AABB) with a dedicated translation and scale struct, and (2) a more generalized 2D 3x3 homogenous transformation with object width and height.

For both variants, scale change is assumed with respect to the center of the bounding box. Both types of bounding boxes also include a trackingStatus flag indicating the tracking validity (1 for invalid) and a templateStatus flag indicating that the template needs an update (1 indicates update).

An AABB takes this format:

A translation with scale takes this format:

The 3x3 transform takes the following format:

The bounding box with transform takes the following format:

The 2D Convolution algorithm performs a 2D convolution operation on the input image with the provided kernel.

The functionality supports both 2D kernels and separable 1D kernels.

For complete description of the following functions and structs, see Vision Programming Interface in the API Reference for this PDK.

For information on the supported layouts and formats, see:

The Get Image Pyramid algorithm computes the image pyramid.

For complete description of the following function, see Vision Programming Interface in the API Reference for this PDK.

For information on the supported layouts and formats, see:

Pitch linear is supported by all functionalities for both input and output surfaces

Block linear is partially supported, depending on whether used for input or output. The following table provides more details.

 

The following table describes VPI support for NvMedia surface format packing types. Support depends on the VPI algorithm. Surface types not listed in this table are not supported.

When an output image is in BL format, its width must be a multiple of 64 pixels.

[1] Edward Rosten and Tom Drummond. “Machine learning for high-speed corner detection.” European Conference on Computer Vision, volume 1, pages 430-443, May 2006.

[2] Edward Rosten, Reid Porter, and Tom Drummond. “Faster and better: A machine learning approach to corner detection.” IEEE Trans. Pattern Analysis and Machine Intelligence, 32:105-119, October 2010.

[3] Jean-Yves Bouguet. Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the Algorithm, 2000.

[4] C. Harris, M. Stephens, “A Combined Corner and Edge Detector,” Proc. Alvey Vision Conf., pp. 147-151, 1988.

[5] Simon Baker, Iain Matthew, “Lucas-Kanade 20 Years On: A Unified Framework,” International Journal of Computer Vision, February 2004, Volume 56, issue 3, pp 221-255.