VPI - Vision Programming Interface

0.4.4 Release

KLT Feature Tracker


The Kanade-Lucas-Tomasi (KLT) Feature Tracker algorithm estimates the 2D translation and scale changes of an image template between original template coordinates and a given reference image using the Inverse Compositional algorithm. For more information, see [1].

Inputs are an array of template bounding boxes, a translation and scale changes predictions array and a reference image. Additionally, a template image input is used to update template patches (see details below).

Outputs are the translation and scale changes estimation array from the input bounding box coordinates to the reference image coordinates and the template bounding box coordinates array in the reference image.

Frame #1Frame #10


Each template bounding box defines a template image patch stored internally with the function descriptor. These template patches are tracked in reference images based on predicted translation and scale changes. An estimated translation and scale change from the original bounding box coordinates to reference image coordinates is computed. Each such estimation includes a tracking validity flag (tracking success or failure) and whether a template update is required, based on user-defined threshold parameters.


Due to PVA restrictions, the created VPI arrays' capacity must be 128.
  1. Initialization phase
    1. Include the header that defines the needed functions and structures.
    2. Define the input frames and input bounding boxes. Refer to VPIBoundingBox documentation for instructions on how to properly fill each bounding box given an axis-aligned bounding box, the reference frames, the input boxes and input predictions.
      size_t frame_count = /*... */;
      VPIImage *frames = /* ... */;
      size_t bbox_count = /* ... */;
      VPIBoundingBox *bboxes = /* ... */;
    3. Create the bounding box array with tracking information. For new bounding boxes, trackingStatus must be 0, indicating that bounding box tracking is valid. templateStatus must be 1, indicating that the template corresponding to this bounding box must be updated.
      VPIKLTTrackedBoundingBox tracked_bboxes[128];
      for (size_t b = 0; b < bbox_count; ++b)
      tracked_bboxes[b].bbox = bboxes[b];
      tracked_bboxes[b].trackingStatus = 0; // valid tracking
      tracked_bboxes[b].templateStatus = 1; // must update
    4. Wrap the tracked bounding box into a VPIArray. The array type must be VPI_ARRAY_TYPE_KLT_TRACKED_BOUNDING_BOX

      VPIArrayData data_bboxes;
      memset(&data_bboxes, 0, sizeof(data_bboxes));
      data_bboxes.capacity = 128;
      data_bboxes.size = bbox_count;
      data_bboxes.data = tracked_bboxes;
      VPIArray inputBoxList;
      vpiArrayCreateHostMemWrapper(&data_bboxes, 0, &inputBoxList);
    5. Create the bounding box transformation prediction array, initially filled with identity transforms, since the template matches exactly the bounding box contents in the template image.
      for (size_t i = 0; i < bbox_count; ++i)
      VPIHomographyTransform2D *xform = preds + i;
      // Identity transform.
      memset(xform, 0, sizeof(*xform));
      xform->mat3[0][0] = 1;
      xform->mat3[1][1] = 1;
      xform->mat3[2][2] = 1;
    6. Wrap this array into a VPIArray. The array type must be VPI_ARRAY_TYPE_HOMOGRAPHY_TRANSFORM_2D.
      VPIArrayData data_preds;
      memset(&data_preds, 0, sizeof(data_preds));
      data_preds.capacity = 128;
      data_preds.size = bbox_count;
      data_preds.data = preds;
      VPIArray inputPredList;
      vpiArrayCreateHostMemWrapper(&data_preds, 0, &inputPredList);
    7. Create the payload that will contain all temporary buffers needed for processing. It is assumed that all input frames have the same size, so the first frame dimensions and type are used to create the payload.
      VPIImageFormat imgFormat;
      vpiImageGetType(frames[0], &imgFormat);
      uint32_t width, height;
      vpiImageGetSize(frames[0], &width, &height);
      vpiCreateKLTFeatureTracker(VPI_BACKEND_CUDA, width, height, imgFormat, &klt);
    8. Define the configuration parameters that guide the KLT tracking process.
      params.nccThresholdUpdate = 0.8f;
      params.nccThresholdKill = 0.6f;
      params.nccThresholdStop = 1.0f;
      params.maxScaleChange = 0.2f;
      params.maxTranslationChange = 1.5f;
    9. Create the output tracked bounding box array. It will contain the estimated current frame's bounding box based on previous frame and the template information gathered so far. It also contains the bounding box current tracking status.
      VPIArray outputBoxList;
    10. Create the output estimated transforms. It will contain the transform that makes the bounding box template match the corresponding bounding box on the current (reference) frame.
      VPIArray outputEstimList;
    11. Create the stream where the algorithm will be submitted for execution.
      VPIStream stream;
      vpiStreamCreate(0, &stream);
  2. Processing phase
    1. Start of the processing loop from the second frame. The previous frame is where the algorithm fetches the tracked templates from, the current frame is where these templates are matched against.
      for (int idframe = 1; idframe < frame_count; ++idframe)
      VPIImage imgTemplate = frames[idframe - 1];
      VPIImage imgReference = frames[idframe];
    2. Submit the algorithm. The first time it's run, it will go through all input bounding boxes, crop them from the template frame and store them in the payload. Subsequent runs will either repeat the cropping and storing process for new bounding boxes added (doesn't happen in this example, but happens in the sample application), or perform the template matching on the reference frame.
      VPI_CHECK_STATUS(vpiSubmitKLTFeatureTracker(stream, klt, imgTemplate, inputBoxList, inputPredList, imgReference,
      outputBoxList, outputEstimList, &params));
    3. Wait until the processing is done.
    4. Lock the output arrays to retrieve the updated bounding boxes and the estimated transforms.
      VPIArrayData updatedBBoxData;
      vpiArrayLock(outputBoxList, VPI_LOCK_READ, &updatedBBoxData);
      VPIArrayData estimData;
      vpiArrayLock(outputEstimList, VPI_LOCK_READ, &estimData);
      VPIKLTTrackedBoundingBox *updated_bbox = (VPIKLTTrackedBoundingBox *)updatedBBoxData.data;
    5. Loop through all bounding boxes.
      for (size_t b = 0; b < bbox_count; ++b)
    6. Update bounding box statuses. If tracking was lost (trackingStatus==1), the input bounding box must also be marked as such, so subsequent KLT iterations ignore it. If the template needs to be updated (templateStatus==1), the next iteration will do the updating, or else it will perform the template matching.
      tracked_bboxes[b].trackingStatus = updated_bbox[b].trackingStatus;
      tracked_bboxes[b].templateStatus = updated_bbox[b].templateStatus;
    7. Skip bounding boxes that aren't being tracked.
      if (updated_bbox[b].trackingStatus)
    8. If template for this bounding box must be updated in next KLT iteration, the user must re-define the bounding box. There are several ways to do it. One can use a feature detector such as Harris keypoint detector to help fetch a brand-new bounding box, use updated_bbox[b] and either refine it through other means to avoid accumulating tracking errors, or simply use it as-is, which is a less robust approach, but still yields decent results. This example chooses this last, simpler approach.
      if (updated_bbox[b].templateStatus)
      tracked_bboxes[b] = updated_bbox[b];
    9. Also reset the corresponding input predicted transforms, setting it to identity, as it's now assumed that the input bounding box matches exactly the object being tracked.
      memset(&preds[b], 0, sizeof(preds[b]));
      preds[b].mat3[0][0] = 1;
      preds[b].mat3[1][1] = 1;
      preds[b].mat3[2][2] = 1;
    10. If the template doesn't need to be updated, set the input predicted transform to the one estimated by this KLT iteration.
      preds[b] = estim[b];
    11. Once all bounding boxes are updated, unlock the output arrays as they aren't needed by this iteration anymore.
    12. Since the input arrays content has been modified externally, invalidate them so that VPI discards the contents of any copies it might have made internally.
  3. Cleanup phase
    1. Free resources held by the stream, the payload, and the input and output arrays.

For more details, consult the KLT Feature Tracker API reference.

Limitations and Constraints

Constraints for specific backends supersede the ones specified for all backends.

All Backends


  • Only available on Jetson Xavier devices.
  • Input images' dimensions must be between 65x65 and 3264x2448.
  • Maximum scale change is 0.2.
  • Minimum input and output array capacities is 128.
  • Maximum number of bounding boxes is 64.
  • Maximum numberOfIterationsScaling is 20.
  • Only accepts VPI_IMAGE_FORMAT_U16 inputs whose pixel values' range is between 0 and 255.


  • Not implemented.


  1. Simon Baker, Iain Matthews, "Lucas-Kanade 20 Years On: A Unified Framework".
    International Journal of Computer Vision, February 2004, Volume 56, issue 3, pp 221-255.
void vpiArrayDestroy(VPIArray array)
Destroy an array instance.
Structure that defines the parameters for vpiCreateKLTFeatureTracker.
Definition: KLTFeatureTracker.h:106
VPIStatus vpiCreateKLTFeatureTracker(VPIBackend backend, uint32_t imageWidth, uint32_t imageHeight, VPIImageFormat imageFormat, VPIPayload *payload)
Creates payload for vpiSubmitKLTFeatureTracker.
VPIStatus vpiArrayCreate(uint32_t capacity, VPIArrayType fmt, uint32_t flags, VPIArray *array)
Create an empty array instance.
float mat3[3][3]
3x3 homogeneous matrix that defines the homography.
Definition: Types.h:385
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
Stores a bounding box that is being tracked by KLT Tracker.
Definition: Types.h:415
uint32_t numberOfIterationsScaling
Number of Inverse compositional iterations of scale estimations.
Definition: KLTFeatureTracker.h:107
Lock memory only for reading.
Definition: Types.h:447
VPIStatus vpiSubmitKLTFeatureTracker(VPIStream stream, VPIPayload payload, VPIImage templateImage, VPIArray inputBoxList, VPIArray inputPredictionList, VPIImage referenceImage, VPIArray outputBoxList, VPIArray outputEstimationList, const VPIKLTFeatureTrackerParams *params)
Runs KLT Feature Tracker on two frames.
VPIArrayType type
Type of each array element.
Definition: Array.h:118
float maxScaleChange
Maximum relative scale change.
Definition: KLTFeatureTracker.h:114
VPIStatus vpiArrayUnlock(VPIArray array)
Releases the lock on array object.
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
VPIStatus vpiArrayCreateHostMemWrapper(const VPIArrayData *arrayData, uint32_t flags, VPIArray *array)
Create an array object by wrapping an existing host memory block.
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
CUDA backend.
Definition: Types.h:91
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:190
VPIBoundingBox bbox
Bounding box being tracked.
Definition: Types.h:417
uint32_t size
Number of elements in the array.
Definition: Array.h:119
VPIHomographyTransform2D element.
Definition: Types.h:234
uint8_t trackingStatus
Tracking status of this bounding box.
Definition: Types.h:424
uint32_t capacity
Maximum number of elements that the array can hold.
Definition: Array.h:120
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
void * data
Points to the first element of the array.
Definition: Array.h:122
float nccThresholdUpdate
Threshold for requiring template update.
Definition: KLTFeatureTracker.h:108
VPIKLTTrackedBoundingBox element.
Definition: Types.h:235
Stores a generic 2D bounding box.
Definition: Types.h:404
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:196
VPIStatus vpiImageGetSize(VPIImage img, uint32_t *width, uint32_t *height)
Get the image size in pixels.
VPIKLTFeatureTrackerType trackingType
Type of KLT tracking that will be performed.
Definition: KLTFeatureTracker.h:121
Declares functions that implement the KLT Feature Tracker algorithm.
Stores information about array characteristics and content.
Definition: Array.h:117
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:208
Inverse compositional algorithm for KLT tracker.
Definition: KLTFeatureTracker.h:92
Stores a generic 2D homography transform.
Definition: Types.h:384
VPIStatus vpiArrayLock(VPIArray array, VPILockMode mode, VPIArrayData *arrayData)
Acquires the lock on array object and returns a pointer to array data.
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:172
VPIStatus vpiArrayInvalidate(VPIArray array)
Informs that the array's wrapped memory was updated outside VPI.
Pre-defined image formats.
Definition: ImageFormat.h:94
float nccThresholdStop
Threshold to stop estimating.
Definition: KLTFeatureTracker.h:110
float maxTranslationChange
Maximum relative translation change.
Definition: KLTFeatureTracker.h:118
float nccThresholdKill
Threshold to consider template tracking was lost.
Definition: KLTFeatureTracker.h:109
uint8_t templateStatus
Status of the template related to this bounding box.
Definition: Types.h:431
VPIStatus vpiImageGetType(VPIImage img, VPIImageFormat *type)
Get the image format.