VPI - Vision Programming Interface

1.0 Release

KLT Bounding Box Tracker

Overview

This application tracks bounding boxes on an input video, draws the frames on each frame and saves them to disk. The user can define what backend will be used for processing.

Note
The output will be in grayscale as the algorithm currently only supports grayscales.

This sample shows the following:

  • Creating and destroying a VPI stream.
  • Wrapping a OpenCV cv::Mat image to be used with VPI.
  • Wrapping an array hosted on CPU (input bounding boxes) to be used by VPI.
  • Creating a VPI-managed 2D image where output will be written to.
  • Use the multi-frame KLT Bounding Box Tracker algorithm.
  • Simple stream synchronization.
  • Array locking to access its contents from CPU side.
  • Error handling.
  • Environment clean up using user-defined context.

Instructions

The usage is:

./vpi_sample_06_klt_tracker <backend> <input video> <input bboxes> <output frames>

where

  • backend: either cpu, cuda or pva; it defines the backend that will perform the processing.
  • input video: input video file name, it accepts all video types that OpenCV's cv::VideoCapture accepts.
  • input bboxes: file with input bounding boxes and in what frame they appear. The file is composed of multiple lines with the following format:
       <frame> <bbox_x> <bbox_y> <bbox_width> <bbox_height>
    It's important that the lines are sorted with frames in ascending order.
  • output frames: the file name that will be used for the output frames. Example: output.png will generate frames output_0000.png, output_0001.png, output_0002.png, and so on.

Here's one example:

./vpi_sample_06_klt_tracker cuda ../assets/dashcam.mp4 ../assets/dashcam_bboxes.txt frame.png

This is using the CUDA backend and one of the provided sample videos and bounding boxes.

Results

Frame 0445Frame 0465

Source Code

For convenience, here's the code that is also installed in the samples directory.

1 /*
2 * Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without
5 * modification, are permitted provided that the following conditions
6 * are met:
7 * * Redistributions of source code must retain the above copyright
8 * notice, this list of conditions and the following disclaimer.
9 * * Redistributions in binary form must reproduce the above copyright
10 * notice, this list of conditions and the following disclaimer in the
11 * documentation and/or other materials provided with the distribution.
12 * * Neither the name of NVIDIA CORPORATION nor the names of its
13 * contributors may be used to endorse or promote products derived
14 * from this software without specific prior written permission.
15 *
16 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 */
28 
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 # include <opencv2/videoio.hpp>
33 #else
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/imgproc/imgproc.hpp>
38 #include <vpi/OpenCVInterop.hpp>
39 
40 #include <vpi/Array.h>
41 #include <vpi/Context.h>
42 #include <vpi/Image.h>
43 #include <vpi/Status.h>
44 #include <vpi/Stream.h>
46 
47 #include <cstring> // for memset
48 #include <fstream>
49 #include <iostream>
50 #include <map>
51 #include <sstream>
52 #include <vector>
53 
54 #define CHECK_STATUS(STMT) \
55  do \
56  { \
57  VPIStatus status = (STMT); \
58  if (status != VPI_SUCCESS) \
59  { \
60  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
61  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
62  std::ostringstream ss; \
63  ss << vpiStatusGetName(status) << ": " << buffer; \
64  throw std::runtime_error(ss.str()); \
65  } \
66  } while (0);
67 
68 // Utility function to wrap a cv::Mat into a VPIImage. If 'image'
69 // is NULL, it'll create the wrapper, or else, it'll update 'image's wrapped
70 // frame with the new frame, without allocating memory.
71 static VPIImage ToVPIImage(VPIImage image, const cv::Mat &frame)
72 {
73  if (image == nullptr)
74  {
75  // Ceate a VPIImage that wraps the frame
76  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(frame, 0, &image));
77  }
78  else
79  {
80  // reuse existing VPIImage wrapper to wrap the new frame.
81  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(image, frame));
82  }
83  return image;
84 }
85 
86 // Utility to draw the bounding boxes into an image and save it to disk.
87 static void SaveKLTBoxes(VPIImage img, VPIArray boxes, VPIArray preds, const std::string &filename, int frame)
88 {
89  // Convert img into a cv::Mat
90  cv::Mat out;
91  {
92  VPIImageData imgdata;
93  CHECK_STATUS(vpiImageLock(img, VPI_LOCK_READ, &imgdata));
94 
95  int cvtype;
96  switch (imgdata.format)
97  {
99  cvtype = CV_8U;
100  break;
101 
102  case VPI_IMAGE_FORMAT_S8:
103  cvtype = CV_8S;
104  break;
105 
107  cvtype = CV_16UC1;
108  break;
109 
111  cvtype = CV_16SC1;
112  break;
113 
114  default:
115  throw std::runtime_error("Image type not supported");
116  }
117 
118  cv::Mat cvimg(imgdata.planes[0].height, imgdata.planes[0].width, cvtype, imgdata.planes[0].data,
119  imgdata.planes[0].pitchBytes);
120 
121  if (cvimg.type() == CV_16U)
122  {
123  cvimg.convertTo(out, CV_8U);
124  cvimg = out;
125  out = cv::Mat();
126  }
127 
128  cvtColor(cvimg, out, cv::COLOR_GRAY2BGR);
129 
130  CHECK_STATUS(vpiImageUnlock(img));
131  }
132 
133  // Now draw the bounding boxes.
134  VPIArrayData boxdata;
135  CHECK_STATUS(vpiArrayLock(boxes, VPI_LOCK_READ, &boxdata));
136 
137  VPIArrayData preddata;
138  CHECK_STATUS(vpiArrayLock(preds, VPI_LOCK_READ, &preddata));
139 
140  auto *pboxes = reinterpret_cast<VPIKLTTrackedBoundingBox *>(boxdata.data);
141  auto *ppreds = reinterpret_cast<VPIHomographyTransform2D *>(preddata.data);
142 
143  srand(0);
144  for (int i = 0; i < *boxdata.sizePointer; ++i)
145  {
146  if (pboxes[i].trackingStatus == 1)
147  {
148  // So that the colors assigned to bounding boxes don't change
149  // when some bbox isn't tracked anymore.
150  rand();
151  rand();
152  rand();
153  continue;
154  }
155 
156  float x, y, w, h;
157  x = pboxes[i].bbox.xform.mat3[0][2] + ppreds[i].mat3[0][2];
158  y = pboxes[i].bbox.xform.mat3[1][2] + ppreds[i].mat3[1][2];
159  w = pboxes[i].bbox.width * pboxes[i].bbox.xform.mat3[0][0] * ppreds[i].mat3[0][0];
160  h = pboxes[i].bbox.height * pboxes[i].bbox.xform.mat3[1][1] * ppreds[i].mat3[1][1];
161 
162  rectangle(out, cv::Rect(x, y, w, h), cv::Scalar(rand() % 256, rand() % 256, rand() % 256), 2);
163  }
164 
165  CHECK_STATUS(vpiArrayUnlock(preds));
166  CHECK_STATUS(vpiArrayUnlock(boxes));
167 
168  // Create the output file name
169  std::string fname = filename;
170  int ext = fname.rfind('.');
171 
172  char buffer[512] = {};
173  snprintf(buffer, sizeof(buffer) - 1, "%s_%04d%s", fname.substr(0, ext).c_str(), frame, fname.substr(ext).c_str());
174 
175  // Finally, write frame to disk
176  if (!imwrite(buffer, out, {cv::IMWRITE_JPEG_QUALITY, 70}))
177  {
178  throw std::runtime_error("Can't write to " + std::string(buffer));
179  }
180 }
181 
182 int main(int argc, char *argv[])
183 {
184  // We'll create all our objects under this context, so that
185  // we don't have to track what objects to destroy. Just destroying
186  // the context will destroy all objects.
187  VPIContext ctx = nullptr;
188 
189  int retval = 0;
190 
191  try
192  {
193  if (argc != 5)
194  {
195  throw std::runtime_error(std::string("Usage: ") + argv[0] +
196  " <cpu|pva|cuda> <input_video> <bbox descr> <output>");
197  }
198 
199  std::string strBackend = argv[1];
200  std::string strInputVideo = argv[2];
201  std::string strInputBBoxes = argv[3];
202  std::string strOutputFiles = argv[4];
203 
204  // Load the input video
205  cv::VideoCapture invid;
206  if (!invid.open(strInputVideo))
207  {
208  throw std::runtime_error("Can't open '" + strInputVideo + "'");
209  }
210 
211  // Create our context.
212  CHECK_STATUS(vpiContextCreate(0, &ctx));
213 
214  // Activate it. From now on all created objects will be owned by it.
215  CHECK_STATUS(vpiContextSetCurrent(ctx));
216 
217  // Load the bounding boxes
218  // Format is: <frame number> <bbox_x> <bbox_y> <bbox_width> <bbox_height>
219  // Important assumption: bboxes must be sorted with increasing frame numbers.
220 
221  // Arrays that will store our input bboxes and predicted transform.
222  VPIArray inputBoxList, inputPredList;
223 
224  // These arrays will actually wrap these vectors.
225  std::vector<VPIKLTTrackedBoundingBox> bboxes;
226  int32_t bboxesSize = 0;
227  std::vector<VPIHomographyTransform2D> preds;
228  int32_t predsSize = 0;
229 
230  // Stores how many bboxes there are in each frame. Only
231  // stores when the bboxes count change.
232  std::map<int, size_t> bboxes_size_at_frame; // frame -> bbox count
233 
234  // PVA requires that array capacity is 128.
235  bboxes.reserve(128);
236  preds.reserve(128);
237 
238  // Read bounding boxes
239  {
240  std::ifstream in(strInputBBoxes);
241  if (!in)
242  {
243  throw std::runtime_error("Can't open '" + strInputBBoxes + "'");
244  }
245 
246  // For each bounding box,
247  int frame, x, y, w, h;
248  while (in >> frame >> x >> y >> w >> h)
249  {
250  if (bboxes.size() == 64)
251  {
252  throw std::runtime_error("Too many bounding boxes");
253  }
254 
255  // Convert the axis-aligned bounding box into our tracking
256  // structure.
257 
258  VPIKLTTrackedBoundingBox track = {};
259  // scale
260  track.bbox.xform.mat3[0][0] = 1;
261  track.bbox.xform.mat3[1][1] = 1;
262  // position
263  track.bbox.xform.mat3[0][2] = x;
264  track.bbox.xform.mat3[1][2] = y;
265  // must be 1
266  track.bbox.xform.mat3[2][2] = 1;
267 
268  track.bbox.width = w;
269  track.bbox.height = h;
270  track.trackingStatus = 0; // valid tracking
271  track.templateStatus = 1; // must update
272 
273  bboxes.push_back(track);
274 
275  // Identity predicted transform.
276  VPIHomographyTransform2D xform = {};
277  xform.mat3[0][0] = 1;
278  xform.mat3[1][1] = 1;
279  xform.mat3[2][2] = 1;
280  preds.push_back(xform);
281 
282  bboxes_size_at_frame[frame] = bboxes.size();
283  }
284 
285  if (!in && !in.eof())
286  {
287  throw std::runtime_error("Can't parse bounding boxes, stopped at bbox #" +
288  std::to_string(bboxes.size()));
289  }
290 
291  // Wrap the input arrays into VPIArray's
292  VPIArrayData data = {};
294  data.capacity = bboxes.capacity();
295  data.sizePointer = &bboxesSize;
296  data.data = &bboxes[0];
297  CHECK_STATUS(vpiArrayCreateHostMemWrapper(&data, 0, &inputBoxList));
298 
300  data.sizePointer = &predsSize;
301  data.data = &preds[0];
302  CHECK_STATUS(vpiArrayCreateHostMemWrapper(&data, 0, &inputPredList));
303  }
304 
305  // Now parse the backend
306  VPIBackend backendType;
307 
308  if (strBackend == "cpu")
309  {
310  backendType = VPI_BACKEND_CPU;
311  }
312  else if (strBackend == "cuda")
313  {
314  backendType = VPI_BACKEND_CUDA;
315  }
316  else if (strBackend == "pva")
317  {
318  backendType = VPI_BACKEND_PVA;
319  }
320  else
321  {
322  throw std::runtime_error("Backend '" + strBackend +
323  "' not recognized, it must be either cpu, cuda or pva.");
324  }
325 
326  // Create the stream for the given backend.
327  VPIStream stream;
328  CHECK_STATUS(vpiStreamCreate(backendType, &stream));
329 
330  // Helper function to fetch a frame from input
331  int nextFrame = 0;
332  auto fetchFrame = [&invid, &nextFrame, backendType]() {
333  cv::Mat frame;
334  if (!invid.read(frame))
335  {
336  return cv::Mat();
337  }
338 
339  // We only support grayscale inputs
340  if (frame.channels() == 3)
341  {
342  cvtColor(frame, frame, cv::COLOR_BGR2GRAY);
343  }
344 
345  if (backendType == VPI_BACKEND_PVA)
346  {
347  // PVA only supports 16-bit unsigned inputs,
348  // where each element is in 0-255 range, so
349  // no rescaling needed.
350  cv::Mat aux;
351  frame.convertTo(aux, CV_16U);
352  frame = aux;
353  }
354  else
355  {
356  assert(frame.type() == CV_8U);
357  }
358 
359  ++nextFrame;
360  return frame;
361  };
362 
363  // Fetch the first frame and wrap it into a VPIImage.
364  // Templates will be based on this frame.
365  cv::Mat cvTemplate = fetchFrame(), cvReference;
366  VPIImage imgTemplate = ToVPIImage(nullptr, cvTemplate);
367 
368  VPIImageFormat imgFormat;
369  CHECK_STATUS(vpiImageGetFormat(imgTemplate, &imgFormat));
370 
371  // Using this first frame's characteristics, create a KLT Bounding Box Tracker payload.
372  // We're limiting the template dimensions to 64x64.
373  VPIPayload klt;
374  CHECK_STATUS(vpiCreateKLTFeatureTracker(backendType, cvTemplate.cols, cvTemplate.rows, imgFormat, NULL, &klt));
375 
376  // Parameters we'll use. No need to change them on the fly, so just define them here.
377  VPIKLTFeatureTrackerParams params = {};
378  params.numberOfIterationsScaling = 20;
379  params.nccThresholdUpdate = 0.8f;
380  params.nccThresholdKill = 0.6f;
381  params.nccThresholdStop = 1.0f;
382  params.maxScaleChange = 0.2f;
383  params.maxTranslationChange = 1.5f;
385 
386  // Output array with estimated bbox for current frame.
387  VPIArray outputBoxList;
388  CHECK_STATUS(vpiArrayCreate(128, VPI_ARRAY_TYPE_KLT_TRACKED_BOUNDING_BOX, 0, &outputBoxList));
389 
390  // Output array with estimated transform of input bbox to match output bbox.
391  VPIArray outputEstimList;
392  CHECK_STATUS(vpiArrayCreate(128, VPI_ARRAY_TYPE_HOMOGRAPHY_TRANSFORM_2D, 0, &outputEstimList));
393 
394  // Reference (current) frame.
395  VPIImage imgReference = nullptr;
396 
397  size_t curNumBoxes = 0;
398 
399  do
400  {
401  size_t curFrame = nextFrame - 1;
402 
403  // Get the number of bounding boxes in current frame.
404  auto tmp = --bboxes_size_at_frame.upper_bound(curFrame);
405  size_t bbox_count = tmp->second;
406 
407  assert(bbox_count >= curNumBoxes && "input bounding boxes must be sorted by frame");
408 
409  // Does current frame have new bounding boxes?
410  if (curNumBoxes != bbox_count)
411  {
412  // Update the input array sizes, the new frame is already there as we populated
413  // these arrays with all input bounding boxes.
414  CHECK_STATUS(vpiArrayLock(inputBoxList, VPI_LOCK_READ_WRITE, nullptr));
415  CHECK_STATUS(vpiArraySetSize(inputBoxList, bbox_count));
416  CHECK_STATUS(vpiArrayUnlock(inputBoxList));
417 
418  CHECK_STATUS(vpiArrayLock(inputPredList, VPI_LOCK_READ_WRITE, nullptr));
419  CHECK_STATUS(vpiArraySetSize(inputPredList, bbox_count));
420  CHECK_STATUS(vpiArrayUnlock(inputPredList));
421 
422  for (size_t i = 0; i < bbox_count - curNumBoxes; ++i)
423  {
424  std::cout << curFrame << " -> new " << curNumBoxes + i << std::endl;
425  }
426  assert(bbox_count <= bboxes.capacity());
427  assert(bbox_count <= preds.capacity());
428 
429  curNumBoxes = bbox_count;
430  }
431 
432  // Save this frame to disk.
433  SaveKLTBoxes(imgTemplate, inputBoxList, inputPredList, strOutputFiles, curFrame);
434 
435  // Fetch a new frame
436  cvReference = fetchFrame();
437 
438  // Video ended?
439  if (cvReference.data == nullptr)
440  {
441  // Just end gracefully.
442  break;
443  }
444 
445  // Wrap frame into a VPIImage
446  imgReference = ToVPIImage(imgReference, cvReference);
447 
448  // Estimate the bounding boxes in current frame (reference) given their position in previous
449  // frame (template).
450  CHECK_STATUS(vpiSubmitKLTFeatureTracker(stream, backendType, klt, imgTemplate, inputBoxList, inputPredList,
451  imgReference, outputBoxList, outputEstimList, &params));
452 
453  // Wait for processing to finish.
454  CHECK_STATUS(vpiStreamSync(stream));
455 
456  // Now we lock the output arrays to properly set up the input for the next iteration.
457  VPIArrayData updatedBBoxData;
458  CHECK_STATUS(vpiArrayLock(outputBoxList, VPI_LOCK_READ, &updatedBBoxData));
459 
460  VPIArrayData estimData;
461  CHECK_STATUS(vpiArrayLock(outputEstimList, VPI_LOCK_READ, &estimData));
462 
463  auto *updated_bbox = reinterpret_cast<VPIKLTTrackedBoundingBox *>(updatedBBoxData.data);
464  auto *estim = reinterpret_cast<VPIHomographyTransform2D *>(estimData.data);
465 
466  // For each bounding box,
467  for (size_t b = 0; b < curNumBoxes; ++b)
468  {
469  // Did tracking failed?
470  if (updated_bbox[b].trackingStatus)
471  {
472  // Do we have to update the input bbox's tracking status too?
473  if (bboxes[b].trackingStatus == 0)
474  {
475  std::cout << curFrame << " -> dropped " << b << std::endl;
476  bboxes[b].trackingStatus = 1;
477  }
478 
479  continue;
480  }
481 
482  // Must update template for this bounding box??
483  if (updated_bbox[b].templateStatus)
484  {
485  std::cout << curFrame << " -> update " << b << std::endl;
486 
487  // There are usually two approaches here:
488  // 1. Redefine the bounding box using a feature detector such as
489  // \ref algo_harris_corners "Harris keypoint detector", or
490  // 2. Use updated_bbox[b], which is still valid, although tracking
491  // errors might accumulate over time.
492  //
493  // We'll go to the second option, less robust, but simple enough
494  // to implement.
495  bboxes[b] = updated_bbox[b];
496 
497  // Signal the input that the template for this bounding box must be updated.
498  bboxes[b].templateStatus = 1;
499 
500  // Predicted transform is now identity as we reset the tracking.
501  preds[b] = VPIHomographyTransform2D{};
502  preds[b].mat3[0][0] = 1;
503  preds[b].mat3[1][1] = 1;
504  preds[b].mat3[2][2] = 1;
505  }
506  else
507  {
508  // Inform that the template for this bounding box doesn't need to be pdated.
509  bboxes[b].templateStatus = 0;
510 
511  // We just update the input transform with the estimated one.
512  preds[b] = estim[b];
513  }
514  }
515 
516  // We're finished working with the output arrays.
517  CHECK_STATUS(vpiArrayUnlock(outputBoxList));
518  CHECK_STATUS(vpiArrayUnlock(outputEstimList));
519 
520  // Since we've updated the input arrays, tell VPI to invalidate
521  // any internal buffers that might still refer to the old data.
522  CHECK_STATUS(vpiArrayInvalidate(inputBoxList));
523  CHECK_STATUS(vpiArrayInvalidate(inputPredList));
524 
525  // Next's reference frame is current's template.
526  std::swap(imgTemplate, imgReference);
527  std::swap(cvTemplate, cvReference);
528  } while (true);
529  }
530  catch (std::exception &e)
531  {
532  std::cerr << e.what() << std::endl;
533  retval = 1;
534  }
535 
536  // Clean up
537  vpiContextDestroy(ctx);
538 
539  return retval;
540 }
Functions and structures for dealing with VPI arrays.
Functions and structures for dealing with VPI contexts.
Functions and structures for dealing with VPI images.
Declares functions that implement the KLT Feature Tracker algorithm.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
int32_t * sizePointer
Points to the number of elements in the array.
Definition: Array.h:121
int32_t capacity
Maximum number of elements that the array can hold.
Definition: Array.h:122
VPIArrayType format
Format of each array element.
Definition: Array.h:120
void * data
Points to the first element of the array.
Definition: Array.h:124
VPIStatus vpiArraySetSize(VPIArray array, int32_t size)
Set the array size in elements.
VPIStatus vpiArrayUnlock(VPIArray array)
Releases the lock on array object.
VPIStatus vpiArrayLock(VPIArray array, VPILockMode mode, VPIArrayData *arrayData)
Acquires the lock on array object and returns a pointer to array data.
VPIStatus vpiArrayCreate(int32_t capacity, VPIArrayType type, uint32_t flags, VPIArray *array)
Create an empty array instance.
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:173
VPIStatus vpiArrayInvalidate(VPIArray array)
Informs that the array's wrapped memory was updated outside VPI.
@ VPI_ARRAY_TYPE_KLT_TRACKED_BOUNDING_BOX
VPIKLTTrackedBoundingBox element.
Definition: ArrayType.h:76
@ VPI_ARRAY_TYPE_HOMOGRAPHY_TRANSFORM_2D
VPIHomographyTransform2D element.
Definition: ArrayType.h:75
Stores information about array characteristics and content.
Definition: Array.h:119
VPIStatus vpiArrayCreateHostMemWrapper(const VPIArrayData *arrayData, uint32_t flags, VPIArray *array)
Create an array object by wrapping an existing host memory block.
VPIStatus vpiContextSetCurrent(VPIContext ctx)
Sets the context for the calling thread.
void vpiContextDestroy(VPIContext ctx)
Destroy a context instance as well as all resources it owns.
VPIStatus vpiContextCreate(uint32_t flags, VPIContext *ctx)
Create a context instance.
struct VPIContextImpl * VPIContext
A handle to a context.
Definition: Types.h:179
VPIImageFormat
Pre-defined image formats.
Definition: ImageFormat.h:99
@ VPI_IMAGE_FORMAT_U16
Single plane with one 16-bit unsigned integer channel.
Definition: ImageFormat.h:110
@ VPI_IMAGE_FORMAT_U8
Single plane with one 8-bit unsigned integer channel.
Definition: ImageFormat.h:104
@ VPI_IMAGE_FORMAT_S16
Single plane with one 16-bit signed integer channel.
Definition: ImageFormat.h:113
@ VPI_IMAGE_FORMAT_S8
Single plane with one 8-bit signed integer channel.
Definition: ImageFormat.h:107
int32_t height
Height of this plane in pixels.
Definition: Image.h:138
int32_t width
Width of this plane in pixels.
Definition: Image.h:137
void * data
Pointer to the first row of this plane.
Definition: Image.h:147
int32_t pitchBytes
Difference in bytes of beginning of one row and the beginning of the previous.
Definition: Image.h:139
VPIImagePlane planes[VPI_MAX_PLANE_COUNT]
Data of all image planes.
Definition: Image.h:166
VPIImageFormat format
Image format.
Definition: Image.h:160
VPIStatus vpiImageLock(VPIImage img, VPILockMode mode, VPIImageData *hostData)
Acquires the lock on an image object and returns a pointer to the image planes.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:197
VPIStatus vpiImageGetFormat(VPIImage img, VPIImageFormat *format)
Get the image format.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
Stores information about image characteristics and content.
Definition: Image.h:159
int8_t templateStatus
Status of the template related to this bounding box.
Definition: Types.h:332
float maxScaleChange
Maximum relative scale change.
int8_t trackingStatus
Tracking status of this bounding box.
Definition: Types.h:325
float maxTranslationChange
Maximum relative translation change.
VPIBoundingBox bbox
Bounding box being tracked.
Definition: Types.h:318
float nccThresholdUpdate
Threshold for requiring template update.
float nccThresholdStop
Threshold to stop estimating.
float nccThresholdKill
Threshold to consider template tracking was lost.
int32_t numberOfIterationsScaling
Number of Inverse compositional iterations of scale estimations.
VPIKLTFeatureTrackerType trackingType
Type of KLT tracking that will be performed.
VPIStatus vpiSubmitKLTFeatureTracker(VPIStream stream, uint32_t backend, VPIPayload payload, VPIImage templateImage, VPIArray inputBoxList, VPIArray inputPredictionList, VPIImage referenceImage, VPIArray outputBoxList, VPIArray outputEstimationList, const VPIKLTFeatureTrackerParams *params)
Runs KLT Feature Tracker on two frames.
VPIStatus vpiCreateKLTFeatureTracker(uint32_t backends, int32_t imageWidth, int32_t imageHeight, VPIImageFormat imageFormat, const VPIKLTFeatureTrackerCreationParams *params, VPIPayload *payload)
Creates payload for vpiSubmitKLTFeatureTracker.
@ VPI_KLT_INVERSE_COMPOSITIONAL
Inverse compositional algorithm for KLT tracker.
Structure that defines the parameters for vpiCreateKLTFeatureTracker.
Stores a bounding box that is being tracked by KLT Tracker.
Definition: Types.h:316
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiImageCreateOpenCVMatWrapper(const cv::Mat &mat, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:209
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:191
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:90
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:92
@ VPI_BACKEND_PVA
PVA backend.
Definition: Types.h:93
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:91
float width
Bounding box width.
Definition: Types.h:307
float height
Bounding box height.
Definition: Types.h:308
VPIHomographyTransform2D xform
Defines the bounding box top left corner and its homography.
Definition: Types.h:306
float mat3[3][3]
3x3 homogeneous matrix that defines the homography.
Definition: Types.h:286
@ VPI_LOCK_READ_WRITE
Lock memory for reading and writing.
Definition: Types.h:362
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:348
Stores a generic 2D homography transform.
Definition: Types.h:285