VPI - Vision Programming Interface

3.0 Release

Pyramidal LK Optical Flow

Overview

This application tracks feature points on an input video, draws the features on each frame and saves them to disk. The user can define what backend will be used for processing.

Note
The output will be in grayscale as the algorithm currently doesn't support color inputs.

Instructions

The command line parameters are:

<backend> <input video> <pyramid levels> <output frames>

where

  • backend: either cpu or cuda; it defines the backend that will perform the processing.
  • input video: input video file name, it accepts all video types that OpenCV's cv::VideoCapture accepts.
  • pyramid levels: specify the number of pyramid that used in the algorithm.
  • output frames: the file name that will be used for the output frames. Example: output.png will generate frames output_0000.png, output_0001.png, output_0002.png, and so on.

Here's one example:

  • C++
    ./vpi_sample_12_optflow_lk cuda ../assets/dashcam.mp4 5 frame.png
  • Python
    python3 main.py cuda ../assets/dashcam.mp4 5 frame.png

This is using the CUDA backend and one of the provided sample videos with pyramid level equals 5.

Results

Frame 0009

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import sys
28 import vpi
29 import numpy as np
30 from os import path
31 from argparse import ArgumentParser
32 from contextlib import contextmanager
33 import cv2
34 
35 
36 # --------------------------------------
37 # Some definitions and utility functions
38 
39 # Maximum number of keypoints that will be tracked
40 MAX_KEYPOINTS = 100
41 
42 def update_mask(mask, trackColors, prevFeatures, curFeatures, status = None):
43  '''Draw keypoint path from previous frame to current one'''
44 
45  numTrackedKeypoints = 0
46 
47  def none_context(a=None): return contextmanager(lambda: (x for x in [a]))()
48 
49  with curFeatures.rlock_cpu(), \
50  (status.rlock_cpu() if status else none_context()), \
51  (prevFeatures.rlock_cpu() if prevFeatures else none_context()):
52 
53  for i in range(curFeatures.size):
54  # keypoint is being tracked?
55  if not status or status.cpu()[i] == 0:
56  color = tuple(trackColors[i,0].tolist())
57 
58  # OpenCV 4.5+ wants integers in the tuple arguments below
59  cf = tuple(np.round(curFeatures.cpu()[i]).astype(int))
60 
61  # draw the tracks
62  if prevFeatures:
63  pf = tuple(np.round(prevFeatures.cpu()[i]).astype(int))
64  cv2.line(mask, pf, cf, color, 2)
65 
66  cv2.circle(mask, cf, 5, color, -1)
67 
68  numTrackedKeypoints += 1
69 
70  return numTrackedKeypoints
71 
72 def save_file_to_disk(frame, mask, baseFileName, frameCounter):
73  '''Apply mask on frame and save it to disk'''
74 
75  frame = frame.convert(vpi.Format.BGR8, backend=vpi.Backend.CUDA)
76  with frame.rlock_cpu() as frameData:
77  frame = cv2.add(frameData, mask)
78 
79  name, ext = path.splitext(baseFileName)
80  fname = "{}_{:04d}{}".format(name, frameCounter, ext)
81 
82  cv2.imwrite(fname, frame, [cv2.IMWRITE_JPEG_QUALITY, 70])
83 
84 # ----------------------------
85 # Parse command line arguments
86 
87 parser = ArgumentParser()
88 parser.add_argument('backend', choices=['cpu', 'cuda'],
89  help='Backend to be used for processing')
90 
91 parser.add_argument('input',
92  help='Input video to be processed')
93 
94 parser.add_argument('pyramid_levels', type=int,
95  help='Number of levels in the pyramid used with the algorithm')
96 
97 parser.add_argument('output',
98  help='Output file name')
99 
100 args = parser.parse_args();
101 
102 if args.backend == 'cuda':
103  backend = vpi.Backend.CUDA
104 else:
105  assert args.backend == 'cpu'
106  backend = vpi.Backend.CPU
107 
108 # adjust output file name to take into account backend used and python version
109 name, ext = path.splitext(args.output)
110 args.output = "{}_python{}_{}{}".format(name, sys.version_info[0], args.backend, ext)
111 
112 # ----------------
113 # Open input video
114 
115 inVideo = cv2.VideoCapture(args.input)
116 
117 # Read first input frame
118 ok, cvFrame = inVideo.read()
119 if not ok:
120  exit('Cannot read first input frame')
121 
122 # ---------------------------
123 # Perform some pre-processing
124 
125 # Retrieve features to be tracked from first frame using
126 # Harris Corners Detector
127 with vpi.Backend.CPU:
128  frame = vpi.asimage(cvFrame, vpi.Format.BGR8).convert(vpi.Format.U8)
129  curFeatures, scores = frame.harriscorners(strength=0.1, sensitivity=0.01)
130 
131 # Limit the number of features we'll track and calculate their colors on the
132 # output image
133 with curFeatures.lock_cpu() as featData, scores.rlock_cpu() as scoresData:
134  # Sort features in descending scores order and keep the first MAX_KEYPOINTS
135  ind = np.argsort(scoresData, kind='mergesort')[::-1]
136  featData[:] = np.take(featData, ind, axis=0)
137  curFeatures.size = min(curFeatures.size, MAX_KEYPOINTS)
138 
139  # Keypoints' have different hues, calculated from their position in the first frame
140  trackColors = np.array([[(int(p[0]) ^ int(p[1])) % 180,255,255] for p in featData], np.uint8).reshape(-1,1,3)
141  # Convert colors from HSV to RGB
142  trackColors = cv2.cvtColor(trackColors, cv2.COLOR_HSV2BGR).astype(int)
143 
144 with backend:
145  optflow = vpi.OpticalFlowPyrLK(frame, curFeatures, args.pyramid_levels)
146 
147 # Counter for the frames
148 idFrame = 0
149 
150 # Create mask with features' tracks over time
151 mask = np.zeros((frame.height, frame.width, 3), np.uint8)
152 numTrackedKeypoints = update_mask(mask, trackColors, None, curFeatures)
153 
154 while True:
155  # Apply mask to frame and save it to disk
156  save_file_to_disk(frame, mask, args.output, idFrame)
157 
158  print("Frame id={}: {} points tracked.".format(idFrame, numTrackedKeypoints))
159 
160  prevFeatures = curFeatures
161 
162  # Read one input frame
163  ret, cvFrame = inVideo.read()
164  if not ret:
165  print("Video ended.")
166  break
167  idFrame += 1
168 
169  # Convert frame to grayscale
170  with vpi.Backend.CUDA:
171  frame = vpi.asimage(cvFrame, vpi.Format.BGR8).convert(vpi.Format.U8);
172 
173  # Calculate where keypoints are in current frame
174  curFeatures, status = optflow(frame)
175 
176  # Update the mask with the current keypoints' position
177  numTrackedKeypoints = update_mask(mask, trackColors, prevFeatures, curFeatures, status)
178 
179  # No more keypoints to track?
180  if numTrackedKeypoints == 0:
181  print("No keypoints to track.")
182  break # nothing else to do
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 # include <opencv2/videoio.hpp>
33 #else
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/imgproc/imgproc.hpp>
38 #include <vpi/OpenCVInterop.hpp>
39 
40 #include <vpi/Array.h>
41 #include <vpi/Image.h>
42 #include <vpi/Pyramid.h>
43 #include <vpi/Status.h>
44 #include <vpi/Stream.h>
47 #include <vpi/algo/HarrisCorners.h>
49 
50 #include <algorithm>
51 #include <cstring> // for memset
52 #include <fstream>
53 #include <iostream>
54 #include <map>
55 #include <numeric>
56 #include <sstream>
57 #include <vector>
58 
59 // Max number of corners detected by harris corner algo
60 constexpr int MAX_HARRIS_CORNERS = 8192;
61 
62 // Max number of keypoints to be tracked
63 constexpr int MAX_KEYPOINTS = 100;
64 
65 #define CHECK_STATUS(STMT) \
66  do \
67  { \
68  VPIStatus status__ = (STMT); \
69  if (status__ != VPI_SUCCESS) \
70  { \
71  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
72  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
73  std::ostringstream ss; \
74  ss << vpiStatusGetName(status__) << ": " << buffer; \
75  throw std::runtime_error(ss.str()); \
76  } \
77  } while (0);
78 
79 static void SaveFileToDisk(VPIImage img, cv::Mat cvMask, std::string baseFileName, int32_t frameCounter)
80 {
81  VPIImageData imgData;
83 
84  cv::Mat cvImage;
85  try
86  {
87  cv::Mat tmp;
88  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgData, &tmp));
89  cvtColor(tmp, cvImage, cv::COLOR_GRAY2BGR);
90 
91  CHECK_STATUS(vpiImageUnlock(img));
92  }
93  catch (...)
94  {
95  CHECK_STATUS(vpiImageUnlock(img));
96  throw;
97  }
98 
99  add(cvImage, cvMask, cvImage);
100 
101  // Create the output file name
102  std::string fname = baseFileName;
103  int ext = fname.rfind('.');
104 
105  char buffer[512] = {};
106  snprintf(buffer, sizeof(buffer) - 1, "%s_%04d%s", fname.substr(0, ext).c_str(), frameCounter,
107  fname.substr(ext).c_str());
108 
109  // Finally, write frame to disk
110  if (!imwrite(buffer, cvImage, {cv::IMWRITE_JPEG_QUALITY, 70}))
111  {
112  throw std::runtime_error("Can't write to " + std::string(buffer));
113  }
114 }
115 
116 // Sort keypoints by decreasing score, and retain only the first 'max'
117 static void SortKeypoints(VPIArray keypoints, VPIArray scores, int max)
118 {
119  VPIArrayData ptsData, scoresData;
120  CHECK_STATUS(vpiArrayLockData(keypoints, VPI_LOCK_READ_WRITE, VPI_ARRAY_BUFFER_HOST_AOS, &ptsData));
121  CHECK_STATUS(vpiArrayLockData(scores, VPI_LOCK_READ_WRITE, VPI_ARRAY_BUFFER_HOST_AOS, &scoresData));
122 
123  VPIArrayBufferAOS &aosKeypoints = ptsData.buffer.aos;
124  VPIArrayBufferAOS &aosScores = scoresData.buffer.aos;
125 
126  std::vector<int> indices(*aosKeypoints.sizePointer);
127  std::iota(indices.begin(), indices.end(), 0);
128 
129  stable_sort(indices.begin(), indices.end(), [&aosScores](int a, int b) {
130  uint32_t *score = reinterpret_cast<uint32_t *>(aosScores.data);
131  return score[a] >= score[b]; // decreasing score order
132  });
133 
134  // keep the only 'max' indexes.
135  indices.resize(std::min<size_t>(indices.size(), max));
136 
137  VPIKeypointF32 *kptData = reinterpret_cast<VPIKeypointF32 *>(aosKeypoints.data);
138 
139  // reorder the keypoints to keep the first 'max' with highest scores.
140  std::vector<VPIKeypointF32> kpt;
141  std::transform(indices.begin(), indices.end(), std::back_inserter(kpt),
142  [kptData](int idx) { return kptData[idx]; });
143  std::copy(kpt.begin(), kpt.end(), kptData);
144 
145  // update keypoint array size.
146  *aosKeypoints.sizePointer = kpt.size();
147 
148  vpiArrayUnlock(scores);
149  vpiArrayUnlock(keypoints);
150 }
151 
152 static int UpdateMask(cv::Mat &cvMask, const std::vector<cv::Scalar> &trackColors, VPIArray prevFeatures,
153  VPIArray curFeatures, VPIArray status)
154 {
155  // Now that optical flow is completed, there are usually two approaches to take:
156  // 1. Add new feature points from current frame using a feature detector such as
157  // \ref algo_harris_corners "Harris Corner Detector"
158  // 2. Keep using the points that are being tracked.
159  //
160  // The sample app uses the valid feature point and continue to do the tracking.
161 
162  // Lock the input and output arrays to draw the tracks to the output mask.
163  VPIArrayData curFeaturesData, statusData;
164  CHECK_STATUS(vpiArrayLockData(curFeatures, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &curFeaturesData));
165  CHECK_STATUS(vpiArrayLockData(status, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &statusData));
166 
167  const VPIArrayBufferAOS &aosCurFeatures = curFeaturesData.buffer.aos;
168  const VPIArrayBufferAOS &aosStatus = statusData.buffer.aos;
169 
170  const VPIKeypointF32 *pCurFeatures = (VPIKeypointF32 *)aosCurFeatures.data;
171  const uint8_t *pStatus = (uint8_t *)aosStatus.data;
172 
173  const VPIKeypointF32 *pPrevFeatures;
174  if (prevFeatures)
175  {
176  VPIArrayData prevFeaturesData;
177  CHECK_STATUS(vpiArrayLockData(prevFeatures, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &prevFeaturesData));
178  pPrevFeatures = (VPIKeypointF32 *)prevFeaturesData.buffer.aos.data;
179  }
180  else
181  {
182  pPrevFeatures = NULL;
183  }
184 
185  int numTrackedKeypoints = 0;
186  int totKeypoints = *curFeaturesData.buffer.aos.sizePointer;
187 
188  for (int i = 0; i < totKeypoints; i++)
189  {
190  // keypoint is being tracked?
191  if (pStatus[i] == 0)
192  {
193  // draw the tracks
194  cv::Point curPoint{(int)round(pCurFeatures[i].x), (int)round(pCurFeatures[i].y)};
195  if (pPrevFeatures != NULL)
196  {
197  cv::Point2f prevPoint{pPrevFeatures[i].x, pPrevFeatures[i].y};
198  line(cvMask, prevPoint, curPoint, trackColors[i], 2);
199  }
200 
201  circle(cvMask, curPoint, 5, trackColors[i], -1);
202 
203  numTrackedKeypoints++;
204  }
205  }
206 
207  // We're finished working with the arrays.
208  if (prevFeatures)
209  {
210  CHECK_STATUS(vpiArrayUnlock(prevFeatures));
211  }
212  CHECK_STATUS(vpiArrayUnlock(curFeatures));
213  CHECK_STATUS(vpiArrayUnlock(status));
214 
215  return numTrackedKeypoints;
216 }
217 
218 int main(int argc, char *argv[])
219 {
220  // OpenCV image that will be wrapped by a VPIImage.
221  // Define it here so that it's destroyed *after* wrapper is destroyed
222  cv::Mat cvFrame;
223 
224  // VPI objects that will be used
225  VPIStream stream = NULL;
226  VPIImage imgTempFrame = NULL;
227  VPIImage imgFrame = NULL;
228  VPIPyramid pyrPrevFrame = NULL, pyrCurFrame = NULL;
229  VPIArray prevFeatures = NULL, curFeatures = NULL, status = NULL;
230  VPIPayload optflow = NULL;
231  VPIArray scores = NULL;
232  VPIPayload harris = NULL;
233 
234  int retval = 0;
235 
236  try
237  {
238  // ============================
239  // Parse command line arguments
240 
241  if (argc != 5)
242  {
243  throw std::runtime_error(std::string("Usage: ") + argv[0] +
244  " <cpu|cuda> <input_video> <pyramid_levels> <output>");
245  }
246 
247  std::string strBackend = argv[1];
248  std::string strInputVideo = argv[2];
249  int32_t pyrLevel = std::stoi(argv[3]);
250  std::string strOutputFiles = argv[4];
251 
252  // Now parse the backend
253  VPIBackend backend;
254 
255  if (strBackend == "cpu")
256  {
257  backend = VPI_BACKEND_CPU;
258  }
259  else if (strBackend == "cuda")
260  {
261  backend = VPI_BACKEND_CUDA;
262  }
263  else
264  {
265  throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be either cpu or cuda.");
266  }
267 
268  {
269  int ext = strOutputFiles.rfind('.');
270  strOutputFiles = strOutputFiles.substr(0, ext) + "_" + strBackend + strOutputFiles.substr(ext);
271  }
272 
273  // ====================
274  // Load the input video
275  cv::VideoCapture invid;
276  if (!invid.open(strInputVideo))
277  {
278  throw std::runtime_error("Can't open '" + strInputVideo + "'");
279  }
280 
281  // Fetch the first frame and wrap it into a VPIImage.
282  // The points to be tracked will be gathered from this frame later on.
283  if (!invid.read(cvFrame))
284  {
285  throw std::runtime_error("Can't retrieve first frame from '" + strInputVideo + "'");
286  }
287 
288  // =================================================
289  // Allocate VPI resources and do some pre-processing
290 
291  // Create the stream where processing will happen.
292  CHECK_STATUS(vpiStreamCreate(0, &stream));
293 
294  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvFrame, 0, &imgTempFrame));
295 
296  // Create grayscale image representation of input.
297  CHECK_STATUS(vpiImageCreate(cvFrame.cols, cvFrame.rows, VPI_IMAGE_FORMAT_U8, 0, &imgFrame));
298 
299  // Create the image pyramids used by the algorithm
300  CHECK_STATUS(
301  vpiPyramidCreate(cvFrame.cols, cvFrame.rows, VPI_IMAGE_FORMAT_U8, pyrLevel, 0.5, 0, &pyrPrevFrame));
302  CHECK_STATUS(vpiPyramidCreate(cvFrame.cols, cvFrame.rows, VPI_IMAGE_FORMAT_U8, pyrLevel, 0.5, 0, &pyrCurFrame));
303 
304  // Create input and output arrays
305  CHECK_STATUS(vpiArrayCreate(MAX_HARRIS_CORNERS, VPI_ARRAY_TYPE_KEYPOINT_F32, 0, &prevFeatures));
306  CHECK_STATUS(vpiArrayCreate(MAX_HARRIS_CORNERS, VPI_ARRAY_TYPE_KEYPOINT_F32, 0, &curFeatures));
307  CHECK_STATUS(vpiArrayCreate(MAX_HARRIS_CORNERS, VPI_ARRAY_TYPE_U8, 0, &status));
308 
309  // Create Optical Flow payload
310  CHECK_STATUS(vpiCreateOpticalFlowPyrLK(backend, cvFrame.cols, cvFrame.rows, VPI_IMAGE_FORMAT_U8, pyrLevel, 0.5,
311  &optflow));
312 
313  // Parameters we'll use. No need to change them on the fly, so just define them here.
314  // We're using the default parameters.
315  VPIOpticalFlowPyrLKParams lkParams;
316  CHECK_STATUS(vpiInitOpticalFlowPyrLKParams(&lkParams));
317 
318  // Create a mask image for drawing purposes
319  cv::Mat cvMask = cv::Mat::zeros(cvFrame.size(), CV_8UC3);
320 
321  // Gather feature points from first frame using Harris Corners on CPU.
322  {
323  CHECK_STATUS(vpiArrayCreate(MAX_HARRIS_CORNERS, VPI_ARRAY_TYPE_U32, 0, &scores));
324 
325  VPIHarrisCornerDetectorParams harrisParams;
326  CHECK_STATUS(vpiInitHarrisCornerDetectorParams(&harrisParams));
327  harrisParams.strengthThresh = 0;
328  harrisParams.sensitivity = 0.01;
329 
330  CHECK_STATUS(vpiCreateHarrisCornerDetector(VPI_BACKEND_CPU, cvFrame.cols, cvFrame.rows, &harris));
331 
332  // Convert input to grayscale to conform with harris corner detector restrictions
333  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgTempFrame, imgFrame, NULL));
334 
335  CHECK_STATUS(vpiSubmitHarrisCornerDetector(stream, VPI_BACKEND_CPU, harris, imgFrame, curFeatures, scores,
336  &harrisParams));
337 
338  CHECK_STATUS(vpiStreamSync(stream));
339 
340  SortKeypoints(curFeatures, scores, MAX_KEYPOINTS);
341  }
342 
343  // Create some random colors
344  std::vector<cv::Scalar> trackColors;
345  {
346  std::vector<cv::Vec3b> tmpTrackColors;
347 
348  VPIArrayData ptsData;
349  CHECK_STATUS(vpiArrayLockData(curFeatures, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &ptsData));
350 
351  const VPIArrayBufferAOS &aosKeypoints = ptsData.buffer.aos;
352 
353  const VPIKeypointF32 *pts = (VPIKeypointF32 *)aosKeypoints.data;
354 
355  for (int i = 0; i < *aosKeypoints.sizePointer; i++)
356  {
357  // Track hue depends on its initial position
358  int hue = ((int)pts[i].x ^ (int)pts[i].y) % 180;
359 
360  tmpTrackColors.push_back(cv::Vec3b(hue, 255, 255));
361  }
362  CHECK_STATUS(vpiArrayUnlock(curFeatures));
363 
364  cvtColor(tmpTrackColors, tmpTrackColors, cv::COLOR_HSV2BGR);
365 
366  for (size_t i = 0; i < tmpTrackColors.size(); i++)
367  {
368  trackColors.push_back(cv::Scalar(tmpTrackColors[i]));
369  }
370  }
371 
372  // Update the mask with info from first frame.
373  int numTrackedKeypoints = UpdateMask(cvMask, trackColors, NULL, curFeatures, status);
374 
375  // =================================================
376  // Main processing stage
377 
378  // Generate pyramid for first frame.
379  CHECK_STATUS(vpiSubmitGaussianPyramidGenerator(stream, backend, imgFrame, pyrCurFrame, VPI_BORDER_CLAMP));
380 
381  // Counter for the frames
382  int idxFrame = 0;
383 
384  while (true)
385  {
386  // Save frame to disk
387  SaveFileToDisk(imgFrame, cvMask, strOutputFiles, idxFrame);
388 
389  printf("Frame id=%d: %d points tracked. \n", idxFrame, numTrackedKeypoints);
390 
391  // Last iteration's current frame/features become this iteration's prev frame/features.
392  // The former will contain information gathered in this iteration.
393  std::swap(prevFeatures, curFeatures);
394  std::swap(pyrPrevFrame, pyrCurFrame);
395 
396  // Fetch a new frame
397  if (!invid.read(cvFrame))
398  {
399  printf("Video ended.\n");
400  break;
401  }
402 
403  ++idxFrame;
404 
405  // Wrap frame into a VPIImage, reusing the existing imgFrame.
406  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(imgTempFrame, cvFrame));
407 
408  // Convert it to grayscale
409  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, backend, imgTempFrame, imgFrame, NULL))
410 
411  // Generate a pyramid out of it
412  CHECK_STATUS(vpiSubmitGaussianPyramidGenerator(stream, backend, imgFrame, pyrCurFrame, VPI_BORDER_CLAMP));
413 
414  // Estimate the features' position in current frame given their position in previous frame
415  CHECK_STATUS(vpiSubmitOpticalFlowPyrLK(stream, 0, optflow, pyrPrevFrame, pyrCurFrame, prevFeatures,
416  curFeatures, status, &lkParams));
417 
418  // Wait for processing to finish.
419  CHECK_STATUS(vpiStreamSync(stream));
420 
421  // Update the output mask
422  numTrackedKeypoints = UpdateMask(cvMask, trackColors, prevFeatures, curFeatures, status);
423 
424  // No more keypoints being tracked?
425  if (numTrackedKeypoints == 0)
426  {
427  printf("No keypoints to track.\n");
428  break; // we can finish procesing.
429  }
430  }
431  }
432  catch (std::exception &e)
433  {
434  std::cerr << e.what() << std::endl;
435  retval = 1;
436  }
437 
438  vpiStreamDestroy(stream);
439  vpiPayloadDestroy(harris);
440  vpiPayloadDestroy(optflow);
441 
442  vpiPyramidDestroy(pyrPrevFrame);
443  vpiImageDestroy(imgTempFrame);
444  vpiImageDestroy(imgFrame);
445  vpiArrayDestroy(prevFeatures);
446  vpiArrayDestroy(curFeatures);
447  vpiArrayDestroy(status);
448  vpiArrayDestroy(scores);
449 
450  return retval;
451 }
Functions and structures for dealing with VPI arrays.
Declares functions that handle image format conversion.
Declares functions that handle gaussian pyramids.
Declares functions that implement the Harris Corner Detector algorithm.
#define VPI_IMAGE_FORMAT_U8
Single plane with one 8-bit unsigned integer channel.
Definition: ImageFormat.h:100
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declares functions that implement the Pyramidal LK Optical Flow algorithm.
Functions and structures for dealing with VPI pyramids.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
void * data
Points to the first element of the array.
Definition: Array.h:135
VPIArrayBuffer buffer
Stores the array contents.
Definition: Array.h:175
int32_t * sizePointer
Points to the number of elements in the array.
Definition: Array.h:122
VPIArrayBufferAOS aos
Array stored in array-of-structures layout.
Definition: Array.h:162
VPIStatus vpiArrayUnlock(VPIArray array)
Releases the lock on array object.
VPIStatus vpiArrayLockData(VPIArray array, VPILockMode mode, VPIArrayBufferType bufType, VPIArrayData *data)
Acquires the lock on an array object and returns the array contents.
void vpiArrayDestroy(VPIArray array)
Destroy an array instance.
VPIStatus vpiArrayCreate(int32_t capacity, VPIArrayType type, uint64_t flags, VPIArray *array)
Create an empty array instance.
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:232
@ VPI_ARRAY_TYPE_U32
Unsigned 32-bit.
Definition: ArrayType.h:76
@ VPI_ARRAY_TYPE_KEYPOINT_F32
VPIKeypointF32 element.
Definition: ArrayType.h:77
@ VPI_ARRAY_TYPE_U8
Unsigned 8-bit.
Definition: ArrayType.h:73
@ VPI_ARRAY_BUFFER_HOST_AOS
Host-accessible array-of-structures.
Definition: Array.h:146
Stores information about array characteristics and content.
Definition: Array.h:116
Stores information about array characteristics and contents.
Definition: Array.h:168
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
VPIStatus vpiSubmitGaussianPyramidGenerator(VPIStream stream, uint64_t backend, VPIImage input, VPIPyramid output, VPIBorderExtension border)
Computes the Gaussian pyramid from the input image.
float strengthThresh
Specifies the minimum threshold with which to eliminate Harris Corner scores.
Definition: HarrisCorners.h:93
float sensitivity
Specifies sensitivity threshold from the Harris-Stephens equation.
Definition: HarrisCorners.h:96
VPIStatus vpiSubmitHarrisCornerDetector(VPIStream stream, uint64_t backend, VPIPayload payload, VPIImage input, VPIArray outFeatures, VPIArray outScores, const VPIHarrisCornerDetectorParams *params)
Submits a Harris Corner Detector operation to the stream.
VPIStatus vpiInitHarrisCornerDetectorParams(VPIHarrisCornerDetectorParams *params)
Initializes VPIHarrisCornerDetectorParams with default values.
VPIStatus vpiCreateHarrisCornerDetector(uint64_t backends, int32_t inputWidth, int32_t inputHeight, VPIPayload *payload)
Creates a Harris Corner Detector payload.
Structure that defines the parameters for vpiSubmitHarrisCornerDetector.
Definition: HarrisCorners.h:82
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:234
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiSubmitOpticalFlowPyrLK(VPIStream stream, uint64_t backend, VPIPayload payload, VPIPyramid prevPyr, VPIPyramid curPyr, VPIArray prevPts, VPIArray curPts, VPIArray trackingStatus, const VPIOpticalFlowPyrLKParams *params)
Runs Pyramidal LK Optical Flow on two frames.
VPIStatus vpiCreateOpticalFlowPyrLK(uint64_t backends, int32_t width, int32_t height, VPIImageFormat fmt, int32_t levels, float scale, VPIPayload *payload)
Creates payload for vpiSubmitOpticalFlowPyrLK.
VPIStatus vpiInitOpticalFlowPyrLKParams(VPIOpticalFlowPyrLKParams *params)
Initializes VPIOpticalFlowPyrLKParams with default values.
Structure that defines the parameters for vpiSubmitOpticalFlowPyrLK.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:268
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
VPIStatus vpiPyramidCreate(int32_t width, int32_t height, VPIImageFormat fmt, int32_t numLevels, float scale, uint64_t flags, VPIPyramid *pyr)
Create an empty image pyramid instance with the specified flags.
struct VPIPyramidImpl * VPIPyramid
A handle to an image pyramid.
Definition: Types.h:262
void vpiPyramidDestroy(VPIPyramid pyr)
Destroy an image pyramid instance as well as all resources it owns.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:92
float x
Keypoint's x coordinate.
Definition: Types.h:316
float y
Keypoint's y coordinate.
Definition: Types.h:317
@ VPI_BORDER_CLAMP
Border pixels are repeated indefinitely.
Definition: Types.h:279
@ VPI_LOCK_READ_WRITE
Lock memory for reading and writing.
Definition: Types.h:609
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:595
Stores a float32 keypoint coordinate The coordinate is relative to the top-left corner of an image.
Definition: Types.h:315