VPI - Vision Programming Interface

2.0 Release - Developer Preview

Harris Corners Detector

Overview

This application detects Harris corners from a given input image. The corners are then drawn onto the input image and the result is saved as an image on disk.

Instructions

The command line parameters are:

<backend> <input image>

where

  • backend: either cpu, cuda or pva; it defines the backend that will perform the processing.
  • input image: input image file name, it accepts png, jpeg and possibly others.

Here's one example:

  • C++
    ./vpi_sample_03_harris_corners pva ../assets/kodim08.png
  • Python
    python3 main.py pva ../assets/kodim08.png

This is using the PVA backend and one of the provided sample images. You can try with other input images, respecting the constraints imposed by the algorithm. If your stream doesn't support PVA, an error is printed. In this case, just try another backend.

Results

Input imageOutput image, Harris keypoints

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import cv2
28 import sys
29 import vpi
30 import numpy as np
31 from argparse import ArgumentParser
32 
33 # ----------------------------
34 # Parse command line arguments
35 
36 parser = ArgumentParser()
37 parser.add_argument('backend', choices=['cpu','cuda','pva'],
38  help='Backend to be used for processing')
39 
40 parser.add_argument('input',
41  help='Input image on which harris corners will be detected')
42 
43 args = parser.parse_args();
44 
45 if args.backend == 'cpu':
46  backend = vpi.Backend.CPU
47 elif args.backend == 'cuda':
48  backend = vpi.Backend.CUDA
49 else:
50  assert args.backend == 'pva'
51  backend = vpi.Backend.PVA
52 
53 # --------------------------------------------------------------
54 # Load input into a vpi.Image and convert it to grayscale, signed 16bpp
55 with vpi.Backend.CUDA:
56  input = vpi.asimage(cv2.imread(args.input), vpi.Format.BGR8).convert(vpi.Format.S16)
57 
58 with backend:
59  corners, scores = input.harriscorners(sensitivity=0.01)
60 
61 # ---------------------------------------
62 # Render the keypoints in the output image
63 
64 out = input.convert(vpi.Format.BGR8, backend=vpi.Backend.CUDA)
65 
66 if corners.size > 0:
67  with out.lock_cpu() as out_data, scores.lock_cpu() as scores_data, corners.lock_cpu() as corners_data:
68  cmap = cv2.applyColorMap(np.arange(0, 256, dtype=np.uint8), cv2.COLORMAP_HOT)
69 
70  maxscore = scores_data.max()
71 
72  for i in range(corners.size):
73  color = tuple([int(x) for x in cmap[255*scores_data[i]//maxscore,0]])
74  kpt = tuple(corners_data[i].astype(np.int16))
75  cv2.circle(out_data, kpt, 3, color, -1)
76 
77 # -------------------
78 # Save result to disk
79 cv2.imwrite('harris_corners_python'+str(sys.version_info[0])+'_'+args.backend+'.png', out.cpu())
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 #else
33 # include <opencv2/contrib/contrib.hpp> // for applyColorMap
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/imgproc/imgproc.hpp>
38 #include <vpi/OpenCVInterop.hpp>
39 
40 #include <vpi/Array.h>
41 #include <vpi/Image.h>
42 #include <vpi/Status.h>
43 #include <vpi/Stream.h>
45 #include <vpi/algo/HarrisCorners.h>
46 
47 #include <cstdio>
48 #include <cstring> // for memset
49 #include <iostream>
50 #include <sstream>
51 
52 #define CHECK_STATUS(STMT) \
53  do \
54  { \
55  VPIStatus status = (STMT); \
56  if (status != VPI_SUCCESS) \
57  { \
58  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
59  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
60  std::ostringstream ss; \
61  ss << vpiStatusGetName(status) << ": " << buffer; \
62  throw std::runtime_error(ss.str()); \
63  } \
64  } while (0);
65 
66 static cv::Mat DrawKeypoints(cv::Mat img, VPIKeypoint *kpts, uint32_t *scores, int numKeypoints)
67 {
68  cv::Mat out;
69  img.convertTo(out, CV_8UC1);
70  cvtColor(out, out, cv::COLOR_GRAY2BGR);
71 
72  if (numKeypoints == 0)
73  {
74  return out;
75  }
76 
77  // prepare our colormap
78  cv::Mat cmap(1, 256, CV_8UC3);
79  {
80  cv::Mat gray(1, 256, CV_8UC1);
81  for (int i = 0; i < 256; ++i)
82  {
83  gray.at<unsigned char>(0, i) = i;
84  }
85  applyColorMap(gray, cmap, cv::COLORMAP_HOT);
86  }
87 
88  float maxScore = *std::max_element(scores, scores + numKeypoints);
89 
90  for (int i = 0; i < numKeypoints; ++i)
91  {
92  cv::Vec3b color = cmap.at<cv::Vec3b>(scores[i] / maxScore * 255);
93  circle(out, cv::Point(kpts[i].x, kpts[i].y), 3, cv::Scalar(color[0], color[1], color[2]), -1);
94  }
95 
96  return out;
97 }
98 
99 int main(int argc, char *argv[])
100 {
101  // OpenCV image that will be wrapped by a VPIImage.
102  // Define it here so that it's destroyed *after* wrapper is destroyed
103  cv::Mat cvImage;
104 
105  // VPI objects that will be used
106  VPIImage imgInput = NULL;
107  VPIImage imgGrayscale = NULL;
108  VPIArray keypoints = NULL;
109  VPIArray scores = NULL;
110  VPIStream stream = NULL;
111  VPIPayload harris = NULL;
112 
113  int retval = 0;
114 
115  try
116  {
117  // =============================
118  // Parse command line parameters
119 
120  if (argc != 3)
121  {
122  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|pva|cuda> <input image>");
123  }
124 
125  std::string strBackend = argv[1];
126  std::string strInputFileName = argv[2];
127 
128  // Now parse the backend
129  VPIBackend backend;
130 
131  if (strBackend == "cpu")
132  {
133  backend = VPI_BACKEND_CPU;
134  }
135  else if (strBackend == "cuda")
136  {
137  backend = VPI_BACKEND_CUDA;
138  }
139  else if (strBackend == "pva")
140  {
141  backend = VPI_BACKEND_PVA;
142  }
143  else
144  {
145  throw std::runtime_error("Backend '" + strBackend +
146  "' not recognized, it must be either cpu, cuda or pva.");
147  }
148 
149  // =====================
150  // Load the input image
151 
152  cvImage = cv::imread(strInputFileName);
153  if (cvImage.empty())
154  {
155  throw std::runtime_error("Can't open '" + strInputFileName + "'");
156  }
157 
158  // =================================
159  // Allocate all VPI resources needed
160 
161  // Create the stream where processing will happen
162  CHECK_STATUS(vpiStreamCreate(0, &stream));
163 
164  // We now wrap the loaded image into a VPIImage object to be used by VPI.
165  // VPI won't make a copy of it, so the original
166  // image must be in scope at all times.
167  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvImage, 0, &imgInput));
168 
169  CHECK_STATUS(vpiImageCreate(cvImage.cols, cvImage.rows, VPI_IMAGE_FORMAT_S16, 0, &imgGrayscale));
170 
171  // Create the output keypoint array. Currently for PVA backend it must have 8192 elements.
172  CHECK_STATUS(vpiArrayCreate(8192, VPI_ARRAY_TYPE_KEYPOINT, 0, &keypoints));
173 
174  // Create the output scores array. It also must have 8192 elements and elements must be uint32_t.
175  CHECK_STATUS(vpiArrayCreate(8192, VPI_ARRAY_TYPE_U32, 0, &scores));
176 
177  // Create the payload for Harris Corners Detector algorithm
178  CHECK_STATUS(vpiCreateHarrisCornerDetector(backend, cvImage.cols, cvImage.rows, &harris));
179 
180  // Define the algorithm parameters. We'll use defaults, expect for sensitivity.
181  VPIHarrisCornerDetectorParams harrisParams;
182  CHECK_STATUS(vpiInitHarrisCornerDetectorParams(&harrisParams));
183  harrisParams.sensitivity = 0.01;
184 
185  // ================
186  // Processing stage
187 
188  // First convert input to grayscale
189  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgInput, imgGrayscale, NULL));
190 
191  // Then get Harris corners
192  CHECK_STATUS(
193  vpiSubmitHarrisCornerDetector(stream, backend, harris, imgGrayscale, keypoints, scores, &harrisParams));
194 
195  // Wait until the algorithm finishes processing
196  CHECK_STATUS(vpiStreamSync(stream));
197 
198  // =======================================
199  // Output processing and saving it to disk
200 
201  // Lock output keypoints and scores to retrieve its data on cpu memory
202  VPIArrayData outKeypointsData;
203  VPIArrayData outScoresData;
204  VPIImageData imgData;
205  CHECK_STATUS(vpiArrayLockData(keypoints, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &outKeypointsData));
206  CHECK_STATUS(vpiArrayLockData(scores, VPI_LOCK_READ, VPI_ARRAY_BUFFER_HOST_AOS, &outScoresData));
207  CHECK_STATUS(vpiImageLockData(imgGrayscale, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &imgData));
208 
209  VPIKeypoint *outKeypoints = (VPIKeypoint *)outKeypointsData.buffer.aos.data;
210  uint32_t *outScores = (uint32_t *)outScoresData.buffer.aos.data;
211 
212  printf("\n%d keypoints found\n", *outKeypointsData.buffer.aos.sizePointer);
213 
214  cv::Mat img;
215  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgData, &img));
216 
217  cv::Mat outImage = DrawKeypoints(img, outKeypoints, outScores, *outKeypointsData.buffer.aos.sizePointer);
218 
219  imwrite("harris_corners_" + strBackend + ".png", outImage);
220 
221  // Done handling outputs, don't forget to unlock them.
222  CHECK_STATUS(vpiImageUnlock(imgGrayscale));
223  CHECK_STATUS(vpiArrayUnlock(scores));
224  CHECK_STATUS(vpiArrayUnlock(keypoints));
225  }
226  catch (std::exception &e)
227  {
228  std::cerr << e.what() << std::endl;
229  retval = 1;
230  }
231 
232  // ========
233  // Clean up
234 
235  // Make sure stream is synchronized before destroying the objects
236  // that might still be in use.
237  if (stream != NULL)
238  {
239  vpiStreamSync(stream);
240  }
241 
242  vpiImageDestroy(imgInput);
243  vpiImageDestroy(imgGrayscale);
244  vpiArrayDestroy(keypoints);
245  vpiArrayDestroy(scores);
246  vpiPayloadDestroy(harris);
247  vpiStreamDestroy(stream);
248 
249  return retval;
250 }
Functions and structures for dealing with VPI arrays.
Declares functions that handle image format conversion.
Declares functions that implement the Harris Corner Detector algorithm.
#define VPI_IMAGE_FORMAT_S16
Single plane with one 16-bit signed integer channel.
Definition: ImageFormat.h:118
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
void * data
Points to the first element of the array.
Definition: Array.h:135
VPIArrayBuffer buffer
Stores the array contents.
Definition: Array.h:175
int32_t * sizePointer
Points to the number of elements in the array.
Definition: Array.h:122
VPIArrayBufferAOS aos
Array stored in array-of-structures layout.
Definition: Array.h:162
VPIStatus vpiArrayUnlock(VPIArray array)
Releases the lock on array object.
VPIStatus vpiArrayLockData(VPIArray array, VPILockMode mode, VPIArrayBufferType bufType, VPIArrayData *data)
Acquires the lock on an array object and returns the array contents.
void vpiArrayDestroy(VPIArray array)
Destroy an array instance.
VPIStatus vpiArrayCreate(int32_t capacity, VPIArrayType type, uint64_t flags, VPIArray *array)
Create an empty array instance.
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:232
@ VPI_ARRAY_TYPE_U32
Unsigned 32-bit.
Definition: ArrayType.h:75
@ VPI_ARRAY_TYPE_KEYPOINT
VPIKeypoint element.
Definition: ArrayType.h:76
@ VPI_ARRAY_BUFFER_HOST_AOS
Host-accessible array-of-structures.
Definition: Array.h:146
Stores information about array characteristics and contents.
Definition: Array.h:168
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
float sensitivity
Specifies sensitivity threshold from the Harris-Stephens equation.
Definition: HarrisCorners.h:96
VPIStatus vpiSubmitHarrisCornerDetector(VPIStream stream, uint64_t backend, VPIPayload payload, VPIImage input, VPIArray outFeatures, VPIArray outScores, const VPIHarrisCornerDetectorParams *params)
Submits a Harris Corner Detector operation to the stream.
VPIStatus vpiInitHarrisCornerDetectorParams(VPIHarrisCornerDetectorParams *params)
Initializes VPIHarrisCornerDetectorParams with default values.
VPIStatus vpiCreateHarrisCornerDetector(uint64_t backends, int32_t inputWidth, int32_t inputHeight, VPIPayload *payload)
Creates a Harris Corner Detector payload.
Structure that defines the parameters for vpiSubmitHarrisCornerDetector.
Definition: HarrisCorners.h:82
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:230
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:268
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_PVA
PVA backend.
Definition: Types.h:94
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:92
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:435
Stores a keypoint coordinate.
Definition: Types.h:316