VPI - Vision Programming Interface

1.0 Release

Stereo Disparity

Overview

The Stereo Disparity application receives left and right stereo pair images and returns the disparity between them, which is a function of image depth. The result is saved into disparity.png.

This sample shows the following:

  • Creating and destroying a VPI stream.
  • Wrapping a OpenCV cv::Mat image to be used with VPI.
  • Creating a VPI-managed 2D image where output will be written to
  • Create a properly-configured StereoProcess algorithm and submit it to the stream.
  • Simple stream synchronization.
  • Image locking to access its contents from CPU side.
  • Error handling.
  • Environment clean up.

Instructions

The usage is:

./vpi_sample_02_stereo_disparity <backend> <input image>

where

  • backend: either cpu, cuda or pva; it defines the backend that will perform the processing.
  • left image: left input image of a stereo pair, it accepts png, jpeg and possibly others.
  • right image: right input image of a stereo pair.

Here's one example:

./vpi_sample_02_stereo_disparity cuda ../assets/chair_stereo_left.png ../assets/chair_stereo_right.png

This is using the CUDA backend and the provided sample images. You can try with other stereo pair images, respecting the constraints imposed by the algorithm.

Results

Left input image Right input image
Stereo disparity

Source Code

For convenience, here's the code that is also installed in the samples directory.

1 /*
2 * Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without
5 * modification, are permitted provided that the following conditions
6 * are met:
7 * * Redistributions of source code must retain the above copyright
8 * notice, this list of conditions and the following disclaimer.
9 * * Redistributions in binary form must reproduce the above copyright
10 * notice, this list of conditions and the following disclaimer in the
11 * documentation and/or other materials provided with the distribution.
12 * * Neither the name of NVIDIA CORPORATION nor the names of its
13 * contributors may be used to endorse or promote products derived
14 * from this software without specific prior written permission.
15 *
16 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 */
28 
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 #else
33 # include <opencv2/highgui/highgui.hpp>
34 #endif
35 
36 #include <opencv2/imgproc/imgproc.hpp>
37 #include <vpi/OpenCVInterop.hpp>
38 
39 #include <vpi/Image.h>
40 #include <vpi/Status.h>
41 #include <vpi/Stream.h>
43 
44 #include <cstring> // for memset
45 #include <iostream>
46 #include <sstream>
47 
48 #define CHECK_STATUS(STMT) \
49  do \
50  { \
51  VPIStatus status = (STMT); \
52  if (status != VPI_SUCCESS) \
53  { \
54  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
55  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
56  std::ostringstream ss; \
57  ss << vpiStatusGetName(status) << ": " << buffer; \
58  throw std::runtime_error(ss.str()); \
59  } \
60  } while (0);
61 
62 int main(int argc, char *argv[])
63 {
64  VPIImage left = NULL;
65  VPIImage right = NULL;
66  VPIImage disparity = NULL;
67  VPIStream stream = NULL;
68  VPIPayload stereo = NULL;
69 
70  int retval = 0;
71 
72  try
73  {
74  if (argc != 4)
75  {
76  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|pva|cuda> <left image> <right image>");
77  }
78 
79  std::string strBackend = argv[1];
80  std::string strLeftFileName = argv[2];
81  std::string strRightFileName = argv[3];
82 
83  // Load the input images
84  cv::Mat cvImageLeft = cv::imread(strLeftFileName, cv::IMREAD_GRAYSCALE);
85  if (cvImageLeft.empty())
86  {
87  throw std::runtime_error("Can't open '" + strLeftFileName + "'");
88  }
89 
90  cv::Mat cvImageRight = cv::imread(strRightFileName, cv::IMREAD_GRAYSCALE);
91  if (cvImageRight.empty())
92  {
93  throw std::runtime_error("Can't open '" + strRightFileName + "'");
94  }
95 
96  // Currently we only accept unsigned 16bpp inputs.
97  cvImageLeft.convertTo(cvImageLeft, CV_16UC1);
98  cvImageRight.convertTo(cvImageRight, CV_16UC1);
99 
100  // Now parse the backend
101  VPIBackend backendType;
102 
103  if (strBackend == "cpu")
104  {
105  backendType = VPI_BACKEND_CPU;
106  }
107  else if (strBackend == "cuda")
108  {
109  backendType = VPI_BACKEND_CUDA;
110  }
111  else if (strBackend == "pva")
112  {
113  backendType = VPI_BACKEND_PVA;
114  }
115  else
116  {
117  throw std::runtime_error("Backend '" + strBackend +
118  "' not recognized, it must be either cpu, cuda or pva.");
119  }
120 
121  // Create the stream for the given backend.
122  CHECK_STATUS(vpiStreamCreate(backendType, &stream));
123 
124  // We now wrap the loaded images into a VPIImage object to be used by VPI.
125  // VPI won't make a copy of it, so the original image must be in scope at all times.
126  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(cvImageLeft, 0, &left));
127  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(cvImageRight, 0, &right));
128 
129  // Create the image where the disparity map will be stored.
130  CHECK_STATUS(vpiImageCreate(cvImageLeft.cols, cvImageLeft.rows, VPI_IMAGE_FORMAT_U16, 0, &disparity));
131 
132  // Create the payload for Harris Corners Detector algorithm
133 
134  CHECK_STATUS(vpiCreateStereoDisparityEstimator(backendType, cvImageLeft.cols, cvImageLeft.rows,
135  VPI_IMAGE_FORMAT_U16, NULL, &stereo));
136 
138  params.windowSize = 5;
139  params.maxDisparity = 64;
140 
141  // Submit it with the input and output images
142  CHECK_STATUS(
143  vpiSubmitStereoDisparityEstimator(stream, backendType, stereo, left, right, disparity, NULL, &params));
144 
145  // Wait until the algorithm finishes processing
146  CHECK_STATUS(vpiStreamSync(stream));
147 
148  // Now let's retrieve the output
149  {
150  // Lock output to retrieve its data on cpu memory
151  VPIImageData data;
152  CHECK_STATUS(vpiImageLock(disparity, VPI_LOCK_READ, &data));
153 
154  // Make an OpenCV matrix out of this image
155  cv::Mat cvOut(data.planes[0].height, data.planes[0].width, CV_16UC1, data.planes[0].data,
156  data.planes[0].pitchBytes);
157 
158  // Scale result and write it to disk
159  double min, max;
160  minMaxLoc(cvOut, &min, &max);
161  cvOut.convertTo(cvOut, CV_8UC1, 255.0 / (max - min), -min);
162 
163  imwrite("disparity_" + strBackend + ".png", cvOut);
164 
165  // Done handling output, don't forget to unlock it.
166  CHECK_STATUS(vpiImageUnlock(disparity));
167  }
168  }
169  catch (std::exception &e)
170  {
171  std::cerr << e.what() << std::endl;
172  retval = 1;
173  }
174 
175  // Clean up
176 
177  // Make sure stream is synchronized before destroying the objects
178  // that might still be in use.
179  if (stream != NULL)
180  {
181  vpiStreamSync(stream);
182  }
183 
184  vpiImageDestroy(left);
185  vpiImageDestroy(right);
186  vpiImageDestroy(disparity);
187  vpiPayloadDestroy(stereo);
188  vpiStreamDestroy(stream);
189 
190  return retval;
191 }
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions that implement stereo disparity estimation algorithms.
Declares functions dealing with VPI streams.
@ VPI_IMAGE_FORMAT_U16
Single plane with one 16-bit unsigned integer channel.
Definition: ImageFormat.h:110
int32_t height
Height of this plane in pixels.
Definition: Image.h:138
int32_t width
Width of this plane in pixels.
Definition: Image.h:137
void * data
Pointer to the first row of this plane.
Definition: Image.h:147
int32_t pitchBytes
Difference in bytes of beginning of one row and the beginning of the previous.
Definition: Image.h:139
VPIImagePlane planes[VPI_MAX_PLANE_COUNT]
Data of all image planes.
Definition: Image.h:166
VPIStatus vpiImageLock(VPIImage img, VPILockMode mode, VPIImageData *hostData)
Acquires the lock on an image object and returns a pointer to the image planes.
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:197
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
Stores information about image characteristics and content.
Definition: Image.h:159
VPIStatus vpiImageCreateOpenCVMatWrapper(const cv::Mat &mat, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:209
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
int32_t windowSize
Width of Census Transform window for disparity features.
int32_t maxDisparity
Maximum disparity for matching search.
VPIStatus vpiSubmitStereoDisparityEstimator(VPIStream stream, uint32_t backend, VPIPayload payload, VPIImage left, VPIImage right, VPIImage disparity, VPIImage confidenceMap, const VPIStereoDisparityEstimatorParams *params)
Runs stereo processing on a pair of images and outputs a disparity map.
VPIStatus vpiCreateStereoDisparityEstimator(uint32_t backends, int32_t imageWidth, int32_t imageHeight, VPIImageFormat inputFormat, const VPIStereoDisparityEstimatorCreationParams *params, VPIPayload *payload)
Creates payload for vpiSubmitStereoDisparityEstimator.
Structure that defines the parameters for vpiSubmitStereoDisparityEstimator.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:191
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:90
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:92
@ VPI_BACKEND_PVA
PVA backend.
Definition: Types.h:93
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:91
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:348