VPI - Vision Programming Interface

1.0 Release

Fisheye Distortion Correction

Overview

This sample application performs a fisheye lens calibration using input images taken with the same camera/lens. Then it uses Remap and the calibration data to correct fisheye lens distortion of these images and save the result to disk. The mapping used for distortion correction is VPI_FISHEYE_EQUIDISTANT, which maps straight lines in the scene to straight lines in the corrected image.

This sample shows the following:

  • Creating and destroying a VPI stream.
  • Use OpenCV to estimate the intrinsic and distortion parameters of a fisheye lens given a set of calibration images.
  • Create a VPIWarpMap and use vpiWarpMapGenerateFromFisheyeLensDistortionModel to initialize it to correct the distortion caused by a fisheye lens.
  • Create pipeline that does Convert Image Format to convert from/to NV12 format and run Remap to perform the lens distortion correction.

Lens Calibration

Lens calibration uses a set of images taken by the same camera/lens, each one showing a checkerboard pattern in a different position, so that taken collectively, the checkerboard appears in almost entire field of view. The more images, the more accurate the calibration will be, but typically 10 to 15 images suffice.

Note
On Ubuntu 16.04, the sample code requires OpenCV >= 2.4.10, which isn't available using apt.

VPI samples include a set of input images that can be used. They are found in /opt/nvidia/vpi1/samples/assets/fisheye directory.

To create a set of calibration images for a given lens, do the following:

  1. Print a checkerboard pattern on a piece of paper. VPI provides in samples' assets directory one 10x7 checkerboard file that can be used, named checkerboard_10x7.pdf.
  2. Mount the fisheye lens on a camera.
  3. With the camera in a fixed position, take several pictures showing the checkerboard in different positions, covering a good part of the field of view.

Instructions

The usage is:

./vpi_sample_11_fisheye -c W,H [-s win] <image1> [image2] [image3] ...

where

  • -c W,H: specifies the number of squares the checkerboard pattern has horizontally (W) and vertically (H).
  • -s win: (optional) the width of a window around each internal vertex of the checkerboard (point where 4 squares meet) to be used in a vertex position refinement stage. The actual vertex position will be searched within this window. If this parameter is omitted, the refinement stage will be skipped.
  • imageN: set of calibration images
Note
Since currently only the PVA backend implements Remap, and only on Jetson Xavier series, this sample can be run solely in these devices.

Here's one invocation example:

./vpi_sample_11_fisheye -c 10,7 -s 22 ../assets/fisheye/*.jpg

This will correct the included set of calibration images, all captured using the checkerboard pattern also included. It's using a 22x22 window around each checkerboard internal vertex to refine the vertex position.

Results

Here are some input and output images produced by the sample application:

InputCorrected

Source Code

For convenience, here's the code that is also installed in the samples directory.

1 /*
2 * Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without
5 * modification, are permitted provided that the following conditions
6 * are met:
7 * * Redistributions of source code must retain the above copyright
8 * notice, this list of conditions and the following disclaimer.
9 * * Redistributions in binary form must reproduce the above copyright
10 * notice, this list of conditions and the following disclaimer in the
11 * documentation and/or other materials provided with the distribution.
12 * * Neither the name of NVIDIA CORPORATION nor the names of its
13 * contributors may be used to endorse or promote products derived
14 * from this software without specific prior written permission.
15 *
16 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 */
28 
29 #include <opencv2/core/version.hpp>
30 
31 #if CV_MAJOR_VERSION >= 3
32 # include <opencv2/imgcodecs.hpp>
33 #else
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/calib3d/calib3d.hpp>
38 #include <opencv2/imgproc/imgproc.hpp>
39 #include <vpi/OpenCVInterop.hpp>
40 
41 #include <string.h> // for basename(3) that doesn't modify its argument
42 #include <unistd.h> // for getopt
43 #include <vpi/Context.h>
44 #include <vpi/Image.h>
46 #include <vpi/Status.h>
47 #include <vpi/Stream.h>
49 #include <vpi/algo/Remap.h>
50 
51 #include <iostream>
52 #include <sstream>
53 
54 #define CHECK_STATUS(STMT) \
55  do \
56  { \
57  VPIStatus status = (STMT); \
58  if (status != VPI_SUCCESS) \
59  { \
60  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
61  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
62  std::ostringstream ss; \
63  ss << vpiStatusGetName(status) << ": " << buffer; \
64  throw std::runtime_error(ss.str()); \
65  } \
66  } while (0);
67 
68 static void PrintUsage(const char *progname, std::ostream &out)
69 {
70  out << "Usage: " << progname << " <-c W,H> [-s win] <image1> [image2] [image3] ...\n"
71  << " where,\n"
72  << " W,H\tcheckerboard with WxH squares\n"
73  << " win\tsearch window width around checkerboard vertex used\n"
74  << "\tin refinement, default is 0 (disable refinement)\n"
75  << " imageN\tinput images taken with a fisheye lens camera" << std::endl;
76 }
77 
78 struct Params
79 {
80  cv::Size vtxCount; // Number of internal vertices the checkerboard has
81  int searchWinSize; // search window size around the checkerboard vertex for refinement.
82  std::vector<const char *> images; // input image names.
83 };
84 
85 static Params ParseParameters(int argc, char *argv[])
86 {
87  Params params = {};
88 
89  cv::Size cbSize;
90 
91  opterr = 0;
92  int opt;
93  while ((opt = getopt(argc, argv, "hc:s:")) != -1)
94  {
95  switch (opt)
96  {
97  case 'h':
98  PrintUsage(basename(argv[0]), std::cout);
99  return {};
100 
101  case 'c':
102  if (sscanf(optarg, "%d,%d", &cbSize.width, &cbSize.height) != 2)
103  {
104  throw std::invalid_argument("Error parsing checkerboard information");
105  }
106 
107  // OpenCV expects number of interior vertices in the checkerboard,
108  // not number of squares. Let's adjust for that.
109  params.vtxCount.width = cbSize.width - 1;
110  params.vtxCount.height = cbSize.height - 1;
111  break;
112 
113  case 's':
114  if (sscanf(optarg, "%d", &params.searchWinSize) != 1)
115  {
116  throw std::invalid_argument("Error parseing search window size");
117  }
118  if (params.searchWinSize < 0)
119  {
120  throw std::invalid_argument("Search window size must be >= 0");
121  }
122  break;
123  case '?':
124  throw std::invalid_argument(std::string("Option -") + (char)optopt + " not recognized");
125  }
126  }
127 
128  for (int i = optind; i < argc; ++i)
129  {
130  params.images.push_back(argv[i]);
131  }
132 
133  if (params.images.empty())
134  {
135  throw std::invalid_argument("At least one image must be defined");
136  }
137 
138  if (cbSize.width <= 3 || cbSize.height <= 3)
139  {
140  throw std::invalid_argument("Checkerboard size must have at least 3x3 squares");
141  }
142 
143  if (params.searchWinSize == 1)
144  {
145  throw std::invalid_argument("Search window size must be 0 (default) or >= 2");
146  }
147 
148  return params;
149 }
150 
151 int main(int argc, char *argv[])
152 {
153  // We'll create all vpi objects under this context, so that
154  // we don't have to track what objects to destroy. Just destroying
155  // the context will destroy all objects.
156  VPIContext ctx = 0;
157 
158  try
159  {
160  // First parse command line paramers
161  Params params = ParseParameters(argc, argv);
162  if (params.images.empty()) // user just wanted the help message?
163  {
164  return 0;
165  }
166 
167  // Where to store checkerboard 2D corners of each input image.
168  std::vector<std::vector<cv::Point2f>> corners2D;
169 
170  // Store image size. All input images must have same size.
171  cv::Size imgSize = {};
172 
173  for (unsigned i = 0; i < params.images.size(); ++i)
174  {
175  // Load input image and do some sanity check
176  cv::Mat img = cv::imread(params.images[i]);
177  if (img.empty())
178  {
179  throw std::runtime_error("Can't read " + std::string(params.images[i]));
180  }
181 
182  if (imgSize == cv::Size{})
183  {
184  imgSize = img.size();
185  }
186  else if (imgSize != img.size())
187  {
188  throw std::runtime_error("All images must have same size");
189  }
190 
191  // Find the checkerboard pattern on the image, saving the 2D
192  // coordinates of checkerboard vertices in cbVertices.
193  // Vertex is the point where 4 squares (2 white and 2 black) meet.
194  std::vector<cv::Point2f> cbVertices;
195 
196  if (findChessboardCorners(img, params.vtxCount, cbVertices,
197  cv::CALIB_CB_ADAPTIVE_THRESH + cv::CALIB_CB_NORMALIZE_IMAGE))
198  {
199  // Needs to perform further corner refinement?
200  if (params.searchWinSize >= 2)
201  {
202  cv::Mat gray;
203  cvtColor(img, gray, cv::COLOR_BGR2GRAY);
204 
205  cornerSubPix(gray, cbVertices, cv::Size(params.searchWinSize / 2, params.searchWinSize / 2),
206  cv::Size(-1, -1),
207  cv::TermCriteria(cv::TermCriteria::EPS + cv::TermCriteria::COUNT, 30, 0.0001));
208  }
209 
210  // save this image's 2D vertices in vector
211  corners2D.push_back(std::move(cbVertices));
212  }
213  else
214  {
215  std::cerr << "Warning: checkerboard pattern not found in image " << params.images[i] << std::endl;
216  }
217  }
218 
219  // Create the vector that stores 3D coordinates for each checkerboard pattern on a space
220  // where X and Y are orthogonal and run along the checkerboard sides, and Z==0 in all points on
221  // checkerboard.
222  std::vector<cv::Point3f> initialCheckerboard3DVertices;
223  for (int i = 0; i < params.vtxCount.height; ++i)
224  {
225  for (int j = 0; j < params.vtxCount.width; ++j)
226  {
227  // since we're not interested in extrinsic camera parameters,
228  // we can assume that checkerboard square size is 1x1.
229  initialCheckerboard3DVertices.emplace_back(j, i, 0);
230  }
231  }
232 
233  // Initialize a vector with initial checkerboard positions for all images
234  std::vector<std::vector<cv::Point3f>> corners3D(corners2D.size(), initialCheckerboard3DVertices);
235 
236  // Camera intrinsic parameters, initially identity (will be estimated by calibration process).
237  using Mat3 = cv::Matx<double, 3, 3>;
238  Mat3 camMatrix = Mat3::eye();
239 
240  // stores the fisheye model coefficients.
241  std::vector<double> coeffs(4);
242 
243  // VPI currently doesn't support skew parameter on camera matrix, make sure
244  // calibration process fixes it to 0.
245  int flags = cv::fisheye::CALIB_FIX_SKEW;
246 
247  // Run calibration
248  {
249  cv::Mat rvecs, tvecs; // stores rotation and translation for each camera, not needed now.
250  double rms = cv::fisheye::calibrate(corners3D, corners2D, imgSize, camMatrix, coeffs, rvecs, tvecs, flags);
251  printf("rms error: %lf\n", rms);
252  }
253 
254  // Output calibration result.
255  printf("Fisheye coefficients: %lf %lf %lf %lf\n", coeffs[0], coeffs[1], coeffs[2], coeffs[3]);
256 
257  printf("Camera matrix:\n");
258  printf("[%lf %lf %lf; %lf %lf %lf; %lf %lf %lf]\n", camMatrix(0, 0), camMatrix(0, 1), camMatrix(0, 2),
259  camMatrix(1, 0), camMatrix(1, 1), camMatrix(1, 2), camMatrix(2, 0), camMatrix(2, 1), camMatrix(2, 2));
260 
261  // Now use VPI to undistort the input images:
262 
263  // Allocate a dense map.
264  VPIWarpMap map = {};
265  map.grid.numHorizRegions = 1;
266  map.grid.numVertRegions = 1;
267  map.grid.regionWidth[0] = imgSize.width;
268  map.grid.regionHeight[0] = imgSize.height;
269  map.grid.horizInterval[0] = 1;
270  map.grid.vertInterval[0] = 1;
271  CHECK_STATUS(vpiWarpMapAllocData(&map));
272 
273  // Initialize the fisheye lens model with the coefficients given by calibration procedure.
274  VPIFisheyeLensDistortionModel distModel = {};
275  distModel.mapping = VPI_FISHEYE_EQUIDISTANT;
276  distModel.k1 = coeffs[0];
277  distModel.k2 = coeffs[1];
278  distModel.k3 = coeffs[2];
279  distModel.k4 = coeffs[3];
280 
281  // Fill up the camera intrinsic parameters given by camera calibration procedure.
283  for (int i = 0; i < 2; ++i)
284  {
285  for (int j = 0; j < 3; ++j)
286  {
287  K[i][j] = camMatrix(i, j);
288  }
289  }
290 
291  // Camera extrinsics is be identity.
292  VPICameraExtrinsic X = {};
293  X[0][0] = X[1][1] = X[2][2] = 1;
294 
295  // Generate a warp map to undistort an image taken from fisheye lens with
296  // given parameters calculated above.
297  vpiWarpMapGenerateFromFisheyeLensDistortionModel(K, X, K, &distModel, &map);
298 
299  // Create out a vpi context to store all vpi objects we'll create.
300  CHECK_STATUS(vpiContextCreate(0, &ctx));
301  // Activate it. From now on all created objects will be owned by it.
302  CHECK_STATUS(vpiContextSetCurrent(ctx));
303 
304  // Create a stream where operations will take place. We're using CUDA
305  // processing.
306  VPIStream stream;
307  CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA, &stream));
308 
309  // Create the Remap payload for undistortion given the map generated above.
310  VPIPayload remap;
311  CHECK_STATUS(vpiCreateRemap(VPI_BACKEND_CUDA, &map, &remap));
312 
313  // Temporary input and output images in NV12 format.
314  VPIImage tmpIn;
315  CHECK_STATUS(vpiImageCreate(imgSize.width, imgSize.height, VPI_IMAGE_FORMAT_NV12_ER, 0, &tmpIn));
316 
317  VPIImage tmpOut;
318  CHECK_STATUS(vpiImageCreate(imgSize.width, imgSize.height, VPI_IMAGE_FORMAT_NV12_ER, 0, &tmpOut));
319 
320  VPIImage vimg = nullptr;
321 
322  // For each input image,
323  for (unsigned i = 0; i < params.images.size(); ++i)
324  {
325  // Read it from disk.
326  cv::Mat img = cv::imread(params.images[i]);
327  assert(!img.empty());
328 
329  // Wrap it into a VPIImage
330  if (vimg == nullptr)
331  {
332  // Now create a VPIImage that wraps it.
333  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(img, 0, &vimg));
334  }
335  else
336  {
337  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(vimg, img));
338  }
339 
340  // Convert BGR -> NV12
341  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, vimg, tmpIn, NULL));
342 
343  // Undistorts the input image.
344  CHECK_STATUS(vpiSubmitRemap(stream, VPI_BACKEND_CUDA, remap, tmpIn, tmpOut, VPI_INTERP_CATMULL_ROM,
345  VPI_BORDER_ZERO, 0));
346 
347  // Convert the result NV12 back to BGR, writing back to the input image.
348  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, tmpOut, vimg, NULL));
349 
350  // Wait until conversion finishes.
351  CHECK_STATUS(vpiStreamSync(stream));
352 
353  // Since vimg is wrapping the OpenCV image, the result is already there.
354  // We just have to save it to disk.
355  char buf[64];
356  snprintf(buf, sizeof(buf), "undistort_%03d.jpg", i);
357  imwrite(buf, img);
358  }
359  }
360  catch (std::exception &e)
361  {
362  std::cerr << "Error: " << e.what() << std::endl;
363  PrintUsage(basename(argv[0]), std::cerr);
364 
365  if (ctx != nullptr)
366  {
367  vpiContextDestroy(ctx);
368  }
369  return 1;
370  }
371 
372  if (ctx != nullptr)
373  {
374  vpiContextDestroy(ctx);
375  }
376  return 0;
377 }
Functions and structures for dealing with VPI contexts.
Declares functions that handle image format conversion.
Functions and structures for dealing with VPI images.
Declares functions to generate warp maps based on common lens distortion models.
Functions for handling OpenCV interoperability with VPI.
Declares functions that implement the Remap algorithm.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
VPIStatus vpiContextSetCurrent(VPIContext ctx)
Sets the context for the calling thread.
void vpiContextDestroy(VPIContext ctx)
Destroy a context instance as well as all resources it owns.
VPIStatus vpiContextCreate(uint32_t flags, VPIContext *ctx)
Create a context instance.
struct VPIContextImpl * VPIContext
A handle to a context.
Definition: Types.h:179
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint32_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
@ VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:152
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:197
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiWarpMapGenerateFromFisheyeLensDistortionModel(const VPICameraIntrinsic Kin, const VPICameraExtrinsic X, const VPICameraIntrinsic Kout, const VPIFisheyeLensDistortionModel *distModel, VPIWarpMap *warpMap)
Generates a mapping that corrects image distortions caused by fisheye lenses.
float VPICameraExtrinsic[3][4]
Camera extrinsic matrix.
Definition: Types.h:399
float VPICameraIntrinsic[2][3]
Camera intrinsic matrix.
Definition: Types.h:386
@ VPI_FISHEYE_EQUIDISTANT
Specifies the equidistant fisheye mapping.
Holds coefficients for fisheye lens distortion model.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiImageCreateOpenCVMatWrapper(const cv::Mat &mat, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:209
VPIStatus vpiSubmitRemap(VPIStream stream, uint32_t backend, VPIPayload payload, VPIImage input, VPIImage output, VPIInterpolationType interp, VPIBorderExtension border, uint32_t flags)
Submits the Remap operation to the stream associated with the payload.
VPIStatus vpiCreateRemap(uint32_t backends, const VPIWarpMap *warpMap, VPIPayload *payload)
Create a payload for Remap algorithm.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:191
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:92
@ VPI_BORDER_ZERO
All pixels outside the image are considered to be zero.
Definition: Types.h:219
@ VPI_INTERP_CATMULL_ROM
Catmull-Rom cubic interpolation.
int8_t numHorizRegions
Number of regions horizontally.
Definition: WarpGrid.h:158
VPIWarpGrid grid
Warp grid control point structure definition.
Definition: WarpMap.h:91
int16_t horizInterval[VPI_WARPGRID_MAX_HORIZ_REGIONS_COUNT]
Horizontal spacing between control points within a given region.
Definition: WarpGrid.h:163
int8_t numVertRegions
Number of regions vertically.
Definition: WarpGrid.h:159
int16_t vertInterval[VPI_WARPGRID_MAX_VERT_REGIONS_COUNT]
Vertical spacing between control points within a given region.
Definition: WarpGrid.h:165
int16_t regionWidth[VPI_WARPGRID_MAX_HORIZ_REGIONS_COUNT]
Width of each region.
Definition: WarpGrid.h:161
int16_t regionHeight[VPI_WARPGRID_MAX_VERT_REGIONS_COUNT]
Height of each region.
Definition: WarpGrid.h:162
VPIStatus vpiWarpMapAllocData(VPIWarpMap *warpMap)
Allocates the warp map's control point array for a given warp grid.
Defines the mapping between input and output images' pixels.
Definition: WarpMap.h:88