VPI - Vision Programming Interface

2.0 Release - Developer Preview

Perspective Warp

Overview

The Perspective Warp sample application takes an input video and outputs a video where for each frame, a different perspective warp is applied. The result is a perspective bouncy effect. Sample application could be modified to get the input from a camera and apply the effect in real-time.

Instructions

The command line parameters are:

<backend> <input video>

where

  • backend: Either cpu, cuda or vic (on Jetson devices only). It defines the backend that will perform the processing.
  • input video: video file to the effect applied on; it accepts .mp4, .avi and possibly others, depending on OpenCV's support.

VPI samples installer includes some sample videos that can be used as input. They are found in /opt/nvidia/vpi2/samples/assets/ directory.

Here's one invocation example:

  • C++
    ./vpi_sample_10_perspwarp cuda ../assets/noisy.mp4
  • Python
    python3 main.py cuda ../assets/noisy.mp4

The application will process noisy.mp4 and create perspwarp_cuda.mp4 with time-variant perspective warps applied to input frames.

Results

Input videoPerspective effect

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import cv2
28 import sys
29 import vpi
30 import numpy as np
31 from math import sin, cos, pi
32 from argparse import ArgumentParser
33 
34 # ----------------------------
35 # Parse command line arguments
36 
37 parser = ArgumentParser()
38 parser.add_argument('backend', choices=['cpu', 'cuda','vic'],
39  help='Backend to be used for processing')
40 
41 parser.add_argument('input',
42  help='Input video to be denoised')
43 
44 args = parser.parse_args();
45 
46 if args.backend == 'cuda':
47  backend = vpi.Backend.CUDA
48 elif args.backend == 'cpu':
49  backend = vpi.Backend.CPU
50 else:
51  assert args.backend == 'vic'
52  backend = vpi.Backend.VIC
53 
54 # -----------------------------
55 # Open input and output videos
56 
57 inVideo = cv2.VideoCapture(args.input)
58 
59 fourcc = cv2.VideoWriter_fourcc(*'MPEG')
60 inSize = (int(inVideo.get(cv2.CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.CAP_PROP_FRAME_HEIGHT)))
61 fps = inVideo.get(cv2.CAP_PROP_FPS)
62 
63 outVideo = cv2.VideoWriter('perspwarp_python'+str(sys.version_info[0])+'_'+args.backend+'.mp4',
64  fourcc, fps, inSize)
65 
66 #--------------------------------------------------------------
67 # Main processing loop
68 curFrame = 1
69 while True:
70  print("Frame: {}".format(curFrame))
71  curFrame+=1
72 
73  # Read one input frame
74  ret, cvFrame = inVideo.read()
75  if not ret:
76  break
77 
78  # Convert it to NV12_ER format to be used by VPI
79  with vpi.Backend.CUDA:
80  frame = vpi.asimage(cvFrame).convert(vpi.Format.NV12_ER)
81 
82  # Calculate the transformation to be applied ------------
83 
84  # Move image's center to origin of coordinate system
85  T1 = np.array([[1, 0, -frame.width/2.0],
86  [0, 1, -frame.height/2.0],
87  [0, 0, 1]])
88 
89  # Apply some time-dependent perspective transform
90  v1 = sin(curFrame/30.0*2*pi/2)*0.0005
91  v2 = cos(curFrame/30.0*2*pi/3)*0.0005
92  P = np.array([[0.66, 0, 0],
93  [0, 0.66, 0],
94  [v1, v2, 1]])
95 
96  # Move image's center back to where it was
97  T2 = np.array([[1, 0, frame.width/2.0],
98  [0, 1, frame.height/2.0],
99  [0, 0, 1]])
100 
101  # Do perspective warp using the backend passed in the command line.
102  with backend:
103  frame = frame.perspwarp(np.matmul(T2, np.matmul(P, T1)))
104 
105  # Convert it to RGB8 for output using the CUDA backend
106  with vpi.Backend.CUDA:
107  frame = frame.convert(vpi.Format.RGB8)
108 
109  # Write the denoised frame to the output video
110  with frame.rlock_cpu() as data:
111  outVideo.write(data)
29 #include <opencv2/core/version.hpp>
30 #include <opencv2/imgcodecs.hpp>
31 #include <opencv2/imgproc/imgproc.hpp>
32 #include <opencv2/videoio.hpp>
33 #include <vpi/OpenCVInterop.hpp>
34 
35 #include <vpi/Image.h>
36 #include <vpi/Status.h>
37 #include <vpi/Stream.h>
40 
41 #include <algorithm>
42 #include <cstring> // for memset
43 #include <fstream>
44 #include <iostream>
45 #include <random>
46 #include <sstream>
47 
48 #define CHECK_STATUS(STMT) \
49  do \
50  { \
51  VPIStatus status = (STMT); \
52  if (status != VPI_SUCCESS) \
53  { \
54  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
55  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
56  std::ostringstream ss; \
57  ss << vpiStatusGetName(status) << ": " << buffer; \
58  throw std::runtime_error(ss.str()); \
59  } \
60  } while (0);
61 
62 static void MatrixMultiply(VPIPerspectiveTransform &r, const VPIPerspectiveTransform &a,
63  const VPIPerspectiveTransform &b)
64 {
65  for (int i = 0; i < 3; ++i)
66  {
67  for (int j = 0; j < 3; ++j)
68  {
69  r[i][j] = a[i][0] * b[0][j];
70  for (int k = 1; k < 3; ++k)
71  {
72  r[i][j] += a[i][k] * b[k][j];
73  }
74  }
75  }
76 }
77 
78 int main(int argc, char *argv[])
79 {
80  // OpenCV image that will be wrapped by a VPIImage.
81  // Define it here so that it's destroyed *after* wrapper is destroyed
82  cv::Mat cvFrame;
83 
84  // VPI objects that will be used
85  VPIStream stream = NULL;
86  VPIImage imgInput = NULL, imgOutput = NULL;
87  VPIImage frameBGR = NULL;
88 
89  int retval = 0;
90 
91  try
92  {
93  // =============================
94  // Parse command line parameters
95 
96  if (argc != 3)
97  {
98  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|vic|cuda> <input_video>");
99  }
100 
101  std::string strBackend = argv[1];
102  std::string strInputVideo = argv[2];
103 
104  // Now parse the backend
105  VPIBackend backend;
106 
107  if (strBackend == "cpu")
108  {
109  backend = VPI_BACKEND_CPU;
110  }
111  else if (strBackend == "cuda")
112  {
113  backend = VPI_BACKEND_CUDA;
114  }
115  else if (strBackend == "vic")
116  {
117  backend = VPI_BACKEND_VIC;
118  }
119  else
120  {
121  throw std::runtime_error("Backend '" + strBackend +
122  "' not recognized, it must be either cpu, cuda or vic.");
123  }
124 
125  // ===============================
126  // Prepare input and output videos
127 
128  // Load the input video
129  cv::VideoCapture invid;
130  if (!invid.open(strInputVideo))
131  {
132  throw std::runtime_error("Can't open '" + strInputVideo + "'");
133  }
134 
135  // Open the output video for writing using input's characteristics
136  int w = invid.get(cv::CAP_PROP_FRAME_WIDTH);
137  int h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
138  int fourcc = cv::VideoWriter::fourcc('M', 'P', 'E', 'G');
139  double fps = invid.get(cv::CAP_PROP_FPS);
140 
141  cv::VideoWriter outVideo("perspwarp_" + strBackend + ".mp4", fourcc, fps, cv::Size(w, h));
142  if (!outVideo.isOpened())
143  {
144  throw std::runtime_error("Can't create output video");
145  }
146 
147  // =================================
148  // Allocate all VPI resources needed
149 
150  // Create the stream for the given backend. We'll be using CUDA for image format conversion.
151  CHECK_STATUS(vpiStreamCreate(backend | VPI_BACKEND_CUDA, &stream));
152 
153  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgInput));
154  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgOutput));
155 
157  memset(&xform, 0, sizeof(xform));
158 
159  // ====================
160  // Main processing loop
161 
162  int curFrame = 1;
163  while (invid.read(cvFrame))
164  {
165  printf("Frame: %d\n", curFrame++);
166 
167  if (frameBGR == NULL)
168  {
169  // Ceate a VPIImage that wraps the frame
170  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvFrame, 0, &frameBGR));
171  }
172  else
173  {
174  // reuse existing VPIImage wrapper to wrap the new frame.
175  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(frameBGR, cvFrame));
176  }
177 
178  // First convert it to NV12 using CUDA
179  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, frameBGR, imgInput, NULL));
180 
181  // move image's center to origin of coordinate system
182  VPIPerspectiveTransform t1 = {{1, 0, -w / 2.0f}, {0, 1, -h / 2.0f}, {0, 0, 1}};
183 
184  // Apply some time-dependent perspective transform
185  float v1 = sin(curFrame / 30.0 * 2 * M_PI / 2) * 0.0005f;
186  float v2 = cos(curFrame / 30.0 * 2 * M_PI / 3) * 0.0005f;
187  VPIPerspectiveTransform P = {{0.66, 0, 0}, {0, 0.66, 0}, {v1, v2, 1}};
188 
189  // move image's center back to where it was.
190  VPIPerspectiveTransform t2 = {{1, 0, w / 2.0f}, {0, 1, h / 2.0f}, {0, 0, 1}};
191 
192  // Apply the transforms defined above.
194  MatrixMultiply(tmp, P, t1);
195  MatrixMultiply(xform, t2, tmp);
196 
197  // Do perspective warp using the backend passed in the command line.
198  // Passing NULL as grid to make it use a dense grid, for better quality.
199  CHECK_STATUS(vpiSubmitPerspectiveWarp(stream, backend, imgInput, xform, imgOutput, NULL, VPI_INTERP_LINEAR,
200  VPI_BORDER_ZERO, 0));
201 
202  // Convert output back to BGR using CUDA
203  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgOutput, frameBGR, NULL));
204  CHECK_STATUS(vpiStreamSync(stream));
205 
206  // Now add it to the output video stream
207  VPIImageData imgdata;
208  CHECK_STATUS(vpiImageLockData(frameBGR, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &imgdata));
209 
210  cv::Mat outFrame;
211  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
212  outVideo << outFrame;
213 
214  CHECK_STATUS(vpiImageUnlock(frameBGR));
215  }
216  }
217  catch (std::exception &e)
218  {
219  std::cerr << e.what() << std::endl;
220  retval = 1;
221  }
222 
223  // =========================
224  // Destroy all VPI resources
225 
226  vpiStreamDestroy(stream);
227  vpiImageDestroy(imgInput);
228  vpiImageDestroy(imgOutput);
229  vpiImageDestroy(frameBGR);
230 
231  return retval;
232 }
Declares functions that handle image format conversion.
#define VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:206
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declares functions that implement the Perspective Warp algorithm.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:230
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiSubmitPerspectiveWarp(VPIStream stream, uint64_t backend, VPIImage input, const VPIPerspectiveTransform xform, VPIImage output, const VPIWarpGrid *grid, VPIInterpolationType interp, VPIBorderExtension border, uint64_t flags)
Submits a Perspective Warp operation to the stream.
float VPIPerspectiveTransform[3][3]
Represents a 2D perspective transform.
Definition: Types.h:507
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_VIC
VIC backend.
Definition: Types.h:95
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:92
@ VPI_BORDER_ZERO
All pixels outside the image are considered to be zero.
Definition: Types.h:278
@ VPI_INTERP_LINEAR
Linear interpolation.
Definition: Interpolation.h:93
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:435