VPI - Vision Programming Interface

2.0 Release - Developer Preview

Dense Optical Flow

Overview

This application fetches frames from input video source, runs the algorithms on the previous and current images, and then calculate the motion vectors for every 4x4 pixel block. The output motion vectors will be mapped to the HSV colorspace, where hue relates to motion angle, value relates to motion speed, and the result will be saved to a video file.

Instructions

The command line parameters are:

<backend> <input video> <quality>

where

  • backend: Defines the backend that will perform the processing. Only nvenc is currently supported.
  • input video: Input video file name, it accepts .mp4, .avi and possibly others, depending on OpenCV's support.
  • quality: Specify the quality that the algorithm will use. Available options are: low (fastest), medium (balanced perf and quality) and high (slowest).

Here's one example:

  • C++
    ./vpi_sample_13_optflow_dense nvenc ../assets/pedestrians.mp4 high
  • Python
    python3 main.py nvenc ../assets/pedestrians.mp4 high

The application will process pedestrians.mp4 and create denseoptflow_mv_nvenc.mp4.

Results

Input videoMotion vector video

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import cv2
28 import sys
29 import vpi
30 import numpy as np
31 from os import path
32 from argparse import ArgumentParser
33 from contextlib import contextmanager
34 
35 # ----------------------------
36 # Some utility functions
37 
38 def process_motion_vectors(mv):
39  with mv.rlock_cpu() as data:
40  # convert S10.5 format to float
41  flow = np.float32(data)/(1<<5)
42 
43  # Create an image where the motion vector angle is
44  # mapped to a color hue, and intensity is proportional
45  # to vector's magnitude
46  magnitude, angle = cv2.cartToPolar(flow[:,:,0], flow[:,:,1], angleInDegrees=True)
47 
48  clip = 5.0
49  cv2.threshold(magnitude, clip, clip, cv2.THRESH_TRUNC, magnitude)
50 
51  # build the hsv image
52  hsv = np.ndarray([flow.shape[0], flow.shape[1], 3], np.float32)
53  hsv[:,:,0] = angle
54  hsv[:,:,1] = np.ones((angle.shape[0], angle.shape[1]), np.float32)
55  hsv[:,:,2] = magnitude / clip
56 
57  # Convert HSV to BGR8
58  bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
59  return np.uint8(bgr*255)
60 
61 # ----------------------------
62 # Parse command line arguments
63 
64 parser = ArgumentParser()
65 parser.add_argument('backend', choices=['nvenc'],
66  help='Backend to be used for processing')
67 
68 parser.add_argument('input',
69  help='Input video to be processed')
70 
71 parser.add_argument('quality', choices=['low', 'medium', 'high'],
72  help='Quality setting')
73 
74 args = parser.parse_args();
75 
76 assert args.backend == 'nvenc'
77 backend = vpi.Backend.NVENC
78 
79 if args.quality == "low":
80  quality = vpi.OptFlowQuality.LOW
81 elif args.quality == "medium":
82  quality = vpi.OptFlowQuality.MEDIUM
83 else:
84  assert args.quality == "high"
85  quality = vpi.OptFlowQuality.HIGH
86 
87 # -----------------------------
88 # Open input and output videos
89 
90 inVideo = cv2.VideoCapture(args.input)
91 
92 fourcc = cv2.VideoWriter_fourcc(*'MPEG')
93 inSize = (int(inVideo.get(cv2.CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.CAP_PROP_FRAME_HEIGHT)))
94 fps = inVideo.get(cv2.CAP_PROP_FPS)
95 
96 if backend == vpi.Backend.NVENC:
97  # NVENC always returns 1/4th resolution
98  outSize = (inSize[0]//4, inSize[1]//4)
99 else:
100  outSize = inSize
101 
102 outVideo = cv2.VideoWriter('denseoptflow_mv_python'+str(sys.version_info[0])+'_'+args.backend+'.mp4',
103  fourcc, fps, outSize)
104 
105 #---------------------------------
106 # Main processing loop
107 
108 prevFrame = None
109 
110 idFrame = 0
111 while True:
112  # Read one input frame
113  ret, cvFrame = inVideo.read()
114  if not ret:
115  break
116 
117  # Convert it to NV12_ER format to be used by VPI
118  # No single backend can convert from OpenCV's BGR8 to NV12_ER_BL
119  # required by the algorithm. We must do in two steps using CUDA and VIC.
120  curFrame = vpi.asimage(cvFrame, vpi.Format.BGR8) \
121  .convert(vpi.Format.NV12_ER, backend=vpi.Backend.CUDA) \
122  .convert(vpi.Format.NV12_ER_BL, backend=vpi.Backend.VIC)
123 
124  # Need at least 2 frames to start processing
125  if prevFrame is not None:
126  print("Processing frame {}".format(idFrame))
127 
128  # Calculate the motion vectors from previous to current frame
129  with backend:
130  motion_vectors = vpi.optflow_dense(prevFrame, curFrame, quality = quality)
131 
132  # Turn motion vectors into an image
133  motion_image = process_motion_vectors(motion_vectors)
134 
135  # Save it to output video
136  outVideo.write(motion_image)
137 
138  # Prepare next iteration
139  prevFrame = curFrame
140  idFrame += 1
29 #include <opencv2/core/version.hpp>
30 #include <opencv2/imgcodecs.hpp>
31 #include <opencv2/imgproc/imgproc.hpp>
32 #include <opencv2/videoio.hpp>
33 #include <vpi/OpenCVInterop.hpp>
34 
35 #include <vpi/Array.h>
36 #include <vpi/Image.h>
37 #include <vpi/ImageFormat.h>
38 #include <vpi/Pyramid.h>
39 #include <vpi/Status.h>
40 #include <vpi/Stream.h>
43 
44 #include <iostream>
45 #include <sstream>
46 
47 #define CHECK_STATUS(STMT) \
48  do \
49  { \
50  VPIStatus status = (STMT); \
51  if (status != VPI_SUCCESS) \
52  { \
53  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
54  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
55  std::ostringstream ss; \
56  ss << vpiStatusGetName(status) << ": " << buffer; \
57  throw std::runtime_error(ss.str()); \
58  } \
59  } while (0);
60 
61 static void ProcessMotionVector(VPIImage mvImg, cv::Mat &outputImage)
62 {
63  // Lock the input image to access it from CPU
64  VPIImageData mvData;
66 
67  // Create a cv::Mat that points to the input image data
68  cv::Mat mvImage;
69  CHECK_STATUS(vpiImageDataExportOpenCVMat(mvData, &mvImage));
70 
71  // Convert S10.5 format to float
72  cv::Mat flow(mvImage.size(), CV_32FC2);
73  mvImage.convertTo(flow, CV_32F, 1.0f / (1 << 5));
74 
75  // Image not needed anymore, we can unlock it.
76  CHECK_STATUS(vpiImageUnlock(mvImg));
77 
78  // Create an image where the motion vector angle is
79  // mapped to a color hue, and intensity is proportional
80  // to vector's magnitude.
81  cv::Mat magnitude, angle;
82  {
83  cv::Mat flowChannels[2];
84  split(flow, flowChannels);
85  cv::cartToPolar(flowChannels[0], flowChannels[1], magnitude, angle, true);
86  }
87 
88  float clip = 5;
89  cv::threshold(magnitude, magnitude, clip, clip, cv::THRESH_TRUNC);
90 
91  // build hsv image
92  cv::Mat _hsv[3], hsv, bgr;
93  _hsv[0] = angle;
94  _hsv[1] = cv::Mat::ones(angle.size(), CV_32F);
95  _hsv[2] = magnitude / clip; // intensity must vary from 0 to 1
96  merge(_hsv, 3, hsv);
97 
98  cv::cvtColor(hsv, bgr, cv::COLOR_HSV2BGR);
99  bgr.convertTo(outputImage, CV_8U, 255.0);
100 }
101 
102 int main(int argc, char *argv[])
103 {
104  // OpenCV image that will be wrapped by a VPIImage.
105  // Define it here so that it's destroyed *after* wrapper is destroyed
106  cv::Mat cvPrevFrame, cvCurFrame;
107 
108  // VPI objects that will be used
109  VPIStream stream = NULL;
110  VPIImage imgPrevFramePL = NULL;
111  VPIImage imgPrevFrameTmp = NULL;
112  VPIImage imgPrevFrameBL = NULL;
113  VPIImage imgCurFramePL = NULL;
114  VPIImage imgCurFrameTmp = NULL;
115  VPIImage imgCurFrameBL = NULL;
116  VPIImage imgMotionVecBL = NULL;
117  VPIPayload payload = NULL;
118 
119  int retval = 0;
120 
121  try
122  {
123  if (argc != 4)
124  {
125  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <nvenc> <input_video> <low|medium|high>");
126  }
127 
128  // Parse input parameters
129  std::string strBackend = argv[1];
130  std::string strInputVideo = argv[2];
131  std::string strQuality = argv[3];
132 
133  VPIOpticalFlowQuality quality;
134  if (strQuality == "low")
135  {
137  }
138  else if (strQuality == "medium")
139  {
141  }
142  else if (strQuality == "high")
143  {
145  }
146  else
147  {
148  throw std::runtime_error("Unknown quality provided");
149  }
150 
151  VPIBackend backend;
152  if (strBackend == "nvenc")
153  {
154  backend = VPI_BACKEND_NVENC;
155  }
156  else
157  {
158  throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be nvenc.");
159  }
160 
161  // Load the input video
162  cv::VideoCapture invid;
163  if (!invid.open(strInputVideo))
164  {
165  throw std::runtime_error("Can't open '" + strInputVideo + "'");
166  }
167 
168  // Create the stream where processing will happen. We'll use user-provided backend
169  // for Optical Flow, and CUDA/VIC for image format conversions.
170  CHECK_STATUS(vpiStreamCreate(backend | VPI_BACKEND_CUDA | VPI_BACKEND_VIC, &stream));
171 
172  // Fetch the first frame
173  if (!invid.read(cvPrevFrame))
174  {
175  throw std::runtime_error("Cannot read frame from input video");
176  }
177 
178  // Create the previous and current frame wrapper using the first frame. This wrapper will
179  // be set to point to every new frame in the main loop.
180  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvPrevFrame, 0, &imgPrevFramePL));
181  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvPrevFrame, 0, &imgCurFramePL));
182 
183  // Define the image formats we'll use throughout this sample.
186 
187  int32_t width = cvPrevFrame.cols;
188  int32_t height = cvPrevFrame.rows;
189 
190  // Create Dense Optical Flow payload to be executed on the given backend
191  CHECK_STATUS(vpiCreateOpticalFlowDense(backend, width, height, imgFmtBL, quality, &payload));
192 
193  // The Dense Optical Flow on NVENC backend expects input to be in block-linear format.
194  // Since Convert Image Format algorithm doesn't currently support direct BGR
195  // pitch-linear (from OpenCV) to NV12 block-linear conversion, it must be done in two
196  // passes, first from BGR/PL to NV12/PL using CUDA, then from NV12/PL to NV12/BL using VIC.
197  // The temporary image buffer below will store the intermediate NV12/PL representation.
198  CHECK_STATUS(vpiImageCreate(width, height, imgFmt, 0, &imgPrevFrameTmp));
199  CHECK_STATUS(vpiImageCreate(width, height, imgFmt, 0, &imgCurFrameTmp));
200 
201  // Now create the final block-linear buffer that'll be used as input to the
202  // algorithm.
203  CHECK_STATUS(vpiImageCreate(width, height, imgFmtBL, 0, &imgPrevFrameBL));
204  CHECK_STATUS(vpiImageCreate(width, height, imgFmtBL, 0, &imgCurFrameBL));
205 
206  // Motion vector image width and height, align to be multiple of 4
207  int32_t mvWidth = (width + 3) / 4;
208  int32_t mvHeight = (height + 3) / 4;
209 
210  // The output video will be heatmap of motion vector image
211  int fourcc = cv::VideoWriter::fourcc('M', 'P', 'E', 'G');
212  double fps = invid.get(cv::CAP_PROP_FPS);
213 
214  cv::VideoWriter outVideo("denseoptflow_mv_" + strBackend + ".mp4", fourcc, fps, cv::Size(mvWidth, mvHeight));
215  if (!outVideo.isOpened())
216  {
217  throw std::runtime_error("Can't create output video");
218  }
219 
220  // Create the output motion vector buffer
221  CHECK_STATUS(vpiImageCreate(mvWidth, mvHeight, VPI_IMAGE_FORMAT_2S16_BL, 0, &imgMotionVecBL));
222 
223  // First convert the first frame to NV12_BL. It'll be used as previous frame when the algorithm is called.
224  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgPrevFramePL, imgPrevFrameTmp, nullptr));
225  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_VIC, imgPrevFrameTmp, imgPrevFrameBL, nullptr));
226 
227  // Create a output image which holds the rendered motion vector image.
228  cv::Mat mvOutputImage;
229 
230  // Fetch a new frame until video ends
231  int idxFrame = 1;
232  while (invid.read(cvCurFrame))
233  {
234  printf("Processing frame %d\n", idxFrame++);
235  // Wrap frame into a VPIImage, reusing the existing imgCurFramePL.
236  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(imgCurFramePL, cvCurFrame));
237 
238  // Convert current frame to NV12_BL format
239  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgCurFramePL, imgCurFrameTmp, nullptr));
240  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_VIC, imgCurFrameTmp, imgCurFrameBL, nullptr));
241 
242  CHECK_STATUS(
243  vpiSubmitOpticalFlowDense(stream, backend, payload, imgPrevFrameBL, imgCurFrameBL, imgMotionVecBL));
244 
245  // Wait for processing to finish.
246  CHECK_STATUS(vpiStreamSync(stream));
247 
248  // Render the resulting motion vector in the output image
249  ProcessMotionVector(imgMotionVecBL, mvOutputImage);
250 
251  // Save to output video
252  outVideo << mvOutputImage;
253 
254  // Swap previous frame and next frame
255  std::swap(cvPrevFrame, cvCurFrame);
256  std::swap(imgPrevFramePL, imgCurFramePL);
257  std::swap(imgPrevFrameBL, imgCurFrameBL);
258  }
259  }
260  catch (std::exception &e)
261  {
262  std::cerr << e.what() << std::endl;
263  retval = 1;
264  }
265 
266  // Destroy all resources used
267  vpiStreamDestroy(stream);
268  vpiPayloadDestroy(payload);
269 
270  vpiImageDestroy(imgPrevFramePL);
271  vpiImageDestroy(imgPrevFrameTmp);
272  vpiImageDestroy(imgPrevFrameBL);
273  vpiImageDestroy(imgCurFramePL);
274  vpiImageDestroy(imgCurFrameTmp);
275  vpiImageDestroy(imgCurFrameBL);
276  vpiImageDestroy(imgMotionVecBL);
277 
278  return retval;
279 }
Functions and structures for dealing with VPI arrays.
Declares functions that handle image format conversion.
Defines types and functions to handle image formats.
#define VPI_IMAGE_FORMAT_NV12_ER_BL
YUV420sp 8-bit block-linear format with full range.
Definition: ImageFormat.h:216
#define VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:206
#define VPI_IMAGE_FORMAT_2S16_BL
Single plane with two interleaved 16-bit signed integer channel.
Definition: ImageFormat.h:127
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declares functions that implement the dense optical flow.
Functions and structures for dealing with VPI pyramids.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
uint64_t VPIImageFormat
Pre-defined image formats.
Definition: ImageFormat.h:94
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:230
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiCreateOpticalFlowDense(uint64_t backends, int32_t width, int32_t height, VPIImageFormat inputFmt, VPIOpticalFlowQuality quality, VPIPayload *payload)
Creates payload for vpiSubmitOpticalFlowDense.
VPIStatus vpiSubmitOpticalFlowDense(VPIStream stream, uint64_t backend, VPIPayload payload, VPIImage prevImg, VPIImage curImg, VPIImage mvImg)
Runs dense Optical Flow on two frames.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:268
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_NVENC
NVENC backend.
Definition: Types.h:96
@ VPI_BACKEND_VIC
VIC backend.
Definition: Types.h:95
VPIOpticalFlowQuality
Defines the quality of the optical flow algorithm.
Definition: Types.h:416
@ VPI_OPTICAL_FLOW_QUALITY_LOW
Fast but low quality optical flow implementation.
Definition: Types.h:418
@ VPI_OPTICAL_FLOW_QUALITY_HIGH
Slow but high quality optical flow implementation.
Definition: Types.h:424
@ VPI_OPTICAL_FLOW_QUALITY_MEDIUM
Speed and quality in between of VPI_OPTICAL_FLOW_QUALITY_LOW and VPI_OPTICAL_FLOW_QUALITY_HIGH.
Definition: Types.h:421
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:435