VPI - Vision Programming Interface

2.0 Release - Developer Preview

Background Subtractor

Overview

This application fetches frames from input video source, runs the algorithms on current images, and then calculate foreground mask. The output foreground mask will be saved to a video file.

Instructions

The command line parameters are:

<backend> <input video>

where

  • backend: Defines the backend that will perform the processing. Only cpu and cuda are currently supported.
  • input video: Input video file name, it accepts .mp4, .avi and possibly others, depending on OpenCV's support.

Here's one example:

  • C++
    ./vpi_sample_14_background_subtractor cpu ../assets/pedestrians.mp4
  • Python
    python3 main.py cpu ../assets/pedestrians.mp4

The application will process pedestrians.mp4 and create fgmask_cpu.mp4 and bgimage_cpu.mp4.

Results

Input videoForeground videoBackground video

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import cv2
28 import sys
29 import vpi
30 import numpy as np
31 from argparse import ArgumentParser
32 
33 # ----------------------------
34 # Parse command line arguments
35 
36 parser = ArgumentParser()
37 parser.add_argument('backend', choices=['cpu','cuda'],
38  help='Backend to be used for processing')
39 
40 parser.add_argument('input',
41  help='Input video to be denoised')
42 
43 args = parser.parse_args();
44 
45 if args.backend == 'cuda':
46  backend = vpi.Backend.CUDA
47 else:
48  assert args.backend == 'cpu'
49  backend = vpi.Backend.CPU
50 
51 # -----------------------------
52 # Open input and output videos
53 
54 inVideo = cv2.VideoCapture(args.input)
55 
56 fourcc = cv2.VideoWriter_fourcc(*'MPEG')
57 inSize = (int(inVideo.get(cv2.CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.CAP_PROP_FRAME_HEIGHT)))
58 fps = inVideo.get(cv2.CAP_PROP_FPS)
59 
60 outVideoFGMask = cv2.VideoWriter('fgmask_python'+str(sys.version_info[0])+'_'+args.backend+'.mp4',
61  fourcc, fps, inSize)
62 
63 outVideoBGImage = cv2.VideoWriter('bgimage_python'+str(sys.version_info[0])+'_'+args.backend+'.mp4',
64  fourcc, fps, inSize)
65 
66 #--------------------------------------------------------------
67 # Create the Background Subtractor object using the backend specified by the user
68 with backend:
69  bgsub = vpi.BackgroundSubtractor(inSize, vpi.Format.BGR8)
70 
71 #--------------------------------------------------------------
72 # Main processing loop
73 idxFrame = 0
74 while True:
75  print("Processing frame {}".format(idxFrame))
76  idxFrame+=1
77 
78  # Read one input frame
79  ret, cvFrame = inVideo.read()
80  if not ret:
81  break
82 
83  # Get the foreground mask and background image estimates
84  fgmask, bgimage = bgsub(vpi.asimage(cvFrame, vpi.Format.BGR8), learnrate=0.01)
85 
86  # Mask needs to be converted to BGR8 for output
87  fgmask = fgmask.convert(vpi.Format.BGR8, backend=vpi.Backend.CUDA);
88 
89  # Write images to output videos
90  with fgmask.rlock_cpu(), bgimage.rlock_cpu():
91  outVideoFGMask.write(fgmask.cpu())
92  outVideoBGImage.write(bgimage.cpu())
29 #include <opencv2/core/version.hpp>
30 #include <opencv2/imgcodecs.hpp>
31 #include <opencv2/imgproc/imgproc.hpp>
32 #include <opencv2/videoio.hpp>
33 #include <vpi/OpenCVInterop.hpp>
34 
35 #include <vpi/Array.h>
36 #include <vpi/Image.h>
37 #include <vpi/ImageFormat.h>
38 #include <vpi/Pyramid.h>
39 #include <vpi/Status.h>
40 #include <vpi/Stream.h>
43 
44 #include <iostream>
45 #include <sstream>
46 
47 #define CHECK_STATUS(STMT) \
48  do \
49  { \
50  VPIStatus status = (STMT); \
51  if (status != VPI_SUCCESS) \
52  { \
53  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
54  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
55  std::ostringstream ss; \
56  ss << vpiStatusGetName(status) << ": " << buffer; \
57  throw std::runtime_error(ss.str()); \
58  } \
59  } while (0);
60 
61 int main(int argc, char *argv[])
62 {
63  // OpenCV image that will be wrapped by a VPIImage.
64  // Define it here so that it's destroyed *after* wrapper is destroyed
65  cv::Mat cvCurFrame;
66 
67  // VPI objects that will be used
68  VPIStream stream = NULL;
69  VPIImage imgCurFrame = NULL;
70  VPIImage bgimage = NULL;
71  VPIImage fgmask = NULL;
72  VPIPayload payload = NULL;
73 
74  int retval = 0;
75 
76  try
77  {
78  if (argc != 3)
79  {
80  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|cuda> <input_video>");
81  }
82 
83  // Parse input parameters
84  std::string strBackend = argv[1];
85  std::string strInputVideo = argv[2];
86 
87  VPIBackend backend;
88  if (strBackend == "cpu")
89  {
90  backend = VPI_BACKEND_CPU;
91  }
92  else if (strBackend == "cuda")
93  {
94  backend = VPI_BACKEND_CUDA;
95  }
96  else
97  {
98  throw std::runtime_error("Backend '" + strBackend + "' not recognized.");
99  }
100 
101  // Load the input video
102  cv::VideoCapture invid;
103  if (!invid.open(strInputVideo))
104  {
105  throw std::runtime_error("Can't open '" + strInputVideo + "'");
106  }
107 
108  int32_t width = invid.get(cv::CAP_PROP_FRAME_WIDTH);
109  int32_t height = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
110 
111  // Create the stream where processing will happen. We'll use user-provided backend.
112  CHECK_STATUS(vpiStreamCreate(backend, &stream));
113 
114  // Create background subtractor payload to be executed on the given backend
115  // OpenCV delivers us BGR8 images, so the algorithm is configured to accept that.
116  CHECK_STATUS(vpiCreateBackgroundSubtractor(backend, width, height, VPI_IMAGE_FORMAT_BGR8, &payload));
117 
118  // Create foreground image
119  CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_U8, 0, &fgmask));
120 
121  // Create background image
122  CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_BGR8, 0, &bgimage));
123 
124  int fourcc = cv::VideoWriter::fourcc('M', 'P', 'E', 'G');
125  double fps = invid.get(cv::CAP_PROP_FPS);
126 
127  cv::VideoWriter outVideo("fgmask_" + strBackend + ".mp4", fourcc, fps, cv::Size(width, height), false);
128  if (!outVideo.isOpened())
129  {
130  throw std::runtime_error("Can't create output video");
131  }
132 
133  cv::VideoWriter bgimageVideo("bgimage_" + strBackend + ".mp4", fourcc, fps, cv::Size(width, height));
134  if (!outVideo.isOpened())
135  {
136  throw std::runtime_error("Can't create output video");
137  }
138 
139  // Fetch a new frame until video ends
140  int idxFrame = 1;
141 
142  while (invid.read(cvCurFrame))
143  {
144  printf("Processing frame %d\n", idxFrame++);
145  // Wrap frame into a VPIImage
146  if (imgCurFrame == NULL)
147  {
148  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvCurFrame, 0, &imgCurFrame));
149  }
150  else
151  {
152  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(imgCurFrame, cvCurFrame));
153  }
154 
156  CHECK_STATUS(vpiInitBackgroundSubtractorParams(&params));
157  params.learningRate = 0.01;
158 
159  CHECK_STATUS(
160  vpiSubmitBackgroundSubtractor(stream, backend, payload, imgCurFrame, fgmask, bgimage, &params));
161 
162  // Wait for processing to finish.
163  CHECK_STATUS(vpiStreamSync(stream));
164 
165  {
166  // Now add it to the output video stream
167  VPIImageData imgdata;
168  CHECK_STATUS(vpiImageLockData(fgmask, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &imgdata));
169 
170  cv::Mat outFrame;
171  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
172 
173  outVideo << outFrame;
174 
175  CHECK_STATUS(vpiImageUnlock(fgmask));
176  }
177 
178  {
179  VPIImageData bgdata;
180  CHECK_STATUS(vpiImageLockData(bgimage, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &bgdata));
181 
182  cv::Mat outFrame;
183  CHECK_STATUS(vpiImageDataExportOpenCVMat(bgdata, &outFrame));
184 
185  bgimageVideo << outFrame;
186 
187  CHECK_STATUS(vpiImageUnlock(bgimage));
188  }
189  }
190  }
191  catch (std::exception &e)
192  {
193  std::cerr << e.what() << std::endl;
194  retval = 1;
195  }
196 
197  // Destroy all resources used
198  vpiStreamDestroy(stream);
199  vpiPayloadDestroy(payload);
200 
201  vpiImageDestroy(imgCurFrame);
202  vpiImageDestroy(fgmask);
203  vpiImageDestroy(bgimage);
204 
205  return retval;
206 }
Functions and structures for dealing with VPI arrays.
Declares functions that implement background subtractor algorithms.
Declares functions that handle image format conversion.
Defines types and functions to handle image formats.
#define VPI_IMAGE_FORMAT_BGR8
Single plane with interleaved BGR 8-bit channel.
Definition: ImageFormat.h:290
#define VPI_IMAGE_FORMAT_U8
Single plane with one 8-bit unsigned integer channel.
Definition: ImageFormat.h:100
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Functions and structures for dealing with VPI pyramids.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
float learningRate
Learning rate that indicates how fast the background model is learnt.
VPIStatus vpiSubmitBackgroundSubtractor(VPIStream stream, uint64_t backend, VPIPayload payload, VPIImage inFrame, VPIImage outFGMask, VPIImage outBGImage, const VPIBackgroundSubtractorParams *params)
Submits a Background Subtractor operation to the stream.
VPIStatus vpiInitBackgroundSubtractorParams(VPIBackgroundSubtractorParams *params)
Initializes VPIBackgroundSubtractorParams with default values.
VPIStatus vpiCreateBackgroundSubtractor(uint64_t backends, int32_t imageWidth, int32_t imageHeight, VPIImageFormat inputFormat, VPIPayload *payload)
Creates payload for vpiSubmitBackgroundSubtractor.
Structure that defines the parameters for vpiCreateBackgroundSubtractor.
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:230
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:268
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:92
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:435