VPI - Vision Programming Interface

1.0 Release

Temporal Noise Reduction

Overview

The Temporal Noise Reduction sample application takes a noisy input video, reduces its noise and writes the result to disk. The user can define what backend will be used for processing.

This sample shows the following:

  • Creating and destroying a VPI stream.
  • Wrapping a OpenCV cv::Mat image to be used with VPI.
  • Wrapping a VPI-managed image into an OpenCV's cv::Mat.
  • Use OpenCV to fetch frames from a video file.
  • Use OpenCV save frames into a video file.
  • Create pipeline that does Convert Image Format to convert from/to NV12 format and run Temporal Noise Reduction.
  • Shows how to properly use Temporal Noise Reduction to process a video sequence.

Instructions

The usage is:

./vpi_sample_09_tnr <backend> <input video> <output video>

where

  • backend: either cuda or pva, as cpu isn't available yet. It defines the backend that will perform the processing.
  • input video: video file to have noise reduced; it accepts .mp4, .avi and possibly others, depending on OpenCV's support.
  • output video: file to write the de-noised result. It'll use the same codec and fps as input video.

VPI samples installer includes a sample video with noise added artificially, found in /opt/nvidia/vpi1/samples/assets/noisy.mp4.

Here's one invocation example:

./vpi_sample_09_tnr cuda ../assets/noisy.mp4 denoised.mp4

Results

Input videoDe-noised video

Source Code

For convenience, here's the code that is also installed in the samples directory.

1 /*
2 * Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without
5 * modification, are permitted provided that the following conditions
6 * are met:
7 * * Redistributions of source code must retain the above copyright
8 * notice, this list of conditions and the following disclaimer.
9 * * Redistributions in binary form must reproduce the above copyright
10 * notice, this list of conditions and the following disclaimer in the
11 * documentation and/or other materials provided with the distribution.
12 * * Neither the name of NVIDIA CORPORATION nor the names of its
13 * contributors may be used to endorse or promote products derived
14 * from this software without specific prior written permission.
15 *
16 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 */
28 
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 # include <opencv2/videoio.hpp>
33 #else
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/imgproc/imgproc.hpp>
38 #include <vpi/OpenCVInterop.hpp>
39 
40 #include <vpi/Context.h>
41 #include <vpi/Image.h>
42 #include <vpi/Status.h>
43 #include <vpi/Stream.h>
46 
47 #include <algorithm>
48 #include <cstring> // for memset
49 #include <fstream>
50 #include <iostream>
51 #include <map>
52 #include <sstream>
53 #include <vector>
54 
55 #define CHECK_STATUS(STMT) \
56  do \
57  { \
58  VPIStatus status = (STMT); \
59  if (status != VPI_SUCCESS) \
60  { \
61  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
62  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
63  std::ostringstream ss; \
64  ss << vpiStatusGetName(status) << ": " << buffer; \
65  throw std::runtime_error(ss.str()); \
66  } \
67  } while (0);
68 
69 // Utility function to wrap a cv::Mat into a VPIImage
70 static VPIImage ToVPIImage(VPIImage image, const cv::Mat &frame)
71 {
72  if (image == nullptr)
73  {
74  // Ceate a VPIImage that wraps the frame
75  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(frame, 0, &image));
76  }
77  else
78  {
79  // reuse existing VPIImage wrapper to wrap the new frame.
80  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(image, frame));
81  }
82  return image;
83 }
84 
85 int main(int argc, char *argv[])
86 {
87  // We'll create all our objects under this context, so that
88  // we don't have to track what objects to destroy. Just destroying
89  // the context will destroy all objects.
90  VPIContext ctx = nullptr;
91 
92  int retval = 0;
93 
94  try
95  {
96  if (argc != 4)
97  {
98  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <cpu|vic|cuda> <input_video> <output>");
99  }
100 
101  std::string strBackend = argv[1];
102  std::string strInputVideo = argv[2];
103  std::string strOutputVideo = argv[3];
104 
105  // Load the input video
106  cv::VideoCapture invid;
107  if (!invid.open(strInputVideo))
108  {
109  throw std::runtime_error("Can't open '" + strInputVideo + "'");
110  }
111 
112  // Create our context.
113  CHECK_STATUS(vpiContextCreate(0, &ctx));
114 
115  // Activate it. From now on all created objects will be owned by it.
116  CHECK_STATUS(vpiContextSetCurrent(ctx));
117 
118  // Now parse the backend
119  VPIBackend backend;
120 
121  if (strBackend == "cpu")
122  {
123  backend = VPI_BACKEND_CPU;
124  }
125  else if (strBackend == "cuda")
126  {
127  backend = VPI_BACKEND_CUDA;
128  }
129  else if (strBackend == "vic")
130  {
131  backend = VPI_BACKEND_VIC;
132  }
133  else
134  {
135  throw std::runtime_error("Backend '" + strBackend +
136  "' not recognized, it must be either cpu, cuda or vic.");
137  }
138 
139  // Create the stream for the given backend.
140  VPIStream streamTNR;
141  CHECK_STATUS(vpiStreamCreate(backend, &streamTNR));
142 
143  VPIStream stream;
144  // PVA backend doesn't have currently Convert Image Format algorithm. We'll use CUDA
145  // backend to do that.
146  CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA | backend, &stream));
147 
148 #if CV_MAJOR_VERSION >= 3
149  int w = invid.get(cv::CAP_PROP_FRAME_WIDTH);
150  int h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
151  int fourcc = invid.get(cv::CAP_PROP_FOURCC);
152  double fps = invid.get(cv::CAP_PROP_FPS);
153 #else
154  int w = invid.get(CV_CAP_PROP_FRAME_WIDTH);
155  int h = invid.get(CV_CAP_PROP_FRAME_HEIGHT);
156  int fourcc = invid.get(CV_FOURCC('M', 'J', 'P', 'G'));
157  double fps = invid.get(CV_CAP_PROP_FPS);
158 #endif
159 
160  cv::VideoWriter outVideo(strOutputVideo, fourcc, fps, cv::Size(w, h));
161 
162  if (!outVideo.isOpened())
163  {
164  throw std::runtime_error("Can't create output video");
165  }
166 
167  VPIImage imgPrevious, imgCurrent, imgOutput;
168  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgPrevious));
169  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgCurrent));
170  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgOutput));
171 
172  // Create a Temporal Noise Reduction payload configured to process NV12
173  // frames under outdoor low light
174  VPIPayload tnr;
177 
178  VPIImage frameBGR = nullptr;
179 
180  int curFrame = 0;
181  cv::Mat cvFrame;
182  while (invid.read(cvFrame))
183  {
184  printf("Frame: %d\n", ++curFrame);
185 
186  frameBGR = ToVPIImage(frameBGR, cvFrame);
187 
188  // First convert it to NV12
189  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, frameBGR, imgCurrent, NULL));
190 
191  // Apply temporal noise reduction
192  // For first frame, we have to pass nullptr as previous frame, this will reset internal
193  // state.
194  CHECK_STATUS(vpiSubmitTemporalNoiseReduction(stream, 0, tnr, curFrame == 1 ? nullptr : imgPrevious,
195  imgCurrent, imgOutput));
196 
197  // Convert output back to BGR
198  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgOutput, frameBGR, NULL));
199  CHECK_STATUS(vpiStreamSync(stream));
200 
201  // Now add it to the output video stream
202  VPIImageData imgdata;
203  CHECK_STATUS(vpiImageLock(frameBGR, VPI_LOCK_READ, &imgdata));
204 
205  cv::Mat outFrame;
206  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
207  outVideo << outFrame;
208 
209  CHECK_STATUS(vpiImageUnlock(frameBGR));
210 
211  // this iteration's output will be next's previous. Previous, which would be discarded, will be reused
212  // to store next frame.
213  std::swap(imgPrevious, imgOutput);
214  };
215  }
216  catch (std::exception &e)
217  {
218  std::cerr << e.what() << std::endl;
219  retval = 1;
220  }
221 
222  // Clean up
223  vpiContextDestroy(ctx);
224 
225  return retval;
226 }
Functions and structures for dealing with VPI contexts.
Declares functions that handle image format conversion.
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
Declares functions that implement the Temporal Noise Reduction algorithm.
VPIStatus vpiContextSetCurrent(VPIContext ctx)
Sets the context for the calling thread.
void vpiContextDestroy(VPIContext ctx)
Destroy a context instance as well as all resources it owns.
VPIStatus vpiContextCreate(uint32_t flags, VPIContext *ctx)
Create a context instance.
struct VPIContextImpl * VPIContext
A handle to a context.
Definition: Types.h:179
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint32_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
@ VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:152
VPIStatus vpiImageLock(VPIImage img, VPILockMode mode, VPIImageData *hostData)
Acquires the lock on an image object and returns a pointer to the image planes.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:197
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
Stores information about image characteristics and content.
Definition: Image.h:159
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiImageCreateOpenCVMatWrapper(const cv::Mat &mat, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:209
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:191
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:90
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:92
@ VPI_BACKEND_VIC
VIC backend.
Definition: Types.h:94
@ VPI_BACKEND_CPU
CPU backend.
Definition: Types.h:91
VPIStatus vpiSubmitTemporalNoiseReduction(VPIStream stream, uint32_t backend, VPIPayload payload, VPIImage prevFrame, VPIImage curFrame, VPIImage outFrame)
Submits a Temporal Noise Reduction operation to the stream associated with the given payload.
VPIStatus vpiCreateTemporalNoiseReduction(uint32_t backends, int32_t width, int32_t height, VPIImageFormat imgFormat, VPITNRVersion version, VPITNRPreset preset, float strength, VPIPayload *payload)
Creates a payload for Temporal Noise Reduction algorithm.
@ VPI_TNR_PRESET_INDOOR_LOW_LIGHT
Low light indoor scene.
@ VPI_TNR_DEFAULT
Chooses the version with best quality available in the current device and given backend.
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:348