VPI - Vision Programming Interface

1.2 Release

Temporal Noise Reduction

Overview

The Temporal Noise Reduction sample application takes a noisy input video, reduces its noise and writes the result to disk. The user can define what backend will be used for processing.

Instructions

The command line parameters are:

<backend> <input video>

where

  • backend: either cuda or vic (on Jetson devices only), as cpu isn't available yet. It defines the backend that will perform the processing.
  • input video: video file to have noise reduced; it accepts .mp4, .avi and possibly others, depending on OpenCV's support.

VPI samples installer includes a sample video with noise added artificially, found in /opt/nvidia/vpi1/samples/assets/noisy.mp4.

Here's one invocation example:

  • C++
    ./vpi_sample_09_tnr cuda ../assets/noisy.mp4
  • Python
    python main.py cuda ../assets/noisy.mp4

The application will process noisy.mp4 and create denoised_cuda.mp4 with the denoised version of the input.

Note
If using OpenCV-2.4 or older (i.e. on Ubuntu 16.04), output file is denoised_cuda.avi.

Results

Input videoDe-noised video

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import cv2
28 import sys
29 import vpi
30 import numpy as np
31 from argparse import ArgumentParser
32 
33 # ----------------------------
34 # Parse command line arguments
35 
36 parser = ArgumentParser()
37 parser.add_argument('backend', choices=['cuda','vic'],
38  help='Backend to be used for processing')
39 
40 parser.add_argument('input',
41  help='Input video to be denoised')
42 
43 args = parser.parse_args();
44 
45 if args.backend == 'cuda':
46  backend = vpi.Backend.CUDA
47 else:
48  assert args.backend == 'vic'
49  backend = vpi.Backend.VIC
50 
51 # -----------------------------
52 # Open input and output videos
53 
54 inVideo = cv2.VideoCapture(args.input)
55 
56 if int(cv2.__version__.split('.')[0]) >= 3:
57  extOutputVideo = '.mp4'
58  fourcc = cv2.VideoWriter_fourcc(*'avc1')
59  inSize = (int(inVideo.get(cv2.CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.CAP_PROP_FRAME_HEIGHT)))
60  fps = inVideo.get(cv2.CAP_PROP_FPS)
61 else:
62  # MP4 support with OpenCV-2.4 has issues, we'll use
63  # avi/mpeg instead.
64  extOutputVideo = '.avi'
65  fourcc = cv2.cv.CV_FOURCC('M','P','E','G')
66  inSize = (int(inVideo.get(cv2.cv.CV_CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT)))
67  fps = inVideo.get(cv2.cv.CV_CAP_PROP_FPS)
68 
69 outVideo = cv2.VideoWriter('denoised_python'+str(sys.version_info[0])+'_'+args.backend+extOutputVideo,
70  fourcc, fps, inSize)
71 
72 #--------------------------------------------------------------
73 # Create the TNR object using the backend specified by the user
74 with backend:
75  tnr = vpi.TemporalNoiseReduction(inSize, vpi.Format.NV12_ER)
76 
77 #--------------------------------------------------------------
78 # Main processing loop
79 curFrame = 0
80 while True:
81  curFrame+=1
82  print("Frame: {}".format(curFrame))
83 
84  # Read one input frame
85  ret, cvFrame = inVideo.read()
86  if not ret:
87  break
88 
89  # Convert it to NV12_ER format to be used by VPI
90  with vpi.Backend.CUDA:
91  frame = vpi.asimage(cvFrame).convert(vpi.Format.NV12_ER)
92 
93  # Retrieve the corresponding denoised frame
94  denoised = tnr(frame, preset=vpi.TNRPreset.INDOOR_MEDIUM_LIGHT, strength=1)
95 
96  # Convert it to RGB8 for output using the CUDA backend
97  with vpi.Backend.CUDA:
98  denoised = denoised.convert(vpi.Format.RGB8)
99 
100  # Write the denoised frame to the output video
101  with denoised.rlock():
102  outVideo.write(denoised.cpu())
103 
104 # vim: ts=8:sw=4:sts=4:et:ai
29 #include <opencv2/core/version.hpp>
30 #if CV_MAJOR_VERSION >= 3
31 # include <opencv2/imgcodecs.hpp>
32 # include <opencv2/videoio.hpp>
33 #else
34 # include <opencv2/highgui/highgui.hpp>
35 #endif
36 
37 #include <opencv2/imgproc/imgproc.hpp>
38 #include <vpi/OpenCVInterop.hpp>
39 
40 #include <vpi/Image.h>
41 #include <vpi/Status.h>
42 #include <vpi/Stream.h>
45 
46 #include <algorithm>
47 #include <cstring> // for memset
48 #include <fstream>
49 #include <iostream>
50 #include <map>
51 #include <sstream>
52 #include <vector>
53 
54 #define CHECK_STATUS(STMT) \
55  do \
56  { \
57  VPIStatus status = (STMT); \
58  if (status != VPI_SUCCESS) \
59  { \
60  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
61  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
62  std::ostringstream ss; \
63  ss << vpiStatusGetName(status) << ": " << buffer; \
64  throw std::runtime_error(ss.str()); \
65  } \
66  } while (0);
67 
68 int main(int argc, char *argv[])
69 {
70  // OpenCV image that will be wrapped by a VPIImage.
71  // Define it here so that it's destroyed *after* wrapper is destroyed
72  cv::Mat cvFrame;
73 
74  // Declare all VPI objects we'll need here so that we
75  // can destroy them at the end.
76  VPIStream stream = NULL;
77  VPIImage imgPrevious = NULL, imgCurrent = NULL, imgOutput = NULL;
78  VPIImage frameBGR = NULL;
79  VPIPayload tnr = NULL;
80 
81  // main return value
82  int retval = 0;
83 
84  try
85  {
86  // =============================
87  // Parse command line parameters
88 
89  if (argc != 3)
90  {
91  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <vic|cuda> <input_video>");
92  }
93 
94  std::string strBackend = argv[1];
95  std::string strInputVideo = argv[2];
96 
97  // Now parse the backend
98  VPIBackend backend;
99 
100  if (strBackend == "cuda")
101  {
102  backend = VPI_BACKEND_CUDA;
103  }
104  else if (strBackend == "vic")
105  {
106  backend = VPI_BACKEND_VIC;
107  }
108  else
109  {
110  throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be either cuda or vic.");
111  }
112 
113  // ===============================
114  // Prepare input and output videos
115 
116  // Load the input video
117  cv::VideoCapture invid;
118  if (!invid.open(strInputVideo))
119  {
120  throw std::runtime_error("Can't open '" + strInputVideo + "'");
121  }
122 
123  // Open the output video for writing using input's characteristics
124 #if CV_MAJOR_VERSION >= 3
125  int w = invid.get(cv::CAP_PROP_FRAME_WIDTH);
126  int h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
127  int fourcc = cv::VideoWriter::fourcc('a', 'v', 'c', '1');
128  double fps = invid.get(cv::CAP_PROP_FPS);
129  std::string extOutputVideo = ".mp4";
130 #else
131  // MP4 support with OpenCV-2.4 has issues, we'll use
132  // avi/mpeg instead.
133  int w = invid.get(CV_CAP_PROP_FRAME_WIDTH);
134  int h = invid.get(CV_CAP_PROP_FRAME_HEIGHT);
135  int fourcc = CV_FOURCC('M', 'P', 'E', 'G');
136  double fps = invid.get(CV_CAP_PROP_FPS);
137  std::string extOutputVideo = ".avi";
138 #endif
139 
140  // Create the output video
141  cv::VideoWriter outVideo("denoised_" + strBackend + extOutputVideo, fourcc, fps, cv::Size(w, h));
142  if (!outVideo.isOpened())
143  {
144  throw std::runtime_error("Can't create output video");
145  }
146 
147  // =================================
148  // Allocate all VPI resources needed
149 
150  // We'll use the backend passed to run remap algorithm and the CUDA to do image format
151  // conversions, therefore we have to force enabling of CUDA backend, along with the
152  // desired backend.
153  CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA | backend, &stream));
154 
155  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgPrevious));
156  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgCurrent));
157  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgOutput));
158 
159  // Create a Temporal Noise Reduction payload configured to process NV12_ER
160  // frames under indoor medium light
161  CHECK_STATUS(vpiCreateTemporalNoiseReduction(backend, w, h, VPI_IMAGE_FORMAT_NV12_ER, VPI_TNR_DEFAULT, &tnr));
162 
163  // ====================
164  // Main processing loop
165 
166  int curFrame = 0;
167  while (invid.read(cvFrame))
168  {
169  printf("Frame: %d\n", ++curFrame);
170 
171  // frameBGR isn't allocated yet?
172  if (frameBGR == NULL)
173  {
174  // Create a VPIImage that wraps the frame
175  CHECK_STATUS(vpiImageCreateOpenCVMatWrapper(cvFrame, 0, &frameBGR));
176  }
177  else
178  {
179  // reuse existing VPIImage wrapper to wrap the new frame.
180  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(frameBGR, cvFrame));
181  }
182 
183  // First convert it to NV12_ER
184  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, frameBGR, imgCurrent, NULL));
185 
186  // Apply temporal noise reduction
187  // For first frame, we have to pass NULL as previous frame,
188  // this will reset internal state.
189  VPITNRParams params;
190  CHECK_STATUS(vpiInitTemporalNoiseReductionParams(&params));
191 
193  params.strength = 1.0f;
194 
195  CHECK_STATUS(vpiSubmitTemporalNoiseReduction(stream, 0, tnr, curFrame == 1 ? NULL : imgPrevious, imgCurrent,
196  imgOutput, &params));
197 
198  // Convert output back to BGR
199  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgOutput, frameBGR, NULL));
200  CHECK_STATUS(vpiStreamSync(stream));
201 
202  // Now add it to the output video stream
203  VPIImageData imgdata;
204  CHECK_STATUS(vpiImageLock(frameBGR, VPI_LOCK_READ, &imgdata));
205 
206  cv::Mat outFrame;
207  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
208  outVideo << outFrame;
209 
210  CHECK_STATUS(vpiImageUnlock(frameBGR));
211 
212  // this iteration's output will be next's previous. Previous, which would be discarded, will be reused
213  // to store next frame.
214  std::swap(imgPrevious, imgOutput);
215  };
216  }
217  catch (std::exception &e)
218  {
219  std::cerr << e.what() << std::endl;
220  retval = 1;
221  }
222 
223  // =========================
224  // Destroy all VPI resources
225  vpiStreamDestroy(stream);
226  vpiPayloadDestroy(tnr);
227  vpiImageDestroy(imgPrevious);
228  vpiImageDestroy(imgCurrent);
229  vpiImageDestroy(imgOutput);
230  vpiImageDestroy(frameBGR);
231 
232  return retval;
233 }
234 
235 // vim: ts=8:sw=4:sts=4:et:ai
Declares functions that handle image format conversion.
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
Declares functions that implement the Temporal Noise Reduction algorithm.
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint32_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
@ VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:194
VPIStatus vpiImageLock(VPIImage img, VPILockMode mode, VPIImageData *hostData)
Acquires the lock on an image object and returns a pointer to the image planes.
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:215
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
Stores information about image characteristics and content.
Definition: Image.h:159
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
VPIStatus vpiImageCreateOpenCVMatWrapper(const cv::Mat &mat, VPIImageFormat fmt, uint32_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:227
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:209
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_VIC
VIC backend.
Definition: Types.h:95
float strength
Noise reduction strength.
VPITNRPreset preset
Scene preset to be used.
VPIStatus vpiInitTemporalNoiseReductionParams(VPITNRParams *params)
Initializes vpiSubmitTemporalNoiseReduction with default values.
VPIStatus vpiCreateTemporalNoiseReduction(uint32_t backends, int32_t width, int32_t height, VPIImageFormat imgFormat, VPITNRVersion version, VPIPayload *payload)
Creates a payload for Temporal Noise Reduction algorithm.
VPIStatus vpiSubmitTemporalNoiseReduction(VPIStream stream, uint32_t backend, VPIPayload payload, VPIImage prevFrame, VPIImage curFrame, VPIImage outFrame, const VPITNRParams *params)
Submits a Temporal Noise Reduction operation to the stream associated with the given payload.
@ VPI_TNR_PRESET_INDOOR_MEDIUM_LIGHT
Medium light indoor scene.
@ VPI_TNR_DEFAULT
Chooses the version with best quality available in the current device and given backend.
Structure that defines the parameters for vpiSubmitTemporalNoiseReduction.
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:383