VPI - Vision Programming Interface

3.0 Release

Temporal Noise Reduction

Overview

The Temporal Noise Reduction sample application takes a noisy input video, reduces its noise and writes the result to disk. The user can define what backend will be used for processing.

Instructions

The command line parameters are:

<backend> <input video>

where

  • backend: either cuda or vic (on Jetson devices only). It defines the backend that will perform the processing.
  • input video: video file to have noise reduced; it accepts .mp4, .avi and possibly others, depending on OpenCV's support.

VPI samples installer includes a sample video with noise added artificially, found in /opt/nvidia/vpi3/samples/assets/noisy.mp4.

Here's one invocation example:

  • C++
    ./vpi_sample_09_tnr cuda ../assets/noisy.mp4
  • Python
    python3 main.py cuda ../assets/noisy.mp4

The application will process noisy.mp4 and create denoised_cuda.mp4 with the denoised version of the input.

Results

Input videoDe-noised video

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language:
27 import sys
28 import vpi
29 import numpy as np
30 from argparse import ArgumentParser
31 import cv2
32 
33 # ----------------------------
34 # Parse command line arguments
35 
36 parser = ArgumentParser()
37 parser.add_argument('backend', choices=['cuda','vic'],
38  help='Backend to be used for processing')
39 
40 parser.add_argument('input',
41  help='Input video to be denoised')
42 
43 args = parser.parse_args();
44 
45 if args.backend == 'cuda':
46  backend = vpi.Backend.CUDA
47 else:
48  assert args.backend == 'vic'
49  backend = vpi.Backend.VIC
50 
51 # -----------------------------
52 # Open input and output videos
53 
54 inVideo = cv2.VideoCapture(args.input)
55 
56 fourcc = cv2.VideoWriter_fourcc(*'MPEG')
57 inSize = (int(inVideo.get(cv2.CAP_PROP_FRAME_WIDTH)), int(inVideo.get(cv2.CAP_PROP_FRAME_HEIGHT)))
58 fps = inVideo.get(cv2.CAP_PROP_FPS)
59 
60 outVideo = cv2.VideoWriter('denoised_python'+str(sys.version_info[0])+'_'+args.backend+'.mp4',
61  fourcc, fps, inSize)
62 
63 #--------------------------------------------------------------
64 # Create the TNR object using the backend specified by the user
65 with backend:
66  tnr = vpi.TemporalNoiseReduction(inSize, vpi.Format.NV12_ER)
67 
68 #--------------------------------------------------------------
69 # Main processing loop
70 curFrame = 0
71 while True:
72  curFrame+=1
73  print("Frame: {}".format(curFrame))
74 
75  # Read one input frame
76  ret, cvFrame = inVideo.read()
77  if not ret:
78  break
79 
80  # Convert it to NV12_ER format to be used by VPI
81  with vpi.Backend.CUDA:
82  frame = vpi.asimage(cvFrame).convert(vpi.Format.NV12_ER)
83 
84  # Retrieve the corresponding denoised frame
85  denoised = tnr(frame, preset=vpi.TNRPreset.INDOOR_MEDIUM_LIGHT, strength=1)
86 
87  # Convert it to RGB8 for output using the CUDA backend
88  with vpi.Backend.CUDA:
89  denoised = denoised.convert(vpi.Format.RGB8)
90 
91  # Write the denoised frame to the output video
92  with denoised.rlock_cpu() as data:
93  outVideo.write(data)
29 #include <opencv2/core/version.hpp>
30 #include <opencv2/imgcodecs.hpp>
31 #include <opencv2/imgproc/imgproc.hpp>
32 #include <opencv2/videoio.hpp>
33 #include <vpi/OpenCVInterop.hpp>
34 
35 #include <vpi/Image.h>
36 #include <vpi/Status.h>
37 #include <vpi/Stream.h>
40 
41 #include <algorithm>
42 #include <cstring> // for memset
43 #include <fstream>
44 #include <iostream>
45 #include <map>
46 #include <sstream>
47 #include <vector>
48 
49 #define CHECK_STATUS(STMT) \
50  do \
51  { \
52  VPIStatus status = (STMT); \
53  if (status != VPI_SUCCESS) \
54  { \
55  char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH]; \
56  vpiGetLastStatusMessage(buffer, sizeof(buffer)); \
57  std::ostringstream ss; \
58  ss << vpiStatusGetName(status) << ": " << buffer; \
59  throw std::runtime_error(ss.str()); \
60  } \
61  } while (0);
62 
63 int main(int argc, char *argv[])
64 {
65  // OpenCV image that will be wrapped by a VPIImage.
66  // Define it here so that it's destroyed *after* wrapper is destroyed
67  cv::Mat cvFrame;
68 
69  // Declare all VPI objects we'll need here so that we
70  // can destroy them at the end.
71  VPIStream stream = NULL;
72  VPIImage imgPrevious = NULL, imgCurrent = NULL, imgOutput = NULL;
73  VPIImage frameBGR = NULL;
74  VPIPayload tnr = NULL;
75 
76  // main return value
77  int retval = 0;
78 
79  try
80  {
81  // =============================
82  // Parse command line parameters
83 
84  if (argc != 3)
85  {
86  throw std::runtime_error(std::string("Usage: ") + argv[0] + " <vic|cuda> <input_video>");
87  }
88 
89  std::string strBackend = argv[1];
90  std::string strInputVideo = argv[2];
91 
92  // Now parse the backend
93  VPIBackend backend;
94 
95  if (strBackend == "cuda")
96  {
97  backend = VPI_BACKEND_CUDA;
98  }
99  else if (strBackend == "vic")
100  {
101  backend = VPI_BACKEND_VIC;
102  }
103  else
104  {
105  throw std::runtime_error("Backend '" + strBackend + "' not recognized, it must be either cuda or vic.");
106  }
107 
108  // ===============================
109  // Prepare input and output videos
110 
111  // Load the input video
112  cv::VideoCapture invid;
113  if (!invid.open(strInputVideo))
114  {
115  throw std::runtime_error("Can't open '" + strInputVideo + "'");
116  }
117 
118  // Open the output video for writing using input's characteristics
119  int w = invid.get(cv::CAP_PROP_FRAME_WIDTH);
120  int h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);
121  int fourcc = cv::VideoWriter::fourcc('M', 'P', 'E', 'G');
122  double fps = invid.get(cv::CAP_PROP_FPS);
123 
124  // Create the output video
125  cv::VideoWriter outVideo("denoised_" + strBackend + ".mp4", fourcc, fps, cv::Size(w, h));
126  if (!outVideo.isOpened())
127  {
128  throw std::runtime_error("Can't create output video");
129  }
130 
131  // =================================
132  // Allocate all VPI resources needed
133 
134  // We'll use the backend passed to run remap algorithm and the CUDA to do image format
135  // conversions, therefore we have to force enabling of CUDA backend, along with the
136  // desired backend.
137  CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA | backend, &stream));
138 
139  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgPrevious));
140  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgCurrent));
141  CHECK_STATUS(vpiImageCreate(w, h, VPI_IMAGE_FORMAT_NV12_ER, 0, &imgOutput));
142 
143  // Create a Temporal Noise Reduction payload configured to process NV12_ER
144  // frames under indoor medium light
145  CHECK_STATUS(vpiCreateTemporalNoiseReduction(backend, w, h, VPI_IMAGE_FORMAT_NV12_ER, VPI_TNR_DEFAULT, &tnr));
146 
147  // ====================
148  // Main processing loop
149 
150  int curFrame = 0;
151  while (invid.read(cvFrame))
152  {
153  printf("Frame: %d\n", ++curFrame);
154 
155  // frameBGR isn't allocated yet?
156  if (frameBGR == NULL)
157  {
158  // Create a VPIImage that wraps the frame
159  CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvFrame, 0, &frameBGR));
160  }
161  else
162  {
163  // reuse existing VPIImage wrapper to wrap the new frame.
164  CHECK_STATUS(vpiImageSetWrappedOpenCVMat(frameBGR, cvFrame));
165  }
166 
167  // First convert it to NV12_ER
168  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, frameBGR, imgCurrent, NULL));
169 
170  // Apply temporal noise reduction
171  // For first frame, we have to pass NULL as previous frame,
172  // this will reset internal state.
173  VPITNRParams params;
174  CHECK_STATUS(vpiInitTemporalNoiseReductionParams(&params));
175 
177  params.strength = 1.0f;
178 
179  CHECK_STATUS(vpiSubmitTemporalNoiseReduction(stream, 0, tnr, curFrame == 1 ? NULL : imgPrevious, imgCurrent,
180  imgOutput, &params));
181 
182  // Convert output back to BGR
183  CHECK_STATUS(vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, imgOutput, frameBGR, NULL));
184  CHECK_STATUS(vpiStreamSync(stream));
185 
186  // Now add it to the output video stream
187  VPIImageData imgdata;
188  CHECK_STATUS(vpiImageLockData(frameBGR, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &imgdata));
189 
190  cv::Mat outFrame;
191  CHECK_STATUS(vpiImageDataExportOpenCVMat(imgdata, &outFrame));
192  outVideo << outFrame;
193 
194  CHECK_STATUS(vpiImageUnlock(frameBGR));
195 
196  // this iteration's output will be next's previous. Previous, which would be discarded, will be reused
197  // to store next frame.
198  std::swap(imgPrevious, imgOutput);
199  };
200  }
201  catch (std::exception &e)
202  {
203  std::cerr << e.what() << std::endl;
204  retval = 1;
205  }
206 
207  // =========================
208  // Destroy all VPI resources
209  vpiStreamDestroy(stream);
210  vpiPayloadDestroy(tnr);
211  vpiImageDestroy(imgPrevious);
212  vpiImageDestroy(imgCurrent);
213  vpiImageDestroy(imgOutput);
214  vpiImageDestroy(frameBGR);
215 
216  return retval;
217 }
Declares functions that handle image format conversion.
#define VPI_IMAGE_FORMAT_NV12_ER
YUV420sp 8-bit pitch-linear format with full range.
Definition: ImageFormat.h:206
Functions and structures for dealing with VPI images.
Functions for handling OpenCV interoperability with VPI.
Declaration of VPI status codes handling functions.
Declares functions dealing with VPI streams.
Declares functions that implement the Temporal Noise Reduction algorithm.
VPIStatus vpiSubmitConvertImageFormat(VPIStream stream, uint64_t backend, VPIImage input, VPIImage output, const VPIConvertImageFormatParams *params)
Converts the image contents to the desired format, with optional scaling and offset.
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:256
VPIStatus vpiImageLockData(VPIImage img, VPILockMode mode, VPIImageBufferType bufType, VPIImageData *data)
Acquires the lock on an image object and returns the image contents.
VPIStatus vpiImageCreate(int32_t width, int32_t height, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Create an empty image instance with the specified flags.
VPIStatus vpiImageUnlock(VPIImage img)
Releases the lock on an image object.
@ VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
Host-accessible with planes in pitch-linear memory layout.
Definition: Image.h:172
Stores information about image characteristics and content.
Definition: Image.h:234
VPIStatus vpiImageCreateWrapperOpenCVMat(const cv::Mat &mat, VPIImageFormat fmt, uint64_t flags, VPIImage *img)
Wraps a cv::Mat in an VPIImage with the given image format.
VPIStatus vpiImageDataExportOpenCVMat(const VPIImageData &imgData, cv::Mat *mat)
Fills an existing cv::Mat with data from VPIImageData coming from a locked VPIImage.
VPIStatus vpiImageSetWrappedOpenCVMat(VPIImage img, const cv::Mat &mat)
Redefines the wrapped cv::Mat of an existing VPIImage wrapper.
struct VPIPayloadImpl * VPIPayload
A handle to an algorithm payload.
Definition: Types.h:268
void vpiPayloadDestroy(VPIPayload payload)
Deallocates the payload object and all associated resources.
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:250
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPIBackend
VPI Backend types.
Definition: Types.h:91
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
VPIStatus vpiStreamCreate(uint64_t flags, VPIStream *stream)
Create a stream instance.
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:93
@ VPI_BACKEND_VIC
VIC backend.
Definition: Types.h:95
float strength
Noise reduction strength.
VPITNRPreset preset
Scene preset to be used.
VPIStatus vpiSubmitTemporalNoiseReduction(VPIStream stream, uint64_t backend, VPIPayload payload, VPIImage prevFrame, VPIImage curFrame, VPIImage outFrame, const VPITNRParams *params)
Submits a Temporal Noise Reduction operation to the stream.
VPIStatus vpiInitTemporalNoiseReductionParams(VPITNRParams *params)
Initializes vpiSubmitTemporalNoiseReduction with default values.
VPIStatus vpiCreateTemporalNoiseReduction(uint64_t backends, int32_t width, int32_t height, VPIImageFormat imgFormat, VPITNRVersion version, VPIPayload *payload)
Creates a payload for Temporal Noise Reduction algorithm.
@ VPI_TNR_PRESET_INDOOR_MEDIUM_LIGHT
Medium light indoor scene.
@ VPI_TNR_DEFAULT
Chooses the version with best quality available in the current device and given backend.
Structure that defines the parameters for vpiSubmitTemporalNoiseReduction.
@ VPI_LOCK_READ
Lock memory only for reading.
Definition: Types.h:595