Background Subtractor

Overview

Background subtraction is an algorithm which is used to separate the foreground objects from the background image in the continuous video sequence. By feeding the frames from the video as input, the binary (or ternary, if including shadow regions) foreground mask and an estimate of the static background image (if enabled) will be produced.

Input	Foreground mask	Background image

Implementation

For each pixel, a group of gaussian models will be created based on the illumination variations of the video sequence. For a pixel from the image, the GMM can be described as:

\[ p(\vec{x}) = \sum_{m=1}^{M}\pi_mN(\vec{x};\vec{\mu}_m, {\vec{\delta}_m}^2) \]

where:
\(\vec{x}\) is the pixel value
\(M\) is the total number of gaussian models
\(\pi_m\) is the weight corresponding to m-th gaussian model
\(\vec{\mu}_m\) is the mean of m-th gaussian model
\(\vec{\delta}_m\) is the standard variance of m-th gaussian model

Mahalanobis distance is used to measure of the distance between a point and a distribution. For more information, see [2]. Here we denote \(D(\vec{x}, m)\) as the Mahalanobis distance between the pixel value x and m-th model.

For any new pixel from the video sequence, it will calculate the Mahalanobis distance \(D(\vec{x}, m)\) against each gaussian model. If the distance is within the threshold, then the model is found. Otherwise it will create new gaussian model if the maximum number of models has not been reached. Based on the weight of the matched model, it will classify the pixel as foreground or background. For more information, see [1].

C API functions

For list of limitations, constraints and backends that implements the algorithm, consult reference documentation of the following functions:

Function	Description
vpiInitBackgroundSubtractorParams	Initializes VPIBackgroundSubtractorParams with default values.
vpiCreateBackgroundSubtractor	Creates payload for vpiSubmitBackgroundSubtractor.
vpiSubmitBackgroundSubtractor	Submits a Background Subtractor operation to the stream.

Usage

Language: C/C++ Python

1. Import VPI module

   import vpi

2. Initialization phase
   1. Create the Background Subtractor object, configuring it to handle RGB8 inputs with dimensions size. The CUDA backend will be used to execute the algorithm.

      with vpi.Backend.CUDA:
          bgsub = vpi.BackgroundSubtractor(size, vpi.Format.RGB8)

3. Processing phase
   1. Fetch a new frame from input video sequence. Here it's assumed that it's already VPI image with RGB8 format.

      while inVideo.read(input)[0]:

   2. Feed it into the bgsub object. The estimation of the foreground mask and the background are returned as VPI images.

          fgmask, bgimage = bgsub(input, learnrate=0.01)

1. Initialization phase:
   1. Include the header that defines the background subtractor function:

      #include <vpi/algo/BackgroundSubtractor.h>

   2. Define the required images:

          VPIImage input     = /* frame from the video sequence */;
          VPIPyramid fgmask  = /* foreground mask. Format has to be VPI_IMAGE_FORMAT_U8 */;
          VPIPyramid bgimage = /* background image. Image format has to be the same as input image format */;

   3. Create the stream to which the algorithm is to be submitted for execution:

          VPIStream stream;
          vpiStreamCreate(0, &stream);

   4. Create the algorithm payload to process input image with provided width, height and format:

          VPIPayload payload;
          vpiCreateBackgroundSubtractor(VPI_BACKEND_CUDA, w, h, VPI_IMAGE_FORMAT_RGB8, &payload);

2. Processing phase:
   1. Start of the processing loop. Fetch the input from the video sequence.

          for (i = 0; i < nframes; ++i)
          {
              input = /* Fetch frame from video sequence */;

   2. Configure the parameters for the submission:

              VPIBackgroundSubtractorParams algoParams;
              vpiInitBackgroundSubtractorParams(&algoParams);
              algoParams.learningRate = 0.01;

   3. Submit the algorithm to the stream, to be executed by the CUDA backend, along with current input frame, the output foreground mask and background image:

              vpiSubmitBackgroundSubtractor(stream, VPI_BACKEND_CUDA, payload, input, fgmask, bgimage, &algoParams);

   4. Optionally, wait until the processing is done:

              vpiStreamSync(stream);

   5. Process the results once they are ready:

              process_results(fgmask, bgimage);
          }

3. Cleanup phase:
   1. Free resources held by the stream, the payload, the input image, the foreground image and background image:

          vpiStreamDestroy(stream);
          vpiPayloadDestroy(payload);
          vpiImageDestroy(input);
          vpiImageDestroy(fgmask);
          vpiImageDestroy(bgimage);

Performance

For information on how to use the performance table below, see Algorithm Performance Tables.
Before comparing measurements, consult Comparing Algorithm Elapsed Times.
For further information on how performance was benchmarked, see Performance Benchmark.

clear filters Device: Jetson AGX Xavier Jetson AGX Orin  -  Streams: 1 2 4 8

References

1. Z. Zivkovic, "Improved Adaptive Gaussian Mixture Model for Background Subtraction."
   Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.
2. Wikipedia article on Mahalanobis distance