Stereo Disparity Estimator

Overview

Given a pair of rectified images from a stereo camera, the Stereo Disparity algorithm uses high-quality dense stereo matching to produce an output image of the same resolution as the input with left-right disparity information. This allows for inferring the depth of the scene captured by the left and right images.

Left image	Right image
Disparity map	Confidence map

Implementation

The stereo disparity estimator uses semi-global matching algorithm (SGM) to compute the disparity for the CUDA backend. The OFA backend (that can be combined with VIC and PVA backends) does an advanced version of the SGM algorithm to optimize memory bandwidth and performance.

The main differences of the advanced version of SGM in OFA backend and the SGM in CUDA backend are:

1. CUDA has a higher memory footprint, which may restrict the algorithm usage as it may exhaust the GPU memory
2. OFA allows up to 8192x8192 input image resolution without consuming CPU or GPU memory resources
3. CUDA allows to control the uniqueness ratio in VPIStereoDisparityEstimatorParams
4. OFA allows to control the number of passes and the P2 alpha in VPIStereoDisparityEstimatorParams

C API functions

For list of limitations, constraints and backends that implements the algorithm, consult reference documentation of the following functions:

Function	Description
vpiInitStereoDisparityEstimatorCreationParams	Initializes VPIStereoDisparityEstimatorCreationParams with default values.
vpiCreateStereoDisparityEstimator	Creates payload for vpiSubmitStereoDisparityEstimator.
vpiInitStereoDisparityEstimatorParams	Initializes VPIStereoDisparityEstimatorParams with default values.
vpiSubmitStereoDisparityEstimator	Runs stereo processing on a pair of images and outputs a disparity map.

Usage

Language: C/C++ Python

1. Import VPI module

   import vpi

2. Estimate the disparity between left and right input VPI images, using 5x5 window and 64 maximum disparity. Optionally, the resulting disparity image is converted to U8, with range [0,255], suited for display.

   with vpi.Backend.CUDA:
       output = vpi.stereodisp(left, right, window=5, maxdisp=64) \
                   .convert(vpi.Format.U8, scale=1.0/(32*64)*255)

1. Initialization phase
   1. Include the header that defines the needed functions and structures. ColorImageFormat algorithm will be needed to process the disparity output for display.

      #include <vpi/algo/ConvertImageFormat.h>
      #include <vpi/algo/StereoDisparity.h>

   2. Define the input rectified stereo pair.

          VPIImage left  = /*...*/;
          VPIImage right = /*...*/;

   3. Create the output disparity and confidence images, and the image used for disparity display (optional).

          int32_t w, h;
          vpiImageGetSize(left, &w, &h);
       
          VPIImage disparity;
          vpiImageCreate(w, h, VPI_IMAGE_FORMAT_S16, 0, &disparity);
       
          VPIImage confidence;
          vpiImageCreate(w, h, VPI_IMAGE_FORMAT_U16, 0, &confidence);
       
          VPIImage display;
          vpiImageCreate(w, h, VPI_IMAGE_FORMAT_U8, 0, &display);

   4. Create the payload that will contain all temporary buffers needed for processing. It'll be created on the CUDA backend.

          VPIPayload stereo;
          vpiCreateStereoDisparityEstimator(VPI_BACKEND_CUDA, 480, 270, VPI_IMAGE_FORMAT_U16, NULL, &stereo);

   5. Create the stream where the algorithm will be submitted for execution.

          VPIStream stream;
          vpiStreamCreate(0, &stream);

2. Processing phase
   1. Define the configuration parameters needed for algorithm execution.

          VPIStereoDisparityEstimatorParams params;
          vpiInitStereoDisparityEstimatorParams(&params);
          params.windowSize   = 5;
          params.maxDisparity = 64;

   2. Submit the payload for execution on the backend associated with it.

          vpiSubmitStereoDisparityEstimator(stream, VPI_BACKEND_CUDA, stereo, left, right, disparity, confidence, &params);

   3. Optionally, the resulting disparity image can be converted to U8 format and the disparity values are rescaled to fit in [0,255] range, suited for display.

          VPIConvertImageFormatParams cvtParams;
          vpiInitConvertImageFormatParams(&cvtParams);
          cvtParams.scale = 1.0f / (32 * params.maxDisparity) * 255;
       
          vpiSubmitConvertImageFormat(stream, VPI_BACKEND_CUDA, disparity, display, &cvtParams);

   4. Wait until the processing is done.

          vpiStreamSync(stream);

3. Cleanup phase
   1. Free resources held by the stream, the payload and the input and ouput images.

          vpiStreamDestroy(stream);
          vpiPayloadDestroy(stereo);
          vpiImageDestroy(left);
          vpiImageDestroy(right);
          vpiImageDestroy(display);
          vpiImageDestroy(disparity);
          vpiImageDestroy(confidence);

Consult the Stereo Disparity Sample for a complete example.

For more information, see Stereo Disparity Estimator in the "C API Reference" section of VPI - Vision Programming Interface.

Performance

For information on how to use the performance table below, see Algorithm Performance Tables.
Before comparing measurements, consult Comparing Algorithm Elapsed Times.
For further information on how performance was benchmarked, see Performance Benchmark.

clear filters Device: Jetson AGX Thor  -  Streams: 1 2 4 8

References

• Hirschmüller, Heiko (2005). "Accurate and efficient stereo processing by semi-global matching and mutual information".
  IEEE Conference on Computer Vision and Pattern Recognition. pp. 807–814.