Multi-Stream Stereo Disparity

Overview

The Multi-Stream Stereo Disparity application demonstrates high-performance stereo processing by utilizing multiple VPI streams concurrently. This sample shows how to process stereo pair images across multiple streams to measure throughput and performance characteristics of the stereo disparity algorithm when running in parallel.

The application loads left and right stereo images, converts them from BGR8 to NV12_BL format through an intermediate Y8_ER conversion, and processes them through multiple VPI streams simultaneously. It measures overall throughput, timing per frame, and can optionally save all processed results.

The C++ implementation provides comprehensive benchmarking capabilities with configurable stream counts and iterations, while the Python version demonstrates the basic multi-stream concept with a simplified interface.

Instructions

The command line parameters differ between the C++ and Python implementations:

C++ Implementation (main.cpp)

<left_image> <right_image> <num_streams> <iterations> [save_outputs]

where

• left_image: left input image of a rectified stereo pair
• right_image: right input image of a rectified stereo pair
• num_streams: number of parallel streams to use for processing
• iterations: number of processing iterations per stream
• save_outputs: true to save all input/output images, false for performance mode (optional, defaults to false)

Python Implementation (main.py)

<left> <right>

where

• left: rectified left input image from a stereo pair
• right: rectified right input image from a stereo pair

Here are some examples:

• C++

  ./vpi_sample_20_multistream_stereo ../assets/chair_stereo_left_960.png ../assets/chair_stereo_right_960.png 4 50

  ./vpi_sample_20_multistream_stereo ../assets/chair_stereo_left_960.png ../assets/chair_stereo_right_960.png 8 100 true

• Python

  python3 main.py ../assets/chair_stereo_left_960.png ../assets/chair_stereo_right_960.png

The C++ examples process the provided stereo pair using 4 streams for 50 iterations (first) and 8 streams for 100 iterations with output saving enabled (second). The Python example demonstrates basic multi-stream processing and saves a single disparity result as disparity.jpg.

Features

• Multi-Stream Processing: Utilizes multiple VPI streams for concurrent stereo processing
• Performance Benchmarking: C++ versions measure throughput (FPS), total processing time, and per-frame timing
• Flexible Input: Accepts PNG, JPEG, and other stereo pair images via command line arguments
  
• Multiple Processing Models:
  • Standard multi-stream processing (main.cpp)
  • Simplified Python demonstration (main.py)
• Format Conversion: Demonstrates BGR8 to NV12_BL conversion pipeline via intermediate Y8_ER format
• Output Saving: Optional mode in C++ versions to save all disparity results per stream and iteration
• Configurable Parameters: C++ versions support adjustable stream count and iteration parameters
• Cross-Language Support: Available in both C++ and Python implementations

Results

C++ Implementation (main.cpp)

The C++ applications print detailed performance metrics including total processing time, average time per frame, and throughput in FPS.

When save_outputs is enabled (set to "true"), the application generates a result per stream and per iteration as follows:

• Disparity results: stream_XX_iter_YYYY_disparity.png
• Confidence results: stream_XX_iter_YYYY_confidence.png

Python Implementation (main.py)

The Python application generates a single output file:

• Disparity result: disparity.jpg

Source Code

For convenience, here's the code that is also installed in the samples directory.

Language: C++ Python

import sys
import vpi
import numpy as np
from PIL import Image
from argparse import ArgumentParser
 
 
def process_arguments():
    parser = ArgumentParser()
    parser.add_argument('left', help='Rectified left input image from a stereo pair')
    parser.add_argument('right', help='Rectified right input image from a stereo pair')
    return parser.parse_args()
 
 
def process_stereo_pair(args):
    """
    Process stereo image pair to generate disparity map.
    
    This is the full script of the python code referenced in the Boston Dynamics Blog (link here)
    
    Args:
        args: Parsed command-line arguments containing left and right image paths
        
    Raises:
        FileNotFoundError: If input images cannot be found
        ValueError: If images are invalid or incompatible
        RuntimeError: If VPI operations fail
    """
    # The VPIStream object is the main entry point to the API. It is loosely based on CUDA's
    # cudaStream_t. This object represents a FIFO command queue which stores a list of commands
    # to be executed by some backend. Here, we create two streams - the left stream handles format
    # conversion for the left image as well as the actual stereo disparity operation, and the right
    # stream handles format conversion for the right image.
    streamLeft = vpi.Stream()
    streamRight = vpi.Stream()
 
    # VPI's Python API natively utilizes numpy as the memory backend for storing images.
    # Here, we load the stereo image pair into numpy arrays and then wrap them as VPI images.
    try:
        left_img = np.asarray(Image.open(args.left))
        right_img = np.asarray(Image.open(args.right))
    except FileNotFoundError as e:
        raise FileNotFoundError(f"Could not open image file: {e}")
    except Exception as e:
        raise ValueError(f"Error loading images: {e}")
 
    try:
        left = vpi.asimage(left_img)
        right = vpi.asimage(right_img)
    except Exception as e:
        raise RuntimeError(f"Failed to create VPI images: {e}")
 
    # We now submit the image conversion operations in parallel on two different streams.
    try:
        left = left.convert(vpi.Format.Y8_ER, scale=1, stream=streamLeft, backend=vpi.Backend.CPU)
        left = left.convert(vpi.Format.NV12_BL, scale=1, stream=streamLeft, backend=vpi.Backend.VIC)
        right = right.convert(vpi.Format.Y8_ER, scale=1, stream=streamRight, backend=vpi.Backend.CPU)
        right = right.convert(vpi.Format.NV12_BL, scale=1, stream=streamRight, backend=vpi.Backend.VIC)
    except Exception as e:
        raise RuntimeError(f"Image format conversion failed: {e}")
 
    # After converting the images to the correct format, we now want to submit the StereoDisparity
    # operation. We are going to submit this on streamRight, which means that it will naturally wait
    # for the right image to be done converting before running. But what about the left image?
    # VPI provides synchronization operations to handle cases just like this - here we can use the
    # most simple one. We call streamLeft.sync(), which will block the calling thread until
    # leftStream is done processing and we can then submit the Stereo Disparity operation safely.
    try:
        streamLeft.sync()
        disparityS16 = vpi.stereodisp(
            left, right, backend=vpi.Backend.OFA | vpi.Backend.PVA | vpi.Backend.VIC, stream=streamRight
        )
    except Exception as e:
        raise RuntimeError(f"Stereo disparity computation failed: {e}")
 
    # After some post-processing steps, we are done! We can then utilize VPI's zero-copy mapping
    # to OpenCV to create a nice OpenCV visualization of our resulting disparity map.
    try:
        disparityU8 = disparityS16.convert(
            vpi.Format.U8, scale=255.0 / (32 * 128), stream=streamRight, backend=vpi.Backend.CPU
        )
        streamRight.sync()
        Image.fromarray(disparityU8.cpu()).save('./disparity.jpg')
    except Exception as e:
        raise RuntimeError(f"Failed to save disparity output: {e}")
 
 
def main():
    """Main entry point for stereo disparity processing."""
    try:
        # Verify VPI can be used before processing arguments
        # This prevents core dumps from VPI initialization issues
        try:
            _ = vpi.Stream()
        except Exception as e:
            print(f"Error: VPI initialization failed: {e}", file=sys.stderr)
            print("Please check VPI installation and hardware support.", file=sys.stderr)
            return 1
        
        args = process_arguments()
        process_stereo_pair(args)
        return 0
    except FileNotFoundError as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1
    except ValueError as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1
    except RuntimeError as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1
    except Exception as e:
        print(f"Unexpected error: {e}", file=sys.stderr)
        return 1
 
 
if __name__ == '__main__':
    sys.exit(main())
 

/*
* Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*  * Redistributions of source code must retain the above copyright
*    notice, this list of conditions and the following disclaimer.
*  * Redistributions in binary form must reproduce the above copyright
*    notice, this list of conditions and the following disclaimer in the
*    documentation and/or other materials provided with the distribution.
*  * Neither the name of NVIDIA CORPORATION nor the names of its
*    contributors may be used to endorse or promote products derived
*    from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
 
#include <opencv2/core/version.hpp>
#if CV_MAJOR_VERSION >= 3
#    include <opencv2/imgcodecs.hpp>
#else
#    include <opencv2/contrib/contrib.hpp> // for colormap
#    include <opencv2/highgui/highgui.hpp>
#endif
 
#include <opencv2/imgproc/imgproc.hpp>
#include <vpi/OpenCVInterop.hpp>
 
#include <vpi/Event.h>
#include <vpi/Image.h>
#include <vpi/Status.h>
#include <vpi/Stream.h>
#include <vpi/algo/ConvertImageFormat.h>
#include <vpi/algo/Rescale.h>
#include <vpi/algo/StereoDisparity.h>
 
#include <algorithm>
#include <cctype>
#include <cstring>
#include <iostream>
#include <sstream>
#include <thread>
 
#define CHECK_STATUS(STMT)                                                                  \
    do                                                                                      \
    {                                                                                       \
        VPIStatus status = (STMT);                                                          \
        if (status != VPI_SUCCESS)                                                          \
        {                                                                                   \
            char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH];                                     \
            vpiGetLastStatusMessage(buffer, sizeof(buffer));                                \
            std::ostringstream ss;                                                          \
            ss << "line " << __LINE__ << " " << vpiStatusGetName(status) << ": " << buffer; \
            throw std::runtime_error(ss.str());                                             \
        }                                                                                   \
    } while (0);
 
int main(int argc, char *argv[])
{
    // OpenCV image that will be wrapped by a VPIImage.
    // Define it here so that it's destroyed *after* wrapper is destroyed
    cv::Mat cvImageLeft, cvImageRight;
 
    try
    {
        // =============================
        // Parse command line parameters
        if (argc != 5 && argc != 6)
        {
            throw std::runtime_error(std::string("Usage: ") + argv[0] + " " +
                                     "<left_image> <right_image> <num_streams> <iterations> [save_outputs:true|false]");
        }
 
        std::string strLeftFileName  = argv[1];
        std::string strRightFileName = argv[2];
        int numStreams               = std::stoi(argv[3]);
        int itersPerStream           = std::stoi(argv[4]);
        bool saveOutput              = false;
 
        if (argc == 6)
        {
            std::string arg5(argv[5]);
            std::transform(arg5.begin(), arg5.end(), arg5.begin(), ::tolower);
            saveOutput = (arg5 == "true");
        }
 
        if (numStreams <= 0 || itersPerStream <= 0)
        {
            throw std::runtime_error("Number of streams and iterations must be positive.");
        }
 
        // =====================
        // Load the input images
        cvImageLeft = cv::imread(strLeftFileName);
        if (cvImageLeft.empty())
        {
            throw std::runtime_error("Can't open '" + strLeftFileName + "'");
        }
        cvImageRight = cv::imread(strRightFileName);
        if (cvImageRight.empty())
        {
            throw std::runtime_error("Can't open '" + strRightFileName + "'");
        }
 
        // ====================
        // Stereo algorithm parameters
        constexpr int MAX_DISPARITY_VALUE = 128;
        constexpr int STEREO_WINDOW_SIZE  = 5;
 
        // ====================
        // Declare VPI Objects
        int totalIterations = itersPerStream * numStreams;
        std::vector<VPIImage> leftInputs(numStreams), rightInputs(numStreams), leftTmps(numStreams),
            rightTmps(numStreams);
        std::vector<VPIImage> leftOuts(numStreams), rightOuts(numStreams);
        std::vector<VPIImage> disparities, confidences;
        std::vector<VPIPayload> stereoPayloads(numStreams);
        std::vector<VPIStream> streamsLeft(numStreams), streamsRight(numStreams);
        std::vector<VPIEvent> events(numStreams);
        int width                        = cvImageLeft.cols;
        int height                       = cvImageLeft.rows;
        const uint64_t supportedBackends = VPI_BACKEND_VIC | VPI_BACKEND_OFA | VPI_BACKEND_PVA;
        VPIStereoDisparityEstimatorCreationParams stereoPayloadParams;
        VPIStereoDisparityEstimatorParams stereoParams;
        CHECK_STATUS(vpiInitStereoDisparityEstimatorCreationParams(&stereoPayloadParams));
        CHECK_STATUS(vpiInitStereoDisparityEstimatorParams(&stereoParams));
        stereoPayloadParams.maxDisparity = MAX_DISPARITY_VALUE;
        stereoParams.maxDisparity        = MAX_DISPARITY_VALUE;
        stereoParams.windowSize          = STEREO_WINDOW_SIZE;
        stereoParams.confidenceType      = VPI_STEREO_CONFIDENCE_RELATIVE;
 
        // ====================
        // Initialize VPI Objects
        for (int i = 0; i < numStreams; i++)
        {
            CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvImageLeft, 0, &leftInputs[i]));
            CHECK_STATUS(vpiImageCreateWrapperOpenCVMat(cvImageRight, 0, &rightInputs[i]));
            CHECK_STATUS(vpiStreamCreate(0, &streamsLeft[i]));
            CHECK_STATUS(vpiStreamCreate(0, &streamsRight[i]));
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_Y8_ER, 0, &leftTmps[i]));
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_NV12_BL, 0, &leftOuts[i]));
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_Y8_ER, 0, &rightTmps[i]));
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_NV12_BL, 0, &rightOuts[i]));
            CHECK_STATUS(vpiCreateStereoDisparityEstimator(supportedBackends, width, height, VPI_IMAGE_FORMAT_NV12_BL,
                                                           &stereoPayloadParams, &stereoPayloads[i]));
            CHECK_STATUS(vpiEventCreate(0, &events[i]));
        }
        int outCount = saveOutput ? (numStreams * itersPerStream) : numStreams;
        disparities.resize(outCount);
        confidences.resize(outCount);
        for (int i = 0; i < outCount; i++)
        {
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_S16, 0, &disparities[i]));
            CHECK_STATUS(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_U16, 0, &confidences[i]));
        }
 
        // ====================
        // Perform format conversion
        for (int i = 0; i < numStreams; i++)
        {
            CHECK_STATUS(
                vpiSubmitConvertImageFormat(streamsLeft[i], VPI_BACKEND_CPU, leftInputs[i], leftTmps[i], NULL));
            CHECK_STATUS(vpiSubmitConvertImageFormat(streamsLeft[i], VPI_BACKEND_VIC, leftTmps[i], leftOuts[i], NULL));
            CHECK_STATUS(vpiEventRecord(events[i], streamsLeft[i]));
            CHECK_STATUS(
                vpiSubmitConvertImageFormat(streamsRight[i], VPI_BACKEND_CPU, rightInputs[i], rightTmps[i], NULL));
            CHECK_STATUS(
                vpiSubmitConvertImageFormat(streamsRight[i], VPI_BACKEND_VIC, rightTmps[i], rightOuts[i], NULL));
            CHECK_STATUS(vpiStreamWaitEvent(streamsRight[i], events[i]));
        }
        for (int i = 0; i < numStreams; i++)
        {
            CHECK_STATUS(vpiStreamSync(streamsLeft[i]));
            CHECK_STATUS(vpiStreamSync(streamsRight[i]));
        }
 
        // ====================
        // Start Benchmarking
        auto benchmarkStart = std::chrono::high_resolution_clock::now();
        for (int iter = 0; iter < itersPerStream; iter++)
        {
            for (int i = 0; i < numStreams; i++)
            {
                int dispIdx = saveOutput ? (i * itersPerStream + iter) : i;
                CHECK_STATUS(vpiSubmitStereoDisparityEstimator(streamsRight[i], supportedBackends, stereoPayloads[i],
                                                               leftOuts[i], rightOuts[i], disparities[dispIdx],
                                                               confidences[dispIdx], &stereoParams));
            }
        }
        // ====================
        // End Benchmarking
        for (int i = 0; i < numStreams; i++)
        {
            CHECK_STATUS(vpiStreamSync(streamsRight[i]));
        }
        auto benchmarkEnd = std::chrono::high_resolution_clock::now();
        int64_t totalTime =
            std::chrono::duration_cast<std::chrono::microseconds>(benchmarkEnd - benchmarkStart).count();
 
        // ====================
        // Save Outputs
        if (saveOutput)
        {
            for (int i = 0; i < numStreams * itersPerStream; i++)
            {
                VPIImageData dispData, confData;
                cv::Mat cvDisparity, cvDisparityColor, cvConfidence, cvMask;
                CHECK_STATUS(
                    vpiImageLockData(disparities[i], VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &dispData));
                vpiImageDataExportOpenCVMat(dispData, &cvDisparity);
                cvDisparity.convertTo(cvDisparity, CV_8UC1, 255.0 / (32 * stereoParams.maxDisparity), 0);
                applyColorMap(cvDisparity, cvDisparityColor, cv::COLORMAP_JET);
                CHECK_STATUS(vpiImageUnlock(disparities[i]));
                std::ostringstream fpStream;
                fpStream << "stream_" << i / itersPerStream << "_iter_" << i % itersPerStream << "_disparity.png";
                imwrite(fpStream.str(), cvDisparityColor);
 
                // Confidence output (U16 -> scale to 8-bit and save)
                CHECK_STATUS(
                    vpiImageLockData(confidences[i], VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &confData));
                vpiImageDataExportOpenCVMat(confData, &cvConfidence);
                cvConfidence.convertTo(cvConfidence, CV_8UC1, 255.0 / 65535.0, 0);
                CHECK_STATUS(vpiImageUnlock(confidences[i]));
                std::ostringstream fpStreamConf;
                fpStreamConf << "stream_" << i / itersPerStream << "_iter_" << i % itersPerStream << "_confidence.png";
                imwrite(fpStreamConf.str(), cvConfidence);
            }
        }
 
        // ====================
        // Clean Up VPI Objects
        // NOTE: This sample uses explicit cleanup for simplicity. In production code,
        // use RAII wrappers for VPI resources to ensure automatic cleanup even if
        // exceptions occur during processing.
        for (int i = 0; i < numStreams; i++)
        {
            CHECK_STATUS(vpiStreamSync(streamsLeft[i]));
            CHECK_STATUS(vpiStreamSync(streamsRight[i]));
            vpiStreamDestroy(streamsLeft[i]);
            vpiStreamDestroy(streamsRight[i]);
            vpiImageDestroy(rightInputs[i]);
            vpiImageDestroy(leftInputs[i]);
            vpiImageDestroy(leftTmps[i]);
            vpiImageDestroy(leftOuts[i]);
            vpiImageDestroy(rightTmps[i]);
            vpiImageDestroy(rightOuts[i]);
            vpiPayloadDestroy(stereoPayloads[i]);
            vpiEventDestroy(events[i]);
        }
        // Destroy all disparity and confidence images
        for (int i = 0; i < (int)disparities.size(); i++)
        {
            vpiImageDestroy(disparities[i]);
        }
        for (int i = 0; i < (int)confidences.size(); i++)
        {
            vpiImageDestroy(confidences[i]);
        }
 
        // ====================
        // Output benchmarking data
        double totalTimeSeconds = totalTime / 1000000.0;
        double avgTimePerFrame  = totalTimeSeconds / totalIterations;
        double throughputFPS    = totalIterations / totalTimeSeconds;
 
        std::cout << "\n" << std::string(70, '=') << std::endl;
        std::cout << "SIMPLE MULTI-STREAM RESULTS" << std::endl;
        std::cout << std::string(70, '=') << std::endl;
        std::cout << "Total time: " << totalTimeSeconds << " seconds" << std::endl;
        std::cout << "Avg time per frame: " << (avgTimePerFrame * 1000) << " ms" << std::endl;
        std::cout << "THROUGHPUT: " << throughputFPS << " FPS" << std::endl;
        std::cout << std::string(70, '=') << std::endl;
 
        return 0;
    }
    catch (std::exception &e)
    {
        std::cerr << e.what() << std::endl;
        return 1;
    }
}