calibration/engine/docs/usecase_stereo.md

Go to the documentation of this file.

# Copyright (c) 2019-2020 NVIDIA CORPORATION.  All rights reserved.

@page calibration_usecase_stereo Epipolar-based Stereo Self-Calibration

@note SW Release Applicability: This tutorial is applicable to modules in both **NVIDIA DriveWorks** and **NVIDIA DRIVE Software** releases.

## Operating Principle

Given a horizontal stereo rig, the right camera's extrinsic pose calibration
with respect to the left camera is represented by both orientation and position
parameters.

Orientation parameters are defined as roll, pitch and yaw, while position
parameters are defined as x, y, z. The orientation parameters identify the
rotation that aligns the right camera orientation to the left camera
orientation. The position parameters identify the translation between the right
camera position and the left camera position. All six parameters are crucial in
tasks such as stereo rectification and disparity estimation that rely on the
relative pose of the right camera with respect to the left camera.

NVIDIA<sup>&reg;</sup> DriveWorks uses visually matched features to calibrate
the right camera's orientation and position in a self-calibration setting. While
orientation is calibrated fully, position is only calibrated up to norm, i.e.,
up to the relative translation between the two cameras specified in the rig
file. All six parameters are calibrated simultaneously by estimating the
transformation that best satisfies the epipolar constraint on the feature
matches.

### Right Camera Roll / Pitch / Yaw / X / Y / Z

The features obtained from the left image are matched to those obtained from the
right image. Using the well-known epipolar constraint, the relative
transformation between the left and the right camera is estimated. This relative
transformation provides roll, pitch, yaw, as well as x, y, z components of the
right camera with respect to the left camera.

The instantaneous calibration for an image pair is accumulated in histograms for
the calibration's roll, pitch, yaw, x, y, z components to estimate a final
calibration in a robust way.

![Matched features are represented by the same color in the top left and top right image tiles. Roll, pitch, yaw, inclination and azimuth histograms of an epipolar-calibrated stereo rig, collected over a period of time, are shown in the bottom tiles. Inclination and azimuth histogram are used to have a compact representation of the x, y, z translation component up to norm.](self_calib_stereo_matches.png)

## Requirements

### Initialization Requirements

- Nominal values on camera calibration
    - Orientation(roll/pitch/yaw): roll/pitch/yaw less than 5 degree error
    - Position(x/y/z): less than 10 mm
    - Intrinsic calibration: accurate to 0.5 pixel

### Runtime Calibration Dependencies

- IMU-based egomotion needs to be based on accurate IMU calibration and odometry properties

### Input Requirements

- Assumption: feature matches (right-to-left) compatible with DriveWork's module @ref imageprocessing_features_mainsection

### Output Requirements

- Corrected calibration for roll/pitch/yaw/x/y/z (mandatory)
- Correction accuracy:
    + roll/yaw: less than 0.10deg error
    + pitch: less than 0.2deg error
    + x/y/z: less than 2cm error
- Time/Events to correction:
    + roll/pitch/yaw/x/y/z: less than 0:27 minutes in 75% of the cases, less than 0:49 minutes in 100% of the cases

## Cross-validation KPI

Several hours of data are used to produce a reference calibration value for
cross-validation. Then, short periods of data are evaluated for whether they can
recover the same values. For example, the graph below shows precision/recall
curves of stereo self-calibration.
Precision indicates that an accepted calibration is within a fixed precision threshold
from the reference calibration, and recall indicates the ratio of accepted calibrations
in the given amount of time.

![](self_calib_stereo_kpi.png)

## Workflow

The following code snippet shows the general structure of a program that performs stereo self-calibration

```{.cpp}
    dwFeatureHistoryArray matchesHistoryGPU;   // used by the features module
    dwFeatureArray matchesDetectedGPU;         // used by the features module

    dwCalibrationEngine_initialize(...);       // depends on sensors from rig configuration module
    dwCalibrationEngine_initializeStereo(...); // depends on nominal calibrations from rig configuration.
    dwCalibrationEngine_startCalibration(...); // runtime calibration dependencies need to be met

    while(true) // main loop
    {
        // track features for current left camera frame
        dwFeature2DTracker_trackFeatures(matchesHistoryGPU, ..., matchesDetectedGPU, ...);

        // detect features for current left camera frame
        dwFeature2DDetector_detectFromPyramid(matchesDetectedGPU, ...);

        // track features for current right camera frame
        dwFeature2DTracker_trackFeatures(matchesHistoryGPU, ..., matchesDetectedGPU, ...);

        // detect features for current right camera frame
        dwFeature2DDetector_detectFromPyramid(matchesDetectedGPU, ...);

        // feed feature matches into self-calibration
        dwCalibrationEngine_addMatches(matchesHistoryGPU, ...);

        // retrieve calibration status
        dwCalibrationEngine_getCalibrationStatus(...);

        // retrieve self-calibration results
        dwCalibrationEngine_getSensorToSensorTransformation(...);
    }

    dwCalibrationEngine_stopCalibration(...);
```

This workflow is demonstrated in the following sample: @ref dwx_stereo_calibration_sample