Epipolar-based Stereo Self-Calibration

Note

SW Release Applicability: This tutorial is applicable to modules in both NVIDIA DriveWorks and NVIDIA DRIVE Software releases.

Operating Principle

Given a horizontal stereo rig, the right camera's extrinsic pose calibration with respect to the left camera is represented by both orientation and position parameters.

Orientation parameters are defined as roll, pitch and yaw, while position parameters are defined as x, y, z. The orientation parameters identify the rotation that aligns the right camera orientation to the left camera orientation. The position parameters identify the translation between the right camera position and the left camera position. All six parameters are crucial in tasks such as stereo rectification and disparity estimation that rely on the relative pose of the right camera with respect to the left camera.

NVIDIA^® DriveWorks uses visually matched features to calibrate the right camera's orientation and position in a self-calibration setting. While orientation is calibrated fully, position is only calibrated up to norm, i.e., up to the relative translation between the two cameras specified in the rig file. All six parameters are calibrated simultaneously by estimating the transformation that best satisfies the epipolar constraint on the feature matches.

Right Camera Roll / Pitch / Yaw / X / Y / Z

The features obtained from the left image are matched to those obtained from the right image. Using the well-known epipolar constraint, the relative transformation between the left and the right camera is estimated. This relative transformation provides roll, pitch, yaw, as well as x, y, z components of the right camera with respect to the left camera.

The instantaneous calibration for an image pair is accumulated in histograms for the calibration's roll, pitch, yaw, x, y, z components to estimate a final calibration in a robust way.

Matched features are represented by the same color in the top left and top right image tiles. Roll, pitch, yaw, inclination and azimuth histograms of an epipolar-calibrated stereo rig, collected over a period of time, are shown in the bottom tiles. Inclination and azimuth histogram are used to have a compact representation of the x, y, z translation component up to norm.

Requirements

Initialization Requirements

• Nominal values on camera calibration
  • Orientation(roll/pitch/yaw): roll/pitch/yaw less than 5 degree error
  • Position(x/y/z): less than 10 mm
  • Intrinsic calibration: accurate to 0.5 pixel

Runtime Calibration Dependencies

• IMU-based egomotion needs to be based on accurate IMU calibration and odometry properties

Input Requirements

• Assumption: feature matches (right-to-left) compatible with DriveWork's module Features

Output Requirements

• Corrected calibration for roll/pitch/yaw/x/y/z (mandatory)
• Correction accuracy:
  • roll/yaw: less than 0.10deg error
  • pitch: less than 0.2deg error
  • x/y/z: less than 2cm error
• Time/Events to correction:
  • roll/pitch/yaw/x/y/z: less than 0:27 minutes in 75% of the cases, less than 0:49 minutes in 100% of the cases

Cross-validation KPI

Several hours of data are used to produce a reference calibration value for cross-validation. Then, short periods of data are evaluated for whether they can recover the same values. For example, the graph below shows precision/recall curves of stereo self-calibration. Precision indicates that an accepted calibration is within a fixed precision threshold from the reference calibration, and recall indicates the ratio of accepted calibrations in the given amount of time.

Workflow

The following code snippet shows the general structure of a program that performs stereo self-calibration

dwFeatureHistoryArray matchesHistoryGPU;   // used by the features module
dwFeatureArray matchesDetectedGPU;         // used by the features module

dwCalibrationEngine_initialize(...);       // depends on sensors from rig configuration module
dwCalibrationEngine_initializeStereo(...); // depends on nominal calibrations from rig configuration.
dwCalibrationEngine_startCalibration(...); // runtime calibration dependencies need to be met

while(true) // main loop
{
    // track features for current left camera frame
    dwFeature2DTracker_trackFeatures(matchesHistoryGPU, ..., matchesDetectedGPU, ...);

    // detect features for current left camera frame
    dwFeature2DDetector_detectFromPyramid(matchesDetectedGPU, ...);

    // track features for current right camera frame
    dwFeature2DTracker_trackFeatures(matchesHistoryGPU, ..., matchesDetectedGPU, ...);

    // detect features for current right camera frame
    dwFeature2DDetector_detectFromPyramid(matchesDetectedGPU, ...);

    // feed feature matches into self-calibration
    dwCalibrationEngine_addMatches(matchesHistoryGPU, ...);

    // retrieve calibration status
    dwCalibrationEngine_getCalibrationStatus(...);

    // retrieve self-calibration results
    dwCalibrationEngine_getSensorToSensorTransformation(...);
}

dwCalibrationEngine_stopCalibration(...);

This workflow is demonstrated in the following sample: Stereo Calibration Sample