The NVIDIA® Static Camera Calibration tutorial describes how to perform intrinsic and extrinsic calibration for a vehicle's cameras using the Camera Calibration Tools.
This step-by-step tutorial covers the following aspects:
The following cameras and targets are required before proceeding in this tutorial.
The following cameras are required for calibration:
These comprise the set of cameras to be calibrated, referred to as "camera" or "cameras" in this tutorial:
An external photo camera is required to support the calibration of the AV Cameras, referred to as "external camera" in this tutorial:
This tutorial requires the use of targets to calibrate the vehicle's cameras.
The AprilTag targets used in this tutorial can be found in the following folder:
data/tools/calibration/aprilTargets
At 100% scaling factor, these target files (incl. boundary!) are of size DIN A4, but they are in vector format and can be scaled in a lossless fashion to A3, A2, A1, or A0.
An example of an AprilTag target used in this tutorial:
The targets must be printed as follows:
data/tools/calibration/aprilTargets
data/tools/calibration/aprilTargets
.Additionally, a checkerboard target is needed for intrinsics calibration. Two checkerboard target versions can be found under data/tools/calibration/checkerboardTargets
, one for A0 and one for ANSI E paper format:
Any checkerboard pattern with the following properties is supported:
For each AprilTag target, the length of the horizontal bar in millimeters (mm) must be measured and noted down in the specified location in the top left corner, as demonstrated in the above image.
Depending on the number and location of the cameras on the vehicle, different scene setups may be used.
In this tutorial, a setup with 12 cameras mounted on a car roof is used as an example.
The following constraints and best-practices should be respected when placing targets during scene setup. Most of these constraints can be verified online with the Static Calibration Recorder Tool tool in extrinsics
mode:
6m
and targets need to be placed with this minimal distance, while the focus region of other camera types usuall allows or demands targets to be placed closer to the sensor.extrinsics
mode, which has to visualize stable detections of all observed targets.Here is an illustration of how to place the AprilTag targets around a car with given sensor set:
Note that two targets are used for both front/rear tele cameras at different depth and further away compared to the other targets for non-tele cameras to fullfill the target plamement requirements.
In this tutorial, a set of 8 large and 4 small AprilTag targets are used as an example.
The following types of data must be captured for static camera calibration:
In order to capture data for intrinsic camera calibration, a checkerboard target has be be moved in front of each camera to be calibrated.
Perform the following for one camera at a time:
There are different options to capture the data for intrinsic camera calibration:
Still images can be captured using Camera Sample. Also see 3.2.1 Extrinsics Images, where it is also necessary to get still images from video or camera.
For the video option, refer to Basic Recording for more information how to use the camera recorder.
format=h264,output-format=yuv
as camera sensor properties. The Intrinsics Constraints Tool will automatically extract all valid frames from such a recorded video (see 4.2 Camera Intrinsics).
The Static Calibration Recorder Tool tool allows to skip the recording step, as it directly extracts the required intrinsics constraints data from the live video stream of the camera.
To calibrate the intrinsics of the supplied external camera, a set of checkerboard images should be captured, following the same process as described above (3.1 Capturing Data for Intrinsic Camera Calibration). It is also possible to move the camera instead of the checkerbard target, which may be more convenient when using the external camera.
Use one of the following methods to capture one image from each car camera. These images should then be named and placed in the appropriate location according to 4.1 Directory Structure.
Run the Static Calibration Recorder Tool tool with the rig file containing the cameras to be calibrated, select the "Extrinsics" mode, set up the scene with the AprilTag targets as described in 2 Scene Setup and save the scene images to files.
Run the Camera Sample to capture a frame.
./sample_camera --rig=<path/to/rig>
Run the DriveWorks Recording Tools to simultaneously record video from all cameras. Use ffmpeg to extract one frame from the video.
ffmpeg -i \<folder\>/camera_X.h264 -t 1 -f image2 camera_X.png
In addition to capturing frames from the vehicle's cameras, a set of images from an outside view is required.
These external images help constructing a calibration graph as they introduce a link between AprilTag targets not observable by the car cameras.
All AprilTag targets need to be linked, including ground targets, wheel targets, and the targets observed by the individual cameras. Links are established between AprilTag targets that are visible in a single image.
Consider the following for external images:
The following demonstrates the types of images to capture with the external camera:
The Static Calibration Tool suite consists of multiple command line tools. This tutorial covers the tools required for static calibration of the vehicle's cameras. The core tool for camera calibration is the Graph Calibration Tool. There are additional calibration tools for auxiliary support and additional sensors. These are not covered in this tutorial. All tools used in this section are documented in Camera Calibration Tools.
The calibration data from 3 Capturing Data is passed to the Graph Calibration Tool through a specific directory structure with a predefined collection of files, see Directory.
Place
Modify the targets.json
file and change the barLength
fields for all AprilTag targets in the scene to the length (in meters) measured in 1.2.3 Target Print Validation. Also see Target Database `targets.json`.
special-targets.json: The AprilTag targets attached to the wheels and the AprilTag targets that lie flat on the floor must also be declared in this file. See Special Targets for the exact file structure.
Ensure that the targets on the ground and the onces attached to the wheels are correctly identified in this file.
extra-constraints.json: Additional prior knowledge (e.g., from CAD data) can be provided to the estimation with this file, which can be beneficial to improve the quality of the estimation. See Extra Constraints for the exact file structure and Extra Constraint Types for a list of currently supported constraints.
Ensure carefully that all provided prior values in this file are accurate (up to their respective expected variances), as inaccurate prior's can only be corrected to a limited extend by the estimation and will likely result in calibration failures.
With the exception of the external camera, this section can be skipped if the Static Calibration Recorder Tool tool has been used in 3.1.1 Recording Options, as that directly outputs the .json files with intrinsics constraints.
Otherwise, if a calibration video or images have been captured in 3.1.1 Recording Options, the usage of the DriveWorks Intrinsics Constraints Tool is required to extract the intrinsics constraints for each individual camera.
The tool exports all required constraints into a .json
file to to be placed into the /intrinsics/
subfolder in the directory structure. The actual calibration of the cameras' intrinsics happens later when running the Graph Calibration Tool.
See these usage Examples on how to extract the intrinsics constraints from the calibration videos or images.
The tool will open a window playing back the input video and indicate with a red or green border if a new intrinsic constraint has been collected (i.e. AprilTag target or checkerboard has been found).
This section requires the usage of the DriveWorks Graph Calibration Tool, to construct a graph-based representation of the data obtained, which is then further optimized.
Running this tool without parameters assumes you have the file structure specified in 4.1 Directory Structure.
The tool determines the names of the cameras from the extrinsics
folder. It then searches the intrinsic
folder for .json
files corresponding with the intrinsics constraints.
The images captured with the external camera will also be used for intrinsic calibration for the external camera, and for extrinsics constraints.
The Calibration Tool will then use these images to construct a graph representation which is later optimized.
Ensure the following before proceeding:
\<calib_data_path\>/extrinsics
directory.\<calib_data_path\>/intrinsics
directory.extrinsics
and the .json
file names in intrinsics
match.external.json
file with the intrinsics constraints of the external camera in the intrinsics
directory.\<calib_data_path\>/external
directory.targets.json
contains the correct barLengths for all AprilTags in the images.special-targets.json
contains the correct IDs for ground and wheel targets.extra-constraints.json
file, ensure that all values related to prior knowledge are accurate and consistent with the sensors being calibrated.Run the Graph Calibration Tool by executing the following:
./calibration-graph-cli --dir=<calib_data_path>
The tool outputs progress to the console. Take note of warnings in yellow and errors in red as they indicate possible problems with the calibration.
The following outputs are placed in the target folder:
calibrated-graph.json
is the main output for this tool. This contains all constraints, the calibrated camera models, the camera poses, the target poses, etc.Validation images can be used to double check if the computed calibration is valid, i.e., if the green re-projections match the targets.
In the following image, a green mask clearly overlays the AprilTag targets, and the corners are well aligned. This indicates a valid computed calibration.
In the following image, a green mask is misaligned with the AprilTag target. This indicates the calibration results might not be correct, or that there is a problem with the calibration target.
targets.json
is not correct.special-targets.json
are not correct. In particular this can cause the coordinate system to be wrong.extrinsic-constraints.json
does not correspond to the actual true values. This can causes the estimation to be restricted to an incorrect space of solutions.Ensure that the coordinate system is as expected. An orange line should be where the rear wheels touch the floor. The other orange line should go through the projection of the rear axle center onto the ground, and go along the forward direction of the car. If this is not the case, double check that the wheel and ground targets in the special-targets.json
are correct.
The Graph Calibration Tool generates a calibrated-graph.json
file as output. This file contains an intermediate calibration result, it cannot yet be consumed by the NVIDIA® DriveWorks SDK. The calibrated graph can be converted into a valid DriveWorks SDK rig.json
file using the Calibrated Graph to Rig File Tool. The rig.json
file contains the full calibrated camera configuration and can be directly consumed by the DriveWorks modules, e.g. Rig Configuration.
To perform the conversion, run the tool by executing the following:
./calibration-graph-to-rig --graph=<calib_data_path>/calibrated-graph.json
The tool generates a rig.json
file in the current folder. This file lists all cameras used during calibration with their final intrinsic and extrinsic calibration data.
If an existing rig.json
is passed as input, all camera entries are modified with the new calibration results.
It is possible to calibrate for camera intrinsics only, if extrinsic calibration is not desired or required.
The procedure is similar to the full calibration workflow, with minor modifications:
intrinsics
directory and the targets.json
file are required.calibrated-graph.json file
and intrinsic validation images. Verify that these validation images meet the requirements described in 4.3.3 Extrinsic Calibration Validation.rig.json
file with the result. Refer to 4.4 Generating DriveWorks `rig.json` File for more information.