AprilTag Detector and Pose Estimator

Overview

AprilTags are fiducial markers designed for visual detection and localization. VPI supports detection and pose estimation of a number of common tag families, and may be used as a drop-in replacement for the reference detection algorithm.

For more information, refer to the landing page for AprilTags at April Robotics.

Input	Detection output

Implementation

Detector

Implementation of the detector is divided into the following stages:

Stage	Description	Backends
Decimation	Input is decimated by a factor of 2	PVA/CPU
Adaptive thresholding	Decide whether pixels are black or white in the decimated image	PVA/CPU
Connected component labeling	Assign unique IDs to contiguous regions of black/white pixels	PVA/CPU
Gradient cluster generation	Cluster all the coordinates which lie on the edge between a given black/white region pair	PVA/CPU
Quad fitting	Attempt to fit quadrilaterals to each gradient cluster	CPU
Decoding	Attempt to decode each quadrilateral as an AprilTag	CPU

When choosing VPI_BACKEND_PVA, VPI will use PVA for the stages which support this backend and fall back to CPU for the remainder. The VPIStream and input/output buffers should be created with support for both PVA and CPU.

Pose estimator

In general, there are eight mathematical solutions to 3D pose estimation from homography decomposition [1]. Of these eight, only two of these are valid subject to the constraints that (1) the camera is in front of the tag and (2) the camera is facing the tag.

The pose estimator attempts to estimate both of these pose candidates via SVD followed by a fixed number of iterations which minimize corner reprojection error. If both pose candidates are valid, the estimator returns the pose which results in the least corner reprojection error.

C API functions

For list of limitations, constraints and backends that implements the algorithm, consult reference documentation of the following functions:

Function	Description
vpiInitAprilTagDecodeParams	Initializes VPIAprilTagDecodeParams with default values.
vpiCreateAprilTagDetector	Creates a AprilTag detector payload.
vpiSubmitAprilTagDetector	Submits a AprilTag detector operation to the stream.
vpiSubmitAprilTagPoseEstimation	Submits a AprilTag pose estimation operation to the stream.

Usage

Language: C/C++

1. Initialization phase:
   1. Include the header that defines AprilTags functions:

      #include <vpi/algo/AprilTags.h>

   2. Define the required input image:

          VPIImage input = /*...*/;

   3. Define the payload to process input image with provided width and height:

          int32_t w, h;
          vpiImageGetSize(input, &w, &h);
       
          VPIPayload payload;
          const int maxHamming           = 1;
          const VPIAprilTagFamily family = VPI_APRILTAG_36H11;
          VPIAprilTagDecodeParams params = {NULL, 0, maxHamming, family};
          vpiCreateAprilTagDetector(VPI_BACKEND_CPU, w, h, &params, &payload);

   4. Define the output arrays:

          const int maxDetections = 64;
       
          VPIArray detections;
          vpiArrayCreate(maxDetections, VPI_ARRAY_TYPE_APRILTAG_DETECTION, VPI_BACKEND_CPU, &detections);
          VPIArray poses;
          vpiArrayCreate(maxDetections, VPI_ARRAY_TYPE_POSE, VPI_BACKEND_CPU, &poses);

   5. Create the stream to which the algorithm is to be submitted for execution:

          VPIStream stream;
          vpiStreamCreate(0, &stream);

2. Processing phase:
   1. Submit the detector:

          vpiSubmitAprilTagDetector(stream, VPI_BACKEND_CPU, payload, maxDetections, input, detections);

   2. Submit the pose estimator:

          const VPICameraIntrinsic intrinsics = {{w / 3.5f, 0.0f, w / 2.f}, {0.0f, h / 3.6f, h / 2.f}};
          const float tagSize       = 0.2f;
          vpiSubmitAprilTagPoseEstimation(stream, VPI_BACKEND_CPU, detections, intrinsics, tagSize, poses);

   3. Wait until the processing is done:

          vpiStreamSync(stream);

3. Cleanup phase:
   1. Free resources held by the stream, the payload, the input image and the output arrays:

          vpiStreamDestroy(stream);
          vpiImageDestroy(input);
          vpiArrayDestroy(detections);
          vpiArrayDestroy(poses);
          vpiPayloadDestroy(payload);

Performance

For information on how to use the performance tables below, see Algorithm Performance Tables.
Before comparing measurements, consult Comparing Algorithm Elapsed Times.
For further information on how performance was benchmarked, see Performance Benchmark.

clear filters Device: Jetson AGX Orin  -  Streams: 1 2 4 8

References

1. Malis, E. and Vargas, M. "Deeper understanding of the homography decomposition for vision-based control"
   Research Report 6303, INRIA 2007