ORB

This gem provides a feature detector and descriptor extractor.

Features are used in applications such as:

• 3D reconstruction using structure-from-motion
• Visual odometry (motion tracking) & SLAM
• Content-based image retrieval
• Image alignment & panorama stitching

“ORB” stands for “Oriented FAST and rotated BRIEF”. This shows that ORB is based on FAST, a feature detector, and BRIEF, a binary descriptor.

The original publication by Rublee, et al., titled “ORB: An efficient alternative to SIFT or SURF”, can be found here: http://www.willowgarage.com/sites/default/files/orb_final.pdf

ORB has the following key qualities compared to other feature types:

• Efficient; Outstanding performance-quality tradeoff
• Resistant to image noise
• Rotation invariant
• Multi-scale

The Isaac SDK ORB gem follows the original publication closely. In addition to that, it also:

• Implements most of the pipeline in CUDA for higher performance by running on the GPU
• Adds a spatial regularization (“grid filtering”) step, which fixes a fundamental flaw of other implementations (e.g., OpenCV’s) by distributing keypoints over the image more evenly. This has significant benefits for applications such as motion tracking.

The algorithm steps are as follows:

1. Downsample input image to different scale levels
2. Extract FAST features on all levels
3. Apply grid filtering
4. Extract feature orientation
5. Extract descriptors

Keypoints

A Keypoint has the following properties:

• x, y: Keypoint coordinates in the image it was extracted from. Note: Since there are multiple scale levels, use the utility function GetCoordsAtTopLevel to get the coordinates at the topmost (initial) image level.
• response: “Strength” of a feature. A higher score means a higher local contrast.
• angle: Orientation angle (in radians)
• scale: Radius of the support region (in pixels)
• level: Scale level from which the feature was extracted. Starts at zero.

The Keypoints type is a vector of keypoints.

Descriptors

A Descriptor has the following properties:

• id: An ID that allows connecting a descriptor back to the keypoint from which it originated
• elements: The binary descriptor data, packed into a small buffer of elements.

The Descriptors type is a vector of descriptors.

The gem provides a single function, ExtractOrbFeatures, which is used to extract ORB features and descriptors from an image.

As an input parameter, an image has to be passed.

As an output parameter, Keypoints and Descriptors objects have to be passed.

These are the additional parameters that can be set:

• Max. features (default 1500): The desired number of best features to retain.
• FAST threshold (default 30): A threshold for the minimum local contrast a pixel has to exhibit in order to be considered a feature
• Grid number of cells linear (default 8): How many bins to use for spatial regularization in horizontal / vertical direction. The total number of bins is the square of the parameter. A higher number means spreading features more evenly. Setting to 1 disables the regularization.
• Downsampling factor (default 0.7): The size of the subsequent scale level, as a ratio of the current level. Values between 0.5 (inclusive) and 1.0 (exclusive) are allowed.
• Max. levels (default 4): How many scale levels to use.

Run the following command to build the gem, the codelet, and the sample application:

            

            
$ bazel build //packages/orb/...
        

The ORB package provides a single codelet, ExtractAndVisualizeOrb, whose sole purpose is to show an example of how to use the gem with a ZED camera.

The gem can of course be used directly in applications without going through the codelet.

ORB features overlaid over ZED camera image stream in Websight.

The example application extracts ORB features from images received from a ZED camera, and visualizes them using Websight.

Whatever platform you run the sample application on, make sure to first connect the ZED camera. While the application is running, open Isaac Sight in a browser by navigating to http://localhost:3000. If you are running the application on a Jetson platform, make sure to use the IP address of the Jetson system instead of localhost.

Host Device

Run the following command to start the demo:

            

            
$ bazel run //packages/orb/apps/orb_demo
        

Embedded Jetson Device

1. Deploy //packages/orb/apps/orb_demo:orb_demo-pkg to the robot as explained in Application Console Options.
2. SSH into the device with the following command:

            

            
$ ssh nvidia@ROBOTIP
        

3. Run the demo application on the device with the following command, after changing to the deployed directory:

            

            
$ cd orb_demo-pkg/
$ ./packages/orb/apps/orb_demo/orb_demo