## VPI - Vision Programming Interface

#### 0.4.4 Release

Harris Corner Detector

# Overview

This algorithm implements the Harris keypoint detection operator that is commonly used to detect keypoints and infer features of an image.

The standard Harris detector algorithm as described in [1] is applied first. After that, a non-max suppression pruning process is applied to the result to remove multiple or spurious keypoints.

Input Parameters Output keypoints

\begin{align*} \mathit{gradientSize} &= 5 \\ \mathit{blockSize} &= 5 \\ \mathit{strengthThresh} &= 20 \\ \mathit{sensitivity} &= 0.01 \\ \mathit{minNMSDistance} &= 8 \end{align*}

# Implementation

1. Compute the spatial gradient of the input using one of the following filters, depending on the value of VPIHarrisCornerDetectorParams::gradientSize :

\begin{align*} \mathit{sobel}_x &= \frac{1}{4} \cdot \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} -1 & 0 & 1 \end{bmatrix} \\ \mathit{sobel}_y &= (\mathit{sobel}_x)^\intercal \end{align*}

\begin{align*} \mathit{sobel}_x &= \frac{1}{16} \cdot \begin{bmatrix} 1 \\ 4 \\ 6 \\ 4 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} -1 & -2 & 0 & 2 & 1 \end{bmatrix} \\ \mathit{sobel}_y &= (\mathit{sobel}_x)^\intercal \end{align*}

\begin{align*} \mathit{sobel}_x &= \frac{1}{64} \cdot \begin{bmatrix} 1 \\ 6 \\ 15 \\ 20 \\ 15 \\ 6 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} -1 & -4 & -5 & 0 & 5 & 4 & 1 \end{bmatrix} \\ \mathit{sobel}_y &= (\mathit{sobel}_x)^\intercal \end{align*}

2. Compute a gradient covariance matrix (structure tensor) for each pixel within a block window, as described by:

$M = \sum_{p \in B}\begin{bmatrix}I_x^2(p) & I_x(p) I_y(p) \\ I_x(p) I_y(p) & I_y^2(p) \end{bmatrix}$

where:

• p is a pixel coordinate within B, a block window of size 3x3, 5x5 or 7x7.
• $$I(p)$$ is the input image
• $$I_x(p) = I(p) * \mathit{sobel}_x$$
• $$I_y(p) = I(p) * \mathit{sobel}_y$$
3. Compute a Harris response score using a sensitivity factor

$R = \mathit{det}(M) - k \cdot \mathit{trace}^2(M )$

where k is the sensitivity factor

4. Applies a threshold-strength criterion, pruning keypoints whose response <= VPIHarrisCornerDetectorParams::strengthThresh.
5. Applies a non-max suppression pruning process.

This process splits the input image into a 2D cell grid. It selects a single corner with the highest response score inside the cell. If several corners within the cell have the same response score, it selects the bottom-right corner.

# Usage

1. Initialization phase
1. Include the header that defines the box filter function.
2. Define the input image object.
VPIImage input = /*...*/;
3. Create the output arrays that will store the keypoints and their scores.
VPIArray keypoints;
vpiArrayCreate(8192, VPI_ARRAY_TYPE_KEYPOINT, 0, &keypoints);
VPIArray scores;
vpiArrayCreate(8192, VPI_ARRAY_TYPE_U32, 0, &scores);
4. Since this algorithm needs temporary memory buffers, create the payload for it on the CUDA backend.
uint32_t w, h;
vpiImageGetSize(input, &w, &h);
5. Create the stream where the algorithm will be submitted for execution.
VPIStream stream;
vpiStreamCreate(0, &stream);
2. Processing phase
1. Fill the configuration structure with parameters for the current algorithm invocation.
params.blockSize = 5;
params.strengthThresh = 20;
params.sensitivity = 0.01;
params.minNMSDistance = 8;
2. Submit the algorithm and its parameters to the stream. It'll be executed by the CUDA backend associated with the payload.
vpiSubmitHarrisCornerDetector(stream, harris, input, keypoints, scores, &params);
3. Optionally, wait until the processing is done.
vpiStreamSync(stream);
3. Cleanup phase
1. Free resources held by the stream, the payload, the input image and the output arrays.
vpiArrayDestroy(keypoints);
vpiArrayDestroy(scores);

For more details, consult the Harris Corners Detector API reference.

# Limitations and Constraints

Constraints for specific backends supersede the ones specified for all backends.

## All Backends

• Input image must have same dimensions as the ones specified during payload creation.
• Only supports Sobel gradient kernels of sizes 3x3, 5x5 and 7x7.
• Output scores and keypoints arrays must have the same capacity.
• Must satisfy $$\mathit{minNMSDistance} \geq 1$$.
• The following image types are accepted:
• On 16-bit inputs, the pixel values must be restricted to 12-bit, or else overflows in score calculation will occur. In this case some keypoints might be invalid.

## PVA

• Only available on Jetson Xavier devices.
• Only supports VPI_IMAGE_FORMAT_S16.
• Output keypoints and scores array capacity must be 8192.
• Only accepts $$\mathit{minNMSDistance} = 8$$.

## VIC

• Not implemented.

# Performance

For information on how to use the performance table below, see Algorithm Performance Tables.
Before comparing measurements, consult Comparing Algorithm Elapsed Times.
For further information on how performance was benchmarked, see Performance Measurement.

-

# References

1. C. Harris, M. Stephens (1988), "A Combined Corner and Edge Detector"
Proceedings of Alvey Vision Conference, pp. 147-151.
vpiArrayDestroy
void vpiArrayDestroy(VPIArray array)
Destroy an array instance.
VPIHarrisCornerDetectorParams::strengthThresh
float strengthThresh
Specifies the minimum threshold with which to eliminate Harris Corner scores.
Definition: HarrisCornerDetector.h:88
vpiArrayCreate
VPIStatus vpiArrayCreate(uint32_t capacity, VPIArrayType fmt, uint32_t flags, VPIArray *array)
Create an empty array instance.
vpiStreamCreate
VPIStatus vpiStreamCreate(uint32_t flags, VPIStream *stream)
Create a stream instance.
vpiCreateHarrisCornerDetector
Creates a Harris Corner Detector payload.
Deallocates the payload object and all associated resources.
vpiStreamSync
VPIStatus vpiStreamSync(VPIStream stream)
Blocks the calling thread until all submitted commands in this stream queue are done (queue is empty)...
VPI_BACKEND_CUDA
@ VPI_BACKEND_CUDA
CUDA backend.
Definition: Types.h:91
VPIStream
struct VPIStreamImpl * VPIStream
A handle to a stream.
Definition: Types.h:190
vpiStreamDestroy
void vpiStreamDestroy(VPIStream stream)
Destroy a stream instance and deallocate all HW resources.
vpiImageDestroy
void vpiImageDestroy(VPIImage img)
Destroy an image instance.
Definition: HarrisCornerDetector.h:82
VPI_ARRAY_TYPE_KEYPOINT
@ VPI_ARRAY_TYPE_KEYPOINT
VPIKeypoint element.
Definition: Types.h:233
vpiSubmitHarrisCornerDetector
VPIStatus vpiSubmitHarrisCornerDetector(VPIStream stream, VPIPayload payload, VPIImage input, VPIArray outFeatures, VPIArray outScores, const VPIHarrisCornerDetectorParams *params)
Submits Harris Corner Detector operation to the stream associated with the payload.
VPIImage
struct VPIImageImpl * VPIImage
A handle to an image.
Definition: Types.h:196
vpiImageGetSize
VPIStatus vpiImageGetSize(VPIImage img, uint32_t *width, uint32_t *height)
Get the image size in pixels.
A handle to an algorithm payload.
Definition: Types.h:208
VPIHarrisCornerDetectorParams::sensitivity
float sensitivity
Specifies sensitivity threshold from the Harris-Stephens equation.
Definition: HarrisCornerDetector.h:91
HarrisCornerDetector.h
Declares functions that implement the Harris Corner Detector algorithm.
VPIHarrisCornerDetectorParams::minNMSDistance
float minNMSDistance
Non-maximum suppression radius, set to 0 to disable it.
Definition: HarrisCornerDetector.h:94
VPIHarrisCornerDetectorParams::blockSize
uint32_t blockSize
Block window size used to compute the Harris Corner score.
Definition: HarrisCornerDetector.h:85
VPIHarrisCornerDetectorParams
Structure that defines the parameters for vpiSubmitHarrisCornerDetector.
Definition: HarrisCornerDetector.h:80
VPIArray
struct VPIArrayImpl * VPIArray
A handle to an array.
Definition: Types.h:172
VPI_ARRAY_TYPE_U32
@ VPI_ARRAY_TYPE_U32
unsigned 32-bit.
Definition: Types.h:232