Gst-nvdewarper#
This plugin dewarps camera input. It accepts gpu-id and config-file as properties. Based on the selected configuration of surfaces, it can generate a maximum of four dewarped surfaces. It currently supports 18 dewarping projection types. Out of these, NVDS_META_SURFACE_FISH_PUSHBROOM
and NVDS_META_SURFACE_FISH_VERTCYL
are used in 360-D use case for dewarping 360° camera input.
The plugin performs its function in these steps:
Reads the configuration file and creates a vector of surface configurations. It supports a maximum of four dewarp surface configurations.
Receives the frame from the decoder. Based on the configuration, generates up to four dewarped surfaces.
Scales these surfaces down to network/selected dewarper output resolution using NPP APIs.
Pushes a buffer containing the dewarped surfaces to the downstream component.
Inputs and Outputs#
Inputs
A buffer containing a frame in RGBA format
Control parameters
gpu-id; selects the GPU ID (dGPU only)
config-file, containing the pathname of the dewarper configuration file
Output
Dewarped RGBA surfaces
NvDewarperSurfaceMeta
with information associated with each surface (projection_type, surface_index, and source_id), and the number of valid dewarped surfaces in the buffer (num_filled_surfaces)
Features#
The following table summarizes the features of the plugin.
Feature |
Description |
Release |
---|---|---|
Configure number of dewarped surfaces |
Supports a maximum of four dewarper surfaces. |
DS 3.0 |
Configure per-surface projection type |
Currently supports FishPushBroom and FishVertRadCyd projections. |
DS 3.0 |
Configure per-surface projection type |
PerspectivePerspective projections |
DS 5.0 |
Configure per-surface projection type |
18 projections |
DS 6.2 |
Configure per-surface index |
Surface index to be set in case of multiple surfaces having same projection type. |
DS 3.0 |
Configure per-surface width and height |
DS 3.0 |
|
Configure per-surface dewarping parameters |
Per surface configurable yaw, roll, pitch, top angle, bottom angle, and focal length and CUDA address mode dewarping parameters. For PerspectivePerspective projection type, field of view, distortion and principal point can additionally be configured. |
DS 3.0 |
Configurable dewarper output resolution |
Creates a batch of up to four surfaces of a specified output resolution; internally scales all dewarper surfaces to output resolution. |
DS 3.0 |
Configurable NVDS CUDA memory type |
— |
DS 3.0 |
Multi-GPU support |
— |
DS 3.0 |
Aisle view CSV calibration file support |
If set, properties in the [surface<n>] group are ignored. Only supported for FishPushBroom and FishVertRadCyd. |
DS 3.0 |
Spot view CSV calibration file support |
If set, properties in the [surface<n>] group are ignored. Only supported for FishPushBroom and FishVertRadCyd. |
DS 3.0 |
Configure source id |
Sets the source ID information in the NvDewarperSurfaceMeta. |
DS 4.0 |
Configurable number of output buffers |
Number of allocated output dewarper buffers. Each buffer contains four dewarped output surfaces. |
DS 4.0 |
Configurable number of surfaces per output buffer |
Number of dewarped output surfaces per buffer. Maximum allowed value is 4. |
DS 5.0 |
Support for advanced configuration parameters |
Fisheye distortion coefficients, destination focal length, destination principal point, rotation axes sequence, rotation matrix |
DS 6.2 |
Configuration File Parameters#
The configuration file located at configs/deepstream-app/config_dewarper.txt.
The configuration file specifies per-surface configuration parameters in [surface<n>]
groups, where <n> is an integer from 0 to 3, representing dewarped surfaces 0 to 3.
Property |
Property Type |
Meaning |
Type and Range |
Example Notes |
---|---|---|---|---|
output-width |
Global |
Scale dewarped surfaces to specified output width |
Integer, >0 |
output-width=960 |
output-height |
Global |
Scale dewarped surfaces to specified output height |
Integer, >0 |
output-height=752 |
dewarp-dump-frames |
Global |
Number of dewarped frames to dump. |
Integer, >0 |
dewarp-dump-frames=10 |
num-batch-buffers |
Global |
Number of dewarped output surfaces per buffer, i.e batch-size of the buffer. |
Integer, 0 to 4 |
num-batch-buffers=4 |
projection-type |
Surface |
Selects projection type. Supported types are:
|
Integer, 1 to 18 |
projection-type=1 |
surface-index |
Surface |
An index that distinguishes surfaces of the same projection type. |
Integer, ≥0 |
surface-index=0 |
width |
Surface |
Dewarped surface width. |
Integer, >0 |
width=3886 |
height |
Surface |
Dewarped surface height. |
Integer, >0 |
height=666 |
top-angle |
Surface |
Top field of view angle, in degrees. |
Float, −180.0 to 180.0 |
top-angle=0 |
bottom-angle |
Surface |
Bottom field of view angle, in degrees. |
Float, −180.0 to 180.0 |
bottom-angle=0 |
pitch |
Surface |
Viewing parameter pitch in degrees. |
Float, 0.0 to 360.0 |
pitch=90 |
yaw |
Surface |
Viewing parameter yaw in degrees. |
Float, 0.0 to 360.0 |
yaw=0 |
roll |
Surface |
Viewing parameter roll in degrees. |
Float, 0.0 to 360.0 |
roll=0 |
focal-length |
Surface |
Focal length of camera lens, in pixels per radian. Can be specified as :
|
Floats, >0.0 |
focal-length=437 OR focal-length=442.63;395.106339 |
src-fov |
Surface |
Source vertical field of view in degrees. Used to compute the source focal-length if the latter is not provided. |
Float, >0.0 |
src-fov=180 |
distortion |
Surface |
Distortion coefficients in the distortion polynomial for a fisheye/perspective source. For perspective source :
For fisheye source : Four radial coefficients
|
Array of 4/5 floats |
distortion= -0.3721;0.1023;-0.00203;0;0
If not provided, all coefficients are set to 0 (no distortion) |
src-x0 |
Surface |
The X coordinate of the center of projection of the source image, in pixel coordinates. Typically, it is (srcWidth-1)/2, but may be different if the image is calibrated or wraps around. |
Float, ≥0 |
src-x0=479.5 If not provided, will take its default value = (srcWidth-1)/2 |
src-y0 |
Surface |
The Y coordinate of the center of projection of the source image, in pixel coordinates. Typically, it is (srcHeight-1)/2, but may be different if the image is calibrated or wraps around. |
Float, ≥0 |
src-y0=375.5 If not provided, will take its default value = (srcHeight-1)/2 |
cuda-address-mode |
Surface |
Cuda Texture addressing mode. Currently supports:
|
Integer, 0 or 1 |
cuda-address-mode= 1 |
aisle-calibration-file |
Global |
Pathname of the configuration file for aisle view. Set for the 360 D application only for PushBroom and/or VertRadCyl projection types. If set, properties in the [surface<n>] group are ignored. The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height. |
String |
aisle-calibration-file= csv_files/nvaisle_2M.csv |
spot-calibration-file |
Global |
Pathname of the configuration file for spot view. Set for the 360 D application only for PushBroom and/or VertRadCyl projection types. If set, properties in the [surface<n>] group are ignored. The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height. |
String |
spot-calibration-file= csv_files/-nvspot_2M.csv For an example of a spot view configuration file, see the file in the example above. |
rot-axes |
Surface |
A three-character string that specifies the sequence of application of rotation angles. Possible values are : XYZ, XZY, YXZ, YZX, ZXY, ZYX where :
Default sequence of rotation is YXZ |
String of three characters |
|
control |
Surface |
A control parameter to vary the amount of warp. Applicable for Panini, Stereographic and Pushbroom projections. |
Float |
control=1.0. control=0 is identical to perspective output. |
rot-matrix |
Surface |
Configures the rotation as a 3X3 rotation matrix. The matrix is specified in a right-handed screen space, with X to the right, Y downwards and Z outwards. If specified, this setting will override yaw, pitch, roll settings. |
Array of 9 floats |
rot-matrix=0.5;-0.86;0;0.86;0.5;0;0;0.0;1.0 |
dst-focal-length |
Surface |
Sets destination image focal length. Can be specified as :
If specified, this setting will override top-angle, bottom-angle settings. |
Floats, >0.0 |
dst-focal-length=350;316 |
dst-principal-point |
Surface |
Set the X coordinate and Y coordinate in pixels of the principal point in destination image If specified, this setting will override top-angle, bottom-angle settings. |
Float, >0.0 |
dst-principal-point=521.02;257.37 |
Note
In case viewing angles (top & bottom)/dst-focal-length/dst-principal-point are not provided, the destination center of projection (Principal Point) and Focal Length are kept the same as the source.
- To increase the flexibility, num-batch-buffers is also a GstProperty. The plugin properties are parsed in order, so if num-batch-buffers is passed both attached to the GstElement in the gstreamer pipeline (eg. with
gst-launch-1.0
command), and in the configuration file, the latest configuration applied will be used (see example below). Example:
Config file (
config_dewarper.txt
) withnum-batch-buffers=2
.Command Line argument with property
num-batch-buffers=1
.
Pipelines:
gst-launch-1.0 … ! nvdewarper config-file=config_dewarper.txt num-batch-buffers=1 ! …
gst-launch-1.0 … ! nvdewarper num-batch-buffers=1 config-file=config_dewarper.txt ! …
Pipeline 1 will result in 1 surface per buffer whereas pipeline 2 will result in 2 surfaces per buffer.
This plugin can be tested with the one of the following pipelines.
For dGPU:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper.txt source-id=6 ! m.sink_0 nvstreammux name=m width=960 height=752 \ batch-size=4 num-surfaces-per-frame=4 ! nvmultistreamtiler ! nveglglessink
Perspective Projection use-case:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! m.sink_0 nvstreammux name=m width=3680 height=2428 \ batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nveglglessink
For Jetson:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper.txt source-id=6 ! m.sink_0 nvstreammux name=m width=960 height=752 batch-size=4 \ num-surfaces-per-frame=4 ! nvmultistreamtiler ! nv3dsink
Perspective Projection use-case:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! m.sink_0 nvstreammux name=m width=3680 height=2428 \ batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nv3dsink
Image File example dGPU:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.jpg ! nvjpegdec ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg
Image File example Jetson:
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.jpg ! nvjpegdec ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg
To better see the effect of cuda-address-mode you can change the dewarping size. For example, in config_dewarper_perspective.txt change [surface0] parameters to:
width=7360 height=4856 cuda-address-mode=1
The Gst-nvdewarper plugin always outputs a GStreamer buffer which contains the number of dewarped surfaces defined by num-batch-buffers (currently maximum of four surfaces are supported). These dewarped surfaces are scaled to the output resolution (output-width × output-height
) set in the configuration file.
Also, the num-surfaces-per-frame and batch-size of Gst-nvstreammux must be set accordingly. Batch-size must be a multiple of the number of dewarped surfaces, which should be set in num-surfaces-per-frame.
Gst Properties#
The following table describes the Gst-nvdewarper plugin’s Gst properties.
Property |
Meaning |
Type and Range |
Example and Notes |
---|---|---|---|
config-file |
Absolute pathname of configuration file for the Gst-nvdewarper element |
String |
config-file= configs/ deepstream-app/ config_dewarper.txt |
gpu-id |
Device ID of the GPU to be used (dGPU only) |
Integer, 0 to 4,294,967,295 |
source-id=6 |
source-id |
Source ID, e.g. camera ID |
Integer, 0 to 4,294,967,295 |
num-output-buffers=4 |
num-batch-buffers |
Number of dewarped output surfaces per buffer, i.e., batch-size of the buffer |
Integer, 0 to 4 |
num-batch-buffers=4 |
nvbuf-memory-type |
Type of memory to be allocated. For dGPU: 0 (nvbuf-mem-default): Default memory, cuda-device 1 (nvbuf-mem-cuda-pinned): Pinned/Host CUDA memory 2 (nvbuf-mem-cuda-device) Device CUDA memory 3 (nvbuf-mem-cuda-unified): Unified CUDA memory For Jetson: 0 (nvbuf-mem-default): Default memory, surface array 4 (nvbuf-mem-surface-array): Surface array memory |
Integer, 0 to 4 |
nvbuf-memory-type=3 |