Skip to navigation
DeepStream Plugin GuidePlugins

Gst-nvdewarper

View as Markdown

This plugin dewarps camera input. It accepts gpu-id and config-file as properties. Based on the selected configuration of surfaces, it can generate a maximum of four dewarped surfaces. It currently supports 18 dewarping projection types. Out of these, NVDS_META_SURFACE_FISH_PUSHBROOM and NVDS_META_SURFACE_FISH_VERTCYL are used in 360-D use case for dewarping 360° camera input. The plugin performs its function in these steps:

  1. Reads the configuration file and creates a vector of surface configurations. It supports a maximum of four dewarp surface configurations.
  2. Receives the frame from the decoder. Based on the configuration, generates up to four dewarped surfaces.
  3. Scales these surfaces down to network/selected dewarper output resolution using NPP APIs.
  4. Pushes a buffer containing the dewarped surfaces to the downstream component.

Gst-nvdewarper

Inputs and Outputs

  • Inputs

    • A buffer containing a frame in RGBA format

  • Control parameters

    • gpu-id; selects the GPU ID (dGPU only)
    • config-file, containing the pathname of the dewarper configuration file

  • Output

    • Dewarped RGBA surfaces
    • NvDewarperSurfaceMeta with information associated with each surface (projection_type, surface_index, and source_id), and the number of valid dewarped surfaces in the buffer (num_filled_surfaces)

Features

The following table summarizes the features of the plugin.

Gst-nvdewarper features

FeatureDescriptionRelease
Configure number of dewarped surfacesSupports a maximum of four dewarper surfaces.DS 3.0
Configure per-surface projection typeCurrently supports FishPushBroom and FishVertRadCyd projections.DS 3.0
Configure per-surface projection typePerspectivePerspective projectionsDS 5.0
Configure per-surface projection type18 projectionsDS 6.2
Configure per-surface indexSurface index to be set in case of multiple surfaces having same projection type.DS 3.0
Configure per-surface width and heightDS 3.0
Configure per-surface dewarping parametersPer surface configurable yaw, roll, pitch, top angle, bottom angle, and focal length and CUDA address mode dewarping parameters.
For PerspectivePerspective projection type, field of view, distortion and principal point can additionally be configured.		DS 3.0
Configurable dewarper output resolutionCreates a batch of up to four surfaces of a specified output resolution; internally scales all dewarper surfaces to output resolution.DS 3.0
Configurable NVDS CUDA memory type—DS 3.0
Multi-GPU support—DS 3.0
Aisle view CSV calibration file supportIf set, properties in the [surface<n>] group are ignored.
Only supported for FishPushBroom and FishVertRadCyd.		DS 3.0
Spot view CSV calibration file supportIf set, properties in the [surface<n>] group are ignored.
Only supported for FishPushBroom and FishVertRadCyd.		DS 3.0
Configure source idSets the source ID information in the NvDewarperSurfaceMeta.DS 4.0
Configurable number of output buffersNumber of allocated output dewarper buffers. Each buffer contains four dewarped output surfaces.DS 4.0
Configurable number of surfaces per output bufferNumber of dewarped output surfaces per buffer. Maximum allowed value is 4.DS 5.0
Support for advanced configuration parametersFisheye distortion coefficients, destination focal length, destination principal point, rotation axes sequence, rotation matrixDS 6.2

Configuration File Parameters

The configuration file located at configs/deepstream-app/config_dewarper.txt. The configuration file specifies per-surface configuration parameters in [surface<n>] groups, where <n> is an integer from 0 to 3, representing dewarped surfaces 0 to 3.

Gst-nvdewarper configuration file [surface<n>] parameters

PropertyProperty TypeMeaningType and RangeExample Notes
output-widthGlobalScale dewarped surfaces to specified output widthInteger, >0output-width=960
output-heightGlobalScale dewarped surfaces to specified output heightInteger, >0output-height=752
dewarp-dump-framesGlobalNumber of dewarped frames to dump.Integer, >0dewarp-dump-frames=10
num-batch-buffersGlobalNumber of dewarped output surfaces per buffer, i.e batch-size of the buffer.Integer,
0 to 4		num-batch-buffers=4
projection-typeSurfaceSelects projection type. Supported types are:
1: Fisheye to PushBroom
2: Fisheye to Vertical Radial Cylindrical
3: Perspective to Perspective
4: Fisheye to Perspective
5: Fisheye to Fisheye
6: Fisheye to Cylindrical
7: Fisheye to Equirectangular
8: Fisheye to Panini
9: Perspective to Equirectangular
10: Perspective to Panini
11: Equirectangular to Cylindrical
12: Equirectangular to Equirectangular
13: Equirectangular to Fisheye
14: Equirectangular to Panini
15: Equirectangular to Perspective
16: Equirectangular to PushBroom
17: Equirectangular to Stereographic
18: Equirectangular to Vertical Radial Cylindrical		Integer, 1 to 18projection-type=1
surface-indexSurfaceAn index that distinguishes surfaces of the same projection type.Integer, ≥0surface-index=0
widthSurfaceDewarped surface width.Integer, >0width=3886
heightSurfaceDewarped surface height.Integer, >0height=666
top-angleSurfaceTop field of view angle, in degrees.Float, −180.0 to 180.0top-angle=0
bottom-angleSurfaceBottom field of view angle, in degrees.Float, −180.0 to 180.0bottom-angle=0
pitchSurfaceViewing parameter pitch in degrees.Float, 0.0 to 360.0pitch=90
yawSurfaceViewing parameter yaw in degrees.Float, 0.0 to 360.0yaw=0
rollSurfaceViewing parameter roll in degrees.Float, 0.0 to 360.0roll=0
focal-lengthSurfaceFocal length of camera lens, in pixels per radian. Can be specified as :
fxy : single float value
<fx, fy> : array of two float values corresponding to focal lengths in x and y direction respectively		Floats, >0.0focal-length=437 OR
focal-length=442.63;395.106339
src-fovSurfaceSource vertical field of view in degrees. Used to compute the source focal-length if the latter is not provided.Float, >0.0src-fov=180
distortionSurfaceDistortion coefficients in the distortion polynomial for a fisheye/perspective source.
For perspective source :
Three radial coefficients:
(k_0,k_1,k_2): r^'=r(1+k_0 r^2+k_1 r^4+k_2 r^6 )
Two tangential coefficients:
(k_3,k_4):x'=x+[1+2k_3 xy+k_4 (r^2 + 2x^2)]
y^'=y+[ k_3 (r^2+2y^2+2k_4 xy)]
For fisheye source :
Four radial coefficients
(k_0,k_1,k_2, k_3): r^'=r(1+k_0 r^2+k_1 r^4+k_2 r^6+k_3 r^8 )		Array of 4/5 floatsdistortion=
-0.3721;0.1023;-0.00203;0;0
distortion=k_0,;k_1,;k_2;k_3;k_4
If not provided, all coefficients are set to 0 (no distortion)
src-x0SurfaceThe X coordinate of the center of projection of the source image, in pixel coordinates.
Typically, it is (srcWidth-1)/2, but may be different if the image is calibrated or wraps around.		Float, ≥0src-x0=479.5
If not provided, will take its default value =
(srcWidth-1)/2
src-y0SurfaceThe Y coordinate of the center of projection of the source image, in pixel coordinates.
Typically, it is (srcHeight-1)/2, but may be different if the image is calibrated or wraps around.		Float, ≥0src-y0=375.5
If not provided, will take its default value =
(srcHeight-1)/2
cuda-address-modeSurfaceCuda Texture addressing mode.
Currently supports:
0: cudaAddressModeClamp (Default) - sets points outside of border to nearest pixel value
1: cudaAddressModeBorder - sets points outside of border to transparent black		Integer, 0 or 1cuda-address-mode=
1
aisle-calibration-fileGlobalPathname of the configuration file for aisle view.
Set for the 360 D application only for PushBroom and/or VertRadCyl projection types.
If set, properties in the [surface<n>] group are ignored.
The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height.		Stringaisle-calibration-file=
csv_files/nvaisle_2M.csv
spot-calibration-fileGlobalPathname of the configuration file for spot view.
Set for the 360 D application only for PushBroom and/or VertRadCyl projection types.
If set, properties in the [surface<n>] group are ignored.
The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height.		Stringspot-calibration-file=
csv_files/nvspot_2M.csv
For an example of a spot view configuration file, see the file in the example above.
rot-axesSurfaceA three-character string that specifies the sequence of application of rotation angles.
Possible values are : XYZ, XZY, YXZ, YZX, ZXY, ZYX where :
X corresponds to pitch,
Y corresponds to yaw and
Z corresponds to roll.
Default sequence of rotation is YXZ		String of three charactersrot-axes=ZXY
controlSurfaceA control parameter to vary the amount of warp. Applicable for Panini, Stereographic and Pushbroom projections.Floatcontrol=1.0.
control=0 is identical to perspective output.
rot-matrixSurfaceConfigures the rotation as a 3X3 rotation matrix. The matrix is specified in a right-handed screen space, with X to the right, Y downwards and Z outwards.
If specified, this setting will override yaw, pitch, roll settings.		Array of 9 floatsrot-matrix=0.5;-0.86;0;0.86;0.5;0;0;0.0;1.0
dst-focal-lengthSurfaceSets destination image focal length.
Can be specified as :
fxy : single float value
<fx, fy> : array of two float values
If specified, this setting will override top-angle, bottom-angle settings.		Floats, >0.0dst-focal-length=350;316
dst-principal-pointSurfaceSet the X coordinate and Y coordinate in pixels of the principal point in destination image
If specified, this setting will override top-angle, bottom-angle settings.		Float, >0.0dst-principal-point=521.02;257.37

In case viewing angles (top & bottom)/dst-focal-length/dst-principal-point are not provided, the destination center of projection (Principal Point) and Focal Length are kept the same as the source.

To increase the flexibility, num-batch-buffers is also a GstProperty. The plugin properties are parsed in order, so if num-batch-buffers is passed both attached to the GstElement in the gstreamer pipeline (eg. with gst-launch-1.0 command), and in the configuration file, the latest configuration applied will be used (see example below). Example:

  • Config file (config_dewarper.txt) with num-batch-buffers=2.
  • Command Line argument with property num-batch-buffers=1.

Pipelines:

  • gst-launch-1.0 … ! nvdewarper config-file=config_dewarper.txt num-batch-buffers=1 ! …
  • gst-launch-1.0 … ! nvdewarper num-batch-buffers=1 config-file=config_dewarper.txt ! …

Pipeline 1 will result in 1 surface per buffer whereas pipeline 2 will result in 2 surfaces per buffer.

This plugin can be tested with the one of the following pipelines.

  • For dGPU:
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper.txt source-id=6 ! m.sink_0 nvstreammux name=m width=960 height=752 \
       batch-size=4 num-surfaces-per-frame=4 ! nvmultistreamtiler ! nveglglessink
Perspective Projection use-case: ::
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! m.sink_0 nvstreammux name=m width=3680 height=2428 \
       batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nveglglessink
  • For Jetson:
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder  ! nvvideoconvert  ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper.txt source-id=6  ! m.sink_0 nvstreammux name=m width=960 height=752 batch-size=4 \
       num-surfaces-per-frame=4 ! nvmultistreamtiler ! nv3dsink
Perspective Projection use-case: ::
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! m.sink_0 nvstreammux name=m width=3680 height=2428 \
       batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nv3dsink
*	Image File example dGPU: ::
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.jpg ! nvjpegdec ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg
*	Image File example Jetson: ::
       gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/yoga.jpg ! nvjpegdec ! nvvideoconvert ! nvdewarper config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg
To better see the effect of cuda-address-mode you can change the dewarping size. For example, in config_dewarper_perspective.txt change [surface0] parameters to: ::
       width=7360
       height=4856
       cuda-address-mode=1

The Gst-nvdewarper plugin always outputs a GStreamer buffer which contains the number of dewarped surfaces defined by num-batch-buffers (currently maximum of four surfaces are supported). These dewarped surfaces are scaled to the output resolution (output-width × output-height) set in the configuration file. Also, the num-surfaces-per-frame and batch-size of Gst-nvstreammux must be set accordingly. Batch-size must be a multiple of the number of dewarped surfaces, which should be set in num-surfaces-per-frame.

Gst Properties

The following table describes the Gst-nvdewarper plugin’s Gst properties.

Gst-nvdewarper gst properties

PropertyMeaningType and RangeExample and Notes
config-fileAbsolute pathname of configuration file for the Gst-nvdewarper elementStringconfig-file= configs/ deepstream-app/ config_dewarper.txt
gpu-idDevice ID of the GPU to be used (dGPU only)Integer,
0 to 4,294,967,295
source-idSource ID, e.g. camera IDInteger,
0 to 4,294,967,295		source-id=6
num-batch-buffersNumber of dewarped output surfaces per buffer, i.e., batch-size of the bufferInteger,
0 to 4		num-batch-buffers=4
num-output-buffersNumber of output buffersInteger,
0 to 4,294,967,295		num-output-buffers=4
silentSuppresses console output when setBooleansilent=1
dewarper-lib-version (read-only)Version of the dewarper library in useString
interpolation-methodInterpolation method used for scaling; see Gst-nvvideoconvert for the shared enum valuesEnuminterpolation-method=1
nvbuf-memory-typeType of memory to be allocated.
For dGPU:
0 (nvbuf-mem-default): Default memory, cuda-device
1 (nvbuf-mem-cuda-pinned): Pinned/Host CUDA memory
2 (nvbuf-mem-cuda-device) Device CUDA memory
3 (nvbuf-mem-cuda-unified): Unified CUDA memory
For Jetson:
0 (nvbuf-mem-default): Default memory, surface array
4 (nvbuf-mem-surface-array): Surface array memory		Integer, 0 to 4nvbuf-memory-type=3