Gst-nvdewarper

This plugin dewarps camera input. It accepts gpu-id and config-file as properties. Based on the selected configuration of surfaces, it can generate a maximum of four dewarped surfaces. It currently supports dewarping of three projection types, NVDS_META_SURFACE_FISH_PUSHBROOM, NVDS_META_SURFACE_FISH_VERTCYL and NVDS_META_SURFACE_PERSPECTIVE_PERSPECTIVE. The first two are used in 360-D use case for dewarping 360° camera input. The plugin performs its function in these steps:

  1. Reads the configuration file and creates a vector of surface configurations. It supports a maximum of four dewarp surface configurations.

  2. Receives the frame from the decoder. Based on the configuration, generates up to four dewarped surfaces.

  3. Scales these surfaces down to network/selected dewarper output resolution using NPP APIs.

  4. Pushes a buffer containing the dewarped surfaces to the downstream component.

Gst-nvdewarper

Inputs and Outputs

  • Inputs

    • A buffer containing a frame in RGBA format

  • Control parameters

    • gpu-id; selects the GPU ID (dGPU only)

    • config-file, containing the pathname of the dewarper configuration file

  • Output

    • Dewarped RGBA surfaces

    • NvDewarperSurfaceMeta with information associated with each surface (projection_type, surface_index, and source_id), and the number of valid dewarped surfaces in the buffer (num_filled_surfaces)

Features

The following table summarizes the features of the plugin.

Gst-nvdewarper features

Feature

Description

Release

Configure number of dewarped surfaces

Supports a maximum of four dewarper surfaces.

DS 3.0

Configure per-surface projection type

Currently supports FishPushBroom and FishVertRadCyd projections.

DS 3.0

Configure per-surface projection type

PerspectivePerspective projections

DS 5.0

Configure per-surface index

Surface index to be set in case of multiple surfaces having same projection type.

DS 3.0

Configure per-surface width and height

DS 3.0

Configure per-surface dewarping parameters

Per surface configurable yaw, roll, pitch, top angle, bottom angle, and focal length and CUDA address mode dewarping parameters. For PerspectivePerspective projection type, field of view, distortion and principal point can additionally be configured.

DS 3.0

Configurable dewarper output resolution

Creates a batch of up to four surfaces of a specified output resolution; internally scales all dewarper surfaces to output resolution.

DS 3.0

Configurable NVDS CUDA memory type

—

DS 3.0

Multi-GPU support

—

DS 3.0

Aisle view CSV calibration file support

If set, properties in the [surface<n>] group are ignored. Only supported for FishPushBroom and FishVertRadCyd.

DS 3.0

Spot view CSV calibration file support

If set, properties in the [surface<n>] group are ignored. Only supported for FishPushBroom and FishVertRadCyd.

DS 3.0

Configure source id

Sets the source ID information in the NvDewarperSurfaceMeta.

DS 4.0

Configurable number of output buffers

Number of allocated output dewarper buffers. Each buffer contains four dewarped output surfaces.

DS 4.0

Configurable number of surfaces per output buffer

Number of dewarped output surfaces per buffer. Maximum allowed value is 4.

DS 5.0

Configuration File Parameters

The configuration file located at configs/deepstream-app/config_dewarper.txt. The configuration file specifies per-surface configuration parameters in [surface<n>] groups, where <n> is an integer from 0 to 3, representing dewarped surfaces 0 to 3.

Gst-nvdewarper configuration file [surface<n>] parameters

Property

Property Type

Meaning

Type and Range

Example Notes

output-width

Global

Scale dewarped surfaces to specified output width

Integer, >0

output-width=960

output-height

Global

Scale dewarped surfaces to specified output height

Integer, >0

output-height=752

dewarp-dump-frames

Global

Number of dewarped frames to dump.

Integer, >0

dewarp-dump-frames=10

num-batch-buffers

Global

Number of dewarped output surfaces per buffer, i.e batch-size of the buffer.

Integer, 0 to 4

num-batch-buffers=4

projection-type

Global
Selects projection type. Supported types are:
1: PushBroom
2: VertRadCyl
3: PerspectivePerspective

Integer, 1 2 or 3

projection-type=1

surface-index

Surface

An index that distinguishes surfaces of the same projection type.

Integer, ≥0

surface-index=0

width

Surface

Dewarped surface width.

Integer, >0

width=3886

height

Surface

Dewarped surface height.

Integer, >0

height=666

top-angle

Surface

Top field of view angle, in degrees.

Float, −180.0 to 180.0

top-angle=0

bottom-angle

Surface

Bottom field of view angle, in degrees.

Float, −180.0 to 180.0

bottom-angle=0

pitch

Surface

Viewing parameter pitch in degrees.

Float, 0.0 to 360.0

pitch=90

yaw

Surface

Viewing parameter yaw in degrees.

Float, 0.0 to 360.0

yaw=0

roll

Surface

Viewing parameter roll in degrees.

Float, 0.0 to 360.0

roll=0

focal-length

Surface

Focal length of camera lens, in pixels per radian.

Float, >0.0

focal-length=437

src-fov

Surface

Source vertical field of view in degrees. Used to compute the source focal-length if the latter is not provided.

Only supported for PerspectivePerspective projection type.

Float, >0.0

src-fov=180

distortion

Surface

Distortion coefficients in the distortion polynomial for a perspective source:

Three radial coefficients (k_0,k_1,k_2): r^'=r(1+k_0 r^2+k_1 r^4+k_2 r^6 )

Two tangential coefficients (k_3,k_4): x'=x+[1+2k_3 xy+k_4 (r^2 + 2x^2)] y^'=y+[ k_3 (r^2+2y^2+2k_4 xy)]

Only supported for PerspectivePerspective projection type.

Array of floats

distortion=

-0.3721;0.1023;-0.00203;0;0

distortion=k_0,;k_1,;k_2;k_3;k_4

If not provided, all coefficients are set to 0 (no distortion)

src-x0

Surface

The X coordinate of the center of projection of the source image, in pixel coordinates.

Typically, it is (srcWidth-1)/2, but may be different if the image is calibrated or wraps around.

Only supported for PerspectivePerspective projection type.

Float, ≥0

src-x0=479.5

If not provided, will take its default value =

(srcWidth-1)/2

src-y0

Surface

The Y coordinate of the center of projection of the source image, in pixel coordinates.

Typically, it is (srcHeight-1)/2, but may be different if the image is calibrated or wraps around.

Only supported for PerspectivePerspective projection type

Float, ≥0

src-y0=375.5

If not provided, will take its default value =

(srcHeight-1)/2

cuda-address-mode

Surface

Cuda Texture addressing mode. Currently supports:

0: cudaAddressModeClamp (Default) - sets points outside of border to nearest pixel value

1: cudaAddressModeBorder - sets points outside of border to transparent black

Integer, 0 or 1

cuda-address-mode=

­1

aisle-calibration-file

Global

Pathname of the configuration file for aisle view.

Set for the 360 D application only for PushBroom and/or VertRadCyl projection types.

If set, properties in the [surface<n>] group are ignored. The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height.

String

aisle-calibration-file=

­csv_files/­nvaisle_2M.csv

spot-calibration-file

Global

Pathname of the configuration file for spot view.

Set for the 360 D application only for PushBroom and/or VertRadCyl projection types.

If set, properties in the [surface<n>] group are ignored. The configuration file is a CSV file with columns like sensorId and cameraId, and dewarping parameters like top-angle, bottom-angle, yaw, roll, pitch, focal-length, width, and height.

String

spot-calibration-file=

­csv_files/-nvspot_2M.csv

For an example of a spot view configuration file, see the file in the example above.

Note

For the Perspective projection type, in case viewing angles (top & bottom) are not provided, the destination center of projection (Principal Point) and Focal Length are kept the same as the source. To increase the flexibility, num-batch-buffers is also a GstProperty. The plugin properties are parsed in order, so if num-batch-buffers is passed both attached to the GstElement in the gstreamer pipeline (eg. with gst-launch-1.0 command), and in the configuration file, the latest configuration applied will be used (see example below). Example:

  • Config file (config_dewarper.txt) with num-batch-buffers=2.

  • Command Line argument with property num-batch-buffers=1.

Pipelines:

  • gst-launch-1.0 … ! nvdewarper config-file=config_dewarper.txt num-batch-buffers=1 ! …

  • gst-launch-1.0 … ! nvdewarper num-batch-buffers=1 config-file=config_dewarper.txt ! …

Pipeline 1 will result in 1 surface per buffer whereas pipeline 2 will result in 2 surfaces per buffer.

This plugin can be tested with the one of the following pipelines.

  • For dGPU:

    gst-launch-1.0 filesrc location=streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=config_dewarper.txt source-id=6 nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m width=960 height=752 \

batch-size=4 num-surfaces-per-frame=4 ! nvmultistreamtiler ! nveglglessink

    Perspective Projection use-case:

    gst-launch-1.0 filesrc location=streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=config_dewarper_perspective.txt nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m width=3680 height=2428 \

batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nveglglessink

  • For Jetson:

    gst-launch-1.0 filesrc location= streams/sample_cam6.mp4 ! qtdemux ! h264parse ! nvv4l2decoder  ! nvvideoconvert  ! nvdewarper config-file=config_dewarper.txt source-id=6  ! m.sink_0 nvstreammux name=m width=960 height=752 batch-size=4 \

num-surfaces-per-frame=4 ! nvmultistreamtiler ! nvegltransform ! nveglglessink

    Perspective Projection use-case:

    gst-launch-1.0 filesrc location=streams/yoga.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nvdewarper config-file=config_dewarper_perspective.txt nvbuf-memory-type=3 ! m.sink_0 nvstreammux name=m width=3680 height=2428 \

batch-size=1 num-surfaces-per-frame=1 ! nvmultistreamtiler ! nvegltransform  ! nveglglessink

    • Image File example dGPU:

      gst-launch-1.0 filesrc location=streams/yoga.jpg ! nvjpegdec ! nvdewarper config-file=config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg

    • Image File example Jetson:

      gst-launch-1.0 filesrc location=streams/yoga.jpg ! nvjpegdec ! nvvideoconvert ! nvdewarper config-file=config_dewarper_perspective.txt ! nvvideoconvert ! jpegenc ! filesink location=output.jpg

    To better see the effect of cuda-address-mode you can change the dewarping size. For example, in config_dewarper_perspective.txt change [surface0] parameters to:

    width=7360
height=4856
cuda-address-mode=1

The Gst-nvdewarper plugin always outputs a GStreamer buffer which contains the number of dewarped surfaces defined by num-batch-buffers (currently maximum of four surfaces are supported). These dewarped surfaces are scaled to the output resolution (output-width × output-height) set in the configuration file. Also, the num-surfaces-per-frame and batch-size of Gst-nvstreammux must be set accordingly. Batch-size must be a multiple of the number of dewarped surfaces, which should be set in num-surfaces-per-frame.

Gst Properties

The following table describes the Gst-nvdewarper plugin’s Gst properties.

Gst-nvdewarper gst properties

Property

Meaning

Type and Range

Example and Notes

config-file

Absolute pathname of configuration file for the Gst-nvdewarper element

String

config-file= configs/ deepstream-app/ config_dewarper.txt

gpu-id

Device ID of the GPU to be used (dGPU only)

Integer, 0 to 4,294,967,295

source-id=6

source-id

Source ID, e.g. camera ID

Integer, 0 to 4,294,967,295

num-output-buffers=4

num-batch-buffers

Number of dewarped output surfaces per buffer, i.e., batch-size of the buffer

Integer, 0 to 4

num-batch-buffers=4

nvbuf-memory-type

Type of memory to be allocated. For dGPU:

0 (nvbuf-mem-default): Default memory, cuda-device

1 (nvbuf-mem-cuda-pinned): Pinned/Host CUDA memory

2 (nvbuf-mem-cuda-device) Device CUDA memory

3 (nvbuf-mem-cuda-unified): Unified CUDA memory

For Jetson:

0 (nvbuf-mem-default): Default memory, surface array

4 (nvbuf-mem-surface-array): Surface array memory

Integer, 0 to 4

nvbuf-memory-type=3