Gst-nvmultiurisrcbin (Alpha)

Note

The DeepStream 6.4 introduces nvmultiurisrcbin with REST API support for dynamic sensor add/remove capability. This is an Alpha feature.

This bin integrates three major DeepStream components viz. nvds_rest_server, nvurisrcbin, and nvstreammux (Gst-nvstreammux New / Gst-nvstreammux) into a single GstBin. For more details on nvurisrcbin, please run gst-inspect-1.0 nvurisrcbin on a machine/docker container with DeepStream installed.

The bin allows users to create simple gstreamer pipelines for AI multimedia analytics. Users can add or remove sensors over REST API calls made to a HTTP endpoint. A sensor is defined as a streaming entity with a valid unique Sensor ID, and URI.

The below High Level Overview diagram and table illustrate how nvmultiurisrcbin integrates nvds_rest_server, nvurisrcbin, and nvstreammux.

Gst-nvmultiurisrcbin High Level Overview
Gst-nvmultiurisrcbin High Level Overview of integrated components

Component Name

Source path in DeepStreamSDK Package for more info

  1. REST API Server; nvds_rest_server

/opt/nvidia/deepstream/deepstream/sources/libs/nvds_rest_server/

  1. Bin Manipulation API Library

/opt/nvidia/deepstream/deepstream/sources/libs/gstnvdscustomhelper/

  1. nvmultiurisrcbin

/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultiurisrcbin/

  1. deepstream-test5-app Demonstrating usage of nvmultiurisrcbin with nvmsgconv and nvmsgbroker

/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/

  1. nvurisrcbin

Source not available. For more info please run gst-inspect-1.0 nvurisrcbin on a machine/docker container with DeepStream installed.

  1. nvstreammux

Source not available. For more info please run gst-inspect-1.0 nvstreammux OR for new nvstreammux USE_NEW_NVSTREAMMUX=yes gst-inspect-1.0 nvstreammux

1. Introduction

This GstBin is a GStreamer source bin. The bin exposes only one source pad that provide batched buffers from an nvstreammux instance.

Users can add/remove streams to the bin for batching over the REST APIs defined in the Section REST API Payload definitions.

Users can also provide a static list of streams and their sensor IDs to start the pipeline with. This can be done using the Gst properties uri-list and sensor-id-list described below.

The nvmultiurisrcbin component output batched buffers from an nvstreammux instance.

Users can set environment variable USE_NEW_NVSTREAMMUX=yes to load the Gst-nvstreammux New instead of the default Gst-nvstreammux.

2. Features

The following table summarizes the features of the bin.

Gst-nvmultiurisrcbin features

Feature

Description

Release

nvmultiurisrcbin integrating REST API Server, nvurisrcbin, and nvstreammux

Introducing nvmultiurisrcbin

DS 6.2

REST API to add and remove sensors runtime

Dynamically add/remove sensors

DS 6.2

3. How to use nvmultiurisrcbin in a pipeline

Below pipeline is a sample gstreamer pipeline which is configured to accept upto 10 input sources and the pipline starts up with two sources:

gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=10 \
drop-pipeline-eos=1 live-source=1 \
uri-list=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4,file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 width=1920 height=1080 \
! nvmultistreamtiler ! nveglglessink

Below diagram illustrates a similar pipeline (with different plugins like nvinfer, nvmsgconv, and nvmsgbroker downstream).

Gst-nvmultiurisrcbin sample video pipeline

REST API payload definitions and sample curl commands for reference

Note:

Currently only few fields from the JSON schema described below are used/mandatory. The remaining fields are optional and ignored.

List of mandatory (and used) fields are:

  • value/camera_id

  • value/camera_url

  • value/change

ADD a new stream to a DeepStream pipeline

Note

DS-6.4 supports REST API version /api/v1. Refer below schema details.

Payload definition and the sample curl command:

1.curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
  "key": "sensor",
  "value": {
     "camera_id": "uniqueSensorID1",
     "camera_name": "front_door",
     "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
     "change": "camera_add",
     "metadata": {
         "resolution": "1920 x1080",
         "codec": "h264",
         "framerate": 30
     }
 },
 "headers": {
     "source": "vst",
     "created_at": "2021-06-01T14:34:13.417Z"
 }
}'

2.curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
 "key": "sensor",
 "event": {
     "camera_id": "uniqueSensorID1",
     "camera_name": "front_door",
     "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
     "change": "camera_streaming",
     "metadata": {
         "resolution": "1920 x1080",
         "codec": "h264",
         "framerate": 30
     }
 },
 "headers": {
     "source": "vst",
     "created_at": "2021-06-01T14:34:13.417Z"
 }
}'

REMOVE a new stream to a DeepStream pipeline

Payload definition and the sample curl command:

curl -XPOST 'http://localhost:9000/api/v1/stream/remove' -d '{
    "key": "sensor",
    "value": {
        "camera_id": "uniqueSensorID1",
        "camera_name": "front_door",
        "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
        "change": "camera_remove",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}'

4. Gst Properties

Gst Properties directly configuring nvmultiurisrcbin

Gst-nvmultiurisrcbin gstreamer properties directly configuring the bin

Property

Meaning

Type and Range

Example Notes

uri-list

comma separated URI list of sources; URI of the file or rtsp source

string

uri-list=file:///opt/file.mp4,file:///opt/file.mp4

sensor-id-list

comma separated list of unique source sensor IDs; this vector is one to one mapped with the uri-list

string

sensor-id-list=uniqueSensorID1,uniqueSensorID2

sensor-name-list

OPTIONAL comma separated list of source sensor names; this vector is one to one mapped with the uri-list

string

sensor-name-list=SensorName1,SensorName2

mode

Video-only or Audio-only modes available

0: for video-only 1: for audio-only

mode=0 (default)

ip-address

Set REST API HTTP IP Address

string

ip-address=localhost (default)

port

Set REST API HTTP Port number; Note: User may pass 0 to disable REST API Server

string

port=9000 (default)

max-batch-size

Set the maximum batch size to be used for nvstreammux; Maximum number of sources to be supported with this instance of nvmultiurisrcbin

Integer, 1 to 4,294,967,295

max-batch-size=30 (default)

Gst Properties to configure each instance of nvurisrcbin created inside this bin

Gst-nvmultiurisrcbin nvurisrcbin specific properties

Property

Meaning

Type and Range

Example Notes

num-extra-surfaces

Set extra decoder surfaces; Number of surfaces in addition to minimum decode surfaces given by the decoder

Integer, 1 to 4,294,967,295

num-extra-surfaces=1 (default)

gpu-id

Set GPU Device ID used by nvurisrcbin components like decoder (nvv4l2decoder) and nvvideoconvert

Integer, 1 to 4,294,967,295

gpu-id=0 (default)

cudadec-memtype

Memory type for cuda decoder buffers, Memory type for CUDA decoder buffers. Represented internally by enum CudaDecMemType. 0 (memtype_device): Device 1 (memtype_pinned): Host Pinned 2 (memtype_unified): Unified

Integer, 0, 1, or 2

cudadec-memtype=2 (default)

drop-frame-interval

Interval to drop the frames, e.g. a value of 5 means the decoder outputs every fifth frame, and others are dropped.

Integer, 1 to 30

Default: 0, dGPU / Jetson

dec-skip-frames

Type of frames to skip during decoding. Represented internally by enum SkipFrame. 0 (decode_all): decode all frames 1 (decode_non_ref): skips non-ref frames (Applicable only on Jetson platform) 2 (decode_key): decode key frames

Integer, 0, 1, or 2

skip-frames=0 Default: 0 ,dGPU / Jetson

select-rtp-protocol

Transport Protocol to use for RTP

Enum RtpProtocol Default: 0, rtp-multi (0): rtp-multi - UDP + UDP Multicast + TCP (4): rtp-tcp - TCP Only

select-rtp-protocol=0 (default)

file-loop

Loop file sources after EOS. Src type must be source-type-uri and uri starting with ‘file:/

boolean

file-loop=false (default)

rtsp-reconnect-interval

Timeout in seconds to wait since last data was received from an RTSP source before forcing a reconnection. 0=disable timeout

Integer, 1 to 4,294,967,295

rtsp-reconnect-interval=0

latency

Jitterbuffer size in milliseconds; applicable only for RTSP streams.

Integer, 1 to 4,294,967,295

latency=100 (default)

udp-buffer-size

UDP Buffer Size in bytes; applicable only for RTSP streams.

Integer, 1 to 4,294,967,295

udp-buffer-size=524288 (default)

smart-record

Enable Smart Record and choose the type of events to respond to. Sources must be of type source-type-rtsp

(0):smart-rec-disable - Disable Smart Record (1):smart-rec-cloud  - Trigger Smart Record through cloud messages only (2):smart-rec-multi  - Trigger Smart Record through cloud and local events

smart-record=0 (default)

smart-rec-dir-path

Path of directory to save the recorded file.

string

Absolute or relative path

smart-rec-file-prefix

By default, Smart_Record is the prefix. For unique file names every source must be provided with a unique prefix

string

Absolute or relative path

smart-rec-video-cache

Size of video cache in seconds. DEPRECATED: Use ‘smart-rec-cache’ instead

Integer, 1 to 4,294,967,295

smart-rec-video-cache=0 (default)

smart-rec-cache

Size of cache in seconds, applies to both audio and video cache

Integer, 1 to 4,294,967,295

smart-rec-cache=0 (default)

smart-rec-container

Container format of recorded video. MP4 and MKV containers are supported. Sources must be of type source-type-rtsp

(0): smart-rec-mp4 - MP4 container (1): smart-rec-mkv - MKV container

smart-rec-container=0 (default)

smart-rec-mode

Smart record mode

(0):smart-rec-mode-av - Record audio and video if available (1):smart-rec-mode-video - Record video only if available (2):smart-rec-mode-audio - Record audio only if available

smart-rec-mode=0 (default)

smart-rec-default-duration

In case a Stop event is not generated. This parameter will ensure the recording is stopped after a predefined default duration.

Integer, 1 to 4,294,967,295

smart-rec-default-duration=20 (default)

disable-passthrough

To disable the passhthrough mode of nvvideoconvert used inside the nvurisrcbin

boolean

disable-passhtorugh=false(default)

drop-pipeline-eos

Boolean property so that EOS is not propagated downstream when all source pads are at EOS.

boolean

drop-pipeline-eos=false(default)

Gst Properties to configure the instance of nvstreammux created inside this bin

All properties except “batch-size” can be configured using the same property names as defined for nvstreammux here and new nvstreammux2 here.

5. Important Notes

nvmultiurisrcbin config recommendations and notes on expected behavior

Gst-nvmultiurisrcbin recommendations

SL

Expected Behavior and Recommendation

1

nvstreammux, live-source property. live-source=1 live-source need to be turned ON to allow batching of available buffers when number of sources is < max-batch-size configuration.

2

REST API stream/add or stream/remove payload value of the change key must contain substring add or remove respectively. Expected Behavior: If ‘add’ is in the change field value, the sensorId and uri will be used to create a new stream. If ‘remove’ is in the change field value, the sensorId and uri will be used to find and remove the matching running stream.

3

Boolean property to inform muxer to skip sending EOS downstream is recommended to be set to 1. When using nvmultiurisrcbin drop-pipeline-eos=1 is preferred default to allow pipeline to stay alive for new sensor add requests after the last added sensor EOS. drop-pipeline-eos=1

6. Known Issues

Gst-nvmultiurisrcbin known issues

SL

Known issues

1

Usage of new nvstreammux for video and audio usecases are not supported at this time and might not work.