GStreamer Plugin Overview

DeepStream SDK is based on the GStreamer framework. This section describes the DeepStream GStreamer plugins and the DeepStream input, outputs, and control parameters. DeepStream SDK is supported on systems that contain an NVIDIA^® Jetson™ module or an NVIDIA dGPU adapter [1]. The manual is intended for engineers who want to develop DeepStream applications or additional plugins using the DeepStream SDK. It also contains information about metadata used in the SDK. Developers can add custom metadata as well. The manual describes the methods defined in the SDK for implementing custom inferencing layers using the IPlugin interface of NVIDIA^® TensorRT™. You can refer the sample examples shipped with the SDK as you use this manual to familiarize yourself with DeepStream application and plugin development.

• MetaData in the DeepStream SDK
  • NvDsBatchMeta: Basic Metadata Structure
  • User/Custom Metadata Addition inside NvDsBatchMeta
  • Adding Custom Meta in Gst Plugins Upstream from Gst-nvstreammux
    • Adding metadata to the plugin before Gst-nvstreammux
    • New metadata fields
• Gst-nvdspreprocess (Alpha)
  • Inputs and Outputs
  • Features
  • Custom library Interfaces
  • Gst-nvdspreprocess File Configuration Specifications
  • Gst Properties
  • Sample pipelines
• Gst-nvinfer
  • Inputs and Outputs
  • Features
  • Gst-nvinfer File Configuration Specifications
  • Gst Properties
  • Clustering algorithms supported by nvinfer
    • cluster-mode = 0 | GroupRectangles
    • cluster-mode = 1 | DBSCAN
    • cluster-mode = 2 | NMS
    • cluster-mode = 3 | Hybrid
    • cluster-mode=4 | No clustering
  • Tensor Metadata
    • To read or parse inference raw tensor data of output layers
  • Segmentation Metadata
• Gst-nvinferserver
  • Inputs and Outputs
  • Gst-nvinferserver Configuration File Specifications
  • Features
    • Low Level libnvds_infer_server.so Configuration File Specifications
  • Gst Properties
  • DeepStream Triton samples
  • DeepStream Triton gRPC support
  • Triton Ensemble Models
  • Custom Process interface IInferCustomProcessor for Extra Input, LSTM Loop, Output Tensor Postprocess
  • Tensor Metadata Output for Downstream Plugins
    • To read or parse inference raw tensor data of output layers
  • Segmentation Metadata
• Gst-nvtracker
  • Sub-batching (Alpha)
  • Inputs and Outputs
  • Gst Properties
  • NvDsTracker API for Low-Level Tracker Library
  • NvMultiObjectTracker : A Reference Low-Level Tracker Library
  • Unified Tracker Architecture for Composable Multi-Object Tracker
  • Workflow and Core Modules in The NvMultiObjectTracker Library
    • Data Association
    • Target Management and Error Handling
    • State Estimation
    • Object Re-Identification
    • Target Re-Association
    • Bounding-box Unclipping
    • Single-View 3D Tracking (Alpha)
    • Configuration Parameters
  • Low-Level Tracker Comparisons and Tradeoffs
  • IOU Tracker
  • NvSORT Tracker
  • NvDeepSORT Tracker
    • Data Association
    • Configuration Parameters
    • Implementation Details and Reference
  • NvDCF Tracker
    • Visual Tracking
    • Data Association
    • Configuration Parameters
  • Miscellaneous Data Output
    • Terminated Track List
    • Shadow Tracking Target Data
    • Past-frame Target Data
  • Setup and Usage of Re-ID Model
    • Setup Sample Re-ID Models
    • Customize Re-ID Model
    • Re-ID Feature Output
  • Setup and Usage of Sub-batching (Alpha)
    • Use-case 1
    • Use-case 2
  • Setup and Visualization of Tracker Sample Pipelines
    • People Tracking
    • Vehicle Tracking
  • Tracker Accuracy Tuning
  • How to Implement a Custom Low-Level Tracker Library
• Gst-nvstreammux
  • Inputs and Outputs
  • Features
  • Gst Properties
  • Known Issues with Solutions and FAQ
    • 1. gst-inspect is not updated properly when switching between legacy and new streammux
• Gst-nvstreammux New
  • Inputs and Outputs
  • Features
  • Gst Properties
  • Mux Config Properties
  • NvStreamMux Tuning Solutions for specific use cases
    • Aim
    • Important Tuning parameters
    • Video and Audio muxing Use cases
    • Troubleshooting
    • Metadata propagation through nvstreammux and nvstreamdemux
    • Cascaded Muxing
  • Known Issues with Solutions and FAQ
    • Observing video and/or audio stutter (low framerate)
    • Sink plugin shall not move asynchronously to PAUSED
    • Heterogeneous batching
    • Adaptive Batching
    • Optimizing nvstreammux config for low-latency vs Compute
    • Latency Measurement API Usage guide for audio
    • gst-inspect is not updated properly when switching between legacy and new streammux
• Gst-nvstreamdemux
  • Inputs and Outputs
    • Use case 1
    • Use case 2
    • Use case 3
• Gst-nvmultistreamtiler
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvdsosd
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvdsmetautils
  • Gst-nvdsmetainsert
  • Inputs and Outputs
  • Features
  • customlib_impl Interfaces
  • Gst Properties
  • Use Cases & Sample Pipelines
  • Gst-nvdsmetaextract
  • Inputs and Outputs
  • Features
  • customlib_impl Interfaces
  • Gst Properties
  • Use Cases & Sample Pipelines
• Gst-nvdsvideotemplate
  • Inputs and Outputs
  • Features
  • customlib_impl Interfaces
  • Gst Properties
  • Sample pipelines
• Gst-nvdsaudiotemplate
  • Inputs and Outputs
  • Features
  • customlib_impl Interfaces
  • Gst Properties
  • Sample pipelines
• Gst-nvvideoconvert
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvdewarper
  • Inputs and Outputs
  • Features
  • Configuration File Parameters
  • Gst Properties
• Gst-nvof
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvofvisual
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvsegvisual
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvvideo4linux2
  • Decoder
    • Inputs and Outputs
    • Features
    • Gst Properties
  • Encoder
    • Inputs and Outputs
    • Features
    • Gst Properties
• Gst-nvjpegdec
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvimagedec
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvjpegenc
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvimageenc
  • Inputs and Outputs
  • Features
  • Gst Properties
• Gst-nvmsgconv
  • Inputs and Outputs
  • Features
  • Gst Properties
  • NvMsgConv low level library API’s
  • Schema Customization
  • Payload with Custom Objects
• Gst-nvmsgbroker
  • Inputs and Outputs
  • Features
  • Gst Properties
  • nvds_msgapi: Protocol Adapter Interface
    • nvds_msgapi_connect(): Create a Connection
    • nvds_msgapi_send() and nvds_msgapi_send_async(): Send an event
    • nvds_msgapi_subscribe(): Consume data by subscribing to topics
    • nvds_msgapi_do_work(): Incremental Execution of Adapter Logic
    • nvds_msgapi_disconnect(): Terminate a Connection
    • nvds_msgapi_getversion(): Get Version Number
    • nvds_msgapi_get_protocol_name(): Get name of the protocol
    • nvds_msgapi_connection_signature(): Get Connection signature
  • nvds_kafka_proto: Kafka Protocol Adapter
    • Installing Dependencies
    • Using the Adapter
    • Configuring Protocol Settings
    • Programmatic Integration
    • Security for Kafka
  • Azure MQTT Protocol Adapter Libraries
    • Installing Dependencies
    • Setting Up Azure IoT
    • Configuring Adapter Settings
    • Using the Adapter
    • Monitor Adapter Execution
    • Message Topics and Routes
  • AMQP Protocol Adapter
    • Installing Dependencies
    • AMQP broker
    • Configure Adapter Settings
    • Using the Adapter
    • User authentication
    • Programmatic Integration
    • Monitor Adapter Execution
  • REDIS Protocol Adapter
    • Installing Dependencies
    • REDIS server
    • Configure Adapter Settings
    • User Authentication
    • Using the Adapter
    • Programmatic Integration
    • Monitor Adapter Execution
  • MQTT Protocol Adapter
    • Installing Dependencies
    • Eclipse Mosquitto
    • Configure Adapter Settings
    • Using the Adapter
    • Programmatic Integration
    • Monitor Adapter Execution
  • nv_msgbroker: Message Broker interface
    • nv_msgbroker_connect(): Create a Connection
    • nv_msgbroker_send_async(): Send an event asynchronously
    • nv_msgbroker_subscribe(): Consume data by subscribing to topics
    • nv_msgbroker_disconnect(): Terminate a Connection
    • nv_msgbroker_version(): Get Version Number
    • Autoreconnect feature
    • Work-interval configuration
    • nvds_logger: Logging Framework
• Gst-nvdsanalytics
  • Inputs and Outputs
  • Features
  • Gst Properties
  • Configuration File Parameters
• Gst-nvdsudpsrc
  • Inputs and Outputs
  • Features
  • Gst Properties
  • Example pipelines
• Gst-nvdsudpsink
  • Inputs and Outputs
  • Features
  • Gst Properties
  • Example pipelines
• Gst-nvdspostprocess (Alpha)
  • Inputs and Outputs
  • Features
  • Gst-nvdspostprocess File Configuration Specifications
  • Gst Properties
  • Sample pipelines
• Gst-nvds3dfilter
  • Custom library Interfaces
  • Inputs and Outputs
  • Configuration file
  • Gst-nvds3dfilter properties Specifications
  • Supported DS3D filter custom libs
  • Custom ds3d::datafilter library: libnvds_3d_alignment_datafilter.so
    • Inputs and Outputs
    • Configuration file
    • Configuration options
  • Custom ds3d::datafilter library: libnvds_3d_depth2point_datafilter.so
• Gst-nvds3dbridge
  • Gst-nvds3dbridge properties Specifications
  • Custom library Interfaces
  • Supported DS3D Data bridges
  • Custom low level library: libnvds_3d_video_databridge.so
    • Inputs and Outputs
    • Configuration file
    • Video bridge configuration options
• Gst-nvds3dmixer
  • Gst-nvds3dmixer properties Specifications
  • Custom library Interfaces
  • Supported DS3D Data mixers
  • Custom low level library: libnvds_3d_multisensor_mixer.so
    • Inputs and Outputs
    • Configuration file
    • Video mixer configuration options
• Gst-NvDsUcx
  • Description
  • Requirements
  • Inputs and Outputs
  • Gst Properties
  • Examples
• Gst-nvdsxfer
  • Inputs and Outputs
  • Gst Properties
  • How to test
  • Use cases
    • Single Stream + Multi-dGPUs Setup
    • Multiple Streams + Multi-dGPU Setup
• Gst-nvvideotestsrc
  • Outputs
  • Gst Properties
  • How to test
• Gst-nvmultiurisrcbin
  • 1. Introduction
  • 2. Features
  • 3. How to use nvmultiurisrcbin in a pipeline
    • REST API payload definitions and sample curl commands for reference
  • 4. Gst Properties
    • Gst Properties directly configuring nvmultiurisrcbin
    • Gst Properties to configure each instance of nvurisrcbin created inside this bin
    • Gst Properties to configure the instance of nvstreammux created inside this bin
  • 5. Important Notes
    • nvmultiurisrcbin config recommendations and notes on expected behavior
  • 6. Limitations
• Gst-nvurisrcbin
  • 1. Introduction
  • 2. How to use nvurisrcbin in a pipeline
  • 3. Gst Properties
    • 3.1 Gst Properties to configure nvurisrcbin

Footnotes

[1]

This manual uses the term dGPU (“discrete GPU”) to refer to NVIDIA GPU expansion card products such as NVIDIA® Tesla® T4, NVIDIA® Ampere and NVIDIA® GeForce® RTX 2080. This version of DeepStream SDK runs on specific dGPU products on x86_64 platforms supported by NVIDIA driver 535+ and NVIDIA® TensorRT™ 10.3.0.26 and later versions.