NVIDIA Tegra
NVIDIA DeepStream Plugin Manual

Application Note
4.0.1 Release


 
Gst-nvstreammux
 
Inputs and Outputs
Features
Gst Properties
The Gst-nvstreammux plugin forms a batch of frames from multiple input sources. When connecting a source to nvstreammux (the muxer), a new pad must be requested from the muxer using gst_element_get_request_pad() and the pad template "sink_%u". For more information, see link_element_to_streammux_sink_pad() in the DeepStream app source code.
The muxer forms a batched buffer of batch-size frames. (batch-size is specified using the gst object property.)
If the muxer’s output format and input format are the same, the muxer forwards the frames from that source as a part of the muxer’s output batched buffer. The frames are returned to the source when muxer gets back its output buffer. If the resolution is not the same, the muxer scales frames from the input into the batched buffer and then returns the input buffers to the upstream component.
The muxer pushes the batch downstream when the batch is filled or the batch formation timeout batched-pushed-timeout is reached. The timeout starts running when the first buffer for a new batch is collected.
The muxer uses a round-robin algorithm to collect frames from the sources. It tries to collect an average of ( batch-size/num-source ) frames per batch from each source (if all sources are live and their frame rates are all the same). The number varies for each source, though, depending on the sources’ frame rates.
The muxer outputs a single resolution (i.e. all frames in the batch have the same resolution). This resolution can be specified using the width and height properties. The muxer scales all input frames to this resolution. The enable-padding property can be set to true to preserve the input aspect ratio while scaling by padding with black bands.
For DGPU platforms, the GPU to use for scaling and memory allocations can be specified with the gpu-id property.
For each source that needs scaling to the muxer’s output resolution, the muxer creates a buffer pool and allocates four buffers each of size:
Where f is 1.5 for NV12 format, or 4.0 for RGBA. The memory type is determined by the nvbuf-memory-type property.
Set the live-source property to true to inform the muxer that the sources are live. In this case the muxer attaches the PTS of the last copied input buffer to the batched Gst Buffer’s PTS. If the property is set to false, the muxer calculates timestamps based on the frame rate of the source which first negotiates capabilities with the muxer.
The muxer attaches an NvDsBatchMeta metadata structure to the output batched buffer. This meta contains information about the frames copied into the batch (e.g. source ID of the frame, original resolutions of the input frames, original buffer PTS of the input frames). The source connected to the Sink_N pad will have pad_index N in NvDsBatchMeta.
The muxer supports addition and deletion of sources at run time. When the muxer receives a buffer from a new source, it sends a GST_NVEVENT_PAD_ADDED event. When a muxer sink pad is removed, the muxer sends a GST_NVEVENT_PAD_DELETED event. Both events contains the source ID of the source being added or removed (see sources/includes/gst-nvevent.h). Downstream elements can reconfigure when they receive these events. Additionally, the muxer also sends a GST_NVEVENT_STREAM_EOS to indicate EOS from the source.
A picture containing screenshot Description automatically generated
Inputs and Outputs
Inputs
NV12/RGBA buffers from an arbitrary number of sources
Control Parameters
batch-size
batched-push-timeout
width
height
enable-padding
gpu-id (dGPU only)
live-source
nvbuf-memory-type
Output
NV12/RGBA batched buffer
GstNvBatchMeta (meta containing information about individual frames in the batched buffer)
Features
The following table summarizes the features of the plugin.
Features of the Gst-nvstreammux plugin
Feature
Description
Release
Configurable batch size
DS 2.0
Configurable batching timeout
DS 2.0
Allows multiple input streams with different resolutions
DS 2.0
Allows multiple input streams with different frame rates
DS 2.0
Scales to user-determined resolution in muxer
DS 2.0
Scales while maintaining aspect ratio with padding
DS 2.0
Multi-GPU support
DS 2.0
Input stream DRC support
DS 3.0
User-configurable CUDA memory type (Pinned/Device/Unified) for output buffers
DS 3.0
Custom message to inform application of EOS from individual sources
DS 3.0
Supports adding and deleting run time sinkpads (input sources) and sending custom events to notify downstream components
DS 3.0
Supports RGBA data handling at output
DS 3.0
Gst Properties
The following table describes the Gst-nvstreammux plugin’s Gst properties.
Gst-nvstreammux plugin, Gst properties
Property
Meaning
Type and Range
Example
Notes
batch-size
Maximum number of frames in a batch.
Integer,
0 to 4,294,967,295
batch-size=30
batched-push-timeout
Timeout in microseconds to wait after the first buffer is available to push the batch even if a complete batch is not formed.
Signed integer,
-1 to 2,147,483,647
batched-push-timeout= 40000
40 msec
width
If non-zero, muxer scales input frames to this width.
Integer,
0 to 4,294,967,295
width=1280
height
If non-zero, muxer scales input frames to this height.
Integer,
0 to 4,294,967,295
height=720
enable-padding
Maintains aspect ratio by padding with black borders when scaling input frames.
Boolean
enable-padding=1
gpu-id
ID of the GPU on which to allocate device or unified memory to be used for copying or scaling buffers. (dGPU only.)
Integer,
0 to 4,294,967,295
gpu-id=1
live-source
Indicates to muxer that sources are live, e.g. live feeds like an RTSP or USB camera.
Boolean
live-source=1
nvbuf-memory-type
Type of memory to be allocated.
For dGPU:
0 (nvbuf-mem-default): Default memory, cuda-device
1 (nvbuf-mem-cuda-pinned): Pinned/Host CUDA memory
2 (nvbuf-mem-cuda-device) Device CUDA memory
3 (nvbuf-mem-cuda-unified): Unified CUDA memory
For Jetson:
0 (nvbuf-mem-default): Default memory, surface array
4 (nvbuf-mem-surface-array): Surface array memory
Integer, 0-4
nvbuf-memory-type=1