Gst-nvstreammux New (Beta) ============================ The Gst-nvstreammux plugin forms a batch of frames from multiple input sources. When connecting a source to nvstreammux (the muxer), a new pad must be requested from the muxer using ``gst_element_get_request_pad()`` and the pad template ``sink_%u``. For more information, see ``link_element_to_streammux_sink_pad()`` in the DeepStream app source code. The muxer forms a batched buffer of batch-size frames. (batch-size is specified using the gst object property.) The muxer forwards the frames from that source as a part of the muxer’s output batched buffer. The frames are returned to the source when muxer gets back its output buffer. The muxer pushes the batch downstream when the batch is filled or the batch formation timeout calculated from the overall and stream specific “fps” control configuration keys in provided streammux config file is reached. The timeout starts running when the first buffer for a new batch is collected. The default overall max and min fps for batch generation are 120 and 5 respectively. The muxer’s default batching uses a round-robin algorithm to collect frames from the sources. It tries to collect an average of ( batch-size/num-source ) frames per batch from each source (if all sources are live and their frame rates are all the same). The number varies for each source, though, depending on the sources’ frame rates. The muxer attaches an ``NvDsBatchMeta`` metadata structure to the output batched buffer. This meta contains information about the frames copied into the batch (e.g. source ID of the frame, original resolutions of the input frames, original buffer PTS of the input frames). The source connected to the ``Sink_N`` pad will have ``pad_index`` ``N`` in ``NvDsBatchMeta``. The muxer supports addition and deletion of sources at run time. When the muxer receives a buffer from a new source, it sends a ``GST_NVEVENT_PAD_ADDED`` event. When a muxer sink pad is removed, the muxer sends a ``GST_NVEVENT_PAD_DELETED`` event. Both events contain the source ID of the source being added or removed (see ``sources/includes/gst-nvevent.h``). Downstream elements can reconfigure when they receive these events. Additionally, the muxer also sends a ``GST_NVEVENT_STREAM_EOS`` to indicate EOS from the source. The muxer supports calculation of NTP timestamps for source frames. It supports two modes. In the system timestamp mode, the muxer attaches the current system time as NTP timestamp. In the RTCP timestamp mode, the muxer uses RTCP Sender Report to calculate NTP timestamp of the frame when the frame was generated at source. The NTP timestamp is set in ``ntp_timestamp`` field of ``NvDsFrameMeta``. The mode can be toggled by setting the ``attach-sys-ts`` property. For more details, refer to :doc: `\DS_NTP_Timestamp` section NTP Timestamp in DeepStream. .. image:: ../content/DS_plugin_gst-nvstreammux2.png :align: center :alt: Gst-nvstreammux .. note:: New nvstreammux is being released as a beta feature (limited support). Users will be able to use the new streammux by setting the environment variable ``export USE_NEW_NVSTREAMMUX=yes``. The existing nvsteammux shall be employed by default. Inputs and Outputs ----------------------- * Inputs * NV12/RGBA buffers from an arbitrary number of sources * mono S16LE/F32LE audio buffers from an arbitrary number of sources * Control Parameters * batch-size * config-file-path [config-keys detailed below] * num-surfaces-per-frame * attach-sys-ts * Output * NV12/RGBA batched video buffer ``NvBufSurface`` or batch audio buffer ``NvBufAudio`` * GstNvBatchMeta (meta containing information about individual frames in the batched buffer) Features ---------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvstreammux plugin features :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_features.csv :widths: 30, 30, 30 :header-rows: 1 Note: New nvstreammux do not scale batched buffers to a single resolution. A batch can have buffers from different streams of different resolutions. So with new mux, a single resolution for the batched buffer is invalid and the muxer's source-pad-caps is not valid either. Gst Properties ------------------ The following table describes the Gst-nvstreammux plugin’s Gst properties. .. csv-table:: Gst-nvstreammux gst-properties :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_gst-properties.csv :widths: 25, 25, 25, 25 :header-rows: 1 Differences between default and new streammux with respect to the GStreamer plugin properties are discussed in the table below: .. csv-table:: Gst-nvstreammux differences from default nvstreammux :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_differences.csv :widths: 30, 30 :header-rows: 1 Mux Config Properties --------------------- Details on Streammux config-file groups and keys are summarized the following table. .. csv-table:: Gst-nvstreammux config-file properties :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_cfg-file-properties.csv :widths: 30, 30, 30 :header-rows: 1 NvStreamMux Tuning Solutions for specific usecases -------------------------------------------------- 1. Aim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1) nvstreammux provide many knobs to tune the way batching algorithm works. This is essential to support a wide range of applications/use-cases the muxer supports. More documentation is available at `Mux Config Properties`_ . 2) Tuning nvstreammux for specific usecases that we work with customers are good learning exercises. 3) Details discussed here include observations, the configs, pipeline changes, etc that worked well for specific use-cases. 4) Users/Contributors - Please feel free to create a `New forum Topic with the contribution here `_ 2. Important Tuning parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To ensure smooth streaming experience, configure/tune the below parameters properly. .. csv-table:: Gst-nvstreammux Tuning parameters :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_tuning_parametes.csv :widths: 30, 30 :header-rows: 1 3. Video + Audio muxing Usecases ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When nvstreammux is fed streams with different frame-rates, tuning is necessary to ensure standard muxer behavior. A sample pipeline diagram below illustrates the use of common components like nvstreammux, nvstreamdemux, flv/qtmux, etc. in a video + audio muxing use-case for reference. .. image:: ../content/DS_plugin_gst-nvstreammux2_sample_av_pipeline.png :align: center :alt: Gst-nvstreammux AV pipeline When the same pipeline includes two nvstreammux modules to mux video and audio from different sources of different video framerate, depending on the type of sources, behaviour could differ. Some of the scenarios and recommended tuning guidance are discussed below. 3.1 Video and Audio muxing; file sources of different fps ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In a single pipeline, we could have file sources with different video framerate, but same audio framerate (typical for most camera footages with reduced video framerate to save bandwidth while keeping the less heavy audio sampling rate intact). Note:  1) In this scenario, video buffers might get mux’d slower than audio buffers. When this happens `GstAggregator `_ based flvmux or qtmux could block the pipeline when the difference between video and audio buffer-timestamps are higher than the set "latency" parameter. 2)  When dealing with file sources/ live sources of different framerates, we need nvstreammux tuned for min-overall-fps. Without this, the muxing always happens at the slowest stream's framerate adding latency to the video buffers. 3) When dealing with file sources of different frame rates and RTMP sources of different framerates, we recommend users to turn on sync-inputs=1 on nvstreammux and tune proper max-latency to ensure video and audio buffers from a single source are regulated and are flowing together in the pipeline after streammux. This is essential for the proper working of GstAggregator based muxers like flvmux, qtmux. etc. 4) To ensure smooth streaming experience, configure/tune the parameters discussed in Section `2. Important Tuning parameters`_ properly. 3.2 Video and Audio muxing; RTMP/RTSP sources ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When using live sources: 1) make sure that nvstreammux/sync-inputs is set to 1. 2) When using RTMP sources, in-built upstream latency query does not work. Thus user need to provide/tune a non-zero nvstreammux/max-latency setting. 3) Tune for nvstreammux/max-latency and other parameters as discussed in Section `2. Important Tuning parameters`_. 4 Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4.1 GstAggregator plugin -> filesink does not write data into the file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Please try increasing the GstAggregator based flvumx/qtmux "latency" setting. Try "latency=18446744073709551614" - the max value to see if it works and then you could tune for an optimal latency according to the type of media source in use. Also, set environment variable "export GST_DEBUG=3" for WARNING logs. Please also check Section `4.2 nvstreammux WARNING “Lot of buffers are being dropped”`_. 4.2 nvstreammux WARNING “Lot of buffers are being dropped” ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Please try increasing the `max-latency` setting to allow late buffers. Please also ensure to set min-overall-fps and max-overall-fps with the nvstreammux config file. Known Issues and FAQ --------------------- 1. Observing video and/or audio stutter (low framerate) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Solution:** You'll need to configure ``max-latency`` parameter on nvstreammux when stutters are observed or when pipeline latency is known to be "non-realtime". 2. Sink plugin shall not move asynchronously to PAUSED ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Solution:** When using new nvstreammux in a GStreamer pipeline, the sink elements shall be configured to set the plugin property `async` to `false`. 3. Heterogeneous batching ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New nvstreammux do not transform/scale batched buffers to a single color-format/resolution unlike the default nvstreammux. A batch can have buffers from different streams of different resolutions and formats. So with new mux, a single resolution for this heterogeneous batched buffer is invalid and the muxer’s source-pad-caps is not valid either. When we have plugins that could transform the input buffers (example: change resolution or color format of video buffers in the batch) between nvstreammux and nvstreamdemux, we need to ensure nvstreammux output is homogeneous (meaning buffers from all sources in the batch shall have same resolution and color format). This is a limitation in the new nvstreammux and shall be addressed in upcoming releases. **Solution:** In this scenario, DeepStream recommends adding nvvideoconvert + capsfiler before each nvstreammux sink pad (enforcing same resolution and format of all sources connecting to new nvstreammux). This ensure that the heterogeneous nvstreammux batch output have buffers of same caps (resolution and format). Example; video use-case: :: gst-launch-1.0 \ uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_0 \ uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_1 \ nvstreammux name=m batch-size=2 ! fakesink async=0 Where the fixed caps: "1920 X 1080; NV12" ensure every buffer in the batch is transformed to this same caps. Example; audio use-case: :: gst-launch-1.0 \ uridecodebin ! audioconvert ! audioresample ! “audio/x-raw, format=S16LE, layout=interleaved, channels=1, rate=48000” ! m.sink_0 \ uridecodebin ! audioconvert ! audioresample ! “audio/x-raw, format=S16LE, layout=interleaved, channels=1, rate=48000” ! m.sink_1 \ nvstreammux name=m batch-size=2 ! fakesink async=0 Where the fixed caps: "48kHz; mono S16LE interleaved" ensure every buffer in the batch is transformed to this same caps. 4. Adaptive Batching ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ nvstreammux supports dynamic batching ? This is in the context of use-cases where we don't know the exact number of inputs initially. Once the pipeline starts, inputs may get connected / disconnected. **Answer:** Yes, nvstreammux support dynamic batch-size when adaptive-batching=1,[property] group in the mux config-file. When adaptive-batching is enabled, batch-size is equal to the number of source pads on the muxer. By default this is enabled. Please refer to `Mux Config Properties`_ for more information.