.. _DS_plugin_gst-nvstreammux2: Gst-nvstreammux New ============================ The Gst-nvstreammux plugin forms a batch of frames from multiple input sources. When connecting a source to nvstreammux (the muxer), a new pad must be requested from the muxer using ``gst_element_get_request_pad()`` and the pad template ``sink_%u``. For more information, see ``link_element_to_streammux_sink_pad()`` in the DeepStream app source code. The muxer forms a batched buffer of batch-size frames. (batch-size is specified using the gst object property.) The muxer forwards the frames from that source as a part of the muxer’s output batched buffer. The frames are returned to the source when muxer gets back its output buffer. The muxer pushes the batch downstream when the batch is filled or the batch formation timeout calculated from the overall and stream specific “fps” control configuration keys in provided streammux config file is reached. The timeout starts running when the first buffer for a new batch is collected. The default overall max and min fps for batch generation are 120 and 5 respectively. The muxer’s default batching uses a round-robin algorithm to collect frames from the sources. It tries to collect an average of ( batch-size/num-source ) frames per batch from each source (if all sources are live and their frame rates are all the same). The number varies for each source, though, depending on the sources’ frame rates. The muxer attaches an ``NvDsBatchMeta`` metadata structure to the output batched buffer. This meta contains information about the frames copied into the batch (e.g. source ID of the frame, original resolutions of the input frames, original buffer PTS of the input frames). The source connected to the ``Sink_N`` pad will have ``pad_index`` ``N`` in ``NvDsBatchMeta``. The muxer supports addition and deletion of sources at run time. When the muxer receives a buffer from a new source, it sends a ``GST_NVEVENT_PAD_ADDED`` event. When a muxer sink pad is removed, the muxer sends a ``GST_NVEVENT_PAD_DELETED`` event. Both events contain the source ID of the source being added or removed (see ``sources/includes/gst-nvevent.h``). Downstream elements can reconfigure when they receive these events. Additionally, the muxer also sends a ``GST_NVEVENT_STREAM_EOS`` to indicate EOS from the source. The muxer supports calculation of NTP timestamps for source frames. It supports two modes. In the system timestamp mode, the muxer attaches the current system time as NTP timestamp. In the RTCP timestamp mode, the muxer uses RTCP Sender Report to calculate NTP timestamp of the frame when the frame was generated at source. The NTP timestamp is set in ``ntp_timestamp`` field of ``NvDsFrameMeta``. The mode can be toggled by setting the ``attach-sys-ts`` property. For more details, refer to :ref:`DS_NTP_Timestamp`. .. image:: ../content/DS_plugin_gst-nvstreammux2.png :align: center :alt: Gst-nvstreammux .. note:: The current nvsteammux shall be employed by default. Users will be able to use the new nvstreammux by setting the environment variable ``export USE_NEW_NVSTREAMMUX=yes``. New nvstreammux is no longer a beta feature. In upcoming DeepStream releases, usage of this environment variable and current nvstreammux will be deprecated to load new nvstreammux by default. Inputs and Outputs ----------------------- * Inputs * NV12/RGBA buffers from an arbitrary number of sources * mono S16LE/F32LE audio buffers from an arbitrary number of sources * Control Parameters * batch-size * config-file-path [config-keys detailed below] * num-surfaces-per-frame * attach-sys-ts * frame-duration * Output * NV12/RGBA batched video buffer ``NvBufSurface`` or batch audio buffer ``NvBufAudio`` * GstNvBatchMeta (meta containing information about individual frames in the batched buffer) Features ---------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvstreammux plugin features :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_features.csv :widths: 30, 30, 30 :header-rows: 1 Note: New nvstreammux do not scale batched buffers to a single resolution. A batch can have buffers from different streams of different resolutions. So with new mux, a single resolution for the batched buffer is invalid and the muxer's source-pad-caps is not valid either. .. _DS_plugin_gst_nvstreammux2_properties: Gst Properties ------------------ The following table describes the Gst-nvstreammux plugin’s Gst properties. .. csv-table:: Gst-nvstreammux gst-properties :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_gst-properties.csv :widths: 25, 25, 25, 25 :header-rows: 1 Differences between default and new streammux with respect to the GStreamer plugin properties are discussed in the table below: .. csv-table:: Gst-nvstreammux differences from default nvstreammux :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_differences.csv :widths: 30, 30 :header-rows: 1 .. _DS_plugin_gst_nvstreammux2_config_file_properties: Mux Config Properties --------------------- Details on Streammux config-file groups and keys are summarized the following table. .. csv-table:: Gst-nvstreammux config-file properties :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_cfg-file-properties.csv :widths: 30, 30, 30 :header-rows: 1 NvStreamMux Tuning Solutions for specific use cases ----------------------------------------------------- 1. Aim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1) nvstreammux provide many knobs to tune the way batching algorithm works. This is essential to support a wide range of applications/use-cases the muxer supports. More documentation is available at `Mux Config Properties`_ . 2) Tuning nvstreammux for specific use cases that we work with customers are good learning exercises. 3) Details discussed here include observations, the configs, pipeline changes, etc that worked well for specific use-cases. 4) Users/Contributors - Please feel free to create a `New forum Topic with the contribution here `_ 2. Important Tuning parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To ensure smooth streaming experience, configure/tune the below parameters properly. .. csv-table:: Gst-nvstreammux Tuning parameters :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_tuning_parametes.csv :widths: 30, 30 :header-rows: 1 3. Video + Audio muxing Use cases ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When nvstreammux is fed streams with different frame-rates, tuning is necessary to ensure standard muxer behavior. A sample pipeline diagram below illustrates the use of common components like nvstreammux, nvstreamdemux, flv/qtmux, etc. in a video + audio muxing use case for reference. .. image:: ../content/DS_plugin_gst-nvstreammux2_sample_av_pipeline.png :align: center :alt: Gst-nvstreammux AV pipeline When the same pipeline includes two nvstreammux modules to mux video and audio from different sources of different video framerate, depending on the type of sources, behavior could differ. Some of the scenarios and recommended tuning guidance are discussed below. 3.1. Video and Audio muxing; file sources of different fps ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In a single pipeline, we could have file sources with different video framerate, but same audio framerate (typical for most camera footages with reduced video framerate to save bandwidth while keeping the less heavy audio sampling rate intact). .. note:: 1. In this scenario, video buffers might get mux’d slower than audio buffers. When this happens `GstAggregator `_ based flvmux or qtmux could block the pipeline when the difference between video and audio buffer-timestamps are higher than the set "latency" parameter. 2. When dealing with file sources/ live sources of different framerates, we need nvstreammux tuned for min-overall-fps. Without this, the muxing always happens at the slowest stream's framerate adding latency to the video buffers. 3. When dealing with file sources of different frame rates and RTMP sources of different framerates, we recommend users to turn on sync-inputs=1 on nvstreammux and tune proper max-latency to ensure video and audio buffers from a single source are regulated and are flowing together in the pipeline after streammux. This is essential for the proper working of GstAggregator based muxers like flvmux, qtmux. etc. 4. To ensure smooth streaming experience, configure/tune the parameters discussed in Section `2. Important Tuning parameters`_ properly. 3.2 Video and Audio muxing; RTMP/RTSP sources ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When using live sources: 1) make sure that ``nvstreammux/sync-inputs`` is set to ``1``. 2) When using RTMP sources, in-built upstream latency query does not work. So you'll need to provide/tune a non-zero nvstreammux/max-latency setting. 3) Tune for nvstreammux/max-latency and other parameters as discussed in Section `2. Important Tuning parameters`_. 4. Troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4.1 GstAggregator plugin -> filesink does not write data into the file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To troubleshoot this issue, try increasing the GstAggregator based flvumx/qtmux "latency" setting. Try ``latency=18446744073709551614`` - the max value to see if it works and then you could tune for an optimal latency according to the type of media source in use. Also, set environment variable ``export GST_DEBUG=3`` for WARNING logs. Please also check Section `4.2 nvstreammux WARNING “Lot of buffers are being dropped”`_. 4.2 nvstreammux WARNING “Lot of buffers are being dropped” ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To troubleshoot this issue, try increasing the `max-latency` setting to allow late buffers. Also ensure to set min-overall-fps and max-overall-fps with the nvstreammux config file. 5. Metadata propagation through nvstreammux and nvstreamdemux ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For NvDsMeta propagation through nvstreammux and sample code, please refer to the deepstream reference application supplied at: ``/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-gst-metadata-test/`` Illustration of GstMeta at Input and Output of nvstreammux: .. image:: ../content/DS_plugin_gst-nvstreammux2_meta_prop_to_mux.png :align: center :alt: Input and Output at nvstreammux plugin (Illustration) Illustration of how GstMeta and NvDsMeta are copied as NvDsUserMeta on the batched buffer's NvDsBatchMeta after nvstreammux. Note: The same illustration holds good for the NvDsBatchMeta available on the demuxed GstBuffer after nvstreamdemux. Only difference is that the GstMeta won't be available as NvDsUserMeta anymore - and will be directly copied on to the demuxed GstBuffer. .. image:: ../content/DS_plugin_gst-nvstreammux2_meta_prop_in_nvdsbatchmeta.png :align: center :alt: Input and Output at nvstreammux plugin (Illustration) Illustration of GstMeta at Input and Output of nvstreammux: .. image:: ../content/DS_plugin_gst-nvstreammux2_meta_prop_from_demux.png :align: center :alt: Input and Output at nvstreamdemux plugin (Illustration) 5.1 Adding GstMeta to buffers before nvstreammux. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Users may add probes on nvstreammux sink pads and attach GstMeta to the GstBuffers flowing into nvstreammux. GstMeta attached on the GstBuffer pushed into nvstreammux sink pads will be copied and available: 1) After nvstreamdemux as GstMeta on the demuxed output GstBuffer. 2) After nvstreammux as NvDsUserMeta on the batched GstBuffer's NvDsBatchMeta->NvDsFrameMeta->user_meta_list. 5.2 Accessing GstMeta post nvstreammux. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The GstMeta on input GstBuffers at nvstreammux will be copied into output batch buffer’s NvDsBatchMeta. Reference code to dereference the NvDsBatchMeta on nvstreammux source pad with an attached GStreamer probe function or downstream plugin is available below: :: #include "gstnvdsmeta.h" static GstPadProbeReturn mux_src_side_probe_video (GstPad * pad, GstPadProbeInfo * info, gpointer u_data) { GstBuffer *buf = (GstBuffer *) info->data; NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf); if (batch_meta == nullptr) { /** Every buffer out of nvstreammux will have batch_meta */ return GST_PAD_PROBE_OK; } /** Now make sure NvDsBatchMeta->NvDsFrameMeta->user_meta_list * has the user meta with meta_type == NVDS_BUFFER_GST_AS_FRAME_USER_META */ for(GList* nodeFrame = batch_meta->frame_meta_list; nodeFrame; nodeFrame = g_list_next(nodeFrame)) { NvDsFrameMeta* frame_meta = static_cast(nodeFrame->data); //Uncomment below line when using nvstreammux to batch audio buffers //NvDsAudioFrameMeta* frame_meta = static_cast(nodeFrame->data); NvDsMetaList* l_user_meta; for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL; l_user_meta = l_user_meta->next) { NvDsUserMeta* user_meta = (NvDsUserMeta *) (l_user_meta->data); if(user_meta->base_meta.meta_type == NVDS_BUFFER_GST_AS_FRAME_USER_META) { /** dereference the empty GstBuffer with GstMeta copied */ GstBuffer* meta_buffer = (GstBuffer*)user_meta->user_meta_data; gpointer state = NULL; GstMeta *gst_meta = NULL; while ((gst_meta = gst_buffer_iterate_meta (meta_buffer, &state))) { /** * Note to users: Here, your GstMeta will be accessible as gst_meta. */ } } } } return GST_PAD_PROBE_OK; } 5.3 Adding GstMeta post nvstreammux. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The user could add GstMeta to each source's batched frame into the NvDsFrameMeta->user_meta_list corresponding to the source's frame. Please copy all GstMeta into a newly created empty GstBuffer and leverage the API's available at ``/opt/nvidia/deepstream/deepstream/sources/includes/gstnvdsmeta.h`` (``/opt/nvidia/deepstream/deepstream/lib/libnvdsgst_meta.so``) : 1) For video: ``nvds_copy_gst_meta_to_frame_meta()`` 2) For audio: ``nvds_copy_gst_meta_to_audio_frame_meta()`` 6. Cascaded Muxing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New streammux supports batching of batched buffers or cascaded muxers and appropriate debatching by demuxers for audio/video. Sample Pipelines mux1(batch-size 2) + mux2(batch-size2) > mux3 (batch-size4) mux1(batch-size 2) > mux1 (batch-size2) > demuxer Following table summarizes important notes for expected nvstreammux configuration for cascaded usecase: .. csv-table:: Gst-nvstreammux cascaded expected configuration :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_cascaded.csv :widths: 3, 20, 30 :header-rows: 1 Special nvmessage/EOS Handling Requirement in the application Note: Only the last nvstreammux instance in the pipeline will send GST_EVENT_EOS. GST_EVENT_EOS from upstream nvstreammux instances will be handled in the downstream nvstreammux instance and will not be forwarded. However, if the application utilize nvmessage EOS from nvstreammux, the application will have to make sure it received this message from all nvstreammux instances before tearing down the pipeline. The nvmessage discussed here is the GST_MESSAGE_ELEMENT event application receive on the bus callback ( API used to parse this message is: gst_nvmessage_is_stream_eos() and gst_nvmessage_parse_stream_eos()). Known Issues and FAQ --------------------- 1. Observing video and/or audio stutter (low framerate) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Solution:** You'll need to configure ``max-latency`` parameter on nvstreammux when stutters are observed or when pipeline latency is known to be "non-real-time". 2. Sink plugin shall not move asynchronously to PAUSED ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Solution:** When using new nvstreammux in a GStreamer pipeline, the sink elements shall be configured to set the plugin property `async` to `false`. 3. Heterogeneous batching ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New nvstreammux do not transform/scale batched buffers to a single color-format/resolution unlike the default nvstreammux. A batch can have buffers from different streams of different resolutions and formats. So with new mux, a single resolution for this heterogeneous batched buffer is invalid and the muxer’s source-pad-caps is not valid either. When we have plugins that could transform the input buffers (example: change resolution or color format of video buffers in the batch) between nvstreammux and nvstreamdemux, we need to ensure nvstreammux output is homogeneous (meaning buffers from all sources in the batch shall have same resolution and color format). This is a limitation in the new nvstreammux and shall be addressed in upcoming releases. **Solution:** In this scenario, DeepStream recommends adding nvvideoconvert + capsfiler before each nvstreammux sink pad (enforcing same resolution and format of all sources connecting to new nvstreammux). This ensure that the heterogeneous nvstreammux batch output have buffers of same caps (resolution and format). Example; video use-case: :: gst-launch-1.0 \ uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_0 \ uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_1 \ nvstreammux name=m batch-size=2 ! fakesink async=0 Where the fixed caps: "1920 X 1080; NV12" ensure every buffer in the batch is transformed to this same caps. Example; audio use-case: :: gst-launch-1.0 \ uridecodebin ! audioconvert ! audioresample ! “audio/x-raw, format=S16LE, layout=interleaved, channels=1, rate=48000” ! m.sink_0 \ uridecodebin ! audioconvert ! audioresample ! “audio/x-raw, format=S16LE, layout=interleaved, channels=1, rate=48000” ! m.sink_1 \ nvstreammux name=m batch-size=2 ! fakesink async=0 Where the fixed caps: "48kHz; mono S16LE interleaved" ensure every buffer in the batch is transformed to this same caps. 4. Adaptive Batching ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Does nvstreammux support dynamic batching ? This is in the context of use-cases where we don't know the exact number of inputs initially. Once the pipeline starts, inputs may get connected / disconnected. **Solution:** Yes, nvstreammux support dynamic batch-size when adaptive-batching=1,[property] group in the mux config-file. When adaptive-batching is enabled, batch-size is equal to the number of source pads on the muxer. By default this is enabled. Please refer to `Mux Config Properties`_ for more information. 5. Optimizing nvstreammux config for low-latency vs Compute ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You may want to design for use-cases where compute resource (GPU) utilization is more important than throughput. On the other hand, there could be use cases where minimum pipeline latency is of utmost importance. The following guidance is intended to help in tuning nvstreammux configuration parameters (passed with `config-file-path` property on nvstreammux) for optimal resource (compute) utilization and low-pipeline-latency. Recommended Config params are shared in the table below. .. csv-table:: Gst-nvstreammux Config file parameters for low-latency vs compute configs. :file: ../text/tables/Gst-nvstreammux tables/DS_Plugin_gst-nvstreammux2_lowlatency_vs_compute_parametes.csv :widths: 30, 30 :header-rows: 1 A sample pipeline diagram below illustrates the use of common OSS components like udpsrc, audiodecoder, audiobuffersplit, udpsink, and Nvidia components like nvstreammux, nvdsaudiotemplate, in an audio pipeline use case for reference. .. image:: ../content/DS_plugin_gst-nvstreammux2_sample_audio_pipeline.png :align: center :alt: Gst-nvstreammux AV pipeline 6. Latency Measurement API Usage guide for audio ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For latency measurement of video buffers, please refer to the usage of ``latency_measurement_buf_prob()`` probe function in the deepstream reference application implementation at ``/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app/deepstream_app.c``. Assume an audio pipeline viz: 32 X udpsrc ! rtpopusdepay ! opusdecode ! audiobuffersplit output-buffer-duration=1/50 ! queue ! mux.sink_%d nvstreammux name=mux ! queue ! nvdsaudiotemplate ! fakesink You may want to measure latency of each buffer from the moment its decoded until the time it reaches final sink plugin in the pipeline. In this example, the latency from `opusdecode` source pad (output) to fakesink sink pad (input). To do this, 1) Add a GStreamer buffer probe programmatically on opusdecode source pad following `documentation here `_ . 2) Inside the probe, call DeepStream API ``nvds_add_reference_timestamp_meta()`` at `/opt/nvidia/deepstream/deepstream/sources/includes/nvds_latency_meta.h` Pseudocode reference: :: static GstPadProbeReturn probe_on_audiodecoder_src_pad (GstPad * pad, GstPadProbeInfo * info, gpointer u_data) { GstBuffer *buf = (GstBuffer *) info->data; /* frame_id/frame_num is passed 0 and ignored here. * Its assigned and available in NvDsFrameMeta by nvstreammux; * Thus not required in this pipeline where nvstreammux is used. */ nvds_add_reference_timestamp_meta(buf, "audiodecoder", 0); return GST_PAD_PROBE_OK; } 3) Next, add a probe on the sink pad of fakesink following documentation `here `_ . Inside this probe, use ``API nvds_measure_buffer_latency()``. Pseudocode reference: :: static GstPadProbeReturn probe_on_fakesink_sink_pad (GstPad * pad, GstPadProbeInfo * info, gpointer u_data) { GstBuffer *buf = (GstBuffer *) info->data; GstMapInfo map_info = {0}; gboolean ok = gst_buffer_map(buf, &map_info, GST_MAP_READ); fail_unless(ok == TRUE); NvBufAudio* bufAudio = (NvBufAudio*)map_info.data; NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf); fail_unless(batch_meta != nullptr); gst_buffer_unmap(buf, &map_info); if(nvds_enable_latency_measurement) { NvDsFrameLatencyInfo* latency_info = (NvDsFrameLatencyInfo*)g_malloc0(sizeof(NvDsFrameLatencyInfo) * batch_meta->max_frames_in_batch); int num_sources_in_batch = nvds_measure_buffer_latency(buf, latency_info); for(int i = 0; i < num_sources_in_batch; i++) { /** Following are the details to profile */ g_print("Source id = %d Frame_num = %d Frame latency = %lf (ms) \n", latency_info[i].source_id, latency_info[i].frame_num, latency_info[i].latency); } } return GST_PAD_PROBE_OK; } .. note:: Latency Measurement relies on ``NvDsUserMeta`` that is added to ``NvDsBatchMeta`` for every batched buffer post ``nvstreammux``. This metadata and hence latency measurement support is available after an ``nvstreammux`` instance until ``nvstreamdemux`` instance in the GStreamer pipeline.