DeepStream Reference Application - deepstream-app

Application Architecture

The image below shows the architecture of the NVIDIA^® DeepStream reference application.

• The Stream Muxer plugin (Gst-nvstreammux) to form a batch of buffers from multiple input sources.
• The Preprocess plugin (Gst-nvdspreprocess) for preprocessing on the pre-defined ROIs for primary inferencing.
• The NVIDIA TensorRT™ based plugin (Gst-nvinfer) for primary and secondary (attribute classification of primary objects) detection and classification respectively.
• The Multi-Object Tracker plugin (Gst-nvtracker) for object tracking with unique ID.
• The Multi Stream Tiler plugin (Gst-nvmultistreamtiler) for forming 2D array of frames.
• The Onscreen Display (OSD) plugin (Gst-nvdsosd) to draw shaded boxes, rectangles and text on the composited frame using the generated metadata.
• The Message Converter (Gst-nvmsgconv) and Message Broker (Gst-nvmsgbroker) plugins in combination to send analytics data to a server in the Cloud.

Reference Application Configuration

The NVIDIA DeepStream SDK reference application uses one of the sample configuration files from the samples/configs/deepstream-app directory in the DeepStream package to:

• Enable or disable components
• Change the properties or behavior of components
• Customize other application configuration settings that are unrelated to the pipeline and its components

The configuration file uses a key file format, based on the freedesktop specifications at: https://specifications.freedesktop.org/desktop-entry-spec/latest

Expected Output for the DeepStream Reference Application (deepstream-app)

The image below shows the expected output with preprocess plugin disabled:

Configuration Groups

The application configuration is divided into groups of configurations for each component and application-specific component. The configuration groups are:

Configuration Groups - deepstream app

Application Group

The application group properties are:

Application group

Tiled-display Group

The tiled-display group properties are:

Tiled display group

Source Group

The source group specifies the source properties. The DeepStream application supports multiple simultaneous sources. For each source, a separate group with the group names such as source%d must be added to the configuration file. For example:

[source0]
key1=value1
key2=value2
...
[source1]
key1=value1
key2=value2
...

The source group properties are:

Source group

Source-list and source-attr-all Groups

The source-list group allows users to provide an initial list of source URI to start streaming with. This group along with [source-attr-all] can replace the need for separate [source] groups for each stream.

Also, DeepStream support (Alpha feature) dynamic sensor provisioning via REST API. For more details on this feature, please refer to the sample configuration file and deepstream reference application run command here.

For example:

[source-list]
num-source-bins=2
list=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4;file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h265.mp4
sgie-batch-size=8
...
[source-attr-all]
enable=1
type=3
num-sources=1
gpu-id=0
cudadec-memtype=0
latency=100
rtsp-reconnect-interval-sec=0
...

Note: [source-list] now support REST Server with use-nvmultiurisrcbin=1

For example:

[source-list]
num-source-bins=2
list=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4;file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h265.mp4
use-nvmultiurisrcbin=1
#sensor-id-list vector is one to one mapped with the uri-list
#identifies each sensor by a unique ID
sensor-id-list=UniqueSensorId1;UniqueSensorId2
#Optional sensor-name-list vector is one to one mapped with the uri-list
sensor-name-list=UniqueSensorName1;UniqueSensorName2
max-batch-size=10
http-ip=localhost
http-port=9000
#sgie batch size is number of sources * fair fraction of number of objects detected per frame per source
#the fair fraction of number of object detected is assumed to be 4
sgie-batch-size=40
[source-attr-all]
enable=1
type=3
num-sources=1
gpu-id=0
cudadec-memtype=0
latency=100
rtsp-reconnect-interval-sec=0

The [source-list] group properties are:

Source List group

The [source-attr-all] group support all properties except uri from the Source Group. A sample config file can be found at /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/configs/test5_config_file_nvmultiurisrcbin_src_list_attr_all.txt.

Streammux Group

The [streammux] group specifies and modifies properties of the Gst-nvstreammux plugin.

Streammux group

Preprocess Group

The [pre-process] group is for adding nvdspreprocess plugin in the pipeline. Supports preprocessing for only Primary GIE.

Preprocess group

Primary GIE and Secondary GIE Group

The DeepStream application supports multiple secondary GIEs. For each secondary GIE, a separate group with the name secondary-gie%d must be added to the configuration file. For example:

[primary-gie]
key1=value1
key2=value2
...
[secondary-gie1]
key1=value1
key2=value2
...

The primary and secondary GIE configurations are as follows. For each configuration, the Valid for column indicates whether the configuration property is valid for the primary or secondary TensorRT model, or for both models.

Primary and Secondary GIE group*

Tracker Group

The tracker group properties include the following, and more details can be found in Gst Properties:

Tracker group

Message Converter Group

Message converter group properties are:

Message converter group

Message Consumer Group

Message consumer group properties are:

Message consumer group

OSD Group

The OSD group specifies the properties and modifies the behavior of the OSD component, which overlays text and rectangles on the video frame.

OSD group

Sink Group

The sink group specifies the properties and modifies the behavior of the sink components for rendering, encoding, and file saving.

Sink group

Tests Group

The tests group is for diagnostics and debugging.

Tests group

NvDs-analytics Group

The [nvds-analytics] group is for adding nvds-analytics plugin in the pipeline.

NvDs-analytics group

Application Tuning for DeepStream SDK

This section provides application tuning tips for the DeepStream SDK using the following parameters in the configuration file.

Performance Optimization

This section covers various performance optimization steps that you can try for maximum performance.

DeepStream best practices

Here are few best practices to optimize DeepStream application to remove bottlenecks in your application:

1. Set the batch size of streammux and primary detector to equal the number of input sources. These settings are under [streammux] and [primary-gie] group of the config file. This keeps the pipeline running at full capacity. Higher or lower batch size than number of input sources can sometimes add latency in the pipeline.

2. Set the height and width of streammux to the input resolution. This is set under [streammux] group of the config file. This ensures that stream doesn’t go through any unwanted image scaling.

3. If you are streaming from live sources such as RTSP or from USB camera, set live-source=1 in [streammux] group of config file. This enables proper timestamping for live sources creating smoother playback

4. Tiling and visual output can take up GPU resource. There are 3 things that you can disable to maximize throughput when you do not need to render the output on your screen. As an example, rendering is not required when you want to run inference on the edge and transmit just the metadata to the cloud for further processing.

5. Disable OSD or on-screen display. OSD plugin is used for drawing bounding boxes and other artifacts and adding labels in the output frame. To disable OSD set enable=0 in the [osd] group of the config file.

6. The tiler creates an NxM grid for displaying the output streams. To disable the tiled output, set enable=0 in the [tiled-display] group of the config file.

7. Disable the output sink for rendering: choose fakesink, that is, type=1 in the [sink] group of the config file. All the performance benchmark in Performance section are ran with tiling, OSD and output sink disabled.

8. If CPU/GPU utilization is low, then one of the possibilities is that the elements in the pipeline are getting starved for buffers. Then try increasing the number of buffers allocated by the decoder by setting the num-extra-surfaces property of the [source#] group in the application or the num-extra-surfaces property of Gst-nvv4l2decoder element.

9. If you are running the application inside docker console and it delivers low FPS, set qos=0 in the configuration file’s [sink0] group. The issue is caused by initial load. When qos set to 1, as the property’s default value in the [sink0] group, decodebin starts dropping frames.

10. If you want to optimize processing pipelines end to end latency you can use latency measurement method in DeepStream.

• To enable frame latency measurement, run this command on the console:

$ export NVDS_ENABLE_LATENCY_MEASUREMENT=1

• To enable latency for all plugins, run this command on the console:

$ export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1

...
guint num_sources_in_batch = nvds_measure_buffer_latency(buf, latency_info);
if (num_sources_in_batch > 0 && latency_info[0].latency > 1e6) {
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  batch_meta->batch_user_meta_list = g_list_reverse (batch_meta->batch_user_meta_list);
  num_sources_in_batch = nvds_measure_buffer_latency(buf, latency_info);
}
...

Jetson optimization

1. Ensure that Jetson clocks are set high. Run these commands to set Jetson clocks high.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2. On Jetson, use Gst-nvdrmvideosink instead of Gst-nv3dsink as nv3dsink requires GPU utilization.

Triton

1. If you are using Triton with DeepStream, tune tf_gpu_memory_fraction values for TensorFlow GPU memory usage per process - suggested range [0.2, 0.6]. Too large value can cause Out-of-memory and too small may cause low perf.
2. Enable TensorRT optimization when using TensorFlow or ONNX with Triton. Update Triton config file to enable TensorFlow/ONNX TensorRT online optimization. This will take several minutes during initialization each time. Alternatively, you can generate TF-TRT graphdef/savedmodel models offline.

Inference Throughput

Here are a few steps to help you increase channel density for your application:

1. With DeepStream, users can infer every other frame or every third frame and use a tracker to predict the location in the object. This can be done with a simple config file change. Users can use one of the 3 available trackers to track the object in the frame. In the inference config file, change the interval parameter under [property]. This is a skip interval, number of frames to skip between inference. Interval of 0 means infer every frames and interval of 1 means skip 1 frame and infer every other frame. This can effectively double your overall channel throughput by going from interval of 0 to 1.
2. Choose lower precision such as FP16 or INT8 for inference. If you want to use FP16, no new model is required. This is a simple change in the DS. To change, update the network-mode option in the inference config file. If you want to run INT8, an INT8 calibration cache is required which contains the FP32 to INT8 quantization table.
3. DeepStream app can also be configured to have cascaded neural network. First network does the detection followed by second network with does some classification on the detection. To enable secondary inference, enable the secondary-gie from the config file. Set the appropriate batch sizes. Batch size will depend on number of objects that are typically sent to the secondary inference from primary inference. You’ll have to experiment to see what the appropriate batch size for their use case is. To reduce the number of inferences of the secondary classifier, the objects to infer on can be filtered by setting input-object-min-width, input-object-min-height, input-object-max-width, input-object-max-height, operate-on-gie-id, operate-on-class-ids appropriately.

Reducing Spurious Detections

Reducing spurious detections