Troubleshooting¶

If you run into to trouble while using DeepStream, consider the following solutions. if you don’t find answers below, post your questions on DeepStream developer forum

You are migrating from DeepStream 4.0+ to DeepStream 5.0¶

Solution:

You must clean up the DeepStream 4.0 libraries and binaries. Use one of these commands to clean up: * For dGPU:

To remove DeepStream 4.0 or later installations:

Open the uninstall.sh file in /opt/nvidia/deepstream/deepstream/

Set PREV_DS_VER as 4.0

Run the script as sudo ./uninstall.sh

For Jetson: Flash the target device with the latest release of JetPack.

“NvDsBatchMeta not found for input buffer” error while running DeepStream pipeline¶

Solution:

The Gst-nvstreammux plugin is not in the pipeline. Starting with DeepStream 4.0, Gst-nvstreammux is a required plugin. This is an example pipeline:

Gst nvv4l2decoder --> Gst nvstreammux --> Gst nvinfer --> Gst nvtracker --> Gst nvmultistreamtiler --> Gst nvvideoconvert --> Gst nvosd --> Gst nveglglessink

The DeepStream reference application fails to launch, or any plugin fails to load¶

Solution:

Try clearing the GStreamer cache by running the command:

$ rm -rf ${HOME}/.cache/gstreamer-1.0

Also run this command if there is an issue with loading any of the plugins. Warnings or errors for failing plugins are displayed on the terminal.

$ gst-inspect-1.0

Then run this command to find missing dependencies:

$ ldd <plugin>.so

where <plugin> is the name of the plugin that failed to load.

Application fails to run when the neural network is changed¶

Solution:

Make sure that the network parameters are updated for the corresponding [GIE] group in the configuration file (e.g. source30_720p_dec_infer-resnet_tiled_display_int8.txt). Also make sure that the Gst-nvinfer plugin’s configuration file is updated accordingly. When the model is changed, make sure that the application is not using old engine files.

The DeepStream application is running slowly (Jetson only)¶

Solution:

Ensure that Jetson clocks are set high. Run these commands to set Jetson clocks high.

$ sudo nvpmodel -m <mode> --for MAX perf and power mode is 0
$ sudo jetson_clocks

The DeepStream application is running slowly¶

Solution 1:

One of the plugins in the pipeline may be running slowly. You can measure the latency of each plugin in the pipeline to determine if one of them is slow. * To enable frame latency measurement, run this command on the console:

$ export NVDS_ENABLE_LATENCY_MEASUREMENT=1

To enable latency for all plugins, run this command on the console:
```
$ export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1
```

Solution 2: (dGPU only)

Ensure that your GPU card is in the PCI slot with the highest bus width.

Solution 3:

In the configuration file’s [streammux] group, set batched-push-timeout to 1/max_fps.

Solution 4:

In the configuration file’s [streammux] group, set width and height to the stream’s resolution.

Solution 5:

For RTSP streaming input, in the configuration file’s [streammux] group, set live-source=1. Also make sure that all [sink#] groups have the sync property set to 0.

Solution 6:

If secondary inferencing is enabled, try to increase batch-size in the the configuration file’s [secondary-gie#] group in case the number of objects to be inferred is greater than the batch-size setting.

Solution 7:

On Jetson, use Gst-nvoverlaysink instead of Gst-nveglglessink as nveglglessink requires GPU utilization.

Solution 8:

If the GPU is the performance bottleneck, try increasing the interval at which the primary detector infers on input frames. You can do this by modifying the interval property of [primary-gie] group in the application configuration, or the interval property of the Gst-nvinfer configuration file.

Solution 9:

If the elements in the pipeline are getting starved for buffers (you can check if CPU/GPU utilization is low), increase the number of buffers allocated by the decoder by setting the num-extra-surfaces property of the [source#] group in the application or the num-extra-surfaces property of Gst-nvv4l2decoder element.

Solution 10:

If you are running the application inside docker/on-console and it delivers low FPS, set qos=0 in the configuration file’s [sink0] group. The issue is caused by initial load. With qos set to 1 as the property’s default value in the [sink0] group, decodebin starts dropping frames.

Solution 11:

For RTSP streaming input, if the input has high jitter the GStreamer rtpjitterbuffer element might drop packets which are late. Increase the latency property of rtspsrc, for deepstream-app set latency in [source*] group. Alternatively, if using RTSP type source (type=4) with deepstream-app, turn off drop-on-latency in deepstream_source_bin.c. These steps may add cumulative delay in frames reaching the renderer and memory accumulation in the rtpjitterbuffer if the pipeline is not fast enough.

NVIDIA Jetson Nano™, deepstream-segmentation-test starts as expected, but crashes after a few minutes rebooting the system¶

Solution:

We recommend you power the Jetson module through the DC power connector when running this app. USB adapters may not be able to handle the transients.

Errors occur when deepstream-app is run with a number of streams greater than 100¶

For example: (deepstream-app:15751): GStreamer-CRITICAL **: 19:25:29.810: gst_poll_write_control: assertion 'set != NULL' failed.

Solution:

Run this command on the console:

ulimit -Sn 4096

Then run the deepstream-app again.

Errors occur when deepstream-app fails to load plugin Gst-nvinferserver on dGPU only¶

For example: (deepstream-app:16632): GStreamer-WARNING **: 13:13:31.201: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtrtserver.so: cannot open shared object file: No such file or directory.

This is a harmless warning indicating that the DeepStream’s nvinferserver plugin cannot be used since “Triton Inference Server” is not installed on x86(dGPU) platforms. Jetson platforms should not have this problem since Triton is installed automatically by DeepStream package.

Solution 1:

Ignore this message if Users do not need Triton support. Otherwise see Solution 2, 3.

Solution 2:

Pull deepstream-triton docker image and start the container. Retry deepstream-app to launch triton models.

Solution 3:

Build Triton server library from source (https://github.com/NVIDIA/triton-inference-server/releases/tag/v1.12.0) and fix dynamic link problem manually.

Tensorflow models are running into OOM (Out-Of-Memory) problem¶

This problem may manifest as other errors like CUDA_ERROR_OUT_OF_MEMORY, core dump, application get killed once GPU memory is set up by tensorflow component.

Solution:

Tune parameter tf_gpu_memory_fraction in config file (e.g. config_infer_primary_detector_ssd_inception_v2_coco_2018_01_28.txt) to a proper value. For more details, see: samples/configs/deepstream-app-trtis/README

Memory usage keeps on increasing when the source is a long duration containerized files(e.g. mp4, mkv)¶

A memory accumulation bug is present in GStreamer’s Base Parse class which potentially affects all codec parsers provided by GStreamer. This bug is seen only with long duration seekable streams (mostly containerized files e.g. mp4). This does not affect live sources like RTSP. An issue has been filed on GStreamer’s gitlab project https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/468

Solution:

Apply the following temporary fix to the GStreamer sources and build the library.

Check the exact gstreamer version installed on the system.

$ gst-inspect-1.0 --version
gst-inspect-1.0 version 1.14.5
GStreamer 1.14.5
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0

Clone the Gstreamer repo and checkout the tag corresponding to the installed version.

$ git clone git@gitlab.freedesktop.org:gstreamer/gstreamer.git
$ cd gstreamer
$ git checkout 1.14.5

Make sure the build dependencies are installed.

$ sudo apt install libbison-dev build-essential flex debhelper

Run autogen.sh and configure script.

$ ./autogen.sh –noconfigure
$ ./configure –prefix=(pwd)/out # Don’t want to overwrite system libs

Save the following patch to a file.

diff --git a/libs/gst/base/gstbaseparse.c b/libs/gst/base/gstbaseparse.c
index 41adf130e..ffc662a45 100644
--- a/libs/gst/base/gstbaseparse.c
+++ b/libs/gst/base/gstbaseparse.c
@@ -1906,6 +1906,9 @@ gst_base_parse_add_index_entry (GstBaseParse * parse, guint64 offset,
GST_LOG_OBJECT (parse, "Adding key=%d index entry %" GST_TIME_FORMAT
    " @ offset 0x%08" G_GINT64_MODIFIER "x", key, GST_TIME_ARGS (ts), offset);

+  if (!key)
+    goto exit;
+
if (G_LIKELY (!force)) {

    if (!parse->priv->upstream_seekable) {

Apply the patch.
```
$ cat patch.txt | patch -p1
```
Build the sources.
```
make -j(nproc) && make install
```

8. Backup the distribution provided library and copy the newly built library. Adjust the library name for version. For jetson replace x86_64-linux-gnu with aarch64-linux-gnu.

$ sudo cp /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0.1405.0 ${HOME}/libgstbase-1.0.so.0.1405.0.backup
$ sudo cp out/lib/libgstbase-1.0.so.0.1405.0 /usr/lib/x86_64-linux-gnu/

Errors occur when deepstream-app is run with a number of RTSP streams and with NvDCF tracker¶

Inconsistency detected by ld.so: dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!

This is due to a known glibc bug - a rare race condition occurs when there are a lot of thread creations and dlopen() executing at the same time. To mitigate the issue, we recommend using persistent threads in NvDCF tracker with the following solution.

Solution:

Set the following param in NvDCF config file (e.g. tracker_config.yml):
```
usePersistentThreads: 1
```
Run DeepStream pipeline again.

Troubleshooting in NvDCF Parameter Tuning¶

Lingering Ghost Bbox¶

When a tracker is not terminated even after the target disappears from the scene, it results in a ghost bbox. This ghost bbox may linger around as it learns more background information.

To mitigate this issue, you may make the termination policy more aggressive by increasing minTrackerConfidence. It is recommended to set PGIE’s parameter interval=0 and adjust minTrackerConfidence. After this, you may try adjusting the PGIE interval while fine-tuning the tracker parameters.

Flickering Bbox¶

Bbox flickering occurs in the video output if the value for minTrackerConfidence is set too low. Increase the value for this parameter to mitigate the issue.

Although the real-time video output may have bbox flickering, you may enable the past-frame data configuration if the tracking IDs are well-maintained over time (i.e., useBufferedOutput: 1 in NvDCF config file and enable-past-frame=1 in deepstream-app config file) to retrieve the missing outputs for post-processing and analysis.

Frequent tracking ID changes although no nearby objects¶

This may occur because the tracker cannot detect the target from the correlation response map. It is recommended to start with lower minimum qualification for the target. First, set minTrackerConfidence with a relatively low value like 0.5. Also, in case the state estimator is enabled, the prediction may not be accurate enough. You may tune the state estimator parameters based on the expected motion dynamics, or disable during debugging.

Frequent tracking ID switches to the nearby objects¶

Make the data association policy stricter by increasing the minimum qualifications such as:

minMatchingScore4SizeSimilarity
minMatchingScore4Iou
minMatchingScore4VisualSimilarity

Consider enabling the instance-awareness to allow the correlation filters learn more discriminatively against the nearby objects. It may be helpful to use larger feature size (i.e., featureImgSizeLevel) and/or use more visual features (i.e., useColorNames: 1 and useHog: 1) if you can spare more resources.

Note

For more FAQs and troubleshooting, see https://forums.developer.nvidia.com/t/deepstream-sdk-faq/