Troubleshooting

This section lists resolutions and tips for troubleshooting across various modules and scenarios.

Setup

Initial deployment taking a long time

containers are downloaded from NGC during first time install, and therefore time to first deployment might be impacted by network bandwidth available to download the containers.

Containers not running after deployment

Check if root filesystem is full by running the df command and verifying usage.

nvidia@tegra-ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk3p1   54G   54G     0 100% /

Examine root filesystem content to determine possible files to delete using command such as du.

This is especially likely if no external storage is attached to the system to support a data partition.

DeepStream FPS degradation

  • With dynamic stream addition mechanism, a slight fps degradation is possible (eg: going down to 29 from 30) especially at larger stream counts.

  • Ensure that your input RTSP stream is healthy in terms of quality and fps. The VLC video player for instance enables display of stream metrics including lost frames, corrupted frames and bitrate. This can be viewed by navigating: Tools > Media Information > Statistics.

  • Verify if your system is over subscribed by verifying utilization metrics using tegrastats. Refer to the supported stream counts for the AI-NVR workflow under the Quick Start Guide section in the documentation, which also describes considerations and tips for performance tuning.

  • Make sure to run at maximum clocks and power settings as mentioned in the setup guide. See instructions for Update settings for performance in Quick Start Guide

DeepStream overlay stream issues

Video is playing but video quality is poor

  • Ensure streaming source like cameras and the Jetson device are connected via High Speed network (~1 GB).

  • Make sure video is getting rendered at the expected FPS of ~30.

  • Ensure that the device from which you are viewing the streams (eg: laptop, mobile app) is connected to a high speed network as well, and that there is sufficient bandwidth between the viewing client (such as VLC) and the Jetson device.

  • Update the UDP ports in the following files:

    if on Orin AGX:

    configs/deepstream/pn26/ds-config-0_agx.txt - Increment udp port under sink3
configs/deepstream/pn26/ds-config-1_agx.txt - Increment udp port under sink8

    if on Orin NX16:

    configs/deepstream/pn26/ds-config-0_nx.txt - Increment udp port under sink3
configs/deepstream/pn26/ds-config-1_nx.txt - Increment udp port under sink8

    By default these are 5401 and 5402, increment by 10 each time the issue is seen and redeploy.

DeepStream RTSP tiled output has video aspect ratio distorted

Use square grid configuration in tiler section within DeepStream config file to ensure that video aspect ratio is preserved during tiling.

Note that depending on the number of input streams, they are distributed across the two DeepStream pipelines each with its own RTSP output stream.

WebRTC Streaming issues

Video is not playing

  • Make sure streaming client (VST webUI browser or mobile app) is in the same network as the device. Otherwise, a relay streaming service such as Twilio is required, and VST must be configured to use it.

Black screen seen during video streaming with mobile app/browser

Ensure that Jetson device and the client (mobile phone) are in the same network. Otherwise video relay streaming has to be setup using service such as Twilio as described in the VST documentation.

Troubleshooting Video Quality - Deep Dive

If you’ve followed the general troubleshooting guidelines but are still experiencing issues with video quality, this section offers a detailed approach to isolate and localize streaming quality issue.

Checking the incoming RTSP stream quality from Camera to Jetson device:

Use the following options to make sure the input RTSP stream from the camera is received at the connected Jetson device with the expected quality,

  • VST metrics:

    • Access streaming metrics via VST webUI by navigating to Debug >> Stream Stats.

    • In the FPS stats, ensure the camera FPS metrics consistently show approximately 30 FPS.

    _images/vst_stream_debug.png

  • FPS from DeepStream logs:

    • Use the following command to view DeepStream logs:

      ` sudo docker logs -f deepstream `

    • Confirm that the average FPS logged is close to 30 FPS.

  • Manual stream inspection:

    • Connect a monitor to the Jetson device.

    • Open a terminal on the Jetson device and execute:

      gst-launch-1.0 playbin uri=rtsp://<camera_url>

    • Manually inspect the input stream from the camera to ensure it’s playing at good quality on the Jetson device.

Checking DeepStream Overlay Output Stream Quality:

Use the following options to make sure the RTSP overlay stream output from Deepstream is of expected quality,

  • Overlay stream FPS:

    Execute the following command to check if the DS overlay output is around 30 FPS:

    ` gst-launch-1.0 rtspsrc location=rtsp://<ds_rtsp_url> ! rtph264depay ! h264parse ! nvv4l2decoder ! fpsdisplaysink fps-update-interval=5000 video-sink=fakesink -v | grep -v octet `

  • Manual Stream Inspection:

    • Connect a monitor to the Jetson device.

    • Open a terminal on the Jetson device and execute: ` gst-launch-1.0 playbin uri=rtsp://<ds_rtsp_url> `

    • Manually inspect the overlay stream from DS to ensure it’s playing with good quality on the Jetson device.

Checking the output stream quality at the client app:

  • VST Metrics:

    • Use the VST webUI to access streaming metrics by navigating to Debug >> Stream Stats.

    • This view displays client bitrate, dropped frame count, and client FPS.

    • Dropped frames (nackCount) may indicate insufficient network bandwidth, while fluctuations in bitrate or client FPS can suggest performance issues with the Jetson system.

_images/webrtc_stats.png

Known issues

  • Cameras outside of supported list specified in the VST documentation may fail to detect. Additionally, camera without s-profile compliance may not work with VST.

  • Occasional crashes are possible in DeepStream container due to a few known issues noted below that occur sporadically. In each case however, docker compose will automatically restart the container and system will get operational.

    • “malloc(): unaligned fastbin chunk detected” resulting in a crash.

    • sometimes crash observed upon stream addition.

    • PVA issue upon DeepStream startup causing crash with error VPI_ERROR_INVALID_OPERATION: PVA is not available and may be oversubscribed in the system. Reboot the system to recover.

  • Newly added streams to VST not getting picked up in DeepStream due to add/remove API timing out; this can particularly occur when VST restarts (such as after a crash), upon which DeepStream streams are removed and re-added by SDR.

  • DeepStream AI microservice in the current release does not have capability to recover from inputs streams going to 0 fps; restart the DeepStream container to recover from this case.

  • VST crashes occasionally seen when adding RTSP streams. However, docker compose will automatically restart the container and system will get operational.

  • Docker compose deployment may not run properly after system reboots. This is due to the required startup ordering of containers not being enforced when containers are automatically started by Docker after reboot.

  • Stream name change in VST does not get reflected in DeepStream.

  • Intermittent freezing and distortion during webRTC streaming; do refer to streaming issues troubleshooting for tips to diagnose the cause.

  • Mobile app video streaming at increased stream count can result in drop in video quality depending on mobile phone specifications.