Quickstart Guide
NVIDIA® DeepStream Software Development Kit (SDK) is an accelerated AI framework to build intelligent video analytics (IVA) pipelines. DeepStream runs on NVIDIA® T4, NVIDIA® Hopper, NVIDIA® Ampere, NVIDIA® ADA and platforms such as NVIDIA® Jetson AGX Orin™, NVIDIA® Jetson Orin™ NX, NVIDIA® Jetson Orin™ Nano.
Jetson Setup
This section explains how to prepare a Jetson device before installing the DeepStream SDK.
Install Jetson SDK components
Download NVIDIA SDK Manager from https://developer.nvidia.com/embedded/jetpack. You will use this to install JetPack 6.0 DP (corresponding to L4T 36.2 release)
NVIDIA SDK Manager is a graphical application which flashes and installs the JetPack packages.
The flashing procedure takes approximately 10-30 minutes, depending on the host system.
Note
If you are using Jetson Orin nano developer kit, you can download the SD card image from https://developer.nvidia.com/embedded/jetpack. This comes packaged with CUDA, TensorRT and cuDNN.
Install Dependencies
Enter the following commands to install the prerequisite packages:
$ sudo apt install \
libssl3 \
libssl-dev \
libgstreamer1.0-0 \
gstreamer1.0-tools \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-plugins-ugly \
gstreamer1.0-libav \
libgstreamer-plugins-base1.0-dev \
libgstrtspserver-1.0-0 \
libjansson4 \
libyaml-cpp-dev
Note
Sometimes with RTSP streams the application gets stuck on reaching EOS. This is because of an issue in rtpjitterbuffer component.
To fix this issue,a script “update_rtpmanager.sh” at /opt/nvidia/deepstream/deepstream/
has been provided with
required details to update gstrtpmanager library. The script should be executed once above mentioned packages are
installed as prerequisite.
Sometimes while running gstreamer pipeline or sample apps,
user can encounter error : GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument. Aborting.
The issue is caused because of a bug in glib 2.0-2.72 version which comes with ubuntu22.04 by default.
The issue is addressed in glib2.76 and its installation is required to fix the issue (https://github.com/GNOME/glib/tree/2.76.6).
Install librdkafka (to enable Kafka protocol adaptor for message broker)
Clone the librdkafka repository from GitHub:
$ git clone https://github.com/confluentinc/librdkafka.git
Configure and build the library:
$ cd librdkafka $ git checkout tags/v2.2.0 $ ./configure --enable-ssl $ make $ sudo make install
Copy the generated libraries to the deepstream directory:
$ sudo mkdir -p /opt/nvidia/deepstream/deepstream-6.4/lib $ sudo cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-6.4/lib $ sudo ldconfig
Install latest NVIDIA BSP packages
Installation of JetPack 6.0 DP will ensure that latest NVIDIA BSP packages are installed.
Install the DeepStream SDK
Method 1: Using SDK Manager
Select
DeepStreamSDK
from theAdditional SDKs
section along with JP 6.0 DP software components for installation.Method 2: Using the DeepStream tar package: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/deepstream
Download the DeepStream 6.4 Jetson tar package
deepstream_sdk_v6.4.0_jetson.tbz2
to the Jetson device.Enter the following commands to extract and install the DeepStream SDK:
$ sudo tar -xvf deepstream_sdk_v6.4.0_jetson.tbz2 -C / $ cd /opt/nvidia/deepstream/deepstream-6.4 $ sudo ./install.sh $ sudo ldconfig
Method 3: Using the DeepStream Debian package: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/deepstream
Download the DeepStream 6.4 Jetson Debian package
deepstream-6.4_6.4.0-1_arm64.deb
to the Jetson device. Enter the following command:$ sudo apt-get install ./deepstream-6.4_6.4.0-1_arm64.deb
Method 4: Use Docker containers
DeepStream docker containers are available on NGC. See the Docker Containers section to learn about developing and deploying DeepStream using docker containers.
Run deepstream-app (the reference application)
Navigate to the samples directory on the development kit.
Enter the following command to run the reference application:
$ deepstream-app -c <path_to_config_file>
Where
<path_to_config_file>
is the pathname of one of the reference application’s configuration files, found inconfigs/deepstream-app/
. See Package Contents for a list of the available files.Config files that can be run with deepstream-app:
source30_1080p_dec_infer-resnet_tiled_display_int8.txt
source30_1080p_dec_preprocess_infer-resnet_tiled_display_int8.txt
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8_gpu1.txt
(dGPU only)source1_usb_dec_infer_resnet_int8.txt
source1_csi_dec_infer_resnet_int8.txt
(Jetson only)source2_csi_usb_dec_infer_resnet_int8.txt
(Jetson only)source6_csi_dec_infer_resnet_int8.txt
(Jetson only)source2_1080p_dec_infer-resnet_demux_int8.txt
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.yml
source30_1080p_dec_infer-resnet_tiled_display_int8.yml
source4_1080p_dec_preprocess_infer-resnet_preprocess_sgie_tiled_display_int8.txt
source2_dewarper_test.txt
Note
You can find sample configuration files under
/opt/nvidia/deepstream/deepstream-6.4/samples
directory. Enter this command to see application usage:$ deepstream-app --help
To save TensorRT Engine/Plan file, run the following command:
$ sudo deepstream-app -c <path_to_config_file>
To show labels in 2D Tiled display view, expand the source of interest with mouse left-click on the source. To return to the tiled display, right-click anywhere in the window.
Keyboard selection of source is also supported. On the console where application is running, press the
z
key followed by the desired row index (0 to 9), then the column index (0 to 9) to expand the source. To restore 2D Tiled display view, pressz
again.
Boost the clocks
After you have installed DeepStream SDK, run these commands on the Jetson device to boost the clocks:
$ sudo nvpmodel -m 0
$ sudo jetson_clocks
You should run these commands before running DeepStream application.
Run precompiled sample applications
Navigate to the chosen application directory inside
sources/apps/sample_apps
.Follow the directory’s README file to run the application.
Note
If the application encounters errors and cannot create Gst elements, remove the GStreamer cache, then try again. To remove the GStreamer cache, enter this command:
$ rm ${HOME}/.cache/gstreamer-1.0/registry.aarch64.bin
When the application is run for a model which does not have an existing engine file, it may take up to a few minutes (depending on the platform and the model) for the file generation and the application launch. For later runs, these generated engine files can be reused for faster loading.
dGPU Setup for Ubuntu
This section explains how to prepare an Ubuntu x86_64
system with NVIDIA dGPU devices before installing the DeepStream SDK.
Note
This document uses the term dGPU (“discrete GPU”) to refer to NVIDIA GPU expansion card products such as NVIDIA Tesla® T4, NVIDIA® Hopper, NVIDIA® Ampere, NVIDIA® ADA, NVIDIA GeForce® GTX 1080, NVIDIA GeForce® RTX 2080, NVIDIA GeForce® RTX 3080, NVIDIA GeForce® RTX 4080 and GeForce®/NVIDIA RTX/QUADRO. This version of DeepStream SDK runs on GPUs supported by NVIDIA driver 535.104.12 and NVIDIA TensorRT™ 8.6.1.6 and later versions.
You must install the following components:
Ubuntu 22.04
GStreamer 1.20.3
NVIDIA driver 535.104.12
CUDA 12.2
TensorRT 8.6.1.6
Remove all previous DeepStream installations
Enter the following commands to remove all previous DeepStream 3.0 or prior installations:
$ sudo rm -rf /usr/local/deepstream /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstnv* /usr/bin/deepstream* /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libnvdsgst*
/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream*
/opt/nvidia/deepstream/deepstream*
$ sudo rm -rf /usr/lib/x86_64-linux-gnu/libv41/plugins/libcuvidv4l2_plugin.so
To remove DeepStream 4.0 or later installations:
Open the
uninstall.sh
file in/opt/nvidia/deepstream/deepstream/
Set PREV_DS_VER as 4.0
Run the following script as
sudo ./uninstall.sh
Install Dependencies
Enter the following commands to install the necessary packages before installing the DeepStream SDK:
$ sudo apt install \
libssl3 \
libssl-dev \
libgstreamer1.0-0 \
gstreamer1.0-tools \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-plugins-ugly \
gstreamer1.0-libav \
libgstreamer-plugins-base1.0-dev \
libgstrtspserver-1.0-0 \
libjansson4 \
libyaml-cpp-dev \
libjsoncpp-dev \
protobuf-compiler \
gcc \
make \
git \
python3
Note
Sometimes with RTSP streams the application gets stuck on reaching EOS. This is because of an issue in rtpjitterbuffer component.
To fix this issue,a script “update_rtpmanager.sh” at /opt/nvidia/deepstream/deepstream/
has been provided with
required details to update gstrtpmanager library. The script should be executed once above mentioned packages are
installed as prerequisite.
Sometimes while running gstreamer pipeline or sample apps,
user can encounter error : GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument. Aborting.
The issue is caused because of a bug in glib 2.0-2.72 version which comes with ubuntu22.04 by default.
The issue is addressed in glib2.76 and its installation is required to fix the issue (https://github.com/GNOME/glib/tree/2.76.6).
Install CUDA Toolkit 12.2
Run the following commands (reference, https://developer.nvidia.com/cuda-downloads):
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
$ sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
$ sudo apt-get update
$ sudo apt-get install cuda-toolkit-12-2
Note
If you observe following errors while CUDA installation, refer to https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/.
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease' is no longer signed.
Install NVIDIA driver 535.104.12
Download and install using NVIDIA driver 535.104.12 from NVIDIA Unix drivers page at: https://www.nvidia.cn/Download/driverResults.aspx/212056/en-us/
Run the following commands:
$chmod 755 NVIDIA-Linux-x86_64-535.104.12.run $sudo ./NVIDIA-Linux-x86_64-535.104.12.run --no-cc-version-check
Note
Ensure gdm, lightdm or Xorg service is stopped while installing nvidia driver
Use command : sudo service gdm stop
sudo service lightdm stop
sudo pkill -9 Xorg
To Run higher number of streams (200+) on Hopper, Ampere and Ada, follow below instructions:
$ sudo service display-manager stop #Make sure no process is running on GPU i.e. Xorg or trition server etc $ sudo pkill -9 Xorg #Remove kernel modules $ sudo rmmod nvidia_drm nvidia_modeset nvidia #Load Modules with Regkeys $ sudo modprobe nvidia NVreg_RegistryDwords="RMDebugOverridePerRunlistChannelRam = 1;RMIncreaseRsvdMemorySizeMB = 1024;RMDisableChIdIsolation = 0x1;RmGspFirmwareHeapSizeMB = 256" $ sudo service display-manager start
Install TensorRT 8.6.1.6
Run the following command to install TensorRT 8.6.1.6:
sudo apt-get install libnvinfer8=8.6.1.6-1+cuda12.0 libnvinfer-plugin8=8.6.1.6-1+cuda12.0 libnvparsers8=8.6.1.6-1+cuda12.0 \ libnvonnxparsers8=8.6.1.6-1+cuda12.0 libnvinfer-bin=8.6.1.6-1+cuda12.0 libnvinfer-dev=8.6.1.6-1+cuda12.0 \ libnvinfer-plugin-dev=8.6.1.6-1+cuda12.0 libnvparsers-dev=8.6.1.6-1+cuda12.0 libnvonnxparsers-dev=8.6.1.6-1+cuda12.0 \ libnvinfer-samples=8.6.1.6-1+cuda12.0 libcudnn8=8.9.4.25-1+cuda12.2 libcudnn8-dev=8.9.4.25-1+cuda12.2Note
It is assumed, “deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/” has been added, as mentioned in CUDA Toolkit 12.2 installation step.
Install librdkafka (to enable Kafka protocol adaptor for message broker)
Clone the librdkafka repository from GitHub:
$ git clone https://github.com/confluentinc/librdkafka.git
Configure and build the library:
$ cd librdkafka $ git checkout tags/v2.2.0 $ ./configure --enable-ssl $ make $ sudo make install
Copy the generated libraries to the deepstream directory:
$ sudo mkdir -p /opt/nvidia/deepstream/deepstream-6.4/lib $ sudo cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-6.4/lib $ sudo ldconfig
Install the DeepStream SDK
Method 1: Using the DeepStream Debian package
Download the DeepStream 6.4 dGPU Debian package
deepstream-6.4_6.4.0-1_amd64.deb
: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/deepstreamEnter the command:
$ sudo apt-get install ./deepstream-6.4_6.4.0-1_amd64.deb
Method 2: Download the DeepStream tar package: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/deepstream
Navigate to the location of the downloaded DeepStream package to extract and install the DeepStream SDK:
$ sudo tar -xvf deepstream_sdk_v6.4.0_x86_64.tbz2 -C / $ cd /opt/nvidia/deepstream/deepstream-6.4/ $ sudo ./install.sh $ sudo ldconfig
Method 3: Use Docker containers
DeepStream docker containers are available on NGC. See the Docker Containers section to learn about developing and deploying DeepStream using docker containers.
Run the deepstream-app (the reference application)
Go to the samples directory and enter this command:
$ deepstream-app -c <path_to_config_file>
Where
<path_to_config_file>
is the pathname of one of the reference application’s configuration files, found inconfigs/deepstream-app
. See Package Contents for a list of the available files.Note
To dump engine file, run the following command:
$ sudo deepstream-app -c <path_to_config_file>
You can find sample configuration files under
/opt/nvidia/deepstream/deepstream-6.4/samples
directory. Enter this command to see application usage:$ deepstream-app --help
To show labels in 2D tiled display view, expand the source of interest with a mouse left-click on the source. To return to the tiled display, right-click anywhere in the window.
Keyboard selection of source is also supported. On the console where application is running, press the
z
key followed by the desired row index (0 to 9), then the column index (0 to 9) to expand the source. To restore the 2D Tiled display view, pressz
again.
Run precompiled sample applications
Navigate to the chosen application directory inside
sources/apps/sample_apps
.Follow that directory’s README file to run the application.
Note
If the application encounters errors and cannot create Gst elements, remove the GStreamer cache, then try again. To remove the GStreamer cache, enter this command:
$ rm ${HOME}/.cache/gstreamer-1.0/registry.x86_64.bin
When the application is run for a model which does not have an existing engine file, it may take up to a few minutes (depending on the platform and the model) for the file generation and application launch. For later runs, these generated engine files can be reused for faster loading.
dGPU Setup for RedHat Enterprise Linux (RHEL)
DeepStream for RHEL is not supported in this release.
How to visualize the output if the display is not attached to the system
1 . Running with an X server by creating virtual display
Refer https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#configuring-xorg-server-on-linux-server for details.
2 . Running without an X server (applicable for applications supporting RTSP streaming output)
The default configuration files provided with the SDK have the EGL based nveglglessink
as the default renderer (indicated by type=2 in the [sink] groups). The renderer requires a running X server and fails without one.
In case of absence of an X server, DeepStream reference applications provide an alternate functionality of streaming the output over RTSP. This can be enabled by adding an RTSP out sink group in the configuration file. Refer to [sink2]
group in source30_1080p_dec_infer-resnet_tiled_display_int8.txt
file for an example. Don’t forget to disable the nveglglessink
renderer by setting enable=0 for the corresponding sink group.
Platform and OS Compatibility
The following table provides information about platform and operating system compatibility in the current and earlier versions of DeepStream.
DS release
DS 6.1
DS 6.1.1
DS 6.2
DS 6.3
DS 6.4
Jetson platforms
AGX Xavier, Jetson NX, Jetson Orin
AGX Xavier, Jetson NX, Jetson Orin
Xavier AGX , Xavier NX, AGX Orin, Orin NX
Xavier AGX , Xavier NX, AGX Orin, Orin NX, Orin nano
AGX Orin, Orin NX, Orin nano
OS
L4T Ubuntu 20.04
L4T Ubuntu 20.04
L4T Ubuntu 20.04
L4T Ubuntu 20.04
L4T Ubuntu 22.04
JetPack release
5.0.1 DP
5.0.2 GA Revision 1
5.1 GA
5.1.2 GA
6.0 DP
L4T release
34.1.1
35.1
35.2.1
35.4
36.2
CUDA release
CUDA 11.4
CUDA 11.4
CUDA 11.4
CUDA 11.4
CUDA 12.2
cuDNN release
cuDNN 8.3.2.49
cuDNN 8.4.1.50+
cuDNN 8.6.0.166+
cuDNN 8.6.0.166+
cuDNN 8.9.4.25+
TensorRT release
TRT 8.4.0.11
TRT 8.4.1.5
TRT 8.5.2.2
TRT 8.5.2.2
TRT 8.6.2.3
OpenCV release
OpenCV 4.2.0
OpenCV 4.2.0
OpenCV 4.2.0
OpenCV 4.5.4
OpenCV 4.8.0
VisionWorks
NA
NA
NA
NA
NA
GStreamer
GStreamer 1.16.2
GStreamer 1.16.2
GStreamer 1.16.3
GStreamer 1.16.3
GStreamer 1.20.3
Docker image
deepstream-l4t:6.1
deepstream-l4t:6.1.1
deepstream-l4t:6.2
deepstream-l4t:6.3
deepstream:6.4
DS release
DS 6.1
DS 6.1.1
DS 6.2
DS 6.3
DS 6.4
GPU platforms
T4, V100, A2, A10, A30, A100, RTX Ampere (Ax000/RTX30x0)
T4, V100, A2, A10, A30, A100, RTX Ampere (Ax000/RTX30x0)
T4, V100, A2, A10, A30, A100, RTX Ampere (Ax000/RTX30x0), Hopper, ADA
T4, V100, A2, A10, A30, A100, RTX Ampere (Ax000/RTX30x0), Hopper, ADA
T4, V100, A2, A10, A30, A100, RTX Ampere (Ax000/RTX30x0), Hopper, ADA
OS
Ubuntu 20.04
Ubuntu 20.04
Ubuntu 20.04
Ubuntu 20.04
Ubuntu 22.04
GCC
GCC 9.4.0
GCC 9.4.0
GCC 9.4.0
GCC 9.4.0
GCC 11.4.0
CUDA release
CUDA 11.6.1
CUDA 11.7.1
CUDA 11.8
CUDA 12.1
CUDA 12.2
cuDNN release
cuDNN 8.4.0.27
cuDNN 8.4.1.50+
cuDNN 8.7.0.84-1+
cuDNN 8.8.1.3-1+
cuDNN 8.9.4.25-1+
TRT release
TRT 8.2.5.1
TRT 8.4.1.5
TRT 8.5.2.2
TRT 8.5.3.1
TRT 8.6.1.6
Display Driver
R510.47.03
R515.65.01
R525.85.12
R525.125.06
R535.104.12
VideoSDK release
SDK 9.1
SDK 9.1
SDK 9.1
SDK 9.1
SDK 9.1
OFSDK release
2.0.23
2.0.23
2.0.23
2.0.23
2.0.23
GStreamer release
GStreamer 1.16.2
GStreamer 1.16.2
GStreamer 1.16.3
GStreamer 1.16.3
GStreamer 1.20.3
OpenCV release
OpenCV 4.2.0
OpenCV 4.2.0
OpenCV 4.2.0
OpenCV 4.2.0
OpenCV 4.5.4
Docker image
deepstream:6.1
deepstream:6.1.1
deepstream:6.2
deepstream:6.3
deepstream:6.4
NVAIE release
NA
NA
NVAIE-3.x
NVAIE-3.x
NVAIE-4.x
Note
By default, OpenCV has been deprecated. However, OpenCV can be enabled in plugins such as nvinfer (nvdsinfer) and dsexample (gst-dsexample) by setting WITH_OPENCV=1 in the Makefile of these components. Please refer component README for more instructions.
NA: “Not Applicable”
DeepStream Triton Inference Server Usage Guidelines
To migrate the Triton version in a DeepStream 6.4 deployment (Triton 23.08) to a newer version (say Triton 23.09 or newer), follow the instructions at DeepStream Triton Migration Guide.
dGPU
Pull the DeepStream Triton Inference Server docker
docker pull nvcr.io/nvidia/deepstream:6.4-triton-multiarch
Start the docker
docker run --gpus "device=0" -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -e CUDA_CACHE_DISABLE=0 nvcr.io/nvidia/deepstream:6.4-triton-multiarch
Note
The triton docker for x86 is based on Tritonserver 23.08 docker and Jetson is based on tritonserver 23.11 docker, and has Ubuntu 22.04.
When the triton docker is launched for the first time, it might take a few minutes to start since it has to generate compute cache.
Jetson
DeepStream Triton container image (nvcr.io/nvidia/deepstream:6.4-triton-multiarch) has Triton Inference Server and supported backend libraries pre-installed.
In order to run the Triton Inference Server directly on device, i.e., without docker, Triton Server setup will be required.
Go to samples directory and run the following commands to set up the Triton Server and backends.
$ cd /opt/nvidia/deepstream/deepstream/samples/
$ sudo ./triton_backend_setup.sh
Note
By default script will download the Triton Server version 2.40. For setting up any other version change the package path accordingly.
Triton backends are installed into /opt/nvidia/deepstream/deepstream/lib/triton_backends
by default by the script. User can update infer_config
settings for specific folders as follows:
model_repo {
backend_dir: /opt/nvidia/tritonserver/backends/
}
Using DLA for inference
DLA is Deep Learning Accelerator present on the Jetson AGX Orin and Jetson Orin NX. These platforms have two DLA engines. DeepStream can be configured to run inference on either of the DLA engines through the Gst-nvinfer plugin. One instance of Gst-nvinfer plugin and thus a single instance of a model can be configured to be executed on a single DLA engine or the GPU. However, multiple Gst-nvinfer plugin instances can be configured to use the same DLA. To configure Gst-nvinfer to use the DLA engine for inference, modify the corresponding property in nvinfer component configuration file (example: samples/configs/deepstream-app/config_infer_primary.txt): Set enable-dla=1 in [property] group. Set use-dla-core=0 or use-dla-core=1 depending on the DLA engine to use.
DeepStream does support inferencing using GPU and DLAs in parallel. You can run this in separate processes or single process. You will need three separate sets of configs configured to run on GPU, DLA0 and DLA1:
Separate processes
When GPU and DLA are run in separate processes, set the environment variable CUDA_DEVICE_MAX_CONNECTIONS
as 1
from the terminal where DLA config is running.
Single process
DeepStream reference application supports multiple configs in the same process. To run DLA and GPU in same process, set environment variable CUDA_DEVICE_MAX_CONNECTIONS
as 32
:
$ deepstream-app -c <gpuconfig> -c <dla0config> -c <dla1config>