The NVIDIA DeepStream SDK delivers a complete streaming analytics toolkit for AI-based video and image perception, as well as multi-sensor processing. DeepStream is an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions for transforming pixels and sensor data into actionable insights.
Isaac SDK comes with a variety of media acquisition, publishing, coding, decoding, inferring, and processing capabilities tailored for robotics use cases. However, multimedia support is not the primary responsibility of the robotics framework. The multitude of multimedia needs for robotics integration makes direct support difficult, so integration with DeepStream as an Isaac component provides a solution.
NVIDIA DeepStream SDK is based on the open-source GStreamer libraries: a pipeline-based multimedia framework that links together a wide variety of media processing systems. GStreamer is itself based on GLib: a set of low-level libraries for providing data-structure handling for C, portability wrappers, execution loops, and interfaces. Visit the GStreamer open source multimedia framework website for details and contributions to the framework.
Internal integration of the open-source GStreamer framework and NVIDIA DeepStream SDK as an Isaac component allows you to reuse their collection of multimedia processing features. The Isaac and DeepStream products naturally complement each other for robotics and media use cases.
GPU-accelerated DeepStream elements can be used as part of the GStreamer pipeline definition. DeepStream can be installed with the Jetson JetPack installer for Jetson Nano and Xavier platforms. Follow the DeepStream installation instructions to make their elements available on your host system.
Isaac does not distribute DeepStream or GStreamer. You must choose which modules to install and comply with their licenses.
The media pipeline is attached to the Isaac engine via the named appsink
and
appsrc
elements for receiving and transmitting, respectively. The component
pipeline
configuration parameter allows you to launch any GStreamer pipeline string
with any configuration and elements. Pipelines may contain zero or more application sources or
sinks. A pipeline containing no application source or sink is managed by Isaac but doesn’t share
data, which is useful for indirect multimedia support for robotic applications.
The application elements support the capabilities, formats, and memory models with the
equivalent Isaac protobuffer messages detailed in the table below. Use appropriate
videoconvert
and nvvideoconvert
elements and caps filters for compatibility. The
capabilities have to be explicitly defined.
Capabilities |
Format |
Messages |
---|---|---|
video/x-raw | RGB | ColorCameraProto |
The acquisition timestamp of the messages transmitted from the DeepStream component to other Isaac components is the node time at the moment of publication.
The component injects an Isaac scheduler thread pool into the GStreamer pipeline. The thread pool follows Isaac thread rules and GStreamer thread rules:
The component runs a single service job for the pipeline main loop.
The pipeline uses one or more jobs for its elements.
One or more elements may run within a single job.
An element may use multiple jobs.
The elements use push- or pull-based scheduling for upstream or downstream threads, respectively.
The jobs use Earliest Deadline First policy.
Each job has an execution group configuration.
A pipeline branch can be configured to run on a different thread with the
queue
element.The job statistics are reported in the scheduler report.
Below is an example job-statistics report for the gstreamer_pipeline_multi
application.
The pipeline is composed of three simple independent branches, which in turn run on three jobs.
|=========================================================================================|
| Job Statistics Report (blocking) |
|=========================================================================================|
| Name | Job Mode | Count | Time (Median - 90% - Max) [ms] |
|-----------------------------------------------------------------------------------------|
| DeepStream Service | Blocking One Shot | 1 | 32816.30 | 32816.30 | 32816.30 |
| DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.40 | 32814.40 | 32814.40 |
| DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.42 | 32814.42 | 32814.42 |
| DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.52 | 32814.52 | 32814.52 |
|=========================================================================================|
The gst-inspect-1.0 command-line tool allows you to view the elements available on the host:
$ gst-inspect-1.0
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
dtls: dtlsenc: DTLS Encoder
[...]
Total count: 265 plugins, 1420 features
The
gst-launch-1.0
command-line tool allows you to build and run a GStreamer pipeline. Enable the debug output for
GStreamer by setting the GST_DEBUG
environment variable to a category level between
0
for no debug output and 8
for a full memory dump. The information level at
4
is usually the most practical.
$ export GST_DEBUG=4
# Your application will now output debug information when you run it.
Obtain a graph of your pipeline by setting the GST_DEBUG_DUMP_DOT_DIR
environment
variable to an existing, writable directory path. Use the dot
tool to generate a
viewable image.
$ sudo apt install graphviz
$ export GST_DEBUG_DUMP_DOT_DIR=/tmp
# Running your application will generate a FILENAME.dot file in the /tmp folder which can be
# converted by the following command:
$ dot -Tpng /tmp/FILENAME.dot > pipeline.png
The following example displays a test video from the GStreamer FAQ with debug information, graph drawing, and verbosity enabled:
$ GST_DEBUG=4 GST_DEBUG_DUMP_DOT_DIR=. gst-launch-1.0 -v \
videotestsrc ! videoconvert ! autovideosink
The original GStreamer example has been altered to demonstrate how to integrate a GStreamer pipeline within an Isaac application. The Isaac example shows how the pipeline media can be imported or exported to Isaac communication channels available in the application configuration.
The pipeline has two bins: a test video source is brought into Isaac through
an application sink, and an application source exposes a video stream to a display video sink.
The
capabilities
of the application element are limited. As such a videoconvert
element ensures
support for the RGB raw format for both the source and sink elements. The pipeline thus changes to
the following:
videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac
appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink

Isaac Application Examples
The following is a minimal Isaac application that passes the video from the sink to the source through Isaac:
{
"name": "example_pipeline_app",
"modules": [
"deepstream",
],
"graph": {
"nodes": [
{
"name": "deepstream",
"components": [
{
"name": "message_ledger",
"type": "isaac::alice::MessageLedger"
},
{
"name": "pipeline",
"type": "isaac::deepstream::Pipeline"
}
]
}
],
"edges": [
{
"source": "deepstream/pipeline/to_isaac",
"target": "deepstream/pipeline/from_isaac"
}
]
},
"config": {
"deepstream": {
"pipeline": {
"pipeline": "videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink"
}
}
}
}
Visualizing an application sink using a video test source would look like the following in Isaac WebSight:

Other demonstration applications are available in the packages/deepstream/apps folder:
Application Name |
Demonstration |
---|---|
gstreamer_pipeline | Trivial GStreamer video test source pipeline |
gstreamer_pipeline_distributed_a | Acquire and publish MPEG-4 compressed video over UDP transport |
gstreamer_pipeline_distributed_b | Receive MPEG-4 compressed video over UDP transport |
gstreamer_pipeline_external | Isaac-managed external media pipelines |
gstreamer_pipeline_multi | Multiple video pipelines in a single component |
logitech_c920pro_cpu | USB connected, V4L2 interfaced, H.264 compressed, CPU decoded camera |
philips_spc1330nc | Trivial USB connected, V4L2 interfaced raw, uncompressed camera |
sony_snchmx70 | Ethernet IP/RTSP transported, H.264 compressed camera |