DeepStream for Robotics

Technology

The NVIDIA DeepStream SDK delivers a complete streaming analytics toolkit for AI-based video and image perception, as well as multi-sensor processing. DeepStream is an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions for transforming pixels and sensor data into actionable insights.

Isaac SDK comes with a variety of media acquisition, publishing, coding, decoding, inferring, and processing capabilities tailored for robotics use cases. However, multimedia support is not the primary responsibility of the robotics framework. The multitude of multimedia needs for robotics integration makes direct support difficult, so integration with DeepStream as an Isaac component provides a solution.

Note

NVIDIA DeepStream SDK is based on the open-source GStreamer libraries: a pipeline-based multimedia framework that links together a wide variety of media processing systems. GStreamer is itself based on GLib: a set of low-level libraries for providing data-structure handling for C, portability wrappers, execution loops, and interfaces. Visit the GStreamer open source multimedia framework website for details and contributions to the framework.

Component

Internal integration of the open-source GStreamer framework and NVIDIA DeepStream SDK as an Isaac component allows you to reuse their collection of multimedia processing features. The Isaac and DeepStream products naturally complement each other for robotics and media use cases.

GPU-accelerated DeepStream elements can be used as part of the GStreamer pipeline definition. DeepStream can be installed with the Jetson JetPack installer for Jetson Nano and Xavier platforms. Follow the DeepStream installation instructions to make their elements available on your host system.

Note

Isaac does not distribute DeepStream or GStreamer. You must choose which modules to install and comply with their licenses.

The media pipeline is attached to the Isaac engine via the named appsink and appsrc elements for receiving and transmitting, respectively. The component pipeline configuration parameter allows you to launch any GStreamer pipeline string with any configuration and elements.

The application elements support the capabilities, formats, and memory models with the equivalent Isaac protobuffer messages detailed in the table below. Use appropriate videoconvert and nvvideoconvert elements and caps filters for compatibility. The capabilities have to be explicitely defined.

Capabilities Format Messages
video/x-raw RGB ColorCameraProto

The acquisition timestamp of the messages transmitted from the DeepStream component to other Isaac components is the node time at the moment of publication.

Tools

The gst-inspect-1.0 command-line tool allows you to view the elements available on the host:

$ gst-inspect-1.0
video4linux2:  v4l2src: Video (video4linux2) Source
video4linux2:  v4l2sink: Video (video4linux2) Sink
video4linux2:  v4l2radio: Radio (video4linux2) Tuner
video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
dtls:  dtlsenc: DTLS Encoder
[...]
Total count: 265 plugins, 1420 features

The gst-launch-1.0 command-line tool allows you to build and run a GStreamer pipeline. Enable the debug output for GStreamer by setting the GST_DEBUG environment variable to a category level between 0 for no debug ouput and 8 for a full memory dump. The information level at 4 is usually the most practical.

$ export GST_DEBUG=4
# Your application will now output debug information when you run it.

Obtain a graph of your pipeline by setting the GST_DEBUG_DUMP_DOT_DIR environment variable to an existing, writable directory path. Use the dot tool to generate a viewable image.

$ sudo apt install graphviz
$ export GST_DEBUG_DUMP_DOT_DIR=/tmp
# Running your application will generate a FILENAME.dot file in the /tmp folder which can be
# converted by the following command:
$ dot -Tpng /tmp/FILENAME.dot > pipeline.png

Examples

The following example displays a test video from the GStreamer FAQ with debug information, graph drawing, and verbosity enabled:

$ GST_DEBUG=4 GST_DEBUG_DUMP_DOT_DIR=. gst-launch-1.0 -v \
  videotestsrc ! videoconvert ! autovideosink

The original GStreamer example has been altered to demonstrate how to integrate a GStreamer pipeline within an Isaac application. The Isaac example shows how the pipeline media can be imported or exported to Isaac communication channels available in the application configuration.

The pipeline has two bins: a test video source is brought into Isaac through an application sink, and an application source exposes a video stream to a display video sink. The capabilities of the application element are limited. As such a videoconvert element ensures support for the RGB raw format for both the source and sink elements. The pipeline thus changes to the following:

videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac

appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink
Multimedia pipeline with input and output to Isaac.

Isaac Application Examples

The following is a minimal Isaac application that passes the video from the sink to the source through Isaac:

{
  "name": "example_pipeline_app",
  "modules": [
    "deepstream",
  ],
  "graph": {
    "nodes": [
      {
        "name": "deepstream",
        "components": [
          {
            "name": "message_ledger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "pipeline",
            "type": "isaac::deepstream::Pipeline"
          }
        ]
      }
    ],
    "edges": [
      {
        "source": "deepstream/pipeline/to_isaac",
        "target": "deepstream/pipeline/from_isaac"
      }
    ]
  },
  "config": {
    "deepstream": {
      "pipeline": {
        "pipeline": "videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac  appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink"
      }
    }
  }
}

Visualizing an application sink using a video test source would look like the following in Isaac WebSight:

WebSight visualization of a video test source through application sink.

Other demonstration applications are available in the packages/deepstream/apps folder:

Application Name Demonstration
gstreamer_pipeline_distributed_a Acquire and publish MPEG-4 compressed video over UDP transport
gstreamer_pipeline_distributed_b Receive MPEG-4 compressed video over UDP transport
gstreamer_pipeline_multi Multiple video pipelines in a single component
gstreamer_pipeline Trivial GStreamer video test source pipeline
logitech_c920pro_cpu USB connected, V4L2 interfaced, H.264 compressed, CPU decoded camera
philips_spc1330nc Trivial USB connected, V4L2 interfaced raw, uncompressed camera
sony_snchmx70 Ethernet IP/RTSP transported, H.264 compressed camera