DeepStream for Robotics

The NVIDIA DeepStream SDK delivers a complete streaming analytics toolkit for AI-based video and image perception, as well as multi-sensor processing. DeepStream is an integral part of NVIDIA Metropolis, the platform for building end-to-end services and solutions for transforming pixels and sensor data into actionable insights.

Isaac SDK comes with a variety of media acquisition, publishing, coding, decoding, inferring, and processing capabilities tailored for robotics use cases. However, multimedia support is not the primary responsibility of the robotics framework. The multitude of multimedia needs for robotics integration makes direct support difficult, so integration with DeepStream as an Isaac component provides a solution.

Internal integration of the open-source GStreamer framework and NVIDIA DeepStream SDK as an Isaac component allows you to reuse their collection of multimedia processing features. The Isaac and DeepStream products naturally complement each other for robotics and media use cases.

GPU-accelerated DeepStream elements can be used as part of the GStreamer pipeline definition. DeepStream can be installed with the Jetson JetPack installer for Jetson Nano and Xavier platforms. Follow the DeepStream installation instructions to make their elements available on your host system.

The media pipeline is attached to the Isaac engine via the named appsink and appsrc elements for receiving and transmitting, respectively. The component pipeline configuration parameter allows you to launch any GStreamer pipeline string with any configuration and elements. Pipelines may contain zero or more application sources or sinks. A pipeline containing no application source or sink is managed by Isaac but doesn’t share data, which is useful for indirect multimedia support for robotic applications.

The application elements support the capabilities, formats, and memory models with the equivalent Isaac protobuffer messages detailed in the table below. Use appropriate videoconvert and nvvideoconvert elements and caps filters for compatibility. The capabilities have to be explicitly defined.

The acquisition timestamp of the messages transmitted from the DeepStream component to other Isaac components is the node time at the moment of publication.

The component injects an Isaac scheduler thread pool into the GStreamer pipeline. The thread pool follows Isaac thread rules and GStreamer thread rules:

• The component runs a single service job for the pipeline main loop.

• The pipeline uses one or more jobs for its elements.

• One or more elements may run within a single job.

• An element may use multiple jobs.

• The elements use push- or pull-based scheduling for upstream or downstream threads, respectively.

• The jobs use Earliest Deadline First policy.

• Each job has an execution group configuration.

• A pipeline branch can be configured to run on a different thread with the queue element.

• The job statistics are reported in the scheduler report.

Below is an example job-statistics report for the gstreamer_pipeline_multi application. The pipeline is composed of three simple independent branches, which in turn run on three jobs.

|=========================================================================================| | Job Statistics Report (blocking) | |=========================================================================================| | Name | Job Mode | Count | Time (Median - 90% - Max) [ms] | |-----------------------------------------------------------------------------------------| | DeepStream Service | Blocking One Shot | 1 | 32816.30 | 32816.30 | 32816.30 | | DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.40 | 32814.40 | 32814.40 | | DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.42 | 32814.42 | 32814.42 | | DeepStream Pipeline Thread | Blocking One Shot | 1 | 32814.52 | 32814.52 | 32814.52 | |=========================================================================================|

The gst-inspect-1.0 command-line tool allows you to view the elements available on the host:

            

            
$ gst-inspect-1.0
video4linux2:  v4l2src: Video (video4linux2) Source
video4linux2:  v4l2sink: Video (video4linux2) Sink
video4linux2:  v4l2radio: Radio (video4linux2) Tuner
video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
dtls:  dtlsenc: DTLS Encoder
[...]
Total count: 265 plugins, 1420 features
        

The gst-launch-1.0 command-line tool allows you to build and run a GStreamer pipeline. Enable the debug output for GStreamer by setting the GST_DEBUG environment variable to a category level between 0 for no debug output and 8 for a full memory dump. The information level at 4 is usually the most practical.

            

            
$ export GST_DEBUG=4
# Your application will now output debug information when you run it.
        

Obtain a graph of your pipeline by setting the GST_DEBUG_DUMP_DOT_DIR environment variable to an existing, writable directory path. Use the dot tool to generate a viewable image.

            

            
$ sudo apt install graphviz
$ export GST_DEBUG_DUMP_DOT_DIR=/tmp
# Running your application will generate a FILENAME.dot file in the /tmp folder which can be
# converted by the following command:
$ dot -Tpng /tmp/FILENAME.dot > pipeline.png
        

The following example displays a test video from the GStreamer FAQ with debug information, graph drawing, and verbosity enabled:

            

            
$ GST_DEBUG=4 GST_DEBUG_DUMP_DOT_DIR=. gst-launch-1.0 -v \
  videotestsrc ! videoconvert ! autovideosink
        

The original GStreamer example has been altered to demonstrate how to integrate a GStreamer pipeline within an Isaac application. The Isaac example shows how the pipeline media can be imported or exported to Isaac communication channels available in the application configuration.

The pipeline has two bins: a test video source is brought into Isaac through an application sink, and an application source exposes a video stream to a display video sink. The capabilities of the application element are limited. As such a videoconvert element ensures support for the RGB raw format for both the source and sink elements. The pipeline thus changes to the following:

            

            
videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac

appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink
        

Isaac Application Examples

The following is a minimal Isaac application that passes the video from the sink to the source through Isaac:

            

            
{
  "name": "example_pipeline_app",
  "modules": [
    "deepstream",
  ],
  "graph": {
    "nodes": [
      {
        "name": "deepstream",
        "components": [
          {
            "name": "message_ledger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "pipeline",
            "type": "isaac::deepstream::Pipeline"
          }
        ]
      }
    ],
    "edges": [
      {
        "source": "deepstream/pipeline/to_isaac",
        "target": "deepstream/pipeline/from_isaac"
      }
    ]
  },
  "config": {
    "deepstream": {
      "pipeline": {
        "pipeline": "videotestsrc ! video/x-raw,format=RGB ! videoconvert ! appsink name=to_isaac  appsrc name=from_isaac ! video/x-raw,format=RGB ! videoconvert ! autovideosink"
      }
    }
  }
}
        

Visualizing an application sink using a video test source would look like the following in Isaac WebSight:

Other demonstration applications are available in the packages/deepstream/apps folder: