DeepStream#

DeepStream SDK is a complete streaming toolkit based on GStreamer for AI-based multi-sensor processing, video, audio, and image understanding. DeepStream is based on GStreamer. GStreamer is a multimedia framework that allows you to quickly build media pipelines by linking media processing elements, with support for many other streaming protocols through GStreamer (RIST, SRT, MPEG-TS).

DeepStream adds connectivity to AI inference and custom elements for ST 2110 uncompressed video support through Rivermax with PTP support, and NMOS registration and connection management capabilities.

DeepStream SDK is provided with reference C/C++ sample applications, including a DeepStream NMOS Application that provides both ST 2110 send and receive capabilities (code sample located in apps/sample_apps/deepstream-nmos, refer to the DeepStream Reference Application - deepstream-nmos app).

This section references a few useful practices and pipelines for DeepStream. You can also refer to the SDK Developer Guide for more details.

Plugins#

GStreamer has a multitude of plugins to support all kinds of streaming protocols. DeepStream adds NVIDIA optimized alternatives of some GStreamer plugins to leverage GPU at best or specific plugins for running AI inference or ST 2110 uncompressed media streaming. You can refer to the DeepStream plugin list and GStreamer plugin list to access a list of these plugins and their capabilities.

ST 2110 Plugins#

Gst-nvdsudpsrc and Gst-nvdsudpsink have been developed to, respectively, send and receive ST 2110 data with Rivermax.

To make ST 2110 on top of any other types of streaming easy to integrate into Holoscan for Media, we created the Media Gateway reference container . The Media Gateway reference container includes a GStreamer plugin called nvdsnmosbin, which creates and controls pipelines using one or more nvdsudpsrc and nvdsudpsink``elements. It provides NMOS dynamic connection support based on AMWA IS-05. This container can be used with any NMOS controller, including Holoscan for Media `NMOS Controller <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/holoscan-for-media/containers/controller-ui>`__ for developers, which can be deployed with the `NMOS Registry <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/holoscan-for-media/containers/nmos-cpp>`__. Media Gateway can be fully reconfigured by using different pipelines in its Helm ``values.yaml file.

Reference Pipelines#

Using the different DeepStream and GStreamer plugins, developers can build pipelines relevant for their workflow. We are listing here a few reference pipelines that are relevant for Holoscan for Media developers. DeepStream plugins can be connected to form a streaming pipeline.

The nvdsnmosbin element leverages the following properties:

  • description: a description for the NMOS node/device (for example, "1080p60-colorbars")

  • domain: the DNS domain to be used for NMOS (for example, local for mDNS-based discovery and registration)

  • hostname: the hostname for the NMOS server (for example, colorbars.local)

  • http-port: the port on which the NMOS server is running, also used for liveness/readiness probes in a Kubernetes deployment (for example, 9010)

  • label: a label for the node/device viewable in the NMOS Controller (for example, "Node1")

  • registration-url: the URL for a fixed IS-04 Registration API (in this case DNS-SD is disabled)

  • seed: a string which is used to generate reproducible NMOS identifiers each time the same seed is used

  • system-url: the URL for a fixed IS-09 System API

Let’s take an example of the pipeline here that generates uncompressed 1080p60 colorbars video:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink payload-size=1220 packets-per-line=4 ptp-src=set" \
    sink_0::sdp="file:///sdps/video"

This pipeline can be understood in two parts:

  • The actual pipeline: Starting with videotestsrc and ending with nmos.sink_0, this section configures the actual GStreamer pipeline from the videotestsrc generator element and passes through the caps (or capabilities) filter to negotiate the appropriate format and is then connected to the nvdsnmosbin element through nmos.sink_0.

  • ``nvdsnmosbin`` configuration: Starting nvdsnmosbin, this section configures the NMOS senders/receivers, using ptp-src=set to set the PTP source clock to a Mellanox interface available to the container.

The nvdsnmosbin GStreamer element can create multiple senders and receivers for a single NMOS device that it is advertising. For example, to add a receiver to the above NMOS node something like the below can be used:

gst-launch-1.0 \

  nvdsnmosbin name=nmos dump-graphviz=false hostname=passthrough.local \

  src_1::sdp=file:///sdps/video \
  src_1::nvdsudpsrc-desc="nvdsudpsrc \
    header-size=20 \
    payload-size=1200 \
    gpu-id=0" \

  sink_1::sdp=file:///sdps/video \
  sink_1::nvdsudpsink-desc="nvdsudpsink \
    payload-size=1220 \
    packets-per-line=4" \

  nmos.src_1 \
  ! "video/x-raw(memory:NVMM),format=UYVP,width=1920,height=1080,framerate=60/1" \
  ! nvvideoconvert \
  ! video/x-raw,format=UYVP,width=1920,height=1080,framerate=60/1 \
  ! queue \
  ! nmos.sink_1

1080p60 Color-Bars Sender#

This pipeline can be used to send live color bar content over ST 2110.

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 } \
      ptp-src=set" \
    sink_0::sdp="file:///sdps/video"

1080p60 Color-Bars Sender Using GPUDirect#

This pipeline can be used to send live color bar content over ST 2110 using GPUDirect.

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! nvvideoconvert ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=60/1" \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 } \
      ptp-src=set gpu-id=0" \
    sink_0::sdp="file:///sdps/video"

1080p60 File Receiver#

This pipeline can be used to receive live ST 2110 content and store it in a file.

gst-launch-1.0 \
  receiver.src_0 \
  ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=60/1" \
  ! nvvideoconvert \
  ! nvv4l2h264enc \
  ! mpegtsmux \
  ! filesink location=/workspace/recv.mp4 \
  nvdsnmosbin name=receiver label=file-receiver hostname=file-receiver.local \
    description="1080p60 file receiver" \
    src_0::nvdsudpsrc-desc="nvdsudpsrc \
      header-size=20 \
      payload-size=1200 \
      gpu-id=0" \
    src_0::rtp-caps="application/x-rtp, \
      media=video, payload=96, clock-rate=90000, encoding-name=raw, \
      depth=(string)10, sampling=YCbCr-4:2:2, \
      width=(string)1920, height=(string)1080, \
      exactframerate=(string)60, colorimetry=BT709, \
      PM=(string)2110GPM, SSN=(string)ST2110-20:2017, \
      TP=(string)2110TPN" \
    src_0::session-name="My File Receiver"

Multi-Sender#

A single pipeline can be used to send multiple streams at the same time as shown in the example below:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  videotestsrc pattern=smpte is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_1 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 ptp-src=set \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 }" \
    sink_0::sdp="file:///sdps/video" \
    sink_1::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 ptp-src=set \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=2 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=3 }" \
    sink_1::sdp="file:///sdps/video"

ST 2110 to SRT#

SRT is often used for secure media transmission. The pipeline below streams a 1080p29.97 ST 2110 input encoded in H264 as SRT:

gst-launch-1.0 \
    receiver0.src_0 \
    ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=30000/1001" \
    ! nvvideoconvert \
    ! nvv4l2h264enc \
    ! h264parse config-interval=1 \
    ! mpegtsmux alignment=7 name=mux0 \
    ! srtsink uri=srt://:9001 \
    nvdsnmosbin name=receiver0 hostname=receiver0.local label="Receive to SRT" http-port="2201" \
      src_0::nvdsudpsrc-desc="nvdsudpsrc header-size=20 payload-size=1200 gpu-id=0" \
      src_0::rtp-caps="application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=raw, depth=(string)10, sampling=YCbCr-4:2:2, width=(string)1920, height=(string)1080, exactframerate=(string)30000/1001, colorimetry=BT709, PM=(string)2110GPM, SSN=(string)ST2110-20:2017, TP=(string)2110TPN" \
      src_0::session-name="Video 1080p29.97"

Audio Receiver and Sender#

The following SDP file can be used for audio workflow:

v=0
o=- 955731720 0 IN IP4 <session-ip>
s=loop audio
t=0 0
m=audio 4000 RTP/AVP 97
c=IN IP4 <destination-ip>/64
a=source-filter: incl IN IP4 <destination-ip> <source-ip>
a=rtpmap:97 L24/48000/2
a=fmtp:97 channel-order=SMPTE2110.(ST)
a=ptime:1.000
a=ts-refclk:ptp=IEEE1588-2008:traceable
a=mediaclk:direct=0

Here is an example of an audio ST 2110 sender using the SDP:

gst-launch-1.0 \
   audiotestsrc \
       ! "audio/x-raw, format=S24BE, rate=48000, channels=2" \
       ! queue \
       ! nvdsudpsink host=<destination-ip> port=4000 local-iface-ip=<local-iface-ip> sdp-file=/workspace/audio

Here is an example of an audio ST 2110 sender using an audio generator:

gst-launch-1.0 \
  audiotestsrc wave=sine \
      ! audio/x-raw,format=S24BE,channels=2,layout=interleaved,rate=48000 \
      ! nmos.sink_0 \
      nvdsnmosbin name=nmos dump-graphviz=true label="2channel-sine-wave-audio" \
        description="2 channel sine wave audio sender" \
        domain=local \
        sink_0::nvdsudpsink-desc="nvdsudpsink \
          payload-size=300" \
        sink_0::session-name="2ch-48k-sine-wave" \
    sink_0::sdp="file:///sdps/audio"

Here is an example of an audio ST 2110 receiver into a file as source:

gst-launch-1.0 \
  receiver.src_0 \
      ! audioconvert \
      ! opusenc \
      ! mpegtsmux alignment=7 name=mux \
      ! filesink location=/workspace/recv.mp4 \
      nvdsnmosbin name=receiver label=audio-file-receiver domain=local \
        description="2 channel audio file receiver" \
        src_0::nvdsudpsrc-desc="nvdsudpsrc payload-size=288 header-size=12 payload-multiple=16" \
        src_0::session-name="2ch-48k-to-file" \
        src_0::sdp="file:///sdps/audio"

Media Gateway#

The Media Gateway reference container is a containerized application built on DeepStream. Its Helm chart values.yaml supports a description property that can be any DeepStream pipeline using the nvdsnmosbin element, or one of the built-in example configurations included in the container and described above, that is 1080p60-sender, 1080p60-receiver, 1080p60-flipper multi1080p60-sender, 2ch-48k-receiver, and 2ch-48k-sender.

You can call one of the example pipelines provided by using the filename stem as the description or you can fill in the description with a custom pipeline. Below is an example of a full custom deployment, including a pipeline that converts 50i video (25 interlaced frames per second) to 50p (50 progressive frames per second).

framerate: 50
height: 1080
width: 1920
mcsrc: "232.128.1.1"
mcport: 20000

imagePullSecrets:
    - name: <your-ngc-secret>
pipelines:
    pipeline1:
        GST_DEBUG: 2,nvdsnmosbin:8,nvdsnmosnode:2,nvdssdp*:8
        description: |
              nmosv.src_1
              ! video/x-raw(memory:NVMM),width={{ .Values.width }},height={{ .Values.height }},format=UYVP
              ! queue max-size-buffers=4 leaky=downstream
              ! nvvideoconvert
              ! deinterlace fields=0 mode=1 method=6 locking=3
              ! nvvideoconvert
              ! video/x-raw,format=UYVP,width={{ .Values.width }},height={{ .Values.height }}
              ! nmosv.sink_1
              nvdsnmosbin name=nmosv dump-graphviz=true hostname=conv.local label=deinterlace-{{ .Values.framerate }}i-{{ .Values.framerate }}p
                src_1::sdp="file:///sdps/video"
                src_1::nvdsudpsrc-desc="nvdsudpsrc header-size=20 payload-size=1200 gpu-id=0"
                sink_1::rtp-caps="application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=raw, depth=(string)10, sampling=YCbCr-4:2:2, width=(string){{ .Values.width }}, height=(string){{ .Values.height }}, exactframerate=(string){{ .Values.framerate }}, colorimetry=BT709, PM=(string)2110GPM, SSN=(string)ST2110-20:2017, TP=(string)2110TPN"
                sink_1::session-name="out-{{ .Values.framerate }}p"
                sink_1::nvdsudpsink-desc="nvdsudpsink payload-size=1220 packets-per-line=4 internal-thread-core=0 render-thread-core=1"
        env:
            - name: CPU_AFFINITY
              value: 2-3
        highSpeedNetwork:
            ip: null
            name: media-a-tx-net
        name: file-receiver
        resources:
            limits:
                cpu: 4
                hugepages-2Mi: 4Gi
                memory: 8Gi
                openshift.io/media_a_tx_pool: 1
            requests:
                cpu: 4
                hugepages-2Mi: 4Gi
                memory: 8Gi
                openshift.io/media_a_tx_pool: 1
        sdps:
            video: |-
                v=0
                o=- 1443716955 1443716955 IN IP4 0.0.0.0
                s=in-{{ .Values.framerate }}i
                t=0 0
                m=video {{ .Values.mcport }} RTP/AVP 96
                c=IN IP4 {{ .Values.mcsrc }}/64
                a=source-filter: incl IN IP4 {{ .Values.mcsrc }} 0.0.0.0
                a=rtpmap:96 raw/90000
                a=fmtp:96 sampling=YCbCr-4:2:2; width={{ .Values.width }}; height={{ .Values.height }}; exactframerate={{ div .Values.framerate 2 }}; depth=10; TCS=SDR; colorimetry=BT709; interlace; PM=2110GPM; SSN=ST2110-20:2017; TP=2110TPN; PAR=1:1;
                a=ts-refclk:ptp=IEEE1588-2008:traceable
                a=mediaclk:direct=0

Debugging Tips#

When running Media Gateway or nvdsnmosbin, the container will automatically be restarted by Kubernetes, deleting the logs. To test and debug DeepStream pipelines without encountering container reboot, deploy a Media Gateway Container using just nvdsnmosbin in the pipeline description of the Helm chart as below:

imagePullSecrets:
  - name: ngc-api-key
pipelines:
  pipeline1:
      description: nvdsnmosbin

The Media Gateway Container will be launched with this basic pipeline. You can start an interactive terminal session in the container using k9s or the associated command. You can then launch reference pipelines using the scripts in the example-gst-launch-commands directory, or write your own pipeline taking inspiration from these examples.

You can follow these recommendations:

  • For these pipelines to properly run, the http-port property of each nvdsnmosbin element should be specified with a different value from the default (9010), as shown in the stream to SRT example.

  • Every script stored in the /workspace/example-gst-launch-commands/ directory as a <pipeline>.sh file can be referenced like description: <pipeline> in the Helm values, so it can be useful to mount a persistent volume in place of this directory to store custom debug pipelines and avoid losing changes at container restart or reinstall or at cluster reboot.

  • Mounting a new persistent volume will override the existing /workspace/example-gst-launch-commands/ directory content. The persistent volume could be mounted to another location first to copy /workspace/example-gst-launch-commands/ existing content manually and ensure the built-in examples can also still be called from Helm Dashboard after the persistent volume is mounted at /workspace/example-gst-launch-commands/.