Is this page helpful?

DeepStream#

The NVIDIA DeepStream SDK is a complete streaming toolkit based on GStreamer for AI-based multi-sensor processing, video, audio, and image understanding. GStreamer is a multimedia framework that lets you quickly build media pipelines by linking media processing elements. GStreamer also supports many other streaming protocols, such as Reliable Internet Stream Transport (RIST), Secure Reliable Transport (SRT), and MPEG transport stream (MPEG-TS).

DeepStream adds connectivity to AI inference. It also adds custom elements for uncompressed ST 2110 video through Rivermax with Precision Time Protocol (PTP) support, and for Networked Media Open Specifications (NMOS) registration and connection management.

The DeepStream SDK includes reference C and C++ sample applications, such as a DeepStream NMOS application that can both send and receive ST 2110. The code sample is in apps/sample_apps/deepstream-nmos. For more information, refer to the DeepStream Reference Application - deepstream-nmos app.

This section describes a few useful practices and pipelines for DeepStream. For more information, refer to the SDK Developer Guide.

Plugins#

GStreamer has many plugins that support a wide range of streaming protocols. DeepStream adds NVIDIA-optimized alternatives to some GStreamer plugins to make the best use of the GPU. It also adds plugins for running AI inference and for uncompressed ST 2110 media streaming. For a list of these plugins and their capabilities, refer to the DeepStream plugin list and the GStreamer plugin list.

ST 2110 Plugins#

The Gst-nvdsudpsrc plugin receives ST 2110 data with Rivermax, and the Gst-nvdsudpsink plugin sends it.

To make ST 2110 easy to integrate with other types of streaming in Holoscan for Media, NVIDIA provides the Media Gateway reference container. The Media Gateway reference container includes a GStreamer plugin called nvdsnmosbin, which creates and controls pipelines that use one or more nvdsudpsrc and nvdsudpsink elements. The plugin provides NMOS dynamic connection support based on Advanced Media Workflow Association (AMWA) IS-05. You can use this container with any NMOS controller, including the Holoscan for Media NMOS Controller for developers, which you can deploy with the NMOS Registry. To fully reconfigure Media Gateway, use different pipelines in its Helm values.yaml file.

Reference Pipelines#

You can connect DeepStream and GStreamer plugins to form streaming pipelines for your workflow. This section lists a few reference pipelines that are relevant for Holoscan for Media developers.

The nvdsnmosbin element has the following properties:

  • description: a description for the NMOS node and device (for example, "1080p60-colorbars")

  • domain: the DNS domain for NMOS (for example, local for mDNS-based discovery and registration)

  • hostname: the hostname for the NMOS server (for example, colorbars.local)

  • http-port: the port that the NMOS server runs on, which liveness and readiness probes in a Kubernetes deployment also use (for example, 9010)

  • label: a label for the node and device that appears in the NMOS Controller (for example, "Node1")

  • registration-url: the URL for a fixed IS-04 Registration API (this setting disables DNS-SD)

  • seed: a string that generates the same NMOS identifiers each time you use the same seed

  • system-url: the URL for a fixed IS-09 System API

The following example pipeline generates uncompressed 1080p60 color bars video:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink payload-size=1220 packets-per-line=4 ptp-src=set" \
    sink_0::sdp="file:///sdps/video"

This pipeline has two parts:

  • The GStreamer pipeline: This part starts with videotestsrc and ends with nmos.sink_0. It starts at the videotestsrc generator element and passes through the caps (capabilities) filter, which negotiates the appropriate format. It then connects to the nvdsnmosbin element through nmos.sink_0.

  • The nvdsnmosbin configuration: This part starts with nvdsnmosbin and configures the NMOS senders and receivers. The ptp-src=set property sets the PTP source clock to an NVIDIA network interface that is available to the container.

The nvdsnmosbin GStreamer element can create multiple senders and receivers for a single NMOS device that it advertises. For example, to add a receiver to the preceding NMOS node, use a pipeline similar to the following:

gst-launch-1.0 \

  nvdsnmosbin name=nmos dump-graphviz=false hostname=passthrough.local \

  src_1::sdp=file:///sdps/video \
  src_1::nvdsudpsrc-desc="nvdsudpsrc \
    header-size=20 \
    payload-size=1200 \
    gpu-id=0" \

  sink_1::sdp=file:///sdps/video \
  sink_1::nvdsudpsink-desc="nvdsudpsink \
    payload-size=1220 \
    packets-per-line=4" \

  nmos.src_1 \
  ! "video/x-raw(memory:NVMM),format=UYVP,width=1920,height=1080,framerate=60/1" \
  ! nvvideoconvert \
  ! video/x-raw,format=UYVP,width=1920,height=1080,framerate=60/1 \
  ! queue \
  ! nmos.sink_1

1080p60 Color-Bars Sender#

Use this pipeline to send live color bar content over ST 2110:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 } \
      ptp-src=set" \
    sink_0::sdp="file:///sdps/video"

1080p60 Color-Bars Sender Using GPUDirect#

Use this pipeline to send live color bar content over ST 2110 with GPUDirect:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! nvvideoconvert ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=60/1" \
  ! nmos.sink_0 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 } \
      ptp-src=set gpu-id=0" \
    sink_0::sdp="file:///sdps/video"

1080p60 File Receiver#

Use this pipeline to receive live ST 2110 content and store it in a file:

gst-launch-1.0 \
  receiver.src_0 \
  ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=60/1" \
  ! nvvideoconvert \
  ! nvv4l2h264enc \
  ! mpegtsmux \
  ! filesink location=/workspace/recv.mp4 \
  nvdsnmosbin name=receiver label=file-receiver hostname=file-receiver.local \
    description="1080p60 file receiver" \
    src_0::nvdsudpsrc-desc="nvdsudpsrc \
      header-size=20 \
      payload-size=1200 \
      gpu-id=0" \
    src_0::rtp-caps="application/x-rtp, \
      media=video, payload=96, clock-rate=90000, encoding-name=raw, \
      depth=(string)10, sampling=YCbCr-4:2:2, \
      width=(string)1920, height=(string)1080, \
      exactframerate=(string)60, colorimetry=BT709, \
      PM=(string)2110GPM, SSN=(string)ST2110-20:2017, \
      TP=(string)2110TPN" \
    src_0::session-name="My File Receiver"

Multi-Sender#

A single pipeline can send multiple streams at the same time, as shown in the following example:

gst-launch-1.0 \
  videotestsrc pattern=smpte100 is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_0 \
  videotestsrc pattern=smpte is-live=true horizontal-speed=2 \
  ! video/x-raw,width=1920,height=1080,format=UYVP,framerate=60/1 \
  ! nmos.sink_1 \
  nvdsnmosbin name=nmos dump-graphviz=true label=1080p60-colorbars \
    description="nmos sender producing 1080p60 colorbars" \
    hostname="simple-video.local" \
    sink_0::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 ptp-src=set \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=0 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=1 }" \
    sink_0::sdp="file:///sdps/video" \
    sink_1::nvdsudpsink-desc="nvdsudpsink \
      payload-size=1220 \
      packets-per-line=4 ptp-src=set \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+internal-thread-core=2 } \
      ${NVDSNMOS_ORIGINAL_CORE_SET:+render-thread-core=3 }" \
    sink_1::sdp="file:///sdps/video"

ST 2110 to SRT#

SRT is a common protocol for secure media transmission. The following pipeline encodes a 1080p29.97 ST 2110 input in H.264 and streams it over SRT:

gst-launch-1.0 \
    receiver0.src_0 \
    ! "video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVP,framerate=30000/1001" \
    ! nvvideoconvert \
    ! nvv4l2h264enc \
    ! h264parse config-interval=1 \
    ! mpegtsmux alignment=7 name=mux0 \
    ! srtsink uri=srt://:9001 \
    nvdsnmosbin name=receiver0 hostname=receiver0.local label="Receive to SRT" http-port="2201" \
      src_0::nvdsudpsrc-desc="nvdsudpsrc header-size=20 payload-size=1200 gpu-id=0" \
      src_0::rtp-caps="application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=raw, depth=(string)10, sampling=YCbCr-4:2:2, width=(string)1920, height=(string)1080, exactframerate=(string)30000/1001, colorimetry=BT709, PM=(string)2110GPM, SSN=(string)ST2110-20:2017, TP=(string)2110TPN" \
      src_0::session-name="Video 1080p29.97"

Audio Receiver and Sender#

You can use the following Session Description Protocol (SDP) file for audio workflows:

v=0
o=- 955731720 0 IN IP4 <session-ip>
s=loop audio
t=0 0
m=audio 4000 RTP/AVP 97
c=IN IP4 <destination-ip>/64
a=source-filter: incl IN IP4 <destination-ip> <source-ip>
a=rtpmap:97 L24/48000/2
a=fmtp:97 channel-order=SMPTE2110.(ST)
a=ptime:1.000
a=ts-refclk:ptp=IEEE1588-2008:traceable
a=mediaclk:direct=0

The following example is an ST 2110 audio sender that uses the SDP file:

gst-launch-1.0 \
   audiotestsrc \
       ! "audio/x-raw, format=S24BE, rate=48000, channels=2" \
       ! queue \
       ! nvdsudpsink host=<destination-ip> port=4000 local-iface-ip=<local-iface-ip> sdp-file=/workspace/audio

The following example is an ST 2110 audio sender that uses an audio generator:

gst-launch-1.0 \
  audiotestsrc wave=sine \
      ! audio/x-raw,format=S24BE,channels=2,layout=interleaved,rate=48000 \
      ! nmos.sink_0 \
      nvdsnmosbin name=nmos dump-graphviz=true label="2channel-sine-wave-audio" \
        description="2 channel sine wave audio sender" \
        domain=local \
        sink_0::nvdsudpsink-desc="nvdsudpsink \
          payload-size=300" \
        sink_0::session-name="2ch-48k-sine-wave" \
    sink_0::sdp="file:///sdps/audio"

The following example is an ST 2110 audio receiver that writes the received stream to a file:

gst-launch-1.0 \
  receiver.src_0 \
      ! audioconvert \
      ! opusenc \
      ! mpegtsmux alignment=7 name=mux \
      ! filesink location=/workspace/recv.mp4 \
      nvdsnmosbin name=receiver label=audio-file-receiver domain=local \
        description="2 channel audio file receiver" \
        src_0::nvdsudpsrc-desc="nvdsudpsrc payload-size=288 header-size=12 payload-multiple=16" \
        src_0::session-name="2ch-48k-to-file" \
        src_0::sdp="file:///sdps/audio"

Media Gateway#

The Media Gateway reference container is a containerized application built on DeepStream. Its Helm chart values.yaml file supports a description property. The value can be any DeepStream pipeline that uses the nvdsnmosbin element, or one of the following built-in example configurations in the container: 1080p60-sender, 1080p60-receiver, 1080p60-flipper, multi1080p60-sender, 2ch-48k-receiver, and 2ch-48k-sender.

To use one of the example pipelines, set description to its file name stem. Alternatively, set description to a custom pipeline. The following example shows a full custom deployment, including a pipeline that converts 50i video (25 interlaced frames per second) to 50p (50 progressive frames per second):

framerate: 50
height: 1080
width: 1920
mcsrc: "232.128.1.1"
mcport: 20000

imagePullSecrets:
    - name: <your-ngc-secret>
pipelines:
    pipeline1:
        GST_DEBUG: 2,nvdsnmosbin:8,nvdsnmosnode:2,nvdssdp*:8
        description: |
              nmosv.src_1
              ! video/x-raw(memory:NVMM),width={{ .Values.width }},height={{ .Values.height }},format=UYVP
              ! queue max-size-buffers=4 leaky=downstream
              ! nvvideoconvert
              ! deinterlace fields=0 mode=1 method=6 locking=3
              ! nvvideoconvert
              ! video/x-raw,format=UYVP,width={{ .Values.width }},height={{ .Values.height }}
              ! nmosv.sink_1
              nvdsnmosbin name=nmosv dump-graphviz=true hostname=conv.local label=deinterlace-{{ .Values.framerate }}i-{{ .Values.framerate }}p
                src_1::sdp="file:///sdps/video"
                src_1::nvdsudpsrc-desc="nvdsudpsrc header-size=20 payload-size=1200 gpu-id=0"
                sink_1::rtp-caps="application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=raw, depth=(string)10, sampling=YCbCr-4:2:2, width=(string){{ .Values.width }}, height=(string){{ .Values.height }}, exactframerate=(string){{ .Values.framerate }}, colorimetry=BT709, PM=(string)2110GPM, SSN=(string)ST2110-20:2017, TP=(string)2110TPN"
                sink_1::session-name="out-{{ .Values.framerate }}p"
                sink_1::nvdsudpsink-desc="nvdsudpsink payload-size=1220 packets-per-line=4 internal-thread-core=0 render-thread-core=1"
        env:
            - name: CPU_AFFINITY
              value: 2-3
        highSpeedNetwork:
            ip: null
            name: media-a-tx-net
        name: file-receiver
        resources:
            limits:
                cpu: 4
                hugepages-2Mi: 4Gi
                memory: 8Gi
                openshift.io/media_a_tx_pool: 1
            requests:
                cpu: 4
                hugepages-2Mi: 4Gi
                memory: 8Gi
                openshift.io/media_a_tx_pool: 1
        sdps:
            video: |-
                v=0
                o=- 1443716955 1443716955 IN IP4 0.0.0.0
                s=in-{{ .Values.framerate }}i
                t=0 0
                m=video {{ .Values.mcport }} RTP/AVP 96
                c=IN IP4 {{ .Values.mcsrc }}/64
                a=source-filter: incl IN IP4 {{ .Values.mcsrc }} 0.0.0.0
                a=rtpmap:96 raw/90000
                a=fmtp:96 sampling=YCbCr-4:2:2; width={{ .Values.width }}; height={{ .Values.height }}; exactframerate={{ div .Values.framerate 2 }}; depth=10; TCS=SDR; colorimetry=BT709; interlace; PM=2110GPM; SSN=ST2110-20:2017; TP=2110TPN; PAR=1:1;
                a=ts-refclk:ptp=IEEE1588-2008:traceable
                a=mediaclk:direct=0

Debugging Tips#

When you run Media Gateway or nvdsnmosbin, Kubernetes automatically restarts the container, which deletes the logs. To test and debug DeepStream pipelines without container restarts, deploy a Media Gateway container with only nvdsnmosbin in the pipeline description of the Helm chart:

imagePullSecrets:
  - name: ngc-api-key
pipelines:
  pipeline1:
      description: nvdsnmosbin

The Media Gateway container starts with this basic pipeline. You can start an interactive terminal session in the container using k9s or the associated command. You can then launch reference pipelines using the scripts in the example-gst-launch-commands directory, or write your own pipeline based on these examples.

Follow these recommendations:

  • For these pipelines to run properly, set the http-port property of each nvdsnmosbin element to a value other than the default (9010), as shown in the stream to SRT example.

  • You can reference any <pipeline>.sh script in the /workspace/example-gst-launch-commands/ directory as description: <pipeline> in the Helm values. To store custom debug pipelines and keep your changes after a container restart, a reinstall, or a cluster reboot, mount a persistent volume in place of this directory.

  • Mounting a new persistent volume overrides the existing content of the /workspace/example-gst-launch-commands/ directory. To keep the built-in examples available from the Helm Dashboard, first mount the persistent volume at another location, and manually copy the existing /workspace/example-gst-launch-commands/ content to it. Then mount the persistent volume at /workspace/example-gst-launch-commands/.