.. _SD.Multimedia.MultimediApis:

.. include:: /content/swdocs.rsts

.. spelling::
   cuDNN
   Multicamera
   Multivideo

Multimedia APIs
!!!!!!!!!!!!!!!

The Multimedia API is a collection of low-level APIs that support
flexible application development. These low-level APIs enable
flexibility by providing better control over the underlying hardware
blocks.

The Multimedia API includes:

-  ``libargus`` for imaging applications
-  V4L2 API for encoding, decoding, scaling, and other media functions
-  NVOSD for On-Screen display
-  Buffer Utility for buffer allocation, management, and sharing, transform, composition, and blending

Example applications are provided to demonstrate:

-  Video decode (dual decode support with NVDEC)
-  Video encode (dual encode support with NVENC)
-  Video decode and DRM based render
-  Video convert
-  Video decode with multi-channels
-  Multivideo decode (decoding of multiple video streams in parallel)
-  JPEG decode and JPEG encode
-  Image and video processing with |NVIDIA(r)| CUDA\ |reg|
-  Camera JPEG capture and video record
-  Camera capture and CUDA processing
-  Multicamera capture with composition
-  Object detection and classification with cuDNN
-  |NVIDIA(r)| TensorRT\ |tm| and OpenCV usage

The Multimedia API provides another application development path for
those not using a framework such as GStreamer, or for developers
leveraging custom frameworks.

.. todo::
   The statement above appears to point in one direction -- away from GStreamer -- and the following sections go off in the opposite direction. New readers will be hopelessly confused. Since GStreamer appears to be an important part of the Multimedia API Way, we should say something that's more appropriate.

The multimedia API is described in the *Jetson Linux API Reference*, available for download from the `Jetson Download Center <https://developer.nvidia.com/embedded/downloads>`_.

Installing GStreamer
@@@@@@@@@@@@@@@@@@@@

.. todo:: Logically I think this section should come later, with the other sections that concern GStreamer.

Download and install GStreamer directly on the target. The wrapper
library, ``gst-openmax``, is an interface between GStreamer and OpenMAX. It
enables accelerated NVIDIA plug-ins to operate in the GStreamer framework.

For more information about GStreamer, see the `GStreamer web site <https://gstreamer.freedesktop.org>`__ sponsored by `freedesktop.org <https://www.freedesktop.org>`__.

Multimedia Demo Applications
@@@@@@@@@@@@@@@@@@@@@@@@@@@@

``NvGstPlayer`` and ``NvGstCapture`` are applications that demonstrate and test GStreamer-based multimedia functionality. ``NvGstPlayer`` is a multimedia player. ``NvGstCapture`` is a multimedia capture application that generates image and video output files.

Before you run ``NvGstPlayer`` and ``NvGstCapture``, complete the prerequisite steps and install GStreamer as directed in the README file included in the release.

.. todo::
   When I first read the above, I thought, "OK, the next section will explain the prerequisites... late, since the *preceding* section documented GStreamer installation, but I can fix that." When I reached the end I realized that *nothing* ever explains the prerequisites. Are there any? If so, we need to explain them -- before the installation instructions, not after. If not, we shouldn't mention them.

About NvGstPlayer
#################

``NvGstPlayer`` is a media file player that runs from the command line. It plays audio and video files encapsulated in these formats:

-  MP4
-  3GP
-  AVI
-  ASF
-  WMA
-  MKV
-  M2TS
-  WEBM
-  MOV

``NvGstPlayer`` supports local file playback and playback over RSTP, HTTP, and UDP.

For more information about ``NvGstPlayer``, see the README ``nvgstplayer-1.0_README.txt``.

.. note::
   The ``gst-omx`` plugin is deprecated as of |NVIDIA(r)| |Jetson(tm)| Linux release 32.1. Use the ``gst-v4l2`` plugin for development instead.

   .. todo::
      If it was deprecated in 32.1, it would normally be unsupported in the next minor release. Has that happened? If so, it shouldn't be mentioned at all.

About NvGstCapture
##################

``NvGstCapture`` is a command line camera capture application. ``NvGstCapture``
captures video data using a camera and encapsulates encoded video data in a
container file.

``NvGstCapture`` supports the ARGUS API using the ``nvarguscamerasrc`` plugin.

.. note::
   Use of the legacy ``nvcamerasrc`` plugin with NvGstCapture is deprecated.

.. todo::
   Again, when was this (or will this be) dropped?

For more information about NvGstCapture, see the README ``nvgstcapture-1.0_README.txt``.