NVIDIA Jetson Linux Developer Guide

32.4.2 Release

Multimedia

Multimedia API

GStreamer-based Multimedia Playback

GStreamer API

GStreamer-based Camera Capture

Accelerated Decode with ffmpeg

Accelerated GStreamer

NVIDIA® Jetson™ Linux Driver Package (L4T) includes the multimedia framework modules for testing purposes.

For GStreamer information, see Accelerated GStreamer.

Multimedia API

The Multimedia API is a collection of low-level APIs that support flexible application development. These low-level APIs enable flexibility by providing better control over the underlying hardware blocks.

The Multimedia API includes:

• libargus for imaging applications

• V4L2 API for encoding, decoding, scaling, and other media functions

• NVOSD for On-Screen display

• Buffer Utility for buffer allocation, management, and sharing, transform, composition, and blending

Example applications are provided to demonstrate:

• Video decode (dual decode support with NVDEC)

• Video encode (dual encode support with NVENC)

• Video decode and DRM based render

• Video convert

• Video decode with multi-channels

• Multivideo decode (decoding of multiple video streams in parallel)

• JPEG decode and JPEG encode

• Image/video processing with CUDA

• Camera JPEG capture and video record

• Camera capture and CUDA processing

• Multicamera capture with composition

• Object detection and classification with cuDNN

• TensorRT and OpenCV usage

The Multimedia API provides another application development path for those not using a framework such as GStreamer, or for developers leveraging custom frameworks.

The Multimedia API reference is available for download at:

https://developer.nvidia.com/embedded/downloads

GStreamer-based Multimedia Playback

Installing GStreamer

Using NvGstPlayer

Use the GStreamer open source multimedia framework and the NvGstPlayer utility for testing multimedia local playback and HTTP/RTSP streaming playback use cases. The NvGstPlayer can be used as a reference implementation.

For more information about the NvGstPlayer application, refer to the ReadMe file included with the release.

Installing GStreamer

Download and install GStreamer directly on the target. The wrapper library, gst-openmax, is an interface between GStreamer and OpenMAX. It enables accelerated NVIDIA plug-ins in the GStreamer framework.

For more information about GStreamer, consult:

http://gstreamer.freedesktop.org

NvGstPlayer is a multimedia player test application.

Before running the NvGstPlayer and NvGstCapture applications, complete prerequisite steps and install GStreamer as directed in the README file available at:

nvgstplayer-1.0_README.tx/nvgstcapture_README.txt

Using NvGstPlayer

Note:

The gst-omx plugin is deprecated in L4T Release 32.1. Use the gst-v4l2 plugin for development instead.

NvGstPlayer is a command line media file player. It plays audio/video files encapsulated in:

• MP4

• 3GP

• AVI

• ASF

• WMA

• MKV

• M2TS

• WEBM

• MOV

NvGstPlayer supports local file playback and playback over RSTP, HTTP, and UDP.

For information consult the README:

nvgstplayer-1.0_README.txt

GStreamer API

GStreamer default support is migrated to version 1.14.0 on Ubuntu 18.04.

GStreamer-based Camera Capture

NvGstCapture is a command line Camera capture application. NvGstCapture captures video data using a camera and encapsulate encoded Video data in the container file.

For information, consult the README:

nvgstcapture-1.0_README.txt

The nvgstcapture application supports the ARGUS API using the nvarguscamerasrc plugin.

Note:

Usage of legacy nvcamerasrc with the nvgstcapture application is deprecated.

Accelerated Decode with ffmpeg

Decode Functional Flow

The NVIDIA ffmpeg package supports hardware-accelerated decode on Jetson platforms.

An application can use accelerated decode to read video files in the following elementary/container formats and dump them in YUV 420 format:

• H.264

• H.265

• VP8

• VP9

• MPEG2

• MPEG4

Note:

The ffmpeg package does not support MPEG4 container files.

To install the ffmpeg binary package

• Enter these commands:

$ sudo apt install ffmpeg

To get source files for the ffmpeg package

• Enter this command:

$ apt source ffmpeg

To include the ffmpeg library in L4T builds

1. Add the following lines to /etc/apt/sources.list:

$ echo "deb https://repo.download.nvidia.com/jetson/ffmpeg main main" | sudo tee -a /etc/apt/sources.list

$ echo "deb-src https://repo.download.nvidia.com/jetson/ffmpeg main main" | sudo tee -a /etc/apt/sources.list

2. Enter the command:

$ sudo apt update

Decode Functional Flow

This section describes the function flow of the ffmpeg decoding process:

3. Call nvv4l2dec_init_decoder() to create a new V4L2 Video Decoder object on the device node /dev/nvhost-nvdec.

4. Call subscribe_event() to subscribe to resolution change events.

5. Call set_output_plane_format() to set the format on the output plane.

6. Call capture_thread() to start a capture thread.

7. Call set_capture_plane_format() to set the format on the capture plane.

8. Call nvv4l2dec_decode() to read buffers from ffmpeg and start the decode process.

9. Call nvv4l2dec_decoder_get_frame() to get the hardware-accelerated decoded data and pass it to ffmpeg for dumping.

10. Call nvv4l2dec_decoder_close() to destroy the buffers and close the device.

This table describes the key structure:

Element	Description
nvPacket	Contains information about input packets.
nvFrame	Contains information about decoded frames.
nvCodingType	Specifies a different codec type.
BufferPlane	Holds the buffer plane parameters.
Buffer	Holds the buffer information.
context_t	Defines the decoder context.

This table describes the key function:

Element	Description
nvv4l2dec_create_decoder	Creates a new V4L2 Video Decoder object on the device node /dev/nvhost-nvdec.
subscribe_event	Subscribes to resolution change events.
set_output_plane_format	Sets the format on the output plane.
req_buffers_on_output_plane	Requests buffers on the output plane to be filled from the input bitstream.
dq_buffer	Dequeues the V4L2 buffer.
q_buffer	Queues the V4L2 buffer.
capture_thread	Starts the capture thread.
dq_event	Dequeues an event reported by the V4L2 device.
req_buffers_on_capture_plane	Requests buffers on the capture plane.
set_capture_plane_format	Sets the capture plane format.