NVIDIA Tegra
NVIDIA DRIVE OS 5.1 Linux SDK

Developer Guide
5.1.12.0 Release


 
NvMedia Architecture
 
NvMedia Stack
NvMedia API Architecture
NvMedia Video Surface
NvMedia Image Surface
NvMedia APIs and Thread Safety
NvMedia provides powerful processing of multimedia data for true hardware acceleration across NVIDIA® Tegra® devices. With the NvMedia and Tegra firmware components, multimedia applications support multiple simultaneous camera feeds for simultaneous processing. The NvMedia features include:
Robust image processing and sensor control blocks for use by applications.
Hardware surface to handle all types of image formats such as RGB, YUV and RAW.
Functional components for image capture, processing, acceleration, encoding, display, and interoperating with other libraries.
Applications leverage the NvMedia Application Programming Interface (API) to process the image and video data. The NvMedia API provides true hardware acceleration of image and video processing hardware components on Tegra® devices. Additionally, NvMedia can route image data to/from other components, such as OpenGL ES and NVIDIA® CUDA®.
NvMedia Stack
The NvMedia software stack supports these types of interactions:
Applications call the NvMedia Framework components to string together a sequence of processing steps for images.
The NvMedia Framework calls low-level hardware drivers to interact with the SOC components on the Tegra chip.
Consult building and running NvMedia sample applications to build an NvMedia sample application.
A screenshot of a cell phone Description generated with very high confidence
Each component functionality is as follows. The drivers expose a subset of the functionality available.
Tegra Hardware
Description
Video Input (VI)
Receives CSI data from the camera.
Image Signal Processor
(ISP)
Produces a processed image from image data captured from an image sensor. For example, it can make pixel-level changes such as inverting pixel bits, auto exposure, and white balance correction.
NVIDIA Encoder (NVENC)
Converts raw image data into one of the supported image formats.
NVIDIA Decoder (NVDEC)
Converts encoded image data into raw image data.
Video Interlace Compositor (VIC)
Converts video data for deinterlacing, composition, and format conversion.
Programmable Vision Accelerator (PVA)
Accelerates computer vision algorithms via the NvMediaVPI APIs.
NvMedia API Architecture
The NvMedia API library is a frame-level, driver-level, threadless library that provides video and image processing pipeline acceleration across NVIDIA® Tegra® devices. The NvMedia API is consistent across all Tegra devices while being operating system middleware and framework agnostic.
The Video API library supports interlaced video for human-machine interface (HMI) applications.
The image API library supports RAW camera processing using Image Signal Processing (ISP) and progressive image processing for Advanced Driver Assistance Systems (ADAS) development.
NvMedia Video Surface
The NvMedia video surface:
Defines and handles YUV, RGB, Progressive, and Interlaced processing
Does not imply a buffer context by the process handling which is required for hardware synchronization.
Is not required for CPU accessing of the NVIDIA hardware buffer format.
The video surface components are as follows:
Video Component
Description
Capture
Captures video CSI data.
Decode
Decodes frame level bitstream; must be parsed externally to provide sequence and slice headers. Supports progressive and interlaced formats.
Encode
Encodes NvMedia video surface inputs (YUV420, RGBA) to H264/H265 format.
Mixer
Supports deinterlacing, format conversion, scaling, and advanced post processing.
Display
Provides the ability to render the YUV/RGB surfaces on the selected display device.
Interoperability of EGLStream
Provides OpenGL and CUDA interoperability to share camera data processed by the GPU (GL or CUDA API).
NvMedia Image Surface
The NvMedia image surface:
Defines and handles YUV, RGB, and RAW (progressive only) image processing.
Carries the image sensor register data for embedded line information.
Carries per image specific metadata that does not have to be per processing handle. The handle is required for hardware synchronization.
Allocation is signaled for CPU access of pitch linear buffer format.
The image surface components are as follows:
Image Component
Description
ISC
The Image Sensor Control provides the ability to control serializer/deserializer (SerDes) blocks and image sensors.
IPP Framework
The Image Processing Pipeline framework processes incoming image data with AE, AWB controls and includes the ISP processing capability. It supports camera tuning tools.
ISP
The Image Signal Processor produces a processed image from image data captured from an image sensor. For example, it can make pixel-level changes such as inverting pixel bits, auto exposure, and white balance correction.
ICP
Image Capture Processing that provides the ability to capture data over the CSI interface.
2D
Image 2D provides the ability to perform manipulation of image data; such as cropping, scaling, copying and converting format.
IEP
Image Encode Processing provides the ability to encode processed YUV 420/RGB surface inputs to H.264, H.265, and JPEG formats.
IDP
Image Display Processing provides the ability to render the YUV/RGB surfaces on the selected display device.
Interoperability of EGLStream
Provides OpenGL and CUDA interoperability to share camera data processed by the GPU (GL or CUDA API).
NvMedia APIs and Thread Safety
NvMedia APIs are not designed to be thread safe. It is the responsibility of the application to maintain thread safety. To ensure thread safety:
NvMedia components can be created and used in any thread but the APIs cannot be used from different threads concurrently.
Different instances of the same component can be used in parallel from different threads.
Encoders are designed to be fed from one thread and get the encoded bitstream from another thread.
The NvMedia IPP framework is also not thread safe. Only get/return output APIs are thread safe.