Image Processing and Management
The NvMedia Image domain performs the following processes:
Handles Image Surface data (H/W buffer). For example, YUV, RGB, RAW (progressive only)
Supports image sensor registers data
Supports per image specific metadata
Handles necessary H/W synchronization
Performs timestamping
Signals surface allocation for CPU access with cached access
Image Capture Processing (ICP)
The NvMedia Image Capture (ICP) component captures the frames coming from the CSI interface. The output of this component contains the captured images.
The NvMedia ICP component provides the following features.
Supports capture formats including: YUV 4:2:2 8/10-Bit, RGB 8:8:8; RAW8, RAW10, RAW12, RAW14, RAW16, RAW20, FP_16ISP, etc.
Supports CSI modes including: x1, x2, x4 for each of the ports
Provides external buffer allocation
Supports capture on Request Processing
Allows embedded line information (image specific metadata)
Image Signal Processing (ISP)
The NvMedia Image Signal Processor (ISP) component processes Bayer images to YUV formatted images. It uses Tegra configurable ISP hardware and supports the following processing operations.
Stuck at Pixel Outlier Removal (SAPOR)
Bad pixel replacement
Spatially varying noise filtering
Areal Processor (AP)
Demosaicer (DM)
Color Artifact Reduction (CAR)
Edge enhancement (EE)
Local Average and Clip (LAC)
Histogram statistics
General Pixel Processor (GPP)
Transfer functions
Affine transform
Auto White Balance (AWB) Gains and black level offset
Bezier lens shading (LS)
Down scalar (DS)
Flicker band detection (FB)
Image 2D
The NvMedia Image 2D component supports image surface processing features such as image copy, image scaling, image cropping. It operates on YUV/RGB input and output surfaces. It also performs format conversion to/from YUV to RGB and supports aggregated image handling.
Image 2D Processing
The Image 2D Processing API provides a variety of functions for 2D image processing.
This diagram shows the overall architecture of the API.
A close up of a sign Description generated with very high confidence
The API uses an opaque object named NvMedia2D to represent 2D objects. The typedef NvMedia2D defines an opaque handle that applications can use to manipulate these objects.
The API provides these functions to manipulate NvMedia2D objects:
NvMedia2Dcreate()creates NvMedia2D objects.
NvMedia2Ddestroy() function destroys NvMedia2D objects.
nvMedia2DWeave()merges odd lines from one Camera Serial Interface (CSI) port with even lines from another. The API utilizes the Video Image Compositor (VIC) to perform the merge through hardware.
nvMedia2dWeave() enables an application to double the effective bandwidth of the CSI ports by dividing the image data in a datastream between two ports. The function reassembles the original image.
NvMedia2DCopyPlane() copies a plane of a YUV image to another YUV image.
NvMedia2DgetVersion() gets the current version of the API.
Image Encode Processing (IEP)
The NvMedia Image Encode (IEP) component supports encoding the incoming NvMedia Image (YUV or RGB) inputs to H.264 or H.265 or JPEG formats.
For a list of supported encoding features, see Video Encode in Video Processing.
Image Display Processing (IDP)
The NvMedia Image display (IDP) component displays YUV and RGB formatted images. It provides mechanisms to render YUV and RGBA surfaces on the display. The display can be selected among the available/connected displays (platform dependent).
Image Processing Pipeline (IPP)
The NvMedia Image Processing Pipeline (IPP) framework provides high dynamic range (HDR) camera processing which outputs images for human and machine vision. It handles individual camera processing or multiple cameras connected to an image aggregator chip.
NvMedia IPP connects individual image components which operate inside the NvMedia Image domain. These components include:
Capture Processing (ICP)
Image Signal Processing (ISP)
Image Sensor Control (ISC)
Control Algorithm
NvMedia IPP components interconnect to one another like a graph, forming a processing pipeline. NvMedia IPP uses image buffers and queues between the components to send the processed images to the next component. Each component maintains its own buffers, called the buffer pool. The IPP framework creates and manages threads for each component.
ISP also outputs image statistics that the Control Algorithm component uses to calculate the proper ISP and sensor exposure settings to achieve proper auto white balance and auto exposure.
EGLStream Interoperability
The NvMedia EGLStream component supports GL/CUDA interoperability of the NvMedia domain. EGLStream provides the interface to post or retrieve the raw YUV or RGB images. It provides the channel of communication to connect the NvMedia domain with GL/CUDA domain. Any NvMedia surface can be rendered on the screen using any type of consumer using EGLStream.
NvMedia EGLStream component provides the following features:
Supports the Khronos EGLStream specification
Maps camera input image data to CPU
Provides images to GL as textures
Provides images to CUDA as cudaPtr or cudaArray
Requires no extra memory copy for GPU processing
Provides multi-threaded or multi-process sharing
Provides ‘mailbox’ or ‘fifo’ mode of transfer
This interoperability feature of NvMedia EGLStream makes it useful in many applications. You can easily realize NvMedia Image, Video producers/consumers interacting with GL/CUDA producers/consumers.
The following diagram show how camera input is processed.
NvMedia Image Sensor Control (ISC)
The NvMedia ISC provides a framework for image sensor control. It includes programming I2C controlled external components such as aggregators, and image sensors. It provides the following features:
Supports addition of custom drivers
Sends the control commands to hardware devices related to image sensor
Reports errors during image capture
Powers on/off the cameras
Supports debugfs
Powers on/off
Checks status of power
NvMedia ISC exposes a user space interface that supports configuring and controlling the sensors, aggregating, and serializing. In addition, it can turn on and configure the camera inputs and respond to interrupts.
ISC power control can be disabled using the ISC_RDEV_CFG_EX macro. In this case the application is responsible for powering on the sensor, serializer and deserializer before attempting to program them.
The application is also responsible for powering these components off after capture if required. The CCP API to control camera power can be found at Library Usage and Communication in the Tegra to Microcontroller Communications topic.