Image Processing and Management
The NvMedia Image domain performs the following processes:
• Handles Image Surface data (H/W buffer). For example, YUV, RGB, RAW (progressive only)
• Supports image sensor registers data
• Supports per image specific metadata
• Handles necessary H/W synchronization
• Performs timestamping
• Signals surface allocation for CPU access with cached access
Image Capture Processing (ICP)
The NvMedia Image Capture (ICP) component captures the frames coming from the CSI interface. The output of this component contains the captured images.
The NvMedia ICP component provides the following features.
• Supports capture formats including: YUV 4:2:2 8/10-Bit, RGB 8:8:8; RAW8, RAW10, RAW12, RAW14, RAW16, RAW20, etc.
• Supports CSI modes including: x1, x2, x4 for each of the ports
• Provides external buffer allocation
• Supports capture on Request Processing
• Allows embedded line information (image specific metadata)
Notes: | Please be aware of the following camera streaming limitations: • Input bitrate for each camera stream is limited to 25.6 gigabits/sec. • Tegra Camera Video Input engine works at a platform-specific frequency and can write 4 or 8 pixels to memory per clock (depending on the data type size). The total peak pixel rate of all connected cameras must not exceed the capacity of the Video Input engine. |
Image 2D
The NvMedia Image 2D component supports image surface processing features such as image copy, image scaling, image cropping. It operates on YUV/RGB input and output surfaces. It also performs format conversion to/from YUV to RGB and supports aggregated image handling.
Image 2D Processing
The Image 2D Processing API provides a variety of functions for 2D image processing.
Architecture
This diagram shows the overall architecture of the API.
The API uses an opaque object named NvMedia2D to represent 2D objects. The typedef NvMedia2D defines an opaque handle that applications can use to manipulate these objects.
The API provides these functions to manipulate NvMedia2D objects:
• NvMedia2Dcreate()creates NvMedia2D objects.
• NvMedia2Ddestroy() function destroys NvMedia2D objects.
• nvMedia2DWeave()merges odd lines from one Camera Serial Interface (CSI) port with even lines from another. The API utilizes the Video Image Compositor (VIC) to perform the merge through hardware.
nvMedia2dWeave() enables an application to double the effective bandwidth of the CSI ports by dividing the image data in a datastream between two ports. The function reassembles the original image.
• NvMedia2DCopyPlane() copies a plane of a YUV image to another YUV image.
• NvMedia2DgetVersion() gets the current version of the API.
Image Encode Processing (IEP)
The NvMedia Image Encode (IEP) component supports encoding the incoming NvMedia Image (YUV) inputs to H.264 or H.265 or JPEG formats.
For a list of supported encoding features, see Video Encode in Video Processing.
Image Display Processing (IDP)
The NvMedia Image display (IDP) component displays YUV and RGB formatted images. It provides mechanisms to render YUV and RGBA surfaces on the display. The display can be selected among the available/connected displays (platform dependent).
SIPL
The NvMedia SIPL framework provides high dynamic range (HDR) camera processing which outputs images for human and machine vision. It handles individual camera processing or multiple cameras connected to an image aggregator chip.
EGLStream Interoperability
The NvMedia EGLStream component supports GL/CUDA interoperability of the NvMedia domain. EGLStream provides the interface to post or retrieve the raw YUV or RGB images. It provides the channel of communication to connect the NvMedia domain with GL/CUDA domain. Any NvMedia surface can be rendered on the screen using any type of consumer using EGLStream.
NvMedia EGLStream component provides the following features:
• Supports the Khronos EGLStream specification
• Maps camera input image data to CPU
• Provides images to GL as textures
• Provides images to CUDA as cudaPtr or cudaArray
• Requires no extra memory copy for GPU processing
• Provides multi-threaded or multi-process sharing
• Provides ‘mailbox’ or ‘fifo’ mode of transfer
This interoperability feature of NvMedia EGLStream makes it useful in many applications. You can easily realize NvMedia Image, Video producers/consumers interacting with GL/CUDA producers/consumers.
The following diagram show how camera input is processed.
NvMedia Image Sensor Control (ISC)
The NvMedia ISC provides a framework for image sensor control. It includes programming I2C controlled external components such as aggregators, and image sensors. It provides the following features:
• Supports addition of custom drivers
• Sends the control commands to hardware devices related to image sensor
• Reports errors during image capture
• Powers on/off the cameras
• Supports debugfs
• Powers on/off
• Checks status of power
NvMedia ISC exposes a user space interface that supports configuring and controlling the sensors, aggregating, and serializing. In addition, it can turn on and configure the camera inputs and respond to interrupts.
ISC power control can be disabled using the ISC_RDEV_CFG_EX macro. In this case the application is responsible for powering on the sensor, serializer and deserializer before attempting to program them.
The application is also responsible for powering these components off after capture if required. The CCP API to control camera power can be found at Library Usage and Communication in the Tegra to Microcontroller Communications topic.