Understanding NvMedia SIPL Framework

The NvMedia SIPL framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX Xavier™ and NVIDIA® DRIVE AGX Pegasus™ platforms. SIPL is implemented in C++ on top of the NvMedia API.

The purpose of SIPL is to abstract the following operations from the application layer:

The following diagram illustrates the architecture of the SIPL framework.

The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.

The SIPL Device Block component initializes and controls the external devices attached to the platform. SIPL Device Block is composed of four primary sub-components: Core, Device Driver Interface (DDI), Camera Device Drivers, and Camera Device Interface (CDI). As a combined component, SIPL Device Block uses I2C to program the settings specified in the PlatformCfg struct. Additionally, it initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link(s) between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. Please note that the application does not directly use SIPL Device Block to control and interact with external devices; instead, it must go through SIPL Core.

The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL Device Block component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to request images both before and after they have gone through the ISP pipeline and provides queues to deliver those images to the user.

This section outlines the process of using the SIPL Framework.

First, use the SIPL Query component through the INvSIPLQuery interface to get a list of supported platform configurations. A platform configuration is structure (a struct) that lists properties of external devices and how they are connected to the platform. This includes but is not limited to:

This information is returned in the form of a list of PlatformCfg structs. Choose the PlatformCfg struct that reflects the current configuration. It is not strictly necessary to use the SIPL Query component, and the PlatformCfg struct may be obtained by other means, or even hard coded into the application.

To initialize SIPL Core, first use the INvSIPLCamera::GetInstance method to get an instance of a class that implements INvSIPLCamera. The following methods can then be used to configure the settings that SIPL Core is to use during capture:

Once SIPL Core has been configured, call INvSIPLCamera::Init()to initialize SIPL. This programs the external devices to prepare them to stream images, and programs the NvMedia ICP and ISP components to prepare them to receive and process images. Note that this step prepares SIPL to capture and process images but does not actually start the process.

Next, register output buffers and configure autocontrol using the following methods:

Once the output buffers have been registered, the pipeline may be started using INvSIPLCamera::Start(). This method programs the externally connected devices to start streaming images. After this method is called, capture is active. SIPL starts receiving images, processing them through ISP, and sending them to the image output queues.

The SIPL Query component implements the INvSIPLQuery interface and is contained in the shared library libnvsipl_query.so. It maintains a database of supported external image devices and their properties. It also maintains a database of platform configurations. The database contains the following types of objects:

SIPL Query provides API calls to:

SIPL Query creates the database at runtime by searching for SIPL Query Drivers. A Query Driver is a shared library in the form libnvsipl_qry_*.so that contains a collection of database objects. SIPL Query searches the paths provided in the $LD_LIBRARY_PATH environment variable for the first directory named nvsipl_drv and opens all Query Drivers in it.

The following table describes the Query Drivers distributed with the PDK and the database objects they contain.

SIPL Device Block is composed of four primary sub-components:

The sub-components will be discussed in more detail below.

Device Block Core provides a unified interface over arbitrary external device configurations to upper layers of the SIPL stack. To accomplish this and to accommodate custom silicon or driver solutions, Device Block Core defines supported external device types as a series of abstract interfaces. Device Block Core exclusively deals with these interfaces in providing its main functionalities of hardware configuration and initialization, stream manipulation, error retrieval, etc.

Third-party Device Block Camera Device Drivers may provide implementations for the interfaces exported by Device Block Core allowing the SIPL stack to handle a wide variety of hardware solutions.

The Device Block Device Drivers Interface (DDI) sub-component is a set of C++ driver interface classes that 3rd-party vendors must inherit and implement in their camera device drivers. Using DDI, hardware-specific device drivers can be dynamically loaded into the client process by SIPL according to the provided platform configuration.

Device Block Core uses Device Block Camera Device Drivers to program external devices. A Device Block Camera Device Driver is a shared library in the form libnvsipl_devblk_drv_*.so that is used to program a specific type of camera module. Device Block Core selects which Device Block Camera Device Driver(s) to use based on the names of the camera modules that are attached to each GMSL link of the deserializer.

At runtime, Device Block Core searches the paths defined in the environment variable $LD_LIBRARY_PATH for the first directory named nvsipl_drv and opens all Device Block Camera Device Drivers in it. This allows you to add support for more devices by adding third-party Device Block Camera Device Drivers to the nvsipl_drv directory.

The Device Block Camera Device Interface (CDI) sub-component provides a common interface for Device Block Camera Device Drivers to access their hardware over an I2C interface. Additionally, CDI provides interfaces that expose various error signals from external devices.

The SIPL Core component implements the INvSIPLCamera interface. It initializes the external devices through the SIPL Device Block component and initializes the platform to capture and process images with the NvMedia ICP and NvMedia ISP APIs.

SIPL Core also provides queues to contain the output buffers produced by the capture processes. You must read the buffers from these queues regularly to avoid starving the system of output buffers.

The application is responsible for allocating and deallocating the plugin object. To avoid resource allocation during safety critical operation, the plugin should only allocate required resources during initialization and deallocate during deinitialization.

To register a user-defined (custom) auto control plugin, the user must derive from the ISiplControlAuto interface class and implement its functions.

The public functions for ISiplControlAuto are as follows.

The Process function is called in the first frame without ISP Statistics or when the ISP has processed the captured image and generated the statistics.

The plugin must receive input information in the SiplControlAutoInputParam structure and populate the results into the SiplControlAutoOutputParam structure. SiplControlAutoInputParam and SiplControlAutoOutputParam structures are described in the Process Function Input Parameters and Process Function Output Parameters sections.

 

When the plugin Process function is called, it must process the input information and generate the output parameters.

The plugin supports the following input categories:

The input information is passed in the following structure:

 

 

The structure members provide embedded data for the image that is being processed. That information is a set of registers of the image sensor.

The definition is as follows:

 

 

The structure members provide attributes data for the image sensor.

 

This structure contains the statistics data.

 

 

When the plugin Process function is called, it must process the input information and generate the output parameters.

The plugin must generate the information in the following categories:

The SiplControlAutoOutputParam structure provides the ISP stats settings, exposure settings, and white balance settings calculated by plugin based on the ISP stats received in Process Function Input Parameters.

The output is passed in the following structure:

 

 

This structure contains the sensor settings.

 

 

This structure contains the AWB, CCT and CCM settings.

 

 

This structure contains the stat block settings for the ISP.

 

The GetNoiseProfile function receives the embedded information for the current frame in SiplControlEmbedInfo structure and the parameter maxSupportedNoiseProfiles variable and puts the generated noise profile number in noiseProfile variable. SiplControlEmbedInfo structure is defined in the SiplControlEmbedData section and maxSupportedNoiseProfiles is an integer ranging from 1 to 32, specifying the max number of allowed noise profiles to be used by the plugin.

 

When the GetNoiseProfile function is called, it must process the following input information and generate the output parameter.

This structure contains the embedded data for the image that is being processed. This structure is explained in the SiplControlEmbedData section.

This parameter is an integer specifying the max number of allowed noise profiles to be used in plugin.

The output of GetNoiseProfile function is the generated noise profile number, noiseProfile. The valid range for the output noiseProfile is:

0<= noiseProfile < maxSupportedNoiseProfiles.

The plugin must check the range for all the input parameters based on the valid ranges provided in the NVIDIA DRIVE OS SDK API Reference Documentation and return an error if any value is not in range. In addition, the plugin must return an error if expTimeValid or gainValid is false in EmbedInfo.

The plugin must output values within the ranges provided in the NVIDIA DRIVE OS SDK API Reference Documentation. In addition, SIPL code would detect the following situations and report an error:

The default CSC_Inv matrix is as follows:

In addition, fusion in ISP is not supported. Therefore, the output SiplControlAutoAwbGain must be filled with valid values for the first plane (SiplControlAutoAwbGain[0]).

NVIDIA recommends the following programming to process 24-bit sensors: The exposure combination is done in the sensor to generate the 24-bit linear pixel values. Following the exposure combination, the sensor performs a PWL compression on the 24-bit linear values to generate the 12-bit compressed pixel values.

The PWL compression on 24-bit values in the sensor is such that when ISP applies a PWL decompression to retrieve 20-bit data, the 20-bit data are effectively the result of 24-bit linear data compressed by a power curve. The power curve is designed to preserve the dynamic range of the sensor.

All the blocks following the PWL decompression (linearization) block in ISP (specifically the statistics data: LAC0, LAC1, HIST0, HIST1, and FB) operate on the 20-bit compressed data.

Therefore, when using the statistics data, you can write software to linearize the data by using the inverse power-curve.

The semi-raw RGBFP16 output generated by ISP is also compressed using this power-curve. Obtain this linear data by writing the software to use the inverse power-curve.