Understanding NvMedia SIPL Framework

Applies to: DRIVE AGX Xavier and DRIVE AGX Pegasus
The NvMedia SIPL framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX Xavier™ and NVIDIA® DRIVE AGX Pegasus™ platforms. SIPL is implemented in C++ on top of the NvMedia API.
The purpose of SIPL is to abstract the following operations from the application layer:
Programming the image sensors, EEPROMs, serializers, and deserializers.
Programming platforms to capture and process the images using hardware image processing pipelines (ISPs).


The following diagram illustrates the architecture of the SIPL framework.
A screenshot of a cell phone Description automatically generated
The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.
The SIPL Device Block component initializes and controls the external devices attached to the platform. SIPL Device Block is composed of three primary sub-components: Core, Camera Device Drivers, and Camera Device Interface (CDI). As a combined component, SIPL Device Block uses I2C to program the settings specified in the PlatformCfg struct. Additionally, it initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link(s) between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. Please note that the application does not directly use SIPL Device Block to control and interact with external devices; instead, it must go through SIPL Core.
The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL Device Block component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to request images both before and after they have gone through the ISP pipeline and provides queues to deliver those images to the user.


This section outlines the process of using the SIPL Framework.

Step 1: Querying Platform Configuration

First, use the SIPL Query component through the INvSIPLQuery interface to get a list of supported platform configurations. A platform configuration is structure (a struct) that lists properties of external devices and how they are connected to the platform. This includes but is not limited to:
Device I2C addresses
Image properties (frame rate, data type, image size, etc.)
Deserializer MIPI properties (PHY mode, number of lanes, etc.)
Which camera modules are connected to which CSI brick
This information is returned in the form of a list of PlatformCfg structs. Choose the PlatformCfg struct that reflects the current configuration. It is not strictly necessary to use the SIPL Query component, and the PlatformCfg struct may be obtained by other means, or even hard coded into the application.

Step 2: Initialize SIPL

To initialize SIPL Core, first use the INvSIPLCamera::GetInstance method to get an instance of a class that implements INvSIPLCamera. The following methods can then be used to configure the settings that SIPL Core is to use during capture:
Once SIPL Core has been configured, call INvSIPLCamera::Init()to initialize SIPL. This programs the external devices to prepare them to stream images, and programs the NvMedia ICP and ISP components to prepare them to receive and process images. Note that this step prepares SIPL to capture and process images but does not actually start the process.

Step 3: Start SIPL

Next, register output buffers and configure autocontrol using the following methods:
Once the output buffers have been registered, the pipeline may be started using INvSIPLCamera::Start(). This method programs the externally connected devices to start streaming images. After this method is called, capture is active. SIPL starts receiving images, processing them through ISP, and sending them to the image output queues.


SIPL Query

The SIPL Query component implements the INvSIPLQuery interface and is contained in the shared library libnvsipl_query.so. It maintains a database of supported external image devices and their properties. It also maintains a database of platform configurations. The database contains the following types of objects:
Database Object Type
An object that stores the settings for an image sensor external device.
An object that stores the settings for an EEPROM external device.
An object that stores the settings for a GMSL serializer external device.
An object that stores the properties for a camera module external device. This consists of one SerInfo object, zero or more EEPROMInfo objects, and zero or more SensorInfo objects.
An object that stores the settings for a GMSL deserializer external device.
An object that describes all external devices that are attached to the CSI ports of the SoC. This includes which DeserInfo objects are to be used for each camera group, and which CameraModuleInfo objects are to be used.
SIPL Query provides API calls to:
Parse the default database
Get information about all devices supported by the library
Get a list of all supported platform configurations
Retrieve a specific platform configuration by name
Parse a user-provided platform_config.json file to override the PlatformCfg structs in the SIPL Query database
Apply a mask to enable only specific deserializer links in a specific platform configuration

SIPL Query Drivers

SIPL Query creates the database at runtime by searching for SIPL Query Drivers. A Query Driver is a shared library in the form libnvsipl_qry_*.so that contains a collection of database objects. SIPL Query searches the paths provided in the $LD_LIBRARY_PATH environment variable for the first directory named nvsipl_drv and opens all Query Drivers in it.
SIPL Query Core and SIPL Query Drivers
The following table describes the Query Drivers distributed with the PDK and the database objects they contain.
SIPL Query Driver Name
Database Objects Contained
MAX96712 DeserInfo
MAX96705 SerInfo
MAX9295 SerInfo
MAX96759 SerInfo
AR0231 SensorInfo
SF3324 CameraModuleInfo
SF3325 CameraModuleInfo
SF3324 PlatformCfg
SF3325 PlatformCfg
AR0144 SensorInfo
AR0144 CameraModule
AR0144 PlatformCfg
Constellation SensorInfo
Constellation CameraModule
Constellation PlatformCfg

SIPL Device Block

SIPL Device Block is composed of three primary sub-components:
1. SIPL Device Block Core
2. (Third-party) SIPL Device Block Camera Device Drivers
3. SIPL Device Block Camera Device Interface (CDI)
The sub-components will be discussed in more detail below.
SIPL Device Block sub-components

SIPL Device Block Core

Device Block Core provides a unified interface over arbitrary external device configurations to upper layers of the SIPL stack. To accomplish this and to accommodate custom silicon or driver solutions, Device Block Core defines supported external device types as a series of abstract interfaces. Device Block Core exclusively deals with these interfaces in providing its main functionalities of hardware configuration and initialization, stream manipulation, error retrieval, etc.
Third-party Device Block Camera Device Drivers may provide implementations for the interfaces exported by Device Block Core allowing the SIPL stack to handle a wide variety of hardware solutions.

SIPL Device Block Camera Device Drivers

Device Block Core uses Device Block Camera Device Drivers to program external devices. A Device Block Camera Device Driver is a shared library in the form libnvsipl_devblk_drv_*.so that is used to program a specific type of camera module. Device Block Core selects which Device Block Camera Device Driver(s) to use based on the names of the camera modules that are attached to each GMSL link of the deserializer.
At runtime, Device Block Core searches the paths defined in the environment variable $LD_LIBRARY_PATH for the first directory named nvsipl_drv and opens all Device Block Camera Device Drivers in it. This allows you to add support for more devices by adding third-party Device Block Camera Device Drivers to the nvsipl_drv directory.
SIPL Device Block Camera Device Drivers

SIPL Device Block Camera Device Interface (CDI)

The Device Block Camera Device Interface (CDI) sub-component provides a common interface for Device Block Camera Device Drivers to access their hardware over an I2C interface. Additionally, CDI provides interfaces that expose various error signals from external devices.


The SIPL Core component implements the INvSIPLCamera interface. It initializes the external devices through the SIPL Device Block component and initializes the platform to capture and process images with the NvMedia ICP and NvMedia ISP APIs.
SIPL Core also provides queues to contain the output buffers produced by the capture processes. You must read the buffers from these queues regularly to avoid starving the system of output buffers.

Registering a User Defined Auto Control Plugin

To register a user-defined (custom) auto control plugin, populate the functions in the class ISiplControlAuto.
class ISiplControlAuto {
virtual SIPLStatus Process(const SiplControlAutoInputParam& inParams,
SiplControlAutoOutputParam& outParams) = 0;
virtual ~ISiplControlAuto() = default;
ISiplControlAuto() = default;
ISiplControlAuto(const ISiplControlAuto&) = delete;
ISiplControlAuto& operator= (const ISiplControlAuto&) = delete;

Class Public Functions

The function is called for every frame by SIPL to process the input parameters (SIPLControlAutoInputParam) and generate the output settings (SiplControlAutoOutputParam) for future images.
Default destructor.
Process Function
The Process function is called in the first frame without ISP Statistics or when the ISP has processed the captured image and generated the statistics.
The plugin receives input information in the SiplControlAutoInputParam structure and puts the results into the SiplControlAutoOutputParam structure. SiplControlAutoInputParam and SiplControlAutoOutputParam structures are described in the Input Parameters and Output Parameters sections.
SIPLStatus Process(const SiplControlAutoInputParam& inParams,
SiplControlAutoOutputParam& outParams)
Specifies the input parameters to the plugin.
Specifies the generated output parameters by the plugin.
An error occurred.

Input Parameters

When the plugin Process function is called, it must process the input information and generate the output parameters.
The plugin supports the following input categories:
Sensor Attribute Properties
Embedded Data and line information
Histogram and LAC statistics
Flicker band statistics
The input information is passed in the following structure:
struct SiplControlAutoInputParam {
SiplControlEmbedData embedData;
DevBlkISCSensorAttributes sensorAttr;
SiplControlIspStatsInfo statsInfo;
Structure Members
Specifies the Embedded Settings of the frame.
Specifies the attributes of the sensor.
Specifies the stats data.
The structure members provide embedded data for the image that is being processed. That information is a set of registers of the image sensor.
The definition is as follows:
struct SiplControlEmbedData {
SiplControlEmbedInfo embedInfo;
DevBlkISCFrameSeqNum frameSeqNum;
DevBlkISCEmbeddedDataChunk topEmbeddedData;
DevBlkISCEmbeddedDataChunk bottomEmbeddedData;
Structure Members
Holds the parsed embedded info for the captured frame.
Holds frame sequence number for the captured frame.
Embedded buffer at the beginning of the frame.
Embedded buffer at the end of the frame.
The structure members provide attributes data for the image sensor.
struct DevBlkISCSensorAttributes {
uint32_t sensorCFA;
uint8_t numActiveExposures;
DevBlkISCAttrRange sensorExpRange[DEVBLK_ISC_MAX_EXPOSURES];
DevBlkISCAttrRange sensorGainRange […];
DevBlkISCAttrRange sensorWhiteBalanceRange […];
float_t sensorGainFactor[DEVBLK_ISC_MAX_EXPOSURES];
uint32_t numFrameReportBytes;
Structure Members
Holds the name attribute.
Holds the CFA attribute.
Holds the fuse ID attribute.
Holds the number of active exposures attribute.
Holds the sensor exposure ranges for active exposures.
Holds the sensor gain ranges for active exposures.
Holds the sensor white balance ranges for active exposures.
Holds the additional sensor gain factor between active exposures. These gain factors describe the sensitivity difference between the exposures.
Holds the number of frame report bytes supported by the sensor.
This structure contains the statistics data.
struct SiplControlIspStatsInfo {
const NvMediaISPLocalAvgClipStatsData* lacData[2];
const NvMediaISPLocalAvgClipStats* lacSettings[2];
const NvMediaISPHistogramStatsData* histData[2];
const NvMediaISPHistogramStats* histSettings[2];
const NvMediaISPFlickerBandStatsData* fbStatsData;
const NvMediaISPFlickerBandStats* fbStatsSettings;
Structure Members
Holds const pointers to LAC stats data from LAC-0, LAC-1 blocks in ISP.
Holds const pointers to LAC stats settings from LAC-0, LAC-1 blocks in ISP.
Holds const pointers to Histogram stats data from HIST-0, HIST-1 blocks in ISP.
Holds const pointers to Histogram stats settings from HIST-0, HIST-1 blocks in ISP
Holds const pointer to Flicker Band stats data from FB block in ISP
Holds const pointer to Flicker Band stats settings from FB block in ISP

Output Parameters

When the plugin Process function is called, it must process the input information and generate the output parameters.
The plugin must generate the information in the following categories:
Exposure control
White balance control
Histogram statistics settings
LAC statistics settings
Flicker-band statistics settings
The SiplControlAutoOutputParam structure provides the ISP stats settings, exposure settings, and white balance settings calculated by plugin based on the ISP stats received in Input Parameters.
The output is passed in the following structure:
struct SiplControlAutoOutputParam {
SiplControlAutoSensorSetting sensorSetting;
SiplControlAutoAwbSetting awbSetting;
SiplControlIspStatsSetting newStatsSetting;
float_t ispDigitalGain;
Structure Members
Sensor exposure and gain settings.
AWB settings.
Stats Settings.
Digital gain to be applied in ISP.
This structure contains the sensor settings.
struct SiplControlAutoSensorSetting {
uint8_t numSensorContexts;
DevBlkISCExposure exposureControl[…];
DevBlkISCWhiteBalance wbControl[…];
bool expProfileValid;
uint32_t expProfile;
Structure Members
Holds the number of sensor contexts to activate.
Holds the sensor exposure settings to set for each context.
Holds the sensor white balance settings to set for each context.
Exposure profile value is valid or not.
Exposure profile value, such as day and night settings.
This structure contains the AWB, CCT and CCM settings.
struct SiplControlAutoAwbSetting {
SiplControlAutoAwbGain wbGainTotal[…];
float_t cct;
float_t ccmMatrix[…][…];
Structure Members
Total white balance gains, including both senor channel gains and ISP gains.
Correlated Color Temperature.
Color Correlation Matrix.
This structure contains the stat block settings for the ISP.
struct SiplControlIspStatsSetting {
bool valid;
NvMediaISPLocalAvgClipStats lac[2];
NvMediaISPHistogramStats hist1;
NvMediaISPFlickerBandStats fbStats;
Structure Members
Settings to control ISP stats blocks are valid or not.
Settings for 2 LAC stats ISP blocks.
Settings for Histogram 1 stats blocks.
Settings for Flicker Band stats block.

ISP Processing and 24-bit Sensor Output

NVIDIA recommends the following programming to process 24-bit sensors: The exposure combination is done in the sensor to generate the 24-bit linear pixel values. Following the exposure combination, the sensor performs a PWL compression on the 24-bit linear values to generate the 12-bit compressed pixel values.
The PWL compression on 24-bit values in the sensor is such that when ISP applies a PWL decompression to retrieve 20-bit data, the 20-bit data are effectively the result of 24-bit linear data compressed by a power curve. The power curve is designed to preserve the dynamic range of the sensor.
All the blocks following the PWL decompression (linearization) block in ISP (specifically the statistics data: LAC0, LAC1, HIST0, HIST1, and FB) operate on the 20-bit compressed data.
Therefore, when using the statistics data, you can write software to linearize the data by using the inverse power-curve.
The semi-raw RGBFP16 output generated by ISP is also compressed using this power-curve. Obtain this linear data by writing the software to use the inverse power-curve.