Understanding NvMedia SIPL Framework
Applies to: DRIVE AGX Xavier and DRIVE AGX Pegasus
The NvMedia SIPL framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX Xavier™ and NVIDIA® DRIVE AGX Pegasus™ platforms. SIPL is implemented in C++ on top of the NvMedia API.
The purpose of SIPL is to abstract the following operations from the application layer:
• Programming the image sensors, EEPROMs, serializers, and deserializers.
• Programming platforms to capture and process the images using hardware image processing pipelines (ISPs).
Architecture
The following diagram illustrates the architecture of the SIPL framework.
The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.
The SIPL Device Block component initializes and controls the external devices attached to the platform. SIPL Device Block is composed of three primary sub-components: Core, Camera Device Drivers, and Camera Device Interface (CDI). As a combined component, SIPL Device Block uses I2C to program the settings specified in the PlatformCfg struct. Additionally, it initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link(s) between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. Please note that the application does not directly use SIPL Device Block to control and interact with external devices; instead, it must go through SIPL Core.
The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL Device Block component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to request images both before and after they have gone through the ISP pipeline and provides queues to deliver those images to the user.
Usage
This section outlines the process of using the SIPL Framework.
Step 1: Querying Platform Configuration
First, use the SIPL Query component through the INvSIPLQuery interface to get a list of supported platform configurations. A platform configuration is structure (a struct) that lists properties of external devices and how they are connected to the platform. This includes but is not limited to:
• Device I2C addresses
• Image properties (frame rate, data type, image size, etc.)
• Deserializer MIPI properties (PHY mode, number of lanes, etc.)
• Which camera modules are connected to which CSI brick
This information is returned in the form of a list of PlatformCfg structs. Choose the PlatformCfg struct that reflects the current configuration. It is not strictly necessary to use the SIPL Query component, and the PlatformCfg struct may be obtained by other means, or even hard coded into the application.
Step 2: Initialize SIPL
To initialize SIPL Core, first use the INvSIPLCamera::GetInstance method to get an instance of a class that implements INvSIPLCamera. The following methods can then be used to configure the settings that SIPL Core is to use during capture:
• INvSIPLCamera::SetPlatformCfg()
• INvSIPLCamera::SetPipelineCfg()
Once SIPL Core has been configured, call INvSIPLCamera::Init()to initialize SIPL. This programs the external devices to prepare them to stream images, and programs the NvMedia ICP and ISP components to prepare them to receive and process images. Note that this step prepares SIPL to capture and process images but does not actually start the process.
Step 3: Start SIPL
Next, register output buffers and configure autocontrol using the following methods:
• INvSIPLCamera::RegisterImageGroups()
• INvSIPLCamera::RegisterImages()
• INvSIPLCamera::RegisterAutoControlPlugin()
Once the output buffers have been registered, the pipeline may be started using INvSIPLCamera::Start(). This method programs the externally connected devices to start streaming images. After this method is called, capture is active. SIPL starts receiving images, processing them through ISP, and sending them to the image output queues.
Components
SIPL Query
The SIPL Query component implements the INvSIPLQuery interface and is contained in the shared library libnvsipl_query.so. It maintains a database of supported external image devices and their properties. It also maintains a database of platform configurations. The database contains the following types of objects:
Database Object Type | Description |
SensorInfo | An object that stores the settings for an image sensor external device. |
EEPROMInfo | An object that stores the settings for an EEPROM external device. |
SerInfo | An object that stores the settings for a GMSL serializer external device. |
CameraModuleInfo | An object that stores the properties for a camera module external device. This consists of one SerInfo object, zero or more EEPROMInfo objects, and zero or more SensorInfo objects. |
DeserInfo | An object that stores the settings for a GMSL deserializer external device. |
PlatformCfg | An object that describes all external devices that are attached to the CSI ports of the SoC. This includes which DeserInfo objects are to be used for each camera group, and which CameraModuleInfo objects are to be used. |
SIPL Query provides API calls to:
• Parse the default database
• Get information about all devices supported by the library
• Get a list of all supported platform configurations
• Retrieve a specific platform configuration by name
• Parse a user-provided platform_config.json file to override the PlatformCfg structs in the SIPL Query database
• Apply a mask to enable only specific deserializer links in a specific platform configuration
SIPL Query Drivers
SIPL Query creates the database at runtime by searching for SIPL Query Drivers. A Query Driver is a shared library in the form libnvsipl_qry_*.so that contains a collection of database objects. SIPL Query searches the paths provided in the $LD_LIBRARY_PATH environment variable for the first directory named nvsipl_drv and opens all Query Drivers in it.
The following table describes the Query Drivers distributed with the PDK and the database objects they contain.
SIPL Query Driver Name | Database Objects Contained |
libnvsipl_qry_serdes.so | MAX96712 DeserInfo MAX96705 SerInfo MAX9295 SerInfo MAX96759 SerInfo |
libnvsipl_qry_ar0231.so | AR0231 SensorInfo SF3324 CameraModuleInfo SF3325 CameraModuleInfo SF3324 PlatformCfg SF3325 PlatformCfg |
libnvsipl_qry_ar0144.so | AR0144 SensorInfo AR0144 CameraModule AR0144 PlatformCfg |
libnvsipl_qry_constellation.so | Constellation SensorInfo Constellation CameraModule Constellation PlatformCfg |
SIPL Device Block
SIPL Device Block is composed of three primary sub-components:
1. SIPL Device Block Core
2. (Third-party) SIPL Device Block Camera Device Drivers
3. SIPL Device Block Camera Device Interface (CDI)
The sub-components will be discussed in more detail below.
SIPL Device Block Core
Device Block Core provides a unified interface over arbitrary external device configurations to upper layers of the SIPL stack. To accomplish this and to accommodate custom silicon or driver solutions, Device Block Core defines supported external device types as a series of abstract interfaces. Device Block Core exclusively deals with these interfaces in providing its main functionalities of hardware configuration and initialization, stream manipulation, error retrieval, etc.
Third-party Device Block Camera Device Drivers may provide implementations for the interfaces exported by Device Block Core allowing the SIPL stack to handle a wide variety of hardware solutions.
SIPL Device Block Camera Device Drivers
Device Block Core uses Device Block Camera Device Drivers to program external devices. A Device Block Camera Device Driver is a shared library in the form libnvsipl_devblk_drv_*.so that is used to program a specific type of camera module. Device Block Core selects which Device Block Camera Device Driver(s) to use based on the names of the camera modules that are attached to each GMSL link of the deserializer.
At runtime, Device Block Core searches the paths defined in the environment variable $LD_LIBRARY_PATH for the first directory named nvsipl_drv and opens all Device Block Camera Device Drivers in it. This allows you to add support for more devices by adding third-party Device Block Camera Device Drivers to the nvsipl_drv directory.
SIPL Device Block Camera Device Interface (CDI)
The Device Block Camera Device Interface (CDI) sub-component provides a common interface for Device Block Camera Device Drivers to access their hardware over an I2C interface. Additionally, CDI provides interfaces that expose various error signals from external devices.
SIPL Core
The SIPL Core component implements the INvSIPLCamera interface. It initializes the external devices through the SIPL Device Block component and initializes the platform to capture and process images with the NvMedia ICP and NvMedia ISP APIs.
SIPL Core also provides queues to contain the output buffers produced by the capture processes. You must read the buffers from these queues regularly to avoid starving the system of output buffers.
Registering a User Defined Auto Control Plugin
To register a user-defined (custom) auto control plugin, populate the functions in the class ISiplControlAuto.
class ISiplControlAuto {
public:
virtual SIPLStatus Process(const SiplControlAutoInputParam& inParams,
SiplControlAutoOutputParam& outParams) = 0;
virtual ~ISiplControlAuto() = default;
protected:
ISiplControlAuto() = default;
private:
ISiplControlAuto(const ISiplControlAuto&) = delete;
ISiplControlAuto& operator= (const ISiplControlAuto&) = delete;
};
Class Public Functions
Member | Description |
Process | The function is called for every frame by SIPL to process the input parameters (SIPLControlAutoInputParam) and generate the output settings (SiplControlAutoOutputParam) for future images. |
~ISiplControlAuto() | Default destructor. |
Process Function
The Process function is called in the first frame without ISP Statistics or when the ISP has processed the captured image and generated the statistics.
The plugin receives input information in the
SiplControlAutoInputParam structure and puts the results into the
SiplControlAutoOutputParam structure.
SiplControlAutoInputParam and
SiplControlAutoOutputParam structures are described in the
Input Parameters and
Output Parameters sections.
Declaration
SIPLStatus Process(const SiplControlAutoInputParam& inParams,
SiplControlAutoOutputParam& outParams)
Parameters
Parameter | Description |
inParams | Specifies the input parameters to the plugin. |
outParams | Specifies the generated output parameters by the plugin. |
Returns
Value | Description |
NVSIPL_STATUS_OK | Success. |
NVSIPL_STATUS_ERROR | An error occurred. |
Input Parameters
When the plugin Process function is called, it must process the input information and generate the output parameters.
The plugin supports the following input categories:
• Sensor Attribute Properties
• Embedded Data and line information
• Histogram and LAC statistics
• Flicker band statistics
SiplControlAutoInputParam
The input information is passed in the following structure:
struct SiplControlAutoInputParam {
SiplControlEmbedData embedData;
DevBlkISCSensorAttributes sensorAttr;
SiplControlIspStatsInfo statsInfo;
};
Structure Members
Member | Description |
embedData | Specifies the Embedded Settings of the frame. |
sensorAttr | Specifies the attributes of the sensor. |
statsInfo | Specifies the stats data. |
SiplControlEmbedData
The structure members provide embedded data for the image that is being processed. That information is a set of registers of the image sensor.
The definition is as follows:
struct SiplControlEmbedData {
SiplControlEmbedInfo embedInfo;
DevBlkISCFrameSeqNum frameSeqNum;
DevBlkISCEmbeddedDataChunk topEmbeddedData;
DevBlkISCEmbeddedDataChunk bottomEmbeddedData;
};
Structure Members
Member | Description |
embedInfo | Holds the parsed embedded info for the captured frame. |
frameSeqNum | Holds frame sequence number for the captured frame. |
topEmbeddedData | Embedded buffer at the beginning of the frame. |
bottomEmbeddedData | Embedded buffer at the end of the frame. |
DevBlkISCSensorAttributes
The structure members provide attributes data for the image sensor.
struct DevBlkISCSensorAttributes {
char sensorName[DEVBLK_ISC_MAX_SENSOR_NAME_LENGTH];
uint32_t sensorCFA;
char sensorFuseId[DEVBLK_ISC_MAX_FUSE_ID_LENGTH];
uint8_t numActiveExposures;
DevBlkISCAttrRange sensorExpRange[DEVBLK_ISC_MAX_EXPOSURES];
DevBlkISCAttrRange sensorGainRange […];
DevBlkISCAttrRange sensorWhiteBalanceRange […];
float_t sensorGainFactor[DEVBLK_ISC_MAX_EXPOSURES];
uint32_t numFrameReportBytes;
};
Structure Members
Member | Description |
sensorName | Holds the name attribute. |
sensorCFA | Holds the CFA attribute. |
sensorFuseId | Holds the fuse ID attribute. |
numActiveExposures | Holds the number of active exposures attribute. |
sensorExpRange | Holds the sensor exposure ranges for active exposures. |
sensorGainRange | Holds the sensor gain ranges for active exposures. |
sensorWhiteBalanceRange | Holds the sensor white balance ranges for active exposures. |
sensorGainFactor | Holds the additional sensor gain factor between active exposures. These gain factors describe the sensitivity difference between the exposures. |
numFrameReportBytes | Holds the number of frame report bytes supported by the sensor. |
SiplControlIspStatsInfo
This structure contains the statistics data.
struct SiplControlIspStatsInfo {
const NvMediaISPLocalAvgClipStatsData* lacData[2];
const NvMediaISPLocalAvgClipStats* lacSettings[2];
const NvMediaISPHistogramStatsData* histData[2];
const NvMediaISPHistogramStats* histSettings[2];
const NvMediaISPFlickerBandStatsData* fbStatsData;
const NvMediaISPFlickerBandStats* fbStatsSettings;
};
Structure Members
Member | Description |
lacData | Holds const pointers to LAC stats data from LAC-0, LAC-1 blocks in ISP. |
lacSettings | Holds const pointers to LAC stats settings from LAC-0, LAC-1 blocks in ISP. |
histData | Holds const pointers to Histogram stats data from HIST-0, HIST-1 blocks in ISP. |
histSettings | Holds const pointers to Histogram stats settings from HIST-0, HIST-1 blocks in ISP |
fbStatsData | Holds const pointer to Flicker Band stats data from FB block in ISP |
fbStatsSettings | Holds const pointer to Flicker Band stats settings from FB block in ISP |
Output Parameters
When the plugin Process function is called, it must process the input information and generate the output parameters.
The plugin must generate the information in the following categories:
• Exposure control
• White balance control
• Histogram statistics settings
• LAC statistics settings
• Flicker-band statistics settings
• SiplControlAutoOutputParam
The
SiplControlAutoOutputParam structure provides the ISP stats settings, exposure settings, and white balance settings calculated by plugin based on the ISP stats received in
Input Parameters.
The output is passed in the following structure:
struct SiplControlAutoOutputParam {
SiplControlAutoSensorSetting sensorSetting;
SiplControlAutoAwbSetting awbSetting;
SiplControlIspStatsSetting newStatsSetting;
float_t ispDigitalGain;
};
Structure Members
Member | Description |
sensorSetting | Sensor exposure and gain settings. |
awbSetting | AWB settings. |
newStatsSetting | Stats Settings. |
ispDigitalGain | Digital gain to be applied in ISP. |
SiplControlAutoSensorSetting
This structure contains the sensor settings.
struct SiplControlAutoSensorSetting {
uint8_t numSensorContexts;
DevBlkISCExposure exposureControl[…];
DevBlkISCWhiteBalance wbControl[…];
bool expProfileValid;
uint32_t expProfile;
};
Structure Members
Member | Description |
numSensorContexts | Holds the number of sensor contexts to activate. |
exposureControl | Holds the sensor exposure settings to set for each context. |
wbControl | Holds the sensor white balance settings to set for each context. |
expProfileValid | Exposure profile value is valid or not. |
expProfile | Exposure profile value, such as day and night settings. |
SiplControlAutoAwbSetting
This structure contains the AWB, CCT and CCM settings.
struct SiplControlAutoAwbSetting {
SiplControlAutoAwbGain wbGainTotal[…];
float_t cct;
float_t ccmMatrix[…][…];
}
Structure Members
Member | Description |
wbGainTotal | Total white balance gains, including both senor channel gains and ISP gains. |
cct | Correlated Color Temperature. |
ccmMatrix | Color Correlation Matrix. |
SiplControlIspStatsSetting
This structure contains the stat block settings for the ISP.
struct SiplControlIspStatsSetting {
bool valid;
NvMediaISPLocalAvgClipStats lac[2];
NvMediaISPHistogramStats hist1;
NvMediaISPFlickerBandStats fbStats;
}
Structure Members
Member | Description |
valid | Settings to control ISP stats blocks are valid or not. |
lac | Settings for 2 LAC stats ISP blocks. |
hist1 | Settings for Histogram 1 stats blocks. |
fbStats | Settings for Flicker Band stats block. |
ISP Processing and 24-bit Sensor Output
NVIDIA recommends the following programming to process 24-bit sensors: The exposure combination is done in the sensor to generate the 24-bit linear pixel values. Following the exposure combination, the sensor performs a PWL compression on the 24-bit linear values to generate the 12-bit compressed pixel values.
The PWL compression on 24-bit values in the sensor is such that when ISP applies a PWL decompression to retrieve 20-bit data, the 20-bit data are effectively the result of 24-bit linear data compressed by a power curve. The power curve is designed to preserve the dynamic range of the sensor.
All the blocks following the PWL decompression (linearization) block in ISP (specifically the statistics data: LAC0, LAC1, HIST0, HIST1, and FB) operate on the 20-bit compressed data.
Therefore, when using the statistics data, you can write software to linearize the data by using the inverse power-curve.
The semi-raw RGBFP16 output generated by ISP is also compressed using this power-curve. Obtain this linear data by writing the software to use the inverse power-curve.