Developer Guide Release

Understanding NvMedia SIPL Framework
Applies to: DRIVE AGX Xavier and DRIVE AGX Pegasus
The NvMedia Sensor Input Processing Library (SIPL) framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX Xavier™ and NVIDIA® DRIVE AGX Pegasus™ platforms. SIPL is implemented in C++ on top of the NvMedia API.
The purpose of SIPL is to abstract the following operations from the application layer:
Programming the image sensors, EEPROMs, serializers, and deserializers.
Programming platforms to capture and process the images using hardware image processing pipelines (ISPs).
The following diagram illustrates the architecture of the SIPL framework.
A screenshot of a cell phone Description automatically generated
The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.
The SIPL DeviceBlock is a component of the SIPL framework which initializes and controls the external devices attached to the platform. It uses I2C to program the settings specified in the PlatformCfg struct. It initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. The application does not need to use this component directly; it is instead used by SIPL Core.
The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL DeviceBlock component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to register callbacks to handle images both before and after they have gone through the ISP pipeline.
Step 1: Querying Platform Configuration
Step 2: Initialize SIPL
Step 3: Start SIPL
This section outlines the process of using the SIPL Framework.
Step 1: Querying Platform Configuration
First, use the SIPL Query component through the INvSIPLQuery interface to get a list of supported platform configurations. A platform configuration is structure (a struct) that lists properties of external devices and how they are connected to the platform. This includes but is not limited to:
Device I2C addresses
Image Properties (framerate, datatype, image size, etc.)
Deserializer MIPI properties (PHY mode, number of lanes, etc.)
Which camera modules are connected to which CSI brick
This information is returned in the form of a list of PlatformCfg structs. Choose the PlatformCfg struct that reflects the current configuration. It is not strictly necessary to use the SIPL Query component, and the PlatformCfg struct may be obtained by other means, or even hard coded into the application.
Step 2: Initialize SIPL
To initialize SIPL Core, first use the INvSIPLCamera::GetInstance method to get an instance of a class that implements INvSIPLCamera. The following methods can then be used to configure the settings that SIPL Core is to use during capture:
Once SIPL Core has been configured, call INvSIPLCamera::Init()to initialize SIPL. This programs the external devices to prepare them to stream images, and programs the NvMedia ICP and ISP components to prepare them to receive and process images. Note that this step prepares SIPL to capture and process images, but does not actually start the process.
Step 3: Start SIPL
Next, create SIPL Client objects to handle the images that are output by SIPL. A SIPL Client object implements the e interface and contains a callback function to handle NvMediaImage objects that are output by SIPL.
Once SIPL client objects have been created, the pipeline may be started using INvSIPLCamera::Start(). This method programs the externally connected devices to start streaming images. After this method is called, capture is active. SIPL starts receiving images, processing them through ISP, and sending them to the SIPL client objects created by the application.
SIPL Query
Query Drivers
SIPL DeviceBlock
Camera Module Drivers
SIPL Query
The SIPL Query component implements the INvSIPLCamera interface and is contained in the shared library libnvsipl_query.so. It maintains a database of supported external image devices and their properties. It also maintains a database of platform configurations. The database contains the following types of objects:
Database Object Type
An object that stores the settings for an image sensor external device.
An object that stores the settings for an EEPROM external device.
An object that stores the settings for a GMSL serializer external device.
An object that stores the properties for a camera module external device. This consists of one SerInfo object, zero or more EEPROMInfo objects, and zero or more SensorInfo objects.
An object that stores the settings for a GMSL deserializer external device.
An object that describes all external devices that are attached to the CSI ports of the SoC. This includes which DeserInfo objects are to be used for each camera group, and which CameraModuleInfo objects are to be used.
SIPL Query provides API calls to:
Parse the default database
Get information about all devices supported by the library
Get a list of all supported platform configurations
Retrieve a specific platform configuration by name
Parse a user-provided platform_config.json file to override the PlatformCfg structs in the SIPL Query Database
Apply a mask to enable only specific deserializer links in a specific platform configuration
Query Drivers
SIPL Query creates the database at runtime by searching for query drivers. A query driver is a shared library in the form libnvsipl_qry_*.so that contains a collection of database objects. SIPL Query searches the paths provided in the $LD_LIBRARY_PATH environment variable for the first directory named nvsipl_drv and opens all query drivers in it.
A screenshot of a cell phone Description automatically generated
The following table describes the query drivers distributed with the PDK and the database objects they contain.
Query Driver Name
Database Objects Contained
MAX96712 DeserInfo
MAX96705 SerInfo
MAX9295 SerInfo
MAX96759 SerInfo
AR0231 SensorInfo
SF3324 CameraModuleInfo
SF3325 CameraModuleInfo
SF3324 PlatformCfg
SF3325 PlatformCfg
AR0144 SensorInfo
AR0144 CameraModule
AR0144 PlatformCfg
Constellation SensorInfo
Constellation CameraModule
Constellation PlatformCfg
SIPL DeviceBlock
In NvMedia the SIPL framework’s DeviceBlock component is represented by the INvSIPLDeviceBlock interface. The libnvsipl_devblk.so shared library contains an implementation of this interface.
An INvSIPLDeviceBlock object is called a device block. It represents all external devices connected to a single deserializer. A device block is used to power on and initialize its associated external devices.
A DeviceBlock is a conceptual component of the SIPL framework. A device block is an instance of a class that implements INvSIPLDeviceBlock, the interface that represents a DeviceBlock in NvMedia. To avoid confusion, keep the distinction between these similar terms in mind.
The methods of INvSIPLDeviceBlock are called in the following sequence:
1. INvSIPLDeviceBlock::Init()powers on the deserializer and camera modules (if power control is enabled), sets up the serdes link between the deserializer and camera modules, and programs the settings on the image sensors. It stops just short of enabling streaming. If this method is successful, it leaves all external devices connected to the CSI brick ready to stream images, but no images are sent to the CSI brick yet. After this point, you can use INvSIPLDeviceBlock::GetProperty()to get handles to the NvMediaISCDevice objects that represent the external devices.
2. INvSIPLDeviceBlock::Start()enables streaming from the image sensors. If this method is successful, the CSI brick associated with the device block begins receiving images from its attached camera modules.
3. INvSIPLDeviceBlock::Stop()disables streaming from the image sensors. If this method is successful, the CSI brick associated with the device block no longer receives frames from its attached camera modules.
4. INvSIPLDeviceBlock::Deinit()powers down the deserializer and camera modules (if power control is enabled) and performs any other steps needed to deinitialize the external devices so they can be reinitialized and used again.
5. INvSIPLDeviceBlock::WaitForError()is a blocking call that waits for an error to occur on the deserializer. When an error occurs, this method is unblocked and returns.
6. INvSIPLDeviceBlock::DetectErrors()queries the deserializer to check for errors that occurred on the camera modules. If there are any errors, it prints the type of error and provides the index of the link on which the error occurred. This method is called after WaitForError().
7. INvSIPLDeviceBlock::ReconfigureLink()attempts to reconfigure a link that may have errors. This method is called after DetectErrors(). The method’s cameraModules parameter is an input that must be set by DetectErrors().
Camera Module Drivers
SIPL DeviceBlock uses camera module drivers to program external devices. A camera module driver is a shared library in the form libnvsipl_drv_*.so that is used to program specific type of camera module. SIPL DeviceBlock selects which camera module drivers to use based on the names of the camera modules that are attached to each GMSL link of the deserializer.
At runtime, SIPL DeviceBlock searches the paths defined in the environment variable $LD_LIBRARY_PATH for the first directory named nvsipl_drv, and opens all camera module drivers in it. This allows you to add support for more devices by adding third-party camera module drivers to the nvsipl_drv directory.
A screenshot of a cell phone Description automatically generated
The SIPL Core component implements the INvSIPLCamera interface. It initializes the external devices through the SIPL DeviceBlock component, and initializes the platform to capture and process images with the NvMedia ICP and NvMedia ISP APIs.
SIPL Core also provides the INvSIPLClient interface, which you can implement to get the images output by the capture processes.