Understanding NvMedia SIPL Framework

The NvMedia Sensor Input Processing Library (SIPL) framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX Xavier™ and NVIDIA® DRIVE AGX Pegasus™ platforms. SIPL is implemented in C++ on top of the NvMedia API.

The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.

The SIPL DeviceBlock is a component of the SIPL framework which initializes and controls the external devices attached to the platform. It uses I2C to program the settings specified in the PlatformCfg struct. It initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. The application does not need to use this component directly; it is instead used by SIPL Core.

The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL DeviceBlock component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to register callbacks to handle images both before and after they have gone through the ISP pipeline.

First, use the SIPL Query component through the INvSIPLQuery interface to get a list of supported platform configurations. A platform configuration is structure (a struct) that lists properties of external devices and how they are connected to the platform. This includes but is not limited to:

This information is returned in the form of a list of PlatformCfg structs. Choose the PlatformCfg struct that reflects the current configuration. It is not strictly necessary to use the SIPL Query component, and the PlatformCfg struct may be obtained by other means, or even hard coded into the application.

To initialize SIPL Core, first use the INvSIPLCamera::GetInstance method to get an instance of a class that implements INvSIPLCamera. The following methods can then be used to configure the settings that SIPL Core is to use during capture:

Once SIPL Core has been configured, call INvSIPLCamera::Init()to initialize SIPL. This programs the external devices to prepare them to stream images, and programs the NvMedia ICP and ISP components to prepare them to receive and process images. Note that this step prepares SIPL to capture and process images but does not actually start the process.

Next, create SIPL Client objects to handle the images that are output by SIPL. A SIPL Client object implements the INvSIPLClient interface and contains a callback function to handle NvMediaImage objects that are output by SIPL.

Once SIPL client objects have been created, the pipeline may be started using INvSIPLCamera::Start(). This method programs the externally connected devices to start streaming images. After this method is called, capture is active. SIPL starts receiving images, processing them through ISP, and sending them to the SIPL client objects created by the application.

The SIPL Query component implements the INvSIPLQuery interface and is contained in the shared library libnvsipl_query.so. It maintains a database of supported external image devices and their properties. It also maintains a database of platform configurations. The database contains the following types of objects:

Database Object Type	Description
SensorInfo	An object that stores the settings for an image sensor external device.
EEPROMInfo	An object that stores the settings for an EEPROM external device.
SerInfo	An object that stores the settings for a GMSL serializer external device.
CameraModuleInfo	An object that stores the properties for a camera module external device. This consists of one SerInfo object, zero or more EEPROMInfo objects, and zero or more SensorInfo objects.
DeserInfo	An object that stores the settings for a GMSL deserializer external device.
PlatformCfg	An object that describes all external devices that are attached to the CSI ports of the SoC. This includes which DeserInfo objects are to be used for each camera group, and which CameraModuleInfo objects are to be used.

SIPL Query creates the database at runtime by searching for query drivers. A query driver is a shared library in the form libnvsipl_qry_*.so that contains a collection of database objects. SIPL Query searches the paths provided in the $LD_LIBRARY_PATH environment variable for the first directory named nvsipl_drv and opens all query drivers in it.

Query Driver Name	Database Objects Contained
libnvsipl_qry_serdes.so	MAX96712 DeserInfo MAX96705 SerInfo MAX9295 SerInfo MAX96759 SerInfo
libnvsipl_qry_ar0231.so	AR0231 SensorInfo SF3324 CameraModuleInfo SF3325 CameraModuleInfo SF3324 PlatformCfg SF3325 PlatformCfg
libnvsipl_qry_ar0144.so	AR0144 SensorInfo AR0144 CameraModule AR0144 PlatformCfg
libnvsipl_qry_constellation.so	Constellation SensorInfo Constellation CameraModule Constellation PlatformCfg

In NvMedia the SIPL framework’s DeviceBlock component is represented by the INvSIPLDeviceBlock interface. The libnvsipl_devblk.so shared library contains an implementation of this interface.

An INvSIPLDeviceBlock object is called a device block. It represents all external devices connected to a single deserializer. A device block is used to power on and initialize its associated external devices.

Note:

A DeviceBlock is a conceptual component of the SIPL framework. A device block is an instance of a class that implements INvSIPLDeviceBlock, the interface that represents a DeviceBlock in NvMedia. To avoid confusion, keep the distinction between these similar terms in mind.

1. INvSIPLDeviceBlock::Init()powers on the deserializer and camera modules (if power control is enabled), sets up the serdes link between the deserializer and camera modules, and programs the settings on the image sensors. It stops just short of enabling streaming. If this method is successful, it leaves all external devices connected to the CSI brick ready to stream images, but no images are sent to the CSI brick yet. After this point, you can use INvSIPLDeviceBlock::GetProperty()to get handles to the NvMediaISCDevice objects that represent the external devices.

2. INvSIPLDeviceBlock::Start()enables streaming from the image sensors. If this method is successful, the CSI brick associated with the device block begins receiving images from its attached camera modules.

3. INvSIPLDeviceBlock::Stop()disables streaming from the image sensors. If this method is successful, the CSI brick associated with the device block no longer receives frames from its attached camera modules.

4. INvSIPLDeviceBlock::Deinit()powers down the deserializer and camera modules (if power control is enabled) and performs any other steps needed to deinitialize the external devices so they can be reinitialized and used again.

5. INvSIPLDeviceBlock::WaitForError()is a blocking call that waits for an error to occur on the deserializer. When an error occurs, this method is unblocked and returns.

6. INvSIPLDeviceBlock::DetectErrors()queries the deserializer to check for errors that occurred on the camera modules. If there are any errors, it prints the type of error and provides the index of the link on which the error occurred. This method is called after WaitForError().

7. INvSIPLDeviceBlock::ReconfigureLink()attempts to reconfigure a link that may have errors. This method is called after DetectErrors(). The method’s cameraModules parameter is an input that must be set by DetectErrors().

SIPL DeviceBlock uses camera module drivers to program external devices. A camera module driver is a shared library in the form libnvsipl_drv_*.so that is used to program a specific type of camera module. SIPL DeviceBlock selects which camera module drivers to use based on the names of the camera modules that are attached to each GMSL link of the deserializer.

At runtime, SIPL DeviceBlock searches the paths defined in the environment variable $LD_LIBRARY_PATH for the first directory named nvsipl_drv, and opens all camera module drivers in it. This allows you to add support for more devices by adding third-party camera module drivers to the nvsipl_drv directory.

The SIPL Core component implements the INvSIPLCamera interface. It initializes the external devices through the SIPL DeviceBlock component, and initializes the platform to capture and process images with the NvMedia ICP and NvMedia ISP APIs.