Understanding NvMedia SIPL Framework
The NvMedia Sensor Input Processing Library (SIPL) framework provides a simplified API to capture the output of image sensors connected to NVIDIA DRIVE AGX Xavier™ platforms. SIPL is implemented in C++ on top of the NvMedia API.
The purpose of SIPL is to abstract the following operations from the application layer:
• Programming the image sensors, EEPROMs, serializers, and deserializers.
• Programming DRIVE AGX Xavier to capture and process the images using hardware image processing pipelines (ISPs)..
Glossary
This section defines some terms that are important to understand the SIPL framework.
• SIPL: The Sensor Input Processing Library.
• External image devices: Sensors, camera module EEPROMs, serializers, and deserializers.
• Platform configuration: A data structure that describes the interconnection of external devices and connection of deserializers to the CSI interfaces of Xavier.
• SIPL query: A SIPL component that retrieves information about supported external devices and platform configurations from a database maintained in JSON files.
• SIPL camera: A SIPL component that provides the API to manage programming of external image devices and Xavier for a given platform configuration
• Camera module: The physical grouping of a serializer, sensor(s), and EEPROM(s).
• SIPL client: An SIPL component that provides an API to receive an output of the sensor pipeline.
• SIPL DeviceBlock: A SIPL component, represented by the InvSIPLDeviceBlock interface class, which provides the API to manage programming of external image devices.
• SIPL pipeline manager: A SIPL component that captures the output of a sensor via NvMedia Image Capture Processing (ICP) and optionally processes the images using NvMedia Image Signal Processing (ISP).
• Streamer: An SIPL component that streams the output of a SIPL pipeline to SIPL clients.
Architecture
The following diagram illustrates the architecture of the SIPL framework.
A query parses the database and retrieves information about external image devices and platform configurations. It provides API calls to select or create a platform configuration.
A DeviceBlock programs a deserializer and the camera modules connected to it using ISC drivers.
The PDK provides code that implements both queries and DeviceBlocks in binary form. Source code may be provided, if needed, on a case-by-case basis.
A camera implements structures and interfaces defined in the INvSIPLCamera interface class. These structures and interfaces connect external image devices to a pipeline manager, using a DeviceBlock. It consists of following sub-components:
• A DeviceBlock manager creates and manages an instance of a DeviceBlock for each deserializer in the platform configuration.
• A pipeline manager creates and manages the NvMediaICP-to-NvMediaISP pipeline for each sensor in the platform configuration.
• A client provides an interface to a SIPL client application which the application can register with SIPL, and through which it can receive output of the image processing pipeline.
Query
A query parses a database containing information about external image devices and their interconnections. The database is stored in JSON files which are kept in the directory:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/query/database/
The following table describes currently supported devices.
JSON file name | Device Type | Supported devices |
image_sensor.json | Image sensor | OnSemi AR0231 OnSemi AR0144 Constellation |
serializer.json | Serializer | Maxim MAX96705 Maxim MAX96759 |
deserializer.json | Deserializer | Maxim MAX96712 |
eeprom.json | EEPROM | OnSemi N24C64 |
camera_module.json | Camera module. Describes a physical grouping of specific serializers, image sensor(s), and EEPROM(s). | Sekonix SF3324 module: 120° FOV, DVP AR0231-RCCB, MAX96705. Sekonix SF3325 module: 60° FOV, DVP AR0231-RCCB, MAX96705. AR0144P module: AR0144, MAX96705. Constellation: Constellation capture, MAX96759. |
The database also includes the file platform_config.json, which contains a list of supported platform configurations. The platform configurations are represented as an array of DeviceBlockInfo objects. Each DeviceBlockInfo object describes a configuration for a specific CSI brick, the associated deserializer, the deserializer links, and the modules connected on each link.
The schemas of these files will be documented in later releases.
A query provides SIPL query API calls to:
• Parse the default database
• Get information about all devices supported by the library
• Get a list of all supported platform configurations
• Retrieve a specific platform configuration by name
• Parse a user-provided platform_config.json file to override the default file
• Apply a mask to enable only specific deserializer links in a specific platform configuration
The PDK defines a reference implementation of a query in:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/query/
• A call to parse the user specified configuration file, e.g. platform_config.json
• A call to modify the configuration defined by the configuration file via masks to enable specific links tp the deserializer
DeviceBlock
The following diagram shows the architecture of a DeviceBlock.
A DeviceBlock manages the programming of external image devices on a specific CSI brick. The PDK provides a reference implementation of a DeviceBlock in the directory:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/devblk/
A DeviceBlock is implemented in the INvSIPLDeviceBlock class described in:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/devblk/include/NvSIPLDeviceBlock.hpp
The INvSIPLDeviceBlock Class
INvSIPLDeviceBlock describes the interfaces offered by a DeviceBlock. It exports these functions:
• static void GetVersion(Version& version)
Returns version information for the library libnvsipl_devblk.so.
• static std::unique_ptr <INvSIPLDeviceBlock> Create(void)
Creates an instance of the implementation class and returns a unique_ptr which points to it.
• virtual SIPLStatus SetConfig(const DeviceBlockInfo* deviceBlockInfo)
eSets the configuration spepcified by deviceBlockInfo.
• virtual SIPLStatus Init(void)
Initializes the DeviceBlock.
• virtual SIPLStatus Start(void)
Starts the DeviceBlock.
• virtual SIPLStatus Stop(void)
Stops the DeviceBlock.
• virtual SIPLStatus Deinit(void)
Deinitializes the DeviceBlock.
• virtual SIPLStatus GetProperty(DeviceBlockProperty& property) const
Returns the property of the DeviceBlock.
Camera
The INvSIPLCamera class defines the interfaces classes used to manage a camera platform configuration.
The DeviceBlock implements e as described in:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/include/NvSIPLCamera.hpp
The INvSIPLCamera Class
INvSIPLCamera describes the interface classes offered by the camera. It exports these functions:
• static std::unique_ptr<INvSIPLCamera> GetInstance(void)
Returns a unique_ptr which points to the instance of NvSIPLCamera.
• SIPLStatus SetPlatformCfg(const PlatformCfg* platformCfg)
Specifies a platform configuration.
• SIPLStatus SetOutputDesc(std::vector<INvSIPLClient::ConsumerDesc>)
Specifies a vector of descriptors of consumers of outputs of pipelines.
• SIPLStatus Init(void)
Initializes the camera platform.
• INvSIPLClient::ClientDesc* GetClientDesc(std::uint32_t index,INvSIPLClient::ConsumerDesc::OutputType outType, INvSIPLClient::ConsumerDesc::BufferType bufType)
Gets a descriptor for the client. The descriptor can be used to create the client.
• SIPLStatus Start(void)
Starts the camera platform.
• SIPLStatus Stop(void)
Stops the camera platform.
• SIPLStatus Deinit(void)
Deinitializes the camera platform.
• SIPLStatus SetImagePoolAttributes(std::uint32_t index,INvSIPLClient::ConsumerDesc::OutputType outType,uint32_t numOfImages,const NvSIPLImageAttr &imageAttr)
Specifies attributes of the image buffer pool used by the pipeline for a specific sensor.
• SIPLStatus SetNotifier(std::uint32_t uIndex, NvSIPLPipelineNotifier* pNotifier)
Specifies a notifier to receive the pipeline event notifications.
• SIPLStatus SetImageGroupWriterCallback(std::uint32_t uIndex, NvSIPLImageGroupWriter* pImageGroupWriter)
Specifies an image group writer callback function to populate the RAW image group to be used for reprocess a use case.
• const PlatformCfg* GetPlatformCfg(void) const
Gets a handle to the camera platform configuration that was specified by SetPlatformCfg.
Sample Application
The PDK contains a sample application, nvsipl_camera, which demonstrates how to use the SIPL API to program the external devices, create pipelines with multiple outputs, and receive output from sensor and image processing pipelines. The test application code is under the directory:
$PDK_INSTALL_DIR/samples/nvmedia/nvsipl/test/camera/
See the SIPL topic
Sensor Input Processing Library) Camera for more details about usage.
Porting from ExtImgDev to SIPL DeviceBlock
A SIPL DeviceBlock is functionally equivalent to the ExtImgDev library, implemented in the PDK as libnvsipl_devblk.so.
A SIPL DeviceBlock has following advantages over the ExtImgDev library:
• Ease of adding support for a new sensor
• Support for heterogeneous platform configurations (those in which different types of camera modules are connected to the same camera port)
ExtImgDev is deprecated for AGX Xavier-based platforms, and will be dropped after the SIPL DeviceBlock is extensively adopted.
The Following table describes an approximate mapping of ExtImgDev API to SIPL.
ExtImgDev API | SIPL DeviceBlock API |
NvMediaStatus ExtImgDevCheckVersion(ExtImgDevVersion *version) | static void GetVersion(Version& version) |
ExtImgDevice* ExtImgDevInit(ExtImgDevParam *configParam) | static std::unique_ptr <INvSIPLDeviceBlock> Create(void) virtual SIPLStatus SetConfig(const DeviceBlockInfo* deviceBlockInfo) virtual SIPLStatus Init(void) virtual SIPLStatus GetProperty(DeviceBlockProperty& property) const |
NvMediaStatus ExtImgDevStart(ExtImgDevice *device) | virtual SIPLStatus Start(void) |
void ExtImgDevStop(ExtImgDevice *device) | virtual SIPLStatus e(void) |
void ExtImgDevDeinit(ExtImgDevice *device); | virtual SIPLStatus Deinit(void) |
NvMediaStatusExtImgDevGetError(ExtImgDevice *device, uint32_t *link, ExtImgDevFailureType *errorType); | To be defined |
NvMediaStatusExtImgDevRegisterCallback(ExtImgDevice *device, uint32_t sigNum, void (*cb) (void *), void *context); | To be defined |
NvMediaStatusExtImgDevWaitForError(ExtImgDevice *device); | To be defined |
NvMediaStatusExtImgDevAbortWaitForError(ExtImgDevice *device); | To be defined |
Dependencies
SIPL depends on the following libraries and components in the NVIDIA® DRIVE™ PDK.
Library | Description |
libnvmedia.so | NvMedia core library |
libnvmedia_isc.so | NvMedia ISC implementation |
libEGL.so | EGL library for EGLStream |
libcuda.so | CUDA library |
libnviscdrv.a | Static library containing ISC drivers for external devices |
libjsoncpp.a | Static library containing an open source implementation libjsoncpp |
libnvmedia_ipp.so | NvMedia IPP implementation |
libnvmedia_acp.so | NvMedia auto control plugin |
