Introduction to SIPL

The Safe Image Processing Library (SIPL) is NVIDIA’s modular, extensible framework for camera and image sensor integration, image processing, and control, which supports continuous streaming of image data from camera sensors.

SIPL provides a unified API and driver model for a wide range of camera hardware, which currently supports modern Camera-over-Ethernet (CoE) architecture for camera sensors based on Holoscan Sensor Bridge (HSB).

SIPL is designed for high reliability, safety, and ease of integration in Jetson embedded platforms and is released as a public SDK package for the Jetson AGX Thor Developer Kit.

CoE is NVIDIA’s next-generation camera connectivity solution, enabling high-performance, flexible, and scalable camera integration using standard Ethernet networks.

For more details on CoE architecture in SIPL framework, refer to Camera-over-Ethernet Overview.

For more details on HSB, refer to Holoscan Sensor Bridge.

SIPL with CoE Architecture Overview

The following diagram shows the SIPL framework supporting CoE architecture with MGBE and Ethernet-based control and data paths:

NVIDIA blocks (shown in green) cannot be modified by the user.

Public blocks (shown in orange) can be updated by the user with their own sample applications and drivers, compiled and deployed on device as needed, using the Jetson SIPL API package.

• For more details on SIPL application development, refer to NvSIPL CoE Camera Application Developer Guide.

• For more details on Unified Device Driver Framework (UDDF) driver development, refer to Guide to Writing UDDF Drivers.

Simplified SIPL Framework Flow Diagram

                           +----------------+
                           | Client Apps    |
                           +-------+--------+
                                   |
                                   v
        +----------------------------------------------------------+
        |                       SIPL Core                          |
        +----------------------------------------------------------+
            |                      |
            |                      v
            |         +-----------------------------+
            |         | Transport Abstraction       |
            |         | (CoE, GMSL, etc.)           |
            |         +------------+----------------+
            |                      |
            v                      v
+----------------+      +---------------------+          +-------------------+
| Image Proc.    |      | UDDF Driver Layer   | <------> | HSL (Hardware     |
| Pipeline       |      +----------+----------+          | Sequence Language)|
+-------+--------+                 |                     +-------------------+
                                   |
                                   |
                                   v
                      +--------------------------------+
                      | Sensor Input (CoE, GMSL, etc.) |
                      +--------------------------------+

SIPL Core also connects to Query API and Control API.

Key Directories (Public Headers and Samples)

Sample code for SIPL application and UDDF driver development is available in the SIPL package, which can be compiled and deployed on a Jetson device as needed.

Refer to the following directories:

sipl/include/                # Main SIPL public API headers
sipl/samples/                # SIPL samples (applications)
uddf/samples/                # UDDF samples (drivers)
uddf/include/uddf/           # UDDF public headers (ddi, cdi, helpers)

Key Components of SIPL Framework

A brief overview of key components of SIPL framework:

• JSON Configuration

  • Camera sensor and transport configuration is described in JSON files (for example, vb1940_config.json).

  • These files define camera names, MAC and IP addresses, HSB IDs, and transport parameters.

  • For more details, see SIPL Query JSON Guide for CoE Camera Development.

• SIPL Query API

  • Query APIs are methods for SIPL applications to enumerate, validate, and configure camera and transport information and configuration.

  • Loads and parses JSON configuration files.

  • For more details, see sipl/include/NvSIPLCameraQuery.hpp.

• SIPL Control API

  • Provides APIs for SIPL applications to control camera and transport.

  • For more details, see sipl/include/NvSIPLCameraControl.hpp.

• SIPL Application Structure

  • SIPL applications use SIPL Query and Control APIs to configure and control camera sensors.

  • The sipl/samples/ directory contains SIPL application samples.

• Camera Sensor Driver Structure

  • Sensor drivers are implemented as UDDF shared libraries.

  • Currently, CoE sensor drivers are provided with the SIPL package.

  • All hardware access is performed via Hardware Sequence Language (HSL) sequences and hololink drivers.

  • Sensor drivers are discovered and loaded dynamically by UDDF and CameraHAL.

  • The sipl/uddf/samples/ directory contains UDDF sensor driver samples.

• Key Interfaces and Classes

  • SIPL interfaces: INvSIPLCamera, INvSIPLCameraQuery, INvSIPLClient, and so on. (See sipl/include/).

  • UDDF interfaces for sensor drivers: IDriver, IHardwareAccess, ISensorControl, and so on. (See uddf/include/uddf/).

• Key Structures

  • CameraConfig, TransportConfig, PlatformCfg, CameraSystemConfig, and so on. (See sipl/include/NvSIPLCameraTypes.hpp).

  • UDDF: DeviceTable, SensorAttributes, and so on. (See uddf/include/uddf/).

SIPL Use Cases

• Industrial and robotics vision systems.

• Smart city and surveillance camera networks, with continuous streaming support.

• High-speed, high-reliability image capture pipelines.

• Rapid prototyping and validation of new camera modules (using CoE camera modules).

• Flexible support for any type of camera transports using Camera HW abstraction layer (for future development)

• Safety, reliability, and certification (for future development)

Benefits and Limitations Compared to Previous Camera Software Architecture

Benefits

• Unified, abstracted API for supported cameras with HSB interface.

• Modular, extensible driver model (UDDF).

• Hardware abstraction via UDDF and HSL for safety and portability.

• JSON-based configuration for easy deployment.

• Designed for safety, reliability, and certification (for future development).

• Better CPU utilization and resource optimization with lower latency for image processing (for future development).

• Better support for multi-camera streaming systems and multi-camera topologies (such as daisy-chain and star).

Limitations

• Requires C++17 toolchain and modern build system.

• Some advanced features might require vendor-specific driver extensions.

• Hardware support is limited to platforms with SIPL integration.

CoE in SIPL

• CoE is a new transport solution provided with the SIPL framework, enabling camera connectivity over standard Ethernet networks.

• Abstracted to applications using SIPL APIs.

• Enables longer cable runs, standard networking, and easier integration with existing infrastructure.

• For more details, see Camera-over-Ethernet Overview.

UDDF

• UDDF is NVIDIA’s standardized driver framework for camera and multimedia devices.

• Provides a clear separation between driver logic (DDI) and hardware access (CDI).

• Enables third-party and vendor driver development with minimal dependencies.

• Supports dynamic driver discovery, loading, and management.

• All hardware access is performed via HSL, ensuring safety and abstraction.

HSL

• HSL is a domain-specific language for describing hardware access sequences (such as I2C and GPIO).

• Used by UDDF drivers to interact with hardware in a safe, abstracted way.

• HSL sequences can be static (precompiled) or dynamic (built at runtime).

• PyHSL is the Python-based source language for authoring HSL sequences.

• HSL enables safety certification and portability by minimizing driver code complexity.

For more details, see the following documentation:

• HSL and UDDF Overview

• Guide to Writing UDDF Drivers

• PyHSL

• Integrating UDDF Drivers with SIPL

Configuration Using SIPL Query and JSON Database

SIPL supports a database-driven configuration model (JSON) to describe cameras, modules, and transports.

Applications can load and validate configurations at runtime via the SIPL Query API, or construct equivalent configurations programmatically.

• SIPL Query decouples camera configuration from application code.

• JSON files define all camera, transport, and module settings, based on the sensor configuration.

• The SIPL Query API loads, validates, and provides these settings to SIPL Core at runtime to initialize the camera pipeline.

• Alternatively, a SIPL application can take JSON files as input to configure the camera and transport settings.

• For more details, see SIPL Query JSON Guide for CoE Camera Development.

Example JSON fragments for a CoE camera:

 {
   "cameraConfigs": [
    {
      "name": "VB1940_Camera",
      "type": "CoE",
      "mac_address": "8c:1f:64:6d:70:03",
      "ip_address": "192.168.1.2",
      "hsb_id": 0
    },
    "transportSettings": [
    {
      "name": "VB1940_Camera",
      "compatible_name": "HsbTransport",
      "type": "CoE",
      "description": "Transport settings for VB1940 camera",
      "if_name": "mgbe0_0",
      "ip_address": "192.168.1.2",
      "vlan_enable": false,
      "hsb_id": 0
    }
  ]
}

Public API Quick Start

The following snippets show a typical flow using SIPL APIs to query configuration and initialize a pipeline. Exact API names and namespaces depend on the installed SIPL package version. For more details, see the README.md files in the SIPL package.

Example code snippet:

#include "NvSIPLCameraQuery.hpp"
#include "NvSIPLCamera.hpp"

int main() {
    // 1) Query and load configuration (JSON database)
    std::unique_ptr query = INvSIPLCameraQuery::GetInstance();
    SIPLStatus status = query->ParseDatabase();
    if (status != NVSIPL_STATUS_OK) { return -1; }

    // Retrieve camera device/transport info as needed
    const CameraDeviceInfoList* deviceList = query->GetDeviceInfoList();
    for (const auto& camera : deviceList->cameraSystemConfig.cameras) {
        std::cout << "Camera: " << camera.name << " (ID: " << camera.sensorInfo.id << ")" << std::endl;
    }

    // ... select desired camera and transport ...

    // 2) Initialize camera pipeline
    auto camera = std::make_unique<nvsipl::CNvSIPLCamera>();
    nvsipl::CameraConfig CameraConfig{}; // Fill from query results or manual config
    camera->SetCameraConfig(CameraConfig);

    status = camera->Init();
    if (status != NVSIPL_STATUS_OK) { return -2; }

    status = camera->Start();
    if (status != NVSIPL_STATUS_OK) { return -3; }

    // 3) Capture loop (simplified)
    // Use SIPL APIs to acquire frames and process them

    // 4) Shutdown
    camera->Stop();
    camera->Deinit();
    return 0;
}

Troubleshooting and Diagnostics

• Validate your JSON configuration using SIPL Query before deployment.

• Ensure all required drivers are installed and referenced in the Camera config, as applicable.

• Check logs for configuration or driver errors.

• Use error handling APIs to retrieve detailed diagnostics.

• Common issues:

  • Incorrect MAC or IP addresses in JSON.

  • Network interface not up or misconfigured.

  • Missing or incompatible driver libraries.

  • Insufficient memory or bandwidth for camera streams.

Quick Start

1. Configure your camera system: Write your JSON file. For more details, refer to SIPL Query JSON Guide for CoE Camera Development.

2. Write your application: For more details, see NvSIPL CoE Camera Application Developer Guide.

3. Troubleshooting:

   • Check logs for configuration or driver errors.

   • Use error handling APIs to retrieve detailed diagnostics.

   • Ensure all required drivers are installed and referenced in the Camera config.