Introduction

The NVIDIA^® DriveWorks Image Signal Processing (ISP) Pipeline processes raw images acquired from GMSL cameras and outputs images that can be used for human vision (HV) or machine vision (MV) applications. Refer to Figure 1 for the DriveWorks ISP Pipeline for common use cases.

Figure 1: DriveWorks ISP Pipeline

The ISP is a hardware component on the DRIVE platform whose interface resides in the lower level NvMedia API (Image Processing Pipeline - IPP) used by the camera sensor (see Camera). This HW ISP is automatically triggered by the camera sensor (gmsl) when a processed image is requested by the user. The output is suitable for HV applications. As it currently stands, the user has no control over this HW ISP component, besides the creation of it by means of camera parameters (see Camera Sample). If no processed output is requested, then the HW ISP component will not be activated and the camera sensor would output non-processed raw images (see Camera for the details about output formats).

For MV vision applications, a more flexible Software ISP component exists, which operates in a different and independent way of the camera sensor. In order to be able to use this component, a RAW image must be acquired by the camera sensor (HW ISP off) or read from a video source (For information on recording and replaying, see: Recording Tools and Recording Sensor Data) and given to the SoftISP. The processed image output depends on the setup of the SoftISP and its use cases include deep learning inference.

Recording using the DriveWorks Recording Tools is available for both of the HW and Soft ISP paths; however, the format of the stored images and subsequent usage is dependent on the path. Images produced through HW ISP can be recorded in .h264 or .h265 format; playback of these recorded images is suitable for HV use cases only. In the Soft ISP path, RAW (or LRAW) images can be captured prior to entering Soft ISP processing. During playback of RAW images, DriveWorks provides the capability to perform Soft ISP processing on these stored images so that they can been used in downstream MV algorithms.

Figure 2: Playback

Assumptions

The SoftISP is optimized for specific modules of ar0231 Sekonix cameras, where the supported module is dependent on the NVIDIA DRIVE platform. These cameras output RAW images with Bayer patterns of RCCB/RCCC. The pipeline also assumes Bracketed Auto Exposure (BAE) control. For more information, see Camera.

SoftISP is available for the following sensors on NVIDIA DRIVE^™ AGX Developer Kit:

ar0231-rccb-bae-sf3325, Sekonix rev 7
ar0231-rccb-bae-sf3324, Sekonix rev 7

There is limited capability with CCCC for ar0144 cameras.

ISP Pipeline

Figure 3: NvMedia AR0231 Processing

Hardware ISP for HV Use Cases

The NvMedia ISP component processes Bayer images to YUV formatted images using Xavier-configurable ISP hardware (HW ISP). Refer to the "Image Processing and Management" section in the NVIDIA DRIVE OS 5.1 Linux Development Guide for more information.

Software ISP for MV Use Cases

The DriveWorks SoftISP module processes RAW12 RCCB images into images that are compatible with subsequent DriveWorks MV applications, such as DriveNet. Unlike the NvMedia IPP framework, DriveWorks ISP processing is not done in the ISP hardware, but in software (Soft ISP). The DriveWorks Soft ISP flow is depicted in Figure 4.

Figure 4: DriveWorks Soft ISP

SoftISP Module Overview

The SoftISP module functions as a pipeline with multiple outcomes and outputs, depending on configuration. The input is a raw image from camera or from raw/lraw video and the input parameters are the size of the input image, the camera raw format, and the tone map type. The output is one, two, or all three of the following images:

Bayer
Demosaic
Tone Map

Bayer

The Bayer output is enabled only if the following conditions are satisfied:

The requested output type, dwSoftISPProcessType enum, is ::DW_SOFTISP_PROCESS_TYPE_BAYER
The dwSoftISPDenoiseMethod enum is set to ::DW_SOFTISP_DENOISE_METHOD_BILATERAL.
The User has allocated a Bayer output image with the properties from dwSoftISP_getBayerImageProperties() and bound with dwSoftISP_bindOutputBayer().

In this case, the module decompands the raw input and applies a bilateral filter. It then stores the result in the bound image.

Demosaic

The demosaic output is enabled if the following conditions are satisfied:

The requested output type, dwSoftISPProcessType enum, is ::DW_SOFTISP_PROCESS_TYPE_DEMOSAIC
The User has allocated an image with the properties from dwSoftISP_getDemosaicImageProperties() and bound with dwSoftISP_bindOutputDemosaic().

In this case, the bound image will contain a demosaiced non-raw, RCB/RCC float16 image.

Tone Map

The tone-mapped output is enabled if the following conditions are satisifed:

The requested output type, dwSoftISPProcessType enum, is ::DW_SOFTISP_PROCESS_TYPE_TONEMAP
The User has allocated an image using the properties of the demosaiced image, with the exception of the format, which can be RGB/RGBA, uint8, or float16.
- A demosaic image must be allocated and bound even if the requested output is not demosaic (see table below).
The SoftISP has been created with a dwTonemapType set in the parameters using dwSoftISP_setTonemapType().

SoftISP Behavior

The following table specifies the behavior and characteristics of the SoftISP.

The Pipeline Behavior column describes the expected processing behavior given the configuration. The "state" of the ISP depends solely on what images are bound and the parameters given, so no memory is implicitly stored. For this reason we show special behaviors that might be less intuitive. Failure cases are also included for completeness. The different behaviors are shown for informative purposes, we advise the user to always go for the most efficient path.

Keywords:

Output : The output image types, based on dwSoftISPProcessType. For convenience, the the values used in this column are given by:
- bayer = ::DW_SOFTISP_PROCESS_TYPE_BAYER, note that when this output is requested, a valid denoise method must be set
- demosiac = ::DW_SOFTISP_PROCESS_TYPE_DEMOSAIC
- tonemap = ::DW_SOFTISP_PROCESS_TYPE_TONEMAP
- Note that there can be 1, 2, or 3 output images, depending on configuration.
bound : the image X corresponding to Bayer, Demosaic or Tonemap has been bound to the SoftISP through dwSoftISP_bindOutputX
unbound : the image X corresponding to Bayer, Demosaic or Tonemap has not been bound (default or by binding a nullptr)
any : any state, won't influence the behavior
-> : the processing steps are sequential
& : the processing steps are performed in one pass
Input :
- raw : raw image bound by dwSoftISP_bindInputRaw()
- bayer : The bayer image currently bound by dwSoftISP_bindOutputBayer() will be used as input instead of the raw image.

Output	Bayer	Demosaic	Tonemap	dwSoftISPDenoiseMethod	Input	Pipeline Behavior	Comment
bayer	bound	any	any	`::DW_SOFTISP_DENOISE_METHOD_BILATERAL`	raw	debayering
bayer+demosaic	bound	bound	any	`::DW_SOFTISP_DENOISE_METHOD_BILATERAL`	raw	debayering&demosaicing
demosaic	unbound	bound	any	any	raw	demosaicing
demosaic	bound	bound	any	any	bayer	demosaicing
demosaic+tonemap	bound	bound	bound	any	raw	demosaicing->tonemapping
demosaic+tonemap	unbound	bound	bound	any	bayer	demosaicing->tonemapping
bayer+demosaic+tonemap	bound	bound	bound	`::DW_SOFTISP_DENOISE_METHOD_BILATERAL`	raw	debayering&demosaicing->tonemapping
bayer+tonemap	bound	any	bound	any	any	FAIL	demosaic must be requested as output when tonemap is requested
tonemap	any	unbound	bound	any	any	FAIL	demosaic must be also bound when tonemap is requested
tonemap	any	bound	bound	any	demosaic	demosaicing->tonemaping

Any deviation from the above table, will result in either failure or undefined behavior.

Example #1

Interpretation: You have set dwSoftISPProcessType to ::DW_SOFTISP_PROCESS_TYPE_BAYER | ::DW_SOFTISP_PROCESS_TYPE_DEMOSAIC, you have called dwSoftISP_bindOutputBayer(), dwSoftISP_bindOutputDemosaic(), and optionally dwSoftISP_bindOutputTonemap(). The resultant pipeline will take the raw input and perform bayer and demosaicing together and output a Bayer image and a Demosaiced image.

Example #2

Interpretation: You have set dwSoftISPProcessType to ::DW_SOFTISP_PROCESS_TYPE_DEMOSAIC, you have called dwSoftISP_bindOutputBayer(), dwSoftISP_bindOutputDemosaic(), and optionally dwSoftISP_bindOutputTonemap(). The resultant pipeline will take Bayer input and output a demosaiced image.

The following table shows outputs based on Pipeline Behavior of the bayer and demosaic steps. (DW_IMAGE_FORMAT is omitted). Please see the above description for Pipeline Behavior.

Pipeline Behavior	Demosaic ROI	dwSoftISPDemosaicMethod	Input	Bayer Output	Demosaic Output	Comment
raw->demosaic	Full Image (WxH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	n.a.	RCB_FLOAT16, W/2xH/2	Optimized path (Fused Kernel)
raw->demosaic	Subregion (CWxCH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	n.a.	RCB_FLOAT16, CW/2xCH/2	-
raw->demosaic	Full Image (WxH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	n.a.	RCB_FLOAT16, WxH	-
raw->demosaic	Subregion (CWxCH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	n.a.	RCB_FLOAT16, CWxCH	-
raw->bayer	Full Image (WxH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	RAW_FLOAT16, WxH	n.a.	-
raw->bayer	Subregion (CWxCH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	RAW_FLOAT16, WxH	n.a.	-
raw->bayer	Full Image (WxH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	RAW_FLOAT16, WxH	n.a.	-
raw->bayer	Subregion (CWxCH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	RAW_FLOAT16, WxH	n.a.	-
bayer->demosaic	Full Image (WxH)	DEMOSAIC_DOWNSAMPLE	RAW_FLOAT16, WxH	n.a.	RCB_FLOAT16, W/2xH/2	-
bayer->demosaic	Subregion (CWxCH)	DEMOSAIC_DOWNSAMPLE	RAW_FLOAT16, WxH	n.a.	RCB_FLOAT16, CW/2xCH/2	-
bayer->demosaic	Full Image (WxH)	DEMOSAIC_INTERPOLATION	RAW_FLOAT16, WxH	n.a.	RCB_FLOAT16, WxH	-
bayer->demosaic	Subregion (CWxCH)	DEMOSAIC_INTERPOLATION	RAW_FLOAT16, WxH	n.a.	RCB_FLOAT16, CWxCH	-
raw->bayer&demosaic	Full Image (WxH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	RAW_FLOAT16, WxH	RCB_FLOAT16, W/2xH/2	Optimized path (Fused Kernel)
raw->bayer&demosaic	Subregion (CWxCH)	DEMOSAIC_DOWNSAMPLE	RAW_UINT16, WxH	RAW_FLOAT16, WxH	RCB_FLOAT16, CW/2xCH/2	-
raw->bayer&demosaic	Full Image (WxH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	RAW_FLOAT16, WxH	RCB_FLOAT16, WxH	-
raw->bayer&demosaic	Subregion (CWxCH)	DEMOSAIC_INTERPOLATION	RAW_UINT16, WxH	RAW_FLOAT16, WxH	RCB_FLOAT16, CWxCH	-

Note: If the starting image format in the Pipeline Behavior section is not raw, then the module assumes that the input is a valid bound image.; For ar0144 cameras, the output can only be TONEMAP RGB/RGBA uint8.

FAQS

Refer to Frequently Asked Questions

Relevant Tutorials

Software ISP Workflow

SoftISP APIs

soft_ISP

Table of Contents