The Software Image Signal Processor (ISP) converts RAW images from a specific camera and converts them into images with a linear response curve.
The output format can be either a Bayer pattern image, a demosaiced image, or both. Bayer image means a linear response image that has the same position varying color filter as the camera physical sensor. The demosiced image is the final result where each pixel has all color channels available, either via interpolation or downsampling. For optimization purposes, if only a subregion of the image is of interest, demosaicing can be run on that sub-region only.
The state of the software ISP pipeline controls the demosaic output. Use the following methods to control the pipeline state:
- Note
- The Bayer image is always the full resolution and not affected by the demosaic state.
Currently, the pipeline supports RAW images from the following types of cameras.
Functionalities Available for the Above cameras
Procss Type | Bayer | Output-Bound Demosaic | Tonemap | Result |
bayer | bound | any | any | raw->bayer |
bayer+demosaic | bound | bound | any | raw->bayer&demosaic |
demosaic | unbound | bound | any | raw->demosaic |
demosaic | bound | bound | any | bayer->demosaic (!) |
demosaic+tonemap | bound | bound | bound | raw->demosaic->tonemap |
demosaic+tonemap | unbound | bound | bound | bayer->demosaic->tonemap |
bayer+demosaic+tonemap | bound | bound | bound | raw->bayer+demosaic->tonemap |
bayer+tonemap | bound | any | bound | FAIL |
tonemap | any | unbound | bound | FAIL |
tonemap | any | bound | bound | demosaic->tonemap (!) |
- Note
- If the starting image in RESULT is not the raw input, then the algorithm will assume that the bound starting image has been used previously and contains a valid image.
Examples
Imput Parameters Based on Underlying Operation
The following table shows the input parameters allowed based on the underlying operation that the softISP executes.
Operation type | Demosaic ROI | Demosaic Method | Input | Ouput 1 | Output 2 | Comment |
raw->demosaic | Full Image (WxH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | RCB, W/2xH/2, fp16 | n.a. | Optimized path (Fused Kernel) |
raw->demosaic | Subregion (CWxCH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | RCB, CW/2xCH/2, fp16 | n.a. | - |
raw->demosaic | Full Image (WxH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | RCB, WxH, fp16 | n.a. | - |
raw->demosaic | Subregion (CWxCH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | RCB, CWxCH, fp16 | n.a. | - |
raw->bayer | Full Image (WxH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | CRCB, WxH, fp16 | n.a. | - |
raw->bayer | Subregion (CWxCH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | CRCB, WxH, fp16 | n.a. | - |
raw->bayer | Full Image (WxH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | CRCB, WxH, fp16 | n.a. | - |
raw->bayer | Subregion (CWxCH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | CRCB, WxH, fp16 | n.a. | - |
bayer->demosaic | Full Image (WxH) | DEMOSAIC_DOWNSAMPLE | CRCB, WxH, fp16 | RCB, W/2xH/2, fp16 | n.a. | - |
bayer->demosaic | Subregion (CWxCH) | DEMOSAIC_DOWNSAMPLE | CRCB, WxH, fp16 | RCB, CW/2xCH/2, fp16 | n.a. | - |
bayer->demosaic | Full Image (WxH) | DEMOSAIC_INTERPOLATION | CRCB, WxH, fp16 | RCB, WxH, fp16 | n.a. | - |
bayer->demosaic | Subregion (CWxCH) | DEMOSAIC_INTERPOLATION | CRCB, WxH, fp16 | RCB, CWxCH, fp16 | n.a. | - |
raw->bayer+demosaic | Full Image (WxH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | CRCB, WxH, fp16 | RCB, W/2xH/2, fp16 | Optimized path (Fused Kernel) |
raw->bayer+demosaic | Subregion (CWxCH) | DEMOSAIC_DOWNSAMPLE | RAW, WxH, uint16 | CRCB, WxH, fp16 | RCB, CW/2xCH/2, fp16 | - |
raw->bayer+demosaic | Full Image (WxH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | CRCB, WxH, fp16 | RCB, WxH, fp16 | - |
raw->bayer+demosaic | Subregion (CWxCH) | DEMOSAIC_INTERPOLATION | RAW, WxH, uint16 | CRCB, WxH, fp16 | RCB, CWxCH, fp16 | - |
|
DW_API_PUBLIC dwStatus | dwSoftISP_bindBayerOutput (dwImageCUDA *bayerImage, dwSoftISPHandle_t obj) |
| Binds a Bayer image to the softISP. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_bindDemosaicOutput (dwImageCUDA *demosaicImage, dwSoftISPHandle_t obj) |
| Binds a demosaic image to the softISP. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_bindRawInput (const dwImageCUDA *rawImage, dwSoftISPHandle_t obj) |
| Binds the input image to the softISP. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_bindTonemapOutput (dwImageCUDA *tonemappedImage, dwSoftISPHandle_t obj) |
| Binds a tone-mapped image to the softISP. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_copyTonemapCUDA (dwImageCUDA *output, const dwImageCUDA *input, dwSoftISPHandle_t obj, cudaStream_t stream) |
| Perform tone-mapping on demosaiced images (DEPRECATED). More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getBayerImageProperties (dwImageProperties *imageProperties, dwSoftISPHandle_t obj) |
| Gets the image properties for the produced Bayer images. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getCUDAStream (cudaStream_t *stream, dwSoftISPHandle_t obj) |
| Gets the CUDA stream for the softISP pipeline. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getDemosaicImageProperties (dwImageProperties *imageProperties, dwSoftISPHandle_t obj) |
| Gets the image properties for the produced demosaiced images. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getDemosaicMethod (dwSoftISPDemosaicMethod *method, dwSoftISPHandle_t obj) |
| Gets the method used for demosaic. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getDemosaicROI (dwRect *cropRegion, dwSoftISPHandle_t obj) |
| Gets the cropping region in the Bayer image used for demosaicing. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getDenoiseMethod (dwSoftISPDenoiseMethod *method, dwSoftISPHandle_t obj) |
| Gets the method used for denoising. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_getTonemapType (dwTonemapType *type, dwSoftISPHandle_t obj) |
| Gets the tonemap type. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_initialize (dwSoftISPHandle_t *obj, const dwSoftISPParams params, const dwContextHandle_t ctx) |
| Creates and initializes the software ISP pipeline. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_initParamsFromCamera (dwSoftISPParams *params, const dwCameraProperties cameraProps) |
| Initializes the parameters for the softISP using dwCameraProperties. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_processDeviceAsync (const int32_t type, dwSoftISPHandle_t obj) |
| Runs the Software ISP to convert the bound raw image into whatever output was bound beforehand. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_release (dwSoftISPHandle_t *obj) |
| Releases the softISP pipeline object. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_reset (dwSoftISPHandle_t obj) |
| Resets the softISP pipeline. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_setCUDAStream (cudaStream_t stream, dwSoftISPHandle_t obj) |
| Sets the CUDA stream for the softISP pipeline. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_setDemosaicMethod (dwSoftISPDemosaicMethod method, dwSoftISPHandle_t obj) |
| Sets the method used to demosaic. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_setDemosaicROI (dwRect cropRegion, dwSoftISPHandle_t obj) |
| Sets the cropping region in the Bayer image used for demosaicing. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_setDenoiseMethod (dwSoftISPDenoiseMethod method, dwSoftISPHandle_t obj) |
| Sets the method used to denoise. More...
|
|
DW_API_PUBLIC dwStatus | dwSoftISP_setTonemapType (dwTonemapType type, dwSoftISPHandle_t obj) |
| Sets the tonemap type. More...
|
|
◆ dwSoftISPParams
Data Fields |
uint32_t |
height |
|
dwCameraRawFormat |
rawFormat |
Raw pixel order. |
dwTonemapType |
type |
Tonemapper type (unknown for no tonemapping) |
uint32_t |
width |
Size of the input image (usually the camera resolution) |
◆ dwSoftISPHandle_t
Handle representing the Software ISP.
Definition at line 150 of file SoftISP.h.
◆ dwSoftISPDemosaicMethod
Enumerator |
---|
DW_SOFT_ISP_DEMOSAIC_METHOD_INTERPOLATION | Full camera resolution using interpolation.
|
DW_SOFT_ISP_DEMOSAIC_METHOD_DOWNSAMPLE | Half camera resolution combining a 2x2 Bayer cell into one output pixel.
|
Definition at line 152 of file SoftISP.h.
◆ dwSoftISPDenoiseMethod
Enumerator |
---|
DW_SOFT_ISP_DENOISE_METHOD_NONE | No denoising.
|
DW_SOFT_ISP_DENOISE_METHOD_BILATERAL | Bilateral denoising.
|
Definition at line 159 of file SoftISP.h.
◆ dwSoftISPProcessType
Enumerator |
---|
DW_SOFT_ISP_PROCESS_TYPE_BAYER | Compute Bayer.
|
DW_SOFT_ISP_PROCESS_TYPE_DEMOSAIC | Compute demosaic.
|
DW_SOFT_ISP_PROCESS_TYPE_TONEMAP | Compute tone map.
|
Definition at line 166 of file SoftISP.h.
◆ dwSoftISP_bindBayerOutput()
Binds a Bayer image to the softISP.
The caller must allocate the image, using the properties returned by dwSoftISP_getBayerImageProperties()
- Parameters
-
[in] | bayerImage | A pointer to the output Bayer image. Passing nullptr will unbound. |
[in] | obj | A pointer to the Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE
◆ dwSoftISP_bindDemosaicOutput()
Binds a demosaic image to the softISP.
The caller must allocate the image, using the properties returned by dwSoftISP_getDemosaicImageProperties().
- Parameters
-
[in] | demosaicImage | A pointer to the output demosaic image. Passing nullptr will unbound. |
[in] | obj | A pointer to the Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE
◆ dwSoftISP_bindRawInput()
Binds the input image to the softISP.
It will be the source used by processDeviceAsync()
.
- Parameters
-
[in] | rawImage | A pointer to the raw image. Passing nullptr will unbound. |
[in] | obj | A pointer to the Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE
◆ dwSoftISP_bindTonemapOutput()
Binds a tone-mapped image to the softISP.
The caller must allocate the image. The properties of the tone-mapped image can be:
- pxlFormat: DW_IMAGE_RGB or DW_IMAGE_RGBA
- pxlType: DW_TYPE_UINT8 or DW_TYPE_UINT16 The rest of the properties match the demosaic image.
- Note
- To have a tone-mapped image, the caller must bind a tone-map image as well. If the tone-mapped image is not bound,
processDeviceAsync()
will skip tone-mapping.
- Parameters
-
[in] | tonemappedImage | A pointer to the output tone-mapped image. Passing nullptr will unbound. |
[in] | obj | A pointer to the Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE
◆ dwSoftISP_copyTonemapCUDA()
Perform tone-mapping on demosaiced images (DEPRECATED).
- Deprecated:
- Parameters
-
[out] | output | Output image allocated by the caller, which must contain RGB/RGBA. |
[in] | input | Input image demosaiced as RCB/RCC. |
[in] | obj | Specifies the RAW camera pipeline handle. |
[in] | stream | Current CUDA stream that is used by the module. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getBayerImageProperties()
Gets the image properties for the produced Bayer images.
- Parameters
-
[out] | imageProperties | A pointer to the image properties for Bayer output. |
[in] | obj | The Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getCUDAStream()
Gets the CUDA stream for the softISP pipeline.
- Parameters
-
[out] | stream | Current CUDA stream that is used by the module. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getDemosaicImageProperties()
Gets the image properties for the produced demosaiced images.
The output size depends on the ROI set via dwSoftISP_setDemosaicROI.
- Parameters
-
[out] | imageProperties | A pointer to the image properties for demosaic output. |
[in] | obj | The Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getDemosaicMethod()
Gets the method used for demosaic.
- Parameters
-
[out] | method | Method used to demosaic. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getDemosaicROI()
Gets the cropping region in the Bayer image used for demosaicing.
- Parameters
-
[out] | cropRegion | Crop region used in the Bayer image. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getDenoiseMethod()
Gets the method used for denoising.
- Parameters
-
[out] | method | Method used to denoise. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_getTonemapType()
Gets the tonemap type.
- Parameters
-
[out] | type | The tonemap type currently used. DW_TONEMAP_UNKNOWN if no tonemap is in use |
[in] | obj | The Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_initialize()
Creates and initializes the software ISP pipeline.
- Parameters
-
[out] | obj | A pointer to the created softISP pipeline handle. |
[in] | params | The parameters of the softISP. |
[in] | ctx | Driveworks context handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST, DW_INVALID_ARGUMENT
◆ dwSoftISP_initParamsFromCamera()
Initializes the parameters for the softISP using dwCameraProperties.
- Parameters
-
[out] | params | A pointer to the initialized parameters. |
[in] | cameraProps | Properties of the camera. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST, DW_INVALID_ARGUMENT
◆ dwSoftISP_processDeviceAsync()
Runs the Software ISP to convert the bound raw image into whatever output was bound beforehand.
The command will process as explained in the table at the beginning of the page (DW_SOFT_ISP_ is omitted).
- Parameters
-
[in] | type | A bitwise mask of the process output chosen (must match or be a subset of the bound images). |
[in] | obj | A pointer to the Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE
◆ dwSoftISP_release()
Releases the softISP pipeline object.
This method releases all resources associated with the RAW camera processing pipeline. Such resources include NVIDIA® CUDA® memory.
- Note
- This method renders the handle unusable.
- Parameters
-
[in] | obj | A pointer to the object handle to release. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_reset()
Resets the softISP pipeline.
(Reserved for future use.)
- Parameters
-
[in] | obj | Specifies the softISP pipeline handle to reset. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_setCUDAStream()
Sets the CUDA stream for the softISP pipeline.
- Parameters
-
[in] | stream | Specifies the CUDA stream. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_setDemosaicMethod()
Sets the method used to demosaic.
Depending on the camera, not all methods may be implemented.
- Parameters
-
[in] | method | Specifies the demosaic method. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_setDemosaicROI()
Sets the cropping region in the Bayer image used for demosaicing.
- Parameters
-
[in] | cropRegion | Specifies the crop region in the Bayer image. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_setDenoiseMethod()
Sets the method used to denoise.
Depending on the camera, not all methods may be implemented.
- Parameters
-
[in] | method | Specifies the denoise method. |
[in] | obj | Specifies the softISP pipeline handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST
◆ dwSoftISP_setTonemapType()
Sets the tonemap type.
All resources are allocated by the softISP at initialization, this call only chooses the current one in use
- Parameters
-
[in] | type | The tonemap type currently used. DW_TONEMAP_UNKNOWN if no tonemap is in use |
[in] | obj | The Software ISP handle. |
- Returns
- DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST