DriveWorks SDK Reference

| 0.6.67 Release

Software ISP

Detailed Description

The Software Image Signal Processor (ISP) converts RAW images from a specific camera and converts them into images with a linear response curve.

The output format can be either a Bayer pattern image, a demosaiced image, or both. Bayer image means a linear response image that has the same position varying color filter as the camera physical sensor. The demosiced image is the final result where each pixel has all color channels available, either via interpolation or downsampling. For optimization purposes, if only a subregion of the image is of interest, demosaicing can be run on that sub-region only.

The state of the software ISP pipeline controls the demosaic output. Use the following methods to control the pipeline state:

Note
The Bayer image is always the full resolution and not affected by the demosaic state.

Currently, the pipeline supports RAW images from the following types of cameras.

Functionalities Available for the Above cameras

Procss TypeBayerOutput-Bound DemosaicTonemapResult
bayer bound any any raw->bayer
bayer+demosaic bound bound any raw->bayer&demosaic
demosaic unbound bound any raw->demosaic
demosaic bound bound any bayer->demosaic (!)
demosaic+tonemap bound bound bound raw->demosaic->tonemap
demosaic+tonemap unbound bound bound bayer->demosaic->tonemap
bayer+demosaic+tonemap bound bound bound raw->bayer+demosaic->tonemap
bayer+tonemap bound any bound FAIL
tonemap any unbound bound FAIL
tonemap any bound bound demosaic->tonemap (!)
Note
If the starting image in RESULT is not the raw input, then the algorithm will assume that the bound starting image has been used previously and contains a valid image.

Examples

// Bind the input image.
dwSoftISP_bindRawInput(input, softISP);
// Set the ROI to full image.
dwSoftISP_setDemosaicROI(fullRegion, softISP);
// Bind the full region output demosaic image.
dwSoftISP_bindDemosaicOutput(fullDemosaicOutput, softISP);
// Bind the Bayer output.
dwSoftISP_bindBayerOutput(bayerOutput, softISP);
// Process (raw->bayer&demosaic).
result = dwSoftISP_processDeviceAsync(DW_SOFT_ISP_PROCESS_BAYER | DW_SOFT_ISP_PROCESS_DEMOSAIC, softISP);
// Set the ROI to a crop region.
dwSoftISP_setDemosaicROI(cropRegion, softISP);
// Bind a different image for cropped demosaic.
dwSoftISP_bindDemosaicOutput(croppedDemosaicOutput, softISP);
// Process (bayer->demosaic) will reuse the previously bound Bayer, more efficient.
result = dwSoftISP_processDeviceAsync(DW_SOFT_ISP_PROCESS_DEMOSAIC, softISP);

Imput Parameters Based on Underlying Operation

The following table shows the input parameters allowed based on the underlying operation that the softISP executes.

Operation typeDemosaic ROIDemosaic MethodInputOuput 1Output 2Comment
raw->demosaic Full Image (WxH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16RCB, W/2xH/2, fp16n.a.Optimized path (Fused Kernel)
raw->demosaic Subregion (CWxCH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16RCB, CW/2xCH/2, fp16n.a.-
raw->demosaic Full Image (WxH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16RCB, WxH, fp16n.a.-
raw->demosaic Subregion (CWxCH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16RCB, CWxCH, fp16n.a.-
raw->bayer Full Image (WxH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16CRCB, WxH, fp16n.a.-
raw->bayer Subregion (CWxCH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16CRCB, WxH, fp16n.a.-
raw->bayer Full Image (WxH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16CRCB, WxH, fp16n.a.-
raw->bayer Subregion (CWxCH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16CRCB, WxH, fp16n.a.-
bayer->demosaic Full Image (WxH)DEMOSAIC_DOWNSAMPLECRCB, WxH, fp16RCB, W/2xH/2, fp16n.a.-
bayer->demosaic Subregion (CWxCH)DEMOSAIC_DOWNSAMPLECRCB, WxH, fp16RCB, CW/2xCH/2, fp16n.a.-
bayer->demosaic Full Image (WxH)DEMOSAIC_INTERPOLATIONCRCB, WxH, fp16RCB, WxH, fp16n.a.-
bayer->demosaic Subregion (CWxCH)DEMOSAIC_INTERPOLATIONCRCB, WxH, fp16RCB, CWxCH, fp16n.a.-
raw->bayer+demosaic Full Image (WxH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16CRCB, WxH, fp16RCB, W/2xH/2, fp16Optimized path (Fused Kernel)
raw->bayer+demosaic Subregion (CWxCH)DEMOSAIC_DOWNSAMPLERAW, WxH, uint16CRCB, WxH, fp16RCB, CW/2xCH/2, fp16-
raw->bayer+demosaic Full Image (WxH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16CRCB, WxH, fp16RCB, WxH, fp16-
raw->bayer+demosaic Subregion (CWxCH)DEMOSAIC_INTERPOLATIONRAW, WxH, uint16CRCB, WxH, fp16RCB, CWxCH, fp16-

Data Structures

struct  dwSoftISPParams
 

Typedefs

typedef struct dwSoftISPObject * dwSoftISPHandle_t
 Handle representing the Software ISP. More...
 

Enumerations

enum  dwSoftISPDemosaicMethod {
  DW_SOFT_ISP_DEMOSAIC_METHOD_INTERPOLATION,
  DW_SOFT_ISP_DEMOSAIC_METHOD_DOWNSAMPLE
}
 
enum  dwSoftISPDenoiseMethod {
  DW_SOFT_ISP_DENOISE_METHOD_NONE,
  DW_SOFT_ISP_DENOISE_METHOD_BILATERAL
}
 
enum  dwSoftISPProcessType {
  DW_SOFT_ISP_PROCESS_TYPE_BAYER = 1 << 0,
  DW_SOFT_ISP_PROCESS_TYPE_DEMOSAIC = 1 << 1,
  DW_SOFT_ISP_PROCESS_TYPE_TONEMAP = 1 << 2
}
 

Functions

DW_API_PUBLIC dwStatus dwSoftISP_bindBayerOutput (dwImageCUDA *bayerImage, dwSoftISPHandle_t obj)
 Binds a Bayer image to the softISP. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_bindDemosaicOutput (dwImageCUDA *demosaicImage, dwSoftISPHandle_t obj)
 Binds a demosaic image to the softISP. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_bindRawInput (const dwImageCUDA *rawImage, dwSoftISPHandle_t obj)
 Binds the input image to the softISP. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_bindTonemapOutput (dwImageCUDA *tonemappedImage, dwSoftISPHandle_t obj)
 Binds a tone-mapped image to the softISP. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_copyTonemapCUDA (dwImageCUDA *output, const dwImageCUDA *input, dwSoftISPHandle_t obj, cudaStream_t stream)
 Perform tone-mapping on demosaiced images (DEPRECATED). More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getBayerImageProperties (dwImageProperties *imageProperties, dwSoftISPHandle_t obj)
 Gets the image properties for the produced Bayer images. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getCUDAStream (cudaStream_t *stream, dwSoftISPHandle_t obj)
 Gets the CUDA stream for the softISP pipeline. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicImageProperties (dwImageProperties *imageProperties, dwSoftISPHandle_t obj)
 Gets the image properties for the produced demosaiced images. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicMethod (dwSoftISPDemosaicMethod *method, dwSoftISPHandle_t obj)
 Gets the method used for demosaic. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicROI (dwRect *cropRegion, dwSoftISPHandle_t obj)
 Gets the cropping region in the Bayer image used for demosaicing. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getDenoiseMethod (dwSoftISPDenoiseMethod *method, dwSoftISPHandle_t obj)
 Gets the method used for denoising. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_getTonemapType (dwTonemapType *type, dwSoftISPHandle_t obj)
 Gets the tonemap type. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_initialize (dwSoftISPHandle_t *obj, const dwSoftISPParams params, const dwContextHandle_t ctx)
 Creates and initializes the software ISP pipeline. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_initParamsFromCamera (dwSoftISPParams *params, const dwCameraProperties cameraProps)
 Initializes the parameters for the softISP using dwCameraProperties. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_processDeviceAsync (const int32_t type, dwSoftISPHandle_t obj)
 Runs the Software ISP to convert the bound raw image into whatever output was bound beforehand. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_release (dwSoftISPHandle_t *obj)
 Releases the softISP pipeline object. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_reset (dwSoftISPHandle_t obj)
 Resets the softISP pipeline. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_setCUDAStream (cudaStream_t stream, dwSoftISPHandle_t obj)
 Sets the CUDA stream for the softISP pipeline. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_setDemosaicMethod (dwSoftISPDemosaicMethod method, dwSoftISPHandle_t obj)
 Sets the method used to demosaic. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_setDemosaicROI (dwRect cropRegion, dwSoftISPHandle_t obj)
 Sets the cropping region in the Bayer image used for demosaicing. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_setDenoiseMethod (dwSoftISPDenoiseMethod method, dwSoftISPHandle_t obj)
 Sets the method used to denoise. More...
 
DW_API_PUBLIC dwStatus dwSoftISP_setTonemapType (dwTonemapType type, dwSoftISPHandle_t obj)
 Sets the tonemap type. More...
 

Data Structure Documentation

◆ dwSoftISPParams

struct dwSoftISPParams
Data Fields
uint32_t height
dwCameraRawFormat rawFormat Raw pixel order.
dwTonemapType type Tonemapper type (unknown for no tonemapping)
uint32_t width Size of the input image (usually the camera resolution)

Typedef Documentation

◆ dwSoftISPHandle_t

typedef struct dwSoftISPObject* dwSoftISPHandle_t

Handle representing the Software ISP.

Definition at line 150 of file SoftISP.h.

Enumeration Type Documentation

◆ dwSoftISPDemosaicMethod

Enumerator
DW_SOFT_ISP_DEMOSAIC_METHOD_INTERPOLATION 

Full camera resolution using interpolation.

DW_SOFT_ISP_DEMOSAIC_METHOD_DOWNSAMPLE 

Half camera resolution combining a 2x2 Bayer cell into one output pixel.

Definition at line 152 of file SoftISP.h.

◆ dwSoftISPDenoiseMethod

Enumerator
DW_SOFT_ISP_DENOISE_METHOD_NONE 

No denoising.

DW_SOFT_ISP_DENOISE_METHOD_BILATERAL 

Bilateral denoising.

Definition at line 159 of file SoftISP.h.

◆ dwSoftISPProcessType

Enumerator
DW_SOFT_ISP_PROCESS_TYPE_BAYER 

Compute Bayer.

DW_SOFT_ISP_PROCESS_TYPE_DEMOSAIC 

Compute demosaic.

DW_SOFT_ISP_PROCESS_TYPE_TONEMAP 

Compute tone map.

Definition at line 166 of file SoftISP.h.

Function Documentation

◆ dwSoftISP_bindBayerOutput()

DW_API_PUBLIC dwStatus dwSoftISP_bindBayerOutput ( dwImageCUDA bayerImage,
dwSoftISPHandle_t  obj 
)

Binds a Bayer image to the softISP.

The caller must allocate the image, using the properties returned by dwSoftISP_getBayerImageProperties()

Parameters
[in]bayerImageA pointer to the output Bayer image. Passing nullptr will unbound.
[in]objA pointer to the Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE

◆ dwSoftISP_bindDemosaicOutput()

DW_API_PUBLIC dwStatus dwSoftISP_bindDemosaicOutput ( dwImageCUDA demosaicImage,
dwSoftISPHandle_t  obj 
)

Binds a demosaic image to the softISP.

The caller must allocate the image, using the properties returned by dwSoftISP_getDemosaicImageProperties().

Parameters
[in]demosaicImageA pointer to the output demosaic image. Passing nullptr will unbound.
[in]objA pointer to the Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE

◆ dwSoftISP_bindRawInput()

DW_API_PUBLIC dwStatus dwSoftISP_bindRawInput ( const dwImageCUDA rawImage,
dwSoftISPHandle_t  obj 
)

Binds the input image to the softISP.

It will be the source used by processDeviceAsync().

Parameters
[in]rawImageA pointer to the raw image. Passing nullptr will unbound.
[in]objA pointer to the Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE

◆ dwSoftISP_bindTonemapOutput()

DW_API_PUBLIC dwStatus dwSoftISP_bindTonemapOutput ( dwImageCUDA tonemappedImage,
dwSoftISPHandle_t  obj 
)

Binds a tone-mapped image to the softISP.

The caller must allocate the image. The properties of the tone-mapped image can be:

  • pxlFormat: DW_IMAGE_RGB or DW_IMAGE_RGBA
  • pxlType: DW_TYPE_UINT8 or DW_TYPE_UINT16 The rest of the properties match the demosaic image.
    Note
    To have a tone-mapped image, the caller must bind a tone-map image as well. If the tone-mapped image is not bound, processDeviceAsync() will skip tone-mapping.
    Parameters
    [in]tonemappedImageA pointer to the output tone-mapped image. Passing nullptr will unbound.
    [in]objA pointer to the Software ISP handle.
    Returns
    DW_SUCCESS, DW_INVALID_HANDLE

◆ dwSoftISP_copyTonemapCUDA()

DW_API_PUBLIC dwStatus dwSoftISP_copyTonemapCUDA ( dwImageCUDA output,
const dwImageCUDA input,
dwSoftISPHandle_t  obj,
cudaStream_t  stream 
)

Perform tone-mapping on demosaiced images (DEPRECATED).

Deprecated:
Parameters
[out]outputOutput image allocated by the caller, which must contain RGB/RGBA.
[in]inputInput image demosaiced as RCB/RCC.
[in]objSpecifies the RAW camera pipeline handle.
[in]streamCurrent CUDA stream that is used by the module.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getBayerImageProperties()

DW_API_PUBLIC dwStatus dwSoftISP_getBayerImageProperties ( dwImageProperties imageProperties,
dwSoftISPHandle_t  obj 
)

Gets the image properties for the produced Bayer images.

Parameters
[out]imagePropertiesA pointer to the image properties for Bayer output.
[in]objThe Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getCUDAStream()

DW_API_PUBLIC dwStatus dwSoftISP_getCUDAStream ( cudaStream_t *  stream,
dwSoftISPHandle_t  obj 
)

Gets the CUDA stream for the softISP pipeline.

Parameters
[out]streamCurrent CUDA stream that is used by the module.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getDemosaicImageProperties()

DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicImageProperties ( dwImageProperties imageProperties,
dwSoftISPHandle_t  obj 
)

Gets the image properties for the produced demosaiced images.

The output size depends on the ROI set via dwSoftISP_setDemosaicROI.

Parameters
[out]imagePropertiesA pointer to the image properties for demosaic output.
[in]objThe Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getDemosaicMethod()

DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicMethod ( dwSoftISPDemosaicMethod method,
dwSoftISPHandle_t  obj 
)

Gets the method used for demosaic.

Parameters
[out]methodMethod used to demosaic.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getDemosaicROI()

DW_API_PUBLIC dwStatus dwSoftISP_getDemosaicROI ( dwRect cropRegion,
dwSoftISPHandle_t  obj 
)

Gets the cropping region in the Bayer image used for demosaicing.

Parameters
[out]cropRegionCrop region used in the Bayer image.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getDenoiseMethod()

DW_API_PUBLIC dwStatus dwSoftISP_getDenoiseMethod ( dwSoftISPDenoiseMethod method,
dwSoftISPHandle_t  obj 
)

Gets the method used for denoising.

Parameters
[out]methodMethod used to denoise.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_getTonemapType()

DW_API_PUBLIC dwStatus dwSoftISP_getTonemapType ( dwTonemapType type,
dwSoftISPHandle_t  obj 
)

Gets the tonemap type.

Parameters
[out]typeThe tonemap type currently used. DW_TONEMAP_UNKNOWN if no tonemap is in use
[in]objThe Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_initialize()

DW_API_PUBLIC dwStatus dwSoftISP_initialize ( dwSoftISPHandle_t obj,
const dwSoftISPParams  params,
const dwContextHandle_t  ctx 
)

Creates and initializes the software ISP pipeline.

Parameters
[out]objA pointer to the created softISP pipeline handle.
[in]paramsThe parameters of the softISP.
[in]ctxDriveworks context handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST, DW_INVALID_ARGUMENT

◆ dwSoftISP_initParamsFromCamera()

DW_API_PUBLIC dwStatus dwSoftISP_initParamsFromCamera ( dwSoftISPParams params,
const dwCameraProperties  cameraProps 
)

Initializes the parameters for the softISP using dwCameraProperties.

Parameters
[out]paramsA pointer to the initialized parameters.
[in]cameraPropsProperties of the camera.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST, DW_INVALID_ARGUMENT

◆ dwSoftISP_processDeviceAsync()

DW_API_PUBLIC dwStatus dwSoftISP_processDeviceAsync ( const int32_t  type,
dwSoftISPHandle_t  obj 
)

Runs the Software ISP to convert the bound raw image into whatever output was bound beforehand.

The command will process as explained in the table at the beginning of the page (DW_SOFT_ISP_ is omitted).

Parameters
[in]typeA bitwise mask of the process output chosen (must match or be a subset of the bound images).
[in]objA pointer to the Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE

◆ dwSoftISP_release()

DW_API_PUBLIC dwStatus dwSoftISP_release ( dwSoftISPHandle_t obj)

Releases the softISP pipeline object.

This method releases all resources associated with the RAW camera processing pipeline. Such resources include NVIDIA® CUDA® memory.

Note
This method renders the handle unusable.
Parameters
[in]objA pointer to the object handle to release.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_reset()

DW_API_PUBLIC dwStatus dwSoftISP_reset ( dwSoftISPHandle_t  obj)

Resets the softISP pipeline.

(Reserved for future use.)

Parameters
[in]objSpecifies the softISP pipeline handle to reset.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_setCUDAStream()

DW_API_PUBLIC dwStatus dwSoftISP_setCUDAStream ( cudaStream_t  stream,
dwSoftISPHandle_t  obj 
)

Sets the CUDA stream for the softISP pipeline.

Parameters
[in]streamSpecifies the CUDA stream.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_setDemosaicMethod()

DW_API_PUBLIC dwStatus dwSoftISP_setDemosaicMethod ( dwSoftISPDemosaicMethod  method,
dwSoftISPHandle_t  obj 
)

Sets the method used to demosaic.

Depending on the camera, not all methods may be implemented.

Parameters
[in]methodSpecifies the demosaic method.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_setDemosaicROI()

DW_API_PUBLIC dwStatus dwSoftISP_setDemosaicROI ( dwRect  cropRegion,
dwSoftISPHandle_t  obj 
)

Sets the cropping region in the Bayer image used for demosaicing.

Parameters
[in]cropRegionSpecifies the crop region in the Bayer image.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_setDenoiseMethod()

DW_API_PUBLIC dwStatus dwSoftISP_setDenoiseMethod ( dwSoftISPDenoiseMethod  method,
dwSoftISPHandle_t  obj 
)

Sets the method used to denoise.

Depending on the camera, not all methods may be implemented.

Parameters
[in]methodSpecifies the denoise method.
[in]objSpecifies the softISP pipeline handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST

◆ dwSoftISP_setTonemapType()

DW_API_PUBLIC dwStatus dwSoftISP_setTonemapType ( dwTonemapType  type,
dwSoftISPHandle_t  obj 
)

Sets the tonemap type.

All resources are allocated by the softISP at initialization, this call only chooses the current one in use

Parameters
[in]typeThe tonemap type currently used. DW_TONEMAP_UNKNOWN if no tonemap is in use
[in]objThe Software ISP handle.
Returns
DW_SUCCESS, DW_INVALID_HANDLE, DW_BAD_CAST