SIPL Guidance on Output Image Formats#

SIPL supports multiple image formats using the ICP and ISP outputs. This topic describes how the output formats are set and which parameters you can modify to override output formats under specific conditions.

ICP Output Formats#

ICP output formats are dependent on the sensor used.

Based on the vendor and sensor module specifics, the output format of ICP is determined using the CameraModuleInfo::SensorInfo::VirtualChannelInfo::cfa and inputFormat fields in the CameraConfig struct provided as input to the INvSIPLCamera::SetPlatformCfg API.

Refer to NvSIPLCapStructs.h for the following:

  • Supported values for input format that correspond to NvSiplCapInputFormatType enum.

  • Supported values for cfa that correspond to the valid NVSIPL_PIXEL_ORDER flags.

SIPL does not add restrictions on the use of various input format type/pixel orders because these are specific to sensors. SIPL does not have a way of determining the expectations of each sensor. We recommend that you adhere to the specifics of the sensors.

However, SIPL does impose restrictions such as rejecting bad combinations of input format and pixel order. Some examples of the invalid combinations are as follows:

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_YUV422 and NVSIPL_PIXEL_ORDER_RGBA

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_RAW6 and NVSIPL_PIXEL_ORDER_RGBA

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_RGB888 and NVSIPL_PIXEL_ORDER_RGGB

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_YUV422 and NVSIPL_PIXEL_ORDER_RGGB

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_RAW6 and NVSIPL_PIXEL_ORDER_YUV

  • NVSIPL_CAP_INPUT_FORMAT_TYPE_RGB888 and NVSIPL_PIXEL_ORDER_YUV

ISP Output Formats#

Refer to the table provided as part of NvSIPLCamera.hpp for documentation on the INvSIPLCamera::RegisterImages API for the various ISP output formats supported by SIPL.

Call INvSIPLCamera::GetImageAttributes before calling INvSIPLCamera::RegisterImages to validate the attributes that you provided. The sequence of API calls is as follows:

  1. Application calls NvSciBufAttrListCreate to create an unreconciled buffer attribute list.

  2. Application calls INvSIPLCamera::GetImageAttrs(NvSciBufAttrs), which checks and validates the buffer attributes in case you chose to override the default format. Otherwise, the application adds attributes for the default formats.

    Note

    Because of ABI compliance requirements, NVIDIA maintains the name of the API. However, a name such as VerifyandSetImageAttributes is more descriptive of the functionality.

  3. Application calls NvSciBufAttrReconcile to reconcile all buffer attributes across buffer attribute lists from other consumers for this buffer, which could be any downstream engine, such as VIC, IEP, and CUDA.

  4. Application calls NvSciBufObjAlloc on the reconciled attribute list to create a buffer object.

  5. Application calls INvSIPLCamera::RegisterImages(NvSciBufObj), which accepts the buffer only if one of the attribute lists used to reconcile and allocate was verified by SIPL using INvSIPLCamera::GetImageAttrs.

Note

  • ISP output images will be a binary match if the default NITO provided by NVIDIA is used while requesting the same output formats with the same resolution.

  • The ISP0 and ISP1 outputs can have different formats as long as they both are YUV type.

  • The camera tuning does not need to be re-run for different formats because formats are simply different ways to represent images in the memory. However, the data itself will differ because the representation of the memory varies across formats.

Override Image Attributes#

ISP supports the following three types of formats:

  • YUV semiplanar images.

  • YUV packed images. (Luma is a type of YUV packed image.)

  • RGBA packed images.

Creating packed type buffers is straightforward: You use NvSciBufImageAttrKey_Plane*** type attributes to override image formats.

For semiplanar type buffers, however, NvSciBuf provides attribute keys to create buffers of the preceding types by using two approaches:

  • Using NvSciBufImageAttrKey_Plane*** type attributes.

  • Using NvSciBufImageAttrKey_Surf*** type attributes.

NvSciBuf added the Surf type attributes to make the task of creating multi-plane image buffers easy for the user. However, in Plane type attributes you must provide the height and width of each plane to determine the memory layout of the image.

In SIPL, the library oversees calculations for height and width of the buffer, factoring in multiple criteria such as sensor output resolution, input crop, and downscaling. This process is complicated if you supply Plane type attributes to SIPL while trying to create multi-plane images. Hence, SIPL restricts you to Surf type attributes when using multi-plane YUV images.

Note

The RGBA FP16 format from ISP2 output is special semi-raw data, which is tapped out right after demosaic in the ISP. This output data does not have standard colors and brightness. It is not suitable for human vision use cases.

Examples#

For a YUV 444 SEMI-PLANAR UINT 16 BLOCK LINEAR image, the following attributes should be set to the corresponding values:

Attribute Name

Attribute Value

NvSciBufGeneralAttrKey_Types

NvSciBufType_Image

NvSciBufImageAttrKey_SurfType

NvSciSurfType_YUV

NvSciBufImageAttrKey_SurfBPC

NvSciSurfBPC_16

NvSciBufImageAttrKey_SurfMemLayout

NvSciSurfMemLayout_SemiPlanar

NvSciBufImageAttrKey_SurfSampleType

NvSciSurfSampleType_444

NvSciBufImageAttrKey_SurfComponentOrder

NvSciSurfComponentOrder_YUV

NvSciBufImageAttrKey_SurfColorStd

NvSciColorStd_REC709_ER

NvSciBufImageAttrKey_Layout

NvSciBufImage_BlockLinearType

For a YUV 444 PACKED UINT 8 BLOCK LINEAR image, the following attributes should be set to the corresponding values

Attribute Name

Attribute Value

NvSciBufGeneralAttrKey_Types

NvSciBufType_Image

NvSciBufImageAttrKey_PlaneCount

1

NvSciBufImageAttrKey_Layout

NvSciBufImage_BlockLinearType

NvSciBufImageAttrKey_PlaneColorFormat

NvSciColor_A8Y8U8V8

NvSciBufImageAttrKey_PlaneColorStd

NvSciColorStd_REC709_ER