Image OpticalFlow/StereoDisparity (OFST) Estimator

Detailed Description

The NvMediaIOFST object takes an uncompressed image frame pair and turns them into opticalflow/stereodisparity estimation data.

Data Structures
struct	NvMediaIOFST
	Holds an OFST object created and returned by NvMediaIOFSTCreate(). More...
struct	NvMediaOFSTInitializeParams
	Holds OFST estimation initialization parameters. More...
struct	NvMediaOFSTExternalHintParams
	Holds OFST estimation parameters. More...

Macros
#define	NVMEDIA_IOFST_VERSION_MAJOR   1
	Major version number. More...
#define	NVMEDIA_IOFST_VERSION_MINOR   14
	Minor version number. More...

Enumerations
enum	NvMediaIOFSTType {   NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HP_MODE,   NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HP_MODE,   NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HQ_MODE,   NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HQ_MODE,   NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_UHP_MODE,   NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_UHP_MODE }
	Defines the image estimation type. More...
enum	NvMediaOFSTConfigFeatures { NVMEDIA_OFST_CONFIG_ENABLE_PROFILING = (1 << 0) }
	Defines OFST estimation configuration features. More...

Functions
NvMediaStatus	NvMediaIOFSTGetVersion (NvMediaVersion *version)
	Returns the version information for the NvMedia IOFST library. More...
NvMediaIOFST *	NvMediaIOFSTCreate (const NvMediaDevice *device, NvMediaIOFSTType estimationType, NvMediaSurfaceType inputFormat, NvMediaSurfaceType outputFormat, const NvMediaOFSTInitializeParams *initParams, uint8_t maxInputBuffering, NvMediaEncoderInstanceId instanceId)
	Creates an NvMediaIOFST object that can compute motion vectors or a disparity map based on differences between two images. More...
void	NvMediaIOFSTDestroy (const NvMediaIOFST *ofst)
	Destroys the created NvMediaIOFST object and frees associated resources. More...
NvMediaStatus	NvMediaIOFSTProcessFrame (const NvMediaIOFST *ofst, const NvMediaImage *frame, const NvMediaImage *refFrame, const NvMediaImage *mvs, const NvMediaOFSTExternalHintParams *extHintParams, NvMediaEncoderInstanceId instanceId)
	Performs OF/ST estimation on a specified frame pair. More...
NvMediaStatus	NvMediaIOFSTImageRegister (const NvMediaIOFST *ofst, const NvMediaImage *image, NvMediaAccessMode accessMode)
	Registers an NvMediaImage for use with an NvMediaIOFST handle. More...
NvMediaStatus	NvMediaIOFSTImageUnRegister (const NvMediaIOFST *ofst, const NvMediaImage *image)
	Un-registers NvMediaImage which was previously registered with NvMediaIOFST using NvMediaIOFSTImageRegister(). More...

Macro Definition Documentation

NVMEDIA_IOFST_VERSION_MAJOR

#define NVMEDIA_IOFST_VERSION_MAJOR   1

Major version number.

Definition at line 38 of file nvmedia_iofst.h.

NVMEDIA_IOFST_VERSION_MINOR

#define NVMEDIA_IOFST_VERSION_MINOR   14

Minor version number.

Definition at line 40 of file nvmedia_iofst.h.

Enumeration Type Documentation

NvMediaIOFSTType

enum NvMediaIOFSTType

Defines the image estimation type.

Enumerator
NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HP_MODE	High Performance OpticalFlow. Relative comparison with respect to other Optical Flow modes: Performance: Moderate Estimation Precision: Moderate
NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HP_MODE	High Performance StereoDisparity. Relative comparison with respect to other Stereo Disparity modes: Performance: Moderate Estimation Precision: Moderate
NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HQ_MODE	High Quality OpticalFlow. Relative comparison with respect to other Optical Flow modes: Performance: Least Estimation Precision: Best
NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HQ_MODE	High Quality StereoDisparity. Relative comparison with respect to other Stereo Disparity modes: Performance: Least Estimation Precision: Best
NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_UHP_MODE	Ultra High Performance OpticalFlow. Relative comparison with respect to other Optical Flow modes: Performance: Best Estimation Precision: Least
NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_UHP_MODE	Ultra High Performance StereoDisparity. Relative comparison with respect to other Stereo Disparity modes: Performance: Best Estimation Precision: Least

Definition at line 45 of file nvmedia_iofst.h.

NvMediaOFSTConfigFeatures

enum NvMediaOFSTConfigFeatures

Defines OFST estimation configuration features.

Enumerator
NVMEDIA_OFST_CONFIG_ENABLE_PROFILING	Enables OFST profiling. This is an internal feature.

Definition at line 81 of file nvmedia_iofst.h.

Function Documentation

NvMediaIOFSTCreate()

NvMediaIOFST* NvMediaIOFSTCreate	(	const NvMediaDevice *	device,
		NvMediaIOFSTType	estimationType,
		NvMediaSurfaceType	inputFormat,
		NvMediaSurfaceType	outputFormat,
		const NvMediaOFSTInitializeParams *	initParams,
		uint8_t	maxInputBuffering,
		NvMediaEncoderInstanceId	instanceId
	)

Creates an NvMediaIOFST object that can compute motion vectors or a disparity map based on differences between two images.

Precondition

NvMediaIOFSTGetVersion()

NvMediaIOFSTNvSciSyncGetVersion() [for use with IOFST-NvSciSync APIs]

Postcondition

NvMediaIOFST object is created

Considerations for Safety:

• Operation Mode: Init

Parameters

[in]	device	A pointer to the NvMediaDevice this OFST is to use. Input range: Non-NULL - valid address
[in]	estimationType	OF/ST estimation type. Input range: One of the following: NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HP_MODE NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HP_MODE NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_HQ_MODE NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_HQ_MODE NVMEDIA_IMAGE_OPTICALFLOW_ESTIMATION_UHP_MODE NVMEDIA_IMAGE_STEREODISPARITY_ESTIMATION_UHP_MODE
[in]	inputFormat	Format of the input NvMediaImage. Input range: Input format returned by a call to NvMediaSurfaceFormatGetType(), with attr being set as follows: QNX Safety build: NVM_SURF_FMT_SET_ATTR_YUV(attr, YUV, 420, SEMI_PLANAR, UINT, [8/16], BL) Non-safety build: NVM_SURF_FMT_SET_ATTR_YUV(attr, LUMA, NONE, PACKED, UINT, [8/10/16], BL) NVM_SURF_FMT_SET_ATTR_YUV(attr, YUV, 420, SEMI_PLANAR, UINT, [8/10/12/16], BL) NVM_SURF_FMT_SET_ATTR_YUV(attr, YUV, 444, SEMI_PLANAR, UINT, [8/10/12/16], BL)
[in]	outputFormat	Format of the output NvMediaImage. Input range: Output format obtained by a call to NvMediaSurfaceFormatGetType(), with attr being set as follows: NVM_SURF_FMT_SET_ATTR_RGBA(attr, RG, INT, 16, BL) for Optical Flow NVM_SURF_FMT_SET_ATTR_RGBA(attr, ALPHA, INT, 16, BL) for Stereo Disparity
[in]	initParams	A pointer to a structure that specifies initialization parameters. Input range: Non-NULL - valid address. Ranges specific to each member in the structure can be found in NvMediaOFSTInitializeParams.
[in]	maxInputBuffering	Maximum number of NvMediaIOFSTProcessFrame() operations that can be queued by NvMediaIOFST. If more than maxInputBuffering operations are queued, NvMediaIOFSTProcessFrame() returns an error to indicate insufficient buffering. Input range: The values between 1 to 16, in increments of 1
[in]	instanceId	ID of the encoder engine instance. Input range: One of the following: NVMEDIA_ENCODER_INSTANCE_0 [Supported only on Non-safety builds] NVMEDIA_ENCODER_INSTANCE_1 NVMEDIA_ENCODER_INSTANCE_AUTO [Supported only on Non-safety builds]

Returns

Created NvMediaIOFST estimator handle if successful, or NULL otherwise.

NvMediaIOFSTDestroy()

void NvMediaIOFSTDestroy

(

const NvMediaIOFST *

ofst

)

Destroys the created NvMediaIOFST object and frees associated resources.

Precondition

NvMediaIOFSTImageUnRegister()

NvMediaIOFSTUnregisterNvSciSyncObj() [for use with IOFST-NvSciSync APIs]

Postcondition

NvMediaIOFST object is destroyed

Considerations for Safety:

• Operation Mode: De-init

Parameters

[in]

ofst

The NvMediaIOFST object to destroy, returned by NvMediaIOFSTCreate(). Input range: Non-NULL - valid pointer address

NvMediaIOFSTGetVersion()

NvMediaStatus NvMediaIOFSTGetVersion

(

NvMediaVersion *

version

)

Returns the version information for the NvMedia IOFST library.

Precondition

None

Postcondition

None

Considerations for Safety:

• Operation Mode: Init

Parameters

[out]

version

A pointer to a NvMediaVersion structure filled by the IOFST library.

Returns

NvMediaStatus, the completion status of the operation:

• NVMEDIA_STATUS_OK if successful.
• NVMEDIA_STATUS_BAD_PARAMETER if the version pointer is NULL.

NvMediaIOFSTImageRegister()

NvMediaStatus NvMediaIOFSTImageRegister	(	const NvMediaIOFST *	ofst,
		const NvMediaImage *	image,
		NvMediaAccessMode	accessMode
	)

Registers an NvMediaImage for use with an NvMediaIOFST handle.

The NvMediaIOFST handle maintains a record of all the images registered using this API.

This is a mandatory API in safety build as skipping it will result in non-deterministic NvMediaIOFSTProcessFrame() execution time.

For deterministic execution time of NvMediaIOFSTProcessFrame() API :

• NvMediaIOFSTImageRegister() must be called for every input and output NvMediaImage that will be used with NvMediaIOFST.
• All NvMediaIOFSTImageRegister() calls must be made before first NvMediaIOFSTProcessFrame() API call.

Maximum of 32 NvMediaImage handles can be registered per access mode.

Precondition

NvMediaIOFSTCreate()

Postcondition

NvMediaImage is registered with NvMediaIOFST object

Considerations for Safety:

• Operation Mode: Init

Parameters

[in]	ofst	NvMediaIOFST handle. Input range: Non-NULL - valid pointer address
[in]	image	A pointer to NvMediaImage object. Input range: Non-NULL - valid pointer address
[in]	accessMode	NvMediaAccessMode required for the image. Input range: One of the following based on the type of NvMediaImage being registered: NVMEDIA_ACCESS_MODE_READ [Input image] NVMEDIA_ACCESS_MODE_READ_WRITE [Output image]

Returns

NvMediaStatus, the completion status of operation:

• NVMEDIA_STATUS_OK if successful.
• NVMEDIA_STATUS_BAD_PARAMETER if ofst, image or accessMode is invalid.
• NVMEDIA_STATUS_ERROR in following cases:
  • User registers more than 32 images.
  • User registers same image with more than one accessModes.
  • User registers same image multiple times.
• NVMEDIA_STATUS_NOT_SUPPORTED if API is not supported in a specific platform configuration.

NvMediaIOFSTImageUnRegister()

NvMediaStatus NvMediaIOFSTImageUnRegister	(	const NvMediaIOFST *	ofst,
		const NvMediaImage *	image
	)

Un-registers NvMediaImage which was previously registered with NvMediaIOFST using NvMediaIOFSTImageRegister().

For all NvMediaImage handles registered with NvMediaIOFST using NvMediaIOFSTImageRegister() API, NvMediaIOFSTImageUnRegister() must be called before calling NvMediaIOFSTDestroy() API. For unregistration to succeed, it should be ensured that none of the submitted tasks on the image are pending prior to calling NvMediaIOFSTImageUnRegister() API. In order to ensure this, NvMediaImageGetStatus() API needs to be called on all output NvMediaImage, to wait for their task completion. Post this NvMediaIOFSTImageUnRegister() can be successfully called on valid NvMediaImage.

For deterministic execution of NvMediaIOFSTProcessFrame() API, NvMediaIOFSTImageUnRegister() must be called only after last NvMediaIOFSTProcessFrame() call.

Precondition

NvMediaIOFSTProcessFrame()

NvMediaImageGetStatus() [verify that processing is complete]

Postcondition

NvMediaImage is un-registered from NvMediaIOFST object

Considerations for Safety:

• Operation Mode: De-init

Parameters

[in]	ofst	NvMediaIOFST handle. Input range: Non-NULL - valid pointer address
[in]	image	A pointer to NvMediaImage to unregister. Input range: Non-NULL - valid pointer address

Returns

NvMediaStatus, the completion status of operation:

• NVMEDIA_STATUS_OK if successful.
• NVMEDIA_STATUS_BAD_PARAMETER if ofst or image is invalid NvMediaIOFSTImageRegister() API.
• NVMEDIA_STATUS_PENDING if the enqueued operations on the NvMediaImage are still pending. In this case, the application can choose to wait for operations to complete on the output surface using NvMediaImageGetStatus() or re-try the NvMediaIOFSTImageUnRegister() API call, untill the status returned is not NVMEDIA_STATUS_PENDING.
• NVMEDIA_STATUS_ERROR in following cases:
  • User unregisters an NvMediaImage which is not previously registered using NvMediaIOFSTImageRegister() API.
  • User unregisters an NvMediaImage multiple times.
• NVMEDIA_STATUS_NOT_SUPPORTED if API is not supported in a specific platform configuration.

NvMediaIOFSTProcessFrame()

NvMediaStatus NvMediaIOFSTProcessFrame	(	const NvMediaIOFST *	ofst,
		const NvMediaImage *	frame,
		const NvMediaImage *	refFrame,
		const NvMediaImage *	mvs,
		const NvMediaOFSTExternalHintParams *	extHintParams,
		NvMediaEncoderInstanceId	instanceId
	)

Performs OF/ST estimation on a specified frame pair.

Estimation is based on the difference between refFrame and frame. The output of Optical Flow processing is motion vectors, and that of Stereo Disparity processing is a disparity surface.

Precondition

NvMediaIOFSTImageRegister()

NvMediaIOFSTSetNvSciSyncObjforEOF() [for use with IOFST-NvSciSync APIs]

NvMediaIOFSTInsertPreNvSciSyncFence() [optional - for use with IOFST- NvSciSync APIs]

Postcondition

Optical Flow/Stereo Disparity estimation task is submitted

Considerations for Safety:

• Operation Mode: Runtime

Parameters

[in]	ofst	A pointer to the NvMediaIOFST estimator to use. Input range: Non-NULL - valid pointer address
[in]	frame	Optical Flow Processing: A pointer to an input NvMediaImage at time T+1 (current frame). Stereo Disparity Processing: A pointer to an input NvMediaImage containing the rectified left view. It must be the same sourceType as ofst->inputformat. frame is allocated through a call to NvMediaImageCreateFromNvSciBuf(). Input range: Non-NULL - valid pointer address.
[in]	refFrame	Optical Flow Processing: A pointer to an input NvMediaImage at time T, which acts as a reference to current frame. Stereo Disparity Processing: A pointer to an input NvMediaImage containing the rectified right view. It must be the same sourceType as ofst->inputFormat. refFrame is allocated through a call to NvMediaImageCreateFromNvSciBuf(). Input range: Non-NULL - valid pointer address.
[in]	mvs	A pointer to an NvMediaImage where motion vector output (for Optical Flow) and disparity surface (for Stereo Disparity) will be populated as a result of OF/ST processing. As the processing in not synchronous mvs will not be populated as soon as this API returns. Clients can optionally call NvMediaImageGetStatus() API to know the status of processing once this API returns. mvs must be of the same sourceType as ofst->outputFormat. mvs is allocated through a call to NvMediaImageCreateFromNvSciBuf(). Motion vector output is in S10.5 format, with a range of [-1024, 1023] and quarter pixel accuracy. This means that the MSB represents the sign of the flow vector, next 10bits represent the integer value and the last 5 bits represent the fractional value. As we support quarter pixel precision, out of the 5 bits that represent the fractional component, only the most significant 2 bits are in use. The last 3 bits of the fractional part are zero all the time. For Stereo Disparity, output corresponds to left view and absolute value of output will provide actual disparity. Input range: Non-NULL - valid pointer address.
[in]	extHintParams	A pointer to a structure containing external hint parameters corresponding to the input frame pair. This feature is not supported on QNX Safety build, the field should be set to NULL. Input range: NULL to disable ME hints Non-NULL - valid pointer address to enable ME hints
[in]	instanceId	The ID of the encoder engine instance. Input range: In the case of QNX safety build, this needs to be set to the same value of instanceId passed to NvMediaIOFSTCreate(). In the case of non-safety builds, if instanceId passed to NvMediaIOFSTCreate() was not NVMEDIA_ENCODER_INSTANCE_AUTO, then the value of to instanceId be passed here, should be the same as that of the value passed to NvMediaIOFSTCreate(). If NVMEDIA_ENCODER_INSTANCE_AUTO was passed to NvMediaIOFSTCreate(), the following instances are supported: NVMEDIA_ENCODER_INSTANCE_0 NVMEDIA_ENCODER_INSTANCE_1 If any other value is passed, an error is returned.

Returns

The completion status of the operation:

• NVMEDIA_STATUS_OK if the call is successful.
• NVMEDIA_STATUS_BAD_PARAMETER if any of the input parameters are invalid.
• NVMEDIA_STATUS_ERROR if there is an internal error in processing.