Detailed Description

Defines Deep Neural Network (DNN) module for performing inference using NVIDIA^® TensorRT^™ models.

Note: SW Release Applicability: These APIs are available in both NVIDIA DriveWorks and NVIDIA DRIVE Software releases.

Data Structures
struct	dwDNNCustomLayer
	Specifies custom layer. More...

struct	dwDNNMetaData
	Specifies TensorRT model header. More...

struct	dwDNNPluginConfiguration
	Specified plugin configuration. More...

Modules
	DNN Plugin
	Provides an interface for supporting non-standard DNN layers.

	DNN Types
	Defines structures used by DNN and DNN related modules.

Typedefs
typedef struct dwDNNObject const *	dwConstDNNHandle_t

typedef struct dwDNNObject *	dwDNNHandle_t
	Handles representing Deep Neural Network interface. More...

Functions
DW_API_PUBLIC dwStatus	dwDNN_getCUDAStream (cudaStream_t *stream, dwDNNHandle_t network)
	Gets the CUDA stream used by the feature list. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputBlobCount (uint32_t *count, dwDNNHandle_t network)
	Gets the input blob count. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputIndex (uint32_t blobIndex, const char8_t blobName, dwDNNHandle_t network)
	Gets the index of an input blob with a given blob name. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputSize (dwBlobSize *blobSize, uint32_t blobIndex, dwDNNHandle_t network)
	Gets the input blob size at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputTensorProperties (dwDNNTensorProperties *tensorProps, uint32_t blobIndex, dwDNNHandle_t network)
	Gets the input tensor properties at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getMetaData (dwDNNMetaData *metaData, dwDNNHandle_t network)
	Returns the metadata for the associated network model. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputBlobCount (uint32_t *count, dwDNNHandle_t network)
	Gets the output blob count. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputIndex (uint32_t blobIndex, const char8_t blobName, dwDNNHandle_t network)
	Gets the index of an output blob with a given blob name. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputSize (dwBlobSize *blobSize, uint32_t blobIndex, dwDNNHandle_t network)
	Gets the output blob size at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputTensorProperties (dwDNNTensorProperties *tensorProps, uint32_t blobIndex, dwDNNHandle_t network)
	Gets the output tensor properties at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_infer (dwDNNTensorHandle_t outputTensors, uint32_t outputTensorCount, dwConstDNNTensorHandle_t inputTensors, uint32_t inputTensorCount, dwDNNHandle_t network)
	Runs inference pipeline on the given input. More...

DW_API_PUBLIC dwStatus	dwDNN_inferRaw (float32_t const d_output, const float32_t const d_input, uint32_t batchsize, dwDNNHandle_t network)
	Forwards pass from all input blobs to all output blobs. More...

DW_API_PUBLIC dwStatus	dwDNN_inferSIO (float32_t d_output, const float32_t d_input, uint32_t batchsize, dwDNNHandle_t network)
	Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network). More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromFile (dwDNNHandle_t network, const char8_t modelFilename, const dwDNNPluginConfiguration *pluginConfiguration, dwProcessorType processorType, dwContextHandle_t context)
	Creates and initializes a TensorRT Network from file. More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromMemory (dwDNNHandle_t network, const char8_t modelContent, uint32_t modelContentSize, const dwDNNPluginConfiguration *pluginConfiguration, dwProcessorType processorType, dwContextHandle_t context)
	Creates and initializes a TensorRT Network from memory. More...

DW_API_PUBLIC dwStatus	dwDNN_release (dwDNNHandle_t network)
	Releases a given network. More...

DW_API_PUBLIC dwStatus	dwDNN_reset (dwDNNHandle_t network)
	Resets a given network. More...

DW_API_PUBLIC dwStatus	dwDNN_setCUDAStream (cudaStream_t stream, dwDNNHandle_t network)
	Sets the CUDA stream for infer operations. More...

Data Structure Documentation

◆ dwDNNCustomLayer

struct dwDNNCustomLayer

Data Fields
const char8_t *	layerName	Name of the custom layer.
const char8_t *	pluginLibraryPath	Path to a plugin shared object. Path must be either absolute path or path relative to DW lib folder.

◆ dwDNNMetaData

struct dwDNNMetaData

Data Fields
dwDataConditionerParams	dataConditionerParams	DataConditioner parameters for running this network.

◆ dwDNNPluginConfiguration

struct dwDNNPluginConfiguration

Data Fields
const dwDNNCustomLayer *	customLayers	Array of custom layers.
size_t	numCustomLayers	Number of custom layers.

Typedef Documentation

◆ dwConstDNNHandle_t

typedef struct dwDNNObject const* dwConstDNNHandle_t

Definition at line 65 of file DNN.h.

◆ dwDNNHandle_t

typedef struct dwDNNObject* dwDNNHandle_t

Handles representing Deep Neural Network interface.

Definition at line 64 of file DNN.h.

Function Documentation

◆ dwDNN_getCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream	(	cudaStream_t *	stream,
		dwDNNHandle_t	network
	)

Gets the CUDA stream used by the feature list.

Parameters

[out]	stream	The CUDA stream currently used.
[in]	network	A handle to the DNN module.

Returns: DW_INVALID_HANDLE if the given network handle or the stream are NULL or of wrong type.
DW_SUCCESS otherwise.

◆ dwDNN_getInputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount	(	uint32_t *	count,
		dwDNNHandle_t	network
	)

Gets the input blob count.

Parameters

[out]	count	A pointer to the number of input blobs.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getInputIndex()

DW_API_PUBLIC dwStatus dwDNN_getInputIndex	(	uint32_t *	blobIndex,
		const char8_t *	blobName,
		dwDNNHandle_t	network
	)

Gets the index of an input blob with a given blob name.

Parameters

[out]	blobIndex	A pointer to the index of the blob with the given name.
[in]	blobName	A pointer to the name of an input blob.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, blobIndex or blobName are null or an input blob with the given name is not found.
DW_SUCCESS otherwise.

◆ dwDNN_getInputSize()

DW_API_PUBLIC dwStatus dwDNN_getInputSize	(	dwBlobSize *	blobSize,
		uint32_t	blobIndex,
		dwDNNHandle_t	network
	)

Gets the input blob size at blobIndex.

Parameters

[out]	blobSize	A pointer to the where the input blob size is returned.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getInputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties	(	dwDNNTensorProperties *	tensorProps,
		uint32_t	blobIndex,
		dwDNNHandle_t	network
	)

Gets the input tensor properties at blobIndex.

Parameters

[out]	tensorProps	Tensor properties.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getMetaData()

DW_API_PUBLIC dwStatus dwDNN_getMetaData	(	dwDNNMetaData *	metaData,
		dwDNNHandle_t	network
	)

Returns the metadata for the associated network model.

Parameters

[out]	metaData	A pointer to a metadata structure.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or metaData are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount	(	uint32_t *	count,
		dwDNNHandle_t	network
	)

Gets the output blob count.

Parameters

[out]	count	A pointer to the number of output blobs.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputIndex()

DW_API_PUBLIC dwStatus dwDNN_getOutputIndex	(	uint32_t *	blobIndex,
		const char8_t *	blobName,
		dwDNNHandle_t	network
	)

Gets the index of an output blob with a given blob name.

Parameters

[out]	blobIndex	A pointer to the index of the blob with the given name.
[in]	blobName	A pointer to the name of an output blob.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_HANDLE - if provided network handle, blobIndex or blobName are NULL or an output blob with the given name is not found or the network handle is of wrong type
DW_SUCCESS otherwise.

◆ dwDNN_getOutputSize()

DW_API_PUBLIC dwStatus dwDNN_getOutputSize	(	dwBlobSize *	blobSize,
		uint32_t	blobIndex,
		dwDNNHandle_t	network
	)

Gets the output blob size at blobIndex.

Parameters

[out]	blobSize	A pointer to the where the output blob size is returned.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getOutputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties	(	dwDNNTensorProperties *	tensorProps,
		uint32_t	blobIndex,
		dwDNNHandle_t	network
	)

Gets the output tensor properties at blobIndex.

Parameters

[out]	tensorProps	Tensor properties.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_infer()

DW_API_PUBLIC dwStatus dwDNN_infer	(	dwDNNTensorHandle_t *	outputTensors,
		uint32_t	outputTensorCount,
		dwConstDNNTensorHandle_t *	inputTensors,
		uint32_t	inputTensorCount,
		dwDNNHandle_t	network
	)

Runs inference pipeline on the given input.

Parameters

[out]	outputTensors	Output tensors.
[in]	outputTensorCount	Number of output tensors.
[in]	inputTensors	Input tensors.
[in]	inputTensorCount	Number of input tensors.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, input or output are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_inferRaw()

DW_API_PUBLIC dwStatus dwDNN_inferRaw	(	float32_t const	d_output,
		const float32_t const	d_input,
		uint32_t	batchsize,
		dwDNNHandle_t	network
	)

Forwards pass from all input blobs to all output blobs.

Note: : The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().; : The umber of blobs must match the respective number of blobs returned by dwDNN_getInputBlobCount() and dwDNN_getOutputBlobCount().

Parameters

[out]	d_output	A pointer to an array of pointers to the input blobs in GPU Memory.
[in]	d_input	A pointer to an array of pointers to the output blobs in GPU Memory.
[in]	batchsize	Batch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle , output or input are NULL.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.

Note: Inference performance might be suboptimal if the network has been generated for a larger batch size.; If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_inferSIO()

DW_API_PUBLIC dwStatus dwDNN_inferSIO	(	float32_t *	d_output,
		const float32_t *	d_input,
		uint32_t	batchsize,
		dwDNNHandle_t	network
	)

Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network).

Note: This method requires the network to be single input - single output.; The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().

Parameters

[out]	d_output	A pointer to the output blob in GPU memory.
[in]	d_input	A pointer to the input blob in GPU memory.
[in]	batchsize	Batch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, output or input are NULL or if provided network is not a single input - single output network.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.

Note: Inference performance might be suboptimal if the network has been generated for a larger batch size.; If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_initializeTensorRTFromFile()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile	(	dwDNNHandle_t *	network,
		const char8_t *	modelFilename,
		const dwDNNPluginConfiguration *	pluginConfiguration,
		dwProcessorType	processorType,
		dwContextHandle_t	context
	)

Creates and initializes a TensorRT Network from file.

Parameters

[out]	network	A pointer to network handle that will be initialized from parameters.
[in]	modelFilename	A pointer to the name of the TensorRT model file.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	context	Specifies the handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.

Note

The network file must be created by the TensorRT_optimization tool.

DNN module will look for metadata file named <modelFilename>.json in the same folder. If it is present, metadata will be loaded from that file. Otherwise, it will be filled with default values. Example metadata:

{
    "dataConditionerParams" : {
        "meanValue" : [0.0, 0.0, 0.0],
        "splitPlanes" : true,
        "pixelScaleCoefficient": 1.0,
        "ignoreAspectRatio" : false,
        "doPerPlaneMeanNormalization" : false
    }
    "tonemapType" : "none",
    "__comment": "tonemapType can be one of {none, agtm}"
}

\

◆ dwDNN_initializeTensorRTFromMemory()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory	(	dwDNNHandle_t *	network,
		const char8_t *	modelContent,
		uint32_t	modelContentSize,
		const dwDNNPluginConfiguration *	pluginConfiguration,
		dwProcessorType	processorType,
		dwContextHandle_t	context
	)

Creates and initializes a TensorRT Network from memory.

Parameters

[out]	network	A pointer to network handle that is initialized from parameters.
[in]	modelContent	A pointer to network content in the memory.
[in]	modelContentSize	Specifies the size of network content in memory, in bytes.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	context	Specifies a handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.

Note: The network file must be created by the TensorRT_optimization tool.; DNN module will fill metadata with default values.

◆ dwDNN_release()

DW_API_PUBLIC dwStatus dwDNN_release ( dwDNNHandle_t network )

Releases a given network.

Parameters

[in] network The network handle to release.

Returns: DW_INVALID_ARGUMENT - if provided network handle is NULL.
DW_SUCCESS otherwise.

◆ dwDNN_reset()

DW_API_PUBLIC dwStatus dwDNN_reset ( dwDNNHandle_t network )

Resets a given network.

Parameters

[in] network Network handle to reset.

Returns: DW_INVALID_ARGUMENT - if provided network handle is null.
DW_SUCCESS otherwise.

◆ dwDNN_setCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_setCUDAStream	(	cudaStream_t	stream,
		dwDNNHandle_t	network
	)

Sets the CUDA stream for infer operations.

Note: The ownership of the stream remains by the callee.

Parameters

[in]	stream	The CUDA stream to be used. Default is the stream 0, resulting in synchronous operations.
[in]	network	A handle to the DNN module to set CUDA stream for.

Returns: DW_INVALID_ARGUMENT if the given network handle is NULL.
DW_SUCCESS otherwise.

Detailed Description

Data Structures

Modules

Typedefs

Functions

Data Structure Documentation

◆ dwDNNCustomLayer

◆ dwDNNMetaData

◆ dwDNNPluginConfiguration

Typedef Documentation

◆ dwConstDNNHandle_t

◆ dwDNNHandle_t

Function Documentation

◆ dwDNN_getCUDAStream()

◆ dwDNN_getInputBlobCount()

◆ dwDNN_getInputIndex()

◆ dwDNN_getInputSize()

◆ dwDNN_getInputTensorProperties()

◆ dwDNN_getMetaData()

◆ dwDNN_getOutputBlobCount()

◆ dwDNN_getOutputIndex()

◆ dwDNN_getOutputSize()

◆ dwDNN_getOutputTensorProperties()

◆ dwDNN_infer()

◆ dwDNN_inferRaw()

◆ dwDNN_inferSIO()

◆ dwDNN_initializeTensorRTFromFile()

◆ dwDNN_initializeTensorRTFromMemory()

◆ dwDNN_release()

◆ dwDNN_reset()

◆ dwDNN_setCUDAStream()