DriveWorks SDK Reference
3.0.4260 Release
For Test and Development only

DNN Interface

Detailed Description

Defines Deep Neural Network (DNN) module for performing inference using NVIDIA® TensorRT models.

Note
SW Release Applicability: These APIs are available in both NVIDIA DriveWorks and NVIDIA DRIVE Software releases.

Data Structures

struct  dwDNNCustomLayer
 Specifies custom layer. More...
 
struct  dwDNNPluginConfiguration
 Specified plugin configuration. More...
 

Modules

 DNN Plugin
 Provides an interface for supporting non-standard DNN layers.
 
 DNN Types
 Defines structures used by DNN and DNN related modules.
 

Typedefs

typedef struct dwDNNObject const * dwConstDNNHandle_t
 
typedef struct dwDNNObject * dwDNNHandle_t
 Handles representing Deep Neural Network interface. More...
 

Functions

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream (cudaStream_t *stream, dwDNNHandle_t network)
 Gets the CUDA stream used by the feature list. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount (uint32_t *count, dwDNNHandle_t network)
 Gets the input blob count. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputIndex (uint32_t *blobIndex, const char8_t *blobName, dwDNNHandle_t network)
 Gets the index of an input blob with a given blob name. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputSize (dwBlobSize *blobSize, uint32_t blobIndex, dwDNNHandle_t network)
 Gets the input blob size at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties (dwDNNTensorProperties *tensorProps, uint32_t blobIndex, dwDNNHandle_t network)
 Gets the input tensor properties at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getMetaData (dwDNNMetaData *metaData, dwDNNHandle_t network)
 Returns the metadata for the associated network model. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount (uint32_t *count, dwDNNHandle_t network)
 Gets the output blob count. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputIndex (uint32_t *blobIndex, const char8_t *blobName, dwDNNHandle_t network)
 Gets the index of an output blob with a given blob name. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputSize (dwBlobSize *blobSize, uint32_t blobIndex, dwDNNHandle_t network)
 Gets the output blob size at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties (dwDNNTensorProperties *tensorProps, uint32_t blobIndex, dwDNNHandle_t network)
 Gets the output tensor properties at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_infer (dwDNNTensorHandle_t *outputTensors, uint32_t outputTensorCount, dwConstDNNTensorHandle_t *inputTensors, uint32_t inputTensorCount, dwDNNHandle_t network)
 Runs inference pipeline on the given input. More...
 
DW_API_PUBLIC dwStatus dwDNN_inferRaw (float32_t *const *d_output, const float32_t *const *d_input, uint32_t batchsize, dwDNNHandle_t network)
 Forwards pass from all input blobs to all output blobs. More...
 
DW_API_PUBLIC dwStatus dwDNN_inferSIO (float32_t *d_output, const float32_t *d_input, uint32_t batchsize, dwDNNHandle_t network)
 Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network). More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile (dwDNNHandle_t *network, const char8_t *modelFilename, const dwDNNPluginConfiguration *pluginConfiguration, dwProcessorType processorType, dwContextHandle_t context)
 Creates and initializes a TensorRT Network from file. More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory (dwDNNHandle_t *network, const char8_t *modelContent, uint32_t modelContentSize, const dwDNNPluginConfiguration *pluginConfiguration, dwProcessorType processorType, dwContextHandle_t context)
 Creates and initializes a TensorRT Network from memory. More...
 
DW_API_PUBLIC dwStatus dwDNN_release (dwDNNHandle_t network)
 Releases a given network. More...
 
DW_API_PUBLIC dwStatus dwDNN_reset (dwDNNHandle_t network)
 Resets a given network. More...
 
DW_API_PUBLIC dwStatus dwDNN_setCUDAStream (cudaStream_t stream, dwDNNHandle_t network)
 Sets the CUDA stream for infer operations. More...
 

Data Structure Documentation

◆ dwDNNCustomLayer

struct dwDNNCustomLayer
Data Fields
const char8_t * layerName Name of the custom layer.
const char8_t * pluginLibraryPath Path to a plugin shared object.

Path must be either absolute path or path relative to DW lib folder.

◆ dwDNNMetaData

struct dwDNNMetaData
Data Fields
dwDataConditionerParams dataConditionerParams DataConditioner parameters for running this network.

◆ dwDNNPluginConfiguration

struct dwDNNPluginConfiguration
Data Fields
const dwDNNCustomLayer * customLayers Array of custom layers.
size_t numCustomLayers Number of custom layers.

Typedef Documentation

◆ dwConstDNNHandle_t

typedef struct dwDNNObject const* dwConstDNNHandle_t

Definition at line 65 of file DNN.h.

◆ dwDNNHandle_t

typedef struct dwDNNObject* dwDNNHandle_t

Handles representing Deep Neural Network interface.

Definition at line 64 of file DNN.h.

Function Documentation

◆ dwDNN_getCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream ( cudaStream_t *  stream,
dwDNNHandle_t  network 
)

Gets the CUDA stream used by the feature list.

Parameters
[out]streamThe CUDA stream currently used.
[in]networkA handle to the DNN module.
Returns
DW_INVALID_HANDLE if the given network handle or the stream are NULL or of wrong type.
DW_SUCCESS otherwise.

◆ dwDNN_getInputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount ( uint32_t *  count,
dwDNNHandle_t  network 
)

Gets the input blob count.

Parameters
[out]countA pointer to the number of input blobs.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getInputIndex()

DW_API_PUBLIC dwStatus dwDNN_getInputIndex ( uint32_t *  blobIndex,
const char8_t blobName,
dwDNNHandle_t  network 
)

Gets the index of an input blob with a given blob name.

Parameters
[out]blobIndexA pointer to the index of the blob with the given name.
[in]blobNameA pointer to the name of an input blob.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, blobIndex or blobName are null or an input blob with the given name is not found.
DW_SUCCESS otherwise.

◆ dwDNN_getInputSize()

DW_API_PUBLIC dwStatus dwDNN_getInputSize ( dwBlobSize blobSize,
uint32_t  blobIndex,
dwDNNHandle_t  network 
)

Gets the input blob size at blobIndex.

Parameters
[out]blobSizeA pointer to the where the input blob size is returned.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getInputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties ( dwDNNTensorProperties tensorProps,
uint32_t  blobIndex,
dwDNNHandle_t  network 
)

Gets the input tensor properties at blobIndex.

Parameters
[out]tensorPropsTensor properties.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getMetaData()

DW_API_PUBLIC dwStatus dwDNN_getMetaData ( dwDNNMetaData metaData,
dwDNNHandle_t  network 
)

Returns the metadata for the associated network model.

Parameters
[out]metaDataA pointer to a metadata structure.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or metaData are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount ( uint32_t *  count,
dwDNNHandle_t  network 
)

Gets the output blob count.

Parameters
[out]countA pointer to the number of output blobs.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputIndex()

DW_API_PUBLIC dwStatus dwDNN_getOutputIndex ( uint32_t *  blobIndex,
const char8_t blobName,
dwDNNHandle_t  network 
)

Gets the index of an output blob with a given blob name.

Parameters
[out]blobIndexA pointer to the index of the blob with the given name.
[in]blobNameA pointer to the name of an output blob.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_HANDLE - if provided network handle, blobIndex or blobName are NULL or an output blob with the given name is not found or the network handle is of wrong type
DW_SUCCESS otherwise.

◆ dwDNN_getOutputSize()

DW_API_PUBLIC dwStatus dwDNN_getOutputSize ( dwBlobSize blobSize,
uint32_t  blobIndex,
dwDNNHandle_t  network 
)

Gets the output blob size at blobIndex.

Parameters
[out]blobSizeA pointer to the where the output blob size is returned.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getOutputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties ( dwDNNTensorProperties tensorProps,
uint32_t  blobIndex,
dwDNNHandle_t  network 
)

Gets the output tensor properties at blobIndex.

Parameters
[out]tensorPropsTensor properties.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_infer()

DW_API_PUBLIC dwStatus dwDNN_infer ( dwDNNTensorHandle_t outputTensors,
uint32_t  outputTensorCount,
dwConstDNNTensorHandle_t inputTensors,
uint32_t  inputTensorCount,
dwDNNHandle_t  network 
)

Runs inference pipeline on the given input.

Parameters
[out]outputTensorsOutput tensors.
[in]outputTensorCountNumber of output tensors.
[in]inputTensorsInput tensors.
[in]inputTensorCountNumber of input tensors.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, input or output are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_inferRaw()

DW_API_PUBLIC dwStatus dwDNN_inferRaw ( float32_t *const *  d_output,
const float32_t *const *  d_input,
uint32_t  batchsize,
dwDNNHandle_t  network 
)

Forwards pass from all input blobs to all output blobs.

Note
: The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().
: The umber of blobs must match the respective number of blobs returned by dwDNN_getInputBlobCount() and dwDNN_getOutputBlobCount().
Parameters
[out]d_outputA pointer to an array of pointers to the input blobs in GPU Memory.
[in]d_inputA pointer to an array of pointers to the output blobs in GPU Memory.
[in]batchsizeBatch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle , output or input are NULL.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.
Note
Inference performance might be suboptimal if the network has been generated for a larger batch size.
If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_inferSIO()

DW_API_PUBLIC dwStatus dwDNN_inferSIO ( float32_t d_output,
const float32_t d_input,
uint32_t  batchsize,
dwDNNHandle_t  network 
)

Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network).

Note
This method requires the network to be single input - single output.
The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().
Parameters
[out]d_outputA pointer to the output blob in GPU memory.
[in]d_inputA pointer to the input blob in GPU memory.
[in]batchsizeBatch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, output or input are NULL or if provided network is not a single input - single output network.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.
Note
Inference performance might be suboptimal if the network has been generated for a larger batch size.
If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_initializeTensorRTFromFile()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile ( dwDNNHandle_t network,
const char8_t modelFilename,
const dwDNNPluginConfiguration pluginConfiguration,
dwProcessorType  processorType,
dwContextHandle_t  context 
)

Creates and initializes a TensorRT Network from file.

Parameters
[out]networkA pointer to network handle that will be initialized from parameters.
[in]modelFilenameA pointer to the name of the TensorRT model file.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]contextSpecifies the handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.
Note
The network file must be created by the TensorRT_optimization tool.
DNN module will look for metadata file named <modelFilename>.json in the same folder. If it is present, metadata will be loaded from that file. Otherwise, it will be filled with default values. Example metadata:
{
    "dataConditionerParams" : {
        "meanValue" : [0.0, 0.0, 0.0],
        "splitPlanes" : true,
        "pixelScaleCoefficient": 1.0,
        "ignoreAspectRatio" : false,
        "doPerPlaneMeanNormalization" : false
    }
    "tonemapType" : "none",
    "__comment": "tonemapType can be one of {none, agtm}"
}
\

◆ dwDNN_initializeTensorRTFromMemory()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory ( dwDNNHandle_t network,
const char8_t modelContent,
uint32_t  modelContentSize,
const dwDNNPluginConfiguration pluginConfiguration,
dwProcessorType  processorType,
dwContextHandle_t  context 
)

Creates and initializes a TensorRT Network from memory.

Parameters
[out]networkA pointer to network handle that is initialized from parameters.
[in]modelContentA pointer to network content in the memory.
[in]modelContentSizeSpecifies the size of network content in memory, in bytes.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]contextSpecifies a handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.
Note
The network file must be created by the TensorRT_optimization tool.
DNN module will fill metadata with default values.

◆ dwDNN_release()

DW_API_PUBLIC dwStatus dwDNN_release ( dwDNNHandle_t  network)

Releases a given network.

Parameters
[in]networkThe network handle to release.
Returns
DW_INVALID_ARGUMENT - if provided network handle is NULL.
DW_SUCCESS otherwise.

◆ dwDNN_reset()

DW_API_PUBLIC dwStatus dwDNN_reset ( dwDNNHandle_t  network)

Resets a given network.

Parameters
[in]networkNetwork handle to reset.
Returns
DW_INVALID_ARGUMENT - if provided network handle is null.
DW_SUCCESS otherwise.

◆ dwDNN_setCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_setCUDAStream ( cudaStream_t  stream,
dwDNNHandle_t  network 
)

Sets the CUDA stream for infer operations.

Note
The ownership of the stream remains by the callee.
Parameters
[in]streamThe CUDA stream to be used. Default is the stream 0, resulting in synchronous operations.
[in]networkA handle to the DNN module to set CUDA stream for.
Returns
DW_INVALID_ARGUMENT if the given network handle is NULL.
DW_SUCCESS otherwise.