Detailed Description

TRT module to perform various deep learning inference.

Definition at line 137 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

Collaboration diagram for TrtModule:

Public Member Functions
	TrtModule ()
	Construct a TrtModule instance. More...

void	initializeModel (std::function< void(nvinfer1::INetworkDefinition network, nvinfer1::IBuilder builder, nvinfer1::IBuilderConfig config, int batchSize)> onnxProfileFunc, std::function< void(TrtModule , const std::vector< int > &sizes)> setRuntimeDimFunc)
	Create TensorRT inference engine; if engine does not exist, serialize from model. More...

void	createInferEngine ()
	Build a TensorRT engine from the configured network model. More...

void	parseOnnxModel (nvinfer1::INetworkDefinition &network, nvinfer1::ILogger &logger, std::unique_ptr< nvonnxparser::IParser > &parser)
	Load ONNX model into network. More...

nvinfer1::ICudaEngine *	decodeTltModel (nvinfer1::IBuilder const builder, nvinfer1::IBuilderConfig const config)
	Decode TAO encrypted model. More...

bool	loadTRTEngine ()
	Load reid TRT engine. More...

void	submitInput (std::vector< void * > &bindings, const std::vector< int > &batchSize, cudaStream_t &cuStream)
	Submit network input for inference. More...

void	printEngineInfo ()
	Print TRT engine information. More...

void	setCudaStream (cudaStream_t cuStream)
	Set the CUDA stream used for TensorRT inference. More...

	TrtModule ()
	Construct a TrtModule instance. More...

void	initializeModel (std::function< void(nvinfer1::INetworkDefinition network, nvinfer1::IBuilder builder, nvinfer1::IBuilderConfig config, int batchSize)> onnxProfileFunc, std::function< void(TrtModule , const std::vector< int > &sizes)> setRuntimeDimFunc)
	Create TensorRT inference engine; if engine does not exist, serialize from model. More...

void	createInferEngine ()
	Build a TensorRT engine from the configured network model. More...

void	parseOnnxModel (nvinfer1::INetworkDefinition &network, nvinfer1::ILogger &logger, std::unique_ptr< nvonnxparser::IParser > &parser)
	Load ONNX model into network. More...

nvinfer1::ICudaEngine *	decodeTltModel (nvinfer1::IBuilder const builder, nvinfer1::IBuilderConfig const config)
	Decode TAO encrypted model. More...

bool	loadTRTEngine ()
	Load reid TRT engine. More...

void	submitInput (std::vector< void * > &bindings, const std::vector< int > &batchSize, cudaStream_t &cuStream)
	Submit network input for inference. More...

void	printEngineInfo ()
	Print TRT engine information. More...

void	setCudaStream (cudaStream_t cuStream)
	Set the CUDA stream used for TensorRT inference. More...

Data Fields
TRTConfigParams	m_params
	TRT module config. More...

std::shared_ptr< nvinfer1::ICudaEngine >	m_engine
	TensorRT engine for reid. More...

std::shared_ptr< nvinfer1::IExecutionContext >	m_context
	TensorRT execution context. More...

cudaStream_t	m_cuStream
	Cuda stream to run all GPU tasks. More...

Protected Attributes
std::shared_ptr< nvinfer1::IRuntime >	m_runtime
	TensorRT runtime used to deserialize engines. More...

std::shared_ptr< TrackerInt8EntropyCalibrator >	m_calibrator
	TensorRT int8 calibrator. More...

std::string	m_absModelFilePath
	Absolute model file path. More...

std::function< void(nvinfer1::INetworkDefinition network, nvinfer1::IBuilder builder, nvinfer1::IBuilderConfig *config, int batchSize)>	m_onnxProfileFunc
	Custom function to perform profiling for dynamic shapes required by TRT. More...

std::function< void(TrtModule *, const std::vector< int > &)>	m_setRuntimeDimFunc
	Custom function to set runtime dimension required by TRT. More...

Constructor & Destructor Documentation

◆ TrtModule() [1/2]

TrtModule::TrtModule ( )

Construct a TrtModule instance.

◆ TrtModule() [2/2]

TrtModule::TrtModule ( )

Construct a TrtModule instance.

Member Function Documentation

◆ createInferEngine() [1/2]

void TrtModule::createInferEngine ( )

Build a TensorRT engine from the configured network model.

◆ createInferEngine() [2/2]

void TrtModule::createInferEngine ( )

Build a TensorRT engine from the configured network model.

◆ decodeTltModel() [1/2]

nvinfer1::ICudaEngine* TrtModule::decodeTltModel	(	nvinfer1::IBuilder *const	builder,
		nvinfer1::IBuilderConfig *const	config
	)

Decode TAO encrypted model.

Parameters

[in]	builder	TensorRT engine builder.
[in]	config	TensorRT builder configuration.

Returns: Decoded engine.

◆ decodeTltModel() [2/2]

nvinfer1::ICudaEngine* TrtModule::decodeTltModel	(	nvinfer1::IBuilder *const	builder,
		nvinfer1::IBuilderConfig *const	config
	)

Decode TAO encrypted model.

Parameters

[in]	builder	TensorRT engine builder.
[in]	config	TensorRT builder configuration.

Returns: Decoded engine.

◆ initializeModel() [1/2]

void TrtModule::initializeModel	(	std::function< void(nvinfer1::INetworkDefinition network, nvinfer1::IBuilder builder, nvinfer1::IBuilderConfig *config, int batchSize)>	onnxProfileFunc,
		std::function< void(TrtModule *, const std::vector< int > &sizes)>	setRuntimeDimFunc
	)

Create TensorRT inference engine; if engine does not exist, serialize from model.

Parameters

[in]	onnxProfileFunc	Callback that sets the ONNX optimization profile before the engine is built.
[in]	setRuntimeDimFunc	Callback that sets runtime dimensions for dynamic input shapes.

◆ initializeModel() [2/2]

void TrtModule::initializeModel	(	std::function< void(nvinfer1::INetworkDefinition network, nvinfer1::IBuilder builder, nvinfer1::IBuilderConfig *config, int batchSize)>	onnxProfileFunc,
		std::function< void(TrtModule *, const std::vector< int > &sizes)>	setRuntimeDimFunc
	)

Create TensorRT inference engine; if engine does not exist, serialize from model.

Parameters

[in]	onnxProfileFunc	Callback that sets the ONNX optimization profile before the engine is built.
[in]	setRuntimeDimFunc	Callback that sets runtime dimensions for dynamic input shapes.

◆ loadTRTEngine() [1/2]

bool TrtModule::loadTRTEngine ( )

Load reid TRT engine.

Returns: True if the engine file was loaded, otherwise the engine needs to be rebuilt.

◆ loadTRTEngine() [2/2]

bool TrtModule::loadTRTEngine ( )

Load reid TRT engine.

Returns: True if the engine file was loaded, otherwise the engine needs to be rebuilt.

◆ parseOnnxModel() [1/2]

void TrtModule::parseOnnxModel	(	nvinfer1::INetworkDefinition &	network,
		nvinfer1::ILogger &	logger,
		std::unique_ptr< nvonnxparser::IParser > &	parser
	)

Load ONNX model into network.

Parameters

[out]	network	Network with ONNX model loaded.
[in]	logger	TensorRT logger.
[out]	parser	ONNX parser to populate.

◆ parseOnnxModel() [2/2]

void TrtModule::parseOnnxModel	(	nvinfer1::INetworkDefinition &	network,
		nvinfer1::ILogger &	logger,
		std::unique_ptr< nvonnxparser::IParser > &	parser
	)

Load ONNX model into network.

Parameters

[out]	network	Network with ONNX model loaded.
[in]	logger	TensorRT logger.
[out]	parser	ONNX parser to populate.

◆ printEngineInfo() [1/2]

void TrtModule::printEngineInfo ( )

Print TRT engine information.

◆ printEngineInfo() [2/2]

void TrtModule::printEngineInfo ( )

Print TRT engine information.

◆ setCudaStream() [1/2]

void TrtModule::setCudaStream ( cudaStream_t cuStream )

inline

Set the CUDA stream used for TensorRT inference.

Parameters

[in] cuStream CUDA stream used for the operation.

Definition at line 198 of file 9.2/sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

References m_cuStream.

◆ setCudaStream() [2/2]

void TrtModule::setCudaStream ( cudaStream_t cuStream )

inline

Set the CUDA stream used for TensorRT inference.

Parameters

[in] cuStream CUDA stream used for the operation.

Definition at line 198 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

References m_cuStream.

◆ submitInput() [1/2]

void TrtModule::submitInput	(	std::vector< void * > &	bindings,
		const std::vector< int > &	batchSize,
		cudaStream_t &	cuStream
	)

Submit network input for inference.

Parameters

[in]	bindings	Input and output tensor addresses for the network.
[in]	batchSize	Runtime batch size for each dynamic input.
[in]	cuStream	CUDA stream that runs the GPU work.

◆ submitInput() [2/2]

void TrtModule::submitInput	(	std::vector< void * > &	bindings,
		const std::vector< int > &	batchSize,
		cudaStream_t &	cuStream
	)

Submit network input for inference.

Parameters

[in]	bindings	Input and output tensor addresses for the network.
[in]	batchSize	Runtime batch size for each dynamic input.
[in]	cuStream	CUDA stream that runs the GPU work.

Field Documentation

◆ m_absModelFilePath

std::string TrtModule::m_absModelFilePath

protected

Absolute model file path.

Definition at line 207 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

◆ m_calibrator

std::shared_ptr< TrackerInt8EntropyCalibrator > TrtModule::m_calibrator

protected

TensorRT int8 calibrator.

Definition at line 206 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

◆ m_context

std::shared_ptr< nvinfer1::IExecutionContext > TrtModule::m_context

TensorRT execution context.

Definition at line 202 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

Referenced by setRuntimeDim4EVSamDecoder(), setRuntimeDim4EVSamEncoder(), setRuntimeDim4NanoSamDecoder(), setRuntimeDim4Sam2MaskDecoder(), setRuntimeDim4Sam2MemAttn(), and setRuntimeDim4Sam2MemEncoder().

◆ m_cuStream

cudaStream_t TrtModule::m_cuStream

Cuda stream to run all GPU tasks.

Definition at line 203 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

Referenced by setCudaStream(), setRuntimeDim4EVSamDecoder(), setRuntimeDim4NanoSamDecoder(), setRuntimeDim4Sam2MaskDecoder(), setRuntimeDim4Sam2MemAttn(), and setRuntimeDim4Sam2MemEncoder().

◆ m_engine

std::shared_ptr< nvinfer1::ICudaEngine > TrtModule::m_engine

TensorRT engine for reid.

Definition at line 201 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

Referenced by setRuntimeDim4EVSamDecoder(), setRuntimeDim4EVSamEncoder(), setRuntimeDim4NanoSamDecoder(), setRuntimeDim4Sam2MaskDecoder(), setRuntimeDim4Sam2MemAttn(), and setRuntimeDim4Sam2MemEncoder().

◆ m_onnxProfileFunc

std::function< void(nvinfer1::INetworkDefinition *network, nvinfer1::IBuilder *builder, nvinfer1::IBuilderConfig *config, int batchSize)> TrtModule::m_onnxProfileFunc

protected

Custom function to perform profiling for dynamic shapes required by TRT.

Definition at line 210 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

◆ m_params

TRTConfigParams TrtModule::m_params

TRT module config.

Definition at line 198 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

◆ m_runtime

std::shared_ptr< nvinfer1::IRuntime > TrtModule::m_runtime

protected

TensorRT runtime used to deserialize engines.

Definition at line 205 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

◆ m_setRuntimeDimFunc

std::function< void(TrtModule *, const std::vector< int > &)> TrtModule::m_setRuntimeDimFunc

protected

Custom function to set runtime dimension required by TRT.

Definition at line 211 of file sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp.

The documentation for this class was generated from the following file:

sources/libs/nvmultiobjecttracker/include/modules/FrameTransform/trtModule.hpp

NVIDIA DeepStream SDK API Reference

9.2 Release

Detailed Description

Public Member Functions

Data Fields

Protected Attributes

Constructor & Destructor Documentation

◆ TrtModule() [1/2]

◆ TrtModule() [2/2]

Member Function Documentation

◆ createInferEngine() [1/2]

◆ createInferEngine() [2/2]

◆ decodeTltModel() [1/2]

◆ decodeTltModel() [2/2]

◆ initializeModel() [1/2]

◆ initializeModel() [2/2]

◆ loadTRTEngine() [1/2]

◆ loadTRTEngine() [2/2]

◆ parseOnnxModel() [1/2]

◆ parseOnnxModel() [2/2]

◆ printEngineInfo() [1/2]

◆ printEngineInfo() [2/2]

◆ setCudaStream() [1/2]

◆ setCudaStream() [2/2]

◆ submitInput() [1/2]

◆ submitInput() [2/2]

Field Documentation

◆ m_absModelFilePath

◆ m_calibrator

◆ m_context

◆ m_cuStream

◆ m_engine

◆ m_onnxProfileFunc

◆ m_params

◆ m_runtime

◆ m_setRuntimeDimFunc