Detailed Description

TensorRT inference engine class.

This class handles the deserialization of TensorRT engines and running inference.

Definition at line 39 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

Public Member Functions
	TensorRTEngine ()
	Constructor. More...

	~TensorRTEngine ()
	Destructor - frees allocated resources. More...

bool	isInitialized () const
	Check if the engine is initialized. More...

void	configure (const ConfigProperties &config)
	Configure the TensorRT engine with properties from config. More...

int	getGpuId () const
	Get the GPU ID used for inference. More...

nvinfer1::ICudaEngine *	getEngine () const
	Get the TensorRT engine object. More...

cudaStream_t	getCudaStream () const
	Get CUDA stream. More...

int	getTensorBindingIndex (const std::string &name) const
	Get binding index for a tensor by name. More...

nvinfer1::Dims	getBindingDimensions (int index) const
	Get binding dimensions for a tensor by index. More...

void *	getDeviceBuffer (const std::string &name) const
	Get device buffer for a tensor by name. More...

std::vector< std::string >	getInputTensorNames () const
	Get all input tensor names. More...

std::vector< std::string >	getOutputTensorNames () const
	Get all output tensor names. More...

bool	updateBindingPointer (const std::string &tensorName, void *newBuffer)
	Updates the binding pointer for a specific tensor. More...

bool	deserializeEngine (const std::string &engineFilePath, TensorCache tensorCache, int batchSize, int numSensors, PrettyLogger logger=nullptr)
	Deserialize a TensorRT engine from a file. More...

bool	runInference (int batch_size, int num_sensors, PrettyLogger *logger=nullptr)
	Run inference using pre-allocated output tensors. More...

	TensorRTEngine ()
	Constructor. More...

	~TensorRTEngine ()
	Destructor - frees allocated resources. More...

bool	isInitialized () const
	Check if the engine is initialized. More...

void	configure (const ConfigProperties &config)
	Configure the TensorRT engine with properties from config. More...

int	getGpuId () const
	Get the GPU ID used for inference. More...

nvinfer1::ICudaEngine *	getEngine () const
	Get the TensorRT engine object. More...

cudaStream_t	getCudaStream () const
	Get CUDA stream. More...

int	getTensorBindingIndex (const std::string &name) const
	Get binding index for a tensor by name. More...

nvinfer1::Dims	getBindingDimensions (int index) const
	Get binding dimensions for a tensor by index. More...

void *	getDeviceBuffer (const std::string &name) const
	Get device buffer for a tensor by name. More...

std::vector< std::string >	getInputTensorNames () const
	Get all input tensor names. More...

std::vector< std::string >	getOutputTensorNames () const
	Get all output tensor names. More...

bool	updateBindingPointer (const std::string &tensorName, void *newBuffer)
	Updates the binding pointer for a specific tensor. More...

bool	deserializeEngine (const std::string &engineFilePath, TensorCache tensorCache, int batchSize, int numSensors, PrettyLogger logger=nullptr)
	Deserialize a TensorRT engine from a file. More...

bool	runInference (int batch_size, int num_sensors, PrettyLogger *logger=nullptr)
	Run inference using pre-allocated output tensors. More...

Constructor & Destructor Documentation

◆ TensorRTEngine() [1/2]

TensorRTEngine::TensorRTEngine ( )

Constructor.

◆ ~TensorRTEngine() [1/2]

TensorRTEngine::~TensorRTEngine ( )

Destructor - frees allocated resources.

◆ TensorRTEngine() [2/2]

TensorRTEngine::TensorRTEngine ( )

Constructor.

◆ ~TensorRTEngine() [2/2]

TensorRTEngine::~TensorRTEngine ( )

Destructor - frees allocated resources.

Member Function Documentation

◆ configure() [1/2]

void TensorRTEngine::configure ( const ConfigProperties & config )

inline

Configure the TensorRT engine with properties from config.

Parameters

config Configuration properties containing GPU ID, logger severity, and profiler settings

Definition at line 63 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References ConfigProperties::enable_profiler, ConfigProperties::gpu_id, and ConfigProperties::trt_logger_severity.

◆ configure() [2/2]

void TensorRTEngine::configure ( const ConfigProperties & config )

inline

Configure the TensorRT engine with properties from config.

Parameters

config Configuration properties containing GPU ID, logger severity, and profiler settings

Definition at line 63 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References ConfigProperties::enable_profiler, ConfigProperties::gpu_id, and ConfigProperties::trt_logger_severity.

◆ deserializeEngine() [1/2]

bool TensorRTEngine::deserializeEngine	(	const std::string &	engineFilePath,
		TensorCache *	tensorCache,
		int	batchSize,
		int	numSensors,
		PrettyLogger *	logger = `nullptr`
	)

Deserialize a TensorRT engine from a file.

Parameters

engineFilePath	Path to the serialized engine file
tensorCache	Tensor cache to store tensors
batchSize	Sparse4D model batch size
numSensors	Number of sensors [Deepstream batch size
logger	Optional logger for detailed status messages

Returns: true if deserialization succeeded, false otherwise

◆ deserializeEngine() [2/2]

bool TensorRTEngine::deserializeEngine	(	const std::string &	engineFilePath,
		TensorCache *	tensorCache,
		int	batchSize,
		int	numSensors,
		PrettyLogger *	logger = `nullptr`
	)

Deserialize a TensorRT engine from a file.

Parameters

engineFilePath	Path to the serialized engine file
tensorCache	Tensor cache to store tensors
batchSize	Sparse4D model batch size
numSensors	Number of sensors [Deepstream batch size
logger	Optional logger for detailed status messages

Returns: true if deserialization succeeded, false otherwise

◆ getBindingDimensions() [1/2]

nvinfer1::Dims TensorRTEngine::getBindingDimensions ( int index ) const

inline

Get binding dimensions for a tensor by index.

Parameters

index Binding index

Returns: Tensor dimensions

Definition at line 115 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getBindingDimensions() [2/2]

nvinfer1::Dims TensorRTEngine::getBindingDimensions ( int index ) const

inline

Get binding dimensions for a tensor by index.

Parameters

index Binding index

Returns: Tensor dimensions

Definition at line 115 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getCudaStream() [1/2]

cudaStream_t TensorRTEngine::getCudaStream ( ) const

inline

Get CUDA stream.

Returns: CUDA stream used for inference

Definition at line 88 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getCudaStream() [2/2]

cudaStream_t TensorRTEngine::getCudaStream ( ) const

inline

Get CUDA stream.

Returns: CUDA stream used for inference

Definition at line 88 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getDeviceBuffer() [1/2]

void* TensorRTEngine::getDeviceBuffer ( const std::string & name ) const

inline

Get device buffer for a tensor by name.

Parameters

name	Tensor name

Returns: Pointer to device buffer, or nullptr if tensor not found

Definition at line 130 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References getTensorBindingIndex().

◆ getDeviceBuffer() [2/2]

void* TensorRTEngine::getDeviceBuffer ( const std::string & name ) const

inline

Get device buffer for a tensor by name.

Parameters

name	Tensor name

Returns: Pointer to device buffer, or nullptr if tensor not found

Definition at line 130 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References getTensorBindingIndex().

◆ getEngine() [1/2]

nvinfer1::ICudaEngine* TensorRTEngine::getEngine ( ) const

inline

Get the TensorRT engine object.

Returns: Pointer to ICudaEngine

Definition at line 81 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getEngine() [2/2]

nvinfer1::ICudaEngine* TensorRTEngine::getEngine ( ) const

inline

Get the TensorRT engine object.

Returns: Pointer to ICudaEngine

Definition at line 81 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getGpuId() [1/2]

int TensorRTEngine::getGpuId ( ) const

inline

Get the GPU ID used for inference.

Returns: The GPU ID currently in use

Definition at line 74 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getGpuId() [2/2]

int TensorRTEngine::getGpuId ( ) const

inline

Get the GPU ID used for inference.

Returns: The GPU ID currently in use

Definition at line 74 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getInputTensorNames() [1/2]

std::vector<std::string> TensorRTEngine::getInputTensorNames ( ) const

inline

Get all input tensor names.

Returns: Vector of input tensor names

Definition at line 143 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getInputTensorNames() [2/2]

std::vector<std::string> TensorRTEngine::getInputTensorNames ( ) const

inline

Get all input tensor names.

Returns: Vector of input tensor names

Definition at line 143 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getOutputTensorNames() [1/2]

std::vector<std::string> TensorRTEngine::getOutputTensorNames ( ) const

inline

Get all output tensor names.

Returns: Vector of output tensor names

Definition at line 176 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getOutputTensorNames() [2/2]

std::vector<std::string> TensorRTEngine::getOutputTensorNames ( ) const

inline

Get all output tensor names.

Returns: Vector of output tensor names

Definition at line 176 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getTensorBindingIndex() [1/2]

int TensorRTEngine::getTensorBindingIndex ( const std::string & name ) const

inline

Get binding index for a tensor by name.

Parameters

name	Tensor name

Returns: Binding index, or -1 if tensor not found

Definition at line 96 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ getTensorBindingIndex() [2/2]

int TensorRTEngine::getTensorBindingIndex ( const std::string & name ) const

inline

Get binding index for a tensor by name.

Parameters

name	Tensor name

Returns: Binding index, or -1 if tensor not found

Definition at line 96 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

Referenced by getDeviceBuffer(), and updateBindingPointer().

◆ isInitialized() [1/2]

bool TensorRTEngine::isInitialized ( ) const

inline

Check if the engine is initialized.

Returns: true if engine is loaded and ready, false otherwise

Definition at line 56 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ isInitialized() [2/2]

bool TensorRTEngine::isInitialized ( ) const

inline

Check if the engine is initialized.

Returns: true if engine is loaded and ready, false otherwise

Definition at line 56 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

◆ runInference() [1/2]

bool TensorRTEngine::runInference	(	int	batch_size,
		int	num_sensors,
		PrettyLogger *	logger = `nullptr`
	)

Run inference using pre-allocated output tensors.

Parameters

batch_size	Sparse4D model batch size
num_sensors	Number of cameras (for dynamic tensor dimensions)
logger	Optional logger for detailed status messages

Returns: true if inference succeeded, false otherwise

◆ runInference() [2/2]

bool TensorRTEngine::runInference	(	int	batch_size,
		int	num_sensors,
		PrettyLogger *	logger = `nullptr`
	)

Run inference using pre-allocated output tensors.

Parameters

batch_size	Sparse4D model batch size
num_sensors	Number of cameras (for dynamic tensor dimensions)
logger	Optional logger for detailed status messages

Returns: true if inference succeeded, false otherwise

◆ updateBindingPointer() [1/2]

bool TensorRTEngine::updateBindingPointer	(	const std::string &	tensorName,
		void *	newBuffer
	)

inline

Updates the binding pointer for a specific tensor.

Instead of copying tensor data, this method directly updates the binding pointer to use an external buffer. The external buffer must be in GPU memory with the same size as the original tensor.

Parameters

tensorName	The name of the tensor to update
newBuffer	Pointer to the new buffer in GPU memory

Returns: true if successful, false otherwise

Definition at line 215 of file 9.2/sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References getTensorBindingIndex().

◆ updateBindingPointer() [2/2]

bool TensorRTEngine::updateBindingPointer	(	const std::string &	tensorName,
		void *	newBuffer
	)

inline

Updates the binding pointer for a specific tensor.

Instead of copying tensor data, this method directly updates the binding pointer to use an external buffer. The external buffer must be in GPU memory with the same size as the original tensor.

Parameters

tensorName	The name of the tensor to update
newBuffer	Pointer to the new buffer in GPU memory

Returns: true if successful, false otherwise

Definition at line 215 of file sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h.

References getTensorBindingIndex().

The documentation for this class was generated from the following file:

sources/gst-plugins/gst-nvdsvideotemplate/ds-sparse4d/infer/tensor_rt_engine.h

NVIDIA DeepStream SDK API Reference

9.2 Release

Detailed Description

Public Member Functions

Constructor & Destructor Documentation

◆ TensorRTEngine() [1/2]

◆ ~TensorRTEngine() [1/2]

◆ TensorRTEngine() [2/2]

◆ ~TensorRTEngine() [2/2]

Member Function Documentation

◆ configure() [1/2]

◆ configure() [2/2]

◆ deserializeEngine() [1/2]

◆ deserializeEngine() [2/2]

◆ getBindingDimensions() [1/2]

◆ getBindingDimensions() [2/2]

◆ getCudaStream() [1/2]

◆ getCudaStream() [2/2]

◆ getDeviceBuffer() [1/2]

◆ getDeviceBuffer() [2/2]

◆ getEngine() [1/2]

◆ getEngine() [2/2]

◆ getGpuId() [1/2]

◆ getGpuId() [2/2]

◆ getInputTensorNames() [1/2]

◆ getInputTensorNames() [2/2]

◆ getOutputTensorNames() [1/2]

◆ getOutputTensorNames() [2/2]

◆ getTensorBindingIndex() [1/2]

◆ getTensorBindingIndex() [2/2]

◆ isInitialized() [1/2]

◆ isInitialized() [2/2]

◆ runInference() [1/2]

◆ runInference() [2/2]

◆ updateBindingPointer() [1/2]

◆ updateBindingPointer() [2/2]