Detailed Description

Base class of inference backend processing.

Definition at line 40 of file infer_base_backend.h.

Inheritance diagram for nvdsinferserver::BaseBackend:

[legend]

Collaboration diagram for nvdsinferserver::BaseBackend:

Public Types
enum	{ kLTpLayerDesc, kTpLayerNum }

enum	{ kInShapeName, kInShapeDims }

using	InferenceDone = std::function< void(NvDsInferStatus, SharedBatchArray)>
	Function wrapper for post inference processing. More...

using	InputsConsumed = std::function< void(SharedBatchArray)>
	Function wrapper called after the input buffer is consumed. More...

using	LayersTuple = std::tuple< const LayerDescription *, int >
	Tuple containing pointer to layer descriptions and the number of layers. More...

using	InputShapeTuple = std::tuple< std::string, InferBatchDims >
	Tuple of layer name and dimensions including batch size. More...

using	InputShapes = std::vector< InputShapeTuple >

Public Member Functions
	~BaseBackend () override=default
	Destructor, default. More...

InferTensorOrder	getInputTensorOrder () const final
	Returns the input tensor order. More...

void	setUniqueId (uint32_t id)
	Set the unique ID for the object instance. More...

int	uniqueId () const
	Get the unique ID of the object instance. More...

void	setFirstDimBatch (bool flag)
	Set the flag indicating that it is a batch input. More...

bool	isFirstDimBatch () const final
	Returns boolean indicating if batched input is expected. More...

uint32_t	getLayerSize () const final
	Returns the total number of layers (input + output) for the model. More...

uint32_t	getInputLayerSize () const final
	Returns the number of input layers for the model. More...

const LayerDescription *	getLayerInfo (const std::string &bindingName) const final
	Retrieve the layer information from the layer name. More...

LayersTuple	getInputLayers () const final
	Get the LayersTuple for input layers. More...

LayersTuple	getOutputLayers () const final
	Get the LayersTuple for output layers. More...

bool	checkInputDims (const InputShapes &shapes) const
	Check that the list of input shapes have fixed dimensions and corresponding layers are marked as input layers. More...

const LayerDescriptionList &	allLayers () const
	Returns the list of all descriptions of all layers, input and output. More...

void	setKeepInputs (bool enable)
	Set the flag indicating whether to keep inputs buffers. More...

int32_t	maxBatchSize () const final
	Returns the maximum batch size set for the backend. More...

bool	isNonBatching () const
	Checks if the batch size indicates batched processing or no. More...

virtual NvDsInferStatus	initialize ()=0
	Initialize the backend for processing. More...

virtual NvDsInferStatus	specifyInputDims (const InputShapes &shapes)=0
	Specify the input layers for the backend. More...

virtual NvDsInferStatus	enqueue (SharedBatchArray inputs, SharedCuStream stream, InputsConsumed bufConsumed, InferenceDone inferenceDone)=0
	Enqueue an array of input batches for inference. More...

Protected Types
using	LayerIdxMap = std::unordered_map< std::string, int >
	Map of layer name to layer index. More...

Protected Member Functions
void	resetLayers (LayerDescriptionList layers, int inputSize)
	Set the layer description list of the backend. More...

LayerDescription *	mutableLayerInfo (const std::string &bindingName)
	Get the mutable layer description structure for the layer name. More...

void	setInputTensorOrder (InferTensorOrder order)
	Set the tensor order for the input layers. More...

bool	needKeepInputs () const
	Check if the keep input flag is set. More...

void	setMaxBatchSize (uint32_t size)
	Set the maximum batch size to be used for the backend. More...

Member Typedef Documentation

◆ InferenceDone

using nvdsinferserver::IBackend::InferenceDone = std::function<void(NvDsInferStatus, SharedBatchArray)>

inherited

Function wrapper for post inference processing.

Definition at line 66 of file infer_ibackend.h.

◆ InputsConsumed

using nvdsinferserver::IBackend::InputsConsumed = std::function<void(SharedBatchArray)>

inherited

Function wrapper called after the input buffer is consumed.

Definition at line 70 of file infer_ibackend.h.

◆ InputShapes

using nvdsinferserver::IBackend::InputShapes = std::vector<InputShapeTuple>

inherited

Definition at line 84 of file infer_ibackend.h.

◆ InputShapeTuple

using nvdsinferserver::IBackend::InputShapeTuple = std::tuple<std::string, InferBatchDims>

inherited

Tuple of layer name and dimensions including batch size.

Definition at line 83 of file infer_ibackend.h.

◆ LayerIdxMap

using nvdsinferserver::BaseBackend::LayerIdxMap = std::unordered_map<std::string, int>

protected

Map of layer name to layer index.

Definition at line 136 of file infer_base_backend.h.

◆ LayersTuple

using nvdsinferserver::IBackend::LayersTuple = std::tuple<const LayerDescription*, int>

inherited

Tuple containing pointer to layer descriptions and the number of layers.

Definition at line 77 of file infer_ibackend.h.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum

inherited

Enumerator
kLTpLayerDesc
kTpLayerNum

Definition at line 72 of file infer_ibackend.h.

◆ anonymous enum

anonymous enum

inherited

Enumerator
kInShapeName
kInShapeDims

Definition at line 79 of file infer_ibackend.h.

Constructor & Destructor Documentation

◆ ~BaseBackend()

nvdsinferserver::BaseBackend::~BaseBackend ( )

overridedefault

Destructor, default.

Member Function Documentation

◆ allLayers()

const LayerDescriptionList& nvdsinferserver::BaseBackend::allLayers ( ) const

inline

Returns the list of all descriptions of all layers, input and output.

Definition at line 113 of file infer_base_backend.h.

◆ checkInputDims()

bool nvdsinferserver::BaseBackend::checkInputDims ( const InputShapes & shapes ) const

Check that the list of input shapes have fixed dimensions and corresponding layers are marked as input layers.

◆ enqueue()

virtual NvDsInferStatus nvdsinferserver::IBackend::enqueue	(	SharedBatchArray	inputs,
		SharedCuStream	stream,
		InputsConsumed	bufConsumed,
		InferenceDone	inferenceDone
	)

pure virtualinherited

Enqueue an array of input batches for inference.

This function adds a input to the inference processing queue of the backend. The post inference function and function to be called after consuming input buffer is provided.

Parameters

[in]	inputs	List of input batch buffers
[in]	stream	The CUDA stream to be used in inference processing.
[in]	bufConsumed	Function to be called once input buffer is consumed.
[in]	inferenceDone	Function to be called after inference.

Returns: Execution status code.

Implemented in nvdsinferserver::TrtISBackend, nvdsinferserver::TritonGrpcBackend, and nvdsinferserver::TritonSimpleRuntime.

◆ getInputLayers()

LayersTuple nvdsinferserver::BaseBackend::getInputLayers ( ) const

finalvirtual

Get the LayersTuple for input layers.

Implements nvdsinferserver::IBackend.

◆ getInputLayerSize()

uint32_t nvdsinferserver::BaseBackend::getInputLayerSize ( ) const

inlinefinalvirtual

Returns the number of input layers for the model.

Implements nvdsinferserver::IBackend.

Definition at line 83 of file infer_base_backend.h.

◆ getInputTensorOrder()

InferTensorOrder nvdsinferserver::BaseBackend::getInputTensorOrder ( ) const

inlinefinalvirtual

Returns the input tensor order.

Implements nvdsinferserver::IBackend.

Definition at line 49 of file infer_base_backend.h.

◆ getLayerInfo()

const LayerDescription* nvdsinferserver::BaseBackend::getLayerInfo ( const std::string & bindingName ) const

finalvirtual

Retrieve the layer information from the layer name.

Implements nvdsinferserver::IBackend.

Referenced by mutableLayerInfo().

◆ getLayerSize()

uint32_t nvdsinferserver::BaseBackend::getLayerSize ( ) const

inlinefinalvirtual

Returns the total number of layers (input + output) for the model.

Implements nvdsinferserver::IBackend.

Definition at line 75 of file infer_base_backend.h.

◆ getOutputLayers()

LayersTuple nvdsinferserver::BaseBackend::getOutputLayers ( ) const

finalvirtual

Get the LayersTuple for output layers.

Implements nvdsinferserver::IBackend.

◆ initialize()

virtual NvDsInferStatus nvdsinferserver::IBackend::initialize ( )

pure virtualinherited

Initialize the backend for processing.

Returns: Status code of the type NvDsInferStatus.

Implemented in nvdsinferserver::TrtISBackend, nvdsinferserver::TritonGrpcBackend, and nvdsinferserver::TritonSimpleRuntime.

◆ isFirstDimBatch()

bool nvdsinferserver::BaseBackend::isFirstDimBatch ( ) const

inlinefinalvirtual

Returns boolean indicating if batched input is expected.

Implements nvdsinferserver::IBackend.

Definition at line 69 of file infer_base_backend.h.

◆ isNonBatching()

bool nvdsinferserver::BaseBackend::isNonBatching ( ) const

inline

Checks if the batch size indicates batched processing or no.

Definition at line 130 of file infer_base_backend.h.

References INFER_EXPORT_API::isNonBatch(), and maxBatchSize().

◆ maxBatchSize()

int32_t nvdsinferserver::BaseBackend::maxBatchSize ( ) const

inlinefinalvirtual

Returns the maximum batch size set for the backend.

Implements nvdsinferserver::IBackend.

Definition at line 125 of file infer_base_backend.h.

Referenced by isNonBatching().

◆ mutableLayerInfo()

LayerDescription* nvdsinferserver::BaseBackend::mutableLayerInfo ( const std::string & bindingName )

inlineprotected

Get the mutable layer description structure for the layer name.

Definition at line 153 of file infer_base_backend.h.

References getLayerInfo().

◆ needKeepInputs()

bool nvdsinferserver::BaseBackend::needKeepInputs ( ) const

inlineprotected

Check if the keep input flag is set.

Definition at line 167 of file infer_base_backend.h.

◆ resetLayers()

void nvdsinferserver::BaseBackend::resetLayers	(	LayerDescriptionList	layers,
		int	inputSize
	)

protected

Set the layer description list of the backend.

This function sets the layer description for the backend and updates the number of input layers, layer name to index map.

Parameters

[in]	layers	The list of descriptions for all layers, input followed by output layers.
[in]	inputSize	The number of input layers in the list.

◆ setFirstDimBatch()

void nvdsinferserver::BaseBackend::setFirstDimBatch ( bool flag )

inline

Set the flag indicating that it is a batch input.

Definition at line 64 of file infer_base_backend.h.

◆ setInputTensorOrder()

void nvdsinferserver::BaseBackend::setInputTensorOrder ( InferTensorOrder order )

inlineprotected

Set the tensor order for the input layers.

Definition at line 162 of file infer_base_backend.h.

◆ setKeepInputs()

void nvdsinferserver::BaseBackend::setKeepInputs ( bool enable )

inline

Set the flag indicating whether to keep inputs buffers.

Definition at line 118 of file infer_base_backend.h.

◆ setMaxBatchSize()

void nvdsinferserver::BaseBackend::setMaxBatchSize ( uint32_t size )

inlineprotected

Set the maximum batch size to be used for the backend.

Definition at line 174 of file infer_base_backend.h.

◆ setUniqueId()

void nvdsinferserver::BaseBackend::setUniqueId ( uint32_t id )

inline

Set the unique ID for the object instance.

Definition at line 54 of file infer_base_backend.h.

◆ specifyInputDims()

virtual NvDsInferStatus nvdsinferserver::IBackend::specifyInputDims ( const InputShapes & shapes )

pure virtualinherited

Specify the input layers for the backend.

Parameters

shapes List of name and shapes of the input layers.

Returns: Status code of the type NvDsInferStatus.

Implemented in nvdsinferserver::TrtISBackend, and nvdsinferserver::TritonSimpleRuntime.

◆ uniqueId()

int nvdsinferserver::BaseBackend::uniqueId ( ) const

inline

Get the unique ID of the object instance.

Definition at line 59 of file infer_base_backend.h.

The documentation for this class was generated from the following file:

infer_base_backend.h


NVIDIA DeepStream SDK API Reference	6.4 Release

NVIDIA DeepStream SDK API Reference

6.4 Release

Detailed Description

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Member Typedef Documentation

◆ InferenceDone

◆ InputsConsumed

◆ InputShapes

◆ InputShapeTuple

◆ LayerIdxMap

◆ LayersTuple

Member Enumeration Documentation

◆ anonymous enum

◆ anonymous enum

Constructor & Destructor Documentation

◆ ~BaseBackend()

Member Function Documentation

◆ allLayers()

◆ checkInputDims()

◆ enqueue()

◆ getInputLayers()

◆ getInputLayerSize()

◆ getInputTensorOrder()

◆ getLayerInfo()

◆ getLayerSize()

◆ getOutputLayers()

◆ initialize()

◆ isFirstDimBatch()

◆ isNonBatching()

◆ maxBatchSize()

◆ mutableLayerInfo()

◆ needKeepInputs()

◆ resetLayers()

◆ setFirstDimBatch()

◆ setInputTensorOrder()

◆ setKeepInputs()

◆ setMaxBatchSize()

◆ setUniqueId()

◆ specifyInputDims()

◆ uniqueId()