Base class of inference backend processing.
Definition at line 40 of file infer_base_backend.h.
Public Types | |
enum | { kLTpLayerDesc, kTpLayerNum } |
enum | { kInShapeName, kInShapeDims } |
using | InferenceDone = std::function< void(NvDsInferStatus, SharedBatchArray)> |
Function wrapper for post inference processing. More... | |
using | InputsConsumed = std::function< void(SharedBatchArray)> |
Function wrapper called after the input buffer is consumed. More... | |
using | LayersTuple = std::tuple< const LayerDescription *, int > |
Tuple containing pointer to layer descriptions and the number of layers. More... | |
using | InputShapeTuple = std::tuple< std::string, InferBatchDims > |
Tuple of layer name and dimensions including batch size. More... | |
using | InputShapes = std::vector< InputShapeTuple > |
Public Member Functions | |
~BaseBackend () override=default | |
Destructor, default. More... | |
InferTensorOrder | getInputTensorOrder () const final |
Returns the input tensor order. More... | |
void | setUniqueId (uint32_t id) |
Set the unique ID for the object instance. More... | |
int | uniqueId () const |
Get the unique ID of the object instance. More... | |
void | setFirstDimBatch (bool flag) |
Set the flag indicating that it is a batch input. More... | |
bool | isFirstDimBatch () const final |
Returns boolean indicating if batched input is expected. More... | |
uint32_t | getLayerSize () const final |
Returns the total number of layers (input + output) for the model. More... | |
uint32_t | getInputLayerSize () const final |
Returns the number of input layers for the model. More... | |
const LayerDescription * | getLayerInfo (const std::string &bindingName) const final |
Retrieve the layer information from the layer name. More... | |
LayersTuple | getInputLayers () const final |
Get the LayersTuple for input layers. More... | |
LayersTuple | getOutputLayers () const final |
Get the LayersTuple for output layers. More... | |
bool | checkInputDims (const InputShapes &shapes) const |
Check that the list of input shapes have fixed dimensions and corresponding layers are marked as input layers. More... | |
const LayerDescriptionList & | allLayers () const |
Returns the list of all descriptions of all layers, input and output. More... | |
void | setKeepInputs (bool enable) |
Set the flag indicating whether to keep inputs buffers. More... | |
int32_t | maxBatchSize () const final |
Returns the maximum batch size set for the backend. More... | |
bool | isNonBatching () const |
Checks if the batch size indicates batched processing or no. More... | |
virtual NvDsInferStatus | initialize ()=0 |
Initialize the backend for processing. More... | |
virtual NvDsInferStatus | specifyInputDims (const InputShapes &shapes)=0 |
Specify the input layers for the backend. More... | |
virtual NvDsInferStatus | enqueue (SharedBatchArray inputs, SharedCuStream stream, InputsConsumed bufConsumed, InferenceDone inferenceDone)=0 |
Enqueue an array of input batches for inference. More... | |
Protected Types | |
using | LayerIdxMap = std::unordered_map< std::string, int > |
Map of layer name to layer index. More... | |
Protected Member Functions | |
void | resetLayers (LayerDescriptionList layers, int inputSize) |
Set the layer description list of the backend. More... | |
LayerDescription * | mutableLayerInfo (const std::string &bindingName) |
Get the mutable layer description structure for the layer name. More... | |
void | setInputTensorOrder (InferTensorOrder order) |
Set the tensor order for the input layers. More... | |
bool | needKeepInputs () const |
Check if the keep input flag is set. More... | |
void | setMaxBatchSize (uint32_t size) |
Set the maximum batch size to be used for the backend. More... | |
|
inherited |
Function wrapper for post inference processing.
Definition at line 66 of file infer_ibackend.h.
|
inherited |
Function wrapper called after the input buffer is consumed.
Definition at line 70 of file infer_ibackend.h.
|
inherited |
Definition at line 84 of file infer_ibackend.h.
|
inherited |
Tuple of layer name and dimensions including batch size.
Definition at line 83 of file infer_ibackend.h.
|
protected |
Map of layer name to layer index.
Definition at line 136 of file infer_base_backend.h.
|
inherited |
Tuple containing pointer to layer descriptions and the number of layers.
Definition at line 77 of file infer_ibackend.h.
|
inherited |
Enumerator | |
---|---|
kLTpLayerDesc | |
kTpLayerNum |
Definition at line 72 of file infer_ibackend.h.
|
inherited |
Enumerator | |
---|---|
kInShapeName | |
kInShapeDims |
Definition at line 79 of file infer_ibackend.h.
|
overridedefault |
Destructor, default.
|
inline |
Returns the list of all descriptions of all layers, input and output.
Definition at line 113 of file infer_base_backend.h.
bool nvdsinferserver::BaseBackend::checkInputDims | ( | const InputShapes & | shapes | ) | const |
Check that the list of input shapes have fixed dimensions and corresponding layers are marked as input layers.
|
pure virtualinherited |
Enqueue an array of input batches for inference.
This function adds a input to the inference processing queue of the backend. The post inference function and function to be called after consuming input buffer is provided.
[in] | inputs | List of input batch buffers |
[in] | stream | The CUDA stream to be used in inference processing. |
[in] | bufConsumed | Function to be called once input buffer is consumed. |
[in] | inferenceDone | Function to be called after inference. |
Implemented in nvdsinferserver::TrtISBackend, nvdsinferserver::TritonGrpcBackend, and nvdsinferserver::TritonSimpleRuntime.
|
finalvirtual |
Get the LayersTuple for input layers.
Implements nvdsinferserver::IBackend.
|
inlinefinalvirtual |
Returns the number of input layers for the model.
Implements nvdsinferserver::IBackend.
Definition at line 83 of file infer_base_backend.h.
|
inlinefinalvirtual |
Returns the input tensor order.
Implements nvdsinferserver::IBackend.
Definition at line 49 of file infer_base_backend.h.
|
finalvirtual |
Retrieve the layer information from the layer name.
Implements nvdsinferserver::IBackend.
Referenced by mutableLayerInfo().
|
inlinefinalvirtual |
Returns the total number of layers (input + output) for the model.
Implements nvdsinferserver::IBackend.
Definition at line 75 of file infer_base_backend.h.
|
finalvirtual |
Get the LayersTuple for output layers.
Implements nvdsinferserver::IBackend.
|
pure virtualinherited |
Initialize the backend for processing.
Implemented in nvdsinferserver::TrtISBackend, nvdsinferserver::TritonGrpcBackend, and nvdsinferserver::TritonSimpleRuntime.
|
inlinefinalvirtual |
Returns boolean indicating if batched input is expected.
Implements nvdsinferserver::IBackend.
Definition at line 69 of file infer_base_backend.h.
|
inline |
Checks if the batch size indicates batched processing or no.
Definition at line 130 of file infer_base_backend.h.
References INFER_EXPORT_API::isNonBatch(), and maxBatchSize().
|
inlinefinalvirtual |
Returns the maximum batch size set for the backend.
Implements nvdsinferserver::IBackend.
Definition at line 125 of file infer_base_backend.h.
Referenced by isNonBatching().
|
inlineprotected |
Get the mutable layer description structure for the layer name.
Definition at line 153 of file infer_base_backend.h.
References getLayerInfo().
|
inlineprotected |
Check if the keep input flag is set.
Definition at line 167 of file infer_base_backend.h.
|
protected |
Set the layer description list of the backend.
This function sets the layer description for the backend and updates the number of input layers, layer name to index map.
[in] | layers | The list of descriptions for all layers, input followed by output layers. |
[in] | inputSize | The number of input layers in the list. |
|
inline |
Set the flag indicating that it is a batch input.
Definition at line 64 of file infer_base_backend.h.
|
inlineprotected |
Set the tensor order for the input layers.
Definition at line 162 of file infer_base_backend.h.
|
inline |
Set the flag indicating whether to keep inputs buffers.
Definition at line 118 of file infer_base_backend.h.
|
inlineprotected |
Set the maximum batch size to be used for the backend.
Definition at line 174 of file infer_base_backend.h.
|
inline |
Set the unique ID for the object instance.
Definition at line 54 of file infer_base_backend.h.
|
pure virtualinherited |
Specify the input layers for the backend.
shapes | List of name and shapes of the input layers. |
Implemented in nvdsinferserver::TrtISBackend, and nvdsinferserver::TritonSimpleRuntime.
|
inline |
Get the unique ID of the object instance.
Definition at line 59 of file infer_base_backend.h.