nvdsinferserver::InferTrtISContext Class Reference

Detailed Description

Inference context accessing Triton Inference Server in C-API mode.

Definition at line 33 of file infer_trtis_context.h.

Inheritance diagram for nvdsinferserver::InferTrtISContext:

Collaboration diagram for nvdsinferserver::InferTrtISContext:

Public Member Functions
	InferTrtISContext ()
	Constructor, default. More...
	~InferTrtISContext () override
	Destructor, default. More...
NvDsInferStatus	deinit () override
	Synchronize on the CUDA stream and call InferCudaContext::deinit(). More...
SharedCuStream &	mainStream () override
	Get the main processing CUDA event. More...
SharedSysMem	acquireTensorHostBuf (const std::string &name, size_t bytes)
	Allocator. More...
SharedCuEvent	acquireTensorHostEvent ()
	Acquire a CUDA event from the events pool. More...
NvDsInferStatus	initialize (const std::string &prototxt, InferLoggingFunc logFunc) final
NvDsInferStatus	run (SharedIBatchArray input, InferOutputCb outputCb) final

Protected Member Functions
NvDsInferStatus	fixateInferenceInfo (const ic::InferenceConfig &config, BaseBackend &backend) override
	Check the tensor order, media format, and datatype for the input tensor. More...
NvDsInferStatus	createPreprocessor (const ic::PreProcessParams &params, std::vector< UniqPreprocessor > &processors) override
	Create the surface converter and network preprocessor. More...
NvDsInferStatus	createPostprocessor (const ic::PostProcessParams &params, UniqPostprocessor &processor) override
	Create the post-processor as per the network output type. More...
NvDsInferStatus	preInference (SharedBatchArray &inputs, const ic::InferenceConfig &config) override
	Initialize non-image input layers if the custom library has implemented the interface. More...
NvDsInferStatus	extraOutputTensorCheck (SharedBatchArray &outputs, SharedOptions inOptions) override
	Post inference steps for the custom processor and LSTM controller. More...
void	notifyError (NvDsInferStatus status) override
	In case of error, notify the waiting threads. More...
void	getNetworkInputInfo (NvDsInferNetworkInfo &networkInfo) override
	Get the network input layer information. More...
int	tensorPoolSize () const
	Get the size of the tensor pool. More...
virtual void	backendConsumedInputs (SharedBatchArray inputs)
const ic::InferenceConfig &	config () const
int	maxBatchSize () const
int	uniqueId () const
BaseBackend *	backend ()
const SharedDllHandle &	customLib () const
bool	needCopyInputToHost () const
void	print (NvDsInferLogLevel l, const char *msg)
bool	needPreprocess () const

Protected Attributes
NvDsInferNetworkInfo	m_NetworkImageInfo {0, 0, 0}
	Network input height, width, channels for preprocessing. More...
InferMediaFormat	m_NetworkImageFormat = InferMediaFormat::kRGB
	The input layer media format. More...
std::string	m_NetworkImageName
	The input layer name. More...
InferTensorOrder	m_InputTensorOrder = InferTensorOrder::kNone
	The input layer tensor order. More...
InferDataType	m_InputDataType = InferDataType::kFp32
	The input layer datatype. More...
std::vector< SharedCudaTensorBuf >	m_ExtraInputs
	Array of buffers of the additional inputs. More...
MapBufferPool< std::string, UniqSysMem >	m_HostTensorPool
	Map of pools for the output tensors. More...
SharedBufPool< std::unique_ptr< CudaEventInPool > >	m_HostTensorEvents
	Pool of CUDA events for host tensor copy. More...
UniqLstmController	m_LstmController
	LSTM controller. More...
UniqStreamManager	m_MultiStreamManager
	stream-id based management. More...
UniqInferExtraProcessor	m_ExtraProcessor
	Extra and custom processing pre/post inference. More...
CropSurfaceConverter *	m_SurfaceConverter = nullptr
	Preprocessor and post-processor handles. More...
NetworkPreprocessor *	m_NetworkPreprocessor = nullptr
Postprocessor *	m_FinalProcessor = nullptr
CropSurfaceConverter *	m_SurfaceConverter = nullptr
	Preprocessor and post-processor handles. More...
NetworkPreprocessor *	m_NetworkPreprocessor = nullptr
Postprocessor *	m_FinalProcessor = nullptr
CropSurfaceConverter *	m_SurfaceConverter = nullptr
	Preprocessor and post-processor handles. More...
NetworkPreprocessor *	m_NetworkPreprocessor = nullptr
Postprocessor *	m_FinalProcessor = nullptr

Constructor & Destructor Documentation

InferTrtISContext()

nvdsinferserver::InferTrtISContext::InferTrtISContext

(

)

Constructor, default.

~InferTrtISContext()

nvdsinferserver::InferTrtISContext::~InferTrtISContext ( )

override

Destructor, default.

Member Function Documentation

acquireTensorHostBuf()

SharedSysMem nvdsinferserver::InferCudaContext::acquireTensorHostBuf ( const std::string &  name, size_t  bytes  )

inherited

Allocator.

Acquire a host buffer for the inference output.

Parameters

[in]	name	Name of the output layer.
[in]	bytes	Size of the buffer.

Returns

Pointer to the memory buffer.

acquireTensorHostEvent()

SharedCuEvent nvdsinferserver::InferCudaContext::acquireTensorHostEvent ( )

inherited

Acquire a CUDA event from the events pool.

backend()

BaseBackend* nvdsinferserver::InferBaseContext::backend ( )

inlineprotectedinherited

Definition at line 101 of file infer_base_context.h.

backendConsumedInputs()

virtual void nvdsinferserver::InferBaseContext::backendConsumedInputs ( SharedBatchArray  inputs )

inlineprotectedvirtualinherited

Definition at line 93 of file infer_base_context.h.

config()

const ic::InferenceConfig& nvdsinferserver::InferBaseContext::config ( ) const

inlineprotectedinherited

Definition at line 98 of file infer_base_context.h.

createPostprocessor()

NvDsInferStatus nvdsinferserver::InferCudaContext::createPostprocessor ( const ic::PostProcessParams &  params, UniqPostprocessor &  processor  )

overrideprotectedvirtualinherited

Create the post-processor as per the network output type.

Parameters

[in]	params	The post processing configuration parameters.
[out]	processor	The handle to the created post processor.

Returns

Status code.

Implements nvdsinferserver::InferBaseContext.

createPreprocessor()

NvDsInferStatus nvdsinferserver::InferCudaContext::createPreprocessor ( const ic::PreProcessParams &  params, std::vector< UniqPreprocessor > &  processors  )

overrideprotectedvirtualinherited

Create the surface converter and network preprocessor.

Parameters

params	The preprocessor configuration.
processors	List of the created preprocessor handles.

Returns

Status code.

Implements nvdsinferserver::InferBaseContext.

customLib()

const SharedDllHandle& nvdsinferserver::InferBaseContext::customLib ( ) const

inlineprotectedinherited

Definition at line 102 of file infer_base_context.h.

deinit()

NvDsInferStatus nvdsinferserver::InferTrtISContext::deinit ( )

override

Synchronize on the CUDA stream and call InferCudaContext::deinit().

extraOutputTensorCheck()

NvDsInferStatus nvdsinferserver::InferCudaContext::extraOutputTensorCheck ( SharedBatchArray &  outputs, SharedOptions  inOptions  )

overrideprotectedvirtualinherited

Post inference steps for the custom processor and LSTM controller.

Parameters

[in,out]	outputs	The output batch buffers array.
[in]	inOptions	The configuration options for the buffers.

Returns

Status code.

Reimplemented from nvdsinferserver::InferBaseContext.

fixateInferenceInfo()

NvDsInferStatus nvdsinferserver::InferCudaContext::fixateInferenceInfo ( const ic::InferenceConfig &  config, BaseBackend &  backend  )

overrideprotectedvirtualinherited

Check the tensor order, media format, and datatype for the input tensor.

Initiate the extra processor and lstm controller if configured.

Parameters

[in]	config	The inference configuration protobuf message.
[in]	backend	The inference backend instance.

Returns

Status code.

Implements nvdsinferserver::InferBaseContext.

getNetworkInputInfo()

void nvdsinferserver::InferCudaContext::getNetworkInputInfo ( NvDsInferNetworkInfo &  networkInfo )

inlineoverrideprotectedinherited

Get the network input layer information.

Definition at line 132 of file infer_cuda_context.h.

References nvdsinferserver::InferCudaContext::m_NetworkImageInfo.

initialize()

NvDsInferStatus nvdsinferserver::InferBaseContext::initialize ( const std::string &  prototxt, InferLoggingFunc  logFunc  )

finalinherited

mainStream()

SharedCuStream& nvdsinferserver::InferTrtISContext::mainStream ( )

inlineoverridevirtual

Get the main processing CUDA event.

Implements nvdsinferserver::InferBaseContext.

Definition at line 53 of file infer_trtis_context.h.

maxBatchSize()

int nvdsinferserver::InferBaseContext::maxBatchSize ( ) const

inlineprotectedinherited

Definition at line 99 of file infer_base_context.h.

needCopyInputToHost()

bool nvdsinferserver::InferBaseContext::needCopyInputToHost ( ) const

protectedinherited

needPreprocess()

bool nvdsinferserver::InferBaseContext::needPreprocess ( ) const

protectedinherited

notifyError()

void nvdsinferserver::InferCudaContext::notifyError ( NvDsInferStatus  status )

overrideprotectedvirtualinherited

In case of error, notify the waiting threads.

Implements nvdsinferserver::InferBaseContext.

preInference()

NvDsInferStatus nvdsinferserver::InferCudaContext::preInference ( SharedBatchArray &  inputs, const ic::InferenceConfig &  config  )

overrideprotectedvirtualinherited

Initialize non-image input layers if the custom library has implemented the interface.

Parameters

[in,out]	inputs	Array of the input batch buffers.
[in]	config	The inference configuration settings.

Returns

Status code.

Reimplemented from nvdsinferserver::InferBaseContext.

print()

void nvdsinferserver::InferBaseContext::print ( NvDsInferLogLevel  l, const char *  msg  )

protectedinherited

run()

NvDsInferStatus nvdsinferserver::InferBaseContext::run ( SharedIBatchArray  input, InferOutputCb  outputCb  )

finalinherited

tensorPoolSize()

int nvdsinferserver::InferCudaContext::tensorPoolSize ( ) const

protectedinherited

Get the size of the tensor pool.

uniqueId()

int nvdsinferserver::InferBaseContext::uniqueId ( ) const

inlineprotectedinherited

Definition at line 100 of file infer_base_context.h.

Field Documentation

m_ExtraInputs

std::vector<SharedCudaTensorBuf> nvdsinferserver::InferCudaContext::m_ExtraInputs

protectedinherited

Array of buffers of the additional inputs.

Definition at line 213 of file infer_cuda_context.h.

m_ExtraProcessor

UniqInferExtraProcessor nvdsinferserver::InferCudaContext::m_ExtraProcessor

protectedinherited

Extra and custom processing pre/post inference.

Definition at line 228 of file infer_cuda_context.h.

m_FinalProcessor

Postprocessor* nvdsinferserver::InferCudaContext::m_FinalProcessor = nullptr

protectedinherited

Definition at line 234 of file infer_cuda_context.h.

m_HostTensorEvents

SharedBufPool<std::unique_ptr<CudaEventInPool> > nvdsinferserver::InferCudaContext::m_HostTensorEvents

protectedinherited

Pool of CUDA events for host tensor copy.

Definition at line 221 of file infer_cuda_context.h.

m_HostTensorPool

MapBufferPool<std::string, UniqSysMem> nvdsinferserver::InferCudaContext::m_HostTensorPool

protectedinherited

Map of pools for the output tensors.

Definition at line 217 of file infer_cuda_context.h.

m_InputDataType

InferDataType nvdsinferserver::InferCudaContext::m_InputDataType = InferDataType::kFp32

protectedinherited

The input layer datatype.

Definition at line 208 of file infer_cuda_context.h.

m_InputTensorOrder

InferTensorOrder nvdsinferserver::InferCudaContext::m_InputTensorOrder = InferTensorOrder::kNone

protectedinherited

The input layer tensor order.

Definition at line 204 of file infer_cuda_context.h.

m_LstmController

UniqLstmController nvdsinferserver::InferCudaContext::m_LstmController

protectedinherited

LSTM controller.

Definition at line 224 of file infer_cuda_context.h.

m_MultiStreamManager

UniqStreamManager nvdsinferserver::InferCudaContext::m_MultiStreamManager

protectedinherited

stream-id based management.

Definition at line 226 of file infer_cuda_context.h.

m_NetworkImageFormat

InferMediaFormat nvdsinferserver::InferCudaContext::m_NetworkImageFormat = InferMediaFormat::kRGB

protectedinherited

The input layer media format.

Definition at line 196 of file infer_cuda_context.h.

m_NetworkImageInfo

NvDsInferNetworkInfo nvdsinferserver::InferCudaContext::m_NetworkImageInfo {0, 0, 0}

protectedinherited

Network input height, width, channels for preprocessing.

Definition at line 192 of file infer_cuda_context.h.

Referenced by nvdsinferserver::InferCudaContext::getNetworkInputInfo().

m_NetworkImageName

std::string nvdsinferserver::InferCudaContext::m_NetworkImageName

protectedinherited

The input layer name.

Definition at line 200 of file infer_cuda_context.h.

m_NetworkPreprocessor

NetworkPreprocessor* nvdsinferserver::InferCudaContext::m_NetworkPreprocessor = nullptr

protectedinherited

Definition at line 233 of file infer_cuda_context.h.

m_SurfaceConverter

CropSurfaceConverter* nvdsinferserver::InferCudaContext::m_SurfaceConverter = nullptr

protectedinherited

Preprocessor and post-processor handles.

Definition at line 232 of file infer_cuda_context.h.

---

The documentation for this class was generated from the following file:

• infer_trtis_context.h