Detailed Description

Class for inference context CUDA processing.

Handles preprocessing, post-processing, extra input processing and LSTM related processing.

Definition at line 44 of file infer_cuda_context.h.

Inheritance diagram for nvdsinferserver::InferCudaContext:

[legend]

Collaboration diagram for nvdsinferserver::InferCudaContext:

[legend]

Public Member Functions
	InferCudaContext ()
	Constructor. More...

	~InferCudaContext () override
	Destructor. More...

SharedSysMem	acquireTensorHostBuf (const std::string &name, size_t bytes)
	Allocator. More...

SharedCuEvent	acquireTensorHostEvent ()
	Acquire a CUDA event from the events pool. More...

NvDsInferStatus	initialize (const std::string &prototxt, InferLoggingFunc logFunc) final

NvDsInferStatus	run (SharedIBatchArray input, InferOutputCb outputCb) final

Protected Member Functions
NvDsInferStatus	fixateInferenceInfo (const ic::InferenceConfig &config, BaseBackend &backend) override
	Check the tensor order, media format, and datatype for the input tensor. More...

NvDsInferStatus	createPreprocessor (const ic::PreProcessParams &params, std::vector< UniqPreprocessor > &processors) override
	Create the surface converter and network preprocessor. More...

NvDsInferStatus	createPostprocessor (const ic::PostProcessParams &params, UniqPostprocessor &processor) override
	Create the post-processor as per the network output type. More...

NvDsInferStatus	allocateResource (const ic::InferenceConfig &config) override
	Allocate resources for the preprocessors and post-processor. More...

NvDsInferStatus	preInference (SharedBatchArray &inputs, const ic::InferenceConfig &config) override
	Initialize non-image input layers if the custom library has implemented the interface. More...

NvDsInferStatus	extraOutputTensorCheck (SharedBatchArray &outputs, SharedOptions inOptions) override
	Post inference steps for the custom processor and LSTM controller. More...

void	notifyError (NvDsInferStatus status) override
	In case of error, notify the waiting threads. More...

NvDsInferStatus	deinit () override
	Release the host tensor pool buffers, extra input buffers, LSTM controller, extra input processor. More...

void	getNetworkInputInfo (NvDsInferNetworkInfo &networkInfo) override
	Get the network input layer information. More...

int	tensorPoolSize () const
	Get the size of the tensor pool. More...

virtual void	backendConsumedInputs (SharedBatchArray inputs)

virtual SharedCuStream &	mainStream ()=0

const ic::InferenceConfig &	config () const

int	maxBatchSize () const

int	uniqueId () const

BaseBackend *	backend ()

const SharedDllHandle &	customLib () const

bool	needCopyInputToHost () const

void	print (NvDsInferLogLevel l, const char *msg)

bool	needPreprocess () const

Protected Attributes
NvDsInferNetworkInfo	m_NetworkImageInfo {0, 0, 0}
	Network input height, width, channels for preprocessing. More...

InferMediaFormat	m_NetworkImageFormat = InferMediaFormat::kRGB
	The input layer media format. More...

std::string	m_NetworkImageName
	The input layer name. More...

InferTensorOrder	m_InputTensorOrder = InferTensorOrder::kNone
	The input layer tensor order. More...

InferDataType	m_InputDataType = InferDataType::kFp32
	The input layer datatype. More...

std::vector< SharedCudaTensorBuf >	m_ExtraInputs
	Array of buffers of the additional inputs. More...

MapBufferPool< std::string, UniqSysMem >	m_HostTensorPool
	Map of pools for the output tensors. More...

SharedBufPool< std::unique_ptr< CudaEventInPool > >	m_HostTensorEvents
	Pool of CUDA events for host tensor copy. More...

UniqLstmController	m_LstmController
	LSTM controller. More...

UniqStreamManager	m_MultiStreamManager
	stream-id based management. More...

UniqInferExtraProcessor	m_ExtraProcessor
	Extra and custom processing pre/post inference. More...

Constructor & Destructor Documentation

◆ InferCudaContext()

nvdsinferserver::InferCudaContext::InferCudaContext ( )

Constructor.

Instantiate the host tensor pool.

◆ ~InferCudaContext()

nvdsinferserver::InferCudaContext::~InferCudaContext ( )

override

Destructor.

Clear the extra input list and host tensor pool.

Member Function Documentation

◆ acquireTensorHostBuf()

SharedSysMem nvdsinferserver::InferCudaContext::acquireTensorHostBuf	(	const std::string &	name,
		size_t	bytes
	)

Allocator.

Acquire a host buffer for the inference output.

Parameters

[in]	name	Name of the output layer.
[in]	bytes	Size of the buffer.

Returns: Pointer to the memory buffer.

◆ acquireTensorHostEvent()

SharedCuEvent nvdsinferserver::InferCudaContext::acquireTensorHostEvent ( )

Acquire a CUDA event from the events pool.

◆ allocateResource()

NvDsInferStatus nvdsinferserver::InferCudaContext::allocateResource ( const ic::InferenceConfig & config )

overrideprotectedvirtual

Allocate resources for the preprocessors and post-processor.

Allocate the host tensor pool buffers.

Parameters

[in] config The inference configuration settings.

Returns: Status code.

Implements nvdsinferserver::InferBaseContext.

◆ backend()

BaseBackend* nvdsinferserver::InferBaseContext::backend ( )

inlineprotectedinherited

Definition at line 101 of file infer_base_context.h.

◆ backendConsumedInputs()

virtual void nvdsinferserver::InferBaseContext::backendConsumedInputs ( SharedBatchArray inputs )

inlineprotectedvirtualinherited

Definition at line 93 of file infer_base_context.h.

◆ config()

const ic::InferenceConfig& nvdsinferserver::InferBaseContext::config ( ) const

inlineprotectedinherited

Definition at line 98 of file infer_base_context.h.

◆ createPostprocessor()

NvDsInferStatus nvdsinferserver::InferCudaContext::createPostprocessor	(	const ic::PostProcessParams &	params,
		UniqPostprocessor &	processor
	)

overrideprotectedvirtual

Create the post-processor as per the network output type.

Parameters

[in]	params	The post processing configuration parameters.
[out]	processor	The handle to the created post processor.

Returns: Status code.

Implements nvdsinferserver::InferBaseContext.

◆ createPreprocessor()

NvDsInferStatus nvdsinferserver::InferCudaContext::createPreprocessor	(	const ic::PreProcessParams &	params,
		std::vector< UniqPreprocessor > &	processors
	)

overrideprotectedvirtual

Create the surface converter and network preprocessor.

Parameters

params	The preprocessor configuration.
processors	List of the created preprocessor handles.

Returns: Status code.

Implements nvdsinferserver::InferBaseContext.

◆ customLib()

const SharedDllHandle& nvdsinferserver::InferBaseContext::customLib ( ) const

inlineprotectedinherited

Definition at line 102 of file infer_base_context.h.

◆ deinit()

NvDsInferStatus nvdsinferserver::InferCudaContext::deinit ( )

overrideprotected

Release the host tensor pool buffers, extra input buffers, LSTM controller, extra input processor.

De-initialize the context.

Returns: Status code.

◆ extraOutputTensorCheck()

NvDsInferStatus nvdsinferserver::InferCudaContext::extraOutputTensorCheck	(	SharedBatchArray &	outputs,
		SharedOptions	inOptions
	)

overrideprotectedvirtual

Post inference steps for the custom processor and LSTM controller.

Parameters

[in,out]	outputs	The output batch buffers array.
[in]	inOptions	The configuration options for the buffers.

Returns: Status code.

Reimplemented from nvdsinferserver::InferBaseContext.

◆ fixateInferenceInfo()

NvDsInferStatus nvdsinferserver::InferCudaContext::fixateInferenceInfo	(	const ic::InferenceConfig &	config,
		BaseBackend &	backend
	)

overrideprotectedvirtual

Check the tensor order, media format, and datatype for the input tensor.

Initiate the extra processor and lstm controller if configured.

Parameters

[in]	config	The inference configuration protobuf message.
[in]	backend	The inference backend instance.

Returns: Status code.

Implements nvdsinferserver::InferBaseContext.

◆ getNetworkInputInfo()

void nvdsinferserver::InferCudaContext::getNetworkInputInfo ( NvDsInferNetworkInfo & networkInfo )

inlineoverrideprotected

Get the network input layer information.

Definition at line 132 of file infer_cuda_context.h.

References m_NetworkImageInfo.

◆ initialize()

NvDsInferStatus nvdsinferserver::InferBaseContext::initialize	(	const std::string &	prototxt,
		InferLoggingFunc	logFunc
	)

finalinherited

◆ mainStream()

virtual SharedCuStream& nvdsinferserver::InferBaseContext::mainStream ( )

protectedpure virtualinherited

Implemented in nvdsinferserver::InferGrpcContext, and nvdsinferserver::InferTrtISContext.

◆ maxBatchSize()

int nvdsinferserver::InferBaseContext::maxBatchSize ( ) const

inlineprotectedinherited

Definition at line 99 of file infer_base_context.h.

◆ needCopyInputToHost()

bool nvdsinferserver::InferBaseContext::needCopyInputToHost ( ) const

protectedinherited

◆ needPreprocess()

bool nvdsinferserver::InferBaseContext::needPreprocess ( ) const

protectedinherited

◆ notifyError()

void nvdsinferserver::InferCudaContext::notifyError ( NvDsInferStatus status )

overrideprotectedvirtual

In case of error, notify the waiting threads.

Implements nvdsinferserver::InferBaseContext.

◆ preInference()

NvDsInferStatus nvdsinferserver::InferCudaContext::preInference	(	SharedBatchArray &	inputs,
		const ic::InferenceConfig &	config
	)

overrideprotectedvirtual

Initialize non-image input layers if the custom library has implemented the interface.

Parameters

[in,out]	inputs	Array of the input batch buffers.
[in]	config	The inference configuration settings.

Returns: Status code.

Reimplemented from nvdsinferserver::InferBaseContext.

◆ print()

void nvdsinferserver::InferBaseContext::print	(	NvDsInferLogLevel	l,
		const char *	msg
	)

protectedinherited

◆ run()

NvDsInferStatus nvdsinferserver::InferBaseContext::run	(	SharedIBatchArray	input,
		InferOutputCb	outputCb
	)

finalinherited

◆ tensorPoolSize()

int nvdsinferserver::InferCudaContext::tensorPoolSize ( ) const

protected

Get the size of the tensor pool.

◆ uniqueId()

int nvdsinferserver::InferBaseContext::uniqueId ( ) const

inlineprotectedinherited

Definition at line 100 of file infer_base_context.h.

Field Documentation

◆ m_ExtraInputs

std::vector<SharedCudaTensorBuf> nvdsinferserver::InferCudaContext::m_ExtraInputs

protected

Array of buffers of the additional inputs.

Definition at line 213 of file infer_cuda_context.h.

◆ m_ExtraProcessor

UniqInferExtraProcessor nvdsinferserver::InferCudaContext::m_ExtraProcessor

protected

Extra and custom processing pre/post inference.

Definition at line 228 of file infer_cuda_context.h.

◆ m_FinalProcessor

Postprocessor* nvdsinferserver::InferCudaContext::m_FinalProcessor = nullptr

protected

Definition at line 234 of file infer_cuda_context.h.

◆ m_HostTensorEvents

SharedBufPool<std::unique_ptr<CudaEventInPool> > nvdsinferserver::InferCudaContext::m_HostTensorEvents

protected

Pool of CUDA events for host tensor copy.

Definition at line 221 of file infer_cuda_context.h.

◆ m_HostTensorPool

MapBufferPool<std::string, UniqSysMem> nvdsinferserver::InferCudaContext::m_HostTensorPool

protected

Map of pools for the output tensors.

Definition at line 217 of file infer_cuda_context.h.

◆ m_InputDataType

InferDataType nvdsinferserver::InferCudaContext::m_InputDataType = InferDataType::kFp32

protected

The input layer datatype.

Definition at line 208 of file infer_cuda_context.h.

◆ m_InputTensorOrder

InferTensorOrder nvdsinferserver::InferCudaContext::m_InputTensorOrder = InferTensorOrder::kNone

protected

The input layer tensor order.

Definition at line 204 of file infer_cuda_context.h.

◆ m_LstmController

UniqLstmController nvdsinferserver::InferCudaContext::m_LstmController

protected

LSTM controller.

Definition at line 224 of file infer_cuda_context.h.

◆ m_MultiStreamManager

UniqStreamManager nvdsinferserver::InferCudaContext::m_MultiStreamManager

protected

stream-id based management.

Definition at line 226 of file infer_cuda_context.h.

◆ m_NetworkImageFormat

InferMediaFormat nvdsinferserver::InferCudaContext::m_NetworkImageFormat = InferMediaFormat::kRGB

protected

The input layer media format.

Definition at line 196 of file infer_cuda_context.h.

◆ m_NetworkImageInfo

NvDsInferNetworkInfo nvdsinferserver::InferCudaContext::m_NetworkImageInfo {0, 0, 0}

protected

Network input height, width, channels for preprocessing.

Definition at line 192 of file infer_cuda_context.h.

Referenced by getNetworkInputInfo().

◆ m_NetworkImageName

std::string nvdsinferserver::InferCudaContext::m_NetworkImageName

protected

The input layer name.

Definition at line 200 of file infer_cuda_context.h.

◆ m_NetworkPreprocessor

NetworkPreprocessor* nvdsinferserver::InferCudaContext::m_NetworkPreprocessor = nullptr

protected

Definition at line 233 of file infer_cuda_context.h.

◆ m_SurfaceConverter

CropSurfaceConverter* nvdsinferserver::InferCudaContext::m_SurfaceConverter = nullptr

protected

Preprocessor and post-processor handles.

Definition at line 232 of file infer_cuda_context.h.

The documentation for this class was generated from the following file:

infer_cuda_context.h


NVIDIA DeepStream SDK API Reference	6.4 Release

NVIDIA DeepStream SDK API Reference

6.4 Release

Detailed Description

Public Member Functions

Protected Member Functions

Protected Attributes

Constructor & Destructor Documentation

◆ InferCudaContext()

◆ ~InferCudaContext()

Member Function Documentation

◆ acquireTensorHostBuf()

◆ acquireTensorHostEvent()

◆ allocateResource()

◆ backend()

◆ backendConsumedInputs()

◆ config()

◆ createPostprocessor()

◆ createPreprocessor()

◆ customLib()

◆ deinit()

◆ extraOutputTensorCheck()

◆ fixateInferenceInfo()

◆ getNetworkInputInfo()

◆ initialize()

◆ mainStream()

◆ maxBatchSize()

◆ needCopyInputToHost()

◆ needPreprocess()

◆ notifyError()

◆ preInference()

◆ print()

◆ run()

◆ tensorPoolSize()

◆ uniqueId()

Field Documentation

◆ m_ExtraInputs

◆ m_ExtraProcessor

◆ m_FinalProcessor

◆ m_HostTensorEvents

◆ m_HostTensorPool

◆ m_InputDataType

◆ m_InputTensorOrder

◆ m_LstmController

◆ m_MultiStreamManager

◆ m_NetworkImageFormat

◆ m_NetworkImageInfo

◆ m_NetworkImageName

◆ m_NetworkPreprocessor

◆ m_SurfaceConverter