Detailed Description

This is a header file for pre-processing cuda kernels with normalization and mean subtraction required by nvdsinfer.

Namespaces
	triton

Data Structures
class	BaseBackend
	Base class of inference backend processing. More...

class	BaseBatchArray
	The base class for array of batch buffers. More...

class	BaseBatchBuffer
	The base class for batch buffers. More...

class	BasePostprocessor
	Base post-processor class. More...

class	BasePreprocessor
	Base preprocessor class. More...

class	BatchSurfaceBuffer

class	BufOptions

class	ClassificationOutput

class	ClassifyPostprocessor
	Post processor class for classification output. More...

class	CpuMem
	Allocates and manages host memory. More...

class	CropSurfaceConverter
	Preprocessor for cropping, scaling and padding the inference input to required height, width. More...

class	CudaDeviceMem
	Allocates and manages CUDA device memory. More...

class	CudaEvent
	Wrapper class for CUDA events. More...

class	CudaHostMem
	Allocates and manages CUDA pinned memory. More...

class	CudaStream
	Wrapper class for CUDA streams. More...

class	CudaTensorBuf
	A batch buffer with CUDA memory allocation. More...

class	DetectionOutput

class	DetectPostprocessor
	Post processor class for detection output. More...

class	IBackend

class	IBatchArray
	Interface class for an array of batch buffers. More...

class	IBatchBuffer
	Interface class for a batch buffer. More...

class	IInferCustomProcessor
	Interface of Custom processor which is created and loaded at runtime through CreateCustomProcessorFunc. More...

class	ImageAlignBuffer

class	InferBaseContext
	The base class for handling the inference context. More...

struct	InferBatchDims
	Holds full dimensions (including batch size) for a layer. More...

struct	InferBufferDescription
	Holds the information about a inference buffer. More...

class	InferCudaContext
	Class for inference context CUDA processing. More...

struct	InferDims
	Holds the information about the dimensions of a neural network layer. More...

class	InferExtraProcessor
	: Extra processing pre/post inference. More...

class	InferGrpcClient
	Wrapper class for the gRPC client of the Triton Inference Server, interfaces with the Triton client library. More...

class	InferGrpcContext
	Inference context class accessing Triton Inference Server in gRPC mode. More...

class	InferSimpleContext

class	InferTrtISContext
	Inference context accessing Triton Inference Server in C-API mode. More...

class	IOptions

class	IPostprocessor
	Post-processor interface class. More...

class	IPreprocessor
	Preprocessor interface class. More...

struct	LayerDescription
	Stores the information of a layer in the inference model. More...

class	LstmController

class	NetworkPreprocessor
	Preprocessor for scaling and normalization of the input and conversion to network media format. More...

class	OtherPostprocessor
	Post processor class for tensor output for custom post processing. More...

class	Postprocessor
	A generic post processor class. More...

class	RefBatchBuffer
	A batch buffer with allocated memory. More...

class	SegmentationOutput

class	SegmentPostprocessor
	Post processor class for segmentation output. More...

class	StreamManager

class	SurfaceBuffer

class	SysMem
	Base class for managing memory allocation. More...

class	ThreadCudaPostprocessor
	A CUDA post processor thread queue template class. More...

class	ThreadHostPostprocessor
	A host post processor thread queue template class. More...

class	ThreadPreprocessor
	Preprocessor thread queue class template. More...

class	TritonGrpcBackend
	Triton gRPC mode backend processing class. More...

class	TritonGrpcRequest
	Triton gRPC inference request class holding data associated with one inference request. More...

class	TritonSimpleRuntime

class	TrtISBackend
	Triton backend processing class. More...

class	TrtIsClassifier
	Post processor class for Triton Classification option. More...

class	TrtISServer
	Wrapper class for creating Triton Inference Server instance. More...

class	TrtServerAllocator
	Wrapper class for Triton server output memory allocator. More...

class	TrtServerRequest
	Wrapper class for Triton inference request. More...

class	TrtServerResponse
	Wrapper class for Triton output parsing. More...

class	WrapCBatchBuffer

Typedefs
using	SharedBufOptions = std::shared_ptr< BufOptions >

using	SharedIBatchBuffer = std::shared_ptr< IBatchBuffer >

using	SharedIBatchArray = std::shared_ptr< IBatchArray >

using	SharedIOptions = std::shared_ptr< IOptions >

using	UniqSurfaceBuf = std::unique_ptr< SurfaceBuffer >

using	SharedBatchSurface = std::shared_ptr< BatchSurfaceBuffer >

using	InferCompleted = std::function< void(NvDsInferStatus, SharedBatchArray)>

using	UniqBackend = std::unique_ptr< BaseBackend >

using	LayerDescriptionList = std::vector< LayerDescription >

typedef std::map< std::string, std::string >	Headers

using	SharedGrpcRequest = std::shared_ptr< TritonGrpcRequest >

using	TritonGrpcAsyncDone = std::function< void(NvDsInferStatus, SharedBatchArray)>

using	SharedRequest = std::shared_ptr< TrtServerRequest >

using	UniqResponse = std::unique_ptr< TrtServerResponse >

using	SharedResponse = std::shared_ptr< TrtServerResponse >

using	TritonInferAsyncDone = std::function< void(SharedRequest, UniqResponse)>

using	SharedClassOutput = std::shared_ptr< ClassificationOutput >

using	TensorMapPool = MapBufferPool< std::string, UniqCudaTensorBuf >

using	TensorMapPoolPtr = std::unique_ptr< TensorMapPool >

Enumerations
enum	OptionType : int { OptionType::oBool = 0, OptionType::oDouble, OptionType::oInt, OptionType::oUint, OptionType::oString, OptionType::oObject, OptionType::oArray, OptionType::oNone = -1 }

enum	InferTensorOrder : int { InferTensorOrder::kNone = 0, InferTensorOrder::kLinear = 1, InferTensorOrder::kNHWC = 2 }
	The type of tensor order. More...

enum	InferMemType : int { InferMemType::kNone = 0, InferMemType::kGpuCuda = 1, InferMemType::kCpu = 2, InferMemType::kCpuCuda = 3, InferMemType::kNvSurface = 5, InferMemType::kNvSurfaceArray = 6 }
	The memory types of inference buffers. More...

enum	InferDataType : int { InferDataType::kFp32 = FLOAT, InferDataType::kFp16 = HALF, InferDataType::kInt8 = INT8, InferDataType::kInt32 = INT32, InferDataType::kInt16 = 7, InferDataType::kUint8, InferDataType::kUint16, InferDataType::kUint32, InferDataType::kFp64, InferDataType::kInt64, InferDataType::kUint64, InferDataType::kString, InferDataType::kBool, InferDataType::kNone = -1 }
	Datatype of the tensor buffer. More...

enum	InferPostprocessType : int { InferPostprocessType::kDetector = 0, InferPostprocessType::kClassifier = 1, InferPostprocessType::kSegmentation = 2, InferPostprocessType::kTrtIsClassifier = 3, InferPostprocessType::kOther = 100 }
	Inference post processing types. More...

enum	InferMediaFormat : int { InferMediaFormat::kRGB = 0, InferMediaFormat::kBGR, InferMediaFormat::kGRAY, InferMediaFormat::kRGBA, InferMediaFormat::kBGRx, InferMediaFormat::kUnknown = -1 }
	Image formats. More...

Functions
template<>
void	BufOptions::D::BasicValue::setV< std::string > (const std::string &v, OptionType t)

void	normalizeDims (InferDims &dims)

SharedBufPool< UniqSurfaceBuf >	createSurfaceBufPool (int width, int height, InferMediaFormat color, int batchSize, int gpuId, int poolSize)

InferDataType	dataTypeFromDsProto (ic::TensorDataType dt)

InferTensorOrder	tensorOrderFromDsProto (ic::TensorOrder o)

InferMediaFormat	mediaFormatFromDsProto (ic::MediaFormat f)

InferMemType	memTypeFromDsProto (ic::MemoryType t)

NvBufSurfTransform_Compute	computeHWFromDsProto (ic::FrameScalingHW h)

NvBufSurfTransform_Inter	scalingFilterFromDsProto (uint32_t filter)

bool	validateProtoConfig (ic::InferenceConfig &c, const std::string &path)

bool	compareModelRepo (const ic::TritonModelRepo &repoA, const ic::TritonModelRepo &repoB)

bool	hasTriton (const ic::BackendParams &params)

const ic::TritonParams &	getTritonParam (const ic::BackendParams &params)

ic::TritonParams *	mutableTriton (ic::BackendParams &params)

InferDataType	DataTypeFromTriton (TRITONSERVER_DataType type)
	Maps the TRITONSERVER_DataType to the InferDataType. More...

TRITONSERVER_DataType	DataTypeToTriton (InferDataType type)
	Maps the InferDataType to TRITONSERVER_DataType. More...

InferDataType	DataTypeFromTritonPb (ni::DataType type)
	Maps the data type from Triton model configuration proto definition to InferDataType. More...

InferTensorOrder	TensorOrderFromTritonPb (ni::ModelInput::Format order)
	Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type. More...

InferTensorOrder	TensorOrderFromTritonMeta (const std::string &format)
	Maps the tensor order from Triton metadata string to the InferTensorOrder type. More...

TRITONSERVER_MemoryType	MemTypeToTriton (InferMemType type)
	Maps the InferMemType to the TRITONSERVER_MemoryType. More...

InferMemType	MemTypeFromTriton (TRITONSERVER_MemoryType type)
	Maps the TRITONSERVER_MemoryType to the InferMemType. More...

template<typename VecDims >
InferDims	DimsFromTriton (const VecDims &shape)
	Converts the input shape vector from Triton to InferDims type. More...

const char *	TritonControlModeToStr (int32_t mode)
	Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll. More...

UniqCudaTensorBuf	createTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name, InferMemType mt, int devId, bool initCuEvent)
	Create a tensor buffer of the specified memory type, dimensions on the given device. More...

UniqCudaTensorBuf	createGpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false)
	Create a CUDA device memory tensor buffer of specified dimensions on the given device. More...

UniqCudaTensorBuf	createCpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false)
	Create a CUDA pinned memory tensor buffer of specified dimensions on the given device. More...

NvDsInferStatus	syncAllCudaEvents (const SharedBatchArray &bufList)
	Synchronize on all events associated with the batch buffer array. More...

Typedef Documentation

◆ Headers

typedef std::map<std::string, std::string> nvdsinferserver::Headers

Definition at line 42 of file infer_grpc_client.h.

◆ InferCompleted

using nvdsinferserver::InferCompleted = typedef std::function<void(NvDsInferStatus, SharedBatchArray)>

Definition at line 45 of file infer_base_context.h.

◆ InferCustomProcessorPtr

using nvdsinferserver::InferCustomProcessorPtr = typedef std::shared_ptr<IInferCustomProcessor>

Definition at line 141 of file infer_common.h.

◆ LayerDescriptionList

using nvdsinferserver::LayerDescriptionList = typedef std::vector<LayerDescription>

Definition at line 58 of file infer_ibackend.h.

◆ SharedBatchArray

using nvdsinferserver::SharedBatchArray = typedef std::shared_ptr<BaseBatchArray>

Definition at line 75 of file infer_common.h.

◆ SharedBatchBuf

using nvdsinferserver::SharedBatchBuf = typedef std::shared_ptr<BaseBatchBuffer>

Common buffer interfaces (internal).

Definition at line 71 of file infer_common.h.

◆ SharedBatchSurface

using nvdsinferserver::SharedBatchSurface = typedef std::shared_ptr<BatchSurfaceBuffer>

Definition at line 198 of file infer_surface_bufs.h.

◆ SharedBufOptions

using nvdsinferserver::SharedBufOptions = typedef std::shared_ptr<BufOptions>

Definition at line 41 of file infer_options.h.

◆ SharedClassOutput

using nvdsinferserver::SharedClassOutput = typedef std::shared_ptr<ClassificationOutput>

Definition at line 132 of file infer_postproc_buf.h.

◆ SharedCudaTensorBuf

using nvdsinferserver::SharedCudaTensorBuf = typedef std::shared_ptr<CudaTensorBuf>

Definition at line 91 of file infer_common.h.

◆ SharedCuEvent

using nvdsinferserver::SharedCuEvent = typedef std::shared_ptr<CudaEvent>

Definition at line 86 of file infer_common.h.

◆ SharedCuStream

using nvdsinferserver::SharedCuStream = typedef std::shared_ptr<CudaStream>

Cuda based pointers.

Definition at line 84 of file infer_common.h.

◆ SharedDllHandle

using nvdsinferserver::SharedDllHandle = typedef std::shared_ptr<DlLibHandle>

Definition at line 111 of file infer_common.h.

◆ SharedGrpcRequest

using nvdsinferserver::SharedGrpcRequest = typedef std::shared_ptr<TritonGrpcRequest>

Definition at line 46 of file infer_grpc_client.h.

◆ SharedIBatchArray

using nvdsinferserver::SharedIBatchArray = typedef std::shared_ptr<IBatchArray>

Definition at line 205 of file infer_datatypes.h.

◆ SharedIBatchBuffer

using nvdsinferserver::SharedIBatchBuffer = typedef std::shared_ptr<IBatchBuffer>

Definition at line 204 of file infer_datatypes.h.

◆ SharedIOptions

using nvdsinferserver::SharedIOptions = typedef std::shared_ptr<IOptions>

Definition at line 206 of file infer_datatypes.h.

◆ SharedOptions

using nvdsinferserver::SharedOptions = typedef std::shared_ptr<IOptions>

Definition at line 73 of file infer_common.h.

◆ SharedRefBatchBuf

using nvdsinferserver::SharedRefBatchBuf = typedef std::shared_ptr<RefBatchBuffer>

Definition at line 77 of file infer_common.h.

◆ SharedRequest

using nvdsinferserver::SharedRequest = typedef std::shared_ptr<TrtServerRequest>

Definition at line 59 of file infer_trtis_server.h.

◆ SharedResponse

using nvdsinferserver::SharedResponse = typedef std::shared_ptr<TrtServerResponse>

Definition at line 61 of file infer_trtis_server.h.

◆ SharedSysMem

using nvdsinferserver::SharedSysMem = typedef std::shared_ptr<SysMem>

Definition at line 88 of file infer_common.h.

◆ ShrTritonAllocator

using nvdsinferserver::ShrTritonAllocator = typedef std::shared_ptr<TrtServerAllocator>

Definition at line 124 of file infer_common.h.

◆ ShrTritonT

template<typename T >

using nvdsinferserver::ShrTritonT = typedef std::shared_ptr<T>

Definition at line 117 of file infer_common.h.

◆ TensorMapPool

using nvdsinferserver::TensorMapPool = typedef MapBufferPool<std::string, UniqCudaTensorBuf>

Definition at line 31 of file infer_extra_processor.h.

◆ TensorMapPoolPtr

using nvdsinferserver::TensorMapPoolPtr = typedef std::unique_ptr<TensorMapPool>

Definition at line 32 of file infer_extra_processor.h.

◆ TritonGrpcAsyncDone

using nvdsinferserver::TritonGrpcAsyncDone = typedef std::function<void(NvDsInferStatus, SharedBatchArray)>

Definition at line 48 of file infer_grpc_client.h.

◆ TritonInferAsyncDone

using nvdsinferserver::TritonInferAsyncDone = typedef std::function<void(SharedRequest, UniqResponse)>

Definition at line 63 of file infer_trtis_server.h.

◆ TrtServerPtr

using nvdsinferserver::TrtServerPtr = typedef std::shared_ptr<TrtISServer>

Definition at line 121 of file infer_common.h.

◆ UniqBackend

using nvdsinferserver::UniqBackend = typedef std::unique_ptr<BaseBackend>

Definition at line 212 of file infer_base_backend.h.

◆ UniqBatchArray

using nvdsinferserver::UniqBatchArray = typedef std::unique_ptr<BaseBatchArray>

Definition at line 76 of file infer_common.h.

◆ UniqBatchBuf

using nvdsinferserver::UniqBatchBuf = typedef std::unique_ptr<BaseBatchBuffer>

Definition at line 72 of file infer_common.h.

◆ UniqCudaTensorBuf

using nvdsinferserver::UniqCudaTensorBuf = typedef std::unique_ptr<CudaTensorBuf>

Definition at line 90 of file infer_common.h.

◆ UniqCuEvent

using nvdsinferserver::UniqCuEvent = typedef std::unique_ptr<CudaEvent>

Definition at line 87 of file infer_common.h.

◆ UniqCuStream

using nvdsinferserver::UniqCuStream = typedef std::unique_ptr<CudaStream>

Definition at line 85 of file infer_common.h.

◆ UniqInferExtraProcessor

using nvdsinferserver::UniqInferExtraProcessor = typedef std::unique_ptr<InferExtraProcessor>

Definition at line 139 of file infer_common.h.

◆ UniqLock

using nvdsinferserver::UniqLock = typedef std::unique_lock<std::mutex>

Miscellaneous declarations.

Definition at line 108 of file infer_common.h.

◆ UniqLstmController

using nvdsinferserver::UniqLstmController = typedef std::unique_ptr<LstmController>

Definition at line 128 of file infer_common.h.

◆ UniqPostprocessor

using nvdsinferserver::UniqPostprocessor = typedef std::unique_ptr<BasePostprocessor>

Processor interfaces.

Definition at line 98 of file infer_common.h.

◆ UniqPreprocessor

using nvdsinferserver::UniqPreprocessor = typedef std::unique_ptr<BasePreprocessor>

Definition at line 99 of file infer_common.h.

◆ UniqResponse

using nvdsinferserver::UniqResponse = typedef std::unique_ptr<TrtServerResponse>

Definition at line 60 of file infer_trtis_server.h.

◆ UniqStreamManager

using nvdsinferserver::UniqStreamManager = typedef std::unique_ptr<StreamManager>

Definition at line 131 of file infer_common.h.

◆ UniqSurfaceBuf

using nvdsinferserver::UniqSurfaceBuf = typedef std::unique_ptr<SurfaceBuffer>

Definition at line 44 of file infer_surface_bufs.h.

◆ UniqSysMem

using nvdsinferserver::UniqSysMem = typedef std::unique_ptr<SysMem>

Definition at line 89 of file infer_common.h.

◆ UniqTritonAllocator

using nvdsinferserver::UniqTritonAllocator = typedef std::unique_ptr<TrtServerAllocator>

Definition at line 123 of file infer_common.h.

◆ UniqTritonT

template<typename T >

using nvdsinferserver::UniqTritonT = typedef std::unique_ptr<T, std::function<void(T*)> >

Definition at line 114 of file infer_common.h.

◆ UniqTrtISBackend

using nvdsinferserver::UniqTrtISBackend = typedef std::unique_ptr<TrtISBackend>

Definition at line 101 of file infer_common.h.

◆ WeakTritonAllocator

using nvdsinferserver::WeakTritonAllocator = typedef std::weak_ptr<TrtServerAllocator>

Definition at line 125 of file infer_common.h.

Enumeration Type Documentation

◆ InferDataType

enum nvdsinferserver::InferDataType : int

strong

Datatype of the tensor buffer.

Enumerator
kFp32
kFp16
kInt8
kInt32
kInt16
kUint8
kUint16
kUint32
kFp64
kInt64
kUint64
kString
kBool
kNone

Definition at line 83 of file infer_datatypes.h.

◆ InferMediaFormat

enum nvdsinferserver::InferMediaFormat : int

strong

Image formats.

Enumerator
kRGB	24-bit interleaved R-G-B
kBGR	24-bit interleaved B-G-R
kGRAY	8-bit Luma
kRGBA	32-bit interleaved R-G-B-A
kBGRx	32-bit interleaved B-G-R-x
kUnknown

Definition at line 129 of file infer_datatypes.h.

◆ InferMemType

enum nvdsinferserver::InferMemType : int

strong

The memory types of inference buffers.

Enumerator
kNone
kGpuCuda	GPU CUDA memory.
kCpu	Host (CPU) memory.
kCpuCuda	CUDA pinned memory.
kNvSurface	NVRM surface memory.
kNvSurfaceArray	NVRM surface array memory.

Definition at line 56 of file infer_datatypes.h.

◆ InferPostprocessType

enum nvdsinferserver::InferPostprocessType : int

strong

Inference post processing types.

Enumerator
kDetector	Post processing for object detection.
kClassifier	Post processing for object classification.
kSegmentation	Post processing for image segmentation.
kTrtIsClassifier	Post processing using Triton Classifier.
kOther	Custom post processing.

Definition at line 103 of file infer_datatypes.h.

◆ InferTensorOrder

enum nvdsinferserver::InferTensorOrder : int

strong

The type of tensor order.

Enumerator
kNone
kLinear	NCHW (batch-channels-height-width) tensor order.
kNHWC	NHWC (batch-height-width-channels) tensor order.

Definition at line 41 of file infer_datatypes.h.

◆ OptionType

enum nvdsinferserver::OptionType : int

strong

Enumerator
oBool
oDouble
oInt
oUint
oString
oObject
oArray
oNone

Definition at line 28 of file infer_ioptions.h.

Function Documentation

◆ BufOptions::D::BasicValue::setV< std::string >()

template<>

void nvdsinferserver::BufOptions::D::BasicValue::setV< std::string >	(	const std::string &	v,
		OptionType	t
	)

inline

Definition at line 183 of file infer_options.h.

References oString.

◆ compareModelRepo()

bool nvdsinferserver::compareModelRepo	(	const ic::TritonModelRepo &	repoA,
		const ic::TritonModelRepo &	repoB
	)

◆ computeHWFromDsProto()

NvBufSurfTransform_Compute nvdsinferserver::computeHWFromDsProto ( ic::FrameScalingHW h )

◆ createCpuTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createCpuTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name = `""`,
		int	devId = `0`,
		bool	initCuEvent = `false`
	)

Create a CUDA pinned memory tensor buffer of specified dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ createGpuTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createGpuTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name = `""`,
		int	devId = `0`,
		bool	initCuEvent = `false`
	)

Create a CUDA device memory tensor buffer of specified dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ createSurfaceBufPool()

SharedBufPool<UniqSurfaceBuf> nvdsinferserver::createSurfaceBufPool	(	int	width,
		int	height,
		InferMediaFormat	color,
		int	batchSize,
		int	gpuId,
		int	poolSize
	)

◆ createTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name,
		InferMemType	mt,
		int	devId,
		bool	initCuEvent
	)

Create a tensor buffer of the specified memory type, dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	mt	Memory type.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ dataTypeFromDsProto()

InferDataType nvdsinferserver::dataTypeFromDsProto ( ic::TensorDataType dt )

◆ DataTypeFromTriton()

InferDataType nvdsinferserver::DataTypeFromTriton ( TRITONSERVER_DataType type )

Maps the TRITONSERVER_DataType to the InferDataType.

Parameters

type	TRITONSERVER_DataType

Returns: InferDataType corresponding to the input type.

◆ DataTypeFromTritonPb()

InferDataType nvdsinferserver::DataTypeFromTritonPb ( ni::DataType type )

Maps the data type from Triton model configuration proto definition to InferDataType.

Parameters

type	The protobuf datatype.

Returns: InferDataType corresponding to the protobuf type.

◆ DataTypeToTriton()

TRITONSERVER_DataType nvdsinferserver::DataTypeToTriton ( InferDataType type )

Maps the InferDataType to TRITONSERVER_DataType.

Parameters

type	InferDataType

Returns: TRITONSERVER_DataType corresponding to the input type.

◆ DimsFromTriton()

template<typename VecDims >

InferDims nvdsinferserver::DimsFromTriton ( const VecDims & shape )

Converts the input shape vector from Triton to InferDims type.

This functions provides a InferDims structure created based on the input dimensions vector. Any negative dimensions are mapped to -1. The numElements value is updated if there is no dynamic size dimension.

Template Parameters

VecDims Input dimensions vector type.

Parameters

shape Input dimensions vector.

Returns: The InferDims structure object created based on the input.

Definition at line 131 of file infer_trtis_utils.h.

References INFER_EXPORT_API::hasWildcard(), normalizeDims(), and NVDSINFER_MAX_DIMS.

◆ getTritonParam()

const ic::TritonParams& nvdsinferserver::getTritonParam ( const ic::BackendParams & params )

inline

Definition at line 61 of file infer_proto_utils.h.

◆ hasTriton()

bool nvdsinferserver::hasTriton ( const ic::BackendParams & params )

inline

Definition at line 56 of file infer_proto_utils.h.

◆ mediaFormatFromDsProto()

InferMediaFormat nvdsinferserver::mediaFormatFromDsProto ( ic::MediaFormat f )

◆ memTypeFromDsProto()

InferMemType nvdsinferserver::memTypeFromDsProto ( ic::MemoryType t )

◆ MemTypeFromTriton()

InferMemType nvdsinferserver::MemTypeFromTriton ( TRITONSERVER_MemoryType type )

Maps the TRITONSERVER_MemoryType to the InferMemType.

◆ MemTypeToTriton()

TRITONSERVER_MemoryType nvdsinferserver::MemTypeToTriton ( InferMemType type )

Maps the InferMemType to the TRITONSERVER_MemoryType.

◆ mutableTriton()

ic::TritonParams* nvdsinferserver::mutableTriton ( ic::BackendParams & params )

inline

Definition at line 71 of file infer_proto_utils.h.

◆ normalizeDims()

void nvdsinferserver::normalizeDims ( InferDims & dims )

Referenced by DimsFromTriton(), and nvdsinferserver::RefBatchBuffer::RefBatchBuffer().

◆ scalingFilterFromDsProto()

NvBufSurfTransform_Inter nvdsinferserver::scalingFilterFromDsProto ( uint32_t filter )

◆ syncAllCudaEvents()

NvDsInferStatus nvdsinferserver::syncAllCudaEvents ( const SharedBatchArray & bufList )

Synchronize on all events associated with the batch buffer array.

Parameters

[in] bufList Array of the batch buffers.

Returns: NVDSINFER_SUCCESS or NVDSINFER_CUDA_ERROR.

◆ tensorOrderFromDsProto()

InferTensorOrder nvdsinferserver::tensorOrderFromDsProto ( ic::TensorOrder o )

◆ TensorOrderFromTritonMeta()

InferTensorOrder nvdsinferserver::TensorOrderFromTritonMeta ( const std::string & format )

Maps the tensor order from Triton metadata string to the InferTensorOrder type.

Parameters

[in] format The Triton metadata tensor order string.

Returns: InferTensorOrder type corresponding to the protobuf order.

◆ TensorOrderFromTritonPb()

InferTensorOrder nvdsinferserver::TensorOrderFromTritonPb ( ni::ModelInput::Format order )

Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type.

Parameters

[in] order The protobuf tensor order type.

Returns: InferTensorOrder type corresponding to the protobuf order.

◆ TritonControlModeToStr()

const char* nvdsinferserver::TritonControlModeToStr ( int32_t mode )

Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll.

◆ validateProtoConfig()

bool nvdsinferserver::validateProtoConfig	(	ic::InferenceConfig &	c,
		const std::string &	path
	)


NVIDIA DeepStream SDK API Reference	7.0 Release

NVIDIA DeepStream SDK API Reference

7.0 Release

Detailed Description

Namespaces

Data Structures

Typedefs

Enumerations

Functions

Typedef Documentation

◆ Headers

◆ InferCompleted

◆ InferCustomProcessorPtr

◆ LayerDescriptionList

◆ SharedBatchArray

◆ SharedBatchBuf

◆ SharedBatchSurface

◆ SharedBufOptions

◆ SharedClassOutput

◆ SharedCudaTensorBuf

◆ SharedCuEvent

◆ SharedCuStream

◆ SharedDllHandle

◆ SharedGrpcRequest

◆ SharedIBatchArray

◆ SharedIBatchBuffer

◆ SharedIOptions

◆ SharedOptions

◆ SharedRefBatchBuf

◆ SharedRequest

◆ SharedResponse

◆ SharedSysMem

◆ ShrTritonAllocator

◆ ShrTritonT

◆ TensorMapPool

◆ TensorMapPoolPtr

◆ TritonGrpcAsyncDone

◆ TritonInferAsyncDone

◆ TrtServerPtr

◆ UniqBackend

◆ UniqBatchArray

◆ UniqBatchBuf

◆ UniqCudaTensorBuf

◆ UniqCuEvent

◆ UniqCuStream

◆ UniqInferExtraProcessor

◆ UniqLock

◆ UniqLstmController

◆ UniqPostprocessor

◆ UniqPreprocessor

◆ UniqResponse

◆ UniqStreamManager

◆ UniqSurfaceBuf

◆ UniqSysMem

◆ UniqTritonAllocator

◆ UniqTritonT

◆ UniqTrtISBackend

◆ WeakTritonAllocator

Enumeration Type Documentation

◆ InferDataType

◆ InferMediaFormat

◆ InferMemType

◆ InferPostprocessType

◆ InferTensorOrder

◆ OptionType

Function Documentation

◆ BufOptions::D::BasicValue::setV< std::string >()

◆ compareModelRepo()

◆ computeHWFromDsProto()

◆ createCpuTensorBuf()

◆ createGpuTensorBuf()

◆ createSurfaceBufPool()

◆ createTensorBuf()

◆ dataTypeFromDsProto()

◆ DataTypeFromTriton()

◆ DataTypeFromTritonPb()

◆ DataTypeToTriton()

◆ DimsFromTriton()

◆ getTritonParam()

◆ hasTriton()

◆ mediaFormatFromDsProto()