This is a header file for pre-processing cuda kernels with normalization and mean subtraction required by nvdsinfer.
Namespaces | |
triton | |
Data Structures | |
class | BaseBackend |
Base class of inference backend processing. More... | |
class | BaseBatchArray |
The base class for array of batch buffers. More... | |
class | BaseBatchBuffer |
The base class for batch buffers. More... | |
class | BasePostprocessor |
Base post-processor class. More... | |
class | BasePreprocessor |
Base preprocessor class. More... | |
class | BatchSurfaceBuffer |
class | BufOptions |
class | ClassificationOutput |
class | ClassifyPostprocessor |
Post processor class for classification output. More... | |
class | CpuMem |
Allocates and manages host memory. More... | |
class | CropSurfaceConverter |
Preprocessor for cropping, scaling and padding the inference input to required height, width. More... | |
class | CudaDeviceMem |
Allocates and manages CUDA device memory. More... | |
class | CudaEvent |
Wrapper class for CUDA events. More... | |
class | CudaHostMem |
Allocates and manages CUDA pinned memory. More... | |
class | CudaStream |
Wrapper class for CUDA streams. More... | |
class | CudaTensorBuf |
A batch buffer with CUDA memory allocation. More... | |
class | DetectionOutput |
class | DetectPostprocessor |
Post processor class for detection output. More... | |
class | IBackend |
class | IBatchArray |
Interface class for an array of batch buffers. More... | |
class | IBatchBuffer |
Interface class for a batch buffer. More... | |
class | IInferCustomProcessor |
Interface of Custom processor which is created and loaded at runtime through CreateCustomProcessorFunc. More... | |
class | ImageAlignBuffer |
class | InferBaseContext |
The base class for handling the inference context. More... | |
struct | InferBatchDims |
Holds full dimensions (including batch size) for a layer. More... | |
struct | InferBufferDescription |
Holds the information about a inference buffer. More... | |
class | InferCudaContext |
Class for inference context CUDA processing. More... | |
struct | InferDims |
Holds the information about the dimensions of a neural network layer. More... | |
class | InferExtraProcessor |
: Extra processing pre/post inference. More... | |
class | InferGrpcClient |
Wrapper class for the gRPC client of the Triton Inference Server, interfaces with the Triton client library. More... | |
class | InferGrpcContext |
Inference context class accessing Triton Inference Server in gRPC mode. More... | |
class | InferSimpleContext |
class | InferTrtISContext |
Inference context accessing Triton Inference Server in C-API mode. More... | |
class | IOptions |
class | IPostprocessor |
Post-processor interface class. More... | |
class | IPreprocessor |
Preprocessor interface class. More... | |
struct | LayerDescription |
Stores the information of a layer in the inference model. More... | |
class | LstmController |
class | NetworkPreprocessor |
Preprocessor for scaling and normalization of the input and conversion to network media format. More... | |
class | OtherPostprocessor |
Post processor class for tensor output for custom post processing. More... | |
class | Postprocessor |
A generic post processor class. More... | |
class | RefBatchBuffer |
A batch buffer with allocated memory. More... | |
class | SegmentationOutput |
class | SegmentPostprocessor |
Post processor class for segmentation output. More... | |
class | StreamManager |
class | SurfaceBuffer |
class | SysMem |
Base class for managing memory allocation. More... | |
class | ThreadCudaPostprocessor |
A CUDA post processor thread queue template class. More... | |
class | ThreadHostPostprocessor |
A host post processor thread queue template class. More... | |
class | ThreadPreprocessor |
Preprocessor thread queue class template. More... | |
class | TritonGrpcBackend |
Triton gRPC mode backend processing class. More... | |
class | TritonGrpcRequest |
Triton gRPC inference request class holding data associated with one inference request. More... | |
class | TritonSimpleRuntime |
class | TrtISBackend |
Triton backend processing class. More... | |
class | TrtIsClassifier |
Post processor class for Triton Classification option. More... | |
class | TrtISServer |
Wrapper class for creating Triton Inference Server instance. More... | |
class | TrtServerAllocator |
Wrapper class for Triton server output memory allocator. More... | |
class | TrtServerRequest |
Wrapper class for Triton inference request. More... | |
class | TrtServerResponse |
Wrapper class for Triton output parsing. More... | |
class | WrapCBatchBuffer |
Typedefs | |
using | SharedBufOptions = std::shared_ptr< BufOptions > |
using | SharedIBatchBuffer = std::shared_ptr< IBatchBuffer > |
using | SharedIBatchArray = std::shared_ptr< IBatchArray > |
using | SharedIOptions = std::shared_ptr< IOptions > |
using | UniqSurfaceBuf = std::unique_ptr< SurfaceBuffer > |
using | SharedBatchSurface = std::shared_ptr< BatchSurfaceBuffer > |
using | InferCompleted = std::function< void(NvDsInferStatus, SharedBatchArray)> |
using | UniqBackend = std::unique_ptr< BaseBackend > |
using | LayerDescriptionList = std::vector< LayerDescription > |
typedef std::map< std::string, std::string > | Headers |
using | SharedGrpcRequest = std::shared_ptr< TritonGrpcRequest > |
using | TritonGrpcAsyncDone = std::function< void(NvDsInferStatus, SharedBatchArray)> |
using | SharedRequest = std::shared_ptr< TrtServerRequest > |
using | UniqResponse = std::unique_ptr< TrtServerResponse > |
using | SharedResponse = std::shared_ptr< TrtServerResponse > |
using | TritonInferAsyncDone = std::function< void(SharedRequest, UniqResponse)> |
using | SharedClassOutput = std::shared_ptr< ClassificationOutput > |
using | TensorMapPool = MapBufferPool< std::string, UniqCudaTensorBuf > |
using | TensorMapPoolPtr = std::unique_ptr< TensorMapPool > |
Functions | |
template<> | |
void | BufOptions::D::BasicValue::setV< std::string > (const std::string &v, OptionType t) |
void | normalizeDims (InferDims &dims) |
SharedBufPool< UniqSurfaceBuf > | createSurfaceBufPool (int width, int height, InferMediaFormat color, int batchSize, int gpuId, int poolSize) |
InferDataType | dataTypeFromDsProto (ic::TensorDataType dt) |
InferTensorOrder | tensorOrderFromDsProto (ic::TensorOrder o) |
InferMediaFormat | mediaFormatFromDsProto (ic::MediaFormat f) |
InferMemType | memTypeFromDsProto (ic::MemoryType t) |
NvBufSurfTransform_Compute | computeHWFromDsProto (ic::FrameScalingHW h) |
NvBufSurfTransform_Inter | scalingFilterFromDsProto (uint32_t filter) |
bool | validateProtoConfig (ic::InferenceConfig &c, const std::string &path) |
bool | compareModelRepo (const ic::TritonModelRepo &repoA, const ic::TritonModelRepo &repoB) |
bool | hasTriton (const ic::BackendParams ¶ms) |
const ic::TritonParams & | getTritonParam (const ic::BackendParams ¶ms) |
ic::TritonParams * | mutableTriton (ic::BackendParams ¶ms) |
InferDataType | DataTypeFromTriton (TRITONSERVER_DataType type) |
Maps the TRITONSERVER_DataType to the InferDataType. More... | |
TRITONSERVER_DataType | DataTypeToTriton (InferDataType type) |
Maps the InferDataType to TRITONSERVER_DataType. More... | |
InferDataType | DataTypeFromTritonPb (ni::DataType type) |
Maps the data type from Triton model configuration proto definition to InferDataType. More... | |
InferTensorOrder | TensorOrderFromTritonPb (ni::ModelInput::Format order) |
Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type. More... | |
InferTensorOrder | TensorOrderFromTritonMeta (const std::string &format) |
Maps the tensor order from Triton metadata string to the InferTensorOrder type. More... | |
TRITONSERVER_MemoryType | MemTypeToTriton (InferMemType type) |
Maps the InferMemType to the TRITONSERVER_MemoryType. More... | |
InferMemType | MemTypeFromTriton (TRITONSERVER_MemoryType type) |
Maps the TRITONSERVER_MemoryType to the InferMemType. More... | |
template<typename VecDims > | |
InferDims | DimsFromTriton (const VecDims &shape) |
Converts the input shape vector from Triton to InferDims type. More... | |
const char * | TritonControlModeToStr (int32_t mode) |
Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll. More... | |
UniqCudaTensorBuf | createTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name, InferMemType mt, int devId, bool initCuEvent) |
Create a tensor buffer of the specified memory type, dimensions on the given device. More... | |
UniqCudaTensorBuf | createGpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false) |
Create a CUDA device memory tensor buffer of specified dimensions on the given device. More... | |
UniqCudaTensorBuf | createCpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false) |
Create a CUDA pinned memory tensor buffer of specified dimensions on the given device. More... | |
NvDsInferStatus | syncAllCudaEvents (const SharedBatchArray &bufList) |
Synchronize on all events associated with the batch buffer array. More... | |
typedef std::map<std::string, std::string> nvdsinferserver::Headers |
Definition at line 42 of file infer_grpc_client.h.
using nvdsinferserver::InferCompleted = typedef std::function<void(NvDsInferStatus, SharedBatchArray)> |
Definition at line 45 of file infer_base_context.h.
using nvdsinferserver::InferCustomProcessorPtr = typedef std::shared_ptr<IInferCustomProcessor> |
Definition at line 141 of file infer_common.h.
using nvdsinferserver::LayerDescriptionList = typedef std::vector<LayerDescription> |
Definition at line 58 of file infer_ibackend.h.
using nvdsinferserver::SharedBatchArray = typedef std::shared_ptr<BaseBatchArray> |
Definition at line 75 of file infer_common.h.
using nvdsinferserver::SharedBatchBuf = typedef std::shared_ptr<BaseBatchBuffer> |
Common buffer interfaces (internal).
Definition at line 71 of file infer_common.h.
using nvdsinferserver::SharedBatchSurface = typedef std::shared_ptr<BatchSurfaceBuffer> |
Definition at line 198 of file infer_surface_bufs.h.
using nvdsinferserver::SharedBufOptions = typedef std::shared_ptr<BufOptions> |
Definition at line 41 of file infer_options.h.
using nvdsinferserver::SharedClassOutput = typedef std::shared_ptr<ClassificationOutput> |
Definition at line 132 of file infer_postproc_buf.h.
using nvdsinferserver::SharedCudaTensorBuf = typedef std::shared_ptr<CudaTensorBuf> |
Definition at line 91 of file infer_common.h.
using nvdsinferserver::SharedCuEvent = typedef std::shared_ptr<CudaEvent> |
Definition at line 86 of file infer_common.h.
using nvdsinferserver::SharedCuStream = typedef std::shared_ptr<CudaStream> |
Cuda based pointers.
Definition at line 84 of file infer_common.h.
using nvdsinferserver::SharedDllHandle = typedef std::shared_ptr<DlLibHandle> |
Definition at line 111 of file infer_common.h.
using nvdsinferserver::SharedGrpcRequest = typedef std::shared_ptr<TritonGrpcRequest> |
Definition at line 46 of file infer_grpc_client.h.
using nvdsinferserver::SharedIBatchArray = typedef std::shared_ptr<IBatchArray> |
Definition at line 205 of file infer_datatypes.h.
using nvdsinferserver::SharedIBatchBuffer = typedef std::shared_ptr<IBatchBuffer> |
Definition at line 204 of file infer_datatypes.h.
using nvdsinferserver::SharedIOptions = typedef std::shared_ptr<IOptions> |
Definition at line 206 of file infer_datatypes.h.
using nvdsinferserver::SharedOptions = typedef std::shared_ptr<IOptions> |
Definition at line 73 of file infer_common.h.
using nvdsinferserver::SharedRefBatchBuf = typedef std::shared_ptr<RefBatchBuffer> |
Definition at line 77 of file infer_common.h.
using nvdsinferserver::SharedRequest = typedef std::shared_ptr<TrtServerRequest> |
Definition at line 59 of file infer_trtis_server.h.
using nvdsinferserver::SharedResponse = typedef std::shared_ptr<TrtServerResponse> |
Definition at line 61 of file infer_trtis_server.h.
using nvdsinferserver::SharedSysMem = typedef std::shared_ptr<SysMem> |
Definition at line 88 of file infer_common.h.
using nvdsinferserver::ShrTritonAllocator = typedef std::shared_ptr<TrtServerAllocator> |
Definition at line 124 of file infer_common.h.
using nvdsinferserver::ShrTritonT = typedef std::shared_ptr<T> |
Definition at line 117 of file infer_common.h.
using nvdsinferserver::TensorMapPool = typedef MapBufferPool<std::string, UniqCudaTensorBuf> |
Definition at line 31 of file infer_extra_processor.h.
using nvdsinferserver::TensorMapPoolPtr = typedef std::unique_ptr<TensorMapPool> |
Definition at line 32 of file infer_extra_processor.h.
using nvdsinferserver::TritonGrpcAsyncDone = typedef std::function<void(NvDsInferStatus, SharedBatchArray)> |
Definition at line 48 of file infer_grpc_client.h.
using nvdsinferserver::TritonInferAsyncDone = typedef std::function<void(SharedRequest, UniqResponse)> |
Definition at line 63 of file infer_trtis_server.h.
using nvdsinferserver::TrtServerPtr = typedef std::shared_ptr<TrtISServer> |
Definition at line 121 of file infer_common.h.
using nvdsinferserver::UniqBackend = typedef std::unique_ptr<BaseBackend> |
Definition at line 212 of file infer_base_backend.h.
using nvdsinferserver::UniqBatchArray = typedef std::unique_ptr<BaseBatchArray> |
Definition at line 76 of file infer_common.h.
using nvdsinferserver::UniqBatchBuf = typedef std::unique_ptr<BaseBatchBuffer> |
Definition at line 72 of file infer_common.h.
using nvdsinferserver::UniqCudaTensorBuf = typedef std::unique_ptr<CudaTensorBuf> |
Definition at line 90 of file infer_common.h.
using nvdsinferserver::UniqCuEvent = typedef std::unique_ptr<CudaEvent> |
Definition at line 87 of file infer_common.h.
using nvdsinferserver::UniqCuStream = typedef std::unique_ptr<CudaStream> |
Definition at line 85 of file infer_common.h.
using nvdsinferserver::UniqInferExtraProcessor = typedef std::unique_ptr<InferExtraProcessor> |
Definition at line 139 of file infer_common.h.
using nvdsinferserver::UniqLock = typedef std::unique_lock<std::mutex> |
Miscellaneous declarations.
Definition at line 108 of file infer_common.h.
using nvdsinferserver::UniqLstmController = typedef std::unique_ptr<LstmController> |
Definition at line 128 of file infer_common.h.
using nvdsinferserver::UniqPostprocessor = typedef std::unique_ptr<BasePostprocessor> |
Processor interfaces.
Definition at line 98 of file infer_common.h.
using nvdsinferserver::UniqPreprocessor = typedef std::unique_ptr<BasePreprocessor> |
Definition at line 99 of file infer_common.h.
using nvdsinferserver::UniqResponse = typedef std::unique_ptr<TrtServerResponse> |
Definition at line 60 of file infer_trtis_server.h.
using nvdsinferserver::UniqStreamManager = typedef std::unique_ptr<StreamManager> |
Definition at line 131 of file infer_common.h.
using nvdsinferserver::UniqSurfaceBuf = typedef std::unique_ptr<SurfaceBuffer> |
Definition at line 44 of file infer_surface_bufs.h.
using nvdsinferserver::UniqSysMem = typedef std::unique_ptr<SysMem> |
Definition at line 89 of file infer_common.h.
using nvdsinferserver::UniqTritonAllocator = typedef std::unique_ptr<TrtServerAllocator> |
Definition at line 123 of file infer_common.h.
using nvdsinferserver::UniqTritonT = typedef std::unique_ptr<T, std::function<void(T*)> > |
Definition at line 114 of file infer_common.h.
using nvdsinferserver::UniqTrtISBackend = typedef std::unique_ptr<TrtISBackend> |
Definition at line 101 of file infer_common.h.
using nvdsinferserver::WeakTritonAllocator = typedef std::weak_ptr<TrtServerAllocator> |
Definition at line 125 of file infer_common.h.
|
strong |
Datatype of the tensor buffer.
Enumerator | |
---|---|
kFp32 | |
kFp16 | |
kInt8 | |
kInt32 | |
kInt16 | |
kUint8 | |
kUint16 | |
kUint32 | |
kFp64 | |
kInt64 | |
kUint64 | |
kString | |
kBool | |
kNone |
Definition at line 83 of file infer_datatypes.h.
|
strong |
Image formats.
Enumerator | |
---|---|
kRGB | 24-bit interleaved R-G-B |
kBGR | 24-bit interleaved B-G-R |
kGRAY | 8-bit Luma |
kRGBA | 32-bit interleaved R-G-B-A |
kBGRx | 32-bit interleaved B-G-R-x |
kUnknown |
Definition at line 129 of file infer_datatypes.h.
|
strong |
The memory types of inference buffers.
Enumerator | |
---|---|
kNone | |
kGpuCuda | GPU CUDA memory. |
kCpu | Host (CPU) memory. |
kCpuCuda | CUDA pinned memory. |
kNvSurface | NVRM surface memory. |
kNvSurfaceArray | NVRM surface array memory. |
Definition at line 56 of file infer_datatypes.h.
|
strong |
Inference post processing types.
Definition at line 103 of file infer_datatypes.h.
|
strong |
The type of tensor order.
Enumerator | |
---|---|
kNone | |
kLinear | NCHW (batch-channels-height-width) tensor order. |
kNHWC | NHWC (batch-height-width-channels) tensor order. |
Definition at line 41 of file infer_datatypes.h.
|
strong |
Enumerator | |
---|---|
oBool | |
oDouble | |
oInt | |
oUint | |
oString | |
oObject | |
oArray | |
oNone |
Definition at line 28 of file infer_ioptions.h.
|
inline |
Definition at line 183 of file infer_options.h.
References oString.
bool nvdsinferserver::compareModelRepo | ( | const ic::TritonModelRepo & | repoA, |
const ic::TritonModelRepo & | repoB | ||
) |
NvBufSurfTransform_Compute nvdsinferserver::computeHWFromDsProto | ( | ic::FrameScalingHW | h | ) |
UniqCudaTensorBuf nvdsinferserver::createCpuTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name = "" , |
||
int | devId = 0 , |
||
bool | initCuEvent = false |
||
) |
Create a CUDA pinned memory tensor buffer of specified dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
UniqCudaTensorBuf nvdsinferserver::createGpuTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name = "" , |
||
int | devId = 0 , |
||
bool | initCuEvent = false |
||
) |
Create a CUDA device memory tensor buffer of specified dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
SharedBufPool<UniqSurfaceBuf> nvdsinferserver::createSurfaceBufPool | ( | int | width, |
int | height, | ||
InferMediaFormat | color, | ||
int | batchSize, | ||
int | gpuId, | ||
int | poolSize | ||
) |
UniqCudaTensorBuf nvdsinferserver::createTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name, | ||
InferMemType | mt, | ||
int | devId, | ||
bool | initCuEvent | ||
) |
Create a tensor buffer of the specified memory type, dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | mt | Memory type. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
InferDataType nvdsinferserver::dataTypeFromDsProto | ( | ic::TensorDataType | dt | ) |
InferDataType nvdsinferserver::DataTypeFromTriton | ( | TRITONSERVER_DataType | type | ) |
Maps the TRITONSERVER_DataType to the InferDataType.
type | TRITONSERVER_DataType |
InferDataType nvdsinferserver::DataTypeFromTritonPb | ( | ni::DataType | type | ) |
Maps the data type from Triton model configuration proto definition to InferDataType.
type | The protobuf datatype. |
TRITONSERVER_DataType nvdsinferserver::DataTypeToTriton | ( | InferDataType | type | ) |
Maps the InferDataType to TRITONSERVER_DataType.
type | InferDataType |
InferDims nvdsinferserver::DimsFromTriton | ( | const VecDims & | shape | ) |
Converts the input shape vector from Triton to InferDims type.
This functions provides a InferDims structure created based on the input dimensions vector. Any negative dimensions are mapped to -1. The numElements value is updated if there is no dynamic size dimension.
VecDims | Input dimensions vector type. |
shape | Input dimensions vector. |
Definition at line 131 of file infer_trtis_utils.h.
References INFER_EXPORT_API::hasWildcard(), normalizeDims(), and NVDSINFER_MAX_DIMS.
|
inline |
Definition at line 61 of file infer_proto_utils.h.
|
inline |
Definition at line 56 of file infer_proto_utils.h.
InferMediaFormat nvdsinferserver::mediaFormatFromDsProto | ( | ic::MediaFormat | f | ) |
InferMemType nvdsinferserver::memTypeFromDsProto | ( | ic::MemoryType | t | ) |
InferMemType nvdsinferserver::MemTypeFromTriton | ( | TRITONSERVER_MemoryType | type | ) |
Maps the TRITONSERVER_MemoryType to the InferMemType.
TRITONSERVER_MemoryType nvdsinferserver::MemTypeToTriton | ( | InferMemType | type | ) |
Maps the InferMemType to the TRITONSERVER_MemoryType.
|
inline |
Definition at line 71 of file infer_proto_utils.h.
void nvdsinferserver::normalizeDims | ( | InferDims & | dims | ) |
Referenced by DimsFromTriton(), and nvdsinferserver::RefBatchBuffer::RefBatchBuffer().
NvBufSurfTransform_Inter nvdsinferserver::scalingFilterFromDsProto | ( | uint32_t | filter | ) |
NvDsInferStatus nvdsinferserver::syncAllCudaEvents | ( | const SharedBatchArray & | bufList | ) |
Synchronize on all events associated with the batch buffer array.
[in] | bufList | Array of the batch buffers. |
InferTensorOrder nvdsinferserver::tensorOrderFromDsProto | ( | ic::TensorOrder | o | ) |
InferTensorOrder nvdsinferserver::TensorOrderFromTritonMeta | ( | const std::string & | format | ) |
Maps the tensor order from Triton metadata string to the InferTensorOrder type.
[in] | format | The Triton metadata tensor order string. |
InferTensorOrder nvdsinferserver::TensorOrderFromTritonPb | ( | ni::ModelInput::Format | order | ) |
Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type.
[in] | order | The protobuf tensor order type. |
const char* nvdsinferserver::TritonControlModeToStr | ( | int32_t | mode | ) |
Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll.
bool nvdsinferserver::validateProtoConfig | ( | ic::InferenceConfig & | c, |
const std::string & | path | ||
) |