Copyright (c) 2021, NVIDIA CORPORATION.
Copyright (c) 2020, NVIDIA CORPORATION.
Copyright (c) 2019-2021, NVIDIA CORPORATION.
This is a header file for pre-processing cuda kernels with normalization and mean subtraction required by nvdsinfer.
Copyright (c) 2020-2021, NVIDIA CORPORATION.
All rights reserved.
NVIDIA Corporation and its licensors retain all intellectual property and proprietary rights in and to this software, related documentation and any modifications thereto. Any use, reproduction, disclosure or distribution of this software and related documentation without an express license agreement from NVIDIA Corporation is strictly prohibited.
Namespaces | |
triton | |
Data Structures | |
class | BaseBackend |
Base class of inference backend processing. More... | |
class | BaseBatchArray |
The base class for array of batch buffers. More... | |
class | BaseBatchBuffer |
The base class for batch buffers. More... | |
class | BasePostprocessor |
Base post-processor class. More... | |
class | BasePreprocessor |
Base preprocessor class. More... | |
class | BatchSurfaceBuffer |
class | BufOptions |
class | ClassificationOutput |
class | ClassifyPostprocessor |
Post processor class for classification output. More... | |
class | CpuMem |
Allocates and manages host memory. More... | |
class | CropSurfaceConverter |
Preprocessor for cropping, scaling and padding the inference input to required height, width. More... | |
class | CudaDeviceMem |
Allocates and manages CUDA device memory. More... | |
class | CudaEvent |
Wrapper class for CUDA events. More... | |
class | CudaHostMem |
Allocates and manages CUDA pinned memory. More... | |
class | CudaStream |
Wrapper class for CUDA streams. More... | |
class | CudaTensorBuf |
A batch buffer with CUDA memory allocation. More... | |
class | DetectionOutput |
class | DetectPostprocessor |
Post processor class for detection output. More... | |
class | IBackend |
class | IBatchArray |
Interface class for an array of batch buffers. More... | |
class | IBatchBuffer |
Interface class for a batch buffer. More... | |
class | IInferCustomProcessor |
Interface of Custom processor which is created and loaded at runtime through CreateCustomProcessorFunc. More... | |
class | ImageAlignBuffer |
class | InferBaseContext |
The base class for handling the inference context. More... | |
struct | InferBatchDims |
Holds full dimensions (including batch size) for a layer. More... | |
struct | InferBufferDescription |
Holds the information about a inference buffer. More... | |
class | InferCudaContext |
Class for inference context CUDA processing. More... | |
struct | InferDims |
Holds the information about the dimensions of a neural network layer. More... | |
class | InferExtraProcessor |
: Extra processing pre/post inference. More... | |
class | InferGrpcClient |
Wrapper class for the gRPC client of the Triton Inference Server, interfaces with the Triton client library. More... | |
class | InferGrpcContext |
Inference context class accessing Triton Inference Server in gRPC mode. More... | |
class | InferSimpleContext |
class | InferTrtISContext |
Inference context accessing Triton Inference Server in C-API mode. More... | |
class | IOptions |
class | IPostprocessor |
Post-processor interface class. More... | |
class | IPreprocessor |
Preprocessor interface class. More... | |
struct | LayerDescription |
Stores the information of a layer in the inference model. More... | |
class | LstmController |
class | NetworkPreprocessor |
Preprocessor for scaling and normalization of the input and conversion to network media format. More... | |
class | OtherPostprocessor |
Post processor class for tensor output for custom post processing. More... | |
class | Postprocessor |
A generic post processor class. More... | |
class | RefBatchBuffer |
A batch buffer with allocated memory. More... | |
class | SegmentationOutput |
class | SegmentPostprocessor |
Post processor class for segmentation output. More... | |
class | StreamManager |
class | SurfaceBuffer |
class | SysMem |
Base class for managing memory allocation. More... | |
class | ThreadCudaPostprocessor |
A CUDA post processor thread queue template class. More... | |
class | ThreadHostPostprocessor |
A host post processor thread queue template class. More... | |
class | ThreadPreprocessor |
Preprocessor thread queue class template. More... | |
class | TritonGrpcBackend |
Triton gRPC mode backend processing class. More... | |
class | TritonGrpcRequest |
Triton gRPC inference request class holding data associated with one inference request. More... | |
class | TritonSimpleRuntime |
class | TrtISBackend |
Triton backend processing class. More... | |
class | TrtIsClassifier |
Post processor class for Triton Classification option. More... | |
class | TrtISServer |
Wrapper class for creating Triton Inference Server instance. More... | |
class | TrtServerAllocator |
Wrapper class for Triton server output memory allocator. More... | |
class | TrtServerRequest |
Wrapper class for Triton inference request. More... | |
class | TrtServerResponse |
Wrapper class for Triton output parsing. More... | |
class | WrapCBatchBuffer |
Typedefs | |
using | SharedBufOptions = std::shared_ptr< BufOptions > |
using | SharedIBatchBuffer = std::shared_ptr< IBatchBuffer > |
using | SharedIBatchArray = std::shared_ptr< IBatchArray > |
using | SharedIOptions = std::shared_ptr< IOptions > |
using | UniqSurfaceBuf = std::unique_ptr< SurfaceBuffer > |
using | SharedBatchSurface = std::shared_ptr< BatchSurfaceBuffer > |
using | InferCompleted = std::function< void(NvDsInferStatus, SharedBatchArray)> |
using | UniqBackend = std::unique_ptr< BaseBackend > |
using | LayerDescriptionList = std::vector< LayerDescription > |
typedef std::map< std::string, std::string > | Headers |
using | SharedGrpcRequest = std::shared_ptr< TritonGrpcRequest > |
using | TritonGrpcAsyncDone = std::function< void(NvDsInferStatus, SharedBatchArray)> |
using | SharedRequest = std::shared_ptr< TrtServerRequest > |
using | UniqResponse = std::unique_ptr< TrtServerResponse > |
using | SharedResponse = std::shared_ptr< TrtServerResponse > |
using | TritonInferAsyncDone = std::function< void(SharedRequest, UniqResponse)> |
using | SharedClassOutput = std::shared_ptr< ClassificationOutput > |
using | TensorMapPool = MapBufferPool< std::string, UniqCudaTensorBuf > |
using | TensorMapPoolPtr = std::unique_ptr< TensorMapPool > |
Functions | |
template<> | |
void | BufOptions::D::BasicValue::setV< std::string > (const std::string &v, OptionType t) |
void | normalizeDims (InferDims &dims) |
SharedBufPool< UniqSurfaceBuf > | createSurfaceBufPool (int width, int height, InferMediaFormat color, int batchSize, int gpuId, int poolSize) |
InferDataType | dataTypeFromDsProto (ic::TensorDataType dt) |
InferTensorOrder | tensorOrderFromDsProto (ic::TensorOrder o) |
InferMediaFormat | mediaFormatFromDsProto (ic::MediaFormat f) |
InferMemType | memTypeFromDsProto (ic::MemoryType t) |
NvBufSurfTransform_Compute | computeHWFromDsProto (ic::FrameScalingHW h) |
NvBufSurfTransform_Inter | scalingFilterFromDsProto (uint32_t filter) |
bool | validateProtoConfig (ic::InferenceConfig &c, const std::string &path) |
bool | compareModelRepo (const ic::TritonModelRepo &repoA, const ic::TritonModelRepo &repoB) |
bool | hasTriton (const ic::BackendParams ¶ms) |
const ic::TritonParams & | getTritonParam (const ic::BackendParams ¶ms) |
ic::TritonParams * | mutableTriton (ic::BackendParams ¶ms) |
InferDataType | DataTypeFromTriton (TRITONSERVER_DataType type) |
Maps the TRITONSERVER_DataType to the InferDataType. More... | |
TRITONSERVER_DataType | DataTypeToTriton (InferDataType type) |
Maps the InferDataType to TRITONSERVER_DataType. More... | |
InferDataType | DataTypeFromTritonPb (ni::DataType type) |
Maps the data type from Triton model configuration proto definition to InferDataType. More... | |
InferTensorOrder | TensorOrderFromTritonPb (ni::ModelInput::Format order) |
Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type. More... | |
InferTensorOrder | TensorOrderFromTritonMeta (const std::string &format) |
Maps the tensor order from Triton metadata string to the InferTensorOrder type. More... | |
TRITONSERVER_MemoryType | MemTypeToTriton (InferMemType type) |
Maps the InferMemType to the TRITONSERVER_MemoryType. More... | |
InferMemType | MemTypeFromTriton (TRITONSERVER_MemoryType type) |
Maps the TRITONSERVER_MemoryType to the InferMemType. More... | |
template<typename VecDims > | |
InferDims | DimsFromTriton (const VecDims &shape) |
Converts the input shape vector from Triton to InferDims type. More... | |
const char * | TritonControlModeToStr (int32_t mode) |
Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll. More... | |
UniqCudaTensorBuf | createTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name, InferMemType mt, int devId, bool initCuEvent) |
Create a tensor buffer of the specified memory type, dimensions on the given device. More... | |
UniqCudaTensorBuf | createGpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false) |
Create a CUDA device memory tensor buffer of specified dimensions on the given device. More... | |
UniqCudaTensorBuf | createCpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false) |
Create a CUDA pinned memory tensor buffer of specified dimensions on the given device. More... | |
NvDsInferStatus | syncAllCudaEvents (const SharedBatchArray &bufList) |
Synchronize on all events associated with the batch buffer array. More... | |
typedef std::map<std::string, std::string> nvdsinferserver::Headers |
Definition at line 42 of file infer_grpc_client.h.
using nvdsinferserver::InferCompleted = typedef std::function<void(NvDsInferStatus, SharedBatchArray)> |
Definition at line 45 of file infer_base_context.h.
using nvdsinferserver::InferCustomProcessorPtr = typedef std::shared_ptr<IInferCustomProcessor> |
Definition at line 141 of file infer_common.h.
using nvdsinferserver::LayerDescriptionList = typedef std::vector<LayerDescription> |
Definition at line 58 of file infer_ibackend.h.
using nvdsinferserver::SharedBatchArray = typedef std::shared_ptr<BaseBatchArray> |
Definition at line 75 of file infer_common.h.
using nvdsinferserver::SharedBatchBuf = typedef std::shared_ptr<BaseBatchBuffer> |
Common buffer interfaces (internal).
Definition at line 71 of file infer_common.h.
using nvdsinferserver::SharedBatchSurface = typedef std::shared_ptr<BatchSurfaceBuffer> |
Definition at line 198 of file infer_surface_bufs.h.
using nvdsinferserver::SharedBufOptions = typedef std::shared_ptr<BufOptions> |
Definition at line 45 of file infer_options.h.
using nvdsinferserver::SharedClassOutput = typedef std::shared_ptr<ClassificationOutput> |
Definition at line 131 of file infer_postproc_buf.h.
using nvdsinferserver::SharedCudaTensorBuf = typedef std::shared_ptr<CudaTensorBuf> |
Definition at line 91 of file infer_common.h.
using nvdsinferserver::SharedCuEvent = typedef std::shared_ptr<CudaEvent> |
Definition at line 86 of file infer_common.h.
using nvdsinferserver::SharedCuStream = typedef std::shared_ptr<CudaStream> |
Cuda based pointers.
Definition at line 84 of file infer_common.h.
using nvdsinferserver::SharedDllHandle = typedef std::shared_ptr<DlLibHandle> |
Definition at line 111 of file infer_common.h.
using nvdsinferserver::SharedGrpcRequest = typedef std::shared_ptr<TritonGrpcRequest> |
Definition at line 46 of file infer_grpc_client.h.
using nvdsinferserver::SharedIBatchArray = typedef std::shared_ptr<IBatchArray> |
Definition at line 205 of file infer_datatypes.h.
using nvdsinferserver::SharedIBatchBuffer = typedef std::shared_ptr<IBatchBuffer> |
Definition at line 204 of file infer_datatypes.h.
using nvdsinferserver::SharedIOptions = typedef std::shared_ptr<IOptions> |
Definition at line 206 of file infer_datatypes.h.
using nvdsinferserver::SharedOptions = typedef std::shared_ptr<IOptions> |
Definition at line 73 of file infer_common.h.
using nvdsinferserver::SharedRefBatchBuf = typedef std::shared_ptr<RefBatchBuffer> |
Definition at line 77 of file infer_common.h.
using nvdsinferserver::SharedRequest = typedef std::shared_ptr<TrtServerRequest> |
Definition at line 59 of file infer_trtis_server.h.
using nvdsinferserver::SharedResponse = typedef std::shared_ptr<TrtServerResponse> |
Definition at line 61 of file infer_trtis_server.h.
using nvdsinferserver::SharedSysMem = typedef std::shared_ptr<SysMem> |
Definition at line 88 of file infer_common.h.
using nvdsinferserver::ShrTritonAllocator = typedef std::shared_ptr<TrtServerAllocator> |
Definition at line 124 of file infer_common.h.
using nvdsinferserver::ShrTritonT = typedef std::shared_ptr<T> |
Definition at line 117 of file infer_common.h.
using nvdsinferserver::TensorMapPool = typedef MapBufferPool<std::string, UniqCudaTensorBuf> |
Definition at line 31 of file infer_extra_processor.h.
using nvdsinferserver::TensorMapPoolPtr = typedef std::unique_ptr<TensorMapPool> |
Definition at line 32 of file infer_extra_processor.h.
using nvdsinferserver::TritonGrpcAsyncDone = typedef std::function<void(NvDsInferStatus, SharedBatchArray)> |
Definition at line 48 of file infer_grpc_client.h.
using nvdsinferserver::TritonInferAsyncDone = typedef std::function<void(SharedRequest, UniqResponse)> |
Definition at line 63 of file infer_trtis_server.h.
using nvdsinferserver::TrtServerPtr = typedef std::shared_ptr<TrtISServer> |
Definition at line 121 of file infer_common.h.
using nvdsinferserver::UniqBackend = typedef std::unique_ptr<BaseBackend> |
Definition at line 212 of file infer_base_backend.h.
using nvdsinferserver::UniqBatchArray = typedef std::unique_ptr<BaseBatchArray> |
Definition at line 76 of file infer_common.h.
using nvdsinferserver::UniqBatchBuf = typedef std::unique_ptr<BaseBatchBuffer> |
Definition at line 72 of file infer_common.h.
using nvdsinferserver::UniqCudaTensorBuf = typedef std::unique_ptr<CudaTensorBuf> |
Definition at line 90 of file infer_common.h.
using nvdsinferserver::UniqCuEvent = typedef std::unique_ptr<CudaEvent> |
Definition at line 87 of file infer_common.h.
using nvdsinferserver::UniqCuStream = typedef std::unique_ptr<CudaStream> |
Definition at line 85 of file infer_common.h.
using nvdsinferserver::UniqInferExtraProcessor = typedef std::unique_ptr<InferExtraProcessor> |
Definition at line 139 of file infer_common.h.
using nvdsinferserver::UniqLock = typedef std::unique_lock<std::mutex> |
Miscellaneous declarations.
Definition at line 108 of file infer_common.h.
using nvdsinferserver::UniqLstmController = typedef std::unique_ptr<LstmController> |
Definition at line 128 of file infer_common.h.
using nvdsinferserver::UniqPostprocessor = typedef std::unique_ptr<BasePostprocessor> |
Processor interfaces.
Definition at line 98 of file infer_common.h.
using nvdsinferserver::UniqPreprocessor = typedef std::unique_ptr<BasePreprocessor> |
Definition at line 99 of file infer_common.h.
using nvdsinferserver::UniqResponse = typedef std::unique_ptr<TrtServerResponse> |
Definition at line 60 of file infer_trtis_server.h.
using nvdsinferserver::UniqStreamManager = typedef std::unique_ptr<StreamManager> |
Definition at line 131 of file infer_common.h.
using nvdsinferserver::UniqSurfaceBuf = typedef std::unique_ptr<SurfaceBuffer> |
Definition at line 44 of file infer_surface_bufs.h.
using nvdsinferserver::UniqSysMem = typedef std::unique_ptr<SysMem> |
Definition at line 89 of file infer_common.h.
using nvdsinferserver::UniqTritonAllocator = typedef std::unique_ptr<TrtServerAllocator> |
Definition at line 123 of file infer_common.h.
using nvdsinferserver::UniqTritonT = typedef std::unique_ptr<T, std::function<void(T*)> > |
Definition at line 114 of file infer_common.h.
using nvdsinferserver::UniqTrtISBackend = typedef std::unique_ptr<TrtISBackend> |
Definition at line 101 of file infer_common.h.
using nvdsinferserver::WeakTritonAllocator = typedef std::weak_ptr<TrtServerAllocator> |
Definition at line 125 of file infer_common.h.
|
strong |
Datatype of the tensor buffer.
Enumerator | |
---|---|
kFp32 | |
kFp16 | |
kInt8 | |
kInt32 | |
kInt16 | |
kUint8 | |
kUint16 | |
kUint32 | |
kFp64 | |
kInt64 | |
kUint64 | |
kString | |
kBool | |
kNone |
Definition at line 83 of file infer_datatypes.h.
|
strong |
Image formats.
Enumerator | |
---|---|
kRGB | 24-bit interleaved R-G-B |
kBGR | 24-bit interleaved B-G-R |
kGRAY | 8-bit Luma |
kRGBA | 32-bit interleaved R-G-B-A |
kBGRx | 32-bit interleaved B-G-R-x |
kUnknown |
Definition at line 129 of file infer_datatypes.h.
|
strong |
The memory types of inference buffers.
Enumerator | |
---|---|
kNone | |
kGpuCuda | GPU CUDA memory. |
kCpu | Host (CPU) memory. |
kCpuCuda | CUDA pinned memory. |
kNvSurface | NVRM surface memory. |
kNvSurfaceArray | NVRM surface array memory. |
Definition at line 56 of file infer_datatypes.h.
|
strong |
Inference post processing types.
Definition at line 103 of file infer_datatypes.h.
|
strong |
The type of tensor order.
Enumerator | |
---|---|
kNone | |
kLinear | NCHW (batch-channels-height-width) tensor order. |
kNHWC | NHWC (batch-height-width-channels) tensor order. |
Definition at line 41 of file infer_datatypes.h.
|
strong |
Enumerator | |
---|---|
oBool | |
oDouble | |
oInt | |
oUint | |
oString | |
oObject | |
oArray | |
oNone |
Definition at line 31 of file infer_ioptions.h.
|
inline |
Definition at line 187 of file infer_options.h.
References oString.
bool nvdsinferserver::compareModelRepo | ( | const ic::TritonModelRepo & | repoA, |
const ic::TritonModelRepo & | repoB | ||
) |
NvBufSurfTransform_Compute nvdsinferserver::computeHWFromDsProto | ( | ic::FrameScalingHW | h | ) |
UniqCudaTensorBuf nvdsinferserver::createCpuTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name = "" , |
||
int | devId = 0 , |
||
bool | initCuEvent = false |
||
) |
Create a CUDA pinned memory tensor buffer of specified dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
UniqCudaTensorBuf nvdsinferserver::createGpuTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name = "" , |
||
int | devId = 0 , |
||
bool | initCuEvent = false |
||
) |
Create a CUDA device memory tensor buffer of specified dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
SharedBufPool<UniqSurfaceBuf> nvdsinferserver::createSurfaceBufPool | ( | int | width, |
int | height, | ||
InferMediaFormat | color, | ||
int | batchSize, | ||
int | gpuId, | ||
int | poolSize | ||
) |
UniqCudaTensorBuf nvdsinferserver::createTensorBuf | ( | const InferDims & | dims, |
InferDataType | dt, | ||
int | batchSize, | ||
const std::string & | name, | ||
InferMemType | mt, | ||
int | devId, | ||
bool | initCuEvent | ||
) |
Create a tensor buffer of the specified memory type, dimensions on the given device.
[in] | dims | Dimensions of the tensor. |
[in] | dt | Datatype. |
[in] | batchSize | Batch size. |
[in] | name | Name of the buffer. |
[in] | mt | Memory type. |
[in] | devId | Device ID for the memory allocation. |
[in] | initCuEvent | Flag to create an associated CUDA event. |
InferDataType nvdsinferserver::dataTypeFromDsProto | ( | ic::TensorDataType | dt | ) |
InferDataType nvdsinferserver::DataTypeFromTriton | ( | TRITONSERVER_DataType | type | ) |
Maps the TRITONSERVER_DataType to the InferDataType.
type | TRITONSERVER_DataType |
InferDataType nvdsinferserver::DataTypeFromTritonPb | ( | ni::DataType | type | ) |
Maps the data type from Triton model configuration proto definition to InferDataType.
type | The protobuf datatype. |
TRITONSERVER_DataType nvdsinferserver::DataTypeToTriton | ( | InferDataType | type | ) |
Maps the InferDataType to TRITONSERVER_DataType.
type | InferDataType |
InferDims nvdsinferserver::DimsFromTriton | ( | const VecDims & | shape | ) |
Converts the input shape vector from Triton to InferDims type.
This functions provides a InferDims structure created based on the input dimensions vector. Any negative dimensions are mapped to -1. The numElements value is updated if there is no dynamic size dimension.
VecDims | Input dimensions vector type. |
shape | Input dimensions vector. |
Definition at line 131 of file infer_trtis_utils.h.
References INFER_EXPORT_API::hasWildcard(), normalizeDims(), and NVDSINFER_MAX_DIMS.
|
inline |
Definition at line 60 of file infer_proto_utils.h.
|
inline |
Definition at line 55 of file infer_proto_utils.h.
InferMediaFormat nvdsinferserver::mediaFormatFromDsProto | ( | ic::MediaFormat | f | ) |
InferMemType nvdsinferserver::memTypeFromDsProto | ( | ic::MemoryType | t | ) |
InferMemType nvdsinferserver::MemTypeFromTriton | ( | TRITONSERVER_MemoryType | type | ) |
Maps the TRITONSERVER_MemoryType to the InferMemType.
TRITONSERVER_MemoryType nvdsinferserver::MemTypeToTriton | ( | InferMemType | type | ) |
Maps the InferMemType to the TRITONSERVER_MemoryType.
|
inline |
Definition at line 70 of file infer_proto_utils.h.
void nvdsinferserver::normalizeDims | ( | InferDims & | dims | ) |
Referenced by DimsFromTriton(), and nvdsinferserver::RefBatchBuffer::RefBatchBuffer().
NvBufSurfTransform_Inter nvdsinferserver::scalingFilterFromDsProto | ( | uint32_t | filter | ) |
NvDsInferStatus nvdsinferserver::syncAllCudaEvents | ( | const SharedBatchArray & | bufList | ) |
Synchronize on all events associated with the batch buffer array.
[in] | bufList | Array of the batch buffers. |
InferTensorOrder nvdsinferserver::tensorOrderFromDsProto | ( | ic::TensorOrder | o | ) |
InferTensorOrder nvdsinferserver::TensorOrderFromTritonMeta | ( | const std::string & | format | ) |
Maps the tensor order from Triton metadata string to the InferTensorOrder type.
[in] | format | The Triton metadata tensor order string. |
InferTensorOrder nvdsinferserver::TensorOrderFromTritonPb | ( | ni::ModelInput::Format | order | ) |
Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type.
[in] | order | The protobuf tensor order type. |
const char* nvdsinferserver::TritonControlModeToStr | ( | int32_t | mode | ) |
Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll.
bool nvdsinferserver::validateProtoConfig | ( | ic::InferenceConfig & | c, |
const std::string & | path | ||
) |