Inference context class accessing Triton Inference Server in gRPC mode.
Definition at line 37 of file infer_grpc_context.h.
Public Member Functions | |
InferGrpcContext () | |
Constructor, default. More... | |
~InferGrpcContext () override | |
Destructor, default. More... | |
SharedSysMem | acquireTensorHostBuf (const std::string &name, size_t bytes) |
Allocator. More... | |
SharedCuEvent | acquireTensorHostEvent () |
Acquire a CUDA event from the events pool. More... | |
NvDsInferStatus | initialize (const std::string &prototxt, InferLoggingFunc logFunc) final |
NvDsInferStatus | run (SharedIBatchArray input, InferOutputCb outputCb) final |
Protected Member Functions | |
NvDsInferStatus | createNNBackend (const ic::BackendParams ¶ms, int maxBatchSize, UniqBackend &backend) |
Create the Triton gRPC mode inference processing backend. More... | |
NvDsInferStatus | deinit () override |
Synchronize on the CUDA stream and call InferCudaContext::deinit(). More... | |
SharedCuStream & | mainStream () override |
Get the main processing CUDA event. More... | |
NvDsInferStatus | fixateInferenceInfo (const ic::InferenceConfig &config, BaseBackend &backend) override |
Check the tensor order, media format, and datatype for the input tensor. More... | |
NvDsInferStatus | createPreprocessor (const ic::PreProcessParams ¶ms, std::vector< UniqPreprocessor > &processors) override |
Create the surface converter and network preprocessor. More... | |
NvDsInferStatus | createPostprocessor (const ic::PostProcessParams ¶ms, UniqPostprocessor &processor) override |
Create the post-processor as per the network output type. More... | |
NvDsInferStatus | preInference (SharedBatchArray &inputs, const ic::InferenceConfig &config) override |
Initialize non-image input layers if the custom library has implemented the interface. More... | |
NvDsInferStatus | extraOutputTensorCheck (SharedBatchArray &outputs, SharedOptions inOptions) override |
Post inference steps for the custom processor and LSTM controller. More... | |
void | notifyError (NvDsInferStatus status) override |
In case of error, notify the waiting threads. More... | |
void | getNetworkInputInfo (NvDsInferNetworkInfo &networkInfo) override |
Get the network input layer information. More... | |
int | tensorPoolSize () const |
Get the size of the tensor pool. More... | |
virtual void | backendConsumedInputs (SharedBatchArray inputs) |
const ic::InferenceConfig & | config () const |
int | maxBatchSize () const |
int | uniqueId () const |
BaseBackend * | backend () |
const SharedDllHandle & | customLib () const |
bool | needCopyInputToHost () const |
void | print (NvDsInferLogLevel l, const char *msg) |
bool | needPreprocess () const |
Protected Attributes | |
NvDsInferNetworkInfo | m_NetworkImageInfo {0, 0, 0} |
Network input height, width, channels for preprocessing. More... | |
InferMediaFormat | m_NetworkImageFormat = InferMediaFormat::kRGB |
The input layer media format. More... | |
std::string | m_NetworkImageName |
The input layer name. More... | |
InferTensorOrder | m_InputTensorOrder = InferTensorOrder::kNone |
The input layer tensor order. More... | |
InferDataType | m_InputDataType = InferDataType::kFp32 |
The input layer datatype. More... | |
std::vector< SharedCudaTensorBuf > | m_ExtraInputs |
Array of buffers of the additional inputs. More... | |
MapBufferPool< std::string, UniqSysMem > | m_HostTensorPool |
Map of pools for the output tensors. More... | |
SharedBufPool< std::unique_ptr< CudaEventInPool > > | m_HostTensorEvents |
Pool of CUDA events for host tensor copy. More... | |
UniqLstmController | m_LstmController |
LSTM controller. More... | |
UniqStreamManager | m_MultiStreamManager |
stream-id based management. More... | |
UniqInferExtraProcessor | m_ExtraProcessor |
Extra and custom processing pre/post inference. More... | |
nvdsinferserver::InferGrpcContext::InferGrpcContext | ( | ) |
Constructor, default.
|
override |
Destructor, default.
|
inherited |
Allocator.
Acquire a host buffer for the inference output.
[in] | name | Name of the output layer. |
[in] | bytes | Size of the buffer. |
|
inherited |
Acquire a CUDA event from the events pool.
|
inlineprotectedinherited |
Definition at line 101 of file infer_base_context.h.
|
inlineprotectedvirtualinherited |
Definition at line 93 of file infer_base_context.h.
|
inlineprotectedinherited |
Definition at line 98 of file infer_base_context.h.
|
protectedvirtual |
Create the Triton gRPC mode inference processing backend.
[in] | params | The backend configuration parameters. |
[in] | maxBatchSize | The maximum batch size configuration. |
[out] | backend | Pointer for the backend handle. |
Implements nvdsinferserver::InferBaseContext.
|
overrideprotectedvirtualinherited |
Create the post-processor as per the network output type.
[in] | params | The post processing configuration parameters. |
[out] | processor | The handle to the created post processor. |
Implements nvdsinferserver::InferBaseContext.
|
overrideprotectedvirtualinherited |
Create the surface converter and network preprocessor.
params | The preprocessor configuration. |
processors | List of the created preprocessor handles. |
Implements nvdsinferserver::InferBaseContext.
|
inlineprotectedinherited |
Definition at line 102 of file infer_base_context.h.
|
overrideprotected |
Synchronize on the CUDA stream and call InferCudaContext::deinit().
|
overrideprotectedvirtualinherited |
Post inference steps for the custom processor and LSTM controller.
[in,out] | outputs | The output batch buffers array. |
[in] | inOptions | The configuration options for the buffers. |
Reimplemented from nvdsinferserver::InferBaseContext.
|
overrideprotectedvirtualinherited |
Check the tensor order, media format, and datatype for the input tensor.
Initiate the extra processor and lstm controller if configured.
[in] | config | The inference configuration protobuf message. |
[in] | backend | The inference backend instance. |
Implements nvdsinferserver::InferBaseContext.
|
inlineoverrideprotectedinherited |
Get the network input layer information.
Definition at line 132 of file infer_cuda_context.h.
References nvdsinferserver::InferCudaContext::m_NetworkImageInfo.
|
finalinherited |
|
inlineoverrideprotectedvirtual |
Get the main processing CUDA event.
Implements nvdsinferserver::InferBaseContext.
Definition at line 69 of file infer_grpc_context.h.
|
inlineprotectedinherited |
Definition at line 99 of file infer_base_context.h.
|
protectedinherited |
|
protectedinherited |
|
overrideprotectedvirtualinherited |
In case of error, notify the waiting threads.
Implements nvdsinferserver::InferBaseContext.
|
overrideprotectedvirtualinherited |
Initialize non-image input layers if the custom library has implemented the interface.
[in,out] | inputs | Array of the input batch buffers. |
[in] | config | The inference configuration settings. |
Reimplemented from nvdsinferserver::InferBaseContext.
|
protectedinherited |
|
finalinherited |
|
protectedinherited |
Get the size of the tensor pool.
|
inlineprotectedinherited |
Definition at line 100 of file infer_base_context.h.
|
protectedinherited |
Array of buffers of the additional inputs.
Definition at line 213 of file infer_cuda_context.h.
|
protectedinherited |
Extra and custom processing pre/post inference.
Definition at line 228 of file infer_cuda_context.h.
|
protectedinherited |
Definition at line 234 of file infer_cuda_context.h.
|
protectedinherited |
Pool of CUDA events for host tensor copy.
Definition at line 221 of file infer_cuda_context.h.
|
protectedinherited |
Map of pools for the output tensors.
Definition at line 217 of file infer_cuda_context.h.
|
protectedinherited |
The input layer datatype.
Definition at line 208 of file infer_cuda_context.h.
|
protectedinherited |
The input layer tensor order.
Definition at line 204 of file infer_cuda_context.h.
|
protectedinherited |
LSTM controller.
Definition at line 224 of file infer_cuda_context.h.
|
protectedinherited |
stream-id based management.
Definition at line 226 of file infer_cuda_context.h.
|
protectedinherited |
The input layer media format.
Definition at line 196 of file infer_cuda_context.h.
|
protectedinherited |
Network input height, width, channels for preprocessing.
Definition at line 192 of file infer_cuda_context.h.
Referenced by nvdsinferserver::InferCudaContext::getNetworkInputInfo().
|
protectedinherited |
The input layer name.
Definition at line 200 of file infer_cuda_context.h.
|
protectedinherited |
Definition at line 233 of file infer_cuda_context.h.
|
protectedinherited |
Preprocessor and post-processor handles.
Definition at line 232 of file infer_cuda_context.h.