nvdsinferserver::TrtISServer Class Reference

Detailed Description

Wrapper class for creating Triton Inference Server instance.

Definition at line 533 of file infer_trtis_server.h.

Inheritance diagram for nvdsinferserver::TrtISServer:

Collaboration diagram for nvdsinferserver::TrtISServer:

Public Member Functions
	~TrtISServer ()
	Destructor. More...
bool	isServerReady ()
	Check if the server is ready. More...
bool	isServerLive ()
	Check if the server is live. More...
bool	isModelReady (const std::string &model, int64_t version)
	Check if the server is ready for inference using specified model. More...
NvDsInferStatus	loadModel (const std::string &modelName)
	Load or reload the specified model. More...
NvDsInferStatus	unloadModel (const std::string &modelName)
	Unload the specified model. More...
NvDsInferStatus	getModelConfig (const std::string &model, int64_t version, ni::ModelConfig &config)
	Get the model configuration for the specified model. More...
SharedRequest	createRequest (const std::string &model, int64_t version, SharedBatchArray &inputs, const std::vector< std::string > &outputs, const std::vector< TritonClassParams > &clasList)
	Create and initializes a new inference request. More...
NvDsInferStatus	inferAsync (SharedRequest request, WeakTritonAllocator allocator, TritonInferAsyncDone done)
	Submit a request for asynchronous inference. More...

Static Public Member Functions
static TrtServerPtr	getInstance (const triton::RepoSettings *repo)
	Get a new or existing instance of the Triton Inference Server. More...

Protected Member Functions
	TrtISServer (const triton::RepoSettings &repo)
	Constructor. More...
NvDsInferStatus	initialize ()
	Create a new instance of the Triton Inference Server. More...
const triton::RepoSettings &	getRepoSettings ()
	Get the model repository settings. More...

Friends
class	TrtServerRequest
class	TrtServerResponse

Constructor & Destructor Documentation

TrtISServer()

nvdsinferserver::TrtISServer::TrtISServer ( const triton::RepoSettings &  repo )

protected

Constructor.

Save the model repository configuration settings.

Parameters

repo

~TrtISServer()

nvdsinferserver::TrtISServer::~TrtISServer

(

)

Destructor.

Stops the Triton server if the server handle is valid.

Member Function Documentation

createRequest()

SharedRequest nvdsinferserver::TrtISServer::createRequest	(	const std::string &	model,
		int64_t	version,
		SharedBatchArray &	inputs,
		const std::vector< std::string > &	outputs,
		const std::vector< TritonClassParams > &	clasList
	)

Create and initializes a new inference request.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[in]	inputs	Array of input batch buffers.
[in]	outputs	List of requested output names.
[in]	clasList	Triton Classification parameters, if any.

Returns

Pointer to the request instance upon success, null pointer on failure.

getInstance()

static TrtServerPtr nvdsinferserver::TrtISServer::getInstance ( const triton::RepoSettings *  repo )

static

Get a new or existing instance of the Triton Inference Server.

This function checks if an instance of Triton is present. If it doesn't exist, a new one is instantiated using the provided model repository configuration. If a Triton instance is already running and repository configuration is provided, the existing and new configuration is checked to be same otherwise a null pointer is returned.

Parameters

[in]

repo

Model repository configuration, can be null when the Triton server is already instantiated.

Returns

Pointer to the Triton server instance.

getModelConfig()

NvDsInferStatus nvdsinferserver::TrtISServer::getModelConfig	(	const std::string &	model,
		int64_t	version,
		ni::ModelConfig &	config
	)

Get the model configuration for the specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[out]	config	ModelConfig protobuf message to be populated.

Returns

NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

getRepoSettings()

const triton::RepoSettings& nvdsinferserver::TrtISServer::getRepoSettings ( )

inlineprotected

Get the model repository settings.

Definition at line 558 of file infer_trtis_server.h.

inferAsync()

NvDsInferStatus nvdsinferserver::TrtISServer::inferAsync	(	SharedRequest	request,
		WeakTritonAllocator	allocator,
		TritonInferAsyncDone	done
	)

Submit a request for asynchronous inference.

This functions sets the release and response callback functions for the request and then triggers the asynchronous inference.

Parameters

[in]	request	Pointer to the request object.
[in]	allocator	Pointer to the response allocator.
[in]	done	Pointer to the function to be called after inference is done.

Returns

NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

initialize()

NvDsInferStatus nvdsinferserver::TrtISServer::initialize ( )

protected

Create a new instance of the Triton Inference Server.

This functions creates a TRITONSERVER_ServerOptions instance and updates it as per the repository configuration. These options are then used to create a new Triton instance and save the handle in m_Impl.

Returns

Error status.

isModelReady()

bool nvdsinferserver::TrtISServer::isModelReady	(	const std::string &	model,
		int64_t	version
	)

Check if the server is ready for inference using specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.

Returns

Boolean indicating readiness.

isServerLive()

bool nvdsinferserver::TrtISServer::isServerLive

(

)

Check if the server is live.

isServerReady()

bool nvdsinferserver::TrtISServer::isServerReady

(

)

Check if the server is ready.

loadModel()

NvDsInferStatus nvdsinferserver::TrtISServer::loadModel

(

const std::string &

modelName

)

Load or reload the specified model.

Parameters

[in]

modelName

Name of the model

Returns

NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

unloadModel()

NvDsInferStatus nvdsinferserver::TrtISServer::unloadModel

(

const std::string &

modelName

)

Unload the specified model.

Parameters

[in]

modelName

Name of the model.

Returns

NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

Friends And Related Function Documentation

TrtServerRequest

friend class TrtServerRequest

friend

Definition at line 534 of file infer_trtis_server.h.

TrtServerResponse

friend class TrtServerResponse

friend

Definition at line 535 of file infer_trtis_server.h.

---

The documentation for this class was generated from the following file:

• infer_trtis_server.h