Detailed Description

Wrapper class for creating Triton Inference Server instance.

Definition at line 533 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

Inheritance diagram for nvdsinferserver::TrtISServer:

Collaboration diagram for nvdsinferserver::TrtISServer:

Public Member Functions
	~TrtISServer ()
	Destructor. More...

bool	isServerReady ()
	Check if the server is ready. More...

bool	isServerLive ()
	Check if the server is live. More...

bool	isModelReady (const std::string &model, int64_t version)
	Check if the server is ready for inference using specified model. More...

NvDsInferStatus	loadModel (const std::string &modelName)
	Load or reload the specified model. More...

NvDsInferStatus	unloadModel (const std::string &modelName)
	Unload the specified model. More...

NvDsInferStatus	getModelConfig (const std::string &model, int64_t version, ni::ModelConfig &config)
	Get the model configuration for the specified model. More...

SharedRequest	createRequest (const std::string &model, int64_t version, SharedBatchArray &inputs, const std::vector< std::string > &outputs, const std::vector< TritonClassParams > &clasList)
	Create and initializes a new inference request. More...

NvDsInferStatus	inferAsync (SharedRequest request, WeakTritonAllocator allocator, TritonInferAsyncDone done)
	Submit a request for asynchronous inference. More...

	~TrtISServer ()
	Destructor. More...

bool	isServerReady ()
	Check if the server is ready. More...

bool	isServerLive ()
	Check if the server is live. More...

bool	isModelReady (const std::string &model, int64_t version)
	Check if the server is ready for inference using specified model. More...

NvDsInferStatus	loadModel (const std::string &modelName)
	Load or reload the specified model. More...

NvDsInferStatus	unloadModel (const std::string &modelName)
	Unload the specified model. More...

NvDsInferStatus	getModelConfig (const std::string &model, int64_t version, ni::ModelConfig &config)
	Get the model configuration for the specified model. More...

SharedRequest	createRequest (const std::string &model, int64_t version, SharedBatchArray &inputs, const std::vector< std::string > &outputs, const std::vector< TritonClassParams > &clasList)
	Create and initializes a new inference request. More...

NvDsInferStatus	inferAsync (SharedRequest request, WeakTritonAllocator allocator, TritonInferAsyncDone done)
	Submit a request for asynchronous inference. More...

Static Public Member Functions
static TrtServerPtr	getInstance (const triton::RepoSettings *repo)
	Get a new or existing instance of the Triton Inference Server. More...

static TrtServerPtr	getInstance (const triton::RepoSettings *repo)
	Get a new or existing instance of the Triton Inference Server. More...

Protected Member Functions
	TrtISServer (const triton::RepoSettings &repo)
	Constructor. More...

NvDsInferStatus	initialize ()
	Create a new instance of the Triton Inference Server. More...

const triton::RepoSettings &	getRepoSettings ()
	Get the model repository settings. More...

	TrtISServer (const triton::RepoSettings &repo)
	Constructor. More...

NvDsInferStatus	initialize ()
	Create a new instance of the Triton Inference Server. More...

const triton::RepoSettings &	getRepoSettings ()
	Get the model repository settings. More...

Friends
class	TrtServerRequest

class	TrtServerResponse

Constructor & Destructor Documentation

◆ TrtISServer() [1/2]

nvdsinferserver::TrtISServer::TrtISServer ( const triton::RepoSettings & repo )

protected

Constructor.

Save the model repository configuration settings.

Parameters

repo

◆ ~TrtISServer() [1/2]

nvdsinferserver::TrtISServer::~TrtISServer ( )

Destructor.

Stops the Triton server if the server handle is valid.

◆ TrtISServer() [2/2]

nvdsinferserver::TrtISServer::TrtISServer ( const triton::RepoSettings & repo )

protected

Constructor.

Save the model repository configuration settings.

Parameters

repo

◆ ~TrtISServer() [2/2]

nvdsinferserver::TrtISServer::~TrtISServer ( )

Destructor.

Stops the Triton server if the server handle is valid.

Member Function Documentation

◆ createRequest() [1/2]

SharedRequest nvdsinferserver::TrtISServer::createRequest	(	const std::string &	model,
		int64_t	version,
		SharedBatchArray &	inputs,
		const std::vector< std::string > &	outputs,
		const std::vector< TritonClassParams > &	clasList
	)

Create and initializes a new inference request.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[in]	inputs	Array of input batch buffers.
[in]	outputs	List of requested output names.
[in]	clasList	Triton Classification parameters, if any.

Returns: Pointer to the request instance upon success, null pointer on failure.

◆ createRequest() [2/2]

SharedRequest nvdsinferserver::TrtISServer::createRequest	(	const std::string &	model,
		int64_t	version,
		SharedBatchArray &	inputs,
		const std::vector< std::string > &	outputs,
		const std::vector< TritonClassParams > &	clasList
	)

Create and initializes a new inference request.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[in]	inputs	Array of input batch buffers.
[in]	outputs	List of requested output names.
[in]	clasList	Triton Classification parameters, if any.

Returns: Pointer to the request instance upon success, null pointer on failure.

◆ getInstance() [1/2]

static TrtServerPtr nvdsinferserver::TrtISServer::getInstance ( const triton::RepoSettings * repo )

static

Get a new or existing instance of the Triton Inference Server.

This function checks if an instance of Triton is present. If it doesn't exist, a new one is instantiated using the provided model repository configuration. If a Triton instance is already running and repository configuration is provided, the existing and new configuration is checked to be same otherwise a null pointer is returned.

Parameters

[in] repo Model repository configuration, can be null when the Triton server is already instantiated.

Returns: Pointer to the Triton server instance.

◆ getInstance() [2/2]

static TrtServerPtr nvdsinferserver::TrtISServer::getInstance ( const triton::RepoSettings * repo )

static

Get a new or existing instance of the Triton Inference Server.

This function checks if an instance of Triton is present. If it doesn't exist, a new one is instantiated using the provided model repository configuration. If a Triton instance is already running and repository configuration is provided, the existing and new configuration is checked to be same otherwise a null pointer is returned.

Parameters

[in] repo Model repository configuration, can be null when the Triton server is already instantiated.

Returns: Pointer to the Triton server instance.

◆ getModelConfig() [1/2]

NvDsInferStatus nvdsinferserver::TrtISServer::getModelConfig	(	const std::string &	model,
		int64_t	version,
		ni::ModelConfig &	config
	)

Get the model configuration for the specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[out]	config	ModelConfig protobuf message to be populated.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ getModelConfig() [2/2]

NvDsInferStatus nvdsinferserver::TrtISServer::getModelConfig	(	const std::string &	model,
		int64_t	version,
		ni::ModelConfig &	config
	)

Get the model configuration for the specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.
[out]	config	ModelConfig protobuf message to be populated.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ getRepoSettings() [1/2]

const triton::RepoSettings& nvdsinferserver::TrtISServer::getRepoSettings ( )

inlineprotected

Get the model repository settings.

Definition at line 558 of file 9.0/sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ getRepoSettings() [2/2]

const triton::RepoSettings& nvdsinferserver::TrtISServer::getRepoSettings ( )

inlineprotected

Get the model repository settings.

Definition at line 558 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ inferAsync() [1/2]

NvDsInferStatus nvdsinferserver::TrtISServer::inferAsync	(	SharedRequest	request,
		WeakTritonAllocator	allocator,
		TritonInferAsyncDone	done
	)

Submit a request for asynchronous inference.

This functions sets the release and response callback functions for the request and then triggers the asynchronous inference.

Parameters

[in]	request	Pointer to the request object.
[in]	allocator	Pointer to the response allocator.
[in]	done	Pointer to the function to be called after inference is done.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ inferAsync() [2/2]

NvDsInferStatus nvdsinferserver::TrtISServer::inferAsync	(	SharedRequest	request,
		WeakTritonAllocator	allocator,
		TritonInferAsyncDone	done
	)

Submit a request for asynchronous inference.

This functions sets the release and response callback functions for the request and then triggers the asynchronous inference.

Parameters

[in]	request	Pointer to the request object.
[in]	allocator	Pointer to the response allocator.
[in]	done	Pointer to the function to be called after inference is done.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ initialize() [1/2]

NvDsInferStatus nvdsinferserver::TrtISServer::initialize ( )

protected

Create a new instance of the Triton Inference Server.

This functions creates a TRITONSERVER_ServerOptions instance and updates it as per the repository configuration. These options are then used to create a new Triton instance and save the handle in m_Impl.

Returns: Error status.

◆ initialize() [2/2]

NvDsInferStatus nvdsinferserver::TrtISServer::initialize ( )

protected

Create a new instance of the Triton Inference Server.

This functions creates a TRITONSERVER_ServerOptions instance and updates it as per the repository configuration. These options are then used to create a new Triton instance and save the handle in m_Impl.

Returns: Error status.

◆ isModelReady() [1/2]

bool nvdsinferserver::TrtISServer::isModelReady	(	const std::string &	model,
		int64_t	version
	)

Check if the server is ready for inference using specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.

Returns: Boolean indicating readiness.

◆ isModelReady() [2/2]

bool nvdsinferserver::TrtISServer::isModelReady	(	const std::string &	model,
		int64_t	version
	)

Check if the server is ready for inference using specified model.

Parameters

[in]	model	Name of the model.
[in]	version	Version of the model.

Returns: Boolean indicating readiness.

◆ isServerLive() [1/2]

bool nvdsinferserver::TrtISServer::isServerLive ( )

Check if the server is live.

◆ isServerLive() [2/2]

bool nvdsinferserver::TrtISServer::isServerLive ( )

Check if the server is live.

◆ isServerReady() [1/2]

bool nvdsinferserver::TrtISServer::isServerReady ( )

Check if the server is ready.

◆ isServerReady() [2/2]

bool nvdsinferserver::TrtISServer::isServerReady ( )

Check if the server is ready.

◆ loadModel() [1/2]

NvDsInferStatus nvdsinferserver::TrtISServer::loadModel ( const std::string & modelName )

Load or reload the specified model.

Parameters

[in] modelName Name of the model

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ loadModel() [2/2]

NvDsInferStatus nvdsinferserver::TrtISServer::loadModel ( const std::string & modelName )

Load or reload the specified model.

Parameters

[in] modelName Name of the model

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ unloadModel() [1/2]

NvDsInferStatus nvdsinferserver::TrtISServer::unloadModel ( const std::string & modelName )

Unload the specified model.

Parameters

[in] modelName Name of the model.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

◆ unloadModel() [2/2]

NvDsInferStatus nvdsinferserver::TrtISServer::unloadModel ( const std::string & modelName )

Unload the specified model.

Parameters

[in] modelName Name of the model.

Returns: NVDSINFER_SUCCESS or NVDSINFER_TRITON_ERROR.

Friends And Related Function Documentation

◆ TrtServerRequest

TrtServerRequest

friend

Definition at line 534 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ TrtServerResponse

TrtServerResponse

friend

Definition at line 535 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

The documentation for this class was generated from the following file:

sources/libs/nvdsinferserver/infer_trtis_server.h

NVIDIA DeepStream SDK API Reference

8.0 Release

Detailed Description

Public Member Functions

Static Public Member Functions

Protected Member Functions

Friends

Constructor & Destructor Documentation

◆ TrtISServer() [1/2]

◆ ~TrtISServer() [1/2]

◆ TrtISServer() [2/2]

◆ ~TrtISServer() [2/2]

Member Function Documentation

◆ createRequest() [1/2]

◆ createRequest() [2/2]

◆ getInstance() [1/2]

◆ getInstance() [2/2]

◆ getModelConfig() [1/2]

◆ getModelConfig() [2/2]

◆ getRepoSettings() [1/2]

◆ getRepoSettings() [2/2]

◆ inferAsync() [1/2]

◆ inferAsync() [2/2]

◆ initialize() [1/2]

◆ initialize() [2/2]

◆ isModelReady() [1/2]

◆ isModelReady() [2/2]

◆ isServerLive() [1/2]

◆ isServerLive() [2/2]

◆ isServerReady() [1/2]

◆ isServerReady() [2/2]

◆ loadModel() [1/2]

◆ loadModel() [2/2]

◆ unloadModel() [1/2]

◆ unloadModel() [2/2]

Friends And Related Function Documentation

◆ TrtServerRequest

◆ TrtServerResponse