Abstract Interface for a functionally safe graph for executing inference on a built network. More...

#include <NvInferSafeRuntime.h>

Public Member Functions
	ITRTGraph (ITRTGraph const &)=delete

	ITRTGraph (ITRTGraph &&)=delete

ITRTGraph &	operator= (ITRTGraph const &) &=delete

ITRTGraph &	operator= (ITRTGraph &&) &=delete

virtual	~ITRTGraph () noexcept=default
	A shallow destructor of ITRTGraph. More...

virtual ErrorCode	clone (ITRTGraph *&graph, ISafeRecorder &recorder) noexcept=0
	Specialized Graph shallow copy. More...

virtual ErrorCode	getScratchMemorySize (size_t &size) const noexcept=0
	This function returns the scratch memory size (in bytes) needed to store all the intermediate tensors for inference. The user could allocate a scratch memory of this size and pass it to TRT via setScratchMemory if not using trtManagedScratch. More...

virtual ErrorCode	getTRTManagedScratch (bool &flag) const noexcept=0
	This function returns the trtManagedScratch flag provided in createTRTGraph call. More...

virtual ErrorCode	setScratchMemory (void *memory) noexcept=0
	This function sets the scratch memory for the graph. This should only be called if scratch memory is not TRT managed (trtManagedScratch is false). An error will be recorded if called on a graph where trtManagedScratch is true. More...

virtual ErrorCode	getScratchMemory (void *&memory) noexcept=0
	This function gets the scratch memory for the graph. This should only be called if scratch memory is not TRT managed (trtManagedScratch is false). An error will be recorded if called on a graph where trtManagedScratch is true. More...

virtual ErrorCode	getNbIOTensors (int64_t &nb) const noexcept=0
	This function returns the total number of input and output tensor for the current graph. More...

virtual ErrorCode	getIOTensorName (AsciiChar const *&name, size_t const index) const noexcept=0
	This function returns the name of a tensor for a given index. More...

virtual ErrorCode	getIOTensorDescriptor (TensorDescriptor &desc, AsciiChar const *const tensorName) const noexcept=0
	This function should return a TensorDescriptor which contains all the information about the tensor based on the name. More...

virtual ErrorCode	getIOTensorDescriptor (TensorDescriptor &desc, int32_t const index) const noexcept=0
	This function should return a TensorDescriptor which contains all the information about the tensor based on the index. More...

virtual ErrorCode	setIOTensorAddress (AsciiChar const *const tensorName, TypedArray const &tensor) noexcept=0
	This function assigns a user allocated device memory block for an input tensor to the graph based on its name. More...

virtual ErrorCode	setIOTensorAddress (int32_t const index, TypedArray const &tensor) noexcept=0
	This function assigns a user allocated device memory block for an input tensor to the graph based on its index. More...

virtual ErrorCode	getIOTensorAddress (AsciiChar const *const tensorName, TypedArray &tensor) noexcept=0
	This function gets the memory address for an user provided input tensor to the graph based on its name. More...

virtual ErrorCode	getIOTensorAddress (int32_t const index, TypedArray &tensor) noexcept=0
	This function gets the memory address for an user provided input tensor to the graph based on its index. More...

virtual ErrorCode	setInputConsumedEvent (cudaEvent_t event) noexcept=0
	This function sets a cudaEvent on the current graph that triggers when the input is consumed. At that point, the input memory can be recycled, i.e. new input for next inference can be loaded. More...

virtual ErrorCode	getInputConsumedEvent (cudaEvent_t &event) const noexcept=0
	This function retrieves the cudaEvent on the current graph that triggers when the input is fully consumed. At that point, the input memory can be recycled, i.e. new input for next inference can be loaded. More...

virtual ErrorCode	getErrorBuffer (RuntimeErrorInformation *&buffer) const noexcept=0
	This function retrieves the RuntimeErrorInformation (for async error) buffer for the current graph. This buffer includes all the runtime error types such as gather out of bound, silently consumed NaN value etc. More...

virtual ErrorCode	getSafeRecorder (ISafeRecorder *&recorder) const noexcept=0
	This function retrieves the ISafeRecorder for the current graph. More...

virtual ErrorCode	getNbIOProfiles (int64_t &nb) const noexcept=0
	This function returns the total number of IO tensor profiles for the current graph. More...

virtual ErrorCode	setIOProfile (int64_t profileIndex) noexcept=0
	This function selects the active IOProfile for the graph. If this function is not called, the TRTGraph will default to profile 0. Each IOProfile on the graph is mutually exclusive, meaning only one IOProfile can be active at a time. More...

virtual ErrorCode	getIOProfile (int64_t &profileIndex) const noexcept=0
	This function retrieves the index of the current active IOProfile for the graph. More...

virtual ErrorCode	getNbAuxStreams (int32_t &nbStreams) const noexcept=0
	Return the number of auxiliary streams used by this graph. More...

virtual ErrorCode	setAuxStreams (cudaStream_t *auxStreams, int32_t nbStreams) noexcept=0
	Set the auxiliary streams that TensorRT should use to run kernels on. More...

virtual ErrorCode	executeAsync (cudaStream_t stream) noexcept=0
	execute one inference of this graph. More...

virtual ErrorCode	sync () noexcept=0
	synchronize one inference of this graph. More...

Protected Member Functions
	ITRTGraph ()=default

Detailed Description

Abstract Interface for a functionally safe graph for executing inference on a built network.

Warning: Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.; APIs are not thread safe: Users must guarantee that each concurrently running thread uses a unique instance (clone) of the API object.

Note: The set APIs are not thread safe and get APIs are thread-safe.

Constructor & Destructor Documentation

◆ ITRTGraph() [1/3]

nvinfer2::safe::ITRTGraph::ITRTGraph ( ITRTGraph const & )

delete

◆ ITRTGraph() [2/3]

nvinfer2::safe::ITRTGraph::ITRTGraph ( ITRTGraph && )

delete

◆ ~ITRTGraph()

virtual nvinfer2::safe::ITRTGraph::~ITRTGraph ( )

virtualdefaultnoexcept

A shallow destructor of ITRTGraph.

Precondition: DEINIT API

◆ ITRTGraph() [3/3]

nvinfer2::safe::ITRTGraph::ITRTGraph ( )

protecteddefault

Member Function Documentation

◆ clone()

virtual ErrorCode nvinfer2::safe::ITRTGraph::clone	(	ITRTGraph *&	graph,
		ISafeRecorder &	recorder
	)

pure virtualnoexcept

Specialized Graph shallow copy.

This function constructs a new ITRTGraph which uses a shared pointer to the persistent part of the graph (pointing to the same set of weights) so we do not duplicate the weights anywhere. Clone will automatically allocate a new scratch memory if the scratch memory is trtManaged (trtManagedScratch set to true in createTRTGraph) otherwise user has to call setScratchMemory before calling execute. The cloned graph will be using the same memory allocator as the original graph.

Precondition: INIT API

Parameters

graph	A reference to an ITRTGraph pointer, that will be initialized after this call.
recorder	An ISafeRecorder that records the errors happening during graph initialization and inference.

Note: Each cloned graph may be given it's own recorder so there is an independent source of truth for each graph.

Returns: ErrorCode

◆ executeAsync()

virtual ErrorCode nvinfer2::safe::ITRTGraph::executeAsync ( cudaStream_t stream )

pure virtualnoexcept

execute one inference of this graph.

Precondition: RUNTIME API

Parameters

stream A CUDA main stream on which the inference kernels will be enqueued. Must be a valid CUDA stream.

Returns

ErrorCode kSUCCESS on success, if any execution error occurred other error code might be returned Errors may include but not be limited to:

Internal errors during executing one engine layer (host side)
CUDA errors
Some input or output tensor addresses have not been set.

◆ getErrorBuffer()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getErrorBuffer ( RuntimeErrorInformation *& buffer ) const

pure virtualnoexcept

This function retrieves the RuntimeErrorInformation (for async error) buffer for the current graph. This buffer includes all the runtime error types such as gather out of bound, silently consumed NaN value etc.

Precondition: RUNTIME API

Parameters

buffer retrieved error buffer.

See also: RuntimeErrorInformation

Returns: ErrorCode

◆ getInputConsumedEvent()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getInputConsumedEvent ( cudaEvent_t & event ) const

pure virtualnoexcept

This function retrieves the cudaEvent on the current graph that triggers when the input is fully consumed. At that point, the input memory can be recycled, i.e. new input for next inference can be loaded.

Precondition: RUNTIME API

Parameters

event retrieved cuda event

Returns: ErrorCode

◆ getIOProfile()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOProfile ( int64_t & profileIndex ) const

pure virtualnoexcept

This function retrieves the index of the current active IOProfile for the graph.

Precondition: RUNTIME API

Parameters

profileIndex retrieved profile index.

Returns: ErrorCode

◆ getIOTensorAddress() [1/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOTensorAddress	(	AsciiChar const *const	tensorName,
		TypedArray &	tensor
	)

pure virtualnoexcept

This function gets the memory address for an user provided input tensor to the graph based on its name.

Precondition: INIT API

Parameters

tensorName	the tensor name that the user would like to obtain memory for.
tensor	set to a device memory pointer allocated by user for the tensor.

Returns: ErrorCode

◆ getIOTensorAddress() [2/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOTensorAddress	(	int32_t const	index,
		TypedArray &	tensor
	)

pure virtualnoexcept

This function gets the memory address for an user provided input tensor to the graph based on its index.

Precondition: INIT API

Parameters

index	the tensor index that the user would like to obtain memory for.
tensor	set to a device memory pointer allocated by user for the tensor.

Returns: ErrorCode

◆ getIOTensorDescriptor() [1/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOTensorDescriptor	(	TensorDescriptor &	desc,
		AsciiChar const *const	tensorName
	)		const

pure virtualnoexcept

This function should return a TensorDescriptor which contains all the information about the tensor based on the name.

Precondition: INIT API

Parameters

desc	corresponding TensorDescriptor that will be filled out after the call.
name	name of the tensor we are interested in.

Returns: ErrorCode

Warning: If the name does not correspond to a valid IO tensor, the function will fail with an error code of ErrorCode::kINVALID_ARGUMENT

◆ getIOTensorDescriptor() [2/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOTensorDescriptor	(	TensorDescriptor &	desc,
		int32_t const	index
	)		const

pure virtualnoexcept

This function should return a TensorDescriptor which contains all the information about the tensor based on the index.

Precondition: INIT API

Parameters

desc	corresponding TensorDescriptor that will be filled out after the call.
index	index of the tensor we are interested in (starting from 0).

Returns: ErrorCode

Warning: If the index does not fall between 0 and getNbIOTensors()-1, the function will fail with an error code of ErrorCode::kINVALID_ARGUMENT

◆ getIOTensorName()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getIOTensorName	(	AsciiChar const *&	name,
		size_t const	index
	)		const

pure virtualnoexcept

This function returns the name of a tensor for a given index.

Precondition: INIT API

Parameters

name	The name string of the tensor that will be filled out after the call.
index	The index of the tensor whose name is to be returned.

Returns: ErrorCode

◆ getNbAuxStreams()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getNbAuxStreams ( int32_t & nbStreams ) const

pure virtualnoexcept

Return the number of auxiliary streams used by this graph.

Precondition: INIT API

Parameters

nbStreams The return value for the number of auxiliary streams.

Returns: ErrorCode

See also: setAuxStreams()

◆ getNbIOProfiles()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getNbIOProfiles ( int64_t & nb ) const

pure virtualnoexcept

This function returns the total number of IO tensor profiles for the current graph.

Precondition: RUNTIME API

Parameters

nb	The return value for the total number of IO tensor profiles for the current graph. A graph will at least have 1 IOProfile

Returns: ErrorCode

◆ getNbIOTensors()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getNbIOTensors ( int64_t & nb ) const

pure virtualnoexcept

This function returns the total number of input and output tensor for the current graph.

Precondition: INIT API

Parameters

nb	The return value for total number of input and output tensor for the current graph.

Returns: ErrorCode

◆ getSafeRecorder()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getSafeRecorder ( ISafeRecorder *& recorder ) const

pure virtualnoexcept

This function retrieves the ISafeRecorder for the current graph.

Precondition: RUNTIME API

Parameters

recorder retrieved ISafeRecorder.

Returns: ErrorCode

◆ getScratchMemory()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getScratchMemory ( void *& memory )

pure virtualnoexcept

This function gets the scratch memory for the graph. This should only be called if scratch memory is not TRT managed (trtManagedScratch is false). An error will be recorded if called on a graph where trtManagedScratch is true.

Precondition: INIT API

Parameters

memory set to a device memory pointer allocated by user or nullptr when the scratch memory is TRT managed.

See also: getScratchMemorySize

Returns: ErrorCode

◆ getScratchMemorySize()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getScratchMemorySize ( size_t & size ) const

pure virtualnoexcept

This function returns the scratch memory size (in bytes) needed to store all the intermediate tensors for inference. The user could allocate a scratch memory of this size and pass it to TRT via setScratchMemory if not using trtManagedScratch.

Precondition: INIT API

See also: setScratchMemory

Parameters

size	The return value for the scratch memory size (in bytes) needed to store all the intermediate tensors for inference.

Returns: ErrorCode

◆ getTRTManagedScratch()

virtual ErrorCode nvinfer2::safe::ITRTGraph::getTRTManagedScratch ( bool & flag ) const

pure virtualnoexcept

This function returns the trtManagedScratch flag provided in createTRTGraph call.

Precondition: INIT API

See also: createTRTGraph

Parameters

flag	A boolean flag that determines if scratch memory is managed by TRT

Returns: ErrorCode

◆ operator=() [1/2]

ITRTGraph & nvinfer2::safe::ITRTGraph::operator= ( ITRTGraph && ) &

delete

◆ operator=() [2/2]

ITRTGraph & nvinfer2::safe::ITRTGraph::operator= ( ITRTGraph const & ) &

delete

◆ setAuxStreams()

virtual ErrorCode nvinfer2::safe::ITRTGraph::setAuxStreams	(	cudaStream_t *	auxStreams,
		int32_t	nbStreams
	)

pure virtualnoexcept

Set the auxiliary streams that TensorRT should use to run kernels on.

TRT will launch the kernels that are supposed to run on the auxiliary streams using the streams provided by the user via this API. The user is responsible for allocating and deallocating these streams.

If getNbAuxStreams() returns a value greater than 0, this API must be called before executeAsync() to provide the required auxiliary streams.

If getNbAuxStreams() returns 0, setAuxStreams() can only be called with an array of size 0.

Precondition: INIT API

Parameters

auxStreams	The pointer to an array of cudaStream_t with the array length equal to nbStreams. All streams in the array must be valid CUDA streams.
nbStreams	The number of auxiliary streams provided. Must be equal to the value returned by getNbAuxStreams(). If nbStreams does not match, kINVALID_ARGUMENT will be returned.

Returns: ErrorCode kSUCCESS on success, kINVALID_ARGUMENT if auxStreams contains nullptr, or if nbStreams does not match getNbAuxStreams().

See also: getNbAuxStreams()

◆ setInputConsumedEvent()

virtual ErrorCode nvinfer2::safe::ITRTGraph::setInputConsumedEvent ( cudaEvent_t event )

pure virtualnoexcept

This function sets a cudaEvent on the current graph that triggers when the input is consumed. At that point, the input memory can be recycled, i.e. new input for next inference can be loaded.

Precondition: INIT API

Parameters

event cuda event to be set

Returns: ErrorCode

◆ setIOProfile()

virtual ErrorCode nvinfer2::safe::ITRTGraph::setIOProfile ( int64_t profileIndex )

pure virtualnoexcept

This function selects the active IOProfile for the graph. If this function is not called, the TRTGraph will default to profile 0. Each IOProfile on the graph is mutually exclusive, meaning only one IOProfile can be active at a time.

Precondition: RUNTIME API

Parameters

profileIndex The index of the profile to select.

Returns: ErrorCode

Note: Other graph APIs are executed in the context of the current IOProfile id, so IOProfile id should be changed only on synchronized inference boundaries.

◆ setIOTensorAddress() [1/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::setIOTensorAddress	(	AsciiChar const *const	tensorName,
		TypedArray const &	tensor
	)

pure virtualnoexcept

This function assigns a user allocated device memory block for an input tensor to the graph based on its name.

Precondition: INIT API

Parameters

tensorName	the tensor name that the user would like to provide memory for.
tensor	a device memory block allocated by user for the tensor.

Returns: ErrorCode

◆ setIOTensorAddress() [2/2]

virtual ErrorCode nvinfer2::safe::ITRTGraph::setIOTensorAddress	(	int32_t const	index,
		TypedArray const &	tensor
	)

pure virtualnoexcept

This function assigns a user allocated device memory block for an input tensor to the graph based on its index.

Precondition: INIT API

Parameters

index	the tensor index that the user would like to provide memory for.
tensor	a device memory block allocated by user for the tensor.

Returns: ErrorCode

◆ setScratchMemory()

virtual ErrorCode nvinfer2::safe::ITRTGraph::setScratchMemory ( void * memory )

pure virtualnoexcept

This function sets the scratch memory for the graph. This should only be called if scratch memory is not TRT managed (trtManagedScratch is false). An error will be recorded if called on a graph where trtManagedScratch is true.

Precondition: INIT API

Parameters

memory pointer to a device memory block allocated by user that is at least as large as getScratchMemorySize, or nullptr which will reset the internal pointer.

See also: getScratchMemorySize

Returns: ErrorCode

Warning: User must guarantee that the allocated scratch memory is large enough.

◆ sync()

virtual ErrorCode nvinfer2::safe::ITRTGraph::sync ( )

pure virtualnoexcept

synchronize one inference of this graph.

Precondition: RUNTIME API

Returns

ErrorCode kSUCCESS on success, if any execution error occurred other error code might be returned. Errors may include but not be limited to:

Internal errors during executing one engine layer (host side)
CUDA errors
Some input or output tensor addresses have not been set.

The documentation for this class was generated from the following file:

NvInferSafeRuntime.h

Public Member Functions

Protected Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ ITRTGraph() [1/3]

◆ ITRTGraph() [2/3]

◆ ~ITRTGraph()

◆ ITRTGraph() [3/3]

Member Function Documentation

◆ clone()

◆ executeAsync()

◆ getErrorBuffer()

◆ getInputConsumedEvent()

◆ getIOProfile()

◆ getIOTensorAddress() [1/2]

◆ getIOTensorAddress() [2/2]

◆ getIOTensorDescriptor() [1/2]

◆ getIOTensorDescriptor() [2/2]

◆ getIOTensorName()

◆ getNbAuxStreams()

◆ getNbIOProfiles()

◆ getNbIOTensors()

◆ getSafeRecorder()

◆ getScratchMemory()

◆ getScratchMemorySize()

◆ getTRTManagedScratch()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ setAuxStreams()

◆ setInputConsumedEvent()

◆ setIOProfile()

◆ setIOTensorAddress() [1/2]

◆ setIOTensorAddress() [2/2]

◆ setScratchMemory()

◆ sync()