A functionally safe engine for executing inference on a built network. More...

#include <NvInferSafeRuntime.h>

Public Member Functions
virtual IExecutionContext *	createExecutionContext () noexcept=0
	Create an execution context. More...

virtual IExecutionContext *	createExecutionContextWithoutDeviceMemory () noexcept=0
	Create an execution context without any device memory allocated. More...

virtual size_t	getDeviceMemorySize () const noexcept=0
	Return the amount of device memory required by an execution context. More...

virtual AsciiChar const *	getName () const noexcept=0
	Returns the name of the network associated with the engine. More...

virtual void	setErrorRecorder (IErrorRecorder *const recorder) noexcept=0
	Set the ErrorRecorder for this interface. More...

virtual IErrorRecorder *	getErrorRecorder () const noexcept=0
	Get the ErrorRecorder assigned to this interface. More...

	ICudaEngine ()=default

virtual	~ICudaEngine () noexcept=default

	ICudaEngine (ICudaEngine const &)=delete

	ICudaEngine (ICudaEngine &&)=delete

ICudaEngine &	operator= (ICudaEngine const &) &=delete

ICudaEngine &	operator= (ICudaEngine &&) &=delete

virtual Dims	getTensorShape (AsciiChar const *const tensorName) const noexcept=0
	Get the extent of an input or output tensor. More...

virtual DataType	getTensorDataType (AsciiChar const *const tensorName) const noexcept=0
	Determine the required data type for a buffer from its tensor name. More...

virtual TensorIOMode	getTensorIOMode (AsciiChar const *const tensorName) const noexcept=0
	Determine whether a tensor is an input or output tensor. More...

virtual std::int32_t	getTensorBytesPerComponent (AsciiChar const *const tensorName) const noexcept=0
	Return the size of the tensor data type in bytes for a vectorized tensor. More...

virtual std::int32_t	getTensorComponentsPerElement (AsciiChar const *const tensorName) const noexcept=0
	Return the number of components included in one element for a vectorized tensor. More...

virtual TensorFormat	getTensorFormat (AsciiChar const *const tensorName) const noexcept=0
	Return the tensor format. More...

virtual std::int32_t	getTensorVectorizedDim (AsciiChar const *const tensorName) const noexcept=0
	Return the dimension index along which the buffer is vectorized. More...

virtual std::int32_t	getNbIOTensors () const noexcept=0
	Return the number of input and output tensors for the network from which the engine was built. More...

virtual AsciiChar const *	getIOTensorName (std::int32_t const index) const noexcept=0
	Return the name of an IO tensor. More...

Detailed Description

A functionally safe engine for executing inference on a built network.

Warning: Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Constructor & Destructor Documentation

◆ ICudaEngine() [1/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( )

default

◆ ~ICudaEngine()

virtual nvinfer1::safe::ICudaEngine::~ICudaEngine ( )

virtualdefaultnoexcept

◆ ICudaEngine() [2/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( ICudaEngine const & )

delete

◆ ICudaEngine() [3/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( ICudaEngine && )

delete

Member Function Documentation

◆ createExecutionContext()

virtual IExecutionContext * nvinfer1::safe::ICudaEngine::createExecutionContext ( )

pure virtualnoexcept

Create an execution context.

See also: safe::IExecutionContext.

Returns: An execution context object if it can be constructed, or nullptr if the construction fails.

Reasons for failure may include but not be limited to:

Heap memory exhaustion
Device memory exhaustion

Usage considerations

Allowed context for the API call
- Thread-safe: Yes; if createExecutionContext fails, users must treat this as a critical error and not perform any subsequent TensorRT operations apart from outputting the error logs.

◆ createExecutionContextWithoutDeviceMemory()

virtual IExecutionContext * nvinfer1::safe::ICudaEngine::createExecutionContextWithoutDeviceMemory ( )

pure virtualnoexcept

Create an execution context without any device memory allocated.

The memory for execution of this device context must be supplied by the application by calling safe::IExecutionContext::setDeviceMemory().

See also: getDeviceMemorySize() safe::IExecutionContext::setDeviceMemory()

Returns: An execution context object if it can be constructed, or nullptr if the construction fails.

Reasons for failure may include but not be limited to heap memory exhaustion.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes; if createExecutionContext fails, users must treat this as a critical error and not perform any subsequent TensorRT operations apart from outputting the error logs.

◆ getDeviceMemorySize()

virtual size_t nvinfer1::safe::ICudaEngine::getDeviceMemorySize ( ) const

pure virtualnoexcept

Return the amount of device memory required by an execution context.

See also: safe::IExecutionContext::setDeviceMemory()

Returns: Size of a contiguous memory buffer (in bytes) that users need to provide to safe::IExecutionContext::setDeviceMemory() if the execution context has been created by calling createExecutionContextWithoutDeviceMemory().

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getErrorRecorder()

virtual IErrorRecorder * nvinfer1::safe::ICudaEngine::getErrorRecorder ( ) const

pure virtualnoexcept

Get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error reporter has not been inherited from the IRuntime, and setErrorReporter() has not been called.

Returns: A pointer to the IErrorRecorder object that has been registered, or nullptr if none has been registered.

See also: setErrorRecorder()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getIOTensorName()

virtual AsciiChar const * nvinfer1::safe::ICudaEngine::getIOTensorName ( std::int32_t const index ) const

pure virtualnoexcept

Return the name of an IO tensor.

If the index does not fall between 0 and getNbIOTensors()-1, the function will fail with an error code of ErrorCode::kINVALID_ARGUMENT(3) that is emitted to the registered IErrorRecorder.

Parameters

index The IO tensor index.

Returns: The name of an IO tensor, which will be a NULL-terminated string of 1024 bytes or less (including the NULL terminator) if the index is in the range (between 0 and getNbIOTensors()-1). nullptr will be returned if the index is not in range.

See also: getNbIOTensors()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getName()

virtual AsciiChar const * nvinfer1::safe::ICudaEngine::getName ( ) const

pure virtualnoexcept

Returns the name of the network associated with the engine.

The name is set during network creation and is retrieved after building or deserialization.

See also: INetworkDefinition::setName(), INetworkDefinition::getName()

Returns: A NULL-terminated C-style string representing the name of the network, which will have a length of 1024 bytes or less including the NULL terminator.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getNbIOTensors()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getNbIOTensors ( ) const

pure virtualnoexcept

Return the number of input and output tensors for the network from which the engine was built.

Returns: The number of IO tensors.

See also: getIOTensorName()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorBytesPerComponent()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorBytesPerComponent ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Return the size of the tensor data type in bytes for a vectorized tensor.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

The size of the tensor data type in bytes if the tensor is vectorized (4 for float and int32, 2 for half, 1 for int8). 0 will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit, or
the tensor of the given name is not vectorized.

See also: safe::ICudaEngine::getTensorVectorizedDim()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorComponentsPerElement()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorComponentsPerElement ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Return the number of components included in one element for a vectorized tensor.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

The vector length (in scalars) for a vectorized tensor, or 1 for a scalar tensor. The invalid value -1 will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit.

See also: safe::ICudaEngine::getTensorVectorizedDim()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorDataType()

virtual DataType nvinfer1::safe::ICudaEngine::getTensorDataType ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Determine the required data type for a buffer from its tensor name.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

The type of the data in the buffer. The default value DataType::kFLOAT will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorFormat()

virtual TensorFormat nvinfer1::safe::ICudaEngine::getTensorFormat ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Return the tensor format.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

The tensor format. TensorFormat::kLINEAR will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorIOMode()

virtual TensorIOMode nvinfer1::safe::ICudaEngine::getTensorIOMode ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Determine whether a tensor is an input or output tensor.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

kINPUT if tensorName is the name of an input tensor, kOUTPUT if tensorName is the name of an output tensor. The invalid value kNONE is returned if

tensorName exceeds the string length limit, or
tensorName is nullptr, or
tensorName does not correspond to any input or output tensor.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorShape()

virtual Dims nvinfer1::safe::ICudaEngine::getTensorShape ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Get the extent of an input or output tensor.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

Extent of the tensor. The invalid value Dims{-1, {}} will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorVectorizedDim()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorVectorizedDim ( AsciiChar const *const tensorName ) const

pure virtualnoexcept

Return the dimension index along which the buffer is vectorized.

Specifically, -1 is returned if the tensor is scalar.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.

Returns

The dimension index along which the buffer is vectorized. -1 will be returned if

name is not the name of an input or output tensor, or
name is nullptr, or
name exceeds the string length limit (1024 bytes or less including the NULL terminator), or
the tensor of given name is not vectorized.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ operator=() [1/2]

ICudaEngine & nvinfer1::safe::ICudaEngine::operator= ( ICudaEngine && ) &

delete

◆ operator=() [2/2]

ICudaEngine & nvinfer1::safe::ICudaEngine::operator= ( ICudaEngine const & ) &

delete

◆ setErrorRecorder()

virtual void nvinfer1::safe::ICudaEngine::setErrorRecorder ( IErrorRecorder *const recorder )

pure virtualnoexcept

Set the ErrorRecorder for this interface.

Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function will call incRefCount of the registered ErrorRecorder at least once. If the recorder is set to nullptr, the error code ErrorCode::kINVALID_ARGUMENT will be emitted if the recorder has been registered.

Parameters

recorder The error recorder to register with this interface, or nullptr to deregister the current. error recorder.

See also: getErrorRecorder()

Usage considerations

Allowed context for the API call
- Thread-safe: No

The documentation for this class was generated from the following file:

NvInferSafeRuntime.h

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ ICudaEngine() [1/3]

◆ ~ICudaEngine()

◆ ICudaEngine() [2/3]

◆ ICudaEngine() [3/3]

Member Function Documentation

◆ createExecutionContext()

◆ createExecutionContextWithoutDeviceMemory()

◆ getDeviceMemorySize()

◆ getErrorRecorder()

◆ getIOTensorName()

◆ getName()

◆ getNbIOTensors()

◆ getTensorBytesPerComponent()

◆ getTensorComponentsPerElement()

◆ getTensorDataType()

◆ getTensorFormat()

◆ getTensorIOMode()

◆ getTensorShape()

◆ getTensorVectorizedDim()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ setErrorRecorder()