Functionally safe context for executing inference using an engine. More...

#include <NvInferSafeRuntime.h>

Public Member Functions
virtual ICudaEngine const &	getEngine () const noexcept=0
	Get the associated engine. More...

virtual void	setName (AsciiChar const *const name) noexcept=0
	Set the name of the execution context. More...

virtual AsciiChar const *	getName () const noexcept=0
	Return the name of the execution context. More...

virtual void	setDeviceMemory (void *const memory) noexcept=0
	Set the device memory for use by this execution context. More...

virtual TRT_DEPRECATED Dims	getStrides (std::int32_t const bindingIndex) const noexcept=0
	Return the strides of the buffer for the given binding. More...

virtual void	setErrorRecorder (IErrorRecorder *const recorder) noexcept=0
	Set the ErrorRecorder for this interface. More...

virtual IErrorRecorder *	getErrorRecorder () const noexcept=0
	get the ErrorRecorder assigned to this interface. More...

virtual TRT_DEPRECATED bool	enqueueV2 (void const const bindings, cudaStream_t const stream, cudaEvent_t const *const inputConsumed) noexcept=0
	Enqueue inference of a batch on a stream. More...

	IExecutionContext ()=default

virtual	~IExecutionContext () noexcept=default

	IExecutionContext (IExecutionContext const &)=delete

	IExecutionContext (IExecutionContext &&)=delete

IExecutionContext &	operator= (IExecutionContext const &) &=delete

IExecutionContext &	operator= (IExecutionContext &&) &=delete

virtual void	setErrorBuffer (FloatingPointErrorInformation *const buffer) noexcept=0
	Set error buffer output for floating point errors. More...

virtual FloatingPointErrorInformation *	getErrorBuffer () const noexcept=0
	Get error buffer output for floating point errors. More...

virtual Dims	getTensorStrides (AsciiChar const *tensorName) const noexcept=0
	Return the strides of the buffer for the given tensor name. More...

virtual bool	setInputTensorAddress (AsciiChar const tensorName, void const data) noexcept=0
	Set memory address for given input tensor. More...

virtual bool	setOutputTensorAddress (AsciiChar const tensorName, void data) noexcept=0
	Set memory address for given output tensor. More...

virtual bool	setInputConsumedEvent (cudaEvent_t event) noexcept=0
	Mark input as consumed. More...

virtual cudaEvent_t	getInputConsumedEvent () const noexcept=0
	Return the event associated with consuming the input. More...

virtual void const *	getInputTensorAddress (AsciiChar const *tensorName) const noexcept=0
	Get memory address for given input tensor. More...

virtual void *	getOutputTensorAddress (AsciiChar const *tensorName) const noexcept=0
	Get memory address for given output tensor. More...

virtual bool	enqueueV3 (cudaStream_t stream) noexcept=0
	Enqueue inference on a stream. More...

Detailed Description

Functionally safe context for executing inference using an engine.

Multiple safe execution contexts may exist for one safe::ICudaEngine instance, allowing the same engine to be used for the execution of multiple inputs simultaneously.

Warning: Do not call the APIs of the same IExecutionContext from multiple threads at any given time. Each concurrent execution must have its own instance of an IExecutionContext.; Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Constructor & Destructor Documentation

◆ IExecutionContext() [1/3]

nvinfer1::safe::IExecutionContext::IExecutionContext ( )

default

◆ ~IExecutionContext()

virtual nvinfer1::safe::IExecutionContext::~IExecutionContext ( )

virtualdefaultnoexcept

◆ IExecutionContext() [2/3]

nvinfer1::safe::IExecutionContext::IExecutionContext ( IExecutionContext const & )

delete

◆ IExecutionContext() [3/3]

nvinfer1::safe::IExecutionContext::IExecutionContext ( IExecutionContext && )

delete

Member Function Documentation

◆ enqueueV2()

virtual TRT_DEPRECATED bool nvinfer1::safe::IExecutionContext::enqueueV2	(	void const const	bindings,
		cudaStream_t const	stream,
		cudaEvent_t const *const	inputConsumed
	)

pure virtualnoexcept

Enqueue inference of a batch on a stream.

This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried using safe::ICudaEngine::getBindingIndex(). This method only works for an execution context built from a network without an implicit batch dimension.

Parameters

bindings	An array of pointers to input and output buffers for the network.
stream	A cuda stream on which the inference kernels will be enqueued.
inputConsumed	An optional event which will be signaled when the input buffers can be refilled with new data.

Returns: True if the kernels were enqueued successfully.

Deprecated:: Deprecated in TensorRT 8.5. Superseded by enqueueV3().

See also: safe::IExecutionContext::enqueueV3()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ enqueueV3()

virtual bool nvinfer1::safe::IExecutionContext::enqueueV3 ( cudaStream_t stream )

pure virtualnoexcept

Enqueue inference on a stream.

Modifying or releasing memory that has been registered for the tensors before stream synchronization or the event passed to setInputConsumedEvent has been being triggered results in undefined behavior.

Parameters

stream A cuda stream on which the inference kernels will be enqueued.

Returns: True on success, false if any execution error occurred.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getEngine()

virtual ICudaEngine const & nvinfer1::safe::IExecutionContext::getEngine ( ) const

pure virtualnoexcept

Get the associated engine.

See also: safe::ICudaEngine

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getErrorBuffer()

virtual FloatingPointErrorInformation * nvinfer1::safe::IExecutionContext::getErrorBuffer ( ) const

pure virtualnoexcept

Get error buffer output for floating point errors.

Returns: Pointer to device memory to use as floating point error buffer or nullptr if not set.

See also: setErrorBuffer()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getErrorRecorder()

virtual IErrorRecorder * nvinfer1::safe::IExecutionContext::getErrorRecorder ( ) const

pure virtualnoexcept

get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will be returned if setErrorRecorder has not been called.

Returns: A pointer to the IErrorRecorder object that has been registered.

See also: setErrorRecorder()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ getInputConsumedEvent()

virtual cudaEvent_t nvinfer1::safe::IExecutionContext::getInputConsumedEvent ( ) const

pure virtualnoexcept

Return the event associated with consuming the input.

Returns: The cuda event, nullptr will be returned if the event is not set yet.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getInputTensorAddress()

virtual void const * nvinfer1::safe::IExecutionContext::getInputTensorAddress ( AsciiChar const * tensorName ) const

pure virtualnoexcept

Get memory address for given input tensor.

Parameters

tensorName The name of an input tensor.

Warning: The string tensorName must be 1024 characters or less including NULL terminator and must be NULL terminated.

Returns: The memory address for the given input tensor. nullptr will be returned if (1) name is not the name of an input tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) the memory address for the given input tensor is not set yet.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getName()

virtual AsciiChar const * nvinfer1::safe::IExecutionContext::getName ( ) const

pure virtualnoexcept

Return the name of the execution context.

See also: setName()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ getOutputTensorAddress()

virtual void * nvinfer1::safe::IExecutionContext::getOutputTensorAddress ( AsciiChar const * tensorName ) const

pure virtualnoexcept

Get memory address for given output tensor.

Parameters

tensorName The name of an output tensor.

Warning: The string tensorName must be 1024 characters or less including NULL terminator and must be NULL terminated.

Returns: Raw output data pointer (void*) for given output tensor, return nullptr if (1) name is not the name of an output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) the memory address for the given output tensor is not set yet.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getStrides()

virtual TRT_DEPRECATED Dims nvinfer1::safe::IExecutionContext::getStrides ( std::int32_t const bindingIndex ) const

pure virtualnoexcept

Return the strides of the buffer for the given binding.

Parameters

bindingIndex The binding index.

Deprecated:: Deprecated in TensorRT 8.5. Superseded by getTensorStrides().

See also: safe::IExecutionContext::getTensorStrides()

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ getTensorStrides()

virtual Dims nvinfer1::safe::IExecutionContext::getTensorStrides ( AsciiChar const * tensorName ) const

pure virtualnoexcept

Return the strides of the buffer for the given tensor name.

The strides are in units of elements, not components or bytes. For example, for TensorFormat::kHWC8, a stride of one spans 8 scalars.

Parameters

tensorName The name of an input or output tensor.

Warning: The string tensorName must be 1024 characters or less including NULL terminator and must be NULL terminated.

Returns: The strides of the buffer for the given tensor name. Dims{-1, {}} will be returned if (1) name is not the name of an input or output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit.

Usage considerations

Allowed context for the API call
- Thread-safe: Yes

◆ operator=() [1/2]

IExecutionContext & nvinfer1::safe::IExecutionContext::operator= ( IExecutionContext && ) &

delete

◆ operator=() [2/2]

IExecutionContext & nvinfer1::safe::IExecutionContext::operator= ( IExecutionContext const & ) &

delete

◆ setDeviceMemory()

virtual void nvinfer1::safe::IExecutionContext::setDeviceMemory ( void *const memory )

pure virtualnoexcept

Set the device memory for use by this execution context.

If using enqueueV2() to run the network, The memory is in use from the invocation of enqueueV2() until network execution is complete. Releasing or otherwise using the memory for other purposes during this time will result in undefined behavior.

Warning: Do not release or use for other purposes the memory set here during network execution.

See also: safe::ICudaEngine::getDeviceMemorySize() safe::ICudaEngine::createExecutionContextWithoutDeviceMemory()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setErrorBuffer()

virtual void nvinfer1::safe::IExecutionContext::setErrorBuffer ( FloatingPointErrorInformation *const buffer )

pure virtualnoexcept

Set error buffer output for floating point errors.

The error buffer output must be allocated in device memory and will be used for subsequent calls to enqueueV2. Checking the contents of the error buffer after inference is the responsibility of the application. The pointer passed here must have alignment adequate for the FloatingPointErrorInformation struct.

Warning: Do not release or use the contents of the error buffer for any other purpose before synchronizing on the CUDA stream passed to enqueueV2.

Parameters

buffer The device memory to use as floating point error buffer

See also: getErrorBuffer()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setErrorRecorder()

virtual void nvinfer1::safe::IExecutionContext::setErrorRecorder ( IErrorRecorder *const recorder )

pure virtualnoexcept

Set the ErrorRecorder for this interface.

Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if a recorder has been registered.

Parameters

recorder The error recorder to register with this interface.

See also: getErrorRecorder()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setInputConsumedEvent()

virtual bool nvinfer1::safe::IExecutionContext::setInputConsumedEvent ( cudaEvent_t event )

pure virtualnoexcept

Mark input as consumed.

Passing event==nullptr removes whatever event was set, if any.

Parameters

event The cuda event that is triggered after all input tensors have been consumed.

Returns: True on success, false if error occurred.

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setInputTensorAddress()

virtual bool nvinfer1::safe::IExecutionContext::setInputTensorAddress	(	AsciiChar const *	tensorName,
		void const *	data
	)

pure virtualnoexcept

Set memory address for given input tensor.

An address defaults to nullptr.

Before calling enqueueV3(), each input must have a non-null address.

Parameters

tensorName	The name of an input tensor.
data	The pointer (void const*) to the const data owned by the user.

Warning: The string tensorName must be 1024 characters or less including NULL terminator and must be NULL terminated.; The pointer must have at least 256-byte alignment.

Returns: True on success, false if (1) name is not the name of an input tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) pointer to the const data is nullptr or not aligned.

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setName()

virtual void nvinfer1::safe::IExecutionContext::setName ( AsciiChar const *const name )

pure virtualnoexcept

Set the name of the execution context.

This method copies the name string.

Warning: Strings passed to the runtime must be 1024 characters or less including NULL terminator and must be NULL terminated.

See also: getName()

Usage considerations

Allowed context for the API call
- Thread-safe: No

◆ setOutputTensorAddress()

virtual bool nvinfer1::safe::IExecutionContext::setOutputTensorAddress	(	AsciiChar const *	tensorName,
		void *	data
	)

pure virtualnoexcept

Set memory address for given output tensor.

An address defaults to nullptr.

Before calling enqueueV3(), each output must have a non-null address.

Parameters

tensorName	The name of an output tensor.
data	The pointer (void*) to the data owned by the user.

Warning: The string tensorName must be 1024 characters or less including NULL terminator and must be NULL terminated.; The pointer must have at least 256-byte alignment.

Returns: True on success. Return false if (1) name is not the name of an output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) pointer to data is nullptr or not aligned.

Usage considerations

Allowed context for the API call
- Thread-safe: No

The documentation for this class was generated from the following file:

NvInferSafeRuntime.h

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ IExecutionContext() [1/3]

◆ ~IExecutionContext()

◆ IExecutionContext() [2/3]

◆ IExecutionContext() [3/3]

Member Function Documentation

◆ enqueueV2()

◆ enqueueV3()

◆ getEngine()

◆ getErrorBuffer()

◆ getErrorRecorder()

◆ getInputConsumedEvent()

◆ getInputTensorAddress()

◆ getName()

◆ getOutputTensorAddress()

◆ getStrides()

◆ getTensorStrides()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ setDeviceMemory()

◆ setErrorBuffer()

◆ setErrorRecorder()

◆ setInputConsumedEvent()

◆ setInputTensorAddress()

◆ setName()

◆ setOutputTensorAddress()