Functionally safe context for executing inference using an engine.
More...
#include <NvInferSafeRuntime.h>
Functionally safe context for executing inference using an engine.
Multiple safe execution contexts may exist for one safe::ICudaEngine instance, allowing the same engine to be used for the execution of multiple inputs simultaneously.
- Warning
- Do not call the APIs of the same IExecutionContext from multiple threads at any given time. Each concurrent execution must have its own instance of an IExecutionContext.
-
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
◆ IExecutionContext() [1/3]
| nvinfer1::safe::IExecutionContext::IExecutionContext |
( |
| ) |
|
|
default |
◆ ~IExecutionContext()
| virtual nvinfer1::safe::IExecutionContext::~IExecutionContext |
( |
| ) |
|
|
virtualdefaultnoexcept |
◆ IExecutionContext() [2/3]
◆ IExecutionContext() [3/3]
◆ enqueueV3()
| virtual bool nvinfer1::safe::IExecutionContext::enqueueV3 |
( |
cudaStream_t const |
stream | ) |
|
|
pure virtualnoexcept |
Enqueue inference on a stream.
Modifying or releasing memory that has been registered for the tensors before stream synchronization or the event passed to setInputConsumedEvent has been signaled results in undefined behavior.
- Parameters
-
| stream | A CUDA stream on which the inference kernels will be enqueued. |
- Returns
- True on success, false if any execution error occurred. Errors may include but not be limited to:
- Internal errors during executing one engine layer
- CUDA errors
- Some input or output tensor addresses have not been set.
Usage considerations
- Allowed context for the API call
◆ getEngine()
| virtual ICudaEngine const & nvinfer1::safe::IExecutionContext::getEngine |
( |
| ) |
const |
|
pure virtualnoexcept |
Get the associated engine.
- See also
- safe::ICudaEngine
Usage considerations
- Allowed context for the API call
◆ getErrorBuffer()
Get error buffer output for runtime errors.
- Returns
- Pointer to device memory to use as runtime error buffer or nullptr if not set.
- See also
- setErrorBuffer()
Usage considerations
- Allowed context for the API call
◆ getErrorRecorder()
| virtual IErrorRecorder * nvinfer1::safe::IExecutionContext::getErrorRecorder |
( |
| ) |
const |
|
pure virtualnoexcept |
get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will be returned if setErrorRecorder has not been called.
- Returns
- A pointer to the IErrorRecorder object that has been registered, or nullptr if the error recorder has been deregistered or not set.
- See also
- setErrorRecorder()
Usage considerations
- Allowed context for the API call
◆ getInputConsumedEvent()
| virtual cudaEvent_t nvinfer1::safe::IExecutionContext::getInputConsumedEvent |
( |
| ) |
const |
|
pure virtualnoexcept |
Return the event associated with consuming the input.
- Returns
- The CUDA event that was passed to setInputConsumedEvent(). nullptr will be returned if the event is not set.
Usage considerations
- Allowed context for the API call
◆ getInputTensorAddress()
| virtual void const * nvinfer1::safe::IExecutionContext::getInputTensorAddress |
( |
AsciiChar const *const |
tensorName | ) |
const |
|
pure virtualnoexcept |
Get memory address for given input tensor.
- Parameters
-
| tensorName | The name of an input tensor. |
- Warning
- The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
- Returns
- The device memory address for the given input tensor. nullptr will be returned if
- name is not the name of an input tensor, or
- name is nullptr, or
- name exceeds the string length limit, or
- the memory address for the given input tensor is not set.
Usage considerations
- Allowed context for the API call
◆ getName()
| virtual AsciiChar const * nvinfer1::safe::IExecutionContext::getName |
( |
| ) |
const |
|
pure virtualnoexcept |
Return the name of the execution context.
- Returns
- The name that was passed to setName(), as a NULL-terminated string of 1024 bytes or less including the NULL terminator. An empty string will be returned as the default value.
- See also
- setName()
Usage considerations
- Allowed context for the API call
◆ getOutputTensorAddress()
| virtual void * nvinfer1::safe::IExecutionContext::getOutputTensorAddress |
( |
AsciiChar const *const |
tensorName | ) |
const |
|
pure virtualnoexcept |
Get memory address for given output tensor.
- Parameters
-
| tensorName | The name of an output tensor. |
- Warning
- The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
- Returns
- The device memory address for the given output tensor. Return nullptr if
- name is not the name of an output tensor, or
- name is nullptr, or
- name exceeds the string length limit, or
- the memory address for the given output tensor is not set.
Usage considerations
- Allowed context for the API call
◆ getTensorStrides()
| virtual Dims nvinfer1::safe::IExecutionContext::getTensorStrides |
( |
AsciiChar const *const |
tensorName | ) |
const |
|
pure virtualnoexcept |
Return the strides of the buffer for the given tensor name.
The strides are in units of elements, not components or bytes. Elements are vectors (for a vectorized format) or scalars (for a scalar format). For example, for TensorFormat::kHWC8, a stride of one spans 8 scalars.
- Parameters
-
| tensorName | The name of an input or output tensor. |
- Warning
- The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
- Returns
- The strides of the buffer for the given tensor name. Dims{-1, {}} will be returned if
- name is not the name of an input or output tensor, or
- name is nullptr, or
- name exceeds the string length limit.
Usage considerations
- Allowed context for the API call
◆ operator=() [1/2]
◆ operator=() [2/2]
◆ setDeviceMemory()
| virtual void nvinfer1::safe::IExecutionContext::setDeviceMemory |
( |
void *const |
memory | ) |
|
|
pure virtualnoexcept |
Set the device memory for use by this execution context.
- Parameters
-
| memory | The start address of a device memory buffer whose size in bytes must be at least the value returned by getEngine().getDeviceMemorySize(). |
If using enqueueV2() to run the network, The memory is in use from the invocation of enqueueV2() until network execution is complete. Releasing or otherwise using the memory for other purposes during this time will result in undefined behavior.
- Warning
- Do not release or use for other purposes the memory set here during network execution.
-
If the execution context has been created by calling createExecutionContext(), this function must not be used and will fail with an error message if called.
- See also
- safe::ICudaEngine::getDeviceMemorySize() safe::ICudaEngine::createExecutionContextWithoutDeviceMemory()
Usage considerations
- Allowed context for the API call
◆ setErrorBuffer()
Set error buffer output for runtime errors.
The error buffer output must be allocated in device memory and will be used for subsequent calls to enqueueV2() or enqueueV3(). Checking the contents of the error buffer after inference is the responsibility of the application. The pointer passed here must have alignment adequate for the RuntimeErrorInformation struct.
- Warning
- The buffer is written if reportable errors are encountered during network execution. Releasing the buffer before network execution is complete will result in undefined behavior. Accessing the memory before network execution is complete may not correctly capture the error state.
- Parameters
-
| buffer | The device memory address of the runtime error information buffer. |
- See also
- getErrorBuffer()
Usage considerations
- Allowed context for the API call
◆ setErrorRecorder()
| virtual void nvinfer1::safe::IExecutionContext::setErrorRecorder |
( |
IErrorRecorder *const |
recorder | ) |
|
|
pure virtualnoexcept |
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function will call incRefCount of the registered ErrorRecorder at least once. If the recorder is set to nullptr, the error code ErrorCode::kINVALID_ARGUMENT will be emitted if the recorder has been registered. The lifetime of the error recorder object must exceed the lifetime of the execution context.
- Parameters
-
| recorder | Either a pointer to a valid error recorder object to register with this interface, or nullptr to deregister the current recorder. |
- See also
- getErrorRecorder()
Usage considerations
- Allowed context for the API call
◆ setInputConsumedEvent()
| virtual bool nvinfer1::safe::IExecutionContext::setInputConsumedEvent |
( |
cudaEvent_t const |
event | ) |
|
|
pure virtualnoexcept |
Set the event to mark inputs as consumed.
Passing event==nullptr removes whatever event was set, if any.
- Parameters
-
| event | The CUDA event that is signaled after all input tensors have been consumed, or nullptr to remove an event that was previously set. |
- Returns
- True on success, false if an error occurred.
Usage considerations
- Allowed context for the API call
◆ setInputTensorAddress()
| virtual bool nvinfer1::safe::IExecutionContext::setInputTensorAddress |
( |
AsciiChar const *const |
tensorName, |
|
|
void const *const |
data |
|
) |
| |
|
pure virtualnoexcept |
Set memory address for given input tensor.
An address defaults to nullptr.
Before calling enqueueV3(), each input must have a non-null address.
- Parameters
-
| tensorName | The name of an input tensor. |
| data | The pointer (void const*) to the input tensor data, which is device memory owned by the user. Users are responsible for ensuring that the buffer size has at least the expected length, which is the product of the tensor dimensions (with the vectorized dimension padded to a multiple of the vector length) times the data type size. |
- Warning
- The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
-
The data pointer must have 256-byte alignment.
- Returns
- True on success, false if
- name is not the name of an input tensor, or
- name is nullptr, or
- name exceeds the string length limit, or
- pointer to the const data is nullptr or not correctly aligned.
Usage considerations
- Allowed context for the API call
◆ setName()
| virtual void nvinfer1::safe::IExecutionContext::setName |
( |
AsciiChar const *const |
name | ) |
|
|
pure virtualnoexcept |
Set the name of the execution context.
This method copies the name string.
- Warning
- Strings passed to the runtime must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator. Otherwise, the operation will not change the execution context name, and an error message will be recorded via the error recorder.
- See also
- getName()
Usage considerations
- Allowed context for the API call
◆ setOutputTensorAddress()
| virtual bool nvinfer1::safe::IExecutionContext::setOutputTensorAddress |
( |
AsciiChar const *const |
tensorName, |
|
|
void *const |
data |
|
) |
| |
|
pure virtualnoexcept |
Set memory address for given output tensor.
An address defaults to nullptr.
Before calling enqueueV3(), each output must have a non-null address.
- Parameters
-
| tensorName | The name of an output tensor. |
| data | The pointer (void*) to the output tensor data, which is device memory owned by the user. Users are responsible for ensuring that the buffer size has at least the expected length, which is the product of the tensor dimensions (with the vectorized dimension padded to a multiple of the vector length) times the data type size. |
- Warning
- The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
-
The data pointer must have 256-byte alignment.
- Returns
- True on success. Return false if
- name is not the name of an output tensor, or
- name is nullptr, or
- name exceeds the string length limit, or
- pointer to data is nullptr or not aligned.
Usage considerations
- Allowed context for the API call
The documentation for this class was generated from the following file:
1.9.2