TensorRT 10.0.1
nvinfer1::safe::ICudaEngine Class Referenceabstract

A functionally safe engine for executing inference on a built network. More...

#include <NvInferSafeRuntime.h>

Public Member Functions

virtual IExecutionContextcreateExecutionContext () noexcept=0
 Create an execution context. More...
 
virtual IExecutionContextcreateExecutionContextWithoutDeviceMemory () noexcept=0
 Create an execution context without any device memory allocated. More...
 
virtual size_t getDeviceMemorySize () const noexcept=0
 Return the amount of device memory required by an execution context. More...
 
virtual AsciiChar const * getName () const noexcept=0
 Returns the name of the network associated with the engine. More...
 
virtual void setErrorRecorder (IErrorRecorder *const recorder) noexcept=0
 Set the ErrorRecorder for this interface. More...
 
virtual IErrorRecordergetErrorRecorder () const noexcept=0
 Get the ErrorRecorder assigned to this interface. More...
 
 ICudaEngine ()=default
 
virtual ~ICudaEngine () noexcept=default
 
 ICudaEngine (ICudaEngine const &)=delete
 
 ICudaEngine (ICudaEngine &&)=delete
 
ICudaEngineoperator= (ICudaEngine const &) &=delete
 
ICudaEngineoperator= (ICudaEngine &&) &=delete
 
virtual Dims getTensorShape (AsciiChar const *const tensorName) const noexcept=0
 Get the extent of an input or output tensor. More...
 
virtual DataType getTensorDataType (AsciiChar const *const tensorName) const noexcept=0
 Determine the required data type for a buffer from its tensor name. More...
 
virtual TensorIOMode getTensorIOMode (AsciiChar const *const tensorName) const noexcept=0
 Determine whether a tensor is an input or output tensor. More...
 
virtual std::int32_t getTensorBytesPerComponent (AsciiChar const *const tensorName) const noexcept=0
 Return the size of the tensor data type in bytes for a vectorized tensor. More...
 
virtual std::int32_t getTensorComponentsPerElement (AsciiChar const *const tensorName) const noexcept=0
 Return the number of components included in one element for a vectorized tensor. More...
 
virtual TensorFormat getTensorFormat (AsciiChar const *const tensorName) const noexcept=0
 Return the tensor format. More...
 
virtual std::int32_t getTensorVectorizedDim (AsciiChar const *const tensorName) const noexcept=0
 Return the dimension index along which the buffer is vectorized. More...
 
virtual std::int32_t getNbIOTensors () const noexcept=0
 Return the number of input and output tensors for the network from which the engine was built. More...
 
virtual AsciiChar const * getIOTensorName (std::int32_t const index) const noexcept=0
 Return the name of an IO tensor. More...
 

Detailed Description

A functionally safe engine for executing inference on a built network.

Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Constructor & Destructor Documentation

◆ ICudaEngine() [1/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( )
default

◆ ~ICudaEngine()

virtual nvinfer1::safe::ICudaEngine::~ICudaEngine ( )
virtualdefaultnoexcept

◆ ICudaEngine() [2/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( ICudaEngine const &  )
delete

◆ ICudaEngine() [3/3]

nvinfer1::safe::ICudaEngine::ICudaEngine ( ICudaEngine &&  )
delete

Member Function Documentation

◆ createExecutionContext()

virtual IExecutionContext * nvinfer1::safe::ICudaEngine::createExecutionContext ( )
pure virtualnoexcept

Create an execution context.

See also
safe::IExecutionContext.
Returns
An execution context object if it can be constructed, or nullptr if the construction fails.

Reasons for failure may include but not be limited to:

  • Heap memory exhaustion
  • Device memory exhaustion


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes; if createExecutionContext fails, users must treat this as a critical error and not perform any subsequent TensorRT operations apart from outputting the error logs.

◆ createExecutionContextWithoutDeviceMemory()

virtual IExecutionContext * nvinfer1::safe::ICudaEngine::createExecutionContextWithoutDeviceMemory ( )
pure virtualnoexcept

Create an execution context without any device memory allocated.

The memory for execution of this device context must be supplied by the application by calling safe::IExecutionContext::setDeviceMemory().

See also
getDeviceMemorySize() safe::IExecutionContext::setDeviceMemory()
Returns
An execution context object if it can be constructed, or nullptr if the construction fails.

Reasons for failure may include but not be limited to heap memory exhaustion.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes; if createExecutionContext fails, users must treat this as a critical error and not perform any subsequent TensorRT operations apart from outputting the error logs.

◆ getDeviceMemorySize()

virtual size_t nvinfer1::safe::ICudaEngine::getDeviceMemorySize ( ) const
pure virtualnoexcept

Return the amount of device memory required by an execution context.

See also
safe::IExecutionContext::setDeviceMemory()
Returns
Size of a contiguous memory buffer (in bytes) that users need to provide to safe::IExecutionContext::setDeviceMemory() if the execution context has been created by calling createExecutionContextWithoutDeviceMemory().


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getErrorRecorder()

virtual IErrorRecorder * nvinfer1::safe::ICudaEngine::getErrorRecorder ( ) const
pure virtualnoexcept

Get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error reporter has not been inherited from the IRuntime, and setErrorReporter() has not been called.

Returns
A pointer to the IErrorRecorder object that has been registered, or nullptr if none has been registered.
See also
setErrorRecorder()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getIOTensorName()

virtual AsciiChar const * nvinfer1::safe::ICudaEngine::getIOTensorName ( std::int32_t const  index) const
pure virtualnoexcept

Return the name of an IO tensor.

If the index does not fall between 0 and getNbIOTensors()-1, the function will fail with an error code of ErrorCode::kINVALID_ARGUMENT(3) that is emitted to the registered IErrorRecorder.

Parameters
indexThe IO tensor index.
Returns
The name of an IO tensor, which will be a NULL-terminated string of 1024 bytes or less (including the NULL terminator) if the index is in the range (between 0 and getNbIOTensors()-1). nullptr will be returned if the index is not in range.
See also
getNbIOTensors()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getName()

virtual AsciiChar const * nvinfer1::safe::ICudaEngine::getName ( ) const
pure virtualnoexcept

Returns the name of the network associated with the engine.

The name is set during network creation and is retrieved after building or deserialization.

See also
INetworkDefinition::setName(), INetworkDefinition::getName()
Returns
A NULL-terminated C-style string representing the name of the network, which will have a length of 1024 bytes or less including the NULL terminator.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getNbIOTensors()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getNbIOTensors ( ) const
pure virtualnoexcept

Return the number of input and output tensors for the network from which the engine was built.

Returns
The number of IO tensors.
See also
getIOTensorName()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorBytesPerComponent()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorBytesPerComponent ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Return the size of the tensor data type in bytes for a vectorized tensor.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
The size of the tensor data type in bytes if the tensor is vectorized (4 for float and int32, 2 for half, 1 for int8). 0 will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit, or
  • the tensor of the given name is not vectorized.
See also
safe::ICudaEngine::getTensorVectorizedDim()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorComponentsPerElement()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorComponentsPerElement ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Return the number of components included in one element for a vectorized tensor.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
The vector length (in scalars) for a vectorized tensor, or 1 for a scalar tensor. The invalid value -1 will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit.
See also
safe::ICudaEngine::getTensorVectorizedDim()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorDataType()

virtual DataType nvinfer1::safe::ICudaEngine::getTensorDataType ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Determine the required data type for a buffer from its tensor name.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
The type of the data in the buffer. The default value DataType::kFLOAT will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorFormat()

virtual TensorFormat nvinfer1::safe::ICudaEngine::getTensorFormat ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Return the tensor format.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
The tensor format. TensorFormat::kLINEAR will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorIOMode()

virtual TensorIOMode nvinfer1::safe::ICudaEngine::getTensorIOMode ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Determine whether a tensor is an input or output tensor.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
kINPUT if tensorName is the name of an input tensor, kOUTPUT if tensorName is the name of an output tensor. The invalid value kNONE is returned if
  • tensorName exceeds the string length limit, or
  • tensorName is nullptr, or
  • tensorName does not correspond to any input or output tensor.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorShape()

virtual Dims nvinfer1::safe::ICudaEngine::getTensorShape ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Get the extent of an input or output tensor.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
Extent of the tensor. The invalid value Dims{-1, {}} will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ getTensorVectorizedDim()

virtual std::int32_t nvinfer1::safe::ICudaEngine::getTensorVectorizedDim ( AsciiChar const *const  tensorName) const
pure virtualnoexcept

Return the dimension index along which the buffer is vectorized.

Specifically, -1 is returned if the tensor is scalar.

Parameters
tensorNameThe name of an input or output tensor.
Warning
The string tensorName must be NULL terminated and have a length of 1024 bytes or less including the NULL terminator.
Returns
The dimension index along which the buffer is vectorized. -1 will be returned if
  • name is not the name of an input or output tensor, or
  • name is nullptr, or
  • name exceeds the string length limit (1024 bytes or less including the NULL terminator), or
  • the tensor of given name is not vectorized.


Usage considerations

  • Allowed context for the API call
    • Thread-safe: Yes

◆ operator=() [1/2]

ICudaEngine & nvinfer1::safe::ICudaEngine::operator= ( ICudaEngine &&  ) &
delete

◆ operator=() [2/2]

ICudaEngine & nvinfer1::safe::ICudaEngine::operator= ( ICudaEngine const &  ) &
delete

◆ setErrorRecorder()

virtual void nvinfer1::safe::ICudaEngine::setErrorRecorder ( IErrorRecorder *const  recorder)
pure virtualnoexcept

Set the ErrorRecorder for this interface.

Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function will call incRefCount of the registered ErrorRecorder at least once. If the recorder is set to nullptr, the error code ErrorCode::kINVALID_ARGUMENT will be emitted if the recorder has been registered.

Parameters
recorderThe error recorder to register with this interface, or nullptr to deregister the current. error recorder.
See also
getErrorRecorder()


Usage considerations

  • Allowed context for the API call
    • Thread-safe: No

The documentation for this class was generated from the following file:

  Copyright © 2024 NVIDIA Corporation
  Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact