nvinfer2::safe Namespace Reference

Namespaces
namespace	consistency
namespace	impl

Classes
class	IPluginV3OneSafeBuild
class	IPluginV3OneSafeBuildMSS
	This class provides build capability and let output shapes be calculated by symbolic expression to support multiple static shape. More...
class	IPluginV3OneSafeCore
	Interface specifying the base plugin capability. More...
class	IPluginV3OneSafeRuntime
	Runtime component of the pluginV3 system, this component needs to be safety certified and is required at runtime. More...
class	ISafeMemAllocator
	Application-implemented class for controlling memory allocation on the GPU/CPU. More...
class	ISafePluginCreatorV3One
	The main interface to be implemented by all plugin creator classes. More...
class	ISafePluginRegistry
	PluginRegistry interface class that contains all the methods user can use to interface with the registry object. More...
class	ISafePluginResourceContext
	Interface for plugins to access per context resources provided by TensorRT. More...
class	ISafeRecorder
	Interface for extended recorder which allows error, warn, debug, or info messages to be recorded. More...
class	ITRTGraph
	Abstract Interface for a functionally safe graph for executing inference on a built network. More...
class	PhysicalDims
	Structure to define the physical dimensions of a tensor with support for up to 9 dimensions. More...
struct	RuntimeErrorInformation
	Holds information about runtime errors that occur during asynchronous kernel execution. More...
struct	TensorDescriptor
	A simple record summarizing various properties of a network IO Tensor. More...
class	TypedArray
	A standard_layout and trivially_copyable typed array that knows the data type and size it is holding. More...

Typedefs
using	Dims = nvinfer1::Dims
using	IVersionedInterface = nvinfer1::IVersionedInterface
using	AsciiChar = nvinfer1::AsciiChar
using	InterfaceInfo = nvinfer1::InterfaceInfo
using	DataType = nvinfer1::DataType
using	TensorIOMode = nvinfer1::TensorIOMode
using	ErrorCode = nvinfer1::ErrorCode
using	IErrorRecorder = nvinfer1::IErrorRecorder
using	IPluginResource = nvinfer1::IPluginResource
using	IPluginCreatorInterface = nvinfer1::IPluginCreatorInterface
using	IPluginV3 = nvinfer1::IPluginV3
using	IPluginCapability = nvinfer1::IPluginCapability
using	PluginField = nvinfer1::PluginField
using	PluginFieldCollection = nvinfer1::PluginFieldCollection
using	TensorRTPhase = nvinfer1::TensorRTPhase
using	ErrorDesc = char const *
	Type alias for error description strings. More...
using	RefCount = int32_t
	Type alias for reference count. More...
using	half_t = __half

Enumerations
enum class	Severity : uint32_t {   kINTERNAL_ERROR = 0 , kERROR = 1 , kWARNING = 2 , kINFO = 3 ,   kVERBOSE = 4 , kDEBUG = 5 }
	Enumerates severity levels for messages issued by the message recorder. More...
enum class	RuntimeErrorType : uint64_t {   kNAN_CONSUMED = 1ULL << 0 , kINF_CONSUMED = 1ULL << 1 , kGATHER_OOB = 1ULL << 2 , kSCATTER_OOB = 1ULL << 3 ,   kSCATTER_RACE = 1ULL << 4 , kDIV_ZERO = 1ULL << 5 }
	Enumerates types of runtime errors that can occur during kernel execution. More...
enum class	MemoryPlacement : uint32_t {   kNONE = 0x0 , kMANAGED = 0x1 , kGPU = 0x2 , kCPU = 0x4 ,   kCPU_PINNED = 0x80 }
	Enum to describe the placement of the memory region. More...
enum class	MemoryUsage : uint32_t { kGENERIC = 0x0 , kIMMUTABLE = 0x3 , kSCRATCH = 0x5 , kIOTENSOR = 0x6 }
	Enum to describe the usage of memory region. More...

Functions
template<typename T >
constexpr int32_t	EnumMax () noexcept
	Maximum number of elements in an enumeration type. More...
nvinfer2::safe::ISafePluginRegistry *	getSafePluginRegistry (nvinfer2::safe::ISafeRecorder &recorder) noexcept
	Get the SafePluginRegistry singleton instance and set the SafeRecorder of the safePluginRegistry. More...
bool	operator== (PhysicalDims const &d0, PhysicalDims const &d1) noexcept
bool	operator!= (PhysicalDims const &d0, PhysicalDims const &d1) noexcept
ErrorCode	createTRTGraph (ITRTGraph *&graph, void const *buffer, int64_t bufferSize, ISafeRecorder &recorder, bool trtManagedScratch=true, ISafeMemAllocator *allocator=nullptr) noexcept
	The C factory function which serves as an entry point to TRT that creates an instance of a ITRTGraph class. It will performs the following operations: More...
ErrorCode	destroyTRTGraph (ITRTGraph *&graph) noexcept
	Toplevel graph destructor. More...

Variables
constexpr uint64_t	kMAXIMUM_ALLOC_SIZE {17179869184U}
	Legacy constant (16 GiB) kept for TensorRT excessiveMemoryDetection test build compatibility. More...

Typedef Documentation

AsciiChar

using nvinfer2::safe::AsciiChar = typedef nvinfer1::AsciiChar

DataType

using nvinfer2::safe::DataType = typedef nvinfer1::DataType

Dims

using nvinfer2::safe::Dims = typedef nvinfer1::Dims

ErrorCode

using nvinfer2::safe::ErrorCode = typedef nvinfer1::ErrorCode

ErrorDesc

using nvinfer2::safe::ErrorDesc = typedef char const*

Type alias for error description strings.

half_t

using nvinfer2::safe::half_t = typedef __half

IErrorRecorder

using nvinfer2::safe::IErrorRecorder = typedef nvinfer1::IErrorRecorder

InterfaceInfo

using nvinfer2::safe::InterfaceInfo = typedef nvinfer1::InterfaceInfo

IPluginCapability

using nvinfer2::safe::IPluginCapability = typedef nvinfer1::IPluginCapability

IPluginCreatorInterface

using nvinfer2::safe::IPluginCreatorInterface = typedef nvinfer1::IPluginCreatorInterface

IPluginResource

using nvinfer2::safe::IPluginResource = typedef nvinfer1::IPluginResource

IPluginV3

using nvinfer2::safe::IPluginV3 = typedef nvinfer1::IPluginV3

IVersionedInterface

using nvinfer2::safe::IVersionedInterface = typedef nvinfer1::IVersionedInterface

PluginField

using nvinfer2::safe::PluginField = typedef nvinfer1::PluginField

PluginFieldCollection

using nvinfer2::safe::PluginFieldCollection = typedef nvinfer1::PluginFieldCollection

RefCount

using nvinfer2::safe::RefCount = typedef int32_t

Type alias for reference count.

TensorIOMode

using nvinfer2::safe::TensorIOMode = typedef nvinfer1::TensorIOMode

TensorRTPhase

using nvinfer2::safe::TensorRTPhase = typedef nvinfer1::TensorRTPhase

Enumeration Type Documentation

MemoryPlacement

enum class nvinfer2::safe::MemoryPlacement : uint32_t

strong

Enum to describe the placement of the memory region.

Enumerator
kNONE	Invalid or unspecified placement (used for error checking)
kMANAGED	CUDA managed memory (not used for safety)
kGPU	Device memory allocated via cudaMalloc.
kCPU	Regular (paged) host memory allocated via malloc, or aligned_alloc.
kCPU_PINNED	Page-locked host memory allocated via cudaHostAlloc, mappable to device (for zero-copy)

MemoryUsage

enum class nvinfer2::safe::MemoryUsage : uint32_t

strong

Enum to describe the usage of memory region.

Enumerator
kGENERIC	Uncommitted memory usage.
kIMMUTABLE	Memory (network weights) that is initialized once and can be shared across different graphs.
kSCRATCH	Scratchpad memory per context used for intermediate tensors.
kIOTENSOR	Memory used for IO Tensors.

RuntimeErrorType

enum class nvinfer2::safe::RuntimeErrorType : uint64_t

strong

Enumerates types of runtime errors that can occur during kernel execution.

• kNAN_CONSUMED error occurs when a NAN value is stored in an INT8 quantized datatype.
• kINF_CONSUMED error occurs when a +/-INF value is stored in an INT8 quantized datatype.
• kGATHER_OOB error occurs when a gather index tensor contains a value that is outside of the data tensor.
• kSCATTER_OOB error occurs when a scatter index tensor contains a value that is outside of the data tensor.
• kSCATTER_RACE error occurs when a scatter index tensor contains duplicate indices with reduction mode kNONE.
• kDIV_ZERO error occurs when a division-by-zero happens and its output is of an integer type.

Enumerator
kNAN_CONSUMED	NaN floating-point value was silently consumed.
kINF_CONSUMED	Inf floating-point value was silently consumed.
kGATHER_OOB	Out-of-bounds access in gather operation.
kSCATTER_OOB	Out-of-bounds access in scatter operation.
kSCATTER_RACE	Race condition in scatter operation.
kDIV_ZERO	Division-by-zero in int division.

Severity

enum class nvinfer2::safe::Severity : uint32_t

strong

Enumerates severity levels for messages issued by the message recorder.

Enumerator
kINTERNAL_ERROR	An internal error has occurred. Execution is unrecoverable.
kERROR	An application error has occurred.
kWARNING	An application error has been discovered, but TensorRT has recovered or fallen back to a default.
kINFO	Informational messages with instructional information.
kVERBOSE	Verbose messages with additional information.
kDEBUG	Trace messages with debugging information.

Function Documentation

createTRTGraph()

ErrorCode nvinfer2::safe::createTRTGraph ( ITRTGraph *&  graph, void const *  buffer, int64_t  bufferSize, ISafeRecorder &  recorder, bool  trtManagedScratch = true, ISafeMemAllocator *  allocator = nullptr  )

noexcept

The C factory function which serves as an entry point to TRT that creates an instance of a ITRTGraph class. It will performs the following operations:

• Deserialize the TRT engine (Serialized binary blob) from the buffer
• Allocate the persistent device memory (for const model weights) though the provided allocator (if not provided then using the default allocator), load the weights into device memory and store it as a shared_ptr member variable (for sharing with clone method)

Precondition

INIT API

Parameters

graph	A pointer to ITRTGraph, that will be initialized after this call.
buffer	A buffer that holds the serialized blob that describes the graph.
bufferSize	Size of the serialized blob
recorder	An ISafeRecorder that sets severity level and records the error messages during the inference. Note this does not include RuntimeError happening during kernel execution, see getErrorBuffer for RuntimeError.
trtManagedScratch	if this parameter is true (default), then TRT will use the memory allocator to allocate the scratch memory that holds the intermediate tensors. When cloning the graph, TRT will also allocate a new scratch memory automatically. If the parameter is set to false (for sharing scratch memory across multiple graphs), user needs to call setScratchMemory for the graph (and its clones) with a memory buffer size at least the size queried from getScratchMemorySize.
allocator	The memory allocator to be used by the runtime. All GPU memory acquired will use this allocator. If nullptr is passed, the default allocator will be used, which calls cudaMalloc and cudaFree.

Returns

ErrorCode

destroyTRTGraph()

ErrorCode nvinfer2::safe::destroyTRTGraph ( ITRTGraph *&  graph )

noexcept

Toplevel graph destructor.

• Frees the shared persistent memory only if this is the last graph instance that holds a reference to it.
• Frees the scratch memory if it is managed by TRT
• Deletes the graph object and all instruction instances underneath it for all IO profiles

Precondition

DEINIT API

Parameters

graph

A pointer reference to a graph object which will be destroyed and reference will be set to nullptr.

Returns

ErrorCode

EnumMax()

template<typename T >

constexpr int32_t nvinfer2::safe::EnumMax ( )

constexprnoexcept

Maximum number of elements in an enumeration type.

getSafePluginRegistry()

nvinfer2::safe::ISafePluginRegistry * nvinfer2::safe::getSafePluginRegistry ( nvinfer2::safe::ISafeRecorder &  recorder )

noexcept

Get the SafePluginRegistry singleton instance and set the SafeRecorder of the safePluginRegistry.

Precondition

INIT API

This function retrieves the singleton instance of the SafePluginRegistry and sets the provided SafeRecorder for error tracking. The SafePluginRegistry is used to manage plugin creators in a safe and thread-safe manner.

Parameters

recorder

The SafeRecorder to be set for the SafePluginRegistry instance.

Returns

A pointer to the ISafePluginRegistry interface of the SafePluginRegistry instance.

• Allowed context for the API call
  • Thread-safe: Yes

operator!=()

bool nvinfer2::safe::operator!= ( PhysicalDims const &  d0, PhysicalDims const &  d1  )

inlinenoexcept

operator==()

bool nvinfer2::safe::operator== ( PhysicalDims const &  d0, PhysicalDims const &  d1  )

inlinenoexcept

Variable Documentation

kMAXIMUM_ALLOC_SIZE

constexpr uint64_t nvinfer2::safe::kMAXIMUM_ALLOC_SIZE {17179869184U}

constexpr

Legacy constant (16 GiB) kept for TensorRT excessiveMemoryDetection test build compatibility.

Remarks

TRTS-11093 removed the max alloc limit from the allocator; this constant is deprecated and will be removed once TensorRT updates its lweSafety tests. Do not use for new code.