TensorRT 8.6.1
nvinfer1 Namespace Reference

The TensorRT API version 1 namespace. More...

Namespaces

namespace  consistency
 
namespace  impl
 
namespace  plugin
 
namespace  safe
 The safety subset of TensorRT's API version 1 namespace.
 
namespace  utils
 

Classes

class  Dims2
 Descriptor for two-dimensional data. More...
 
class  Dims3
 Descriptor for three-dimensional data. More...
 
class  Dims32
 
class  Dims4
 Descriptor for four-dimensional data. More...
 
class  DimsExprs
 
class  DimsHW
 Descriptor for two-dimensional spatial data. More...
 
class  DynamicPluginTensorDesc
 
class  IActivationLayer
 An Activation layer in a network definition. More...
 
class  IAlgorithm
 Describes a variation of execution of a layer. An algorithm is represented by IAlgorithmVariant and the IAlgorithmIOInfo for each of its inputs and outputs. An algorithm can be selected or reproduced using AlgorithmSelector::selectAlgorithms().". More...
 
class  IAlgorithmContext
 Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm. More...
 
class  IAlgorithmIOInfo
 Carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and output along with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce an algorithm using IAlgorithmSelector::selectAlgorithms(). More...
 
class  IAlgorithmSelector
 Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder. More...
 
class  IAlgorithmVariant
 provides a unique 128-bit identifier, which along with the input and output information denotes the variation of algorithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector::selectAlgorithms() More...
 
class  IAssertionLayer
 An assertion layer in a network. More...
 
class  IBuilder
 Builds an engine from a network definition. More...
 
class  IBuilderConfig
 Holds properties for configuring a builder to produce an engine. More...
 
class  ICastLayer
 A cast layer in a network. More...
 
class  IConcatenationLayer
 A concatenation layer in a network definition. More...
 
class  IConditionLayer
 
class  IConstantLayer
 Layer that represents a constant value. More...
 
class  IConvolutionLayer
 A convolution layer in a network definition. More...
 
class  ICudaEngine
 An engine for executing inference on a built network, with functionally unsafe features. More...
 
class  IDeconvolutionLayer
 A deconvolution layer in a network definition. More...
 
class  IDequantizeLayer
 A Dequantize layer in a network definition. More...
 
class  IDimensionExpr
 
class  IEinsumLayer
 An Einsum layer in a network. More...
 
class  IElementWiseLayer
 A elementwise layer in a network definition. More...
 
class  IEngineInspector
 An engine inspector which prints out the layer information of an engine or an execution context. More...
 
class  IErrorRecorder
 Reference counted application-implemented error reporting interface for TensorRT objects. More...
 
class  IExecutionContext
 Context for executing inference using an engine, with functionally unsafe features. More...
 
class  IExprBuilder
 
class  IFillLayer
 Generate an output tensor with specified mode. More...
 
class  IFullyConnectedLayer
 A fully connected layer in a network definition. This layer expects an input tensor of three or more non-batch dimensions. The input is automatically reshaped into an MxV tensor X, where V is a product of the last three dimensions and M is a product of the remaining dimensions (where the product over 0 dimensions is defined as 1). For example: More...
 
class  IGatherLayer
 A Gather layer in a network definition. Supports several kinds of gathering. More...
 
class  IGpuAllocator
 Application-implemented class for controlling allocation on the GPU. More...
 
class  IGridSampleLayer
 A GridSample layer in a network definition. More...
 
class  IHostMemory
 Class to handle library allocated memory that is accessible to the user. More...
 
class  IIdentityLayer
 A layer that represents the identity function. More...
 
class  IIfConditional
 
class  IIfConditionalBoundaryLayer
 
class  IIfConditionalInputLayer
 
class  IIfConditionalOutputLayer
 
class  IInt8Calibrator
 Application-implemented interface for calibration. More...
 
class  IInt8EntropyCalibrator
 
class  IInt8EntropyCalibrator2
 
class  IInt8LegacyCalibrator
 
class  IInt8MinMaxCalibrator
 
class  IIteratorLayer
 
class  ILayer
 Base class for all layer classes in a network definition. More...
 
class  ILogger
 Application-implemented logging interface for the builder, refitter and runtime. More...
 
class  ILoggerFinder
 A virtual base class to find a logger. Allows a plugin to find an instance of a logger if it needs to emit a log message. A pointer to an instance of this class is passed to a plugin shared library on initialization when that plugin is serialized as part of a version-compatible plan. See the plugin chapter in the developer guide for details. More...
 
class  ILoop
 
class  ILoopBoundaryLayer
 
class  ILoopOutputLayer
 
class  ILRNLayer
 A LRN layer in a network definition. More...
 
class  IMatrixMultiplyLayer
 Layer that represents a Matrix Multiplication. More...
 
class  INetworkDefinition
 A network definition for input to the builder. More...
 
class  INMSLayer
 A non-maximum suppression layer in a network definition. More...
 
class  INoCopy
 Forward declaration of IEngineInspector for use by other interfaces. More...
 
class  INonZeroLayer
 
class  INormalizationLayer
 A normalization layer in a network definition. More...
 
class  IOneHotLayer
 A OneHot layer in a network definition. More...
 
class  IOptimizationProfile
 Optimization profile for dynamic input dimensions and shape tensors. More...
 
class  IOutputAllocator
 Callback from ExecutionContext::enqueueV3() More...
 
class  IPaddingLayer
 Layer that represents a padding operation. More...
 
class  IParametricReLULayer
 Layer that represents a parametric ReLU operation. More...
 
class  IPluginCreator
 Plugin creator class for user implemented layers. More...
 
class  IPluginRegistry
 Single registration point for all plugins in an application. It is used to find plugin implementations during engine deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also have a corresponding IPluginCreator implementation. More...
 
class  IPluginV2
 Plugin class for user-implemented layers. More...
 
class  IPluginV2DynamicExt
 
class  IPluginV2Ext
 Plugin class for user-implemented layers. More...
 
class  IPluginV2IOExt
 Plugin class for user-implemented layers. More...
 
class  IPluginV2Layer
 Layer type for pluginV2. More...
 
class  IPoolingLayer
 A Pooling layer in a network definition. More...
 
class  IProfiler
 Application-implemented interface for profiling. More...
 
class  IQuantizeLayer
 A Quantize layer in a network definition. More...
 
class  IRaggedSoftMaxLayer
 A RaggedSoftmax layer in a network definition. More...
 
class  IRecurrenceLayer
 
class  IReduceLayer
 Layer that represents a reduction across a non-bool tensor. More...
 
class  IRefitter
 Updates weights in an engine. More...
 
class  IResizeLayer
 A resize layer in a network definition. More...
 
class  IReverseSequenceLayer
 A ReverseSequence layer in a network definition. More...
 
class  IRNNv2Layer
 An RNN layer in a network definition, version 2. More...
 
class  IRuntime
 Allows a serialized functionally unsafe engine to be deserialized. More...
 
class  IScaleLayer
 A Scale layer in a network definition. More...
 
class  IScatterLayer
 A scatter layer in a network definition. Supports several kinds of scattering. More...
 
class  ISelectLayer
 
class  IShapeLayer
 Layer type for getting shape of a tensor. More...
 
class  IShuffleLayer
 Layer type for shuffling data. More...
 
class  ISliceLayer
 Slices an input tensor into an output tensor based on the offset and strides. More...
 
class  ISoftMaxLayer
 A Softmax layer in a network definition. More...
 
class  ITensor
 A tensor in a network definition. More...
 
class  ITimingCache
 Class to handle tactic timing info collected from builder. More...
 
class  ITopKLayer
 Layer that represents a TopK reduction. More...
 
class  ITripLimitLayer
 
class  IUnaryLayer
 Layer that represents an unary operation. More...
 
struct  Permutation
 
class  PluginField
 Structure containing plugin attribute field names and associated data This information can be parsed to decode necessary plugin metadata. More...
 
struct  PluginFieldCollection
 Plugin field collection struct. More...
 
class  PluginRegistrar
 Register the plugin creator to the registry The static registry object will be instantiated when the plugin library is loaded. This static object will register all creators available in the library to the registry. More...
 
struct  PluginTensorDesc
 Fields that a plugin might see for an input or output. More...
 
class  Weights
 An array of weights used as a layer parameter. More...
 

Typedefs

using TensorFormats = uint32_t
 It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormat::kCHW4 | 1U << TensorFormat::kCHW32. More...
 
using SliceMode = SampleMode
 
using ResizeMode = InterpolationMode
 
using QuantizationFlags = uint32_t
 Represents one or more QuantizationFlag values using binary OR operations. More...
 
using BuilderFlags = uint32_t
 Represents one or more BuilderFlag values using binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U << BuilderFlag::kDEBUG. More...
 
using NetworkDefinitionCreationFlags = uint32_t
 Represents one or more NetworkDefinitionCreationFlag flags using binary OR operations. e.g., 1U << NetworkDefinitionCreationFlag::kEXPLICIT_BATCH. More...
 
using TempfileControlFlags = uint32_t
 Represents a collection of one or more TempfileControlFlag values combined using bitwise-OR operations. More...
 
using TacticSources = uint32_t
 Represents a collection of one or more TacticSource values combine using bitwise-OR operations. More...
 
using char_t = char
 char_t is the type used by TensorRT to represent all valid characters. More...
 
using AsciiChar = char_t
 
using Dims = Dims32
 
using AllocatorFlags = uint32_t
 
using PluginFormat = TensorFormat
 PluginFormat is reserved for backward compatibility. More...
 

Enumerations

enum class  LayerType : int32_t {
  kCONVOLUTION = 0 , kFULLY_CONNECTED = 1 , kACTIVATION = 2 , kPOOLING = 3 ,
  kLRN = 4 , kSCALE = 5 , kSOFTMAX = 6 , kDECONVOLUTION = 7 ,
  kCONCATENATION = 8 , kELEMENTWISE = 9 , kPLUGIN = 10 , kUNARY = 11 ,
  kPADDING = 12 , kSHUFFLE = 13 , kREDUCE = 14 , kTOPK = 15 ,
  kGATHER = 16 , kMATRIX_MULTIPLY = 17 , kRAGGED_SOFTMAX = 18 , kCONSTANT = 19 ,
  kRNN_V2 = 20 , kIDENTITY = 21 , kPLUGIN_V2 = 22 , kSLICE = 23 ,
  kSHAPE = 24 , kPARAMETRIC_RELU = 25 , kRESIZE = 26 , kTRIP_LIMIT = 27 ,
  kRECURRENCE = 28 , kITERATOR = 29 , kLOOP_OUTPUT = 30 , kSELECT = 31 ,
  kFILL = 32 , kQUANTIZE = 33 , kDEQUANTIZE = 34 , kCONDITION = 35 ,
  kCONDITIONAL_INPUT = 36 , kCONDITIONAL_OUTPUT = 37 , kSCATTER = 38 , kEINSUM = 39 ,
  kASSERTION = 40 , kONE_HOT = 41 , kNON_ZERO = 42 , kGRID_SAMPLE = 43 ,
  kNMS = 44 , kREVERSE_SEQUENCE = 45 , kNORMALIZATION = 46 , kCAST = 47
}
 The type values of layer classes. More...
 
enum class  ActivationType : int32_t {
  kRELU = 0 , kSIGMOID = 1 , kTANH = 2 , kLEAKY_RELU = 3 ,
  kELU = 4 , kSELU = 5 , kSOFTSIGN = 6 , kSOFTPLUS = 7 ,
  kCLIP = 8 , kHARD_SIGMOID = 9 , kSCALED_TANH = 10 , kTHRESHOLDED_RELU = 11
}
 Enumerates the types of activation to perform in an activation layer. More...
 
enum class  PaddingMode : int32_t {
  kEXPLICIT_ROUND_DOWN = 0 , kEXPLICIT_ROUND_UP = 1 , kSAME_UPPER = 2 , kSAME_LOWER = 3 ,
  kCAFFE_ROUND_DOWN = 4 , kCAFFE_ROUND_UP = 5
}
 Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, padding mode takes precedence if setPaddingMode() and setPrePadding() are also used. More...
 
enum class  PoolingType : int32_t { kMAX = 0 , kAVERAGE = 1 , kMAX_AVERAGE_BLEND = 2 }
 The type of pooling to perform in a pooling layer. More...
 
enum class  ScaleMode : int32_t { kUNIFORM = 0 , kCHANNEL = 1 , kELEMENTWISE = 2 }
 Controls how shift, scale and power are applied in a Scale layer. More...
 
enum class  ElementWiseOperation : int32_t {
  kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,
  kSUB = 4 , kDIV = 5 , kPOW = 6 , kFLOOR_DIV = 7 ,
  kAND = 8 , kOR = 9 , kXOR = 10 , kEQUAL = 11 ,
  kGREATER = 12 , kLESS = 13
}
 Enumerates the binary operations that may be performed by an ElementWise layer. More...
 
enum class  GatherMode : int32_t { kDEFAULT = 0 , kELEMENT = 1 , kND = 2 }
 Control form of IGatherLayer. More...
 
enum class  RNNOperation : int32_t { kRELU = 0 , kTANH = 1 , kLSTM = 2 , kGRU = 3 }
 Enumerates the RNN operations that may be performed by an RNN layer. More...
 
enum class  RNNDirection : int32_t { kUNIDIRECTION = 0 , kBIDIRECTION = 1 }
 Enumerates the RNN direction that may be performed by an RNN layer. More...
 
enum class  RNNInputMode : int32_t { kLINEAR = 0 , kSKIP = 1 }
 Enumerates the RNN input modes that may occur with an RNN layer. More...
 
enum class  RNNGateType : int32_t {
  kINPUT = 0 , kOUTPUT = 1 , kFORGET = 2 , kUPDATE = 3 ,
  kRESET = 4 , kCELL = 5 , kHIDDEN = 6
}
 Identifies an individual gate within an RNN cell. More...
 
enum class  UnaryOperation : int32_t {
  kEXP = 0 , kLOG = 1 , kSQRT = 2 , kRECIP = 3 ,
  kABS = 4 , kNEG = 5 , kSIN = 6 , kCOS = 7 ,
  kTAN = 8 , kSINH = 9 , kCOSH = 10 , kASIN = 11 ,
  kACOS = 12 , kATAN = 13 , kASINH = 14 , kACOSH = 15 ,
  kATANH = 16 , kCEIL = 17 , kFLOOR = 18 , kERF = 19 ,
  kNOT = 20 , kSIGN = 21 , kROUND = 22 , kISINF = 23
}
 Enumerates the unary operations that may be performed by a Unary layer. More...
 
enum class  ReduceOperation : int32_t {
  kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,
  kAVG = 4
}
 Enumerates the reduce operations that may be performed by a Reduce layer. More...
 
enum class  SampleMode : int32_t {
  kSTRICT_BOUNDS = 0 , kDEFAULT = kSTRICT_BOUNDS , kWRAP = 1 , kCLAMP = 2 ,
  kFILL = 3 , kREFLECT = 4
}
 Controls how ISliceLayer and IGridSample handle out-of-bounds coordinates. More...
 
enum class  TopKOperation : int32_t { kMAX = 0 , kMIN = 1 }
 Enumerates the operations that may be performed by a TopK layer. More...
 
enum class  MatrixOperation : int32_t { kNONE , kTRANSPOSE , kVECTOR }
 Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplication. More...
 
enum class  InterpolationMode : int32_t { kNEAREST = 0 , kLINEAR = 1 , kCUBIC = 2 }
 Enumerates various modes of interpolation. More...
 
enum class  ResizeCoordinateTransformation : int32_t { kALIGN_CORNERS = 0 , kASYMMETRIC = 1 , kHALF_PIXEL = 2 }
 The resize coordinate transformation function. More...
 
enum class  ResizeSelector : int32_t { kFORMULA = 0 , kUPPER = 1 }
 The coordinate selector when resize to single pixel output. More...
 
enum class  ResizeRoundMode : int32_t { kHALF_UP = 0 , kHALF_DOWN = 1 , kFLOOR = 2 , kCEIL = 3 }
 The rounding mode for nearest neighbor resize. More...
 
enum class  LoopOutput : int32_t { kLAST_VALUE = 0 , kCONCATENATE = 1 , kREVERSE = 2 }
 Enum that describes kinds of loop outputs. More...
 
enum class  TripLimit : int32_t { kCOUNT = 0 , kWHILE = 1 }
 Enum that describes kinds of trip limits. More...
 
enum class  FillOperation : int32_t { kLINSPACE = 0 , kRANDOM_UNIFORM = 1 , kRANDOM_NORMAL = 2 }
 Enumerates the tensor fill operations that may performed by a fill layer. More...
 
enum class  ScatterMode : int32_t { kELEMENT = 0 , kND = 1 }
 Control form of IScatterLayer. More...
 
enum class  BoundingBoxFormat : int32_t { kCORNER_PAIRS = 0 , kCENTER_SIZES = 1 }
 Representation of bounding box data used for the Boxes input tensor in INMSLayer. More...
 
enum class  CalibrationAlgoType : int32_t { kLEGACY_CALIBRATION = 0 , kENTROPY_CALIBRATION = 1 , kENTROPY_CALIBRATION_2 = 2 , kMINMAX_CALIBRATION = 3 }
 Version of calibration algorithm to use. More...
 
enum class  QuantizationFlag : int32_t { kCALIBRATE_BEFORE_FUSION = 0 }
 List of valid flags for quantizing the network to int8. More...
 
enum class  BuilderFlag : int32_t {
  kFP16 = 0 , kINT8 = 1 , kDEBUG = 2 , kGPU_FALLBACK = 3 ,
  kSTRICT_TYPES = 4 , kREFIT = 5 , kDISABLE_TIMING_CACHE = 6 , kTF32 = 7 ,
  kSPARSE_WEIGHTS = 8 , kSAFETY_SCOPE = 9 , kOBEY_PRECISION_CONSTRAINTS = 10 , kPREFER_PRECISION_CONSTRAINTS = 11 ,
  kDIRECT_IO = 12 , kREJECT_EMPTY_ALGORITHMS = 13 , kENABLE_TACTIC_HEURISTIC = 14 , kVERSION_COMPATIBLE = 15 ,
  kEXCLUDE_LEAN_RUNTIME = 16 , kFP8 = 17
}
 List of valid modes that the builder can enable when creating an engine from a network definition. More...
 
enum class  MemoryPoolType : int32_t {
  kWORKSPACE = 0 , kDLA_MANAGED_SRAM = 1 , kDLA_LOCAL_DRAM = 2 , kDLA_GLOBAL_DRAM = 3 ,
  kTACTIC_DRAM = 4
}
 The type for memory pools used by TensorRT. More...
 
enum class  PreviewFeature : int32_t { kFASTER_DYNAMIC_SHAPES_0805 = 0 , kDISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805 = 1 , kPROFILE_SHARING_0806 = 2 }
 Define preview features. More...
 
enum class  HardwareCompatibilityLevel : int32_t { kNONE = 0 , kAMPERE_PLUS = 1 }
 
enum class  NetworkDefinitionCreationFlag : int32_t { kEXPLICIT_BATCH = 0 , kEXPLICIT_PRECISION = 1 }
 List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFlag is used with createNetworkV2() to specify immutable properties of the network. Creating a network without NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag has been deprecated. More...
 
enum class  EngineCapability : int32_t {
  kSTANDARD = 0 , kDEFAULT = kSTANDARD , kSAFETY = 1 , kSAFE_GPU = kSAFETY ,
  kDLA_STANDALONE = 2 , kSAFE_DLA = kDLA_STANDALONE
}
 List of supported engine capability flows. More...
 
enum class  DimensionOperation : int32_t {
  kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,
  kSUB = 4 , kEQUAL = 5 , kLESS = 6 , kFLOOR_DIV = 7 ,
  kCEIL_DIV = 8
}
 An operation on two IDimensionExpr, which represent integer expressions used in dimension computations. More...
 
enum class  TensorLocation : int32_t { kDEVICE = 0 , kHOST = 1 }
 The location for tensor data storage, device or host. More...
 
enum class  WeightsRole : int32_t {
  kKERNEL = 0 , kBIAS = 1 , kSHIFT = 2 , kSCALE = 3 ,
  kCONSTANT = 4 , kANY = 5
}
 How a layer uses particular Weights. More...
 
enum class  DeviceType : int32_t { kGPU , kDLA }
 The device that this layer/network will execute on. More...
 
enum class  TempfileControlFlag : int32_t { kALLOW_IN_MEMORY_FILES = 0 , kALLOW_TEMPORARY_FILES = 1 }
 Flags used to control TensorRT's behavior when creating executable temporary files. More...
 
enum class  OptProfileSelector : int32_t { kMIN = 0 , kOPT = 1 , kMAX = 2 }
 When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), select whether we are interested in the minimum, optimum, or maximum values for these parameters. The minimum and maximum specify the permitted range that is supported at runtime, while the optimum value is used for the kernel selection. This should be the "typical" value that is expected to occur at runtime. More...
 
enum class  TacticSource : int32_t {
  kCUBLAS = 0 , kCUBLAS_LT = 1 , kCUDNN = 2 , kEDGE_MASK_CONVOLUTIONS = 3 ,
  kJIT_CONVOLUTIONS = 4
}
 List of tactic sources for TensorRT. More...
 
enum class  ProfilingVerbosity : int32_t {
  kLAYER_NAMES_ONLY = 0 , kNONE = 1 , kDETAILED = 2 , kDEFAULT = kLAYER_NAMES_ONLY ,
  kVERBOSE = kDETAILED
}
 List of verbosity levels of layer information exposed in NVTX annotations and in IEngineInspector. More...
 
enum class  LayerInformationFormat : int32_t { kONELINE = 0 , kJSON = 1 }
 The format in which the IEngineInspector prints the layer information. More...
 
enum class  DataType : int32_t {
  kFLOAT = 0 , kHALF = 1 , kINT8 = 2 , kINT32 = 3 ,
  kBOOL = 4 , kUINT8 = 5 , kFP8 = 6
}
 The type of weights and tensors. More...
 
enum class  TensorFormat : int32_t {
  kLINEAR = 0 , kCHW2 = 1 , kHWC8 = 2 , kCHW4 = 3 ,
  kCHW16 = 4 , kCHW32 = 5 , kDHWC8 = 6 , kCDHW32 = 7 ,
  kHWC = 8 , kDLA_LINEAR = 9 , kDLA_HWC4 = 10 , kHWC16 = 11 ,
  kDHWC = 12
}
 Format of the input/output tensors. More...
 
enum class  AllocatorFlag : int32_t { kRESIZABLE = 0 }
 
enum class  ErrorCode : int32_t {
  kSUCCESS = 0 , kUNSPECIFIED_ERROR = 1 , kINTERNAL_ERROR = 2 , kINVALID_ARGUMENT = 3 ,
  kINVALID_CONFIG = 4 , kFAILED_ALLOCATION = 5 , kFAILED_INITIALIZATION = 6 , kFAILED_EXECUTION = 7 ,
  kFAILED_COMPUTATION = 8 , kINVALID_STATE = 9 , kUNSUPPORTED_STATE = 10
}
 Error codes that can be returned by TensorRT during execution. More...
 
enum class  TensorIOMode : int32_t { kNONE = 0 , kINPUT = 1 , kOUTPUT = 2 }
 Definition of tensor IO Mode. More...
 
enum class  PluginVersion : uint8_t { kV2 = 0 , kV2_EXT = 1 , kV2_IOEXT = 2 , kV2_DYNAMICEXT = 3 }
 
enum class  PluginFieldType : int32_t {
  kFLOAT16 = 0 , kFLOAT32 = 1 , kFLOAT64 = 2 , kINT8 = 3 ,
  kINT16 = 4 , kINT32 = 5 , kCHAR = 6 , kDIMS = 7 ,
  kUNKNOWN = 8
}
 The possible field types for custom layer. More...
 

Functions

template<>
constexpr int32_t EnumMax< LayerType > () noexcept
 
template<>
constexpr int32_t EnumMax< ScaleMode > () noexcept
 
template<>
constexpr int32_t EnumMax< GatherMode > () noexcept
 
template<>
constexpr int32_t EnumMax< RNNOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< RNNDirection > () noexcept
 
template<>
constexpr int32_t EnumMax< RNNInputMode > () noexcept
 
template<>
constexpr int32_t EnumMax< RNNGateType > () noexcept
 
template<>
constexpr int32_t EnumMax< UnaryOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< ReduceOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< SampleMode > () noexcept
 
template<>
constexpr int32_t EnumMax< TopKOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< MatrixOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< LoopOutput > () noexcept
 
template<>
constexpr int32_t EnumMax< TripLimit > () noexcept
 
template<>
constexpr int32_t EnumMax< FillOperation > () noexcept
 
template<>
constexpr int32_t EnumMax< ScatterMode > () noexcept
 
template<>
constexpr int32_t EnumMax< BoundingBoxFormat > () noexcept
 
template<>
constexpr int32_t EnumMax< CalibrationAlgoType > () noexcept
 
template<>
constexpr int32_t EnumMax< QuantizationFlag > () noexcept
 
template<>
constexpr int32_t EnumMax< BuilderFlag > () noexcept
 
template<>
constexpr int32_t EnumMax< MemoryPoolType > () noexcept
 
template<>
constexpr int32_t EnumMax< NetworkDefinitionCreationFlag > () noexcept
 
nvinfer1::IPluginRegistrygetBuilderPluginRegistry (nvinfer1::EngineCapability capability) noexcept
 Return the plugin registry for building a Standard engine, or nullptr if no registry exists. More...
 
nvinfer1::safe::IPluginRegistrygetBuilderSafePluginRegistry (nvinfer1::EngineCapability capability) noexcept
 Return the plugin registry for building a Safety engine, or nullptr if no registry exists. More...
 
template<>
constexpr int32_t EnumMax< DimensionOperation > () noexcept
 Maximum number of elements in DimensionOperation enum. More...
 
template<>
constexpr int32_t EnumMax< WeightsRole > () noexcept
 Maximum number of elements in WeightsRole enum. More...
 
template<>
constexpr int32_t EnumMax< DeviceType > () noexcept
 Maximum number of elements in DeviceType enum. More...
 
template<>
constexpr int32_t EnumMax< TempfileControlFlag > () noexcept
 Maximum number of elements in TempfileControlFlag enum. More...
 
template<>
constexpr int32_t EnumMax< OptProfileSelector > () noexcept
 Number of different values of OptProfileSelector enum. More...
 
template<>
constexpr int32_t EnumMax< TacticSource > () noexcept
 Maximum number of tactic sources in TacticSource enum. More...
 
template<>
constexpr int32_t EnumMax< ProfilingVerbosity > () noexcept
 Maximum number of profile verbosity levels in ProfilingVerbosity enum. More...
 
template<>
constexpr int32_t EnumMax< LayerInformationFormat > () noexcept
 
template<typename T >
constexpr int32_t EnumMax () noexcept
 Maximum number of elements in an enumeration type. More...
 

Detailed Description

The TensorRT API version 1 namespace.

Typedef Documentation

◆ AllocatorFlags

using nvinfer1::AllocatorFlags = typedef uint32_t

◆ AsciiChar

using nvinfer1::AsciiChar = typedef char_t

AsciiChar is the type used by TensorRT to represent valid ASCII characters. This type is used by IPluginV2, PluginField, IPluginCreator, IPluginRegistry, and ILogger due to their use in automotive safety context.

◆ BuilderFlags

using nvinfer1::BuilderFlags = typedef uint32_t

Represents one or more BuilderFlag values using binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U << BuilderFlag::kDEBUG.

See also
IBuilderConfig::setFlags(), IBuilderConfig::getFlags()

◆ char_t

using nvinfer1::char_t = typedef char

char_t is the type used by TensorRT to represent all valid characters.

◆ Dims

using nvinfer1::Dims = typedef Dims32

Alias for Dims32.

Warning
: This alias might change in the future.

◆ NetworkDefinitionCreationFlags

using nvinfer1::NetworkDefinitionCreationFlags = typedef uint32_t

Represents one or more NetworkDefinitionCreationFlag flags using binary OR operations. e.g., 1U << NetworkDefinitionCreationFlag::kEXPLICIT_BATCH.

See also
IBuilder::createNetworkV2

◆ PluginFormat

PluginFormat is reserved for backward compatibility.

See also
IPluginV2::supportsFormat()

◆ QuantizationFlags

using nvinfer1::QuantizationFlags = typedef uint32_t

Represents one or more QuantizationFlag values using binary OR operations.

See also
IBuilderConfig::getQuantizationFlags(), IBuilderConfig::setQuantizationFlags()

◆ ResizeMode

Deprecated:
Deprecated in TensorRT 8.5. Superseded by InterpolationMode.

◆ SliceMode

Deprecated:
Deprecated in TensorRT 8.5. Superseded by SampleMode.

◆ TacticSources

using nvinfer1::TacticSources = typedef uint32_t

Represents a collection of one or more TacticSource values combine using bitwise-OR operations.

See also
IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()

◆ TempfileControlFlags

using nvinfer1::TempfileControlFlags = typedef uint32_t

Represents a collection of one or more TempfileControlFlag values combined using bitwise-OR operations.

See also
TempfileControlFlag, IRuntime::setTempfileControlFlags(), IRuntime::getTempfileControlFlags()

◆ TensorFormats

using nvinfer1::TensorFormats = typedef uint32_t

It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormat::kCHW4 | 1U << TensorFormat::kCHW32.

See also
ITensor::getAllowedFormats(), ITensor::setAllowedFormats(),

Enumeration Type Documentation

◆ ActivationType

enum class nvinfer1::ActivationType : int32_t
strong

Enumerates the types of activation to perform in an activation layer.

Enumerator
kRELU 

Rectified linear activation.

kSIGMOID 

Sigmoid activation.

kTANH 

TanH activation.

kLEAKY_RELU 

LeakyRelu activation: x>=0 ? x : alpha * x.

kELU 

Elu activation: x>=0 ? x : alpha * (exp(x) - 1).

kSELU 

Selu activation: x>0 ? beta * x : beta * (alpha*exp(x) - alpha)

kSOFTSIGN 

Softsign activation: x / (1+|x|)

kSOFTPLUS 

Parametric softplus activation: alpha*log(exp(beta*x)+1)

kCLIP 

Clip activation: max(alpha, min(beta, x))

kHARD_SIGMOID 

Hard sigmoid activation: max(0, min(1, alpha*x+beta))

kSCALED_TANH 

Scaled tanh activation: alpha*tanh(beta*x)

kTHRESHOLDED_RELU 

Thresholded ReLU activation: x>alpha ? x : 0.

◆ AllocatorFlag

enum class nvinfer1::AllocatorFlag : int32_t
strong
Enumerator
kRESIZABLE 

TensorRT may call realloc() on this allocation.

◆ BoundingBoxFormat

enum class nvinfer1::BoundingBoxFormat : int32_t
strong

Representation of bounding box data used for the Boxes input tensor in INMSLayer.

See also
INMSLayer
Enumerator
kCORNER_PAIRS 

(x1, y1, x2, y2) where (x1, y1) and (x2, y2) are any pair of diagonal corners

kCENTER_SIZES 

(x_center, y_center, width, height) where (x_center, y_center) is the center point of the box

◆ BuilderFlag

enum class nvinfer1::BuilderFlag : int32_t
strong

List of valid modes that the builder can enable when creating an engine from a network definition.

See also
IBuilderConfig::setFlags(), IBuilderConfig::getFlags()
Enumerator
kFP16 

Enable FP16 layer selection, with FP32 fallback.

kINT8 

Enable Int8 layer selection, with FP32 fallback with FP16 fallback if kFP16 also specified.

kDEBUG 

Enable debugging of layers via synchronizing after every layer.

kGPU_FALLBACK 

Enable layers marked to execute on GPU if layer cannot execute on DLA.

kSTRICT_TYPES 

Legacy flag with effect similar to setting all of these three flags:

* kPREFER_PRECISION_CONSTRAINTS
* kDIRECT_IO
* kREJECT_EMPTY_ALGORITHMS

except that if the direct I/O requirement cannot be met and kDIRECT_IO was not explicitly set,
instead of the build failing, the build falls back as if kDIRECT_IO was not set.

\deprecated Deprecated in TensorRT 8.2.
kREFIT 

Enable building a refittable engine.

kDISABLE_TIMING_CACHE 

Disable reuse of timing information across identical layers.

kTF32 

Allow (but not require) computations on tensors of type DataType::kFLOAT to use TF32. TF32 computes inner products by rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas. Enabled by default.

kSPARSE_WEIGHTS 

Allow the builder to examine weights and use optimized functions when weights have suitable sparsity.

kSAFETY_SCOPE 

Change the allowed parameters in the EngineCapability::kSTANDARD flow to match the restrictions that EngineCapability::kSAFETY check against for DeviceType::kGPU and EngineCapability::kDLA_STANDALONE check against the DeviceType::kDLA case. This flag is forced to true if EngineCapability::kSAFETY at build time if it is unset.

This flag is only supported in NVIDIA Drive(R) products.

kOBEY_PRECISION_CONSTRAINTS 

Require that layers execute in specified precisions. Build fails otherwise.

kPREFER_PRECISION_CONSTRAINTS 

Prefer that layers execute in specified precisions. Fall back (with warning) to another precision if build would otherwise fail.

kDIRECT_IO 

Require that no reformats be inserted between a layer and a network I/O tensor for which ITensor::setAllowedFormats was called. Build fails if a reformat is required for functional correctness.

kREJECT_EMPTY_ALGORITHMS 

Fail if IAlgorithmSelector::selectAlgorithms returns an empty set of algorithms.

kENABLE_TACTIC_HEURISTIC 

Enable heuristic-based tactic selection for shorter engine generation time. The engine may not be as performant as when built with a profiling-based builder.

This flag is only supported by NVIDIA Ampere and later GPUs.

Deprecated:
Superseded by builder optimization level 2. Deprecated in TensorRT 8.6
kVERSION_COMPATIBLE 

Restrict to lean runtime operators to provide version forward compatibility for the plan.

Using this flag with ICudaEngine::serialize() and BuilderFlag::kREFIT would result in error. This flag is only supported by NVIDIA Volta and later GPUs. This flag is not supported in NVIDIA Drive(R) products. This flag is not supported with implicit batch mode. Network must be created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH.

kEXCLUDE_LEAN_RUNTIME 

Exclude lean runtime from the plan when version forward compatability is enabled. By default, this flag is unset, so the lean runtime will be included in the plan.

If BuilderFlag::kVERSION_COMPATIBLE is not set then the value of this flag will be ignored.

This flag is not supported with implicit batch mode. Network must be created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH.

kFP8 

Enable FP8 layer selection, with FP32 fallback.

Warning
kFP8 is not supported yet and will result in an error or undefined behavior.

◆ CalibrationAlgoType

enum class nvinfer1::CalibrationAlgoType : int32_t
strong

Version of calibration algorithm to use.

enum CalibrationAlgoType

Enumerator
kLEGACY_CALIBRATION 
kENTROPY_CALIBRATION 
kENTROPY_CALIBRATION_2 
kMINMAX_CALIBRATION 

◆ DataType

enum class nvinfer1::DataType : int32_t
strong

The type of weights and tensors.

Enumerator
kFLOAT 

32-bit floating point format.

kHALF 

IEEE 16-bit floating-point format.

kINT8 

Signed 8-bit integer representing a quantized floating-point value.

kINT32 

Signed 32-bit integer format.

kBOOL 

8-bit boolean. 0 = false, 1 = true, other values undefined.

kUINT8 

Unsigned 8-bit integer format. Cannot be used to represent quantized floating-point values. Use the IdentityLayer to convert kUINT8 network-level inputs to {kFLOAT, kHALF} prior to use with other TensorRT layers, or to convert intermediate output before kUINT8 network-level outputs from {kFLOAT, kHALF} to kUINT8. kUINT8 conversions are only supported for {kFLOAT, kHALF}. kUINT8 to {kFLOAT, kHALF} conversion will convert the integer values to equivalent floating point values. {kFLOAT, kHALF} to kUINT8 conversion will convert the floating point values to integer values by truncating towards zero. This conversion has undefined behavior for floating point values outside the range [0.0f, 256.0f) after truncation. kUINT8 conversions are not supported for {kINT8, kINT32, kBOOL}.

kFP8 

Signed 8-bit floating point with 1 sign bit, 4 exponent bits, 3 mantissa bits, and exponent-bias 7.

Warning
kFP8 is not supported yet and will result in an error or undefined behavior.

◆ DeviceType

enum class nvinfer1::DeviceType : int32_t
strong

The device that this layer/network will execute on.

Enumerator
kGPU 

GPU Device.

kDLA 

DLA Core.

◆ DimensionOperation

enum class nvinfer1::DimensionOperation : int32_t
strong

An operation on two IDimensionExpr, which represent integer expressions used in dimension computations.

For example, given two IDimensionExpr x and y and an IExprBuilder& eb, eb.operation(DimensionOperation::kSUM, x, y) creates a representation of x+y.

See also
IDimensionExpr, IExprBuilder
Enumerator
kSUM 

Sum of the two operands.

kPROD 

Product of the two operands.

kMAX 

Maximum of the two operands.

kMIN 

Minimum of the two operands.

kSUB 

Substract the second element from the first.

kEQUAL 

1 if operands are equal, 0 otherwise.

kLESS 

1 if first operand is less than second operand, 0 otherwise.

kFLOOR_DIV 

Floor division of the first element by the second.

kCEIL_DIV 

Division rounding up.

◆ ElementWiseOperation

enum class nvinfer1::ElementWiseOperation : int32_t
strong

Enumerates the binary operations that may be performed by an ElementWise layer.

Operations kAND, kOR, and kXOR must have inputs of DataType::kBOOL.

Operation kPOW must have inputs of DataType::kFLOAT, DataType::kHALF, or DataType::kINT8.

All other operations must have inputs of DataType::kFLOAT, DataType::kHALF, DataType::kINT8, or DataType::kINT32.

See also
IElementWiseLayer
Enumerator
kSUM 

Sum of the two elements.

kPROD 

Product of the two elements.

kMAX 

Maximum of the two elements.

kMIN 

Minimum of the two elements.

kSUB 

Subtract the second element from the first.

kDIV 

Divide the first element by the second.

kPOW 

The first element to the power of the second element.

kFLOOR_DIV 

Floor division of the first element by the second.

kAND 

Logical AND of two elements.

kOR 

Logical OR of two elements.

kXOR 

Logical XOR of two elements.

kEQUAL 

Check if two elements are equal.

kGREATER 

Check if element in first tensor is greater than corresponding element in second tensor.

kLESS 

Check if element in first tensor is less than corresponding element in second tensor.

◆ EngineCapability

enum class nvinfer1::EngineCapability : int32_t
strong

List of supported engine capability flows.

The EngineCapability determines the restrictions of a network during build time and what runtime it targets. When BuilderFlag::kSAFETY_SCOPE is not set (by default), EngineCapability::kSTANDARD does not provide any restrictions on functionality and the resulting serialized engine can be executed with TensorRT's standard runtime APIs in the nvinfer1 namespace. EngineCapability::kSAFETY provides a restricted subset of network operations that are safety certified and the resulting serialized engine can be executed with TensorRT's safe runtime APIs in the nvinfer1::safe namespace. EngineCapability::kDLA_STANDALONE provides a restricted subset of network operations that are DLA compatible and the resulting serialized engine can be executed using standalone DLA runtime APIs. See sampleCudla for an example of integrating cuDLA APIs with TensorRT APIs.

Enumerator
kSTANDARD 

Standard: TensorRT flow without targeting the safety runtime. This flow supports both DeviceType::kGPU and DeviceType::kDLA.

kDEFAULT 
Deprecated:
Deprecated in TensorRT 8.0. Superseded by kSTANDARD.
kSAFETY 

Safety: TensorRT flow with restrictions targeting the safety runtime. See safety documentation for list of supported layers and formats. This flow supports only DeviceType::kGPU.

This flag is only supported in NVIDIA Drive(R) products.

kSAFE_GPU 
Deprecated:
Deprecated in TensorRT 8.0. Superseded by kSAFETY.
kDLA_STANDALONE 

DLA Standalone: TensorRT flow with restrictions targeting external, to TensorRT, DLA runtimes. See DLA documentation for list of supported layers and formats. This flow supports only DeviceType::kDLA.

kSAFE_DLA 
Deprecated:
Deprecated in TensorRT 8.0. Superseded by kDLA_STANDALONE.

◆ ErrorCode

enum class nvinfer1::ErrorCode : int32_t
strong

Error codes that can be returned by TensorRT during execution.

Enumerator
kSUCCESS 

Execution completed successfully.

kUNSPECIFIED_ERROR 

An error that does not fall into any other category. This error is included for forward compatibility.

kINTERNAL_ERROR 

A non-recoverable TensorRT error occurred. TensorRT is in an invalid internal state when this error is emitted and any further calls to TensorRT will result in undefined behavior.

kINVALID_ARGUMENT 

An argument passed to the function is invalid in isolation. This is a violation of the API contract.

kINVALID_CONFIG 

An error occurred when comparing the state of an argument relative to other arguments. For example, the dimensions for concat differ between two tensors outside of the channel dimension. This error is triggered when an argument is correct in isolation, but not relative to other arguments. This is to help to distinguish from the simple errors from the more complex errors. This is a violation of the API contract.

kFAILED_ALLOCATION 

An error occurred when performing an allocation of memory on the host or the device. A memory allocation error is normally fatal, but in the case where the application provided its own memory allocation routine, it is possible to increase the pool of available memory and resume execution.

kFAILED_INITIALIZATION 

One, or more, of the components that TensorRT relies on did not initialize correctly. This is a system setup issue.

kFAILED_EXECUTION 

An error occurred during execution that caused TensorRT to end prematurely, either an asynchronous error or other execution errors reported by CUDA/DLA. In a dynamic system, the data can be thrown away and the next frame can be processed or execution can be retried. This is either an execution error or a memory error.

kFAILED_COMPUTATION 

An error occurred during execution that caused the data to become corrupted, but execution finished. Examples of this error are NaN squashing or integer overflow. In a dynamic system, the data can be thrown away and the next frame can be processed or execution can be retried. This is either a data corruption error, an input error, or a range error. This is not used in safety but may be used in standard.

kINVALID_STATE 

TensorRT was put into a bad state by incorrect sequence of function calls. An example of an invalid state is specifying a layer to be DLA only without GPU fallback, and that layer is not supported by DLA. This can occur in situations where a service is optimistically executing networks for multiple different configurations without checking proper error configurations, and instead throwing away bad configurations caught by TensorRT. This is a violation of the API contract, but can be recoverable.

Example of a recovery: GPU fallback is disabled and conv layer with large filter(63x63) is specified to run on DLA. This will fail due to DLA not supporting the large kernel size. This can be recovered by either turning on GPU fallback or setting the layer to run on the GPU.

kUNSUPPORTED_STATE 

An error occurred due to the network not being supported on the device due to constraints of the hardware or system. An example is running a unsafe layer in a safety certified context, or a resource requirement for the current network is greater than the capabilities of the target device. The network is otherwise correct, but the network and hardware combination is problematic. This can be recoverable. Examples:

  • Scratch space requests larger than available device memory and can be recovered by increasing allowed workspace size.
  • Tensor size exceeds the maximum element count and can be recovered by reducing the maximum batch size.

◆ FillOperation

enum class nvinfer1::FillOperation : int32_t
strong

Enumerates the tensor fill operations that may performed by a fill layer.

See also
IFillLayer
Enumerator
kLINSPACE 

Generate evenly spaced numbers over a specified interval.

kRANDOM_UNIFORM 

Generate a tensor with random values drawn from a uniform distribution.

kRANDOM_NORMAL 

Generate a tensor with random values drawn from a normal distribution.

◆ GatherMode

enum class nvinfer1::GatherMode : int32_t
strong

Control form of IGatherLayer.

See also
IGatherLayer
Enumerator
kDEFAULT 

Similar to ONNX Gather.

kELEMENT 

Similar to ONNX GatherElements.

kND 

Similar to ONNX GatherND.

◆ HardwareCompatibilityLevel

enum class nvinfer1::HardwareCompatibilityLevel : int32_t
strong

Describes requirements of compatibility with GPU architectures other than that of the GPU on which the engine was built. Levels except kNONE are only supported for engines built on NVIDIA Ampere and later GPUs. Note that compatibility with future hardware depends on CUDA forward compatibility support.

Enumerator
kNONE 

Do not require hardware compatibility with GPU architectures other than that of the GPU on which the engine was built.

kAMPERE_PLUS 

Require that the engine is compatible with Ampere and newer GPUs. This will limit the max shared memory usage to 48KiB, may reduce the number of available tactics for each layer, and may prevent some fusions from occurring. Thus this can decrease the performance, especially for tf32 models. This option will disable cuDNN, cuBLAS, and cuBLAS LT as tactic sources.

◆ InterpolationMode

enum class nvinfer1::InterpolationMode : int32_t
strong

Enumerates various modes of interpolation.

Enumerator
kNEAREST 

ND (0 < N <= 8) nearest neighbor resizing.

kLINEAR 

Supports linear (1D), bilinear (2D), and trilinear (3D) interpolation.

kCUBIC 

Supports bicubic (2D) interpolation.

◆ LayerInformationFormat

enum class nvinfer1::LayerInformationFormat : int32_t
strong

The format in which the IEngineInspector prints the layer information.

See also
IEngineInspector::getLayerInformation(), IEngineInspector::getEngineInformation()
Enumerator
kONELINE 

Print layer information in one line per layer.

kJSON 

Print layer information in JSON format.

◆ LayerType

enum class nvinfer1::LayerType : int32_t
strong

The type values of layer classes.

See also
ILayer::getType()
Enumerator
kCONVOLUTION 

Convolution layer.

kFULLY_CONNECTED 

Fully connected layer.

kACTIVATION 

Activation layer.

kPOOLING 

Pooling layer.

kLRN 

LRN layer.

kSCALE 

Scale layer.

kSOFTMAX 

SoftMax layer.

kDECONVOLUTION 

Deconvolution layer.

kCONCATENATION 

Concatenation layer.

kELEMENTWISE 

Elementwise layer.

kPLUGIN 

Plugin layer.

kUNARY 

UnaryOp operation Layer.

kPADDING 

Padding layer.

kSHUFFLE 

Shuffle layer.

kREDUCE 

Reduce layer.

kTOPK 

TopK layer.

kGATHER 

Gather layer.

kMATRIX_MULTIPLY 

Matrix multiply layer.

kRAGGED_SOFTMAX 

Ragged softmax layer.

kCONSTANT 

Constant layer.

kRNN_V2 

RNNv2 layer.

kIDENTITY 

Identity layer.

kPLUGIN_V2 

PluginV2 layer.

kSLICE 

Slice layer.

kSHAPE 

Shape layer.

kPARAMETRIC_RELU 

Parametric ReLU layer.

kRESIZE 

Resize Layer.

kTRIP_LIMIT 

Loop Trip limit layer.

kRECURRENCE 

Loop Recurrence layer.

kITERATOR 

Loop Iterator layer.

kLOOP_OUTPUT 

Loop output layer.

kSELECT 

Select layer.

kFILL 

Fill layer.

kQUANTIZE 

Quantize layer.

kDEQUANTIZE 

Dequantize layer.

kCONDITION 

Condition layer.

kCONDITIONAL_INPUT 

Conditional Input layer.

kCONDITIONAL_OUTPUT 

Conditional Output layer.

kSCATTER 

Scatter layer.

kEINSUM 

Einsum layer.

kASSERTION 

Assertion layer.

kONE_HOT 

OneHot layer.

kNON_ZERO 

NonZero layer.

kGRID_SAMPLE 

Grid sample layer.

kNMS 

NMS layer.

kREVERSE_SEQUENCE 

Reverse sequence layer.

kNORMALIZATION 

Normalization layer.

kCAST 

Cast layer.

◆ LoopOutput

enum class nvinfer1::LoopOutput : int32_t
strong

Enum that describes kinds of loop outputs.

Enumerator
kLAST_VALUE 

Output value is value of tensor for last iteration.

kCONCATENATE 

Output value is concatenation of values of tensor for each iteration, in forward order.

kREVERSE 

Output value is concatenation of values of tensor for each iteration, in reverse order.

◆ MatrixOperation

enum class nvinfer1::MatrixOperation : int32_t
strong

Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplication.

Enumerator
kNONE 

Treat x as a matrix if it has two dimensions, or as a collection of matrices if x has more than two dimensions, where the last two dimensions are the matrix dimensions. x must have at least two dimensions.

kTRANSPOSE 

Like kNONE, but transpose the matrix dimensions.

kVECTOR 

Treat x as a vector if it has one dimension, or as a collection of vectors if x has more than one dimension. x must have at least one dimension.

The first input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is equivalent to a tensor with dimensions [M, 1, K] with MatrixOperation::kNONE, i.e. is treated as M row vectors of length K, or dimensions [M, K, 1] with MatrixOperation::kTRANSPOSE.

The second input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is equivalent to a tensor with dimensions [M, K, 1] with MatrixOperation::kNONE, i.e. is treated as M column vectors of length K, or dimensions [M, 1, K] with MatrixOperation::kTRANSPOSE.

◆ MemoryPoolType

enum class nvinfer1::MemoryPoolType : int32_t
strong

The type for memory pools used by TensorRT.

See also
IBuilderConfig::setMemoryPoolLimit, IBuilderConfig::getMemoryPoolLimit
Enumerator
kWORKSPACE 

kWORKSPACE is used by TensorRT to store intermediate buffers within an operation. This is equivalent to the deprecated IBuilderConfig::setMaxWorkspaceSize and overrides that value. This defaults to max device memory. Set to a smaller value to restrict tactics that use over the threshold en masse. For more targeted removal of tactics use the IAlgorithmSelector interface.

kDLA_MANAGED_SRAM 

kDLA_MANAGED_SRAM is a fast software managed RAM used by DLA to communicate within a layer. The size of this pool must be at least 4 KiB and must be a power of 2. This defaults to 1 MiB. Orin has capacity of 1 MiB per core, and Xavier shares 4 MiB across all of its accelerator cores.

kDLA_LOCAL_DRAM 

kDLA_LOCAL_DRAM is host RAM used by DLA to share intermediate tensor data across operations. The size of this pool must be at least 4 KiB and must be a power of 2. This defaults to 1 GiB.

kDLA_GLOBAL_DRAM 

kDLA_GLOBAL_DRAM is host RAM used by DLA to store weights and metadata for execution. The size of this pool must be at least 4 KiB and must be a power of 2. This defaults to 512 MiB.

kTACTIC_DRAM 

kTACTIC_DRAM is the device DRAM used by the optimizer to run tactics. On embedded devices, where host and device memory are unified, this includes all host memory required by TensorRT to build the network up to the point of each memory allocation. This defaults to 75% of totalGlobalMem as reported by cudaGetDeviceProperties when cudaGetDeviceProperties.embedded is true, and 100% otherwise.

◆ NetworkDefinitionCreationFlag

enum class nvinfer1::NetworkDefinitionCreationFlag : int32_t
strong

List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFlag is used with createNetworkV2() to specify immutable properties of the network. Creating a network without NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag has been deprecated.

See also
IBuilder::createNetworkV2
Enumerator
kEXPLICIT_BATCH 

Mark the network to be an explicit batch network. Dynamic shape support requires that the kEXPLICIT_BATCH flag is set. With dynamic shapes, any of the input dimensions can vary at run-time, and there are no implicit dimensions in the network specification. Varying dimensions are specified by using the wildcard dimension value -1.

kEXPLICIT_PRECISION 

Deprecated. This flag has no effect now, but is only kept for backward compatability.

◆ OptProfileSelector

enum class nvinfer1::OptProfileSelector : int32_t
strong

When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), select whether we are interested in the minimum, optimum, or maximum values for these parameters. The minimum and maximum specify the permitted range that is supported at runtime, while the optimum value is used for the kernel selection. This should be the "typical" value that is expected to occur at runtime.

See also
IOptimizationProfile::setDimensions(), IOptimizationProfile::setShapeValues()
Enumerator
kMIN 

This is used to set or get the minimum permitted value for dynamic dimensions etc.

kOPT 

This is used to set or get the value that is used in the optimization (kernel selection).

kMAX 

This is used to set or get the maximum permitted value for dynamic dimensions etc.

◆ PaddingMode

enum class nvinfer1::PaddingMode : int32_t
strong

Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, padding mode takes precedence if setPaddingMode() and setPrePadding() are also used.

There are three padding styles, EXPLICIT, SAME, and CAFFE, with each style having two variants. The EXPLICIT and CAFFE styles determine if the final sampling location is used or not. The SAME style determine if the asymmetry in the padding is on the pre or post padding.

Shorthand:
I = dimensions of input image.
B = prePadding, before the image data. For deconvolution, prePadding is set before output.
A = postPadding, after the image data. For deconvolution, postPadding is set after output.
P = delta between input and output
S = stride
F = filter
O = output
D = dilation
M = I + B + A ; The image data plus any padding
DK = 1 + D * (F - 1)

Formulas for Convolution:

  • EXPLICIT_ROUND_DOWN:
    O = floor((M - DK) / S) + 1
  • CAFFE_ROUND_DOWN:
    O = floor((I + B * 2 - DK) / S) + 1
  • EXPLICIT_ROUND_UP:
    O = ceil((M - DK) / S) + 1
  • CAFFE_ROUND_UP:
    O = ceil((I + B * 2 - DK) / S) + 1
  • SAME_UPPER:
    O = ceil(I / S)
    P = floor((I - 1) / S) * S + DK - I;
    B = floor(P / 2)
    A = P - B
  • SAME_LOWER:
    O = ceil(I / S)
    P = floor((I - 1) / S) * S + DK - I;
    A = floor(P / 2)
    B = P - A

Formulas for Deconvolution:

  • EXPLICIT_ROUND_DOWN:
  • CAFFE_ROUND_DOWN:
  • EXPLICIT_ROUND_UP:
  • CAFFE_ROUND_UP:
    O = (I - 1) * S + DK - (B + A)
  • SAME_UPPER:
    O = min(I * S, (I - 1) * S + DK)
    P = max(DK - S, 0)
    B = floor(P / 2)
    A = P - B
  • SAME_LOWER:
    O = min(I * S, (I - 1) * S + DK)
    P = max(DK - S, 0)
    A = floor(P / 2)
    B = P - A

Formulas for Pooling:

  • EXPLICIT_ROUND_DOWN:
    O = floor((M - F) / S) + 1
  • EXPLICIT_ROUND_UP:
    O = ceil((M - F) / S) + 1
  • SAME_UPPER:
    O = ceil(I / S)
    P = floor((I - 1) / S) * S + F - I;
    B = floor(P / 2)
    A = P - B
  • SAME_LOWER:
    O = ceil(I / S)
    P = floor((I - 1) / S) * S + F - I;
    A = floor(P / 2)
    B = P - A
  • CAFFE_ROUND_DOWN:
    EXPLICIT_ROUND_DOWN - ((EXPLICIT_ROUND_DOWN - 1) * S >= I + B)
  • CAFFE_ROUND_UP:
    EXPLICIT_ROUND_UP - ((EXPLICIT_ROUND_UP - 1) * S >= I + B)

Pooling Example 1:

Given I = {6, 6}, B = {3, 3}, A = {2, 2}, S = {2, 2}, F = {3, 3}. What is O?
(B, A can be calculated for SAME_UPPER and SAME_LOWER mode)
  • EXPLICIT_ROUND_DOWN:
    Computation:
    M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11}
    O ==> floor((M - F) / S) + 1
    ==> floor(({11, 11} - {3, 3}) / {2, 2}) + {1, 1}
    ==> floor({8, 8} / {2, 2}) + {1, 1}
    ==> {5, 5}
  • EXPLICIT_ROUND_UP:
    Computation:
    M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11}
    O ==> ceil((M - F) / S) + 1
    ==> ceil(({11, 11} - {3, 3}) / {2, 2}) + {1, 1}
    ==> ceil({8, 8} / {2, 2}) + {1, 1}
    ==> {5, 5}
    The sample points are {0, 2, 4, 6, 8} in each dimension.
  • SAME_UPPER:
    Computation:
    I = {6, 6}
    S = {2, 2}
    O = ceil(I / S) = {3, 3}
    P = floor((I - 1) / S) * S + F - I
    ==> floor(({6, 6} - {1, 1}) / {2, 2}) * {2, 2} + {3, 3} - {6, 6}
    ==> {4, 4} + {3, 3} - {6, 6}
    ==> {1, 1}
    B = floor({1, 1} / {2, 2})
    ==> {0, 0}
    A = {1, 1} - {0, 0}
    ==> {1, 1}
  • SAME_LOWER:
    Computation:
    I = {6, 6}
    S = {2, 2}
    O = ceil(I / S) = {3, 3}
    P = floor((I - 1) / S) * S + F - I
    ==> {1, 1}
    A = floor({1, 1} / {2, 2})
    ==> {0, 0}
    B = {1, 1} - {0, 0}
    ==> {1, 1}
    The sample pointers are {0, 2, 4} in each dimension. SAMPLE_UPPER has {O0, O1, O2, pad} in output in each dimension. SAMPLE_LOWER has {pad, O0, O1, O2} in output in each dimension.

Pooling Example 2:

Given I = {6, 6}, B = {3, 3}, A = {3, 3}, S = {2, 2}, F = {3, 3}. What is O?
  • CAFFE_ROUND_DOWN:
    Computation:
    M = {6, 6} + {3, 3} + {3, 3} ==> {12, 12}
    EXPLICIT_ROUND_DOWN ==> floor((M - F) / S) + 1
    ==> floor(({12, 12} - {3, 3}) / {2, 2}) + {1, 1}
    ==> {5, 5}
    DIFF = (((EXPLICIT_ROUND_DOWN - 1) * S >= I + B) ? {1, 1} : {0, 0})
    ==> ({5, 5} - {1, 1}) * {2, 2} >= {6, 6} + {3, 3} ? {1, 1} : {0,0}
    ==> {0, 0}
    O ==> EXPLICIT_ROUND_DOWN - DIFF
    ==> {5, 5} - {0, 0}
    ==> {5, 5}
  • CAFFE_ROUND_UP:
    Computation:
    M = {6, 6} + {3, 3} + {3, 3} ==> {12, 12}
    EXPLICIT_ROUND_UP ==> ceil((M - F) / S) + 1
    ==> ceil(({12, 12} - {3, 3}) / {2, 2}) + {1, 1}
    ==> {6, 6}
    DIFF = (((EXPLICIT_ROUND_UP - 1) * S >= I + B) ? {1, 1} : {0, 0})
    ==> ({6, 6} - {1, 1}) * {2, 2} >= {6, 6} + {3, 3} ? {1, 1} : {0,0}
    ==> {1, 1}
    O ==> EXPLICIT_ROUND_UP - DIFF
    ==> {6, 6} - {1, 1}
    ==> {5, 5}

The sample points are {0, 2, 4, 6, 8} in each dimension.
CAFFE_ROUND_DOWN and CAFFE_ROUND_UP have two restrictions each on usage with pooling operations. This will cause getDimensions to return an empty dimension and also to reject the network at validation time.
For more information on original reference code, see https://github.com/BVLC/caffe/blob/master/src/caffe/layers/pooling_layer.cpp

  • Restriction 1:
    CAFFE_ROUND_DOWN: B >= F is an error if (B - S) < F
    CAFFE_ROUND_UP: (B + S) >= (F + 1) is an error if B < (F + 1)
  • Restriction 2:
    CAFFE_ROUND_DOWN: (B - S) >= F is an error if B >= F
    CAFFE_ROUND_UP: B >= (F + 1) is an error if (B + S) >= (F + 1)
Enumerator
kEXPLICIT_ROUND_DOWN 

Use explicit padding, rounding output size down.

kEXPLICIT_ROUND_UP 

Use explicit padding, rounding output size up.

kSAME_UPPER 

Use SAME padding, with prePadding <= postPadding.

kSAME_LOWER 

Use SAME padding, with prePadding >= postPadding.

kCAFFE_ROUND_DOWN 

Use CAFFE padding, rounding output size down, uses prePadding value.

kCAFFE_ROUND_UP 

Use CAFFE padding, rounding output size up, uses prePadding value.

◆ PluginFieldType

enum class nvinfer1::PluginFieldType : int32_t
strong

The possible field types for custom layer.

Enumerator
kFLOAT16 

FP16 field type.

kFLOAT32 

FP32 field type.

kFLOAT64 

FP64 field type.

kINT8 

INT8 field type.

kINT16 

INT16 field type.

kINT32 

INT32 field type.

kCHAR 

char field type.

kDIMS 

nvinfer1::Dims field type.

kUNKNOWN 

Unknown field type.

◆ PluginVersion

enum class nvinfer1::PluginVersion : uint8_t
strong
Enumerator
kV2 

IPluginV2.

kV2_EXT 

IPluginV2Ext.

kV2_IOEXT 

IPluginV2IOExt.

kV2_DYNAMICEXT 

IPluginV2DynamicExt.

◆ PoolingType

enum class nvinfer1::PoolingType : int32_t
strong

The type of pooling to perform in a pooling layer.

Enumerator
kMAX 
kAVERAGE 
kMAX_AVERAGE_BLEND 

◆ PreviewFeature

enum class nvinfer1::PreviewFeature : int32_t
strong

Define preview features.

Preview Features have been fully tested but are not yet as stable as other features in TensorRT. They are provided as opt-in features for at least one release.

Enumerator
kFASTER_DYNAMIC_SHAPES_0805 

Optimize runtime dimensions with TensorRT's DL Compiler. Potentially reduces run time and decreases device memory usage and engine size. Models most likely to benefit from enabling kFASTER_DYNAMIC_SHAPES_0805 are transformer-based models, and models containing dynamic control flows.

The default value for this flag is on.

Deprecated:
Turning it off is deprecated in TensorRT 8.6. The flag kFASTER_DYNAMIC_SHAPES_0805 will be removed in 9.0.
kDISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805 

Disable usage of cuDNN/cuBLAS/cuBLASLt tactics in the TensorRT core library.

When the flag is enabled, TensorRT core will not use these tactics even if they are specified in IBuilderConfig::setTacticSources(), but cudnnContext and cublasContext handles will still be passed to plugins via IPluginV2Ext::attachToContext() if the appropriate tactic sources are set.

This allows users to experiment with disabling external library tactics without having to modify their application's plugins to support nullptr handles.

The default value for this flag is on.

See also
TacticSource
kPROFILE_SHARING_0806 

Allows optimization profiles to be shared across execution contexts. This flag defaults to false and will become the default behavior in TensorRT 9.0. At that point this flag will do nothing.

◆ ProfilingVerbosity

enum class nvinfer1::ProfilingVerbosity : int32_t
strong

List of verbosity levels of layer information exposed in NVTX annotations and in IEngineInspector.

See also
IBuilderConfig::setProfilingVerbosity(), IBuilderConfig::getProfilingVerbosity(), IEngineInspector
Enumerator
kLAYER_NAMES_ONLY 

Print only the layer names. This is the default setting.

kNONE 

Do not print any layer information.

kDETAILED 

Print detailed layer information including layer names and layer parameters.

kDEFAULT 
Deprecated:
Deprecated in TensorRT 8.0. Superseded by kLAYER_NAMES_ONLY.
kVERBOSE 
Deprecated:
Deprecated in TensorRT 8.0. Superseded by kDETAILED.

◆ QuantizationFlag

enum class nvinfer1::QuantizationFlag : int32_t
strong

List of valid flags for quantizing the network to int8.

See also
IBuilderConfig::setQuantizationFlag(), IBuilderConfig::getQuantizationFlag()
Enumerator
kCALIBRATE_BEFORE_FUSION 

Run int8 calibration pass before layer fusion. Only valid for IInt8LegacyCalibrator and IInt8EntropyCalibrator. The builder always runs the int8 calibration pass before layer fusion for IInt8MinMaxCalibrator and IInt8EntropyCalibrator2. Disabled by default.

◆ ReduceOperation

enum class nvinfer1::ReduceOperation : int32_t
strong

Enumerates the reduce operations that may be performed by a Reduce layer.

The table shows the result of reducing across an empty volume of a given type.

Operation kFLOAT and kHALF kINT32 kINT8
kSUM 0 0 0
kPROD 1 1 1
kMAX negative infinity INT_MIN -128
kMIN positive infinity INT_MAX 127
kAVG NaN 0 -128

The current version of TensorRT usually performs reduction for kINT8 via kFLOAT or kHALF. The kINT8 values show the quantized representations of the floating-point values.

Enumerator
kSUM 
kPROD 
kMAX 
kMIN 
kAVG 

◆ ResizeCoordinateTransformation

enum class nvinfer1::ResizeCoordinateTransformation : int32_t
strong

The resize coordinate transformation function.

See also
IResizeLayer::setCoordinateTransformation()
Enumerator
kALIGN_CORNERS 

Think of each value in the tensor as a unit volume, and the coordinate is a point inside this volume. The coordinate point is drawn as a star (*) in the below diagram, and multiple values range has a length. Define x_origin as the coordinate of axis x in the input tensor, x_resized as the coordinate of axis x in the output tensor, length_origin as length of the input tensor in axis x, and length_resize as length of the output tensor in axis x.

|<--------------length---------->|
|    0     |    1     |    2     |    3     |
*          *          *          *

x_origin = x_resized * (length_origin - 1) / (length_resize - 1)
kASYMMETRIC 

|<-----------—length------------------—>| | 0 | 1 | 2 | 3 |


x_origin = x_resized * (length_origin / length_resize)

kHALF_PIXEL 

|<-----------—length------------------—>| | 0 | 1 | 2 | 3 |


x_origin = (x_resized + 0.5) * (length_origin / length_resize) - 0.5

◆ ResizeRoundMode

enum class nvinfer1::ResizeRoundMode : int32_t
strong

The rounding mode for nearest neighbor resize.

See also
IResizeLayer::setNearestRounding()
Enumerator
kHALF_UP 

Round half up.

kHALF_DOWN 

Round half down.

kFLOOR 

Round to floor.

kCEIL 

Round to ceil.

◆ ResizeSelector

enum class nvinfer1::ResizeSelector : int32_t
strong

The coordinate selector when resize to single pixel output.

See also
IResizeLayer::setSelectorForSinglePixel()
Enumerator
kFORMULA 

Use formula to map the original index.

kUPPER 

Select the upper left pixel.

◆ RNNDirection

enum class nvinfer1::RNNDirection : int32_t
strong

Enumerates the RNN direction that may be performed by an RNN layer.

See also
IRNNv2Layer
Enumerator
kUNIDIRECTION 

Network iterations from first input to last input.

kBIDIRECTION 

Network iterates from first to last and vice versa and outputs concatenated.

◆ RNNGateType

enum class nvinfer1::RNNGateType : int32_t
strong

Identifies an individual gate within an RNN cell.

See also
RNNOperation
Enumerator
kINPUT 

Input gate (i).

kOUTPUT 

Output gate (o).

kFORGET 

Forget gate (f).

kUPDATE 

Update gate (z).

kRESET 

Reset gate (r).

kCELL 

Cell gate (c).

kHIDDEN 

Hidden gate (h).

◆ RNNInputMode

enum class nvinfer1::RNNInputMode : int32_t
strong

Enumerates the RNN input modes that may occur with an RNN layer.

If the RNN is configured with RNNInputMode::kLINEAR, then for each gate g in the first layer of the RNN, the input vector X[t] (length E) is left-multiplied by the gate's corresponding weight matrix W[g] (dimensions HxE) as usual, before being used to compute the gate output as described by RNNOperation.

If the RNN is configured with RNNInputMode::kSKIP, then this initial matrix multiplication is "skipped" and W[g] is conceptually an identity matrix. In this case, the input vector X[t] must have length H (the size of the hidden state).

See also
IRNNv2Layer
Enumerator
kLINEAR 

Perform the normal matrix multiplication in the first recurrent layer.

kSKIP 

No operation is performed on the first recurrent layer.

◆ RNNOperation

enum class nvinfer1::RNNOperation : int32_t
strong

Enumerates the RNN operations that may be performed by an RNN layer.

Equation definitions

The equations below have the following naming convention:

t := current time step
i := input gate
o := output gate
f := forget gate
z := update gate
r := reset gate
c := cell gate
h := hidden gate
g[t] denotes the output of gate g at timestep t, e.g.
f[t] is the output of the forget gate f.
X[t] := input tensor for timestep t
C[t] := cell state for timestep t
H[t] := hidden state for timestep t
W[g] := W (input) parameter weight matrix for gate g
R[g] := U (recurrent) parameter weight matrix for gate g
Wb[g] := W (input) parameter bias vector for gate g
Rb[g] := U (recurrent) parameter bias vector for gate g
Unless otherwise specified, all operations apply pointwise
to elements of each operand tensor.
ReLU(X) := max(X, 0)
tanh(X) := hyperbolic tangent of X
sigmoid(X) := 1 / (1 + exp(-X))
exp(X) := e^X
A.B denotes matrix multiplication of A and B.
A*B denotes pointwise multiplication of A and B.

Equations

Depending on the value of RNNOperation chosen, each sub-layer of the RNN layer will perform one of the following operations:

::kRELU
H[t] := ReLU(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])
H[t] := tanh(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])
i[t] := sigmoid(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])
f[t] := sigmoid(W[f].X[t] + R[f].H[t-1] + Wb[f] + Rb[f])
o[t] := sigmoid(W[o].X[t] + R[o].H[t-1] + Wb[o] + Rb[o])
c[t] := tanh(W[c].X[t] + R[c].H[t-1] + Wb[c] + Rb[c])
C[t] := f[t]*C[t-1] + i[t]*c[t]
H[t] := o[t]*tanh(C[t])
z[t] := sigmoid(W[z].X[t] + R[z].H[t-1] + Wb[z] + Rb[z])
r[t] := sigmoid(W[r].X[t] + R[r].H[t-1] + Wb[r] + Rb[r])
h[t] := tanh(W[h].X[t] + r[t]*(R[h].H[t-1] + Rb[h]) + Wb[h])
H[t] := (1 - z[t])*h[t] + z[t]*H[t-1]
@ kTANH
TanH activation.
@ kGRU
Three-gate network consisting of Gated Recurrent Units.
@ kLSTM
Four-gate LSTM network w/o peephole connections.
See also
IRNNv2Layer
Enumerator
kRELU 

Single gate RNN w/ ReLU activation function.

kTANH 

Single gate RNN w/ TANH activation function.

kLSTM 

Four-gate LSTM network w/o peephole connections.

kGRU 

Three-gate network consisting of Gated Recurrent Units.

◆ SampleMode

enum class nvinfer1::SampleMode : int32_t
strong

Controls how ISliceLayer and IGridSample handle out-of-bounds coordinates.

See also
ISliceLayer and IGridSample
Enumerator
kSTRICT_BOUNDS 

Fail with error when the coordinates are out of bounds.

kDEFAULT 
kWRAP 

Coordinates wrap around periodically.

Deprecated:
Use kSTRICT_BOUNDS.
kCLAMP 

Out of bounds indices are clamped to bounds.

kFILL 

Use fill input value when coordinates are out of bounds.

kREFLECT 

Coordinates reflect. The axis of reflection is the middle of the perimeter pixel and the reflections are repeated indefinitely within the padded regions. Repeats values for a single pixel and throws error for zero pixels.

◆ ScaleMode

enum class nvinfer1::ScaleMode : int32_t
strong

Controls how shift, scale and power are applied in a Scale layer.

See also
IScaleLayer
Enumerator
kUNIFORM 

Identical coefficients across all elements of the tensor.

kCHANNEL 

Per-channel coefficients.

kELEMENTWISE 

Elementwise coefficients.

◆ ScatterMode

enum class nvinfer1::ScatterMode : int32_t
strong

Control form of IScatterLayer.

See also
IScatterLayer
Enumerator
kELEMENT 

Similar to ONNX ScatterElements.

kND 

Similar to ONNX ScatterND.

◆ TacticSource

enum class nvinfer1::TacticSource : int32_t
strong

List of tactic sources for TensorRT.

See also
TacticSources, IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources(), PreviewFeature::kDISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805
Enumerator
kCUBLAS 

cuBLAS tactics. Enabled by default.

Note
Disabling kCUBLAS will cause the cublas handle passed to plugins in attachToContext to be null.
kCUBLAS_LT 

cuBLAS LT tactics. Enabled for x86 platforms and only enabled for non-x86 platforms when CUDA >= 11.0 by default.

kCUDNN 

cuDNN tactics. Enabled by default.

Note
Disabling kCUDNN will cause the cuDNN handle passed to plugins in attachToContext to be null.
kEDGE_MASK_CONVOLUTIONS 

Enables convolution tactics implemented with edge mask tables. These tactics tradeoff memory for performance by consuming additional memory space proportional to the input size. Enabled by default.

kJIT_CONVOLUTIONS 

Enables convolution tactics implemented with source-code JIT fusion. The engine building time may increase when this is enabled. Enabled by default.

◆ TempfileControlFlag

enum class nvinfer1::TempfileControlFlag : int32_t
strong

Flags used to control TensorRT's behavior when creating executable temporary files.

On some platforms the TensorRT runtime may need to create files in a temporary directory or use platform-specific APIs to create files in-memory to load temporary DLLs that implement runtime code. These flags allow the application to explicitly control TensorRT's use of these files. This will preclude the use of certain TensorRT APIs for deserializing and loading lean runtimes.

Enumerator
kALLOW_IN_MEMORY_FILES 

Allow creating and loading files in-memory (or unnamed files).

kALLOW_TEMPORARY_FILES 

Allow creating and loading named files in a temporary directory on the filesystem.

\see IRuntime::setTemporaryDirectory() 

◆ TensorFormat

enum class nvinfer1::TensorFormat : int32_t
strong

Format of the input/output tensors.

This enum is used by both plugins and network I/O tensors.

See also
IPluginV2::supportsFormat(), safe::ICudaEngine::getBindingFormat()

For more information about data formats, see the topic "Data Format Description" located in the TensorRT Developer Guide.

Enumerator
kLINEAR 

Row major linear format. For a tensor with dimensions {N, C, H, W} or {numbers, channels, columns, rows}, the dimensional index corresponds to {3, 2, 1, 0} and thus the order is W minor.

For DLA usage, the tensor sizes are limited to C,H,W in the range [1,8192].

kCHW2 

Two wide channel vectorized row major format. This format is bound to FP16. It is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+1)/2][H][W][2], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/2][h][w][c%2].

kHWC8 

Eight channel format where C is padded to a multiple of 8. This format is bound to FP16. It is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to the array with dimensions [N][H][W][(C+7)/8*8], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kCHW4 

Four wide channel vectorized row major format. This format is bound to INT8 or FP16. It is only available for dimensions >= 3. For INT8, the C dimension must be a build-time constant. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+3)/4][H][W][4], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/4][h][w][c%4].

Deprecated usage:

If running on the DLA, this format can be used for acceleration with the caveat that C must be equal or lesser than 4. If used as DLA input and the build option kGPU_FALLBACK is not specified, it needs to meet line stride requirement of DLA format. Column stride in bytes should be a multiple of 32 on Xavier and 64 on Orin.

kCHW16 

Sixteen wide channel vectorized row major format. This format is bound to FP16. It is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+15)/16][H][W][16], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/16][h][w][c%16].

For DLA usage, this format maps to the native feature format for FP16, and the tensor sizes are limited to C,H,W in the range [1,8192].

kCHW32 

Thirty-two wide channel vectorized row major format. This format is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+31)/32][H][W][32], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/32][h][w][c%32].

For DLA usage, this format maps to the native feature format for INT8, and the tensor sizes are limited to C,H,W in the range [1,8192].

kDHWC8 

Eight channel format where C is padded to a multiple of 8. This format is bound to FP16, and it is only available for dimensions >= 4. For a tensor with dimensions {N, C, D, H, W}, the memory layout is equivalent to an array with dimensions [N][D][H][W][(C+7)/8*8], with the tensor coordinates (n, c, d, h, w) mapping to array subscript [n][d][h][w][c].

kCDHW32 

Thirty-two wide channel vectorized row major format. This format is bound to FP16 and INT8 and is only available for dimensions >= 4. For a tensor with dimensions {N, C, D, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+31)/32][D][H][W][32], with the tensor coordinates (n, c, d, h, w) mapping to array subscript [n][c/32][d][h][w][c%32].

kHWC 

Non-vectorized channel-last format. This format is bound to either FP32 or UINT8, and is only available for dimensions >= 3.

kDLA_LINEAR 

DLA planar format. For a tensor with dimension {N, C, H, W}, the W axis always has unit stride. The stride for stepping along the H axis is rounded up to 64 bytes.

The memory layout is equivalent to a C array with dimensions [N][C][H][roundUp(W, 64/elementSize)] where elementSize is 2 for FP16 and 1 for Int8, with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c][h][w].

kDLA_HWC4 

DLA image format. For a tensor with dimension {N, C, H, W} the C axis always has unit stride. The stride for stepping along the H axis is rounded up to 32 bytes on Xavier and 64 bytes on Orin. C can only be 1, 3 or 4. If C == 1, it will map to grayscale format. If C == 3 or C == 4, it will map to color image format. And if C == 3, the stride for stepping along the W axis needs to be padded to 4 in elements.

When C is {1, 3, 4}, then C' is {1, 4, 4} respectively, the memory layout is equivalent to a C array with dimensions [N][H][roundUp(W, 32/C'/elementSize)][C'] on Xavier and [N][H][roundUp(W, 64/C'/elementSize)][C'] on Orin where elementSize is 2 for FP16 and 1 for Int8. The tensor coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kHWC16 

Sixteen channel format where C is padded to a multiple of 16. This format is bound to FP16. It is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to the array with dimensions [N][H][W][(C+15)/16*16], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kDHWC 

Non-vectorized channel-last format. This format is bound to FP32. It is only available for dimensions >= 4.

◆ TensorIOMode

enum class nvinfer1::TensorIOMode : int32_t
strong

Definition of tensor IO Mode.

Enumerator
kNONE 

Tensor is not an input or output.

kINPUT 

Tensor is input to the engine.

kOUTPUT 

Tensor is output by the engine.

◆ TensorLocation

enum class nvinfer1::TensorLocation : int32_t
strong

The location for tensor data storage, device or host.

Enumerator
kDEVICE 

Data stored on device.

kHOST 

Data stored on host.

◆ TopKOperation

enum class nvinfer1::TopKOperation : int32_t
strong

Enumerates the operations that may be performed by a TopK layer.

Enumerator
kMAX 

Maximum of the elements.

kMIN 

Minimum of the elements.

◆ TripLimit

enum class nvinfer1::TripLimit : int32_t
strong

Enum that describes kinds of trip limits.

Enumerator
kCOUNT 

Tensor is scalar of type kINT32 that contains the trip count.

kWHILE 

Tensor is a scalar of type kBOOL. Loop terminates when value is false.

◆ UnaryOperation

enum class nvinfer1::UnaryOperation : int32_t
strong

Enumerates the unary operations that may be performed by a Unary layer.

Operations kNOT must have inputs of DataType::kBOOL.

Operation kSIGN must have inputs of DataType::kFLOAT, DataType::kHALF, DataType::kINT8, or DataType::kINT32.

Operation kISINF must have inputs of DataType::kFLOAT or DataType::kHALF.

All other operations must have inputs of DataType::kFLOAT, DataType::kHALF, or DataType::kINT8.

Operations kSIGN and kROUND are not supported in implicit batch mode.

See also
IUnaryLayer
Enumerator
kEXP 

Exponentiation.

kLOG 

Log (base e).

kSQRT 

Square root.

kRECIP 

Reciprocal.

kABS 

Absolute value.

kNEG 

Negation.

kSIN 

Sine.

kCOS 

Cosine.

kTAN 

Tangent.

kSINH 

Hyperbolic sine.

kCOSH 

Hyperbolic cosine.

kASIN 

Inverse sine.

kACOS 

Inverse cosine.

kATAN 

Inverse tangent.

kASINH 

Inverse hyperbolic sine.

kACOSH 

Inverse hyperbolic cosine.

kATANH 

Inverse hyperbolic tangent.

kCEIL 

Ceiling.

kFLOOR 

Floor.

kERF 

Gauss error function.

kNOT 

Logical NOT.

kSIGN 

Sign, If input > 0, output 1; if input < 0, output -1; if input == 0, output 0.

kROUND 

Round to nearest even for floating-point data type.

kISINF 

Return true if input value equals +/- infinity for floating-point data type.

◆ WeightsRole

enum class nvinfer1::WeightsRole : int32_t
strong

How a layer uses particular Weights.

The power weights of an IScaleLayer are omitted. Refitting those is not supported.

Enumerator
kKERNEL 

kernel for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer

kBIAS 

bias for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer

kSHIFT 

shift part of IScaleLayer

kSCALE 

scale part of IScaleLayer

kCONSTANT 

weights for IConstantLayer

kANY 

Any other weights role.

Function Documentation

◆ EnumMax()

template<typename T >
constexpr int32_t nvinfer1::EnumMax ( )
constexprnoexcept

Maximum number of elements in an enumeration type.

◆ EnumMax< BoundingBoxFormat >()

template<>
constexpr int32_t nvinfer1::EnumMax< BoundingBoxFormat > ( )
inlineconstexprnoexcept

Maximum number of elements in BoundingBoxFormat enum.

See also
BoundingBoxFormat

◆ EnumMax< BuilderFlag >()

template<>
constexpr int32_t nvinfer1::EnumMax< BuilderFlag > ( )
inlineconstexprnoexcept

Maximum number of builder flags in BuilderFlag enum.

See also
BuilderFlag

◆ EnumMax< CalibrationAlgoType >()

template<>
constexpr int32_t nvinfer1::EnumMax< CalibrationAlgoType > ( )
inlineconstexprnoexcept

Maximum number of elements in CalibrationAlgoType enum.

See also
DataType

◆ EnumMax< DeviceType >()

template<>
constexpr int32_t nvinfer1::EnumMax< DeviceType > ( )
inlineconstexprnoexcept

Maximum number of elements in DeviceType enum.

See also
DeviceType

◆ EnumMax< DimensionOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< DimensionOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in DimensionOperation enum.

See also
DimensionOperation

◆ EnumMax< FillOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< FillOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in FillOperation enum.

See also
FillOperation

◆ EnumMax< GatherMode >()

template<>
constexpr int32_t nvinfer1::EnumMax< GatherMode > ( )
inlineconstexprnoexcept

Maximum number of elements in GatherMode enum.

See also
GatherMode

◆ EnumMax< LayerInformationFormat >()

template<>
constexpr int32_t nvinfer1::EnumMax< LayerInformationFormat > ( )
inlineconstexprnoexcept

Maximum number of layer information formats in LayerInformationFormat enum.

See also
LayerInformationFormat

◆ EnumMax< LayerType >()

template<>
constexpr int32_t nvinfer1::EnumMax< LayerType > ( )
inlineconstexprnoexcept

Maximum number of elements in LayerType enum.

See also
LayerType

◆ EnumMax< LoopOutput >()

template<>
constexpr int32_t nvinfer1::EnumMax< LoopOutput > ( )
inlineconstexprnoexcept

Maximum number of elements in LoopOutput enum.

See also
DataType

◆ EnumMax< MatrixOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< MatrixOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in MatrixOperation enum.

See also
DataType

◆ EnumMax< MemoryPoolType >()

template<>
constexpr int32_t nvinfer1::EnumMax< MemoryPoolType > ( )
inlineconstexprnoexcept

Maximum number of memory pool types in the MemoryPoolType enum.

See also
MemoryPoolType

◆ EnumMax< NetworkDefinitionCreationFlag >()

template<>
constexpr int32_t nvinfer1::EnumMax< NetworkDefinitionCreationFlag > ( )
inlineconstexprnoexcept

Maximum number of elements in NetworkDefinitionCreationFlag enum.

See also
NetworkDefinitionCreationFlag

◆ EnumMax< OptProfileSelector >()

template<>
constexpr int32_t nvinfer1::EnumMax< OptProfileSelector > ( )
inlineconstexprnoexcept

Number of different values of OptProfileSelector enum.

See also
OptProfileSelector

◆ EnumMax< ProfilingVerbosity >()

template<>
constexpr int32_t nvinfer1::EnumMax< ProfilingVerbosity > ( )
inlineconstexprnoexcept

Maximum number of profile verbosity levels in ProfilingVerbosity enum.

See also
ProfilingVerbosity

◆ EnumMax< QuantizationFlag >()

template<>
constexpr int32_t nvinfer1::EnumMax< QuantizationFlag > ( )
inlineconstexprnoexcept

Maximum number of quantization flags in QuantizationFlag enum.

See also
QuantizationFlag

◆ EnumMax< ReduceOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< ReduceOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in ReduceOperation enum.

See also
ReduceOperation

◆ EnumMax< RNNDirection >()

template<>
constexpr int32_t nvinfer1::EnumMax< RNNDirection > ( )
inlineconstexprnoexcept

Maximum number of elements in RNNDirection enum.

See also
RNNDirection

◆ EnumMax< RNNGateType >()

template<>
constexpr int32_t nvinfer1::EnumMax< RNNGateType > ( )
inlineconstexprnoexcept

Maximum number of elements in RNNGateType enum.

See also
RNNGateType

◆ EnumMax< RNNInputMode >()

template<>
constexpr int32_t nvinfer1::EnumMax< RNNInputMode > ( )
inlineconstexprnoexcept

Maximum number of elements in RNNInputMode enum.

See also
RNNInputMode

◆ EnumMax< RNNOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< RNNOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in RNNOperation enum.

See also
RNNOperation

◆ EnumMax< SampleMode >()

template<>
constexpr int32_t nvinfer1::EnumMax< SampleMode > ( )
inlineconstexprnoexcept

Maximum number of elements in SampleMode enum.

See also
SampleMode

◆ EnumMax< ScaleMode >()

template<>
constexpr int32_t nvinfer1::EnumMax< ScaleMode > ( )
inlineconstexprnoexcept

Maximum number of elements in ScaleMode enum.

See also
ScaleMode

◆ EnumMax< ScatterMode >()

template<>
constexpr int32_t nvinfer1::EnumMax< ScatterMode > ( )
inlineconstexprnoexcept

Maximum number of elements in ScatterMode enum.

See also
ScatterMode

◆ EnumMax< TacticSource >()

template<>
constexpr int32_t nvinfer1::EnumMax< TacticSource > ( )
inlineconstexprnoexcept

Maximum number of tactic sources in TacticSource enum.

See also
TacticSource

◆ EnumMax< TempfileControlFlag >()

template<>
constexpr int32_t nvinfer1::EnumMax< TempfileControlFlag > ( )
inlineconstexprnoexcept

Maximum number of elements in TempfileControlFlag enum.

See also
TempfileControlFlag

◆ EnumMax< TopKOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< TopKOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in TopKOperation enum.

See also
TopKOperation

◆ EnumMax< TripLimit >()

template<>
constexpr int32_t nvinfer1::EnumMax< TripLimit > ( )
inlineconstexprnoexcept

Maximum number of elements in TripLimit enum.

See also
DataType

◆ EnumMax< UnaryOperation >()

template<>
constexpr int32_t nvinfer1::EnumMax< UnaryOperation > ( )
inlineconstexprnoexcept

Maximum number of elements in UnaryOperation enum.

See also
UnaryOperation

◆ EnumMax< WeightsRole >()

template<>
constexpr int32_t nvinfer1::EnumMax< WeightsRole > ( )
inlineconstexprnoexcept

Maximum number of elements in WeightsRole enum.

See also
WeightsRole

◆ getBuilderPluginRegistry()

nvinfer1::IPluginRegistry * nvinfer1::getBuilderPluginRegistry ( nvinfer1::EngineCapability  capability)
noexcept

Return the plugin registry for building a Standard engine, or nullptr if no registry exists.

Also return nullptr if the input argument is not EngineCapability::kSTANDARD. Engine capabilities EngineCapability::kSTANDARD and EngineCapability::kSAFETY have distinct plugin registries. When building a Safety engine, use nvinfer1::getBuilderSafePluginRegistry(). Use IPluginRegistry::registerCreator from the registry to register plugins. Plugins registered in a registry associated with a specific engine capability are only available when building engines with that engine capability.

There is no plugin registry for EngineCapability::kDLA_STANDALONE.

◆ getBuilderSafePluginRegistry()

nvinfer1::safe::IPluginRegistry * nvinfer1::getBuilderSafePluginRegistry ( nvinfer1::EngineCapability  capability)
noexcept

Return the plugin registry for building a Safety engine, or nullptr if no registry exists.

Also return nullptr if the input argument is not EngineCapability::kSAFETY. When building a Standard engine, use nvinfer1::getBuilderPluginRegistry(). Use safe::IPluginRegistry::registerCreator from the registry to register plugins.