TensorRT
6.0.1.5
|
The TensorRT API version 1 namespace. More...
Classes | |
class | Dims |
Structure to define the dimensions of a tensor. More... | |
class | Dims2 |
Descriptor for two-dimensional data. More... | |
class | Dims3 |
Descriptor for three-dimensional data. More... | |
class | Dims4 |
Descriptor for four-dimensional data. More... | |
class | DimsCHW |
Descriptor for data with one channel dimension and two spatial dimensions. More... | |
class | DimsExprs |
class | DimsHW |
Descriptor for two-dimensional spatial data. More... | |
class | DimsNCHW |
Descriptor for data with one index dimension, one channel dimension and two spatial dimensions. More... | |
class | DynamicPluginTensorDesc |
class | IActivationLayer |
An Activation layer in a network definition. More... | |
class | IBuilder |
Builds an engine from a network definition. More... | |
class | IBuilderConfig |
Holds properties for configuring a builder to produce an engine. More... | |
class | IConcatenationLayer |
A concatenation layer in a network definition. More... | |
class | IConstantLayer |
Layer that represents a constant value. More... | |
class | IConvolutionLayer |
A convolution layer in a network definition. More... | |
class | ICudaEngine |
An engine for executing inference on a built network, with functionally unsafe features. More... | |
class | IDeconvolutionLayer |
A deconvolution layer in a network definition. More... | |
class | IDimensionExpr |
class | IElementWiseLayer |
A elementwise layer in a network definition. More... | |
class | IErrorRecorder |
Reference counted application-implemented error reporting interface for TensorRT objects. More... | |
class | IExecutionContext |
Context for executing inference using an engine, with functionally unsafe features. More... | |
class | IExprBuilder |
class | IFullyConnectedLayer |
A fully connected layer in a network definition. This layer expects an input tensor of three or more non-batch dimensions. The input is automatically reshaped into an MxV tensor X , where V is a product of the last three dimensions and M is a product of the remaining dimensions (where the product over 0 dimensions is defined as 1). For example: More... | |
class | IGatherLayer |
class | IGpuAllocator |
Application-implemented class for controlling allocation on the GPU. More... | |
class | IHostMemory |
Class to handle library allocated memory that is accessible to the user. More... | |
class | IIdentityLayer |
A layer that represents the identity function. More... | |
class | IInt8Calibrator |
Application-implemented interface for calibration. More... | |
class | IInt8EntropyCalibrator |
class | IInt8EntropyCalibrator2 |
class | IInt8LegacyCalibrator |
class | IInt8MinMaxCalibrator |
class | ILayer |
Base class for all layer classes in a network definition. More... | |
class | ILogger |
Application-implemented logging interface for the builder, engine and runtime. More... | |
class | ILRNLayer |
A LRN layer in a network definition. More... | |
class | IMatrixMultiplyLayer |
Layer that represents a Matrix Multiplication. More... | |
class | INetworkDefinition |
A network definition for input to the builder. More... | |
class | IOptimizationProfile |
Optimization profile for dynamic input dimensions and shape tensors. More... | |
class | IOutputDimensionsFormula |
Application-implemented interface to compute layer output sizes. More... | |
class | IPaddingLayer |
Layer that represents a padding operation. More... | |
class | IParametricReLULayer |
Layer that represents a parametric ReLU operation. More... | |
class | IPlugin |
Plugin class for user-implemented layers. More... | |
class | IPluginCreator |
Plugin creator class for user implemented layers. More... | |
class | IPluginExt |
Plugin class for user-implemented layers. More... | |
class | IPluginFactory |
Plugin factory for deserialization. More... | |
class | IPluginLayer |
Layer type for plugins. More... | |
class | IPluginRegistry |
Single registration point for all plugins in an application. It is used to find plugin implementations during engine deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also have a corresponding IPluginCreator implementation. More... | |
class | IPluginV2 |
Plugin class for user-implemented layers. More... | |
class | IPluginV2DynamicExt |
class | IPluginV2Ext |
Plugin class for user-implemented layers. More... | |
class | IPluginV2IOExt |
Plugin class for user-implemented layers. More... | |
class | IPluginV2Layer |
Layer type for pluginV2. More... | |
class | IPoolingLayer |
A Pooling layer in a network definition. More... | |
class | IProfiler |
Application-implemented interface for profiling. More... | |
class | IRaggedSoftMaxLayer |
A RaggedSoftmax layer in a network definition. More... | |
class | IReduceLayer |
Layer that represents a reduction operator. More... | |
class | IRefitter |
Updates weights in an engine. More... | |
class | IResizeLayer |
A resize layer in a network definition. More... | |
class | IRNNLayer |
A RNN layer in a network definition. More... | |
class | IRNNv2Layer |
An RNN layer in a network definition, version 2. More... | |
class | IRuntime |
Allows a serialized functionally unsafe engine to be deserialized. More... | |
class | IScaleLayer |
A Scale layer in a network definition. More... | |
class | IShapeLayer |
Layer type for getting shape of a tensor. More... | |
class | IShuffleLayer |
Layer type for shuffling data. More... | |
class | ISliceLayer |
Slices an input tensor into an output tensor based on the offset and strides. More... | |
class | ISoftMaxLayer |
A Softmax layer in a network definition. More... | |
class | ITensor |
A tensor in a network definition. More... | |
class | ITopKLayer |
Layer that represents a TopK reduction. More... | |
class | IUnaryLayer |
Layer that represents an unary operation. More... | |
struct | Permutation |
class | PluginField |
Structure containing plugin attribute field names and associated data This information can be parsed to decode necessary plugin metadata. More... | |
struct | PluginFieldCollection |
class | PluginRegistrar |
Register the plugin creator to the registry The static registry object will be instantiated when the plugin library is loaded. This static object will register all creators available in the library to the registry. More... | |
struct | PluginTensorDesc |
Fields that a plugin might see for an input or output. More... | |
class | Weights |
An array of weights used as a layer parameter. More... | |
Typedefs | |
typedef uint32_t | BuilderFlags |
It is capable of representing one or more BuilderFlags by binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U << BuilderFlag::kDEBUG. More... | |
typedef uint32_t | NetworkDefinitionCreationFlags |
This bitset is capable of representing one or more NetworkDefinitionCreationFlag flags constructed with binary OR operations. e.g., 1U << NetworkDefinitionCreationFlag::kEXPLICIT_BATCH. More... | |
typedef uint32_t | TensorFormats |
It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormats::kCHW4 | 1U << TensorFormats::kCHW32. More... | |
using | PluginFormat = TensorFormat |
PluginFormat is reserved for backward compatibility. More... | |
Functions | |
template<> | |
constexpr int | EnumMax< LayerType > () |
Maximum number of elements in LayerType enum. More... | |
template<> | |
constexpr int | EnumMax< PaddingMode > () |
Maximum number of elements in PaddingMode enum. More... | |
template<> | |
constexpr int | EnumMax< PoolingType > () |
Maximum number of elements in PoolingType enum. More... | |
template<> | |
constexpr int | EnumMax< ScaleMode > () |
Maximum number of elements in ScaleMode enum. More... | |
template<> | |
constexpr int | EnumMax< ElementWiseOperation > () |
Maximum number of elements in ElementWiseOperation enum. More... | |
template<> | |
constexpr int | EnumMax< RNNOperation > () |
Maximum number of elements in RNNOperation enum. More... | |
template<> | |
constexpr int | EnumMax< RNNDirection > () |
Maximum number of elements in RNNDirection enum. More... | |
template<> | |
constexpr int | EnumMax< RNNInputMode > () |
Maximum number of elements in RNNInputMode enum. More... | |
template<> | |
constexpr int | EnumMax< RNNGateType > () |
Maximum number of elements in RNNGateType enum. More... | |
template<> | |
constexpr int | EnumMax< UnaryOperation > () |
Maximum number of elements in UnaryOperation enum. More... | |
template<> | |
constexpr int | EnumMax< ReduceOperation > () |
Maximum number of elements in ReduceOperation enum. More... | |
template<> | |
constexpr int | EnumMax< TopKOperation > () |
Maximum number of elements in TopKOperation enum. More... | |
template<> | |
constexpr int | EnumMax< MatrixOperation > () |
Maximum number of elements in MatrixOperation enum. More... | |
template<> | |
constexpr int | EnumMax< ResizeMode > () |
Maximum number of elements in ResizeMode enum. More... | |
template<> | |
constexpr int | EnumMax< CalibrationAlgoType > () |
Maximum number of elements in CalibrationAlgoType enum. More... | |
template<> | |
constexpr int | EnumMax< BuilderFlag > () |
Maximum number of builder flags in BuilderFlag enum. More... | |
template<> | |
constexpr int | EnumMax< NetworkDefinitionCreationFlag > () |
template<> | |
constexpr int | EnumMax< PluginType > () |
template<> | |
constexpr int | EnumMax< EngineCapability > () |
Maximum number of elements in EngineCapability enum. More... | |
template<> | |
constexpr int | EnumMax< DimensionOperation > () |
Maximum number of elements in DimensionOperation enum. More... | |
template<> | |
constexpr int | EnumMax< WeightsRole > () |
Maximum number of elements in WeightsRole enum. More... | |
template<> | |
constexpr int | EnumMax< DeviceType > () |
Maximum number of elements in DeviceType enum. More... | |
template<> | |
constexpr int | EnumMax< OptProfileSelector > () |
Number of different values of OptProfileSelector enum. More... | |
template<typename T > | |
constexpr int | EnumMax () |
Maximum number of elements in an enumeration type. | |
template<> | |
constexpr int | EnumMax< ActivationType > () |
Maximum number of elements in ActivationType enum. More... | |
template<> | |
constexpr int | EnumMax< DataType > () |
Maximum number of elements in DataType enum. More... | |
template<> | |
constexpr int | EnumMax< DimensionType > () |
Maximum number of elements in DimensionType enum. More... | |
template<> | |
constexpr int | EnumMax< TensorFormat > () |
Maximum number of elements in TensorFormat enum. More... | |
template<> | |
constexpr int | EnumMax< TensorLocation > () |
Maximum number of elements in TensorLocation enum. More... | |
template<> | |
constexpr int | EnumMax< ILogger::Severity > () |
Maximum number of elements in ILogger::Severity enum. More... | |
template<> | |
constexpr int | EnumMax< ErrorCode > () |
Maximum number of elements in ErrorCode enum. More... | |
The TensorRT API version 1 namespace.
typedef uint32_t nvinfer1::BuilderFlags |
It is capable of representing one or more BuilderFlags by binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U << BuilderFlag::kDEBUG.
This bitset is capable of representing one or more NetworkDefinitionCreationFlag flags constructed with binary OR operations. e.g., 1U << NetworkDefinitionCreationFlag::kEXPLICIT_BATCH.
using nvinfer1::PluginFormat = typedef TensorFormat |
PluginFormat is reserved for backward compatibility.
typedef uint32_t nvinfer1::TensorFormats |
It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormats::kCHW4 | 1U << TensorFormats::kCHW32.
|
strong |
Forward declare IGpuAllocator for use in other interfaces.
Enumerates the types of activation to perform in an activation layer.
|
strong |
|
strong |
Version of calibration algorithm to use.
enum CalibrationAlgoType
|
strong |
|
strong |
|
strong |
An operation on two IDimensionExpr, which represent integer expressions used in dimension computations.
For example, given two IDimensionExpr x and y and an IExprBuilder& eb, eb.operation(DimensionOperation::kSUM, x, y) creates a representation of x+y.
|
strong |
|
strong |
Enumerates the binary operations that may be performed by an ElementWise layer.
|
strong |
Forward declaration of IPluginFactory for use by other interfaces.
List of supported engine capability flows.
|
strong |
Error codes that can be returned by TensorRT during execution.
|
strong |
The type values of layer classes.
|
strong |
Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplication.
Enumerator | |
---|---|
kNONE | Treat x as a matrix if it has two dimensions, or as a collection of matrices if x has more than two dimensions, where the last two dimensions are the matrix dimensions. x must have at least two dimensions. |
kTRANSPOSE | Like kNONE, but transpose the matrix dimensions. |
kVECTOR | Treat x as a vector if it has one dimension, or as a collection of vectors if x has more than one dimension. x must have at least one dimension. The first input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is equivalent to a tensor with dimensions [M, 1, K] with MatrixOperation::kNONE, i.e. is treated as M row vectors of length K. If MatrixOperation::kTRANSPOSE is specified, then the dimensions are [M, K, 1]. The second input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is equivalent to a tensor with dimensions [M, K, 1] with MatrixOperation::kNONE, i.e. is treated as M column vectors of length K. If MatrixOperation::kTRANSPOSE is specified, then the dimensions are [M, 1, K]. |
|
strong |
List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFlag is used with createNetworkV2 to specify immutable properties of the network. The createNetwork() function always had an implicit batch dimension being specified by the maxBatchSize builder parameter. createNetworkV2 with kDEFAULT flag mimics that behaviour.
Enumerator | |
---|---|
kEXPLICIT_BATCH | Mark the network to be an explicit batch network. Dynamic shape support requires that the kEXPLICIT_BATCH flag is set. With dynamic shapes, any of the input dimensions can vary at run-time, and there are no implicit dimensions in the network specification. This is specified by using the wildcard dimension value -1. |
kEXPLICIT_PRECISION | Mark the network to be an explicit precision network. Setting the network to be an explicit precision network has the following implications: 1) Precision of all input tensors to the network have to be specified with ITensor::setType() function 2) Precision of all layer output tensors in the network have to be specified using ILayer::setOutputType() function 3) The builder will not quantize the weights of any layer including those running in lower precision(INT8). It will simply cast the weights into the required precision. 4) Dynamic ranges must not be provided to run the network in int8 mode. Dynamic ranges of each tensor in the explicit precision network is [-127,127]. 5) Quantizing and dequantizing activation values between higher (FP32) and lower (INT8) precision will be performed using explicit Scale layers with input/output precision set appropriately. |
|
strong |
When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), select whether we are interested in the minimum, optimum, or maximum values for these parameters. The minimum and maximum specify the permitted range that is supported at runtime, while the optimum value is used for the kernel selection. This should be the "typical" value that is expected to occur at runtime.
|
strong |
Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, padding mode takes precedence if setPaddingMode() and setPrePadding() are also used.
kEXPLICIT* padding is to use explicit padding. kSAME* padding is to implicitly calculate padding to keep output dim to be the "same" with input dim. For convolution and pooling, output dim is ceil(input dim, stride), for deconvolution it is inverse, then use the output dim to calculate padding size. kCAFFE* padding is symmetric padding.
|
strong |
Enumerator | |
---|---|
kFLOAT16 | FP16 field type. |
kFLOAT32 | FP32 field type. |
kFLOAT64 | FP64 field type. |
kINT8 | INT8 field type. |
kINT16 | INT16 field type. |
kINT32 | INT32 field type. |
kCHAR | char field type. |
kDIMS | nvinfer1::Dims field type. |
|
strong |
The type values for the various plugins.
|
strong |
|
strong |
Enumerates the RNN direction that may be performed by an RNN layer.
Enumerator | |
---|---|
kUNIDIRECTION | Network iterations from first input to last input. |
kBIDIRECTION | Network iterates from first to last and vice versa and outputs concatenated. |
|
strong |
Identifies an individual gate within an RNN cell.
Enumerator | |
---|---|
kINPUT | Input gate (i). |
kOUTPUT | Output gate (o). |
kFORGET | Forget gate (f). |
kUPDATE | Update gate (z). |
kRESET | Reset gate (r). |
kCELL | Cell gate (c). |
kHIDDEN | Hidden gate (h). |
|
strong |
Enumerates the RNN input modes that may occur with an RNN layer.
If the RNN is configured with RNNInputMode::kLINEAR, then for each gate g
in the first layer of the RNN, the input vector X[t]
(length E
) is left-multiplied by the gate's corresponding weight matrix W[g]
(dimensions HxE
) as usual, before being used to compute the gate output as described by RNNOperation.
If the RNN is configured with RNNInputMode::kSKIP, then this initial matrix multiplication is "skipped" and W[g]
is conceptually an identity matrix. In this case, the input vector X[t]
must have length H
(the size of the hidden state).
Enumerator | |
---|---|
kLINEAR | Perform the normal matrix multiplication in the first recurrent layer. |
kSKIP | No operation is performed on the first recurrent layer. |
|
strong |
Enumerates the RNN operations that may be performed by an RNN layer.
Equation definitions
In the equations below, we use the following naming convention:
Equations
Depending on the value of RNNOperation chosen, each sub-layer of the RNN layer will perform one of the following operations:
|
strong |
Controls how shift, scale and power are applied in a Scale layer.
Enumerator | |
---|---|
kUNIFORM | Identical coefficients across all elements of the tensor. |
kCHANNEL | Per-channel coefficients. |
kELEMENTWISE | Elementwise coefficients. |
|
strong |
Format of the input/output tensors.
This enum is extended to be used by both plugins and reformat-free network I/O tensors.
For more information about data formats, see the topic "Data Format Description" located in the TensorRT Developer Guide (https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html).
|
strong |
|
strong |
|
strong |
Enumerates the unary operations that may be performed by a Unary layer.
|
strong |
How a layer uses particular Weights.
The power weights of an IScaleLayer are omitted. Refitting those is not supported.
Enumerator | |
---|---|
kKERNEL | kernel for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer |
kBIAS | bias for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer |
kSHIFT | shift part of IScaleLayer |
kSCALE | scale part of IScaleLayer |
kCONSTANT | weights for IConstantLayer |
|
inline |
Maximum number of elements in ActivationType enum.
|
inline |
Maximum number of builder flags in BuilderFlag enum.
|
inline |
Maximum number of elements in CalibrationAlgoType enum.
|
inline |
Maximum number of elements in DataType enum.
|
inline |
Maximum number of elements in DeviceType enum.
|
inline |
Maximum number of elements in DimensionOperation enum.
|
inline |
Maximum number of elements in DimensionType enum.
|
inline |
Maximum number of elements in ElementWiseOperation enum.
|
inline |
Maximum number of elements in EngineCapability enum.
|
inline |
Maximum number of elements in ErrorCode enum.
|
inline |
Maximum number of elements in ILogger::Severity enum.
|
inline |
Maximum number of elements in LayerType enum.
|
inline |
Maximum number of elements in MatrixOperation enum.
|
inline |
Number of different values of OptProfileSelector enum.
|
inline |
Maximum number of elements in PaddingMode enum.
|
inline |
Maximum number of elements in PoolingType enum.
|
inline |
Maximum number of elements in ReduceOperation enum.
|
inline |
Maximum number of elements in ResizeMode enum.
|
inline |
Maximum number of elements in RNNDirection enum.
|
inline |
Maximum number of elements in RNNGateType enum.
|
inline |
Maximum number of elements in RNNInputMode enum.
|
inline |
Maximum number of elements in RNNOperation enum.
|
inline |
Maximum number of elements in ScaleMode enum.
|
inline |
Maximum number of elements in TensorFormat enum.
|
inline |
Maximum number of elements in TensorLocation enum.
|
inline |
Maximum number of elements in TopKOperation enum.
|
inline |
Maximum number of elements in UnaryOperation enum.
|
inline |
Maximum number of elements in WeightsRole enum.