Layer Base Classes¶
ITensor¶
-
tensorrt.
TensorLocation
¶ The physical location of the data.
Members:
DEVICE : Data is stored on the device.
HOST : Data is stored on the device.
-
tensorrt.
TensorFormat
¶ Format of the input/output tensors.
This enum is extended to be used by both plugins and reformat-free network I/O tensors.
For more information about data formats, see the topic “Data Format Description” located in the TensorRT Developer Guide (https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html).
Members:
- CHW32 :
Thirty-two wide channel vectorized row major format.
This format is bound to INT8. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+31)/32][H][W][32], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/32][h][w][c%32].
- CHW4 :
Four wide channel vectorized row major format. This format is bound to INT8. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+3)/4][H][W][4], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/4][h][w][c%4].
- CHW2 :
Two wide channel vectorized row major format.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+1)/2][H][W][2], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/2][h][w][c%2].
- LINEAR :
Row major linear format.
For a tensor with dimensions {N, C, H, W} or {numbers, channels, columns, rows}, the dimensional index corresponds to {3, 2, 1, 0} and thus the order is W major.
- HWC8 :
Eight channel format where C is padded to a multiple of 8.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, H, W, C}, the memory layout is equivalent to the array with dimensions [N][H][W][(C+7)/8*8], with the tensor coordinates (n, h, w, c) mapping to array subscript [n][h][w][c].
- CHW16 :
Sixteen wide channel vectorized row major format.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with dimensions [N][(C+15)/16][H][W][16], with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c/16][h][w][c%16].
-
class
tensorrt.
ITensor
¶ A tensor in an
INetworkDefinition
.- Variables
name –
str
The tensor name. For a network input, the name is assigned by the application. For tensors which are layer outputs, a default name is assigned consisting of the layer name followed by the index of the output in brackets.shape –
Dims
The shape of a tensor. For a network input the shape is assigned by the application. For a network output it is computed based on the layer parameters and the inputs to the layer. If a tensor size or a parameter is modified in the network, the shape of all dependent tensors will be recomputed. This call is only legal for network input tensors, since the shape of layer output tensors are inferred based on layer inputs and parameters.dtype –
DataType
The data type of a tensor. The type is unchanged if the type is invalid for the given tensor. If the tensor is a network input or output, then the tensor type cannot betensorrt.int8
.broadcast_across_batch –
bool
Whether to enable broadcast of tensor across the batch. When a tensor is broadcast across a batch, it has the same value for every member in the batch. Memory is only allocated once for the single member. This method is only valid for network input tensors, since the flags of layer output tensors are inferred based on layer inputs and parameters. If this state is modified for a tensor in the network, the states of all dependent tensors will be recomputed.location –
TensorLocation
The storage location of a tensor.is_network_input –
bool
Whether the tensor is a network input.is_network_output –
bool
Whether the tensor is a network output.dynamic_range –
Tuple[float, float]
A tuple containing the [minimum, maximum] of the dynamic range, orNone
if the range was not set.is_shape –
bool
Whether the tensor is a shape tensor.allowed_formats –
int
The allowed set of TensorFormat candidates. This should be an integer consisting of one or moreTensorFormat
s, combined via bitwise OR after bit shifting. For example,1 << int(TensorFormats.CHW4) | 1 << int(TensorFormat.CHW32)
.
-
get_dynamic_range
(self: tensorrt.tensorrt.ITensor) → float¶ Get dynamic range for the tensor. NOTE: It is suggested to use
tensor.dynamic_range
instead, which is a tuple including both the minimum and maximum of the dynamic range.- Returns
The absolute maximum of the dynamic range.
-
reset_dynamic_range
(self: tensorrt.tensorrt.ITensor) → None¶ Undo the effect of setting the dynamic range.
-
set_dynamic_range
(self: tensorrt.tensorrt.ITensor, min: float, max: float) → bool¶ Set dynamic range for the tensor. NOTE: It is suggested to use
tensor.dynamic_range = (min, max)
instead.- Parameters
min – Minimum of the dynamic range.
max – Maximum of the dyanmic range.
- Returns
true if succeed in setting range. Otherwise false.
ILayer¶
-
tensorrt.
LayerType
¶ Type of Layer
Members:
ITERATOR : Loop Iterator layer
CONCATENATION : Concatenation layer
FILL : Fill layer
CONVOLUTION : Convolution layer
TOPK : TopK layer
RNN : RNN layer
PADDING : Padding layer
MATRIX_MULTIPLY : Matrix multiply layer
GATHER : Gather layer
FULLY_CONNECTED : Fully connected layer
TRIP_LIMIT : Loop Trip limit layer
RAGGED_SOFTMAX : Ragged softmax layer
RECURRENCE : Loop Recurrence layer
POOLING : Pooling layer
LRN : LRN layer
RESIZE : Resize layer
UNARY : Unary layer
PLUGIN : Plugin layer
SOFTMAX : Softmax layer
PARAMETRIC_RELU : Parametric ReLU layer
CONSTANT : Constant layer
SLICE : Slice layer
SCALE : Scale layer
PLUGIN_V2 : PluginV2 layer
LOOP_OUTPUT : Loop output layer
DECONVOLUTION : Deconvolution layer
SHAPE : Shape layer
IDENTITY : Identity layer
REDUCE : Reduce layer
ELEMENTWISE : Elementwise layer
SHUFFLE : Shuffle layer
SELECT : Select layer
RNN_V2 : RNNv2 layer
ACTIVATION : Activation layer
-
class
tensorrt.
ILayer
¶ Base class for all layer classes in an
INetworkDefinition
.- Variables
-
get_input
(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.ITensor¶ Get the layer input corresponding to the given index.
- Parameters
index – The index of the input tensor.
- Returns
The input tensor, or
None
if the index is out of range.
-
get_output
(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.ITensor¶ Get the layer output corresponding to the given index.
- Parameters
index – The index of the output tensor.
- Returns
The output tensor, or
None
if the index is out of range.
-
get_output_type
(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.DataType¶ Get the output type of the layer.
- Parameters
index – The index of the output tensor.
- Returns
The output precision. Default : DataType.FLOAT.
-
output_type_is_set
(self: tensorrt.tensorrt.ILayer, index: int) → bool¶ Whether the output type has been set for this layer.
- Parameters
index – The index of the output.
- Returns
Whether the output type has been explicitly set.
-
reset_output_type
(self: tensorrt.tensorrt.ILayer, index: int) → None¶ Reset output type of this layer.
- Parameters
index – The index of the output.
-
reset_precision
(self: tensorrt.tensorrt.ILayer) → None¶ Reset the computation precision of the layer.
-
set_input
(self: tensorrt.tensorrt.ILayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None¶ Set the layer input corresponding to the given index.
- Parameters
index – The index of the input tensor.
tensor – The input tensor.
-
set_output_type
(self: tensorrt.tensorrt.ILayer, index: int, dtype: tensorrt.tensorrt.DataType) → None¶ Constraint layer to generate output data with given type. Note that this method cannot be used to set the data type of the second output tensor of the topK layer. The data type of the second output tensor of the topK layer is always Int32.
- Parameters
index – The index of the output tensor to set the type.
dtype – DataType of the output.
IOutputDimensionsFormula¶
-
class
tensorrt.
IOutputDimensionsFormula
¶ Application-implemented interface to compute layer output sizes.
-
compute
(self: tensorrt.tensorrt.IOutputDimensionsFormula, input_shape: tensorrt.tensorrt.DimsHW, kernel_shape: tensorrt.tensorrt.DimsHW, stride: tensorrt.tensorrt.DimsHW, padding: tensorrt.tensorrt.DimsHW, dilation: tensorrt.tensorrt.DimsHW, layer_name: str) → tensorrt.tensorrt.DimsHW¶ Application-implemented interface to compute the HW output dimensions of a layer from the layer input and parameters.
- Parameters
input_shape – The input shape of the layer.
kernel_shape – The kernel shape (or window size, for a pooling layer) parameter of the layer operation.
stride – The stride parameter for the layer.
padding – The padding parameter of the layer.
dilation – The dilation parameter of the layer (only applicable to convolutions).
layer_name – The name of the layer.
- Returns
The output size of the layer
-