A tensor in a network definition. More...

#include <NvInfer.h>

Inheritance diagram for nvinfer1::ITensor:

Public Member Functions
void	setName (char const *name) noexcept
	Set the tensor name. More...

char const *	getName () const noexcept
	Get the tensor name. More...

void	setDimensions (Dims const &dimensions) noexcept
	Set the dimensions of a tensor. More...

Dims	getDimensions () const noexcept
	Get the dimensions of a tensor. More...

void	setType (DataType type) noexcept
	Set the data type of a tensor. More...

DataType	getType () const noexcept
	Get the data type of a tensor. More...

TRT_DEPRECATED bool	setDynamicRange (float min, float max) noexcept
	Set dynamic range for the tensor. More...

bool	isNetworkInput () const noexcept
	Whether the tensor is a network input. More...

bool	isNetworkOutput () const noexcept
	Whether the tensor is a network output. More...

TRT_DEPRECATED void	setBroadcastAcrossBatch (bool broadcastAcrossBatch) noexcept
	Set whether to enable broadcast of tensor across the implicit batch dimension. More...

TRT_DEPRECATED bool	getBroadcastAcrossBatch () const noexcept
	Check if tensor is broadcast across the implicit batch dimension. More...

TensorLocation	getLocation () const noexcept
	Get the storage location of a tensor. More...

TRT_DEPRECATED void	setLocation (TensorLocation location) noexcept
	Set the storage location of a tensor. More...

TRT_DEPRECATED bool	dynamicRangeIsSet () const noexcept
	Query whether dynamic range is set. More...

void	resetDynamicRange () noexcept
	Undo effect of setDynamicRange. More...

float	getDynamicRangeMin () const noexcept
	Get minimum of dynamic range. More...

float	getDynamicRangeMax () const noexcept
	Get maximum of dynamic range. More...

void	setAllowedFormats (TensorFormats formats) noexcept
	Set allowed formats for an input or output tensor. By default all formats are allowed. Shape tensors (for which isShapeTensor() returns true) may only have row-major linear format. More...

TensorFormats	getAllowedFormats () const noexcept
	Get a bitmask of TensorFormat values that the tensor supports. For a shape tensor, only row-major linear format is allowed. More...

bool	isShapeTensor () const noexcept
	Whether the tensor is a shape tensor. More...

bool	isExecutionTensor () const noexcept
	Whether the tensor is an execution tensor. More...

void	setDimensionName (int32_t index, char const *name) noexcept
	Name a dimension of an input tensor. More...

char const *	getDimensionName (int32_t index) const noexcept
	Get the name of an input dimension. More...

Protected Member Functions
virtual	~ITensor () noexcept=default

Protected Member Functions inherited from nvinfer1::INoCopy
	INoCopy ()=default

virtual	~INoCopy ()=default

	INoCopy (INoCopy const &other)=delete

INoCopy &	operator= (INoCopy const &other)=delete

	INoCopy (INoCopy &&other)=delete

INoCopy &	operator= (INoCopy &&other)=delete

Protected Attributes
apiv::VTensor *	mImpl

Detailed Description

A tensor in a network definition.

To remove a tensor from a network definition, use INetworkDefinition::removeTensor().

When using the DLA, the cumulative size of all Tensors that are not marked as Network Input or Output tensors, must be less than 1GB in size to fit into a single subgraph. If the build option kGPU_FALLBACK is specified, then multiple subgraphs can be created, with each subgraph limited to less than 1GB of internal tensors data.

Warning: Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Constructor & Destructor Documentation

◆ ~ITensor()

virtual nvinfer1::ITensor::~ITensor ( )

protectedvirtualdefaultnoexcept

Member Function Documentation

◆ dynamicRangeIsSet()

TRT_DEPRECATED bool nvinfer1::ITensor::dynamicRangeIsSet ( ) const

inlinenoexcept

Query whether dynamic range is set.

Returns: True if dynamic range is set, false otherwise.

Deprecated:: Deprecated in TensorRT 10.1. Superseded by explicit quantization.

◆ getAllowedFormats()

TensorFormats nvinfer1::ITensor::getAllowedFormats ( ) const

inlinenoexcept

Get a bitmask of TensorFormat values that the tensor supports. For a shape tensor, only row-major linear format is allowed.

Returns: The value specified by setAllowedFormats or all possible formats.

See also: ITensor::setAllowedFormats()

◆ getBroadcastAcrossBatch()

TRT_DEPRECATED bool nvinfer1::ITensor::getBroadcastAcrossBatch ( ) const

inlinenoexcept

Check if tensor is broadcast across the implicit batch dimension.

Returns: Always false since TensorRT 10.0 does not support an implicit batch dimension.

See also: setBroadcastAcrossBatch()

Deprecated:: Deprecated in TensorRT 10.0. Implicit batch is not supported since TensorRT 10.0.

◆ getDimensionName()

char const * nvinfer1::ITensor::getDimensionName ( int32_t index ) const

inlinenoexcept

Get the name of an input dimension.

Parameters

index index of the dimension

Returns: The name of the input dimension, or nullptr if the dimension has no name. The name is a pointer to a null-terminated character sequence.

See also: setDimensionName()

◆ getDimensions()

Dims nvinfer1::ITensor::getDimensions ( ) const

inlinenoexcept

Get the dimensions of a tensor.

Returns: The dimensions of the tensor.

Warning: getDimensions() returns a -1 for dimensions that are derived from a wildcard dimension.

See also: setDimensions()

◆ getDynamicRangeMax()

float nvinfer1::ITensor::getDynamicRangeMax ( ) const

inlinenoexcept

Get maximum of dynamic range.

Returns: Maximum of dynamic range, or quiet NaN if range was not set.

◆ getDynamicRangeMin()

float nvinfer1::ITensor::getDynamicRangeMin ( ) const

inlinenoexcept

Get minimum of dynamic range.

Returns: Minimum of dynamic range, or quiet NaN if range was not set.

◆ getLocation()

TensorLocation nvinfer1::ITensor::getLocation ( ) const

inlinenoexcept

Get the storage location of a tensor.

Returns: The location of tensor data.

See also: setLocation()

◆ getName()

char const * nvinfer1::ITensor::getName ( ) const

inlinenoexcept

Get the tensor name.

Returns: The name as a null-terminated C-style string.

See also: setName()

◆ getType()

DataType nvinfer1::ITensor::getType ( ) const

inlinenoexcept

Get the data type of a tensor.

Returns: The data type of the tensor.

The type is the type set by setType if the tensor is a network input or output. Otherwise the type is the inferred type.

See also: setType()

◆ isExecutionTensor()

bool nvinfer1::ITensor::isExecutionTensor ( ) const

inlinenoexcept

Whether the tensor is an execution tensor.

Tensors are usually execution tensors. The exceptions are tensors used solely for shape calculations or whose contents not needed to compute the outputs.

The result of isExecutionTensor() is reliable only when network construction is complete. For example, if a partially built network has no path from a tensor to a network output, isExecutionTensor() returns false. Completing the path would cause it to become true.

A tensor with isShapeTensor() == false and isExecutionTensor() == false can still show up as an input to the engine if its dimensions are required. In that case, only its dimensions need to be set at runtime and a nullptr can be passed instead of a pointer to its contents.

◆ isNetworkInput()

bool nvinfer1::ITensor::isNetworkInput ( ) const

inlinenoexcept

Whether the tensor is a network input.

◆ isNetworkOutput()

bool nvinfer1::ITensor::isNetworkOutput ( ) const

inlinenoexcept

Whether the tensor is a network output.

◆ isShapeTensor()

bool nvinfer1::ITensor::isShapeTensor ( ) const

inlinenoexcept

Whether the tensor is a shape tensor.

A shape tensor is a tensor that is related to shape calculations. It must have type Int32, Int64, Bool, or Float, and its shape must be determinable at build time. Furthermore, it must be needed as a shape tensor, either marked as a network shape output via markOutputForShapes(), or as a layer input that is required to be a shape tensor, such as the second input to IShuffleLayer. Some layers are "polymorphic" in this respect. For example, the inputs to IElementWiseLayer must be shape tensors if the output is a shape tensor.

The TensorRT Developer Guide give the formal rules for what tensors are shape tensors.

The result of isShapeTensor() is reliable only when network construction is complete. For example, if a partially built network sums two tensors T1 and T2 to create tensor T3, and none are yet needed as shape tensors, isShapeTensor() returns false for all three tensors. Setting the second input of IShuffleLayer to be T3 would cause all three tensors to be shape tensors, because IShuffleLayer requires that its second optional input be a shape tensor, and IElementWiseLayer is "polymorphic".

It is possible for a tensor to be both a shape tensor and an execution tensor.

Returns: True if tensor is a shape tensor, false otherwise.

See also: INetworkDefinition::markOutputForShapes()

◆ resetDynamicRange()

void nvinfer1::ITensor::resetDynamicRange ( )

inlinenoexcept

Undo effect of setDynamicRange.

◆ setAllowedFormats()

void nvinfer1::ITensor::setAllowedFormats ( TensorFormats formats )

inlinenoexcept

Set allowed formats for an input or output tensor. By default all formats are allowed. Shape tensors (for which isShapeTensor() returns true) may only have row-major linear format.

When running network on DLA and the build option kGPU_FALLBACK is not specified, if DLA format(kCHW4 with Int8, kCHW4 with FP16, kCHW16 with FP16, kCHW32 with Int8) is set, the input format is treated as native DLA format with line stride requirement. Input/output binding with these format should have correct layout during inference.

Tensor formats are determined at build time by TensorRT for tensors not marked as input or output.

Parameters

formats A bitmask of TensorFormat values that are supported for this tensor.

See also: ITensor::getAllowedFormats(); TensorFormats

◆ setBroadcastAcrossBatch()

TRT_DEPRECATED void nvinfer1::ITensor::setBroadcastAcrossBatch ( bool broadcastAcrossBatch )

inlinenoexcept

Set whether to enable broadcast of tensor across the implicit batch dimension.

Warning: This method has no effect other than issuing a warning.

Parameters

broadcastAcrossBatch Whether to broadcast the tensor across the implicit batch dimension that was a feature of TensorRT 9.x and prior.

See also: getBroadcastAcrossBatch()

Deprecated:: Deprecated in TensorRT 10.0. Implicit batch is not supported since TensorRT 10.0.

◆ setDimensionName()

void nvinfer1::ITensor::setDimensionName	(	int32_t	index,
		char const *	name
	)

inlinenoexcept

Name a dimension of an input tensor.

Associate a runtime dimension of an input tensor with a symbolic name. Dimensions with the same non-empty name must be equal at runtime. Knowing this equality for runtime dimensions may help the TensorRT optimizer. Both runtime and build-time dimensions can be named.

For example, setDimensionName(0, "n") associates the symbolic name "n" with the leading dimension.

This method copies the name string. If the function is called again, with the same index, it will overwrite the previous name. If nullptr is passed as name, it will clear the name of the dimension.

Parameters

index	index of the dimension
name	of the dimension, as a pointer to a null-terminated character sequence.

Warning: The string name must be null-terminated, and be at most 4096 bytes including the terminator.

See also: getDimensionName()

◆ setDimensions()

void nvinfer1::ITensor::setDimensions ( Dims const & dimensions )

inlinenoexcept

Set the dimensions of a tensor.

For a network input, the dimensions are assigned by the application. For a network output, the dimensions are computed based on the layer parameters and the inputs to the layer. If a tensor size or a parameter is modified in the network, the dimensions of all dependent tensors will be recomputed.

This call is only legal for network input tensors, since the dimensions of layer output tensors are inferred based on layer inputs and parameters.

Parameters

dimensions The dimensions of the tensor.

See also: getDimensions()

◆ setDynamicRange()

TRT_DEPRECATED bool nvinfer1::ITensor::setDynamicRange	(	float	min,
		float	max
	)

inlinenoexcept

Set dynamic range for the tensor.

Currently, only symmetric ranges are supported. Therefore, the larger of the absolute values of the provided bounds is used.

Returns: Whether the dynamic range was set successfully.

Requires that min and max be finite, and min <= max.

Deprecated:: Deprecated in TensorRT 10.1. Superseded by explicit quantization.

◆ setLocation()

TRT_DEPRECATED void nvinfer1::ITensor::setLocation ( TensorLocation location )

inlinenoexcept

Set the storage location of a tensor.

Parameters

location the location of tensor data

Only network input tensors for storing sequence lengths for RNNv2 are supported. Using host storage for layers that do not support it will generate errors at build time.

See also: getLocation()

Deprecated:: Deprecated in TensorRT 10.0. RNNv2 is not supported and the location must always be TensorLocation::kDEVICE since TensorRT 10.0.

◆ setName()

void nvinfer1::ITensor::setName ( char const * name )

inlinenoexcept

Set the tensor name.

For a network input, the name is assigned by the application. For tensors which are layer outputs, a default name is assigned consisting of the layer name followed by the index of the output in brackets. Each input and output tensor must have a unique name.

This method copies the name string.

Parameters

name The name.

Warning: The string name must be null-terminated, and be at most 4096 bytes including the terminator.

See also: getName()

◆ setType()

void nvinfer1::ITensor::setType ( DataType type )

inlinenoexcept

Set the data type of a tensor.

Parameters

type	The data type of the tensor when the type is not inferred.

For strongly typed networks, this method should be used only for network inputs, since the types of all other tensors are inferred. Setting the type of a network output is tolerated if the type equals the inferred type, otherwise an error occurs and the type is not updated.

For weakly typed networks, this method can be used for network outputs too, but the type merely has to be implicitly convertible from the inferred type to the specified type. In this case it does not matter whether the type is set first or the tensor is marked as an output first (via INetworkDefinition::markOutput or INetworkDefinition::markOutputForShapes).

However, marking it first has two advantages:

* It avoids warnings that the tensor is not yet a network I/O tensor.
* It causes method `getType()` to return the type that was set instead of the inferred type.

See also: getType()

Note: This function does more than just set the type, so t.setType(t.getType()) is not necessarily a no-op, particularly for input and output tensors!; Repeated consecutive applications of t.setType(t.getType()) would be idempotent, provided the state of the ITensor isn't changed between calls.

Member Data Documentation

◆ mImpl

apiv::VTensor* nvinfer1::ITensor::mImpl

protected

The documentation for this class was generated from the following file:

NvInfer.h

Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ ~ITensor()

Member Function Documentation

◆ dynamicRangeIsSet()

◆ getAllowedFormats()

◆ getBroadcastAcrossBatch()

◆ getDimensionName()

◆ getDimensions()

◆ getDynamicRangeMax()

◆ getDynamicRangeMin()

◆ getLocation()

◆ getName()

◆ getType()

◆ isExecutionTensor()

◆ isNetworkInput()

◆ isNetworkOutput()

◆ isShapeTensor()

◆ resetDynamicRange()

◆ setAllowedFormats()

◆ setBroadcastAcrossBatch()

◆ setDimensionName()

◆ setDimensions()

◆ setDynamicRange()

◆ setLocation()

◆ setName()

◆ setType()

Member Data Documentation

◆ mImpl