TensorRT
6.0.1.5
|
An engine for executing inference on a built network, with functionally unsafe features. More...
#include <NvInferRuntime.h>
Public Member Functions | |
virtual int | getNbBindings () const noexcept=0 |
Get the number of binding indices. More... | |
virtual int | getBindingIndex (const char *name) const noexcept=0 |
Retrieve the binding index for a named tensor. More... | |
virtual const char * | getBindingName (int bindingIndex) const noexcept=0 |
Retrieve the name corresponding to a binding index. More... | |
virtual bool | bindingIsInput (int bindingIndex) const noexcept=0 |
Determine whether a binding is an input binding. More... | |
virtual Dims | getBindingDimensions (int bindingIndex) const noexcept=0 |
Get the dimensions of a binding. More... | |
virtual DataType | getBindingDataType (int bindingIndex) const noexcept=0 |
Determine the required data type for a buffer from its binding index. More... | |
virtual int | getMaxBatchSize () const noexcept=0 |
Get the maximum batch size which can be used for inference. More... | |
virtual int | getNbLayers () const noexcept=0 |
Get the number of layers in the network. More... | |
virtual TRT_DEPRECATED std::size_t | getWorkspaceSize () const noexcept=0 |
Get the amount of workspace the engine uses. More... | |
virtual IHostMemory * | serialize () const noexcept=0 |
Serialize the network to a stream. More... | |
virtual IExecutionContext * | createExecutionContext () noexcept=0 |
Create an execution context. More... | |
virtual void | destroy () noexcept=0 |
Destroy this object;. | |
virtual TensorLocation | getLocation (int bindingIndex) const noexcept=0 |
Get location of binding. More... | |
virtual IExecutionContext * | createExecutionContextWithoutDeviceMemory () noexcept=0 |
create an execution context without any device memory allocated More... | |
virtual size_t | getDeviceMemorySize () const noexcept=0 |
Return the amount of device memory required by an execution context. More... | |
virtual bool | isRefittable () const noexcept=0 |
Return true if engine can be refit. More... | |
virtual int | getBindingBytesPerComponent (int bindingIndex) const noexcept=0 |
Return the number of bytes per component of an element. More... | |
virtual int | getBindingComponentsPerElement (int bindingIndex) const noexcept=0 |
Return the number of components included in one element. More... | |
virtual TensorFormat | getBindingFormat (int bindingIndex) const noexcept=0 |
Return the binding format. More... | |
virtual const char * | getBindingFormatDesc (int bindingIndex) const noexcept=0 |
Return the human readable description of the tensor format. More... | |
virtual int | getBindingVectorizedDim (int bindingIndex) const noexcept=0 |
Return the dimension index that the buffer is vectorized. More... | |
virtual const char * | getName () const noexcept=0 |
Returns the name of the network associated with the engine. More... | |
virtual int | getNbOptimizationProfiles () const noexcept=0 |
Get the number of optimization profiles defined for this engine. More... | |
virtual Dims | getProfileDimensions (int bindingIndex, int profileIndex, OptProfileSelector select) const noexcept=0 |
Get the minimum / optimum / maximum dimensions for a particular binding under an optimization profile. More... | |
virtual const int32_t * | getProfileShapeValues (int profileIndex, int inputIndex, OptProfileSelector select) const noexcept=0 |
Get minimum / optimum / maximum values for an input shape binding under an optimization profile. More... | |
virtual bool | isShapeBinding (int bindingIndex) const noexcept=0 |
True if tensor is required as input for shape calculations or output from them. More... | |
virtual bool | isExecutionBinding (int bindingIndex) const noexcept=0 |
True if pointer to tensor data is required for execution phase, false if nullptr can be supplied. More... | |
virtual EngineCapability | getEngineCapability () const noexcept=0 |
determine that execution capability this engine has. More... | |
virtual void | setErrorRecorder (IErrorRecorder *recorder) noexcept=0 |
Set the ErrorRecorder for this interface. More... | |
virtual IErrorRecorder * | getErrorRecorder () const noexcept=0 |
get the ErrorRecorder assigned to this interface. More... | |
An engine for executing inference on a built network, with functionally unsafe features.
|
pure virtualnoexcept |
Determine whether a binding is an input binding.
bindingIndex | The binding index. |
|
pure virtualnoexcept |
Create an execution context.
|
pure virtualnoexcept |
create an execution context without any device memory allocated
The memory for execution of this device context must be supplied by the application.
|
pure virtualnoexcept |
Return the number of bytes per component of an element.
The vector component size is returned if getBindingVectorizedDim() != -1.
bindingIndex | The binding Index. |
|
pure virtualnoexcept |
Return the number of components included in one element.
The number of elements in the vectors is returned if getBindingVectorizedDim() != -1.
bindingIndex | The binding Index. |
|
pure virtualnoexcept |
Determine the required data type for a buffer from its binding index.
bindingIndex | The binding index. |
|
pure virtualnoexcept |
Get the dimensions of a binding.
bindingIndex | The binding index. |
|
pure virtualnoexcept |
Return the binding format.
bindingIndex | The binding Index. |
|
pure virtualnoexcept |
Return the human readable description of the tensor format.
The description includes the order, vectorization, data type, strides, and etc. Examples are shown as follows: Example 1: kCHW + FP32 "Row major linear FP32 format" Example 2: kCHW2 + FP16 "Two wide channel vectorized row major FP16 format" Example 3: kHWC8 + FP16 + Line Stride = 32 "Channel major FP16 format where C % 8 == 0 and H Stride % 32 == 0"
bindingIndex | The binding Index. |
|
pure virtualnoexcept |
Retrieve the binding index for a named tensor.
IExecutionContext::enqueue() and IExecutionContext::execute() require an array of buffers.
Engine bindings map from tensor names to indices in this array. Binding indices are assigned at engine build time, and take values in the range [0 ... n-1] where n is the total number of inputs and outputs.
name | The tensor name. |
|
pure virtualnoexcept |
Retrieve the name corresponding to a binding index.
This is the reverse mapping to that provided by getBindingIndex().
bindingIndex | The binding index. |
|
pure virtualnoexcept |
Return the dimension index that the buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
bindingIndex | The binding Index. |
|
pure virtualnoexcept |
Return the amount of device memory required by an execution context.
|
pure virtualnoexcept |
determine that execution capability this engine has.
If the engine has EngineCapability::kDEFAULT, then all engine functionality is valid.. If the engine has EngineCapability::kSAFE_GPU, then only the functionality in safe::ICudaEngine is valid. If the engine has EngineCapability::kSAFE_DLA, then only serialize, destroy, and const-accessor functions are valid.
|
pure virtualnoexcept |
get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will be returned if setErrorRecorder has not been called.
|
pure virtualnoexcept |
Get location of binding.
This lets you know whether the binding should be a pointer to device or host memory.
bindingIndex | The binding index. |
|
pure virtualnoexcept |
Get the maximum batch size which can be used for inference.
|
pure virtualnoexcept |
Returns the name of the network associated with the engine.
The name is set during network creation and is retrieved after building or deserialization.
|
pure virtualnoexcept |
Get the number of binding indices.
|
pure virtualnoexcept |
Get the number of layers in the network.
The number of layers in the network is not necessarily the number in the original network definition, as layers may be combined or eliminated as the engine is optimized. This value can be useful when building per-layer tables, such as when aggregating profiling data over a number of executions.
|
pure virtualnoexcept |
Get the number of optimization profiles defined for this engine.
|
pure virtualnoexcept |
Get the minimum / optimum / maximum dimensions for a particular binding under an optimization profile.
bindingIndex | The binding index (must be between 0 and getNbBindings() - 1) |
profileIndex | The profile index (must be between 0 and getNbOptimizationProfiles()-1) |
select | Whether to query the minimum, optimum, or maximum dimensions for this binding. |
|
pure virtualnoexcept |
Get minimum / optimum / maximum values for an input shape binding under an optimization profile.
profileIndex | The profile index (must be between 0 and getNbOptimizationProfiles()-1) |
inputIndex | The input index (must be between 0 and getNbBindings() - 1) |
select | Whether to query the minimum, optimum, or maximum shape values for this binding. |
|
pure virtualnoexcept |
Get the amount of workspace the engine uses.
The workspace size will be no greater than the value provided to the builder when the engine was built, and will typically be smaller. Workspace will be allocated for each execution context.
|
pure virtualnoexcept |
True if pointer to tensor data is required for execution phase, false if nullptr can be supplied.
For example, if a network uses an input tensor with binding i ONLY as the "reshape dimensions" input of IShuffleLayer, then isExecutionBinding(i) is false, and a nullptr can be supplied for it when calling IExecutionContext::execute or IExecutionContext::enqueue.
|
pure virtualnoexcept |
Return true if engine can be refit.
|
pure virtualnoexcept |
True if tensor is required as input for shape calculations or output from them.
TensorRT evaluates a network in two phases:
Some tensors are required in phase 1. These tensors are called "shape tensors", and always have type Int32 and no more than one dimension. These tensors are not always shapes themselves, but might be used to calculate tensor shapes for phase 2.
isShapeBinding(i) returns true if the tensor is a required input or an output computed in phase 1. isExecutionBinding(i) returns true if the tensor is a required input or an output computed in phase 2.
For example, if a network uses an input tensor with binding i as an addend to an IElementWiseLayer that computes the "reshape dimensions" for IShuffleLayer, then isShapeBinding(i) == true.
It's possible to have a tensor be required by both phases. For instance, a tensor can be used for the "reshape dimensions" and as the indices for an IGatherLayer collecting floating-point data.
It's also possible to have a tensor be required by neither phase, but nonetheless shows up in the engine's inputs. For example, if an input tensor is used only as an input to IShapeLayer, only its shape matters and its values are irrelevant.
|
pure virtualnoexcept |
Serialize the network to a stream.
The network may be deserialized with IRuntime::deserializeCudaEngine() and also safe::IRuntime::deserializeCudaEngine() if only functional-safe features are used in the engine.
|
pure virtualnoexcept |
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder with the interface, resulting in a call to decRefCount if a recorder has been registered.
recorder | The error recorder to register with this interface. |