Refitter
- class tensorrt.Refitter(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger) None
Updates weights in an
ICudaEngine
.- Variables
error_recorder –
IErrorRecorder
Application-implemented error reporting interface for TensorRT objects.logger –
ILogger
The logger provided when creating the refitter.max_threads –
int
The maximum thread that can be used by theRefitter
.weights_validation –
bool
The flag to indicate whether to validate weights in the refitting process.
- Parameters
engine – The engine to refit.
logger – The logger to use.
- __del__(self: tensorrt.tensorrt.Refitter) None
- __init__(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger) None
- Parameters
engine – The engine to refit.
logger – The logger to use.
- get_all(self: tensorrt.tensorrt.Refitter) Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]
Get description of all weights that could be refitted.
- Returns
The names of layers with refittable weights, and the roles of those weights.
- get_all_weights(self: tensorrt.tensorrt.Refitter) List[str]
Get names of all weights that could be refitted.
- Returns
The names of refittable weights.
- get_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str) tuple
Gets the dynamic range of a tensor. If the dynamic range was never set, returns the range computed during calibration.
- Parameters
tensor_name – The name of the tensor whose dynamic range to retrieve.
- Returns
Tuple[float, float]
A tuple containing the [minimum, maximum] of the dynamic range.
- get_missing(self: tensorrt.tensorrt.Refitter) Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]
Get description of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.
- Returns
The names of layers with missing weights, and the roles of those weights.
- get_missing_weights(self: tensorrt.tensorrt.Refitter) List[str]
Get names of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.
- Returns
The names of missing weights, empty string for unnamed weights.
- get_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.Weights
Get weights associated with the given name.
If the weights were never set, returns null weights and reports an error to the refitter errorRecorder.
- Parameters
weights_name – The name of the weights to be refitted.
- Returns
Weights associated with the given name.
- get_tensors_with_dynamic_range(self: tensorrt.tensorrt.Refitter) List[str]
Get names of all tensors that have refittable dynamic ranges.
- Returns
The names of tensors with refittable dynamic ranges.
- get_weights_location(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.TensorLocation
Get location for the weights associated with the given name.
If the weights were never set, returns TensorLocation.HOST and reports an error to the refitter errorRecorder.
- Parameters
weights_name – The name of the weights to be refitted.
- Returns
Location for the weights associated with the given name.
- get_weights_prototype(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.Weights
Get the weights prototype associated with the given name.
The dtype and size of weights prototype is the same as weights used for engine building. The size of the weights prototype is -1 when the name of the weights is None or does not correspond to any refittable weights.
- Parameters
weights_name – The name of the weights to be refitted.
- Returns
weights prototype associated with the given name.
- refit_cuda_engine(self: tensorrt.tensorrt.Refitter) bool
Refits associated engine.
If
False
is returned, a subset of weights may have been refitted.The behavior is undefined if the engine has pending enqueued work. Provided weights on CPU or GPU can be unset and released, or updated after refit_cuda_engine returns.
IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead.
- Returns
True
on success, orFalse
if new weights validation fails or get_missing_weights() != 0 before the call.
- refit_cuda_engine_async(self: tensorrt.tensorrt.Refitter, stream_handle: int) bool
Enqueue weights refitting of the associated engine on the given stream.
If
False
is returned, a subset of weights may have been refitted.The behavior is undefined if the engine has pending enqueued work on a different stream from the provided one. Provided weights on CPU can be unset and released, or updated after refit_cuda_engine_async returns. Freeing or updating of the provided weights on GPU can be enqueued on the same stream after refit_cuda_engine_async returns.
IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead. The weights updating task should use the the same stream as the one used for the refit call.
- Parameters
stream – The stream to enqueue the weights updating task.
- Returns
True
on success, orFalse
if new weights validation fails or get_missing_weights() != 0 before the call.
- set_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str, range: List[float]) bool
Update dynamic range for a tensor.
- Parameters
tensor_name – The name of the tensor whose dynamic range to update.
range – The new range.
- Returns
True
if successful,False
otherwise.
Returns false if there is no Int8 engine tensor derived from a network tensor of that name. If successful, then
get_missing()
may report that some weights need to be supplied.
- set_named_weights(*args, **kwargs)
Overloaded function.
set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights) -> bool
Specify new weights of given name. Possible reasons for rejection are:
The name of weights is empty or does not correspond to any refittable weights.
The size of the weights is inconsistent with the size returned from calling
get_weights_prototype()
with the same name.The dtype of the weights is inconsistent with the dtype returned from calling
get_weights_prototype()
with the same name.
Modifying the weights before
refit_cuda_engine()
orrefit_cuda_engine_async()
returns will result in undefined behavior.- arg name
The name of the weights to be refitted.
- arg weights
The new weights to associate with the name.
- returns
True
on success, orFalse
if new weights are rejected.
set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights, location: tensorrt.tensorrt.TensorLocation) -> bool
Specify new weights on a specified device of given name. Possible reasons for rejection are:
The name of weights is empty or does not correspond to any refittable weights.
The size of the weights is inconsistent with the size returned from calling
get_weights_prototype()
with the same name.The dtype of the weights is inconsistent with the dtype returned from calling
get_weights_prototype()
with the same name.
It is allowed to provide some weights on CPU and others on GPU. Modifying the weights before
refit_cuda_engine()
orrefit_cuda_engine_async()
returns will result in undefined behavior.- arg name
The name of the weights to be refitted.
- arg weights
The new weights on the specified device.
- arg location
The location (host vs. device) of the new weights.
- returns
True
on success, orFalse
if new weights are rejected.
- set_weights(self: tensorrt.tensorrt.Refitter, layer_name: str, role: tensorrt.tensorrt.WeightsRole, weights: tensorrt.tensorrt.Weights) bool
Specify new weights for a layer of given name. Possible reasons for rejection are:
There is no such layer by that name.
The layer does not have weights with the specified role.
The size of weights is inconsistent with the layer’s original specification.
Modifying the weights before
refit_cuda_engine()
orrefit_cuda_engine_async()
returns will result in undefined behavior.- Parameters
layer_name – The name of the layer.
role – The role of the weights. See
WeightsRole
for more information.weights – The weights to refit with.
- Returns
True
on success, orFalse
if new weights are rejected.
- unset_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) bool
Unset weights associated with the given name.
Unset weights before releasing them.
- Parameters
weights_name – The name of the weights to be refitted.
- Returns
False
if the weights were never set, returnsTrue
otherwise.