Refitter¶

class tensorrt.Refitter(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger)¶

Updates weights in an ICudaEngine .

Variables:

error_recorder – IErrorRecorder Application-implemented error reporting interface for TensorRT objects.
logger – ILogger The logger provided when creating the refitter.
max_threads – int The maximum thread that can be used by the Refitter.
weights_validation – bool The flag to indicate whether to validate weights in the refitting process.

Parameters:

engine – The engine to refit.
logger – The logger to use.

__del__(self: tensorrt.tensorrt.Refitter) → None¶

__init__(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger) → None¶

Parameters:

engine – The engine to refit.
logger – The logger to use.

get_all(self: tensorrt.tensorrt.Refitter) → Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]¶

Get description of all weights that could be refitted.

Returns:: The names of layers with refittable weights, and the roles of those weights.

get_all_weights(self: tensorrt.tensorrt.Refitter) → List[str]¶

Get names of all weights that could be refitted.

Returns:: The names of refittable weights.

get_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str) → tuple¶

[DEPRECATED] Deprecated in TensorRT 10.1. Superseded by explicit quantization.

Gets the dynamic range of a tensor. If the dynamic range was never set, returns the range computed during calibration.

Parameters:: tensor_name – The name of the tensor whose dynamic range to retrieve.
Returns:: Tuple[float, float] A tuple containing the [minimum, maximum] of the dynamic range.

get_missing(self: tensorrt.tensorrt.Refitter) → Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]¶

Get description of missing weights.

For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.

Returns:: The names of layers with missing weights, and the roles of those weights.

get_missing_weights(self: tensorrt.tensorrt.Refitter) → List[str]¶

Get names of missing weights.

For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.

Returns:: The names of missing weights, empty string for unnamed weights.

get_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) → tensorrt.tensorrt.Weights¶

Get weights associated with the given name.

If the weights were never set, returns null weights and reports an error to the refitter errorRecorder.

Parameters:: weights_name – The name of the weights to be refitted.
Returns:: Weights associated with the given name.

get_tensors_with_dynamic_range(self: tensorrt.tensorrt.Refitter) → List[str]¶

[DEPRECATED] Deprecated in TensorRT 10.1. Superseded by explicit quantization.

Get names of all tensors that have refittable dynamic ranges.

Returns:: The names of tensors with refittable dynamic ranges.

get_weights_location(self: tensorrt.tensorrt.Refitter, weights_name: str) → tensorrt.tensorrt.TensorLocation¶

Get location for the weights associated with the given name.

If the weights were never set, returns TensorLocation.HOST and reports an error to the refitter errorRecorder.

Parameters:: weights_name – The name of the weights to be refitted.
Returns:: Location for the weights associated with the given name.

get_weights_prototype(self: tensorrt.tensorrt.Refitter, weights_name: str) → tensorrt.tensorrt.Weights¶

Get the weights prototype associated with the given name.

The dtype and size of weights prototype is the same as weights used for engine building. The size of the weights prototype is -1 when the name of the weights is None or does not correspond to any refittable weights.

Parameters:: weights_name – The name of the weights to be refitted.
Returns:: weights prototype associated with the given name.

refit_cuda_engine(self: tensorrt.tensorrt.Refitter) → bool¶

Refits associated engine.

If False is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work. Provided weights on CPU or GPU can be unset and released, or updated after refit_cuda_engine returns.

Returns:: True on success, or False if new weights validation fails or get_missing_weights() != 0 before the call.

refit_cuda_engine_async(self: tensorrt.tensorrt.Refitter, stream_handle: int) → bool¶

Enqueue weights refitting of the associated engine on the given stream.

If False is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work on a different stream from the provided one. Provided weights on CPU can be unset and released, or updated after refit_cuda_engine_async returns. Freeing or updating of the provided weights on GPU can be enqueued on the same stream after refit_cuda_engine_async returns.

IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead. The weights updating task should use the the same stream as the one used for the refit call.

Parameters:: stream – The stream to enqueue the weights updating task.
Returns:: True on success, or False if new weights validation fails or get_missing_weights() != 0 before the call.

set_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str, range: List[float]) → bool¶

[DEPRECATED] Deprecated in TensorRT 10.1. Superseded by explicit quantization.

Update dynamic range for a tensor.

Parameters:

tensor_name – The name of the tensor whose dynamic range to update.
range – The new range.

Returns:

True if successful, False otherwise.

Returns false if there is no Int8 engine tensor derived from a network tensor of that name. If successful, then get_missing() may report that some weights need to be supplied.

set_named_weights(*args, **kwargs)¶

Overloaded function.

set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights) -> bool
Specify new weights of given name. Possible reasons for rejection are:
- The name of weights is empty or does not correspond to any refittable weights.
- The size of the weights is inconsistent with the size returned from calling get_weights_prototype() with the same name.
- The dtype of the weights is inconsistent with the dtype returned from calling get_weights_prototype() with the same name.
Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

arg name:

The name of the weights to be refitted.

arg weights:

The new weights to associate with the name.

returns:

True on success, or False if new weights are rejected.
set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights, location: tensorrt.tensorrt.TensorLocation) -> bool
Specify new weights on a specified device of given name. Possible reasons for rejection are:
- The name of weights is empty or does not correspond to any refittable weights.
- The size of the weights is inconsistent with the size returned from calling get_weights_prototype() with the same name.
- The dtype of the weights is inconsistent with the dtype returned from calling get_weights_prototype() with the same name.
It is allowed to provide some weights on CPU and others on GPU. Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

arg name:

The name of the weights to be refitted.

arg weights:

The new weights on the specified device.

arg location:

The location (host vs. device) of the new weights.

returns:

True on success, or False if new weights are rejected.

set_weights(self: tensorrt.tensorrt.Refitter, layer_name: str, role: tensorrt.tensorrt.WeightsRole, weights: tensorrt.tensorrt.Weights) → bool¶

Specify new weights for a layer of given name. Possible reasons for rejection are:

There is no such layer by that name.
The layer does not have weights with the specified role.
The size of weights is inconsistent with the layer’s original specification.

Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

Parameters:

layer_name – The name of the layer.
role – The role of the weights. See WeightsRole for more information.
weights – The weights to refit with.

Returns:

True on success, or False if new weights are rejected.

unset_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) → bool¶

Unset weights associated with the given name.

Unset weights before releasing them.

Parameters:: weights_name – The name of the weights to be refitted.
Returns:: False if the weights were never set, returns True otherwise.