Loaders
Module: polygraphy.backend.trt
- class LoadPlugins(plugins=None, obj=None)[source]
Bases:
BaseLoader
A passthrough loader that loads plugins from the specified paths. Passthrough here means that it can be used to wrap any other loader. The purpose of wrapping another loader is that you can control the order of execution when lazily evaluating.
For immediate evaluation, use load_plugins instead:
load_plugins(plugins=["/path/to/my/plugin.so", "/path/to/my/other_plugin.so"])
Loads plugins from the specified paths.
- Parameters:
plugins (List[str]) – A list of paths to plugin libraries to load before inference.
obj – An object or callable to return or call respectively. If
obj
is callable, extra parameters will be forwarded toobj
. Ifobj
is not callable, it will be returned.
- call_impl(*args, **kwargs)[source]
- Returns:
The provided
obj
argument, or its return value if it is callable. ReturnsNone
ifobj
was not set.- Return type:
object
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- load_plugins(plugins=None, obj=None, *args, **kwargs)
Immediately evaluated functional variant of
LoadPlugins
.Loads plugins from the specified paths.
- Parameters:
plugins (List[str]) – A list of paths to plugin libraries to load before inference.
obj – An object or callable to return or call respectively. If
obj
is callable, extra parameters will be forwarded toobj
. Ifobj
is not callable, it will be returned.
- Returns:
The provided
obj
argument, or its return value if it is callable. ReturnsNone
ifobj
was not set.- Return type:
object
- class CreateNetwork(explicit_batch=None, strongly_typed=None)[source]
Bases:
BaseLoader
Functor that creates an empty TensorRT network.
Creates an empty TensorRT network.
- Parameters:
explicit_batch (bool) – Whether to create the network with explicit batch mode. Defaults to True.
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- call_impl()[source]
- Returns:
The builder and empty network.
- Return type:
(trt.Builder, trt.INetworkDefinition)
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- create_network(explicit_batch=None, strongly_typed=None)
Immediately evaluated functional variant of
CreateNetwork
.Creates an empty TensorRT network.
- Parameters:
explicit_batch (bool) – Whether to create the network with explicit batch mode. Defaults to True.
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- Returns:
The builder and empty network.
- Return type:
(trt.Builder, trt.INetworkDefinition)
- class NetworkFromOnnxBytes(model_bytes, flags=None, plugin_instancenorm=None, strongly_typed=None)[source]
Bases:
BaseNetworkFromOnnx
Functor that parses an ONNX model to create a trt.INetworkDefinition.
Parses an ONNX model.
- Parameters:
model_bytes (Union[bytes, Callable() -> bytes]) – A serialized ONNX model or a callable that returns one.
flags (List[trt.OnnxParserFlag]) – A list of
OnnxParserFlag
s to modify the default parsing behavior of the ONNX parser. Defaults to None.plugin_instancenorm (bool) – Whether to force usage of the plugin implementation of ONNX InstanceNorm by clearing the NATIVE_INSTANCENORM flag in the parser. Defaults to False
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- call_impl()[source]
- Returns:
A TensorRT network, as well as the builder used to create it, and the parser used to populate it.
- Return type:
(trt.IBuilder, trt.INetworkDefinition, trt.OnnxParser)
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- network_from_onnx_bytes(model_bytes, flags=None, plugin_instancenorm=None, strongly_typed=None)
Immediately evaluated functional variant of
NetworkFromOnnxBytes
.Parses an ONNX model.
- Parameters:
model_bytes (Union[bytes, Callable() -> bytes]) – A serialized ONNX model or a callable that returns one.
flags (List[trt.OnnxParserFlag]) – A list of
OnnxParserFlag
s to modify the default parsing behavior of the ONNX parser. Defaults to None.plugin_instancenorm (bool) – Whether to force usage of the plugin implementation of ONNX InstanceNorm by clearing the NATIVE_INSTANCENORM flag in the parser. Defaults to False
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- Returns:
A TensorRT network, as well as the builder used to create it, and the parser used to populate it.
- Return type:
(trt.IBuilder, trt.INetworkDefinition, trt.OnnxParser)
- class NetworkFromOnnxPath(path, flags=None, plugin_instancenorm=None, strongly_typed=None)[source]
Bases:
BaseNetworkFromOnnx
Functor that parses an ONNX model to create a trt.INetworkDefinition. This loader supports models with weights stored in an external location.
Parses an ONNX model from a file.
- Parameters:
path (str) – The path from which to load the model.
flags (List[trt.OnnxParserFlag]) – A list of
OnnxParserFlag
s to modify the default parsing behavior of the ONNX parser. Defaults to None.plugin_instancenorm (bool) – Whether to force usage of the plugin implementation of ONNX InstanceNorm by clearing the NATIVE_INSTANCENORM flag in the parser. Defaults to False
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- call_impl()[source]
- Returns:
A TensorRT network, as well as the builder used to create it, and the parser used to populate it.
- Return type:
(trt.IBuilder, trt.INetworkDefinition, trt.OnnxParser)
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- network_from_onnx_path(path, flags=None, plugin_instancenorm=None, strongly_typed=None)
Immediately evaluated functional variant of
NetworkFromOnnxPath
.Parses an ONNX model from a file.
- Parameters:
path (str) – The path from which to load the model.
flags (List[trt.OnnxParserFlag]) – A list of
OnnxParserFlag
s to modify the default parsing behavior of the ONNX parser. Defaults to None.plugin_instancenorm (bool) – Whether to force usage of the plugin implementation of ONNX InstanceNorm by clearing the NATIVE_INSTANCENORM flag in the parser. Defaults to False
strongly_typed (bool) – Whether to mark the network as being strongly typed. Defaults to False.
- Returns:
A TensorRT network, as well as the builder used to create it, and the parser used to populate it.
- Return type:
(trt.IBuilder, trt.INetworkDefinition, trt.OnnxParser)
- class PostprocessNetwork(network, func, name=None)[source]
Bases:
BaseLoader
[EXPERIMENTAL] Functor that applies a given post-processing function to a TensorRT
INetworkDefinition
.Applies a given post-processing function to a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
func (Callable[[trt.INetworkDefinition], None]) – A callable which accepts a named network argument. PostprocessNetwork will pass in the parsed network via this argument, which can then be modified by the callable.
name (Optional[str]) – The name of this postprocessing step, used for logging purposes.
- call_impl()[source]
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- postprocess_network(network, func, name=None)
Immediately evaluated functional variant of
PostprocessNetwork
.Applies a given post-processing function to a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
func (Callable[[trt.INetworkDefinition], None]) – A callable which accepts a named network argument. PostprocessNetwork will pass in the parsed network via this argument, which can then be modified by the callable.
name (Optional[str]) – The name of this postprocessing step, used for logging purposes.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- class ModifyNetworkOutputs(network, outputs=None, exclude_outputs=None)[source]
Bases:
PostprocessNetwork
Functor that modifies outputs in a TensorRT
INetworkDefinition
.Modifies outputs in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
outputs (Sequence[str]) – Names of tensors to mark as outputs. If provided, this will override the outputs already marked in the network. If a value of constants.MARK_ALL is used instead of a list, all tensors in the network are marked.
exclude_outputs (Sequence[str]) – Names of tensors to exclude as outputs. This can be useful in conjunction with
outputs=constants.MARK_ALL
to omit outputs.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- call_impl()
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- modify_network_outputs(network, outputs=None, exclude_outputs=None)
Immediately evaluated functional variant of
ModifyNetworkOutputs
.Modifies outputs in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
outputs (Sequence[str]) – Names of tensors to mark as outputs. If provided, this will override the outputs already marked in the network. If a value of constants.MARK_ALL is used instead of a list, all tensors in the network are marked.
exclude_outputs (Sequence[str]) – Names of tensors to exclude as outputs. This can be useful in conjunction with
outputs=constants.MARK_ALL
to omit outputs.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- class SetLayerPrecisions(network, layer_precisions)[source]
Bases:
PostprocessNetwork
Functor that sets layer precisions in a TensorRT
INetworkDefinition
.Sets layer precisions in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
layer_precisions (Dict[str, trt.DataType]) – A mapping of layer names to their desired compute precision.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- call_impl()
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- set_layer_precisions(network, layer_precisions)
Immediately evaluated functional variant of
SetLayerPrecisions
.Sets layer precisions in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
layer_precisions (Dict[str, trt.DataType]) – A mapping of layer names to their desired compute precision.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- class SetTensorDatatypes(network, tensor_datatypes)[source]
Bases:
PostprocessNetwork
Functor that sets tensor datatypes for network I/O tensors in a TensorRT
INetworkDefinition
.Sets network I/O tensor datatypes in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
tensor_datatypes (Dict[str, trt.DataType]) – A mapping of tensor names to their desired data types.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- call_impl()
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- set_tensor_datatypes(network, tensor_datatypes)
Immediately evaluated functional variant of
SetTensorDatatypes
.Sets network I/O tensor datatypes in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
tensor_datatypes (Dict[str, trt.DataType]) – A mapping of tensor names to their desired data types.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- class SetTensorFormats(network, tensor_formats)[source]
Bases:
PostprocessNetwork
Functor that sets network I/O tensor formats in a TensorRT
INetworkDefinition
.Sets network I/O tensor formats in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
tensor_formats (Dict[str, List[trt.TensorFormat]]) – A mapping of tensor names to their allowed formats.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- call_impl()
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- set_tensor_formats(network, tensor_formats)
Immediately evaluated functional variant of
SetTensorFormats
.Sets network I/O tensor formats in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
tensor_formats (Dict[str, List[trt.TensorFormat]]) – A mapping of tensor names to their allowed formats.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- class LoadRuntime(path)[source]
Bases:
BaseLoader
Functor that loads a TensorRT
IRuntime
.Loads a TensorRT
IRuntime
.The loaded runtime can be used to execute a version compatible engine that excludes the lean runtime.
- Parameters:
path (str) – The path to a shared library from which to load the runtime.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- load_runtime(path)
Immediately evaluated functional variant of
LoadRuntime
.Loads a TensorRT
IRuntime
.The loaded runtime can be used to execute a version compatible engine that excludes the lean runtime.
- Parameters:
path (str) – The path to a shared library from which to load the runtime.
- Returns:
The runtime that was loaded.
- Return type:
trt.Runtime
- class EngineBytesFromNetwork(network, config=None, save_timing_cache=None)[source]
Bases:
BaseLoader
Functor that uses a TensorRT
INetworkDefinition
to build a serialized engine.Builds and serializes TensorRT engine.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
config (Callable(trt.Builder, trt.INetworkDefinition) -> trt.IBuilderConfig) – A TensorRT builder configuration or a callable that returns one. If not supplied, a CreateConfig instance with default parameters is used.
save_timing_cache (Union[str, file-like]) – A path or file-like object at which to save a tactic timing cache. Any existing cache will be appended to. If a path is provided, the file will be locked for exclusive access to prevent multiple processes from attempting to update the timing cache at the same time.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- engine_bytes_from_network(network, config=None, save_timing_cache=None)
Immediately evaluated functional variant of
EngineBytesFromNetwork
.Builds and serializes TensorRT engine.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
config (Callable(trt.Builder, trt.INetworkDefinition) -> trt.IBuilderConfig) – A TensorRT builder configuration or a callable that returns one. If not supplied, a CreateConfig instance with default parameters is used.
save_timing_cache (Union[str, file-like]) – A path or file-like object at which to save a tactic timing cache. Any existing cache will be appended to. If a path is provided, the file will be locked for exclusive access to prevent multiple processes from attempting to update the timing cache at the same time.
- Returns:
The serialized engine that was created.
- Return type:
bytes
- class EngineFromNetwork(network, config=None, save_timing_cache=None, runtime=None)[source]
Bases:
EngineBytesFromNetwork
Functor similar to EngineBytesFromNetwork, but deserializes the engine before returning.
Builds a TensorRT serialized engine and then deserializes it.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
config (Callable(trt.Builder, trt.INetworkDefinition) -> trt.IBuilderConfig) – A TensorRT builder configuration or a callable that returns one. If not supplied, a CreateConfig instance with default parameters is used.
save_timing_cache (Union[str, file-like]) – A path or file-like object at which to save a tactic timing cache. Any existing cache will be appended to. If a path is provided, the file will be locked for exclusive access to prevent multiple processes from attempting to update the timing cache at the same time.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- engine_from_network(network, config=None, save_timing_cache=None, runtime=None)
Immediately evaluated functional variant of
EngineFromNetwork
.Builds a TensorRT serialized engine and then deserializes it.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
config (Callable(trt.Builder, trt.INetworkDefinition) -> trt.IBuilderConfig) – A TensorRT builder configuration or a callable that returns one. If not supplied, a CreateConfig instance with default parameters is used.
save_timing_cache (Union[str, file-like]) – A path or file-like object at which to save a tactic timing cache. Any existing cache will be appended to. If a path is provided, the file will be locked for exclusive access to prevent multiple processes from attempting to update the timing cache at the same time.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- Returns:
The engine that was created.
- Return type:
trt.ICudaEngine
- class EngineFromBytes(serialized_engine, runtime=None)[source]
Bases:
BaseLoader
Functor that deserializes an engine from a buffer.
Deserializes an engine from a buffer.
- Parameters:
serialized_engine (Union[Union[str, bytes], Callable() -> Union[str, bytes]]) – The serialized engine bytes or a callable that returns them.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- engine_from_bytes(serialized_engine, runtime=None)
Immediately evaluated functional variant of
EngineFromBytes
.Deserializes an engine from a buffer.
- Parameters:
serialized_engine (Union[Union[str, bytes], Callable() -> Union[str, bytes]]) – The serialized engine bytes or a callable that returns them.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- Returns:
The deserialized engine.
- Return type:
trt.ICudaEngine
- class EngineFromPath(path: str, runtime=None)[source]
Bases:
BaseLoader
Functor that deserializes an engine from a path.
Deserializes an engine from a path.
- Parameters:
path (Union[str, Callable() -> str]) – The file path to the serialized engine or a callable that returns it.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- engine_from_path(path, runtime=None)
Immediately evaluated functional variant of
EngineFromPath
.Deserializes an engine from a path.
- Parameters:
path (Union[str, Callable() -> str]) – The file path to the serialized engine or a callable that returns it.
runtime (Union[trt.Runtime, Callable() -> trt.Runtime]) – The runtime to use when deserializing the engine or a callable that returns one. If no runtime is provided, one will be created.
- Returns:
The deserialized engine.
- Return type:
trt.ICudaEngine
- class BytesFromEngine(engine)[source]
Bases:
BaseLoader
Functor that serializes an engine to bytes.
Returned bytes copy the serialized engine’s memory. Use BufferFromEngine to directly reference the memory.
Serializes an engine to bytes.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- bytes_from_engine(engine)
Immediately evaluated functional variant of
BytesFromEngine
.Serializes an engine to bytes.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
- Returns:
The serialized engine copied to bytes.
- Return type:
bytes
- class BufferFromEngine(engine: ICudaEngine | Callable[[], ICudaEngine])[source]
Bases:
BaseLoader
Functor that serializes an engine to a buffer.
Returned buffer directly references the serialized engine’s memory and does not copy it.
Serializes an engine to a memoryview.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
- call_impl() IHostMemory [source]
- Returns:
The serialized engine as a buffer.
- Return type:
IHostMemory
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- buffer_from_engine(engine)
Immediately evaluated functional variant of
BufferFromEngine
.Serializes an engine to a memoryview.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
- Returns:
The serialized engine as a buffer.
- Return type:
IHostMemory
- class SaveEngine(engine, path)[source]
Bases:
BaseLoader
Functor that saves an engine to the provided path.
Saves an engine to the provided path.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
path (str) – The path at which to save the engine.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- save_engine(engine, path)
Immediately evaluated functional variant of
SaveEngine
.Saves an engine to the provided path.
- Parameters:
engine (Union[trt.ICudaEngine, Callable() -> trt.ICudaEngine]) – An engine or a callable that returns one.
path (str) – The path at which to save the engine.
- Returns:
The engine that was saved.
- Return type:
trt.ICudaEngine
- class OnnxLikeFromNetwork(network)[source]
Bases:
BaseLoader
Functor that creates an ONNX-like, but not valid ONNX, model based on a TensorRT network.
[HIGHLY EXPERIMENTAL] Creates an ONNX-like, but not valid ONNX, model from a TensorRT network. This uses the ONNX format, but generates nodes that are not valid ONNX operators. Hence, this should be used only for visualization or debugging purposes.
The resulting model does not include enough information to faithfully reconstruct the TensorRT network, but does preserve the structure of the network and many of the layer parameters.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
- call_impl()[source]
- Returns:
The ONNX-like, but not valid ONNX, representation of the TensorRT network.
- Return type:
onnx.ModelProto
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- onnx_like_from_network(network)
Immediately evaluated functional variant of
OnnxLikeFromNetwork
.[HIGHLY EXPERIMENTAL] Creates an ONNX-like, but not valid ONNX, model from a TensorRT network. This uses the ONNX format, but generates nodes that are not valid ONNX operators. Hence, this should be used only for visualization or debugging purposes.
The resulting model does not include enough information to faithfully reconstruct the TensorRT network, but does preserve the structure of the network and many of the layer parameters.
- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
- Returns:
The ONNX-like, but not valid ONNX, representation of the TensorRT network.
- Return type:
onnx.ModelProto
- class MarkDebug(network, mark_debug)[source]
Bases:
PostprocessNetwork
Functor that mark tensors as debug tensors in a TensorRT
INetworkDefinition
.Mark tensors as debug tensors in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
mark_debug (List[str]) – List of tensor names to mark as debug tensors.
- __call__(*args, **kwargs)
Invokes the loader by forwarding arguments to
call_impl
.Note:
call_impl
should not be called directly - use this function instead.
- call_impl()
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]
- mark_debug(network, mark_debug)
Immediately evaluated functional variant of
MarkDebug
.Mark tensors as debug tensors in a TensorRT
INetworkDefinition
.- Parameters:
network (Union[Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]], Callable() -> Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]) – A tuple containing a TensorRT builder, network and optionally parser or a callable that returns one. To omit the parser, return a tuple containing just the builder and network.
mark_debug (List[str]) – List of tensor names to mark as debug tensors.
- Returns:
The modified network along with the builder and parser if provided.
- Return type:
Tuple[trt.Builder, trt.INetworkDefinition, Optional[parser]]