Runtime

tensorrt.TempfileControlFlag

Flags used to control TensorRT’s behavior when creating executable temporary files.

On some platforms the TensorRT runtime may need to create files in a temporary directory or use platform-specific APIs to create files in-memory to load temporary DLLs that implement runtime code. These flags allow the application to explicitly control TensorRT’s use of these files. This will preclude the use of certain TensorRT APIs for deserializing and loading lean runtimes.

These should be treated as bit offsets, e.g. in order to allow in-memory files for a given IRuntime:

runtime.tempfile_control_flags |= (1 << int(TempfileControlFlag.ALLOW_IN_MEMORY_FILES))

Members:

ALLOW_IN_MEMORY_FILES : Allow creating and loading files in-memory (or unnamed files).

ALLOW_TEMPORARY_FILES : Allow creating and loading named files in a temporary directory on the filesystem.

class tensorrt.Runtime(self: tensorrt.tensorrt.Runtime, logger: tensorrt.tensorrt.ILogger)

Allows a serialized ICudaEngine to be deserialized.

Variables:

  • error_recorder – IErrorRecorder Application-implemented error reporting interface for TensorRT objects.

  • gpu_allocator – IGpuAllocator The GPU allocator to be used by the Runtime . All GPU memory acquired will use this allocator. If set to None, the default allocator will be used (Default: cudaMalloc/cudaFree).

  • DLA_core – int The DLA core used by the network. Defaults to -1. This property specifies which DLA core to use by index when multiple cores are available. Its current value is the assigned DLA core, or -1 if DLA is unavailable or no core is set. The value must be in the range [0, num_DLA_cores). If no DLA core is set when DLAWorkspaceAllocationStrategy.SHARED_STATIC is selected, core 0 is used automatically. If num_DLA_cores is 0, setting this property does nothing.

  • num_DLA_cores – int The number of DLA hardware cores accessible, or 0 if DLA is unavailable.

  • dla_workspace_allocation_strategy – DLAWorkspaceAllocationStrategy The DLA workspace allocation strategy used for subsequent engine deserializations. Engines already deserialized by this runtime retain the strategy that was selected when they were deserialized. Selecting DLAWorkspaceAllocationStrategy.SHARED_STATIC creates or reuses a shared workspace for the selected DLA core. If no DLA core is selected, TensorRT selects core 0. Changing DLA_core while shared workspace is selected creates or reuses a separate shared workspace for the newly selected core. Setting this property raises RuntimeError if the workspace allocation strategy cannot be changed successfully.

  • logger – ILogger The logger provided when creating the refitter.

  • max_threads – int The maximum thread that can be used by the Runtime.

  • temporary_directory – str The temporary directory to use when loading executable code for engines. If set to None (the default), TensorRT will attempt to find a suitable directory for use using platform-specific heuristics: - On UNIX/Linux platforms, TensorRT will first try the TMPDIR environment variable, then fall back to /tmp - On Windows, TensorRT will try the TEMP environment variable.

  • tempfile_control_flags – int Flags which control whether TensorRT is allowed to create in-memory or temporary files. See TempfileControlFlag for details.

  • engine_host_code_allowed – bool Whether this runtime is allowed to deserialize engines that contain host executable code (Default: False).

  • defer_weights_loading – bool When True, the next deserialize_cuda_engine() call will skip GPU weight allocation. Call ICudaEngine.load_weights() or ICudaEngine.load_weights_async() to bring weights to the GPU when ready (Default: False).

Parameters:

logger – The logger to use.

__del__(self: tensorrt.tensorrt.Runtime) → None
__exit__(exc_type, exc_value, traceback)

Context managers are deprecated and have no effect. Objects are automatically freed when the reference count reaches 0.

__init__(self: tensorrt.tensorrt.Runtime, logger: tensorrt.tensorrt.ILogger) → None
Parameters:

logger – The logger to use.

deserialize_cuda_engine(*args, **kwargs)

Overloaded function.

  1. deserialize_cuda_engine(self: tensorrt.tensorrt.Runtime, serialized_engine: collections.abc.Buffer) -> tensorrt.tensorrt.ICudaEngine

    Deserialize an ICudaEngine from host memory.

    arg serialized_engine:

    The buffer that holds the serialized ICudaEngine.

    returns:

    The ICudaEngine, or None if it could not be deserialized.

  2. deserialize_cuda_engine(self: tensorrt.tensorrt.Runtime, stream_reader_v2: tensorrt.tensorrt.IStreamReaderV2) -> tensorrt.tensorrt.ICudaEngine

    Deserialize an ICudaEngine from a stream reader v2.

    arg stream_reader:

    The PyStreamReaderV2 that will read the serialized ICudaEngine. This enables deserialization from a file directly, with possible benefits to performance.

    returns:

    The ICudaEngine, or None if it could not be deserialized.

get_plugin_registry(self: tensorrt.tensorrt.Runtime) → tensorrt.tensorrt.IPluginRegistry

Get the local plugin registry that can be used by the runtime.

Returns:

The local plugin registry that can be used by the runtime.

load_runtime(self: tensorrt.tensorrt.Runtime, path: str) → tensorrt.tensorrt.Runtime

Load IRuntime from the file.

This method loads a runtime library from a shared library file. The runtime can then be used to execute a plan file built with BuilderFlag.VERSION_COMPATIBLE and BuilderFlag.EXCLUDE_LEAN_RUNTIME both set and built with the same version of TensorRT as the loaded runtime library.

Variables:

path – Path to the runtime lean library.

Returns:

The IRuntime, or None if it could not be loaded.