Runtime

tensorrt_rtx.TempfileControlFlag

Flags used to control TensorRT-RTX’s behavior when creating executable temporary files.

On some platforms the TensorRT-RTX runtime may need to create files in a temporary directory or use platform-specific APIs to create files in-memory to load temporary DLLs that implement runtime code. These flags allow the application to explicitly control TensorRT-RTX’s use of these files. This will preclude the use of certain TensorRT-RTX APIs for deserializing and loading lean runtimes.

These should be treated as bit offsets, e.g. in order to allow in-memory files for a given IRuntime:

runtime.tempfile_control_flags |= (1 << int(TempfileControlFlag.ALLOW_IN_MEMORY_FILES))

Members:

ALLOW_IN_MEMORY_FILES : Allow creating and loading files in-memory (or unnamed files).

ALLOW_TEMPORARY_FILES : Allow creating and loading named files in a temporary directory on the filesystem.

class tensorrt_rtx.Runtime(self: tensorrt_rtx.tensorrt_rtx.Runtime, logger: tensorrt_rtx.tensorrt_rtx.ILogger)

Allows a serialized ICudaEngine to be deserialized.

Variables:

  • error_recorder – IErrorRecorder Application-implemented error reporting interface for TensorRT-RTX objects.

  • gpu_allocator – IGpuAllocator The GPU allocator to be used by the Runtime . All GPU memory acquired will use this allocator. If set to None, the default allocator will be used (Default: cudaMalloc/cudaFree).

  • DLA_core – int The DLA core used by the network. Defaults to -1. This property specifies which DLA core to use by index when multiple cores are available. Its current value is the assigned DLA core, or -1 if DLA is unavailable or no core is set. The value must be in the range [0, num_DLA_cores). If no DLA core is set when DLAWorkspaceAllocationStrategy.SHARED_STATIC is selected, core 0 is used automatically. If num_DLA_cores is 0, setting this property does nothing.

  • num_DLA_cores – int The number of DLA hardware cores accessible, or 0 if DLA is unavailable.

  • dla_workspace_allocation_strategy – DLAWorkspaceAllocationStrategy The DLA workspace allocation strategy used for subsequent engine deserializations. Engines already deserialized by this runtime retain the strategy that was selected when they were deserialized. Selecting DLAWorkspaceAllocationStrategy.SHARED_STATIC creates or reuses a shared workspace for the selected DLA core. If no DLA core is selected, TensorRT-RTX selects core 0. Changing DLA_core while shared workspace is selected creates or reuses a separate shared workspace for the newly selected core. Setting this property raises RuntimeError if the workspace allocation strategy cannot be changed successfully.

  • logger – ILogger The logger provided when creating the refitter.

  • max_threads – int The maximum thread that can be used by the Runtime.

  • temporary_directory – str The temporary directory to use when loading executable code for engines. If set to None (the default), TensorRT-RTX will attempt to find a suitable directory for use using platform-specific heuristics: - On UNIX/Linux platforms, TensorRT-RTX will first try the TMPDIR environment variable, then fall back to /tmp - On Windows, TensorRT-RTX will try the TEMP environment variable.

  • tempfile_control_flags – int Flags which control whether TensorRT-RTX is allowed to create in-memory or temporary files. See TempfileControlFlag for details.

  • engine_host_code_allowed – bool Whether this runtime is allowed to deserialize engines that contain host executable code (Default: False).

  • defer_weights_loading – bool When True, the next deserialize_cuda_engine() call will skip GPU weight allocation. Call ICudaEngine.load_weights() or ICudaEngine.load_weights_async() to bring weights to the GPU when ready (Default: False).

Parameters:

logger – The logger to use.

__del__(self: tensorrt_rtx.tensorrt_rtx.Runtime) → None
__exit__(exc_type, exc_value, traceback)

Context managers are deprecated and have no effect. Objects are automatically freed when the reference count reaches 0.

__init__(self: tensorrt_rtx.tensorrt_rtx.Runtime, logger: tensorrt_rtx.tensorrt_rtx.ILogger) → None
Parameters:

logger – The logger to use.

property defer_weights_loading

When set to True, the next call to deserialize_cuda_engine() will defer GPU weight allocation. The engine can then be used to create execution contexts (driving JIT compilation) without weights resident on the GPU. Call ICudaEngine.load_weights() (host blob) or ICudaEngine.load_weights_async() (streamed from an IStreamReaderV2 on a CUDA stream) when ready to bring weights to the GPU and enable inference.

deserialize_cuda_engine(*args, **kwargs)

Overloaded function.

  1. deserialize_cuda_engine(self: tensorrt_rtx.tensorrt_rtx.Runtime, serialized_engine: collections.abc.Buffer) -> tensorrt_rtx.tensorrt_rtx.ICudaEngine

    Deserialize an ICudaEngine from host memory.

    arg serialized_engine:

    The buffer that holds the serialized ICudaEngine.

    returns:

    The ICudaEngine, or None if it could not be deserialized.

  2. deserialize_cuda_engine(self: tensorrt_rtx.tensorrt_rtx.Runtime, stream_reader_v2: tensorrt_rtx.tensorrt_rtx.IStreamReaderV2) -> tensorrt_rtx.tensorrt_rtx.ICudaEngine

    Deserialize an ICudaEngine from a stream reader v2.

    arg stream_reader:

    The PyStreamReaderV2 that will read the serialized ICudaEngine. This enables deserialization from a file directly, with possible benefits to performance.

    returns:

    The ICudaEngine, or None if it could not be deserialized.

get_engine_validity(self: tensorrt_rtx.tensorrt_rtx.Runtime, buffer: collections.abc.Buffer) → tuple

Get engine validity and diagnostics from serialized engine buffer. Returns tuple of (EngineValidity, diagnostics).

get_plugin_registry(self: tensorrt_rtx.tensorrt_rtx.Runtime) → tensorrt_rtx.tensorrt_rtx.IPluginRegistry

Get the local plugin registry that can be used by the runtime.

Returns:

The local plugin registry that can be used by the runtime.

load_runtime(self: tensorrt_rtx.tensorrt_rtx.Runtime, path: str) → tensorrt_rtx.tensorrt_rtx.Runtime

Load IRuntime from the file.

This method loads a runtime library from a shared library file. The runtime can then be used to execute a plan file built with BuilderFlag.VERSION_COMPATIBLE and BuilderFlag.EXCLUDE_LEAN_RUNTIME both set and built with the same version of TensorRT-RTX as the loaded runtime library.

Variables:

path – Path to the runtime lean library.

Returns:

The IRuntime, or None if it could not be loaded.