Appendix: Python Migration Patterns for TensorRT 10.x

Note

The Python API is not supported on QNX.

Migrating I/O Buffer Allocation to Named Tensors

TensorRT 10.x replaces the binding-based API with a name-based tensor API. Use num_io_tensors and get_tensor_name() to iterate over I/O tensors, and get_tensor_mode() to check whether a tensor is an input or output.

Migrating I/O Buffer Allocation to Named Tensors (Before / After / Summary)

Before (TensorRT 8.x)

def allocate_buffers(self, engine):
    '''
    Allocates all buffers required for an engine; that is, host and device inputs and outputs.
    '''
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()

    # binding is the name of input/output
    for binding in engine:
        size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size
        dtype = trt.nptype(engine.get_binding_dtype(binding))

        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't be swapped to disk)
        device_mem = cuda.mem_alloc(host_mem.nbytes)

        # Append the device buffer address to device bindings.
        # When cast to int, it's a linear index into the context's memory (like memory address).
        bindings.append(int(device_mem))

        # Append to the appropriate input/output list.
        if engine.binding_is_input(binding):
            inputs.append(self.HostDeviceMem(host_mem, device_mem))
        else:
            outputs.append(self.HostDeviceMem(host_mem, device_mem))

    return inputs, outputs, bindings, stream

After (TensorRT 10.x)

def allocate_buffers(self, engine):
    '''
    Allocates all buffers required for an engine; that is, host and device inputs and outputs.
    '''
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()

    for i in range(engine.num_io_tensors):
        tensor_name = engine.get_tensor_name(i)
        size = trt.volume(engine.get_tensor_shape(tensor_name))
        dtype = trt.nptype(engine.get_tensor_dtype(tensor_name))

        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't be swapped to disk)
        device_mem = cuda.mem_alloc(host_mem.nbytes)

        # Append the device buffer address to device bindings.
        # When cast to int, it's a linear index into the context's memory (like memory address).
        bindings.append(int(device_mem))

        # Append to the appropriate input/output list.
        if engine.get_tensor_mode(tensor_name) == trt.TensorIOMode.INPUT:
            inputs.append(self.HostDeviceMem(host_mem, device_mem))
        else:
            outputs.append(self.HostDeviceMem(host_mem, device_mem))

    return inputs, outputs, bindings, stream

Summary of Changes

• Changed from binding-based iteration to name-based API using num_io_tensors and get_tensor_name()

• Replaced binding_is_input() with get_tensor_mode() to check I/O mode

Note

The HostDeviceMem helper class used in the examples above is a container that pairs host (CPU) and device (GPU) memory allocations. It is not part of the TensorRT API. A minimal implementation is:

from collections import namedtuple

HostDeviceMem = namedtuple("HostDeviceMem", ["host", "device"])

Migrating from execute_async_v2 to execute_async_v3

The examples below show TensorRT 8.x first, then TensorRT 10.x, for the same inference task. In TensorRT 10.x, execute_async_v3 (enqueueV3 in C++) replaces execute_async_v2 (enqueueV2 in C++): call set_tensor_address for each I/O tensor before execute_async_v3, as shown in the After example below.

Migrating from execute_async_v2 to execute_async_v3 (Before / After / Summary)

Before (TensorRT 8.x)

# Allocate device memory for inputs.
d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)]

# Allocate device memory for outputs.
h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32)
d_output = cuda.mem_alloc(h_output.nbytes)

# Transfer data from host to device.
cuda.memcpy_htod_async(d_inputs[0], input_a, stream)
cuda.memcpy_htod_async(d_inputs[1], input_b, stream)
cuda.memcpy_htod_async(d_inputs[2], input_c, stream)

# Run inference
context.execute_async_v2(bindings=[int(d_inp) for d_inp in d_inputs] + [int(d_output)], stream_handle=stream.handle)

# Synchronize the stream
stream.synchronize()

After (TensorRT 10.x)

# Allocate device memory for inputs.
d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)]

# Allocate device memory for outputs.
h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32)
d_output = cuda.mem_alloc(h_output.nbytes)

# Transfer data from host to device.
cuda.memcpy_htod_async(d_inputs[0], input_a, stream)
cuda.memcpy_htod_async(d_inputs[1], input_b, stream)
cuda.memcpy_htod_async(d_inputs[2], input_c, stream)

# Setup tensor address
bindings = [int(d_inputs[i]) for i in range(3)] + [int(d_output)]

for i in range(engine.num_io_tensors):
    context.set_tensor_address(engine.get_tensor_name(i), bindings[i])

# Run inference
context.execute_async_v3(stream_handle=stream.handle)

# Synchronize the stream
stream.synchronize()

Summary of Changes

• Added explicit tensor address setup using set_tensor_address() with tensor names

• Changed from execute_async_v2() to execute_async_v3()

• The bindings parameter is no longer passed to execute_async_v3(); tensor addresses must be set beforehand

Migrating Engine Builds to build_serialized_network

The examples below show TensorRT 8.x first, then TensorRT 10.x, for the same engine build path. In TensorRT 10.x, build_serialized_network() is the standard build path and is always available. Omit the build_engine() / serialize() fallback from the 8.x sample, and check for a None return instead of catching AttributeError, as shown in the After example below.

Migrating Engine Builds to build_serialized_network (Before / After / Summary)

Before (TensorRT 8.x)

engine_bytes = None
try:
    engine_bytes = self.builder.build_serialized_network(self.network, self.config)
except AttributeError:
    engine = self.builder.build_engine(self.network, self.config)
    engine_bytes = engine.serialize()
    del engine
assert engine_bytes

After (TensorRT 10.x)

engine_bytes = self.builder.build_serialized_network(self.network, self.config)
if engine_bytes is None:
    log.error("Failed to create engine")
    sys.exit(1)

Summary of Changes

• The fallback to build_engine() and serialize() is no longer needed in TensorRT 10.x

• build_serialized_network() is now the standard method and always available

• Error handling should check for None return value instead of catching AttributeError