Migration Guide :: NVIDIA Deep Learning TensorRT Documentation

Abstract

This document highlights the TensorRT API modifications. If you are unfamiliar with these changes, refer to our sample code for clarification.

For previously released TensorRT documentation, refer to the TensorRT Archives.

1. Python

1.1. Python API Changes

Table 1. Allocating Buffers and Using a Name-Based Engine API
TensorRT 8.x	TensorRT 10.0
def allocate_buffers(self, engine): ''' Allocates all buffers required for an engine, i.e. host/device inputs/outputs. ''' inputs = [] outputs = [] bindings = [] stream = cuda.Stream() # binding is the name of input/output for binding in engine: size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size dtype = trt.nptype(engine.get_binding_dtype(binding)) # Allocate host and device buffers host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't swapped to disk) device_mem = cuda.mem_alloc(host_mem.nbytes) # Append the device buffer address to device bindings. # When cast to int, it's a linear index into the context's memory (like memory address). bindings.append(int(device_mem)) # Append to the appropriate input/output list. if engine.binding_is_input(binding): inputs.append(self.HostDeviceMem(host_mem, device_mem)) else: outputs.append(self.HostDeviceMem(host_mem, device_mem)) return inputs, outputs, bindings, stream	def allocate_buffers(self, engine): ''' Allocates all buffers required for an engine, i.e. host/device inputs/outputs. ''' inputs = [] outputs = [] bindings = [] stream = cuda.Stream() for i in range(engine.num_io_tensors): tensor_name = engine.get_tensor_name(i) size = trt.volume(engine.get_tensor_shape(tensor_name)) dtype = trt.nptype(engine.get_tensor_dtype(tensor_name)) # Allocate host and device buffers host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't swapped to disk) device_mem = cuda.mem_alloc(host_mem.nbytes) # Append the device buffer address to device bindings. # When cast to int, it's a linear index into the context's memory (like memory address). bindings.append(int(device_mem)) # Append to the appropriate input/output list. if engine.get_tensor_mode(tensor_name) == trt.TensorIOMode.INPUT: inputs.append(self.HostDeviceMem(host_mem, device_mem)) else: outputs.append(self.HostDeviceMem(host_mem, device_mem)) return inputs, outputs, bindings, stream

Table 1. Allocating Buffers and Using a Name-Based Engine API

TensorRT 8.x

TensorRT 10.0

def allocate_buffers(self, engine):
    '''
    Allocates all buffers required for an engine, i.e. host/device inputs/outputs.
    '''
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()


    # binding is the name of input/output
    for binding in engine: 
        size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size
        dtype = trt.nptype(engine.get_binding_dtype(binding))


        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't swapped to disk)
        device_mem = cuda.mem_alloc(host_mem.nbytes)

        # Append the device buffer address to device bindings. 
        # When cast to int, it's a linear index into the context's memory (like memory address).
        bindings.append(int(device_mem))

        # Append to the appropriate input/output list.
        if engine.binding_is_input(binding):
            inputs.append(self.HostDeviceMem(host_mem, device_mem))
        else:
            outputs.append(self.HostDeviceMem(host_mem, device_mem))

    return inputs, outputs, bindings, stream

def allocate_buffers(self, engine):
    '''
    Allocates all buffers required for an engine, i.e. host/device inputs/outputs.
    '''
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()

    for i in range(engine.num_io_tensors):
        tensor_name = engine.get_tensor_name(i)
        size = trt.volume(engine.get_tensor_shape(tensor_name))
        dtype = trt.nptype(engine.get_tensor_dtype(tensor_name))

        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype) # page-locked memory buffer (won't swapped to disk)
        device_mem = cuda.mem_alloc(host_mem.nbytes)

        # Append the device buffer address to device bindings. 
        # When cast to int, it's a linear index into the context's memory (like memory address). 
        bindings.append(int(device_mem))

       # Append to the appropriate input/output list.
        if engine.get_tensor_mode(tensor_name) == trt.TensorIOMode.INPUT:
            inputs.append(self.HostDeviceMem(host_mem, device_mem))
        else:
            outputs.append(self.HostDeviceMem(host_mem, device_mem))

    return inputs, outputs, bindings, stream

Table 2. Transition from `enqueueV2` to `enqueueV3` for Python
TensorRT 8.x	TensorRT 10.0
# Allocate device memory for inputs. d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)] # Allocate device memory for outputs. h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32) d_output = cuda.mem_alloc(h_output.nbytes) # Transfer data from host to device. cuda.memcpy_htod_async(d_inputs[0], input_a, stream) cuda.memcpy_htod_async(d_inputs[1], input_b, stream) cuda.memcpy_htod_async(d_inputs[2], input_c, stream) # Run inference context.execute_async_v2(bindings=[int(d_inp) for d_inp in d_inputs] + [int(d_output)], stream_handle=stream.handle) # Synchronize the stream stream.synchronize()	# Allocate device memory for inputs. d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)] # Allocate device memory for outputs. h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32) d_output = cuda.mem_alloc(h_output.nbytes) # Transfer data from host to device. cuda.memcpy_htod_async(d_inputs[0], input_a, stream) cuda.memcpy_htod_async(d_inputs[1], input_b, stream) cuda.memcpy_htod_async(d_inputs[2], input_c, stream) # Setup tensor address bindings = [int(d_inputs[i]) for i in range(3)] + [int(d_output)] for i in range(engine.num_io_tensors): context.set_tensor_address(engine.get_tensor_name(i), bindings[i]) # Run inference context.execute_async_v3(stream_handle=stream.handle) # Synchronize the stream stream.synchronize()

Table 2. Transition from enqueueV2 to enqueueV3 for Python

TensorRT 8.x

TensorRT 10.0

# Allocate device memory for inputs.
d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)]

# Allocate device memory for outputs.
h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32)
d_output = cuda.mem_alloc(h_output.nbytes)

# Transfer data from host to device.
cuda.memcpy_htod_async(d_inputs[0], input_a, stream)
cuda.memcpy_htod_async(d_inputs[1], input_b, stream)
cuda.memcpy_htod_async(d_inputs[2], input_c, stream)

# Run inference
context.execute_async_v2(bindings=[int(d_inp) for d_inp in d_inputs] + [int(d_output)], stream_handle=stream.handle)

# Synchronize the stream
stream.synchronize()

# Allocate device memory for inputs.
d_inputs = [cuda.mem_alloc(input_nbytes) for binding in range(input_num)]

# Allocate device memory for outputs.
h_output = cuda.pagelocked_empty(output_nbytes, dtype=np.float32)
d_output = cuda.mem_alloc(h_output.nbytes)

# Transfer data from host to device.
cuda.memcpy_htod_async(d_inputs[0], input_a, stream)
cuda.memcpy_htod_async(d_inputs[1], input_b, stream)
cuda.memcpy_htod_async(d_inputs[2], input_c, stream)

# Setup tensor address
bindings = [int(d_inputs[i]) for i in range(3)] + [int(d_output)]

for i in range(engine.num_io_tensors):
    context.set_tensor_address(engine.get_tensor_name(i), bindings[i])

# Run inference
context.execute_async_v3(stream_handle=stream.handle)

# Synchronize the stream
stream.synchronize()

Table 3. Engine Building, use only `build_serialized_network`
TensorRT 8.x	TensorRT 10.0
engine_bytes = None try: engine_bytes = self.builder.build_serialized_network(self.network, self.config) except AttributeError: engine = self.builder.build_engine(self.network, self.config) engine_bytes = engine.serialize() del engine assert engine_bytes	engine_bytes = self.builder.build_serialized_network(self.network, self.config) if engine_bytes is None: log.error("Failed to create engine") sys.exit(1)

Table 3. Engine Building, use only build_serialized_network

TensorRT 8.x

TensorRT 10.0

engine_bytes = None
try:
    engine_bytes = self.builder.build_serialized_network(self.network, self.config)
except AttributeError:
    engine = self.builder.build_engine(self.network, self.config)
    engine_bytes = engine.serialize()
    del engine
assert engine_bytes

engine_bytes = self.builder.build_serialized_network(self.network, self.config)
if engine_bytes is None:
    log.error("Failed to create engine")
    sys.exit(1)

Types

APILanguage
ExecutionContextAllocationStrategy
IGpuAsyncAllocator
InterfaceInfo
IPluginResource
IPluginV3
IStreamReader
IVersionedInterface

Methods and Properties

ICudaEngine.is_debug_tensor()
ICudaEngine.minimum_weight_streaming_budget
ICudaEngine.streamable_weights_size
ICudaEngine.weight_streaming_budget
IExecutionContext.get_debug_listener()
IExecutionContext.get_debug_state()
IExecutionContext.set_all_tensors_debug_state()
IExecutionContext.set_debug_listener()
IExecutionContext.set_tensor_debug_state()
IExecutionContext.update_device_memory_size_for_shapes()
IGpuAllocator.allocate_async()
IGpuAllocator.deallocate_async()
INetworkDefinition.add_plugin_v3()
INetworkDefinition.is_debug_tensor()
INetworkDefinition.mark_debug()
INetworkDefinition.unmark_debug()
IPluginRegistry.acquire_plugin_resource()
IPluginRegistry.all_creators
IPluginRegistry.deregister_creator()
IPluginRegistry.get_creator()
IPluginRegistry.register_creator()
IPluginRegistry.release_plugin_resource()

1.3. Removed Python APIs

Table 4. Removed Python APIs and their Suggested Superseded API
Python API	Superseded API
`BuilderFlag.ENABLE_TACTIC_HEURISTIC`	Builder optimization level 2
`BuilderFlag.STRICT_TYPES`	Use all three flags: `BuilderFlag.DIRECT_IO` `BuilderFlag.PREFER_PRECISION_CONSTRAINTS` `BuilderFlag.REJECT_EMPTY_ALGORITHMS`
`EngineCapability.DEFAULT` `EngineCapability.kSAFE_DLA` `EngineCapability.SAFE_GPU`	`EngineCapability.STANDARD` `EngineCapability.DLA_STANDALONE` `EngineCapability.SAFETY`
`IAlgorithmIOInfo.tensor_format`	The strides, data type, and vectorization information is sufficient to uniquely identify tensor formats.
`IBuilder.max_batch_size`	Implicit batch is no longer supported.
`IBuilderConfig.max_workspace_size`	`IBuilderConfig.set_memory_pool_limit()` with `MemoryPoolType.WORKSPACE` `IBuilderConfig.get_memory_pool_limit()` with `MemoryPoolType.WORKSPACE`
`IBuilderConfig.min_timing_iterations`	`IBuilderConfig.avg_timing_iterations`
`ICudaEngine.binding_is_input()` `ICudaEngine.get_binding_bytes_per_component()` `ICudaEngine.get_binding_components_per_element()` `ICudaEngine.get_binding_dtype()` `ICudaEngine.get_binding_format()` `ICudaEngine.get_binding_format_desc()` `ICudaEngine.get_binding_index()` `ICudaEngine.get_binding_name()` `ICudaEngine.get_binding_shape()` `ICudaEngine.get_binding_vectorized_dim()` `ICudaEngine.get_location()` `ICudaEngine.get_profile_shape()` `ICudaEngine.get_profile_shape_input()` `ICudaEngine.has_implicit_batch_dimension()` `ICudaEngine.is_execution_binding()` `ICudaEngine.is_shape_binding()` `ICudaEngine.max_batch_size()` `ICudaEngine.num_bindings()`	`ICudaEngine.get_tensor_mode()` `ICudaEngine.get_tensor_bytes_per_component()` `ICudaEngine.get_tensor_components_per_element()` `ICudaEngine.get_tensor_dtype()` `ICudaEngine.get_tensor_format()` `ICudaEngine.get_tensor_format_desc()` No name-based equivalent replacement No name-based equivalent replacement `ICudaEngine.get_tensor_shape()` `ICudaEngine.get_tensor_vectorized_dim()` `ITensor.location` `ICudaEngine.get_tensor_profile_shape()` `ICudaEngine.get_tensor_profile_values()` Implicit batch is no longer supported No name-based equivalent replacement `ICudaEngine.is_shape_inference_io()` Implicit batch is no longer supported `ICudaEngine.num_io_tensors()`
`IExecutionContext.get_binding_shape()` `IExecutionContext.get_strides()` `IExecutionContext.set_binding_shape()`	`IExecutionContext.get_tensor_shape()` `IExecutionContext.get_tensor_strides()` `IExecutionContext.set_input_shape()`
`IFullyConnectedLayer`	`IMatrixMultiplyLayer`
`INetworkDefinition.add_convolution()` `INetworkDefinition.add_deconvolution()` `INetworkDefinition.add_fully_connected()` `INetworkDefinition.add_padding()` `INetworkDefinition.add_pooling()` `INetworkDefinition.add_rnn_v2()` `INetworkDefinition.has_explicit_precision` `INetworkDefinition.has_implicit_batch_dimension`	`INetworkDefinition.add_convolution_nd()` `INetworkDefinition.add_deconvolution_nd()` `INetworkDefinition.add_matrix_multiply()` `INetworkDefinition.add_padding_nd()` `INetworkDefinition.add_pooling_nd()` `INetworkDefinition.add_loop()` Explicit precision support is removed in 10.0 Implicit batch is no longer supported
`IRNNv2Layer`	`ILoop`
`NetworkDefinitionCreationFlag.EXPLICIT_BATCH` `NetworkDefinitionCreationFlag.EXPLICIT_PRECISION`	Support is removed in 10.0
`PaddingMode.CAFFE_ROUND_DOWN` `PaddingMode.CAFFE_ROUND_UP`	Caffe is not supported since 9.0
`PreviewFeature.DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805` `PreviewFeature.FASTER_DYNAMIC_SHAPES_0805`	External tactics are always disabled for core code This flag is on by default
`ProfilingVerbosity.DEFAULT` `ProfilingVerbosity.VERBOSE`	`ProfilingVerbosity.LAYER_NAMES_ONLY` `ProfilingVerbosity.DETAILED`
`ResizeMode`	Use `InterpolationMode`, alias is removed
`SampleMode.DEFAULT`	`SampleMode.STRICT_BOUNDS`
`SliceMode`	Use `SampleMode`, alias is removed

2. C++

2.1. C++ API Changes

Table 5. Transition from `enqueueV2` to `enqueueV3` for C++
TensorRT 8.x	TensorRT 10.0
// Create RAII buffer manager object. samplesCommon::BufferManager buffers(mEngine); auto context = SampleUniquePtr<nvinfer1::IExecutionContext>(mEngine->createExecutionContext()); if (!context) { return false; } // Pick a random digit to try to infer. srand(time(NULL)); int32_t const digit = rand() % 10; // Read the input data into the managed buffers. // There should be just 1 input tensor. ASSERT(mParams.inputTensorNames.size() == 1); if (!processInput(buffers, mParams.inputTensorNames[0], digit)) { return false; } // Create CUDA stream for the execution of this inference. cudaStream_t stream; CHECK(cudaStreamCreate(&stream)); // Asynchronously copy data from host input buffers to device input buffers buffers.copyInputToDeviceAsync(stream); // Asynchronously enqueue the inference work if (!context->enqueueV2(buffers.getDeviceBindings().data(), stream, nullptr)) { return false; } // Asynchronously copy data from device output buffers to host output buffers. buffers.copyOutputToHostAsync(stream); // Wait for the work in the stream to complete. CHECK(cudaStreamSynchronize(stream)); // Release stream. CHECK(cudaStreamDestroy(stream));	// Create RAII buffer manager object. samplesCommon::BufferManager buffers(mEngine); auto context = SampleUniquePtr<nvinfer1::IExecutionContext>(mEngine->createExecutionContext()); if (!context) { return false; } for (int32_t i = 0, e = mEngine->getNbIOTensors(); i < e; i++) { auto const name = mEngine->getIOTensorName(i); context->setTensorAddress(name, buffers.getDeviceBuffer(name)); } // Pick a random digit to try to infer. srand(time(NULL)); int32_t const digit = rand() % 10; // Read the input data into the managed buffers. // There should be just 1 input tensor. ASSERT(mParams.inputTensorNames.size() == 1); if (!processInput(buffers, mParams.inputTensorNames[0], digit)) { return false; } // Create CUDA stream for the execution of this inference. cudaStream_t stream; CHECK(cudaStreamCreate(&stream)); // Asynchronously copy data from host input buffers to device input buffers buffers.copyInputToDeviceAsync(stream); // Asynchronously enqueue the inference workif (!context->enqueueV3(stream)) { return false; } // Asynchronously copy data from device output buffers to host output buffers. buffers.copyOutputToHostAsync(stream); // Wait for the work in the stream to complete. CHECK(cudaStreamSynchronize(stream)); // Release stream. CHECK(cudaStreamDestroy(stream));

Table 5. Transition from enqueueV2 to enqueueV3 for C++

TensorRT 8.x

TensorRT 10.0

// Create RAII buffer manager object.
samplesCommon::BufferManager buffers(mEngine);

auto context = SampleUniquePtr<nvinfer1::IExecutionContext>(mEngine->createExecutionContext());
if (!context)
{
    return false;
}








// Pick a random digit to try to infer.
srand(time(NULL));
int32_t const digit = rand() % 10;

// Read the input data into the managed buffers.
// There should be just 1 input tensor.
ASSERT(mParams.inputTensorNames.size() == 1);

if (!processInput(buffers, mParams.inputTensorNames[0], digit))
{
    return false;
}
// Create CUDA stream for the execution of this inference.
cudaStream_t stream;
CHECK(cudaStreamCreate(&stream));

// Asynchronously copy data from host input buffers to device input buffers
buffers.copyInputToDeviceAsync(stream);

// Asynchronously enqueue the inference work

if (!context->enqueueV2(buffers.getDeviceBindings().data(), stream, nullptr))
{
    return false;
}
// Asynchronously copy data from device output buffers to host output buffers.
buffers.copyOutputToHostAsync(stream);

// Wait for the work in the stream to complete.
CHECK(cudaStreamSynchronize(stream));

// Release stream.
CHECK(cudaStreamDestroy(stream));

// Create RAII buffer manager object.
samplesCommon::BufferManager buffers(mEngine);

auto context = SampleUniquePtr<nvinfer1::IExecutionContext>(mEngine->createExecutionContext());
if (!context)
{
    return false;
}

for (int32_t i = 0, e = mEngine->getNbIOTensors(); i < e; i++)
{
    auto const name = mEngine->getIOTensorName(i);
    context->setTensorAddress(name, buffers.getDeviceBuffer(name));
}


// Pick a random digit to try to infer.
srand(time(NULL));
int32_t const digit = rand() % 10;

// Read the input data into the managed buffers.
// There should be just 1 input tensor.
ASSERT(mParams.inputTensorNames.size() == 1);

if (!processInput(buffers, mParams.inputTensorNames[0], digit))
{
    return false;
}
// Create CUDA stream for the execution of this inference.
cudaStream_t stream;
CHECK(cudaStreamCreate(&stream));

// Asynchronously copy data from host input buffers to device input buffers
buffers.copyInputToDeviceAsync(stream);

// Asynchronously enqueue the inference workif (!context->enqueueV3(stream))
{
    return false;
}
// Asynchronously copy data from device output buffers to host output buffers.
buffers.copyOutputToHostAsync(stream);

// Wait for the work in the stream to complete.
CHECK(cudaStreamSynchronize(stream));

// Release stream.
CHECK(cudaStreamDestroy(stream));

The dimensions held by Dims changed from int32_t to int64_t. However, in TensorRT 10.0, TensorRT will generally reject networks that use dimensions exceeding the range of int32_t. The tensor type returned by IShapeLayer is now DataType::kINT64. Use ICastLayer to cast the result to the tensor of type DataType::kINT32 if 32-bit dimensions are required.

Inspect code that bitwise copies to and from Dims to ensure it is correct for int64_t dimensions.

2.3. Added C++ APIs

Enums

ActivationType::kGELU_ERF
ActivationType::kGELU_TANH
BuilderFlag::kREFIT_IDENTICAL
BuilderFlag::kSTRIP_PLAN
BuilderFlag::kWEIGHT_STREAMING
Datatype::kINT4
LayerType::kPLUGIN_V3

Types

APILanguage
Dims64
ExecutionContextAllocationStrategy
IGpuAsyncAllocator
InterfaceInfo
IPluginResource
IPluginV3
IStreamReader
IVersionedInterface

Methods and Properties

getInferLibBuildVersion
getInferLibMajorVersion
getInferLibMinorVersion
getInferLibPatchVersion
ICudaEngine::createRefitter
IcudaEngine::getMinimumWeightStreamingBudget
IcudaEngine::getStreamableWeightsSize
ICudaEngine::getWeightStreamingBudget
IcudaEngine::isDebugTensor
ICudaEngine::setWeightStreamingBudget
IExecutionContext::getDebugListener
IExecutionContext::getTensorDebugState
IExecutionContext::setAllTensorsDebugState
IExecutionContext::setDebugListener
IExecutionContext::setOuputTensorAddress
IExecutionContext::setTensorDebugState
IExecutionContext::updateDeviceMemorySizeForShapes
IGpuAllocator::allocateAsync
IGpuAllocator::deallocateAsync
INetworkDefinition::addPluginV3
INetworkDefinition::isDebugTensor
INetworkDefinition::markDebug
INetworkDefinition::unmarkDebug
IPluginRegistry::acquirePluginResource
IPluginRegistry::deregisterCreator
IPluginRegistry::getAllCreators
IPluginRegistry::getCreator
IPluginRegistry::registerCreator
IPluginRegistry::releasePluginResource

2.4. Removed C++ APIs

Table 6. Removed C++ APIs and their Suggested Superseded API
C++ API	Superseded API
`BuilderFlag::kENABLE_TACTIC_HEURISTIC`	Builder optimization level 2
`BuilderFlag::kSTRICT_TYPES`	Use for all three flags: `kREJECT_EMPTY_ALGORITHMS` `kDIRECT_IO` `kPREFER_PRECISION_CONSTRAINTS` Note: When removing enum members (for all enums in this list) we will be changing enumeration in the enum to have sequential numbers.
`EngineCapability::kDEFAULT` `EngineCapability::kSAFE_DLA` `EngineCapability::kSAFE_GPU`	`EngineCapability::kSTANDARD` `EngineCapability::kDLA_STANDALONE` `EngineCapability::kSAFETY`
`IAlgorithm::getAlgorithmIOInfo()`	`IAlgorithm::getAlgorithmIOInfoByIndex()`
`IAlgorithmIOInfo::getTensorFormat()`	The strides, data type, and vectorization information is sufficient to uniquely identify tensor formats.
`IBuilder::buildEngineWithConfig()` `IBuilder::destroy()` `IBuilder::getMaxBatchSize()` `IBuilder::setMaxBatchSize()`	`IBuilder::buildSerializedNetwork()` `delete ObjectName` Implicit batch is no longer supported Implicit batch is no longer supported
`IBuilderConfig::destroy()` `IBuilderConfig::getMaxWorkspaceSize()` `IBuilderConfig::getMinTimingIterations()` `IBuilderConfig::setMaxWorkspaceSize()` `IBuilderConfig::setMinTimingIterations()`	`delete ObjectName` `IBuilderConfig::getMemoryPoolLimit()` with `MemoryPoolType::kWORKSPACE` `IBuilderConfig::getAvgTimingIterations()` `IBuilderConfig::setMemoryPoolLimit()` with `MemoryPoolType::kWORKSPACE` `IBuilderConfig::setAvgTimingIterations()`
`IConvolutionLayer::getDilation()` `IConvolutionLayer::getKernelSize()` `IConvolutionLayer::getPadding()` `IConvolutionLayer::getStride()` `IConvolutionLayer::setDilation()` `IConvolutionLayer::setKernelSize()` `IConvolutionLayer::setPadding()` `IConvolutionLayer::setStride()`	`IConvolutionLayer::getDilationNd()` `IConvolutionLayer::getKernelSizeNd()` `IConvolutionLayer::getPaddingNd()` `IConvolutionLayer::getStrideNd()` `IConvolutionLayer::setDilationNd()` `IConvolutionLayer::setKernelSizeNd()` `IConvolutionLayer::setPaddingNd()` `IConvolutionLayer::setStrideNd()`
`ICudaEngine::bindingIsInput()` `ICudaEngine::destroy()` `ICudaEngine::getBindingBytesPerComponent()` `ICudaEngine::getBindingComponentsPerElement()` `ICudaEngine::getBindingDataType()` `ICudaEngine::getBindingDimensions()` `ICudaEngine::getBindingFormat()` `ICudaEngine::getBindingFormatDesc()` `ICudaEngine::getBindingIndex()` `ICudaEngine::getBindingName()` `ICudaEngine::getBindingVectorizedDim()` `ICudaEngine::getLocation()` `ICudaEngine::getMaxBatchSize()` `ICudaEngine::getNbBindings()` `ICudaEngine::getProfileDimensions()` `ICudaEngine::getProfileShapeValues()` `ICudaEngine::hasImplicitBatchDimension()` `ICudaEngine::isExecutionBinding()` `ICudaEngine::isShapeBinding()`	`ICudaEngine::getTensorIOMode()` `delete ObjectName` `ICudaEngine::getTensorBytesPerComponent()` `ICudaEngine::getTensorComponentsPerElement()` `ICudaEngine::getTensorDataType()` `ICudaEngine::getTensorShape()` `ICudaEngine::getTensorFormat()` `ICudaEngine::getTensorFormatDesc()` Name-based methods Name-based methods `ICudaEngine::getTensorVectorizedDim()` `ITensor::getLocation()` Implicit batch is no longer supported `ICudaEngine::getNbIOTensors()` `ICudaEngine::getProfileShape()` `ICudaEngine::getShapeValues()` Implicit batch is no longer supported No name-based equivalent replacement `ICudaEngine::isShapeInferenceIO()`
`IDeconvolutionLayer::getKernelSize()` `IDeconvolutionLayer::getPadding()` `IDeconvolutionLayer::getStride()` `IDeconvolutionLayer::setKernelSize()` `IDeconvolutionLayer::setPadding()` `IDeconvolutionLayer::setStride()`	`IDeconvolutionLayer::getKernelSizeNd()` `IDeconvolutionLayer::getPaddingNd()` `IDeconvolutionLayer::getStrideNd()` `IDeconvolutionLayer::setKernelSizeNd()` `IDeconvolutionLayer::setPaddingNd()` `IDeconvolutionLayer::setStrideNd()`
`IExecutionContext::destroy()` `IExecutionContext::enqueue()` `IExecutionContext::enqueueV2()` `IExecutionContext::execute()` `IExecutionContext::getBindingDimensions()` `IExecutionContext::getShapeBinding()` `IExecutionContext::getStrides()` `IExecutionContext::setBindingDimensions()` `IExecutionContext::setInputShapeBinding()` `IExecutionContext::setOptimizationProfile()`	`delete ObjectName` `IExecutionContext::enqueueV3()` `IExecutionContext::enqueueV3()` `IExecutionContext::executeV2()` `IExecutionContext::getTensorShape()` `IExecutionContext::getTensorAddress()` or `getOutputTensorAddress()` `IExecutionContext::getTensorStrides()` `IExecutionContext::setInputShape()` `IExecutionContext::setInputTensorAddress()` or `setTensorAddress()` `IExecutionContext::setOptimizationProfileAsync()`
`IFullyConnectedLayer`	`IMatrixMultiplyLayer`
`IGpuAllocator::free()`	`IGpuAllocator::deallocate()`
`IHostMemory::destroy()`	`delete ObjectName`
`INetworkDefinition::addConvolution()` `INetworkDefinition::addDeconvolution()` `INetworkDefinition::addFullyConnected()` `INetworkDefinition::addPadding()` `INetworkDefinition::addPooling()` `INetworkDefinition::addRNNv2()` `INetworkDefinition::destroy()` `INetworkDefinition::hasExplicitPrecision()` `INetworkDefinition::hasImplicitBatchDimension()`	`INetworkDefinition::addConvolutionNd()` `INetworkDefinition::addDeconvolutionNd()` `INetworkDefinition::addMatrixMultiply()` `INetworkDefinition::addPaddingNd()` `INetworkDefinition::addPoolingNd()` `INetworkDefinition::addLoop()` `delete ObjectName` Explicit precision support is removed in 10.0 Implicit batch support is removed
`IOnnxConfig::destroy()`	`delete ObjectName`
`IPaddingLayer::getPostPadding()` `IPaddingLayer::getPrePadding()` `IPaddingLayer::setPostPadding()` `IPaddingLayer::setPrePadding()`	`IPaddingLayer::getPostPaddingNd()` `IPaddingLayer::getPrePaddingNd()` `IPaddingLayer::setPostPaddingNd()` `IPaddingLayer::setPrePaddingNd()`
`IPoolingLayer::getPadding()` `IPoolingLayer::getStride()` `IPoolingLayer::getWindowSize()` `IPoolingLayer::setPadding()` `IPoolingLayer::setStride()` `IPoolingLayer::setWindowSize()`	`IPoolingLayer::getPaddingNd()` `IPoolingLayer::getStrideNd()` `IPoolingLayer::getWindowSizeNd()` `IPoolingLayer::setPaddingNd()` `IPoolingLayer::setStrideNd()` `IPoolingLayer::setWindowSizeNd()`
`IRefitter::destroy()`	`delete ObjectName`
`IResizeLayer::getAlignCorners()` `IResizeLayer::setAlignCorners()`	`IResizeLayer::getAlignCornersNd()` `IResizeLayer::setAlignCornersNd()`
IRuntime::deserializeCudaEngine(void const* blob, std::size_t size, IPluginFactory* pluginFactory) `IRuntime::destroy()`	Use `deserializeCudaEngine` with two parameters `delete ObjectName`
`IRNNv2Layer`	`ILoop`
`kNV_TENSORRT_VERSION_IMPL`	#define NV_TENSORRT_VERSION_INT(major, minor, patch) ((major) 10000L + (minor) 100L + (patch) *1L) Note: TensorRT version encoding was changed to accommodate a two digit minor version.
`NetworkDefinitionCreationFlag::kEXPLICIT_BATCH` `NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION`	Support is removed in 10.0
`NV_TENSORRT_SONAME_MAJOR` `NV_TENSORRT_SONAME_MINOR` `NV_TENSORRT_SONAME_PATCH`	`NV_TENSORRT_MAJOR` `NV_TENSORRT_MINOR` `NV_TENSORRT_PATCH`
`PaddingMode::kCAFFE_ROUND_DOWN` `PaddingMode::kCAFFE_ROUND_UP`	Caffe is not supported since 9.0
`PreviewFeature::kDISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805` `PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805`	External tactics are always disabled for core code This flag is on by default
`ProfilingVerbosity::kDEFAULT` `ProfilingVerbosity::kVERBOSE`	`ProfilingVerbosity::kLAYER_NAMES_ONLY` `ProfilingVerbosity::kDETAILED`
`ResizeMode`	Use `InterpolationMode`, alias is removed
`RNNDirection` `RNNGateType` `RNNInputMode` `RNNOperation`	RNN related data structures are removed
`SampleMode::kDEFAULT`	`SampleMode::kSTRICT_BOUNDS`
`SliceMode`	Use `SampleMode`, alias is removed

2.5. Removed C++ Plugins

Table 7. Removed C++ Plugins and their Suggested Superseded Plugin
C++ Plugin	Superseded Plugin
`createAnchorGeneratorPlugin()` `createBatchedNMSPlugin()` `createInstanceNormalizationPlugin()` `createNMSPlugin()` `createNormalizePlugin()` `createPriorBoxPlugin()` `createRegionPlugin()` `createReorgPlugin()` `createRPNROIPlugin()` `createSplitPlugin()`	`GridAnchorPluginCreator::createPlugin()` `BatchedNMSPluginCreator::createPlugin()` `InstanceNormalizationPluginCreator::createPlugin()` `NMSPluginCreator::createPlugin()` `NormalizePluginCreator::createPlugin()` `PriorBoxPluginCreator::createPlugin()` `RegionPluginCreator::createPlugin()` `ReorgPluginCreator::createPlugin()` `RPROIPluginCreator::createPlugin()` `INetworkDefinition::addSlice()`
`struct Quadruple`	Related plugins are removed

2.6. Removed Safety C++ APIs

Table 8. Removed Safety C++ APIs and their Suggested Superseded Safety API
Safety C++ API	Superseded Safety API
`safe::ICudaEngine::bindingIsInput()` `safe::ICudaEngine::getBindingBytesPerComponent()` `safe::ICudaEngine::getBindingComponentsPerElement()` `safe::ICudaEngine::getBindingDataType()` `safe::ICudaEngine::getBindingDimensions()` `safe::ICudaEngine::getBindingIndex()` `safe::ICudaEngine::getBindingName()` `safe::ICudaEngine::getBindingVectorizedDim()` `safe::ICudaEngine::getNbBindings()` `safe::ICudaEngine::getTensorFormat()`	`safe::ICudaEngine::tensorIOMode()` `safe::ICudaEngine::getTensorBytesPerComponent()` `safe::ICudaEngine::getTensorComponentsPerElement()` `safe::ICudaEngine::getTensorDataType()` `safe::ICudaEngine::getTensorShape()` `safe::name-based methods` `safe::name-based methods` `safe::ICudaEngine::getTensorVectorizedDim()` `safe::ICudaEngine::getNbIOTensors()` `safe::ICudaEngine::getBindingFormat()`
`safe::IExecutionContext::enqueueV2()` `safe::IExecutionContext::getStrides()`	`safe::IExecutionContext::enqueueV3()` `safe::IExecutionContext::getTensorStrides()`

3. trtexec

3.1. trtexec Flag Changes

Table 9. Changes to flag `workspace` and `minTiming`
TensorRT 8.x	TensorRT 10.0
trtexec \ --onnx=/path/to/model.onnx \ --saveEngine=/path/to/engine.trt \ --optShapes=input:$INPUT_SHAPE \ --avgTiming=1 \ --workspace=1024 \ --minTiming=1	trtexec \ --onnx=/path/to/model.onnx \ --saveEngine=/path/to/engine.trt \ --optShapes=input:$INPUT_SHAPE \ --avgTiming=1 \ --memPoolSize=workspace:1024

Table 9. Changes to flag workspace and minTiming

TensorRT 8.x

TensorRT 10.0

trtexec \
    --onnx=/path/to/model.onnx \
    --saveEngine=/path/to/engine.trt \
    --optShapes=input:$INPUT_SHAPE \
    --avgTiming=1 \
    --workspace=1024 \
    --minTiming=1

trtexec \
    --onnx=/path/to/model.onnx \
    --saveEngine=/path/to/engine.trt \
    --optShapes=input:$INPUT_SHAPE \
    --avgTiming=1 \
        --memPoolSize=workspace:1024

3.2. Removed trtexec Flags

Table 10. Removed trtexec Flags and their Suggested Superseded Flag
trtexec Flag	Superseded Flag
`--minTiming`	`avgTiming`
`--preview=features` options: `disableExternalTacticSourcesForCore0805` `fasterDynamicShapes0805`	N/A
`--workspace=N`	`--memPoolSize=poolspec`

3.3. Deprecated trtexec Flags

Table 11. Deprecated trtexec Flags
trtexec Flag
`--buildOnly`
`--explicitPrecision`
`--heuristic`
`--nvtxMode`

Notices

Notice

This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.

NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.

Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.

NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.

NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.

NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.

No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.

Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.

THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.

Arm

Arm, AMBA and Arm Powered are registered trademarks of Arm Limited. Cortex, MPCore and Mali are trademarks of Arm Limited. "Arm" is used to represent Arm Holdings plc; its operating company Arm Limited; and the regional subsidiaries Arm Inc.; Arm KK; Arm Korea Limited.; Arm Taiwan Limited; Arm France SAS; Arm Consulting (Shanghai) Co. Ltd.; Arm Germany GmbH; Arm Embedded Technologies Pvt. Ltd.; Arm Norway, AS and Arm Sweden AB.

HDMI

HDMI, the HDMI logo, and High-Definition Multimedia Interface are trademarks or registered trademarks of HDMI Licensing LLC.

Blackberry/QNX

Trademarks, including but not limited to BLACKBERRY, EMBLEM Design, QNX, AVIAGE, MOMENTICS, NEUTRINO and QNX CAR are the trademarks or registered trademarks of BlackBerry Limited, used under license, and the exclusive rights to such trademarks are expressly reserved.

Google

Android, Android TV, Google Play and the Google Play logo are trademarks of Google, Inc.

Trademarks

NVIDIA, the NVIDIA logo, and BlueField, CUDA, DALI, DRIVE, Hopper, JetPack, Jetson AGX Xavier, Jetson Nano, Maxwell, NGC, Nsight, Orin, Pascal, Quadro, Tegra, TensorRT, Triton, Turing and Volta are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

Abstract

1. Python

1.1. Python API Changes

1.2. Added Python APIs

Types

Methods and Properties

1.3. Removed Python APIs

2. C++

2.1. C++ API Changes

2.2. 64-Bit Dimension Changes

2.3. Added C++ APIs

Enums

Types

Methods and Properties

2.4. Removed C++ APIs

2.5. Removed C++ Plugins

2.6. Removed Safety C++ APIs

3. trtexec

3.1. trtexec Flag Changes

3.2. Removed trtexec Flags

3.3. Deprecated trtexec Flags

Notices

Notice

Arm

HDMI

Blackberry/QNX

Google

Trademarks

Copyright