C++ Migration Patterns for TensorRT 11.x#

Migrating from Weak Typing to Strong Typing#

TensorRT 11.x removes all precision-enabling builder flags such as BuilderFlag::kFP16 and BuilderFlag::kINT8. Use ModelOpt AutoCast to convert your ONNX model to mixed precision before building.

Migrating INT8 Calibration to Explicit Quantization#

TensorRT 11.x removes IInt8Calibrator and all its subclasses, along with setInt8Calibrator(). Use ModelOpt or manual Q/DQ nodes for explicit quantization instead.

Migrating INT8 Calibration to Explicit Quantization (Before / After / Summary)

Before (TensorRT 10.x)

class MyCalibrator : public nvinfer1::IInt8EntropyCalibrator2
{
public:
    int32_t getBatchSize() const noexcept override { return 1; }

    bool getBatch(void* bindings[], char const* names[], int32_t nbBindings) noexcept override
    {
        // Fill bindings with calibration data
        if (mCurrentBatch >= mNumBatches)
            return false;
        // ... copy data to GPU
        mCurrentBatch++;
        return true;
    }

    void const* readCalibrationCache(size_t& length) noexcept override { return nullptr; }
    void writeCalibrationCache(void const* ptr, size_t length) noexcept override {}

private:
    int32_t mCurrentBatch{0};
    int32_t mNumBatches{100};
};

// Usage
auto config = SampleUniquePtr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());
config->setFlag(BuilderFlag::kINT8);

MyCalibrator calibrator;
config->setInt8Calibrator(&calibrator);

auto plan = SampleUniquePtr<IHostMemory>(builder->buildSerializedNetwork(*network, *config));

In TensorRT 11.x, IInt8Calibrator and all subclasses have been removed along with setInt8Calibrator(). Use ModelOpt or manual Q/DQ nodes.

After (TensorRT 11.x)

// Step 1: Quantize the model offline using ModelOpt:
//   python -m modelopt.onnx.quantization --onnx_path model.onnx --calibration_data data.npz
//
// Alternatively, add QuantizeLinear/DequantizeLinear nodes to the ONNX graph manually,
// or use the INetworkDefinition::addQuantize() and addDequantize() APIs.

// Step 2: Build the pre-quantized model
auto builder = SampleUniquePtr<nvinfer1::IBuilder>(nvinfer1::createInferBuilder(logger));
auto network = SampleUniquePtr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(0));
auto config = SampleUniquePtr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());

auto parser = SampleUniquePtr<nvonnxparser::IParser>(
    nvonnxparser::createParser(*network, logger));
parser->parseFromFile("model_quantized.onnx",
    static_cast<int>(nvinfer1::ILogger::Severity::kWARNING));

auto plan = SampleUniquePtr<IHostMemory>(builder->buildSerializedNetwork(*network, *config));

Summary of Changes

Removed the IInt8Calibrator subclass entirely
Removed config->setFlag(BuilderFlag::kINT8) and config->setInt8Calibrator()
Quantization is applied to the model offline (Q/DQ nodes in the ONNX graph) or through the addQuantize() / addDequantize() network definition APIs

Migrating Plugins from `IPluginV2DynamicExt` to `IPluginV3`#

The following example shows a complete plugin migration from V2 to V3 using a NonZero plugin that computes the indices of non-zero elements. This demonstrates V3’s support for data-dependent output shapes, which was not possible with V2.

See also

Side-by-Side V2 ↔ V3 API Mapping: Method-by-method mapping table grouped by lifecycle phase (core, build, runtime, serialization, network attachment).
Known Migration Issues: Known issues encountered when porting V2 plugins, including the empty PluginField initializer crash and the strongly-typed network requirement.
Performance: Resolving V2 → V3 Regressions: Checklist for resolving performance regressions after migrating a plugin from IPluginV2DynamicExt to IPluginV3.

Migrating Plugins from IPluginV2DynamicExt to IPluginV3 (Before / After / Summary)

Before (TensorRT 10.x, ``IPluginV2DynamicExt``)

class NonZeroPluginV2 : public nvinfer1::IPluginV2DynamicExt
{
public:
    // IPluginV2 core methods
    char const* getPluginType() const noexcept override { return "NonZeroPlugin"; }
    char const* getPluginVersion() const noexcept override { return "1"; }
    int32_t getNbOutputs() const noexcept override { return 1; }

    // Output dimensions - limited to expressions of input dimensions only
    DimsExprs getOutputDimensions(int32_t outputIndex, DimsExprs const* inputs,
        int32_t nbInputs, IExprBuilder& exprBuilder) noexcept override
    {
        // Cannot express data-dependent shapes - must use an upper bound
        DimsExprs output;
        output.nbDims = 2;
        output.d[0] = exprBuilder.operation(DimensionOperation::kPROD,
            *inputs[0].d[0], *inputs[0].d[1]); // Upper bound: R * C
        output.d[1] = exprBuilder.constant(2);
        return output;
    }

    bool supportsFormatCombination(int32_t pos, PluginTensorDesc const* inOut,
        int32_t nbInputs, int32_t nbOutputs) noexcept override
    {
        return inOut[pos].format == TensorFormat::kLINEAR;
    }

    void configurePlugin(DynamicPluginTensorDesc const* in, int32_t nbInputs,
        DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept override {}

    int32_t enqueue(PluginTensorDesc const* inputDesc, PluginTensorDesc const* outputDesc,
        void const* const* inputs, void* const* outputs,
        void* workspace, cudaStream_t stream) noexcept override
    {
        // Execute kernel
        return 0;
    }

    size_t getWorkspaceSize(PluginTensorDesc const* inputs, int32_t nbInputs,
        PluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept override
    {
        return 0;
    }

    // Serialization
    size_t getSerializationSize() const noexcept override { return sizeof(bool); }
    void serialize(void* buffer) const noexcept override
    {
        *reinterpret_cast<bool*>(buffer) = mRowOrder;
    }

    IPluginV2DynamicExt* clone() const noexcept override
    {
        return new NonZeroPluginV2(mRowOrder);
    }

    // ... other required IPluginV2 methods, including destroy and setPluginNamespace

private:
    bool mRowOrder{true};
};

// V2 Plugin Creator
class NonZeroCreatorV2 : public nvinfer1::IPluginCreator
{
public:
    char const* getPluginName() const noexcept override { return "NonZeroPlugin"; }
    char const* getPluginVersion() const noexcept override { return "1"; }
    PluginFieldCollection const* getFieldNames() noexcept override { return &mFC; }

    IPluginV2* createPlugin(char const* name, PluginFieldCollection const* fc) noexcept override
    {
        return new NonZeroPluginV2(/*rowOrder=*/true);
    }

    IPluginV2* deserializePlugin(char const* name, void const* data,
        size_t length) noexcept override
    {
        bool rowOrder = *reinterpret_cast<bool const*>(data);
        return new NonZeroPluginV2(rowOrder);
    }

    // ... other required methods

private:
    PluginFieldCollection mFC{};
};

IPluginV2Layer* addNonZeroLayer(
    INetworkDefinition& network, ITensor& inputTensor)
{
    NonZeroPluginV2 plugin(/*rowOrder=*/true);
    ITensor* inputs[] = {&inputTensor};
    return network.addPluginV2(inputs, 1, plugin);
}

After (TensorRT 11.x, ``IPluginV3``)

class NonZeroPlugin : public IPluginV3, public IPluginV3OneCore,
                  public IPluginV3OneBuild, public IPluginV3OneRuntime
{
public:
    NonZeroPlugin(int32_t rowOrder) : mRowOrder(rowOrder) {}

    // IPluginV3 - return the appropriate capability interface
    IPluginCapability* getCapabilityInterface(PluginCapabilityType type) noexcept override
    {
        if (type == PluginCapabilityType::kBUILD)
            return static_cast<IPluginV3OneBuild*>(this);
        if (type == PluginCapabilityType::kRUNTIME)
            return static_cast<IPluginV3OneRuntime*>(this);
        return static_cast<IPluginV3OneCore*>(this);
    }

    // IPluginV3OneCore
    AsciiChar const* getPluginName() const noexcept override { return "NonZeroPlugin"; }
    AsciiChar const* getPluginVersion() const noexcept override { return "1"; }
    AsciiChar const* getPluginNamespace() const noexcept override { return ""; }

    // IPluginV3OneBuild
    int32_t getNbOutputs() const noexcept override { return 2; } // data + size tensor

    int32_t getOutputDataTypes(DataType* outputTypes, int32_t nbOutputs,
        DataType const* inputTypes, int32_t nbInputs) const noexcept override
    {
        outputTypes[0] = DataType::kINT32; // non-zero indices
        outputTypes[1] = DataType::kINT64; // size tensor
        return 0;
    }

    // Output shapes - V3 supports data-dependent shapes through declareSizeTensor
    int32_t getOutputShapes(DimsExprs const* inputs, int32_t nbInputs,
        DimsExprs const* shapeInputs, int32_t nbShapeInputs,
        DimsExprs* outputs, int32_t nbOutputs,
        IExprBuilder& exprBuilder) noexcept override
    {
        auto upperBound = exprBuilder.operation(DimensionOperation::kPROD,
            *inputs[0].d[0], *inputs[0].d[1]);
        auto optValue = exprBuilder.operation(DimensionOperation::kFLOOR_DIV,
            *upperBound, *exprBuilder.constant(2));

        // Declare a size tensor - enables data-dependent output shapes
        auto numNonZero = exprBuilder.declareSizeTensor(1, *optValue, *upperBound);

        outputs[0].nbDims = 2;
        outputs[0].d[0] = numNonZero; // Data-dependent dimension
        outputs[0].d[1] = exprBuilder.constant(2);

        outputs[1].nbDims = 0; // Size tensor is a scalar
        return 0;
    }

    bool supportsFormatCombination(int32_t pos, DynamicPluginTensorDesc const* inOut,
        int32_t nbInputs, int32_t nbOutputs) noexcept override
    {
        return inOut[pos].desc.format == TensorFormat::kLINEAR;
    }

    int32_t configurePlugin(DynamicPluginTensorDesc const* in, int32_t nbInputs,
        DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept override
    {
        return 0;
    }

    size_t getWorkspaceSize(DynamicPluginTensorDesc const* inputs, int32_t nbInputs,
        DynamicPluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept override
    {
        return 0;
    }

    // IPluginV3OneRuntime
    int32_t enqueue(PluginTensorDesc const* inputDesc, PluginTensorDesc const* outputDesc,
        void const* const* inputs, void* const* outputs,
        void* workspace, cudaStream_t stream) noexcept override
    {
        // Execute kernel - same as V2
        return 0;
    }

    int32_t onShapeChange(PluginTensorDesc const* in, int32_t nbInputs,
        PluginTensorDesc const* out, int32_t nbOutputs) noexcept override
    {
        return 0;
    }

    // Serialization - uses PluginFieldCollection instead of raw bytes
    PluginFieldCollection const* getFieldsToSerialize() noexcept override
    {
        mDataToSerialize.clear();
        mDataToSerialize.emplace_back("rowOrder", &mRowOrder, PluginFieldType::kINT32, 1);
        mFCToSerialize.nbFields = mDataToSerialize.size();
        mFCToSerialize.fields = mDataToSerialize.data();
        return &mFCToSerialize;
    }

    IPluginV3* attachToContext(IPluginResourceContext* context) noexcept override
    {
        return clone();
    }

    IPluginV3* clone() noexcept override
    {
        return new NonZeroPlugin(mRowOrder);
    }

private:
    int32_t mRowOrder{1};
    std::vector<PluginField> mDataToSerialize;
    PluginFieldCollection mFCToSerialize{};
};

// V3 Plugin Creator
class NonZeroCreator : public nvinfer1::IPluginCreatorV3One
{
public:
    NonZeroCreator()
    {
        mPluginAttributes.emplace_back("rowOrder", nullptr, PluginFieldType::kINT32, 1);
        mFC.nbFields = mPluginAttributes.size();
        mFC.fields = mPluginAttributes.data();
    }

    char const* getPluginName() const noexcept override { return "NonZeroPlugin"; }
    char const* getPluginVersion() const noexcept override { return "1"; }
    char const* getPluginNamespace() const noexcept override { return ""; }
    PluginFieldCollection const* getFieldNames() noexcept override { return &mFC; }

    // Phase-aware creation - no separate deserializePlugin needed
    IPluginV3* createPlugin(char const* name, PluginFieldCollection const* fc,
        TensorRTPhase phase) noexcept override
    {
        int32_t rowOrder = 1;
        for (int32_t i = 0; i < fc->nbFields; ++i)
        {
            if (std::string_view(fc->fields[i].name) == "rowOrder")
                rowOrder = *static_cast<int32_t const*>(fc->fields[i].data);
        }
        return new NonZeroPlugin(rowOrder);
    }

private:
    PluginFieldCollection mFC{};
    std::vector<PluginField> mPluginAttributes;
};

// addPluginV3 accepts both data inputs and shape inputs.
IPluginV3Layer* addNonZeroLayer(
    INetworkDefinition& network, ITensor& inputTensor)
{
    NonZeroPlugin plugin(/*rowOrder=*/1);
    ITensor* inputs[] = {&inputTensor};
    return network.addPluginV3(inputs, 1, nullptr, 0, plugin);
}

Summary of Changes

Plugin class inherits from IPluginV3, IPluginV3OneCore, IPluginV3OneBuild, and IPluginV3OneRuntime instead of IPluginV2DynamicExt
Added getCapabilityInterface() to return the appropriate interface for each phase (core, build, runtime)
getOutputDimensions() replaced by getOutputShapes(), which supports data-dependent output shapes using exprBuilder.declareSizeTensor()
Added required getOutputDataTypes() method
serialize() / getSerializationSize() replaced by getFieldsToSerialize(), which returns a PluginFieldCollection for structured serialization
Added onShapeChange() and attachToContext() methods
Creator inherits from IPluginCreatorV3One instead of IPluginCreator; createPlugin() takes a TensorRTPhase parameter, and deserializePlugin() is no longer needed, because createPlugin() handles both build and runtime phases
addPluginV2(inputs, nbInputs, plugin) replaced by addPluginV3(inputs, nbInputs, shapeInputs, nbShapeInputs, plugin)

Known Issues When Migrating Plugins

Empty PluginField initializers can crash V3 dispatch. When a plugin advertises a PluginField with a nullptr data pointer and length == 0, the V3 creator dispatch path can dereference the pointer during build or deserialization. Populate every entry with a non-null sentinel buffer, even when the value is unused at runtime:
```
// Bad: empty initializer
mPluginAttributes.emplace_back("flag", nullptr, PluginFieldType::kINT32, 0);

// Good: non-null sentinel keeps the dispatch path safe
static int32_t kDummy = 0;
mPluginAttributes.emplace_back("flag", &kDummy, PluginFieldType::kINT32, 1);
```
Use strongly-typed networks with IPluginV3. Mixing IPluginV3 plugins with weakly-typed networks can hit fusion paths that were not exercised by IPluginV2DynamicExt and trigger crashes. In TensorRT 11.0.0 all precision-enabling builder flags (BuilderFlag::kFP16, kINT8, kBF16, kFP8, kINT4, kFP4) have been removed, so any network you build is strongly typed by default; no action required for fresh 11.x builds. Authors back-porting V3 plugins to a 10.x build for evaluation must explicitly opt in with createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)). The parameter is a bitmask of type NetworkDefinitionCreationFlags (an alias for uint32_t), and the enumerators are bit positions, so shift the enumerator rather than passing it directly.

Migrating Weight Streaming APIs#

The weight streaming API has been updated in TensorRT 11.x. The getMinimumWeightStreamingBudget() method has been removed; compute a budget from getStreamableWeightsSize() and available device memory instead.

The C++ fragments in this section require <algorithm> and <cuda_runtime.h>. SampleUniquePtr is the helper from the TensorRT OSS samples; use your application’s equivalent owning pointer outside the sample tree.

#include <algorithm>
#include <cuda_runtime.h>

Migrating Memory Management APIs#

TensorRT 11.x replaces getDeviceMemorySize() with getDeviceMemorySizeV2() (which returns int64_t), removes createExecutionContextWithoutDeviceMemory(), and replaces setDeviceMemory(void*) with setDeviceMemoryV2(void*, int64_t).