Extending TensorRT with Custom Layers#
NVIDIA TensorRT supports many layers, and its functionality is continually extended; however, there can be cases in which the layers supported do not cater to a model’s specific needs. In such cases, TensorRT can be extended by implementing custom layers, often called plugins.
TensorRT contains standard plugins that can be loaded into your application. For a list of open-source plugins, refer to GitHub: TensorRT plugins.
To use standard TensorRT plugins in your application, the libnvinfer_plugin.so (nvinfer_plugin.dll on Windows) library must be loaded, and all plugins must be registered by calling initLibNvInferPlugins in your application code. For more information about these plugins, refer to the NvInferPlugin.h file.
You can write and add your own if these plugins do not meet your needs.
- Adding Custom Layers Using the C++ API
- Implementing a Plugin Class
- Implementing a Plugin Creator Class
- Registering a Plugin Creator with the Plugin Registry
- Adding a Plugin Instance to a TensorRT Network
- Example: Adding a Custom Layer with Dynamic Shapes Using C++
- Example: Adding a Custom Layer with Data-Dependent and Shape Input-Dependent Shapes Using C++
- Example: Adding a Custom Layer with INT8 I/O Support Using C++
- Adding Custom Layers using the Python API (TensorRT >= 10.6)
- Adding Custom Layers using the Python API (Advanced/TensorRT <= 10.5)
- Registration of a Python Plugin
- Building and Running TensorRT Engines Containing Python Plugins
- Implementing
enqueueof a Python Plugin - Translating Device Buffers/CUDA Stream Pointers in
enqueueto other Frameworks - Automatic Downcasting
- Example: Adding a Custom Layer to a TensorRT Network Using Python
- Enabling Timing Caching and Using Custom Tactics
- Sharing Custom Resources Among Plugins
- Using Custom Layers When Importing a Model with a Parser
- Plugin API Description
- Migrating V2 Plugins to
IPluginV3 - Coding Guidelines for Plugins
- Plugin Shared Libraries