Extending TensorRT with Custom Layers#
NVIDIA TensorRT supports many layers, and its functionality is continually extended; however, there can be cases in which the layers supported do not cater to a model’s specific needs. In such cases, TensorRT can be extended by implementing custom layers, often called plugins.
TensorRT contains standard plugins that can be loaded into your application. For a list of open-source plugins, refer to GitHub: TensorRT plugins.
To use standard TensorRT plugins in your application, the libnvinfer_plugin.so (nvinfer_plugin.dll on Windows) library must be loaded, and all plugins must be registered by calling initLibNvInferPlugins in your application code. For more information about these plugins, refer to the NvInferPlugin.h file.
You can write and add your own if these plugins do not meet your needs.
Adding Custom Layers using the Python API (TensorRT >= 10.6)#
For most use cases, defining Python plugins with a decorator-based approach is recommended (available starting in TensorRT 10.6). Refer to the Writing Custom Operators with TensorRT Python Plugins in the TensorRT Python API documentation for a manual describing different use cases and best practices. Note that embedding Python-defined plugins to TensorRT engines such that the engine is independent of Python and the plugin source itself, is only possible with this approach.
- Adding Custom Layers Using the C++ API
- Implementing a Plugin Class
- Implementing a Plugin Creator Class
- Registering a Plugin Creator with the Plugin Registry
- Adding a Plugin Instance to a TensorRT Network
- Example: Adding a Custom Layer with Dynamic Shapes
- Example: Adding a Custom Layer with Data-Dependent and Shape Input-Dependent Shapes Using C++
- Example: Adding a Custom Layer with INT8 I/O Support Using C++
- Enabling Timing Caching and Using Custom Tactics
- Plugin API Description