Working with Transformers

TensorRT provides built-in layers and fusions for common transformer workloads, so attention, position embeddings, and expert routing map onto dedicated APIs instead of hand-built subgraphs.

In this guide

• Rotary Position Embedding: express RoPE with IRotaryEmbeddingLayer and import it from ONNX

• KV Cache: update the K/V cache during LLM inference with IKVCacheUpdateLayer

• MoE (Mixture of Experts): route tokens to experts with IMoELayer on SM100, SM103, and SM110

• Fused Attention: trigger attention fusion through IAttention or a primitive layer graph

• Multi-Device Attention: split the key-value sequence across GPUs with context parallelism