Working with Transformers

TensorRT provides built-in layers and fusions for common transformer workloads, including RoPE, KV cache updates, MoE, fused attention, and multi-device attention. The pages below cover each topic in depth.

• Rotary Position Embedding
• KV Cache
• MoE (Mixture of Experts)
• Fused Attention
  • IO Form (Tensor Layout)
  • Supported Attention Fusions
  • Constructing a Fused Attention
  • Example Workflow: Quantize an ONNX Attention Model to FP8 with ModelOpt
  • Example: Exploiting Sparsity in Attention Masks
• Example: Using Transformer-Oriented APIs
  • Padded Format
  • Packed (Ragged) Format
  • Fully Packed (Ragged) Format
• Multi-Device Attention