Best Practices

This guide provides best practices for optimizing performance with TensorRT. It covers benchmarking, profiling, optimization techniques, and hardware/software configuration for achieving optimal inference performance.

What You Will Learn:

• How to benchmark performance using trtexec

• Techniques for analyzing and profiling your models

• Strategies for optimizing inference speed

• Hardware and software environment best practices

This guide is organized from basic benchmarking to advanced optimization techniques, allowing you to progressively improve your model’s performance.

• Performance Benchmarking using trtexec
  • Performance Benchmarking with an ONNX File
  • Performance Benchmarking with ONNX+Quantization
  • Per-Layer Runtime and Layer Information
  • Performance Benchmarking with TensorRT Plan File
  • Duration and Number of Iterations
• Advanced Performance Measurement Techniques
  • Wall-Clock Timing
  • CUDA Events
  • Built-In TensorRT Profiling
  • ONNX Profiling Tools
  • CUDA Profiling Tools
  • Tracking Memory
• Hardware/Software Environment for Performance Measurements
  • GPU Information Query and GPU Monitoring
  • GPU Clock Locking and Floating Clock
  • GPU Power Consumption and Power Throttling
  • GPU Temperature and Thermal Throttling
  • H2D/D2H Data Transfers and PCIe Bandwidth
  • TCC Mode and WDDM Mode
  • Enqueue-Bound Workloads and CUDA Graphs
  • BlockingSync and SpinWait Synchronization Modes
• Optimizing TensorRT Performance
  • Batching
  • Within-Inference Multi-Streaming
  • Cross-Inference Multi-Streaming
  • CUDA Graphs
  • Enabling Fusion
  • Limiting Compute Resources
  • Deterministic Tactic Selection
• Overhead of Shape Change and Optimization Profile Switching
  • Optimizing Layer Performance
  • Optimizing for Tensor Cores
  • Optimizing Plugins
  • Optimizing Python Performance
• Improving Model Accuracy
• Optimizing Builder Performance
  • Timing Cache
  • Builder Optimization Level