Examples

NVIDIA AITune Examples

View as Markdown

This directory contains practical examples demonstrating how to use NVIDIA AITune to tune different types of AI models for inference performance.

ESM2 and Parakeet CTC use AITune’s default strategy without overrides. For AOT tuning, this selects MaxThroughputStrategy with automatic backend candidates and maximum-batch-size discovery. JIT also defaults to MaxThroughputStrategy, with fewer candidates and maximum-batch-size discovery disabled. Stable Diffusion and FLUX disable maximum-batch-size discovery to tune their recorded image batches. Stable Diffusion uses the default AOT backend candidates, as do FLUX’s non-transformer modules. Explicit backend lists remain where models need quantization, precision settings, or specific compiler options.

ResNet

Computer Vision - Image Classification

Shows how to tune ResNet models for image classification tasks. This example demonstrates model tuning and inference tuning for convolutional neural networks.

  • Location: ResNet
  • Model: ResNet50 image classification
  • Use Case: Optimizing CNN models for computer vision tasks
  • Key Features:
    • Model tuning with AITune
    • Image classification inference
    • Performance comparison before/after tuning
  • More Info:

StableDiffusion

Generative AI - Text-to-Image

Demonstrates tuning of Stable Diffusion models for text-to-image generation. This example shows how to tune diffusion models for faster and more efficient image generation.

FLUX

Generative AI - Advanced Text-to-Image

Shows tuning of the FLUX text-to-image model, demonstrating advanced diffusion model tuning techniques for high-quality image generation.

  • Location: FLUX
  • Model: FLUX.1-dev from Black Forest Labs
  • Use Case: Optimizing state-of-the-art text-to-image models
  • Key Features:
    • Advanced diffusion model tuning
    • Single-GPU and Diffusers context-parallel execution
    • High-quality image generation
    • Efficient inference pipeline tuning
  • More Info:

WAN

Generative AI - Text-to-Video

Demonstrates distributed tuning and serving of the WAN 2.1 text-to-video model with Diffusers context parallelism.

  • Location: WAN
  • Model: WAN 2.1 T2V 1.3B from Hugging Face
  • Use Case: Optimizing large text-to-video diffusion models across multiple GPUs
  • Key Features:
    • Multi-GPU Ulysses and ring context parallelism
    • Independent backend selection for the WAN transformer
    • Rank-local distributed checkpoints
    • Original and tuned MP4 generation
    • Collective NVIDIA Dynamo serving
  • More Info:

ParakeetCTC

Speech AI - Automatic Speech Recognition

Demonstrates tuning of ASR (Automatic Speech Recognition) models using NVIDIA’s Parakeet CTC model for speech-to-text conversion.

ParakeetRNNT

Speech AI - Automatic Speech Recognition

Demonstrates tuning of ASR (Automatic Speech Recognition) models using NVIDIA’s Parakeet RNNT model for speech-to-text conversion.

ESM2

Text AI - Advanced Text Embedding

Demonstrates tuning of ESM2 model for text embedding tasks.

E5Large

Text AI - Advanced Text Embedding

Demonstrates tuning of E5Large model for text embedding tasks.

LLM

Large Language Models - Text Generation

Demonstrates tuning of Large Language Models for text generation tasks. This example shows how to optimize LLMs for efficient inference with KV cache support.

  • Location: LLM
  • Model: Qwen3.5-0.8B from HuggingFace
  • Use Case: Optimizing LLMs for text generation and inference
  • Key Features:
    • LLM model tuning with AITune
    • Single-GPU and Transformers native tensor-parallel execution
    • Static and dynamic KV cache optimization
    • Prefill and decode phase optimization
    • HuggingFace integration
  • More Info:

JIT Tuning

Just-In-Time Model Tuning

Demonstrates how to use NVIDIA AITune with JIT (Just-In-Time) tuning that requires no code changes. This example shows how to enable automatic tuning through environment variables without modifying existing code.

Each example includes:

  • Complete setup instructions
  • Usage examples with CLI commands
  • Model-specific tuning parameters
  • AI Dynamo deployment instructions where supported

To get started, navigate to any example directory and follow the README instructions for that specific model type.