LLM Inference Quick Start Recipes

View as Markdown

Optimized deployment guides for NVIDIA hardware for the most popular open source LLMs.

Dynamo Disaggregated Recipes

ModelDynamo + SGLangDynamo + TRT-LLMDynamo + vLLM
DeepSeek-R1✅ (Slurm)✅✅
DeepSeek-V3.2 (FP4)—✅—
DeepSeek-V4-Flash✅—✅
DeepSeek-V4-Pro✅—✅
GLM-5 (NVFP4)✅——
GPT-OSS—✅—
Kimi-K2.5—✅—
Llama-3-70B——✅
Nemotron-3-Super✅✅✅
Nemotron-3-Ultra——✅
Qwen3-235B-A22B (FP8)—✅—
Qwen3-32B——✅
Qwen3-32B (FP8)—✅✅

Framework Recipes

ModelSGLangTRT-LLMvLLM
DeepSeek-R1✅✅ (0528)✅ (R1/V3)
DeepSeek-V3.2✅——
DeepSeek-V4✅——
DeepSeek-V4-Pro——✅
DiffusionGemma——✅
GLM-4.7✅—✅
GLM-5✅✅✅
GLM-5.1✅—✅
GLM-5.2✅—✅
GPT-OSS✅✅✅
GPT-OSS + Eagle3—✅—
Kimi-K2-Thinking—✅—
Kimi-K2.5✅—✅
Kimi-K2.6✅—✅
Kimi-K2.7-Code✅—✅
Llama-3.3-70B✅✅✅
Llama-4-Scout—✅✅
MiniMax-M2——✅
MiniMax-M2.5✅—✅
MiniMax-M3✅✅✅
Nemotron-3-Nano✅✅✅
Nemotron-3-Super✅✅✅
Nemotron-3-Ultra✅—✅
Qwen—✅—
Qwen3—✅—
Qwen3-Coder-480B-A35B——✅
Qwen3-Next✅✅—
Qwen3.5✅✅✅