Keeps rollout workers busy and healthy with KV-aware routing, fast policy refresh through ModelExpress, and recovery from worker loss.
Faster multi-turn sessions by keeping each session’s KV cache on one worker and putting interactive turns ahead of background work.
20-30% better throughput and 10-90% lower TTFT by processing images in parallel, avoiding redundant encoding and prefill, and disaggregating the encoder.
Up to 40% lower TTFT by overlapping the ASR and LLM stages, and better resiliency by recovering realtime transcription after worker loss.
Faster video requests by decoding in the Rust frontend or on the GPU’s hardware decoder, and routing repeated clips to the worker that already caches them.
8-17x faster generation with step-distilled models, and 5 seconds of 1080p video in about 4.5 seconds with FastVideo.
60-80% of a dedicated GPU’s throughput by letting many lightly used models share a GPU.
The performance improvements come from benchmarks on real production use cases and vary with the model, hardware, and workload. See the respective docs for details.