NVIDIA Dynamo Glossary

B

Block - A fixed-size chunk of tokens (typically 16 or 64 tokens) used for efficient KV cache management and memory allocation, serving as the fundamental unit for techniques like PagedAttention.

C

Component - The fundamental deployable unit in Dynamo. A discoverable service entity that can host multiple endpoints and typically maps to a Docker container (such as VllmWorker, Router, Processor).

Conditional Disaggregation - Dynamo’s intelligent decision-making process within disaggregated serving that determines whether a request is processed locally or sent to a remote prefill engine based on prefill length and queue status.

D

Decode Phase - The second phase of LLM inference that generates output tokens one at a time.

depends() - A Dynamo function that creates dependencies between services, enabling automatic client generation and service discovery.

Disaggregated Serving - Dynamo’s core architecture that separates prefill and decode phases into specialized engines to maximize GPU throughput and improve performance.

Distributed Runtime - Dynamo’s Rust-based core system that manages service discovery, communication, and component lifecycle across distributed clusters.

Dynamo - NVIDIA’s high-performance distributed inference framework for Large Language Models (LLMs) and generative AI models, designed for multinode environments with disaggregated serving and cache-aware routing.

Dynamo Artifact - A packaged archive containing an inference graph and its dependencies, created using dynamo build. It’s the containerized, deployable version of a Graph.

Dynamo Cloud - A Kubernetes platform providing managed deployment experience for Dynamo inference graphs.

dynamo build - The CLI command to containerize inference graphs or parts of graphs into Docker containers.

dynamo deploy - The CLI command to deploy inference graphs to Kubernetes with Helm charts or custom operators.

dynamo run - The CLI command to quickly experiment and test models with various LLM engines.

dynamo serve - The CLI command to compose and serve inference graphs locally.

E

@endpoint - A Python decorator used to define service endpoints within a Dynamo component.

Endpoint - A specific network-accessible API within a Dynamo component, such as generate or load_metrics.

F

Frontend - Dynamo’s API server component that receives user requests and provides OpenAI-compatible HTTP endpoints.

G

Graph - A collection of interconnected Dynamo components that form a complete inference pipeline with request paths (single-in) and response paths (many-out for streaming). A graph can be packaged into a Dynamo Artifact for deployment.

I

Instance - A running process with a unique instance_id. Multiple instances can serve the same namespace, component, and endpoint for load balancing

K

KV Block Manager (KVBM) - Dynamo’s scalable runtime component that handles memory allocation, management, and remote sharing of Key-Value blocks across heterogeneous and distributed environments.

KV Cache - Key-Value cache that stores computed attention states from previous tokens to avoid recomputation during inference.

KV Router - Dynamo’s intelligent routing system that directs requests to workers with the highest cache overlap to maximize KV cache reuse. Determines routing based on KV cache hit rates and worker metrics.

KVIndexer - Dynamo component that maintains a global view of cached blocks across all workers using a prefix tree structure to calculate cache hit rates.

KVPublisher - Dynamo component that emits KV cache events (stored/removed) from individual workers to the global KVIndexer.

N

Namespace - Dynamo’s logical grouping mechanism for related components. Similar to directories in a file system, they prevent collisions between different deployments.

NIXL (NVIDIA Inference tranXfer Library) - High-performance data transfer library optimized for inference workloads, supporting direct GPU-to-GPU transfers and multiple memory hierarchies.

P

PagedAttention - Memory management technique from vLLM that efficiently manages KV cache by chunking requests into blocks.

Planner - Dynamo component that performs dynamic resource scaling based on real-time demand signals and system metrics.

Prefill Phase - The first phase of LLM inference that processes the input prompt and generates KV cache.

Prefix Caching - Optimization technique that reuses previously computed KV cache for common prompt prefixes.

Processor - Dynamo component that handles request preprocessing, tokenization, and routing decisions.

R

RadixAttention - Technique from SGLang that uses a prefix tree structure for efficient KV cache matching, insertion, and eviction.

RDMA (Remote Direct Memory Access) - Technology that allows direct memory access between distributed systems, used for efficient KV cache transfers.

S

@service - Python decorator used to define a Dynamo service class.

SGLang - Fast LLM inference framework with native embedding support and RadixAttention.

T

Tensor Parallelism (TP) - Model parallelism technique where model weights are distributed across multiple GPUs.

TensorRT-LLM - NVIDIA’s optimized LLM inference engine with multinode MPI distributed support.

Time-To-First-Token (TTFT) - The latency from receiving a request to generating the first output token.

V

vLLM - High-throughput LLM serving engine with Ray distributed support and PagedAttention.

X

xPyD (x Prefill y Decode) - Dynamo notation describing disaggregated serving configurations where x prefill workers serve y decode workers. Dynamo supports runtime-reconfigurable xPyD.