For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Digest
  • Getting Started
    • Quickstart
    • Introduction
    • Local Installation
    • Building from Source
    • Contribution Guide
  • Resources
    • Support Matrix
    • Feature Matrix
    • Release Artifacts
    • Examples
  • Kubernetes Deployment
    • Deployment Guide
  • User Guides
    • KV Cache Aware Routing
    • Disaggregated Serving
    • KV Cache Offloading
    • Dynamo Benchmarking
    • Multimodal
    • Diffusion (Preview)
    • Tool Calling
    • LoRA Adapters
    • Agents
    • Observability (Local)
    • Fault Tolerance
    • Writing Python Workers
  • Backends
    • SGLang
    • TensorRT-LLM
    • vLLM
  • Components
    • Frontend
    • Router
    • Planner
    • Profiler
    • KVBM
  • Integrations
    • LMCache
    • SGLang HiCache
    • FlexKV
    • KV Events for Custom Engines
  • Design Docs
    • Overall Architecture
    • Architecture Flow
    • Disaggregated Serving
    • Distributed Runtime
    • Blog
  • Documentation
    • Dynamo Docs Guide
NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoDocumentation
Digest
On this page
  • Overview
  • Initialization
  • Service Discovery Backends
  • Runtime Initialization
  • Calling Endpoints
  • Examples
Design Docs

Distributed Runtime

||View as Markdown|
Edit this page
Previous

Disaggregated Serving

Next

Discovery Plane

Overview

Dynamo’s DistributedRuntime is the core infrastructure in the framework that enables distributed communication and coordination between different Dynamo components. It is implemented in Rust (/lib/runtime) and exposed to other programming languages via bindings (i.e., Python bindings can be found in /lib/bindings/python). The runtime supports multiple discovery backends (Kubernetes-native or etcd) and request planes (TCP, HTTP, or NATS). DistributedRuntime follows a hierarchical structure:

  • DistributedRuntime: This is the highest level object that exposes the distributed runtime interface. It manages connections to discovery backends (K8s API or etcd) and optional messaging (NATS for KV events), and handles lifecycle with cancellation tokens.
  • Namespace: A Namespace is a logical grouping of components that isolate between different model deployments.
  • Component: A Component is a discoverable object within a Namespace that represents a logical unit of workers.
  • Endpoint: An Endpoint is a network-accessible service that provides a specific service or function.

While theoretically each DistributedRuntime can have multiple Namespaces as long as their names are unique (similar logic also applies to Component/Namespace and Endpoint/Component), in practice, each dynamo components typically are deployed with its own process and thus has its own DistributedRuntime object. However, they share the same namespace to discover each other.

For example, a typical deployment configuration (like examples/backends/vllm/deploy/agg.yaml or examples/backends/sglang/deploy/agg.yaml) has multiple components:

  • Frontend: Starts an HTTP server (OpenAI-compatible API on port 8000), handles incoming requests, applies chat templates, performs tokenization, and routes requests to workers. The make_engine function encapsulates this functionality.
  • Worker components (e.g., VllmDecodeWorker, VllmPrefillWorker, SGLangDecodeWorker, TRTLLMWorker): Perform the actual inference computation using their respective engines (SGLang, TensorRT-LLM, vLLM).

Since these components are deployed in different processes, each has its own DistributedRuntime. Within their own DistributedRuntime, they all share the same Namespace (e.g., vllm-agg, sglang-disagg). Under their namespace, each has its own Component:

  • Frontend uses the make_engine function which handles HTTP serving, request preprocessing, and worker discovery automatically
  • Worker components register with names like backend, prefill, decode, or encoder depending on their role
  • Workers register endpoints like generate, clear_kv_blocks, or load_metrics

Their DistributedRuntimes are initialized in their respective main functions, their Namespaces are configured in the deployment YAML, and their Endpoints are obtained by path. In Python, use runtime.endpoint("namespace.component.endpoint") (e.g., runtime.endpoint("dynamo.backend.generate")).

Initialization

In this section, we explain what happens under the hood when DistributedRuntime/Namespace/Component/Endpoint objects are created. There are multiple modes for DistributedRuntime initialization based on the deployment environment.

1The hierarchy and naming may change over time, and this document might not reflect the latest changes. Regardless of such changes, the main concepts would remain the same.

Service Discovery Backends

The DistributedRuntime supports two service discovery backends, configured via DYN_DISCOVERY_BACKEND:

  • KV Store Discovery (DYN_DISCOVERY_BACKEND=etcd): Uses etcd for service discovery. This is the default for all deployments unless explicitly overridden. Other KV store backends (file, mem) are also available.

  • Kubernetes Discovery (DYN_DISCOVERY_BACKEND=kubernetes): Uses native Kubernetes resources (DynamoWorkerMetadata CRD, EndpointSlices) for service discovery. Must be explicitly set. The Dynamo operator automatically sets this environment variable for Kubernetes deployments. No etcd required.

Note: There is no automatic detection of the deployment environment. The runtime defaults to etcd. For Kubernetes deployments, the operator injects DYN_DISCOVERY_BACKEND=kubernetes into pod environments.

When using Kubernetes discovery, the KV store backend automatically switches to in-memory storage since etcd is not needed.

Runtime Initialization

  • DistributedRuntime: When a DistributedRuntime object is created, it establishes connections based on the discovery backend:
    • Kubernetes mode: Uses K8s API for service registration via DynamoWorkerMetadata CRD. No external dependencies required.
    • KV Store mode: Connects to etcd for service discovery. Creates a primary lease with a background keep-alive task. All objects registered under this DistributedRuntime use this lease_id to maintain their lifecycle.
    • NATS (optional): Used for KV event messaging when using KV-aware routing. Can be disabled via --no-kv-events flag, which enables prediction-based routing without event persistence.
    • Request Plane: TCP by default. Can be configured to use HTTP or NATS via DYN_REQUEST_PLANE environment variable.
  • Namespace: Namespaces are primarily a logical grouping mechanism. They provide the root path for all components under this Namespace.
  • Component: When a Component object is created, it registers a service in the internal registry of the DistributedRuntime, which tracks all services and endpoints.
  • Endpoint: When an Endpoint object is created and started, it performs registration based on the discovery backend:
    • Kubernetes mode: Endpoint information is stored in DynamoWorkerMetadata CRD resources, which are watched by other components for discovery.
    • KV Store mode: Endpoint information is stored in etcd at a path following the naming: /services/{namespace}/{component}/{endpoint}-{lease_id}. Note that endpoints of different workers of the same type (i.e., two VllmPrefillWorkers in one deployment) share the same Namespace, Component, and Endpoint name. They are distinguished by their different primary lease_id.

Calling Endpoints

Dynamo uses a Client object to call an endpoint. When a Client is created, it is given the name of the Namespace, Component, and Endpoint. It then watches for endpoint changes:

  • Kubernetes mode: Watches DynamoWorkerMetadata CRD resources for endpoint updates.
  • KV Store mode: Sets up an etcd watcher to monitor the prefix /services/{namespace}/{component}/{endpoint}.

The watcher continuously updates the Client with information about available Endpoints.

The user can decide which load balancing strategy to use when calling the Endpoint from the Client, which is done in push_router.rs. Dynamo supports three load balancing strategies:

  • random: randomly select an endpoint to hit
  • round_robin: select endpoints in round-robin order
  • direct: direct the request to a specific endpoint by specifying the instance ID

After selecting which endpoint to hit, the Client sends the request using the configured request plane (TCP by default). The request plane handles the actual transport:

  • TCP (default): Direct TCP connection with connection pooling
  • HTTP: HTTP/2-based transport
  • NATS: Message broker-based transport (legacy)

Examples

We provide native rust and python (through binding) examples for basic usage of DistributedRuntime:

  • Rust: /lib/runtime/examples/
  • Python: We also provide complete examples of using DistributedRuntime. Please refer to the engines in components/src/dynamo for full implementation details.