Knowledge BaseConcepts

Sidecar Backends

Run Dynamo beside a stock inference engine through its native gRPC API.
View as Markdown

Experimental. Sidecar packaging, launchers, and API coverage are still evolving. The sidecar path does not yet match every feature of the in-process backends.

A Dynamo sidecar runs beside the inference engine process. It registers the engine with Dynamo discovery, forwards engine events into the Dynamo event plane, and serves requests from the Dynamo frontend by calling the engine’s native gRPC API.

Design Goals

  • Keep the upstream engine’s native serve path and argument surface.
  • Move toward public, versioned gRPC contracts with explicit backward compatibility instead of importing private engine APIs.
  • Isolate Dynamo and engine dependencies in separate processes.
  • Attribute failures through engine-specific and Dynamo-specific logs and health checks.
  • Reuse Dynamo’s frontend, routing, planning, and disaggregated-serving orchestration.

Architecture

Dynamo Sidecar architecture. A client sends OpenAI-compatible HTTP to the Dynamo Frontend, which tokenizes, routes, and sends token IDs over the request plane to the Dynamo Sidecar.

The frontend and router discover sidecars and send requests to them over the Dynamo request plane. Each sidecar converts requests to the engine’s native gRPC API and streams responses back.

Responsibilities

LayerResponsibility
Dynamo frontend and routerOpenAI-compatible API, preprocessing, and routing to sidecars
Dynamo sidecarEngine registration and discovery, request forwarding over native gRPC, and event forwarding
Inference engineNative gRPC request serving, scheduling, sampling, token generation, KV cache, and GPU execution

Container Packaging

The sidecar Dockerfile builds all three engine-specific sidecar executables into one CPU-only image. Deployments select an engine by setting the container command:

EngineContainer command
vLLMdynamo-vllm-sidecar
SGLangdynamo-sglang-sidecar
TensorRT-LLMdynamo-trtllm-sidecar

The image’s default entrypoint, dynamo-sidecar, maps the short names vllm, sglang, and trtllm onto those executables. It is a convenience for ad-hoc docker run; the deployment manifests override it with command. The inference engine remains in a separate GPU container, so the sidecar image does not include vLLM, SGLang, TensorRT-LLM, CUDA, or engine-specific Python dependencies. The image runs as the non-root dynamo user with numeric user ID 1000 and declares port 9090 for Dynamo system endpoints, so Kubernetes can enforce runAsNonRoot.

The sidecar image is published to NGC for each release, starting with 1.6.0:

docker pull nvcr.io/nvidia/ai-dynamo/dynamo-sidecar:<version>  # 1.6.0 or later

To build it from source instead, run from the repository root with the sidecar Dockerfile:

docker build -f lib/sidecar/Dockerfile -t dynamo-sidecar:1.6.0-dev .

Pod Layout

Kubernetes sidecar mode is a work in progress.

The engine and sidecar share one worker pod. The engine is the main container; the sidecar is a native sidecar: an init container named runtime with restartPolicy: Always, which requires Kubernetes 1.29 or later. The two connect over loopback.

podTemplate:
  spec:
    initContainers:
    - name: runtime
      image: nvcr.io/nvidia/ai-dynamo/dynamo-sidecar:<version>
      command: [dynamo-vllm-sidecar]
      args: [--grpc-endpoint, 127.0.0.1:50051]
      restartPolicy: Always
    containers:
    - name: main
      image: vllm/vllm-openai:<version>

Kubernetes starts the sidecar before the engine and keeps it running for the life of the pod. Declaring the runtime init container enables sidecar mode in the Dynamo operator, which gives the sidecar /live and /health probes on port 9090. These probes do not track engine loading, so keep the engine’s own probes on main. Sidecar mode supports worker, prefill, and decode components; multinode deployments are not yet supported.

Topologies

Each engine page shows its single-node TP, multi-node TP, and multi-node DP topologies: vLLM, SGLang, TensorRT-LLM.