SGLang Sidecar

dynamo-sglang-sidecar connects a Dynamo worker to SGLang’s native gRPC server. See Sidecar Backends for the architecture.

Support Matrix

Launch Locally

See lib/sidecar/sglang/launch/ for all topologies. For example, aggregated serving on one GPU:

export DYN_DISCOVERY_BACKEND=file   # single host: no etcd or NATS needed
./lib/sidecar/sglang/launch/agg.sh

In a second terminal:

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Deploy on Kubernetes

See lib/sidecar/sglang/deploy/ for all manifests. For example, aggregated serving:

Before applying, replace <your-registry>/dynamo-sidecar in the manifest with a sidecar image and create the hf-token-secret Secret that the manifest reads.

kubectl create secret generic hf-token-secret -n <namespace> \
  --from-literal=HF_TOKEN=<your-hf-token>
kubectl apply -f lib/sidecar/sglang/deploy/agg.yaml -n <namespace>
kubectl port-forward -n <namespace> svc/sglang-sidecar-agg-frontend 8000:8000

Multinode deployments require backendFramework: sglang and complete leader and worker role PodTemplates. The templates own the native engine commands and sidecar placement. See the TP example and attention-DP example. Attention-DP requires SGLang 0.5.22 or later and a Dynamo sidecar with matching attention-DP metadata support. Engine-failure KV recovery remains incomplete and is tracked in DEP #14897.

Migrate an existing DGD to multinode sidecars

The multinode examples use DYNAMO_RANK and DYNAMO_LEADER_ADDRESS. The operator injects these variables only into DGDs originally created by operator 1.6.0 or later. Upgrading the operator or switching to sidecar mode does not enable them for older DGDs.

For a DGD created before operator 1.6.0, choose one of these options:

• Delete and recreate the DGD after upgrading the operator, then use the examples with their existing topology arguments.
• Keep the DGD and replace the --node-rank and --dist-init-addr arguments in the main container of both the leader and worker role templates with the provider-specific arguments below.

For Grove, use these arguments. The group-wide pod index requires Grove v0.1.0-alpha.15 or later and matching CRDs.

- --node-rank
- $(GROVE_PCSG_POD_INDEX)
- --dist-init-addr
- $(GROVE_PCSG_NAME)-$(GROVE_PCSG_INDEX)-sglangworker-ldr-0.$(GROVE_HEADLESS_SERVICE):29500

Here, sglangworker is the example component name, SGLangWorker, in lowercase. Replace it with your component name in lowercase if you rename the component.

For LeaderWorkerSet, use these arguments:

- --node-rank
- $(LWS_WORKER_INDEX)
- --dist-init-addr
- $(LWS_LEADER_ADDRESS):29500

Kubernetes expands these variable references in container arguments. No shell wrapper is needed.

Topologies

The frontend reaches each sidecar over Dynamo’s request, discovery, and event planes; the sidecar reaches the engine over its native gRPC API. Dashed arrows carry KV events.

Single-Node TP

One engine on one node, with one sidecar.

Multi-Node TP

One engine spans two nodes. Only the leader node has a sidecar; the follower node holds the remaining TP ranks.

Multi-Node DP

The leader sidecar registers every DP rank and serves all requests; SGLang dispatches each one to the right scheduler. The follower sidecar accepts no requests and relays its node’s KV events directly to the frontend.