Reference Guide

Overview

The SGLang backend in Dynamo uses a modular architecture where main.py dispatches to specialized initialization modules based on the worker type. Each worker type has its own init module, request handler, health check, and registration logic.

Dynamo SGLang uses SGLang’s native argument parser — all SGLang engine arguments (e.g., --model-path, --tp, --trust-remote-code) are passed through directly. Dynamo adds its own arguments for worker mode selection, tokenizer control, and disaggregation configuration.

Worker Types

Argument Reference

Dynamo-Specific Arguments

These arguments are added by Dynamo on top of SGLang’s native arguments. For the complete field-by-field reference of every Dynamo SGLang flag, default, and environment variable, see SGLang Configuration.

The current supported parser names for both flags are documented in Tool Call Parsing (Dynamo) and Reasoning Parsing (Dynamo).

Tokenizer Behavior

By default, Dynamo handles tokenization and detokenization through its Rust-based frontend, passing input_ids to SGLang. This enables all frontend endpoints (v1/chat/completions, v1/completions, v1/embeddings).

For SGLang-native preprocessing (tool calling, reasoning parsing, chat templates), use --dyn-chat-processor sglang on the frontend. See SGLang Chat Processor for architecture and usage.

Set --default-chat-template-kwargs on the frontend, not on the SGLang worker. The frontend renders the chat template, so SGLang’s flag has no effect behind Dynamo, and the worker refuses it at startup. The SGLang sidecar logs a warning instead, because its engine can also serve SGLang’s own HTTP server. The frontend applies the defaults to every chat request, for workers and sidecars, before it reads the thinking controls. A key in the request’s chat_template_kwargs takes precedence unless its value is null or "auto", and a request thinking control replaces the thinking keys in the defaults. The flag cannot set chat_template; use --custom-jinja-template to replace the chat template.

Request Cancellation

When a client disconnects, Dynamo automatically cancels the in-flight request across all workers, freeing compute resources. A background cancellation monitor detects disconnection and aborts the SGLang request.

For details on the cancellation architecture, see Request Cancellation.

Graceful Shutdown

SGLang workers use Dynamo’s graceful shutdown mechanism. When a SIGTERM or SIGINT is received:

1. Discovery unregister: The worker is removed from service discovery so no new requests are routed to it
2. Grace period: In-flight requests are allowed to complete
3. Deferred handlers: SGLang’s internal signal handlers (captured during startup via monkey-patching loop.add_signal_handler) are invoked after the graceful period

This ensures zero dropped requests during rolling updates or scale-down events.

For more details, see Graceful Shutdown.

Health Checks

Each worker type has a specialized health check payload that validates the full inference pipeline:

Health checks are registered with the Dynamo runtime and called by the frontend or Kubernetes liveness probes. See Observability Architecture for the active health-check design.

Metrics and KV Events

Prometheus Metrics

Enable metrics with --enable-metrics on the worker. Set DYN_SYSTEM_PORT to expose the /metrics endpoint:

DYN_SYSTEM_PORT=8081 python -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --enable-metrics

Both SGLang engine metrics (sglang:* prefix) and Dynamo runtime metrics (dynamo_* prefix) are served from the same endpoint.

For metric details, see SGLang Observability. For a visualization walkthrough, see Metrics and Dashboards.

KV Events

When configured with --kv-events-config, workers publish KV cache events (block creation/deletion) for the KV-aware router. Events are published via ZMQ from SGLang’s scheduler and relayed through Dynamo’s event plane.

For DP attention mode (--enable-dp-attention), the publisher handles multiple DP ranks per node, each with its own KV event stream.

Engine Routes

SGLang workers expose operational endpoints via Dynamo’s system server:

Configurable SGLang Method Routes

Set --engine-route or DYN_SGLANG_ENGINE_ROUTES to expose additional SGLang methods on a worker’s system server without changing Dynamo. Each descriptor uses this grammar:

<path>[=<method>][:engine|tm]

• <path> becomes /engine/<path> and can contain /-separated segments.
• <method> defaults to <path>. Use server_info=get_server_info to expose an alias. A path containing / therefore needs an explicit method.
• The target defaults to the sgl.Engine instance. Set :tm to target its tokenizer manager.
• Repeat --engine-route for multiple routes. Separate multiple DYN_SGLANG_ENGINE_ROUTES descriptors with whitespace.

For per-worker generation control with NVIDIA Collective Communications Library (NCCL) weight updates, expose this allowlist:

export DYN_SGLANG_ENGINE_ROUTES="server_info=get_server_info pause_generation:tm flush_cache init_weights_update_group update_weights_from_distributed:tm destroy_weights_update_group continue_generation:tm"
python -m dynamo.sglang --model-path Qwen/Qwen3-0.6B

Call these routes on each worker’s system-server address. Send inference requests to the Dynamo frontend and reserve the system server for worker control.

SGLang 0.5.15 exposes get_internal_state only on the tokenizer manager, where it returns a list with one state dictionary per data-parallel rank. The Engine get_server_info adapter merges the server arguments and scheduler topology into the top-level dictionary expected by control clients, so /engine/server_info returns that dictionary directly without a result envelope.

The update_weights_from_distributed:tm form accepts SGLang’s tokenizer-manager request schema, including weight_version. The public SGLang Engine wrapper omits that field. This configurable /engine/update_weights_from_distributed route is separate from Dynamo’s fixed /engine/control/update_weights_from_distributed route.

Configured Engine methods receive JSON object keys as keyword arguments. For tokenizer-manager methods with an annotated dataclass or msgspec request, Dynamo constructs the request object from the JSON body and supplies None for an HTTP request parameter when the SGLang signature requires one. Async methods are awaited, sync Engine wrappers run outside the server loop, and nested results are converted to JSON. Empty request bodies become {}. Configured routes accept the ordinary HTTP methods handled by the Dynamo system server.

Dynamo parses and resolves the complete route list during worker startup. Startup fails for malformed descriptors, unknown targets, duplicate paths, private methods, missing methods, or non-callable attributes. An unconfigured method has no route. Request content cannot select or replace the method captured for a configured path, and failure values such as {"success": false} remain failures in the response.

See Also

• Examples: Local deployment launch scripts
• Disaggregation: P/D architecture and KV transfer
• Diffusion: image, video, and language diffusion models
• Configuration and Tuning: KV-aware routing configuration

Experimental Worker Drain

Experimental. Manual drain currently supports standalone aggregated SGLang workers using the TCP response plane. Workers that share an engine across multiple Dynamo gateway processes do not expose these controls.

There is no separate drain enable flag. On a supported worker, enabling the system listener with DYN_SYSTEM_PORT also exposes the lifecycle API below. Experimental describes the support and validation scope, not an additional opt-in requirement. Setting DYN_SYSTEM_PORT=-1 disables the entire system listener, including its health and metrics endpoints; this is the default when no system port is configured. Normal SIGTERM shutdown still works without the listener.

Use the selected worker’s system listener directly and restrict access to trusted administrators. The frontend HTTP address does not expose these worker-local routes.

Send no body or an empty JSON object. Drain returns 202 while completion is pending. Status and a successful resume return 200. Conflicts return 409, and discovery RPC failures return 503. Manual drain requires the TCP response plane. Other response planes return 409.

Drain and resume a worker

Start discovery services and the frontend as described in the SGLang launch scripts. For a standalone aggregated worker, enable its system listener before startup:

DYN_SYSTEM_PORT=8081 python3 -m dynamo.sglang \
  --model-path Qwen/Qwen3-0.6B \
  --request-plane tcp \
  --response-plane tcp

When the worker runs locally, set its control address:

WORKER_URL=http://127.0.0.1:8081

For Kubernetes, select one worker Pod. Replace default and my-sglang-worker with its namespace and Pod name, and run this command in a separate terminal. The worker must already have DYN_SYSTEM_PORT=8081 in its container environment.

kubectl -n default port-forward pod/my-sglang-worker 18081:8081

Use WORKER_URL=http://127.0.0.1:18081 for that forwarded worker. Send the drain request:

curl --fail-with-body -sS -i -X POST "$WORKER_URL/engine/drain" \
  -H 'Content-Type: application/json' -d '{}'

With the default routing grace, the response is HTTP 202. Example body:

{"worker_id":"1234","state":"draining","reason":"routing_grace"}

Poll the same worker’s status until state is drained:

curl --fail-with-body -sS "$WORKER_URL/engine/status"

Example completion response:

{"worker_id":"1234","state":"drained","reason":null}

To cancel an administrative drain or restore the worker after it drains, send:

curl --fail-with-body -sS -i -X POST "$WORKER_URL/engine/resume" \
  -H 'Content-Type: application/json' -d '{}'

A successful resume returns HTTP 200 with state="serving". After discovery converges, verify inference through your frontend. For the model above and a frontend listening on port 8000:

FRONTEND_URL=http://127.0.0.1:8000
curl --fail-with-body -sS "$FRONTEND_URL/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"max_tokens":8}'

Expect a successful Chat Completions response with a choices array. Polling or resuming a different worker does not complete the selected worker’s operation.

Admission and completion semantics

Drain lets previously admitted requests run and keeps internal health probes available. During routing convergence the worker can still accept requests. Configure that interval with DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS (default 5 seconds). After admission closes, late requests receive a worker-unavailable error. Frontend pipelines that support migration can reselect using the existing --migration-limit and ordinary routing and replay rules. A zero limit disables reselection.

The grace period starts after discovery confirms withdrawal. Admission checks its deadline at request ingress, even while an engine-control operation holds the lifecycle lock. Repeated drain calls do not extend the deadline.

Discovery RPC errors leave the write outcome unconfirmed. A failed drain keeps admission open and reports discovery_unregister_pending; retry drain to confirm withdrawal. Every resume call reasserts discovery registration, including when admission is already open. Resume opens admission before registration so requests can arrive as soon as the endpoint becomes visible. If registration fails, the call returns an error and admission stays open unless shutdown supersedes resume. Retry resume to confirm registration.

SGLang’s pause_generation and drain are mutually exclusive. Continue generation before draining a paused worker. Resume a draining or drained worker before pausing it. SIGTERM moves the worker to stopping, prevents resume, and follows the existing shutdown sequence.

Poll status for drained. The worker reaches this state after business admission closes, worker ingress requests and their response streams finish, and the SGLang tokenizer manager has no remaining request IDs. While draining, reason indicates whether the worker is waiting for routing grace, Dynamo ingress, SGLang requests, or an available completion check.

Replacement admission and Pod deletion remain part of the manual Grove OnDelete release workflow.