Reference Guide

Overview

The SGLang backend in Dynamo uses a modular architecture where main.py dispatches to specialized initialization modules based on the worker type. Each worker type has its own init module, request handler, health check, and registration logic.

Dynamo SGLang uses SGLang’s native argument parser — all SGLang engine arguments (e.g., --model-path, --tp, --trust-remote-code) are passed through directly. Dynamo adds its own arguments for worker mode selection, tokenizer control, and disaggregation configuration.

Worker Types

Argument Reference

Dynamo-Specific Arguments

These arguments are added by Dynamo on top of SGLang’s native arguments. For the complete field-by-field reference of every Dynamo SGLang flag, default, and environment variable, see SGLang Configuration.

The current supported parser names for both flags are documented in Tool Call Parsing (Dynamo) and Reasoning Parsing (Dynamo).

Tokenizer Behavior

By default, Dynamo handles tokenization and detokenization through its Rust-based frontend, passing input_ids to SGLang. This enables all frontend endpoints (v1/chat/completions, v1/completions, v1/embeddings).

For SGLang-native preprocessing (tool calling, reasoning parsing, chat templates), use --dyn-chat-processor sglang on the frontend. See SGLang Chat Processor for architecture and usage.

Request Cancellation

When a client disconnects, Dynamo automatically cancels the in-flight request across all workers, freeing compute resources. A background cancellation monitor detects disconnection and aborts the SGLang request.

For details on the cancellation architecture, see Request Cancellation.

Graceful Shutdown

SGLang workers use Dynamo’s graceful shutdown mechanism. When a SIGTERM or SIGINT is received:

1. Discovery unregister: The worker is removed from service discovery so no new requests are routed to it
2. Grace period: In-flight requests are allowed to complete
3. Deferred handlers: SGLang’s internal signal handlers (captured during startup via monkey-patching loop.add_signal_handler) are invoked after the graceful period

This ensures zero dropped requests during rolling updates or scale-down events.

For more details, see Graceful Shutdown.

Health Checks

Each worker type has a specialized health check payload that validates the full inference pipeline:

Health checks are registered with the Dynamo runtime and called by the frontend or Kubernetes liveness probes. See Observability Architecture for the active health-check design.

Metrics and KV Events

Prometheus Metrics

Enable metrics with --enable-metrics on the worker. Set DYN_SYSTEM_PORT to expose the /metrics endpoint:

$
DYN_SYSTEM_PORT=8081 python -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --enable-metrics

Both SGLang engine metrics (sglang:* prefix) and Dynamo runtime metrics (dynamo_* prefix) are served from the same endpoint.

For metric details, see SGLang Observability. For a visualization walkthrough, see Metrics and Dashboards.

KV Events

When configured with --kv-events-config, workers publish KV cache events (block creation/deletion) for the KV-aware router. Events are published via ZMQ from SGLang’s scheduler and relayed through Dynamo’s event plane.

For DP attention mode (--enable-dp-attention), the publisher handles multiple DP ranks per node, each with its own KV event stream.

Engine Routes

SGLang workers expose operational endpoints via Dynamo’s system server:

Configurable SGLang Method Routes

Set --engine-route or DYN_SGLANG_ENGINE_ROUTES to expose additional SGLang methods on a worker’s system server without changing Dynamo. Each descriptor uses this grammar:

<path>[=<method>][:engine|tm]

• <path> becomes /engine/<path> and can contain /-separated segments.
• <method> defaults to <path>. Use server_info=get_server_info to expose an alias. A path containing / therefore needs an explicit method.
• The target defaults to the sgl.Engine instance. Set :tm to target its tokenizer manager.
• Repeat --engine-route for multiple routes. Separate multiple DYN_SGLANG_ENGINE_ROUTES descriptors with whitespace.

For per-worker generation control with NVIDIA Collective Communications Library (NCCL) weight updates, expose this allowlist:

$
export DYN_SGLANG_ENGINE_ROUTES="server_info=get_server_info pause_generation:tm flush_cache init_weights_update_group update_weights_from_distributed:tm destroy_weights_update_group continue_generation:tm"
$
python -m dynamo.sglang --model-path Qwen/Qwen3-0.6B

Call these routes on each worker’s system-server address. Send inference requests to the Dynamo frontend and reserve the system server for worker control.

SGLang 0.5.15 exposes get_internal_state only on the tokenizer manager, where it returns a list with one state dictionary per data-parallel rank. The Engine get_server_info adapter merges the server arguments and scheduler topology into the top-level dictionary expected by control clients, so /engine/server_info returns that dictionary directly without a result envelope.

The update_weights_from_distributed:tm form accepts SGLang’s tokenizer-manager request schema, including weight_version. The public SGLang Engine wrapper omits that field. This configurable /engine/update_weights_from_distributed route is separate from Dynamo’s fixed /engine/control/update_weights_from_distributed route.

Configured Engine methods receive JSON object keys as keyword arguments. For tokenizer-manager methods with an annotated dataclass or msgspec request, Dynamo constructs the request object from the JSON body and supplies None for an HTTP request parameter when the SGLang signature requires one. Async methods are awaited, sync Engine wrappers run outside the server loop, and nested results are converted to JSON. Empty request bodies become {}. Configured routes accept the ordinary HTTP methods handled by the Dynamo system server.

Dynamo parses and resolves the complete route list during worker startup. Startup fails for malformed descriptors, unknown targets, duplicate paths, private methods, missing methods, or non-callable attributes. An unconfigured method has no route. Request content cannot select or replace the method captured for a configured path, and failure values such as {"success": false} remain failures in the response.

See Also

• Examples: Local deployment launch scripts
• Disaggregation: P/D architecture and KV transfer
• Diffusion: LLM, image, and video diffusion models
• Configuration and Tuning: KV-aware routing configuration