SGLang Chat Processor

The SGLang chat processor enables SGLang-native preprocessing and postprocessing in the Dynamo frontend. It uses SGLang’s tokenizer, chat templates, tool call parser, and reasoning parser directly — bypassing the default Rust preprocessor for v1/chat/completions requests.

When to Use

Use --dyn-chat-processor sglang when Dynamo’s built-in Rust preprocessor does not yet support a tool call parser or reasoning parser you need. The SGLang processor delegates to SGLang’s Python implementations, so any parser SGLang supports works immediately.

Common cases:

• A tool call format not yet in the Rust tool_calling library
• A reasoning parser not yet supported natively
• A chat template that the Rust preprocessor doesn’t handle correctly

If the parser you need is missing from the Rust preprocessor, consider opening an issue or PR to add native support — native parsers avoid the Python GIL overhead entirely.

Quick Start

# Frontend with SGLang processor, tool calling, and reasoning
python -m dynamo.frontend \
  --router-mode kv \
  --dyn-chat-processor sglang \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
# Workers (unchanged)
CUDA_VISIBLE_DEVICES=0 python -m dynamo.sglang \
  --model-path Qwen/Qwen3-14B-FP8 \
  --served-model-name Qwen/Qwen3-14B-FP8 \
  --tp 1 --trust-remote-code \
  --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'

DeepSeek Prompt Encoding

For DeepSeek V4 and V4.1 checkpoints identified by checkpoint metadata, the processor prefers SGLang’s recipe renderer when that module is available, including when the tokenizer has a chat template. An explicit --chat-template override takes precedence over checkpoint encoding. This path requires the checkpoint’s fast tokenizer and the deepseek-recipe package supplied by the SGLang installation. SGLang v0.5.22 keeps the existing V4 encoder path. V4.1 checkpoints without a chat template require the recipe renderer available in SGLang v0.5.23 or newer; they cannot fall back to the V4 encoder.

Recipe encoding rejects continue_final_message=true combined with thinking because the current output parser does not track the continued prompt’s reasoning state. Continuation with thinking disabled remains supported.

Dynamo continues to own sampling settings, constrained decoding, and output parsing. Image URLs must match the recipe’s placeholder order; requests with a different order or UUID-only images are rejected before inference.

Frontend Arguments

These arguments are passed to the frontend (not the worker) when using --dyn-chat-processor sglang:

Environment Variables

Tool Calling

The processor supports all SGLang tool call formats. Pass --tool-call-parser on the frontend:

python -m dynamo.frontend \
  --dyn-chat-processor sglang \
  --tool-call-parser hermes

Any parser supported by SGLang can be used. See the SGLang documentation for the full list of available tool call parsers.

Example: Tool Call Request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-14B-FP8",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

Response:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_8cd24396f3671048",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\": \"Paris\"}"
        }
      }],
      "reasoning_content": "The user wants weather info for Paris..."
    },
    "finish_reason": "tool_calls"
  }]
}

Reasoning Parsing

For models that produce chain-of-thought reasoning (e.g., Qwen3, DeepSeek-R1), pass --reasoning-parser:

python -m dynamo.frontend \
  --dyn-chat-processor sglang \
  --reasoning-parser qwen3

The parser separates think tag content into the reasoning_content field and regular content into the content field.

Migration from --use-sglang-tokenizer

--use-sglang-tokenizer on the worker is deprecated. Replace with --dyn-chat-processor sglang on the frontend:

  # Before (deprecated)
- python -m dynamo.sglang --model-path <model> --use-sglang-tokenizer
- python -m dynamo.frontend
  # After
  python -m dynamo.sglang --model-path <model>
+ python -m dynamo.frontend --dyn-chat-processor sglang

Key differences:

See Also

• Tool Calling: General tool calling guide
• Reference Guide: Full SGLang backend reference
• Agentic Workloads: Priority scheduling and KV cache tuning