SGLang Multimodal

This document provides a comprehensive guide for multimodal inference using SGLang backend in Dynamo. SGLang multimodal supports native EPD and EP/D flows where the SGLang engine performs media encoding, plus explicit encode-worker E/PD and E/P/D flows with NIXL (RDMA) for zero-copy tensor transfer.

Support Matrix

Supported URL Formats

Deployment Patterns

SGLang supports EPD, EP/D, E/PD, and E/P/D patterns. See Multimodal Model Serving for detailed explanations.

Component Flags

SGLang-Specific Characteristics

• Vision Encoder in Python: SGLang runs model-specific vision encoding after the frontend or worker decodes the media
• Token Expansion: Single <|image_pad|> token replaced with N tokens based on embedding shape
• NIXL Transfer: Embeddings transferred from Encoder → PD Worker using NIXL
• Optional Rust Frontend Decode: --frontend-decoding moves media fetch and decode to the Rust frontend while retaining SGLang preprocessing and vision encoding

Multimodal KV Routing

Multimodal KV routing works with SGLang’s aggregated worker topology. It is independent of the E/PD and E/P/D encoder-disaggregation patterns described later in this guide.

SGLang RadixAttention includes a per-media pad_value token in its prefix-cache key. Dynamo must use that same token in the routing view:

1. The Rust frontend hashes each image or sampled video and calculates its expanded token count.
2. It derives pad_value = MM_PAD_SHIFT_VALUE + (mm_hash % 2^30).
3. It substitutes that value for the media placeholder in the routing-only token view.
4. It forwards the original hash list through GenerateReqInput.mm_hashes.
5. SGLang uses the supplied hash when constructing its own pad_value, keeping the router and RadixAttention cache keys aligned.

The frontend selects this behavior automatically when the worker’s ModelDeploymentCard reports backend_framework="sglang".

For Qwen3-VL and Qwen3.5 video, --frontend-decoding also makes an aggregated worker publish its effective frame-sampling, resize, timestamp, and placeholder contract. The frontend enables exact video routing only when every worker in the discovery cohort publishes the same compatible contract. Unsupported models, legacy workers, mixed contracts, or a processor override fall back to text-prefix routing without failing the request.

Step 4 needs an SGLang build that accepts mm_hashes. Dynamo’s SGLang image carries the upstream patch that adds it; a custom build without it is still supported, but Dynamo detects the missing argument when the worker starts and routes on the text prefix alone, so media identity no longer contributes to cache overlap.

Launch an aggregated deployment with multimodal KV routing:

cd $DYNAMO_HOME
bash examples/backends/sglang/launch/agg_multimodal_router.sh

The launcher enables --frontend-decoding, configures KV events on each worker, and sets --router-mode kv with a matching frontend and worker block size.

Version Requirements

The Dynamo SGLang image includes both routing prerequisites:

• Dynamo is built with the mm-routing Rust feature.
• SGLang 0.5.13 or later includes GenerateReqInput.mm_hashes support. Dynamo currently pins 0.5.19.
• The CUDA image includes the codec-limited VP8/VP9 frontend decoder.

Custom installations on SGLang 0.5.12 or earlier need the mm_hashes change from sgl-project/sglang#25300. Without it, requests still complete but fall back to text-prefix-only routing. Prefer upgrading to 0.5.13 or later instead of patching an installed package.

Enable DYN_LOG=info,mm_routing=debug to inspect media-token counts, multimodal hashes, the video processor contract, and the selected worker’s overlap. A repeated request should select the same worker with high block overlap.

For the user-facing workflow, see Multimodal KV Routing.

Use the Latest Release

We recommend using the latest stable release of dynamo to avoid breaking changes:

You can find the latest release and check out the corresponding branch with:

git checkout $(git describe --tags $(git rev-list --tags --max-count=1))

EPD Serving (Simple Aggregated)

Components

• worker: DecodeWorkerHandler handles encoding, prefilling, and decoding in a single process.

Workflow

The DecodeWorkerHandler receives multimodal requests with image/video URLs and passes them directly to SGLang’s engine. SGLang’s internal mm_data_processor handles image/video fetching, loading, encoding, and token expansion.

Launch

cd $DYNAMO_HOME/examples/backends/sglang
./launch/agg_vision.sh --model-path Qwen/Qwen2-VL-7B-Instruct

Client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 50,
    "stream": false
  }' | jq

Video requests use the same aggregated path:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe the video in detail"
          },
          {
            "type": "video_url",
            "video_url": {
              "url": "https://samplelib.com/mp4/sample-5s.mp4"
            }
          }
        ]
      }
    ],
    "max_tokens": 50,
    "stream": false
  }' | jq

EP/D or P/D Serving (No Separate Encode Worker)

Components

• workers:
  • PrefillWorkerHandler receives raw multimodal metadata and lets SGLang perform media loading, encoding, token expansion, and KV production during prefill.
  • DecodeWorkerHandler receives matching multimodal metadata so token layout stays aligned with the transferred KV cache.

Workflow

The Rust frontend tokenizes the request and forwards image/video URLs as multi_modal_data. There is no encode worker. The prefill worker passes those URLs to SGLang’s normal multimodal engine path, so the vision context is produced inside the prefill worker. The decode worker also passes the same URLs to SGLang so the tokenizer manager can reproduce the multimodal token layout while consuming KV cache from prefill.

This native EP/D or P/D path intentionally trades simplicity for duplicated raw-media preprocessing: both prefill and decode call SGLang with image_data/video_data, so SGLang’s multimodal processor may fetch/load/preprocess the same media twice. Use the E/PD or E/P/D encode-worker topology when the deployment must preprocess media once and forward precomputed embeddings.

Launch

Native P/D and EP/D use the same launchers. The topology is selected by the model: text-only models run P/D, while VLMs run native EP/D where the prefill worker performs the media encode step. Neither path has a separate encode worker.

cd $DYNAMO_HOME/examples/backends/sglang
# P/D: text-only model, prefill on GPU 0 and decode on GPU 1.
./launch/disagg.sh --model Qwen/Qwen3-0.6B
# EP/D: VLM, same launcher; prefill performs media encoding and decode runs separately.
./launch/disagg.sh --model Qwen/Qwen3-VL-4B-Instruct
# Single-GPU smoke test: same EP/D behavior with prefill and decode packed together.
./launch/disagg_same_gpu.sh --model Qwen/Qwen3-VL-4B-Instruct

These launchers pass --enable-multimodal to the prefill and decode workers but deliberately do not pass --dedicated-mm-encoder.

E/PD Serving (Encode Separate)

Components

• workers:
  • MultimodalEncodeWorkerHandler for image encoding and embeddings generation
  • MultimodalWorkerHandler for prefilling and decoding.

Workflow

The Rust frontend tokenizes the request and extracts image URLs into multi_modal_data. The MultimodalEncodeWorker receives the pre-tokenized request, downloads and encodes the image, and passes the embeddings to the MultimodalWorker. The work complete event is sent via NATS, while the embeddings tensor is transferred via RDMA through the NIXL interface. The MultimodalWorker then prefills and decodes the prompt in the same engine, as in the LLM aggregated serving example. Only the encode worker is registered to the Dynamo frontend as an available endpoint. The PD worker does NOT register - it is an internal component and communicates via NATS.

Launch

cd $DYNAMO_HOME/examples/backends/sglang
./launch/multimodal_epd.sh

Client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 50,
    "stream": false
  }' | jq

E/P/D Serving (Full Disaggregation)

Components

• workers:
  • MultimodalEncodeWorkerHandler for image encoding and embeddings generation
  • MultimodalWorkerHandler for decoding
  • MultimodalPrefillWorkerHandler for prefilling

Workflow

In models like Qwen2.5-VL, embeddings are only required during the prefill stage. The Rust frontend tokenizes and extracts image URLs. The MultimodalEncodeWorker receives the pre-tokenized request, encodes images, and transfers embeddings via NIXL to the Decode Worker (the entry point for disaggregation), which then coordinates with the Prefill Worker. The Prefill Worker processes the embeddings and forwards the KV cache back to the Decode Worker for token generation.

Launch

cd $DYNAMO_HOME/examples/backends/sglang
./launch/multimodal_disagg.sh

Client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 50,
    "stream": false
  }' | jq

Bootstrap Coordination

SGLang disaggregation uses a bootstrap mechanism for P->D coordination:

Request Flow (Important)

Client → Frontend → Processor → Encode → DECODE Worker → Prefill Worker
                                               ↑
                                    Entry point for disaggregation!

Bootstrap Process

1. Decode Worker receives request from Encode Worker
2. Decode Worker calls Prefill Worker via NATS to request bootstrap info
3. Prefill Worker generates {host, port, room} and returns immediately
4. Both workers connect to same “room” using bootstrap coordinates
5. SGLang internally transfers KV cache state via bootstrap connection (not NIXL)

Key Difference from vLLM

• vLLM: Frontend → Prefill → Decode (Prefill is entry point)
• SGLang: Frontend → Processor → Encode → Decode → Prefill (Decode is entry point)

Inter-Component Communication

Control Flow (NATS)

All component-to-component communication happens via NATS:

E/PD Mode (Encode Separate)

Processor → Encode Worker → PD Worker
  (NATS)        (NATS + NIXL embeddings)

E/P/D Mode (Full Disaggregation)

Processor → Encode Worker → DECODE Worker → Prefill Worker
  (NATS)        (NATS)            (NATS)
                             ↓
                    Decode requests bootstrap
                             ↓
                    Prefill returns {host, port, room}
                             ↓
                    Both connect via bootstrap
                             ↓
                    SGLang internal KV cache transfer

Detailed Message Flow

Processor → Encode Worker:
  - NATS round_robin with SglangMultimodalRequest
  - Contains: tokenized input_ids, image URL, sampling params
Encode Worker → Decode/PD Worker:
  - NATS round_robin to "backend" component
  - Contains: expanded token_ids, NIXL metadata, embeddings shape
  - NIXL transfer: embeddings tensor
Decode Worker → Prefill Worker (disagg only):
  - NATS call to "prefill" component
  - Decode requests bootstrap coordinates
  - Prefill returns: {bootstrap_host, bootstrap_port, bootstrap_room}
Prefill ↔ Decode (via bootstrap):
  - SGLang internal connection (not NATS)
  - KV cache state shared via bootstrap mechanism

Data Transfer (NIXL)

NIXL is used only for embedding transfer:

# Encode Worker
descriptor = connect.Descriptor(precomputed_embeddings)
with connector.create_readable(descriptor) as readable:
    request.serialized_request = readable.metadata()
    await pd_worker_client.round_robin(request)
    await readable.wait_for_completion()
# PD Worker
embeddings = torch.empty(request.embeddings_shape, dtype=torch.float16)
descriptor = connect.Descriptor(embeddings)
read_op = await connector.begin_read(request.serialized_request, descriptor)
await read_op.wait_for_completion()

Vision Encoding Details

Encode Worker Components

The encode worker uses SGLang’s MMEncoder for model-agnostic vision encoding. MMEncoder handles vision model loading, image preprocessing, and feature extraction internally:

from sglang.srt.disaggregation.encode_server import MMEncoder
self.encoder = MMEncoder(
    server_args=config.server_args,
    dist_init_method="tcp://127.0.0.1:0",
    rank=0,
)
# At request time:
image_grid_dim, mm_embedding = await self.encoder._encode([image_url])

Token Expansion Process

1. Processor inserts single image token (e.g., <|image_pad|>)
2. Encode worker generates embeddings: shape = (batch, num_patches, hidden_dim)
3. Encode worker replaces single token with num_patches tokens
4. Downstream worker receives expanded token sequence

Example:

# Before: ["Hello", "<|image_pad|>", "world"]
# After:  ["Hello", "<|image_pad|>", "<|image_pad|>", ...(576 tokens), "world"]

Chat Template Processing

SGLang uses its own chat template system:

from sglang.srt.parser.conversation import chat_templates
conv = chat_templates["qwen2-vl"].copy()
conv.append_message(conv.roles[0], f"{conv.image_token} Describe this image")
processed = tokenizer(text=conv.get_prompt(), return_tensors="pt")

Supported templates: qwen2-vl, llama-3, vicuna, etc.

NIXL Usage

Key Difference: SGLang native EP/D or P/D uses bootstrap mechanism, not NIXL for KV cache like vLLM.

Environment Variables

SGLANG_ENCODER_MM_LOAD_WORKERS

Controls how many threads the encoder uses to fetch and load images concurrently. When a request contains multiple images (URLs, file paths, or base64 data), each image is loaded in a separate thread. Default is 4. Increase if image loading (network fetch or disk I/O) is the bottleneck rather than GPU compute. Has no effect if the vision encoder itself is the bottleneck, since encoding is sequential on GPU after all images are loaded.

# Example: allow up to 16 concurrent image loads per encoder
export SGLANG_ENCODER_MM_LOAD_WORKERS=16

Only applies to the EPD encode worker (which uses SGLang’s MMEncoder internally).

Profiling

Dynamo’s SGLang multimodal workers include NVTX markers for nsys profiling. They are disabled by default (zero overhead) and enabled by setting DYN_NVTX=1.

cd $DYNAMO_HOME/examples/backends/sglang
DYN_NVTX=1 nsys profile --trace=cuda,nvtx -o profile.nsys-rep \
  bash launch/multimodal_epd.sh ...

Key NVTX ranges emitted:

Known Limitations

• No Data URL support - Only HTTP/HTTPS URLs supported; data:image/... base64 URLs not supported
• No pre-computed embeddings - Cannot use .pt, .pth, .bin embedding files; vision encoder runs for every request
• No audio support - No audio encoder implementation
• Only Processor registers with Dynamo - Workers are internal components, frontend routes to Processor only
• Disaggregated routing - Decode Worker is the entry point (calls Prefill), cannot route directly to Prefill workers
• Limited model generalization - Token expansion logic is model-specific; adding new models may require implementation updates

Supported Models

SGLang multimodal only supports image-based vision-language models:

• Qwen2-VL / Qwen2.5-VL - Qwen/Qwen2.5-VL-7B-Instruct
• Qwen3-VL - Qwen/Qwen3-VL-30B-A3B-Instruct
• Models supported by SGLang’s MMEncoder

Key Files