Feature GuidesMultimodal

vLLM Multimodal

View as Markdown

This document provides a comprehensive guide for multimodal inference using the vLLM backend in Dynamo.

Security Requirement: All multimodal workers require the --enable-multimodal flag to be explicitly set at startup. This prevents unintended processing of multimodal data from untrusted sources. Media requests are rejected when the flag is absent, and workers configured with a multimodal role fail at startup. This flag is analogous to --enable-mm-embeds in vLLM serve but also extends it to all multimodal content (URL, embeddings, and base64 data).

Support Matrix

ModalityAggregatedP/DSeparate encode worker
ImageYesYesLegacy entry point only
VideoYesYesProcessed by the language-model worker
AudioYesYes, with decode reloadNot routed to the separate encoder

Supported URL Formats

FormatExampleDescription
HTTP/HTTPShttp://example.com/image.jpgRemote media files
Data URLdata:image/jpeg;base64,/9j/4AAQ...Base64-encoded inline data

Deployment Patterns

The main multimodal vLLM launchers in this repo are:

PatternDeviceLaunch scriptUnified selectionBest for
AggregatedCUDAagg_multimodal.sh--unifiedSimplest image/video serving from one worker
AggregatedXPUxpu/agg_multimodal_xpu.shNoImage/video serving on XPU devices
P/DCUDAdisagg_multimodal_p_d.sh--unifiedPrefill/decode separation without a dedicated encoder
E/PD (Encode + PD)CUDAdisagg_multimodal_e_pd.shNoSeparate encoder and embedding-cache workflows
E/P/D (Full Disaggregation)CUDAdisagg_multimodal_epd.shNoSeparate encode, prefill, and decode workers

Custom Vision Encoders

The legacy aggregated vLLM worker can load an author-provided vision tower in process, batch images across concurrent requests, and splice the resulting embeddings into the language-model prompt. See Custom Vision Encoders for the backend contract, launch instructions, batch sizing guidance, and current limitations.

Image/Video Serving

Dynamo supports multimodal image and video requests for Vision Language Models (VLMs). Qwen/Qwen3-VL-2B-Instruct is a good example because the same model can handle both image_url and video_url requests through the standard OpenAI chat endpoint.

Aggregated Serving

Use the single-worker aggregated launcher for the simplest image/video setup:

$cd $DYNAMO_HOME/examples/backends/vllm
$
$# GPU deployment
$bash launch/agg_multimodal.sh --model Qwen/Qwen3-VL-2B-Instruct
$
$# Unified backend
$bash launch/agg_multimodal.sh --unified --model Qwen/Qwen3-VL-2B-Instruct
$
$# XPU deployment
$bash launch/xpu/agg_multimodal_xpu.sh --model Qwen/Qwen3-VL-2B-Instruct

Image request:

$curl http://localhost:8000/v1/chat/completions \
>  -H "Content-Type: application/json" \
>  -d '{
>      "model": "Qwen/Qwen3-VL-2B-Instruct",
>      "messages": [
>        {
>          "role": "user",
>          "content": [
>            {
>              "type": "text",
>              "text": "What is in this image?"
>            },
>            {
>              "type": "image_url",
>              "image_url": {
>                "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
>              }
>            }
>          ]
>        }
>      ],
>      "max_tokens": 64,
>      "temperature": 0.0,
>      "stream": false
>    }'

Video request:

$curl http://localhost:8000/v1/chat/completions \
>  -H "Content-Type: application/json" \
>  -d '{
>      "model": "Qwen/Qwen3-VL-2B-Instruct",
>      "messages": [
>        {
>          "role": "user",
>          "content": [
>            {
>              "type": "text",
>              "text": "Describe the video in detail"
>            },
>            {
>              "type": "video_url",
>              "video_url": {
>                "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4"
>              }
>            }
>          ]
>        }
>      ],
>      "max_tokens": 64,
>      "stream": false
>    }' | jq

P/D Serving

Use the P/D launcher to separate prefill and decode without deploying a dedicated multimodal encoder:

$cd $DYNAMO_HOME/examples/backends/vllm
$
$# Legacy vLLM worker path
$bash launch/disagg_multimodal_p_d.sh --model Qwen/Qwen3-VL-2B-Instruct
$
$# Unified vLLM worker path
$bash launch/disagg_multimodal_p_d.sh --unified \
>  --model Qwen/Qwen3-VL-2B-Instruct

For Qwen-VL images, prefill sends grid and embedding-shape metadata so decode can construct schema-valid placeholder embeddings and initialize mRoPE. Other model families use the expanded prompt token IDs produced during prefill.

The P/D handoff does not carry video embeddings. Video and audio inputs are loaded again on the decode worker. This preserves current behavior but adds media download and processing work. Mixed image-and-video P/D requests retain the same model-specific limitations as the legacy vLLM path.

Unified vLLM Backend

Pass --unified to the aggregated or P/D launchers to run python -m dynamo.vllm.unified_main. The unified path supports HTTP URLs, data URLs, frontend-decoded images, mm_processor_kwargs, frontend-provided multimodal hashes, and Kimi-style vision_chunk inputs.

The Python vLLM frontend can pre-render multimodal processor inputs and send them to an aggregated unified worker. Shared memory is the same-node default; NIXL supports the transfer channel used by cross-node deployments:

$# Same-node shared-memory transfer
$DYN_CHAT_PROCESSOR=vllm DYNAMO_MM_TRANSFER=shm \
>  bash launch/agg_multimodal.sh --unified \
>  --model Qwen/Qwen3-VL-2B-Instruct
$
$# NIXL transfer
$DYN_CHAT_PROCESSOR=vllm DYNAMO_MM_TRANSFER=nixl \
>  bash launch/agg_multimodal.sh --unified \
>  --model Qwen/Qwen3-VL-2B-Instruct

The frontend includes the original media references when transfer preparation is unavailable or partial. Fully transferred requests omit those references to avoid duplicating large inline data URIs in the backend payload. A receiver-side failure after a full transfer does not currently have a raw-media fallback.

P/D prefill deliberately uses the original media because it still needs raw-media-derived metadata for the decode handoff.

The unified vLLM entry point does not provide a separate Encode worker and rejects both --disaggregation-mode encode and --route-to-encoder. Use the legacy E/PD or E/P/D launchers when a dedicated encoder is required.

E/PD Serving (Encode + PD)

Use disagg_multimodal_e_pd.sh when you want a separate encode worker and a combined prefill/decode worker. This path is primarily useful for image-centric workloads and embedding-cache experiments.

When a separate encode worker is deployed with the current vLLM path, only image_url inputs are routed to it. video_url inputs are still processed on the combined PD worker.

$cd $DYNAMO_HOME/examples/backends/vllm
$
$# Multi-GPU deployment
$bash launch/disagg_multimodal_e_pd.sh --model Qwen/Qwen3-VL-2B-Instruct
$
$# Single-GPU (functional testing with small models)
$bash launch/disagg_multimodal_e_pd.sh --model Qwen/Qwen3-VL-2B-Instruct --single-gpu

E/P/D Serving (Full Disaggregation)

Use disagg_multimodal_epd.sh when you want separate encode, prefill, and decode workers for multimodal workloads.

In the current vLLM implementation, the separate encode worker is only used for image_url inputs. video_url inputs are still processed on the prefill worker, not on the encode worker.

$cd $DYNAMO_HOME/examples/backends/vllm
$
$# Multi-GPU deployment
$bash launch/disagg_multimodal_epd.sh --model Qwen/Qwen3-VL-2B-Instruct
$
$# Single-GPU (functional testing with small models)
$bash launch/disagg_multimodal_epd.sh --model Qwen/Qwen3-VL-2B-Instruct --single-gpu

Audio Serving

Dynamo supports audio_url requests for audio-capable models. Audio is loaded by the backend worker via vLLM’s AudioMediaIO at native sample rate — vLLM’s model-specific processor handles resampling and feature extraction internally. Omni models can handle image_url, video_url, and audio_url in the same request.

Aggregated Serving

Use the same aggregated multimodal launcher with an audio-capable model:

$pip install 'vllm[audio]'  # installs librosa and other audio dependencies
$cd $DYNAMO_HOME/examples/backends/vllm
$
$# GPU deployment
$bash launch/agg_multimodal.sh --model Qwen/Qwen3-Omni-30B-A3B-Instruct
$
$# XPU deployment
$DYN_CHAT_PROCESSOR=vllm \
>  bash launch/xpu/agg_multimodal_xpu.sh --model Qwen/Qwen3-Omni-30B-A3B-Instruct

Audio request:

$curl http://localhost:8000/v1/chat/completions \
>  -H "Content-Type: application/json" \
>  -d '{
>      "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
>      "messages": [
>        {
>          "role": "user",
>          "content": [
>            {
>              "type": "text",
>              "text": "What sound is this?"
>            },
>            {
>              "type": "audio_url",
>              "audio_url": {
>                "url": "https://raw.githubusercontent.com/yuekaizhang/Triton-ASR-Client/main/datasets/mini_en/wav/1221-135766-0002.wav"
>              }
>            }
>          ]
>        }
>      ],
>      "max_tokens": 100,
>      "stream": false
>    }' | jq

Embedding Cache

Dynamo supports embedding cache in both aggregated and disaggregated settings:

SettingImplementationLaunch Script
AggregatedSupported via vLLM ECConnector in vLLM 0.17+agg_multimodal.sh (or with vllm serve directly)
Disaggregated encoderDynamo-managed cache in the worker layer on top of vLLM enginedisagg_multimodal_e_pd.sh

Aggregated Worker

A single vLLM instance caches encoded embeddings on CPU so repeated images skip encoding entirely. Supported natively with vLLM 0.17+.

Launch with Dynamo:

$bash examples/backends/vllm/launch/agg_multimodal.sh \
>    --unified \
>    --model Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
>    --multimodal-embedding-cache-capacity-gb 10

Both dynamo.vllm and dynamo.vllm.unified_main automatically configure ec_both mode with DynamoMultimodalEmbeddingCacheConnector when capacity is greater than zero. A capacity of zero disables the CPU cache. Frontend-provided multimodal hashes are reused as cache identities so routing and embedding-cache lookups agree.

Launch with vllm serve (standalone, no Dynamo):

$vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
>    --ec-transfer-config "{
>        \"ec_role\": \"ec_both\",
>        \"ec_connector\": \"DynamoMultimodalEmbeddingCacheConnector\",
>        \"ec_connector_module_path\": \"dynamo.vllm.multimodal_utils.multimodal_embedding_cache_connector\",
>        \"ec_connector_extra_config\": {\"multimodal_embedding_cache_capacity_gb\": 10}
>    }"

The multimodal_embedding_cache_capacity_gb parameter controls the CPU-side LRU cache size in GB (0 = disabled). Requires vLLM 0.17+.

Disaggregated Encoder (Embedding Cache in Prefill Worker)

In the disaggregated setting, the Prefill Worker (P) owns a CPU-side LRU embedding cache (EmbeddingCacheManager). On each request P checks the cache first — on a hit, the Encode Worker is skipped entirely. On a miss, P routes to the Encode Worker (E), receives embeddings via NIXL, saves them to the cache, and then feeds the embeddings along with the request into the vLLM Instance for prefill.

Launch:

$cd $DYNAMO_HOME/examples/backends/vllm
$bash launch/disagg_multimodal_e_pd.sh --multimodal-embedding-cache-capacity-gb 10

Client: Use the same image_url request format shown in Aggregated Serving.

LoRA Adapters on Multimodal Workers

Multimodal workers support dynamic loading and unloading of LoRA adapters at runtime via the management API. This enables serving fine-tuned multimodal models alongside the base model.

Loading a LoRA Adapter

Load an adapter on a running multimodal worker via the load_lora endpoint:

$# For components workers (URI-based, requires DYN_LORA_ENABLED=true)
$curl -X POST http://<worker-host>:<port>/load_lora \
>  -H "Content-Type: application/json" \
>  -d '{
>    "lora_name": "my-vlm-adapter",
>    "source": {"uri": "s3://my-bucket/adapters/my-vlm-adapter"}
>  }'
$
$# For example workers (path-based)
$curl -X POST http://<worker-host>:<port>/load_lora \
>  -H "Content-Type: application/json" \
>  -d '{
>    "lora_name": "my-vlm-adapter",
>    "lora_path": "/path/to/adapter"
>  }'

Sending Requests with a LoRA

Set the model field in the request to the LoRA adapter name:

$curl -X POST http://<frontend-host>:<port>/v1/chat/completions \
>  -H "Content-Type: application/json" \
>  -d '{
>    "model": "my-vlm-adapter",
>    "messages": [
>      {"role": "user", "content": [
>        {"type": "text", "text": "Describe this image"},
>        {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
>      ]}
>    ]
>  }'

Requests without a LoRA name (or with the base model name) will use the base model.

Unloading a LoRA Adapter

$curl -X POST http://<worker-host>:<port>/unload_lora \
>  -H "Content-Type: application/json" \
>  -d '{"lora_name": "my-vlm-adapter"}'

Listing Loaded Adapters

$curl -X POST http://<worker-host>:<port>/list_loras

Disaggregated Mode

In disaggregated (prefill/decode) deployments, the same LoRA adapter must be loaded on both the prefill and decode workers. The LoRA identity (model field) is automatically propagated from the prefill worker to the decode worker in the forwarded request.

$# Load on prefill worker
$curl -X POST http://<prefill-worker>/load_lora \
>  -d '{"lora_name": "my-adapter", "source": {"uri": "s3://bucket/adapter"}}'
$
$# Load on decode worker (same adapter)
$curl -X POST http://<decode-worker>/load_lora \
>  -d '{"lora_name": "my-adapter", "source": {"uri": "s3://bucket/adapter"}}'

If a LoRA is loaded on the prefill worker but not on the decode worker, the decode worker will fall back to the base model for that request.

Supported Models

For a list of multimodal models supported by vLLM, see vLLM Supported Multimodal Models. Models listed there should generally work with aggregated serving, though they may not all be explicitly tested in this repo.