CLI Templates

vLLM Local Deployment Examples

Copyable commands for running the vLLM launch scripts in the Dynamo repository.
View as Markdown

Clone the NVIDIA Dynamo repository before running an example. The launch scripts load supporting files through paths relative to examples/backends/vllm.

$git clone https://github.com/ai-dynamo/dynamo.git
$cd dynamo/examples/backends/vllm

Review the selected script and confirm its model, accelerator, and environment requirements before running the command. Pass supported flags after the script name or set its documented environment variables to override defaults.

Aggregated

Starts the frontend and one vLLM worker that handles both prefill and decode. The default configuration uses one GPU and is the baseline local text-generation deployment.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path; --unified uses the unified backend entry point.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg.sh

View the launch script.

Starts a vLLM pooling worker and exposes the OpenAI-compatible embeddings endpoint. Use it for vector search, retrieval, and other embedding workloads.

The default model is Qwen/Qwen3-Embedding-0.6B. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
MAX_MODEL_LEN2048Maximum model context length.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_embed.sh

View the launch script.

Starts two embedding workers behind one frontend. Use identical model arguments to test load balancing, or different models to test model-name-based request dispatch.

The script declares a GPU requirement of 2.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT1${DYN_SYSTEM_PORT:-8081}System and metrics port for the first worker.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
MAX_MODEL_LEN2048Maximum model context length.

Positional and pass-through arguments: Supply two model names. Any remaining arguments are forwarded to both vLLM workers.

$./launch/agg_embed_multiworker.sh \
>  Qwen/Qwen3-Embedding-0.6B \
>  Qwen/Qwen3-Embedding-0.6B

View the launch script.

Serves a multimodal model with media preprocessing in the Dynamo frontend and one vLLM worker. Use it as the baseline image or video deployment.

The default model is Qwen/Qwen3-VL-30B-A3B-Instruct-FP8.

Configuration

VariableDefaultDescription
DYN_MODEL_NAMEQwen/Qwen3-VL-30B-A3B-Instruct-FP8Model identifier or local model path loaded by the worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
CUDA_VISIBLE_DEVICES0GPU devices visible to the launched worker processes.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_multimodal.sh

View the launch script.

Expands media placeholders into routing tokens, hashes each media item, and routes requests by text and media cache overlap. This path uses the Rust multimodal router in the frontend.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
NAMESPACEmm-routerDynamo discovery namespace used by the frontend and workers.
HTTP_PORT${DYN_HTTP_PORT:-8000}Port used by the corresponding service or worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
BLOCK_SIZE16KV-cache block size; it must match the backend block size for KV routing.
GPU_MEMORY_UTILIZATION0.20Fraction of GPU memory available to each backend worker.
MAX_MODEL_LEN4096Maximum model context length.
NUM_WORKERS2Number of backend workers launched by the script.
SINGLE_GPUfalseWhen true, places all workers on GPU 0 for functional testing.
NATS_SERVERnats://127.0.0.1:4222NATS server URL used for Dynamo discovery and messaging.
ETCD_ENDPOINTShttp://127.0.0.1:2379etcd endpoint used for Dynamo discovery state.
VLLM_SYSTEM_PORT_BASE18081Starting system and metrics port for vLLM workers.
KV_EVENTS_PORT_BASE5557Starting port for per-worker KV event publishers.
DYN_LOGinfo,mm_routing=debug,dynamo_kv_router::scheduling=debug,dynamo_llm::kv_router=debugDynamo log filter used by the launched processes.
VLLM_EXTRA_ARGSEmptyAdditional arguments appended to each dynamo.vllm worker command.

Command-line flags: --model sets the model identifier or path; --num-workers sets the worker count; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_multimodal_router.sh

View the launch script.

Uses vLLM’s Python chat processor in the frontend to expand media tokens and compute media hashes for KV routing. The script launches the requested frontend and worker replicas, but requires NATS and etcd to be running before it starts.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
NAMESPACEdynamoDynamo discovery namespace used by the frontend and workers.
HTTP_PORT${DYN_HTTP_PORT:-8000}Port used by the corresponding service or worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
BLOCK_SIZE16KV-cache block size; it must match the backend block size for KV routing.
GPU_MEMORY_UTILIZATION0.40Fraction of GPU memory available to each backend worker.
MAX_MODEL_LEN4096Maximum model context length.
NUM_WORKERS2Number of backend workers launched by the script.
SINGLE_GPUfalseWhen true, places all workers on GPU 0 for functional testing.
NUM_FRONTENDS1Number of frontend replicas launched by the script.
NATS_SERVERnats://127.0.0.1:4222NATS server URL used for Dynamo discovery and messaging.
ETCD_ENDPOINTShttp://127.0.0.1:2379etcd endpoint used for Dynamo discovery state.
VLLM_SYSTEM_PORT_BASE18081Starting system and metrics port for vLLM workers.
KV_EVENTS_PORT_BASE29080Starting port for per-worker KV event publishers.
DYN_MM_IMAGE_CACHE_SIZE32Frontend multimodal image-cache capacity.
DYNAMO_MM_TRANSFERshmTransport used to transfer preprocessed multimodal data.
VLLM_EXTRA_ARGSEmptyAdditional arguments appended to each dynamo.vllm worker command.
FRONTEND_EXTRA_ARGSEmptyAdditional arguments appended to each frontend command.
FRONTEND_SYSTEM_PORT_BASE9080Starting system port for frontend replicas.

Command-line flags: --model sets the model identifier or path; --num-workers sets the worker count; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_multimodal_router_chat_processor.sh

View the launch script.

Starts the baseline worker through the request-plane mode selected by the script flags. Use it to compare or validate the available frontend-to-worker transports.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --tcp uses the TCP request plane (the default); --nats uses the NATS request plane.

$./launch/agg_request_planes.sh

View the launch script.

Starts two workers behind the KV router and publishes cache events over ZeroMQ (ZMQ) for prefix-aware placement. The default topology uses two GPUs.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
$./launch/agg_router.sh

View the launch script.

Runs the KV router in approximate mode, which predicts cache placement without consuming backend KV events. Use it when event publishing is unavailable.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
$./launch/agg_router_approx.sh

View the launch script.

Starts two frontend and KV-router replicas on separate ports while sharing routing state. Use it to exercise multiple ingress replicas against the same worker pool.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT_R18000HTTP port for the first frontend replica.
DYN_HTTP_PORT_R28001HTTP port for the second frontend replica.
DYN_SYSTEM_PORT_R18091System port for the first frontend or router replica.
DYN_SYSTEM_PORT_R28092System port for the second frontend or router replica.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
$./launch/agg_router_replicas.sh

View the launch script.

Runs meta-llama/Meta-Llama-3.1-8B-Instruct with the yuhuili/EAGLE3-LLaMA3.1-Instruct-8B draft model on one GPU. The worker uses the eagle3 method and proposes two speculative tokens per step.

The default model is meta-llama/Meta-Llama-3.1-8B-Instruct.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/agg_spec_decoding.sh

View the launch script.

Enables tracing on the frontend and worker so requests appear in the local observability stack. Start the tracing stack before running this example.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_tracing.sh

View the launch script.

Starts multiple data-parallel workers for a Mixture-of-Experts model and lets Dynamo route among the data-parallel ranks. The default Qwen configuration uses four GPUs.

The default model is Qwen/Qwen3-30B-A3B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/dep.sh

View the launch script.

Runs DeepSeek-R1 with data-parallel attention and expert parallelism across multiple nodes. Each node starts its assigned data-parallel ranks and joins the same distributed worker group.

The default model is deepseek-ai/DeepSeek-R1.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.

Command-line flags: --num-nodes sets the total node count; --node-rank sets this node’s rank; --gpus-per-node sets the GPU count on each node; --master-addr sets the distributed rendezvous address; --log-dir sets the log directory; --model sets the model identifier or path.

Replace 192.0.2.10 with the head node’s reachable IP address. Run on the head node:

$./launch/dsr1_dep.sh \
>  --num-nodes 2 \
>  --node-rank 0 \
>  --gpus-per-node 8 \
>  --master-addr 192.0.2.10

Run on the second node:

$./launch/dsr1_dep.sh \
>  --num-nodes 2 \
>  --node-rank 1 \
>  --gpus-per-node 8 \
>  --master-addr 192.0.2.10

View the launch script.

Runs one tensor-parallel vLLM worker across a head node and a worker node using vLLM multiprocessing. Use it when the tensor-parallel size exceeds the GPUs available on one host.

The default model is meta-llama/Llama-3.1-8B-Instruct. The script requires eight GPUs on each of two nodes, with NATS and etcd reachable from the head node.

Configuration

VariableDefaultDescription
MODELmeta-llama/Llama-3.1-8B-InstructModel identifier or local model path loaded by the worker.
TENSOR_PARALLEL_SIZE16Tensor-parallel degree used by the backend.
NNODES2Number of nodes participating in the tensor-parallel deployment.

Command-line flags: --head starts the head-node process; --worker starts a worker-node process; --head-ip sets the reachable head-node IP address.

Replace 192.0.2.10 with the head node’s reachable IP address. Run on the head node:

$./launch/multi_node_tp.sh --head --head-ip 192.0.2.10

Run the worker command on the second node:

$./launch/multi_node_tp.sh --worker --head-ip 192.0.2.10

View the launch script.

Disaggregated

Runs prefill and decode in separate vLLM workers and transfers KV cache through NIXL. Use --unified to exercise the unified worker entry point with the same topology.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --unified uses the unified backend entry point.

$./launch/disagg.sh

View the launch script.

Runs media encoding separately from a worker that performs both prefill and decode. Use it when encoding should scale independently but language-model stages can remain together.

The default model is Qwen/Qwen3-VL-30B-A3B-Instruct-FP8.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_ENCODE_WORKER_GPU1GPU for the encode worker; --single-gpu changes the default to 0.
DYN_PD_WORKER_GPU2GPU for the combined prefill/decode worker; --single-gpu changes the default to 0.
DYN_ENCODE_GPU_MEM0.9Encode-worker GPU-memory fraction; --single-gpu changes the default to 0.1.
DYN_PD_GPU_MEM0.9Prefill/decode GPU-memory fraction; --single-gpu changes the default to 0.7.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/disagg_multimodal_e_pd.sh

View the launch script.

Splits multimodal inference into dedicated encode, prefill, and decode workers. The script offers single-GPU and two-GPU modes for functional testing in addition to the full layout.

The default model is llava-hf/llava-1.5-7b-hf.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_ENCODE_WORKER_GPU0GPU for the encode worker in the default three-GPU layout.
DYN_PREFILL_WORKER_GPU1GPU for the prefill worker; packed modes change this default to 0.
DYN_DECODE_WORKER_GPU2GPU for the decode worker; --two-gpu changes this to 1 and --single-gpu to 0.
DYN_ENCODE_GPU_MEM0.9Encode-worker GPU-memory fraction; packed modes change the default to 0.1.
DYN_PREFILL_GPU_MEM0.9Prefill-worker GPU-memory fraction; two- and one-GPU modes use 0.7 and 0.4.
DYN_DECODE_GPU_MEM0.9Decode-worker GPU-memory fraction; single-GPU mode changes the default to 0.4.
VLLM_NIXL_SIDE_CHANNEL_PORT_ENCODE20097NIXL side-channel port used by the encode worker.
VLLM_NIXL_SIDE_CHANNEL_PORT_PREFILL20098NIXL side-channel port used by the prefill worker.
VLLM_NIXL_SIDE_CHANNEL_PORT_DECODE20099NIXL side-channel port used by the decode worker.
VLLM_ZMQ_PORT_ENCODE20080ZeroMQ port used by the encode worker.
VLLM_ZMQ_PORT_PREFILL20081ZeroMQ port used by the prefill worker.
VLLM_ZMQ_PORT_DECODE20082ZeroMQ port used by the decode worker.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
DYN_SYSTEM_PORT38083System and metrics port for the third worker.

Command-line flags: --model sets the model identifier or path; --single-gpu places workers on one accelerator; --two-gpu places encode and prefill on GPU 0 and decode on GPU 1.

$./launch/disagg_multimodal_epd.sh

View the launch script.

Runs a two-worker multimodal topology in which the prefill worker also performs vision encoding. It uses fewer processes than encode-prefill-decode at the cost of more work on prefill.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_PREFILL_WORKER_GPU0GPU device assigned to the corresponding worker.
DYN_DECODE_WORKER_GPU1GPU device assigned to the corresponding worker.
DYN_PREFILL_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_DECODE_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_SYSTEM_PORT1${DYN_SYSTEM_PORT:-8081}System and metrics port for the first worker.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path; --single-gpu places workers on one accelerator.

$./launch/disagg_multimodal_p_d.sh

View the launch script.

Routes requests across separate prefill and decode pools using cache events from the workers. The frontend also discovers prefill workers and activates its internal prefill router.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_router.sh

View the launch script.

Runs the disaggregated KV-routing topology with the device settings required for Intel Gaudi accelerators.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_router_gaudi.sh

View the launch script.

Runs separate prefill and decode processes on the same GPU with reduced model-length and concurrency limits. Use it for functional testing of KV transfer without a multi-GPU host.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DEFAULT_KV_CACHE_BYTES1023525000Per-worker KV-cache capacity used when no profiler override is present.
CUDA_VISIBLE_DEVICES0GPU devices visible to the launched worker processes.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_CHAT_PROCESSOREmptyChat processor used by the frontend for multimodal preprocessing.
DYN_ROUTER_MODEEmptyOptional router mode passed to the frontend for this topology.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_VLLM_NIXL_SIDE_CHANNEL_PORT15600NIXL side-channel port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
DYN_VLLM_NIXL_SIDE_CHANNEL_PORT220097NIXL side-channel port for the second worker.
DYN_VLLM_KV_EVENT_PORT20081Port used by the vLLM KV event publisher.
$./launch/disagg_same_gpu.sh

View the launch script.

Enables tracing on the frontend, prefill worker, and decode worker so a request can be followed across the full disaggregated path. Start the tracing stack first.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path.

$./launch/disagg_tracing.sh

View the launch script.

KV Cache Offloading

Aggregated

Enables FlexKV on a worker that handles both prefill and decode. Use it to test external KV-cache storage without a router.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/agg_flexkv.sh

View the launch script.

Combines FlexKV with KV-aware routing so the frontend places requests by cache locality while workers offload cache blocks.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/agg_flexkv_router.sh

View the launch script.

Enables the KV Block Manager (KVBM) on a single worker and configures CPU-backed cache capacity. Use it to test local KV-cache offloading.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/agg_kvbm.sh

View the launch script.

Starts multiple KVBM-enabled workers behind the KV router. Each worker uses distinct event and coordination ports so the router can track its cache state.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/agg_kvbm_router.sh

View the launch script.

Connects one vLLM worker to LMCache through the in-process connector. Use it as the baseline LMCache deployment.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
$./launch/agg_lmcache.sh

View the launch script.

Starts an LMCache server as a separate process and connects the vLLM worker through the multiprocess connector. Use it to isolate cache management from the inference worker.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
LMCACHE_PORT5555LMCache data-plane port.
LMCACHE_HTTP_PORT8080LMCache HTTP administration port.
LMCACHE_L1_SIZE_GB16LMCache L1 cache capacity, in GiB.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
$./launch/agg_lmcache_mp.sh

View the launch script.

Runs the in-process LMCache connector while configuring a Prometheus multiprocess directory. Use it to validate metric collection in a multi-process-style environment.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
PROMETHEUS_MULTIPROC_DIR/tmp/prometheus_multiproc_$$_$RANDOMDirectory used by Prometheus multiprocess metrics.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
$./launch/agg_lmcache_multiproc.sh

View the launch script.

Disaggregated

Runs separate prefill and decode workers and enables FlexKV on prefill. Use it to test offloading cache created during prompt processing before transfer to decode.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_flexkv.sh

View the launch script.

Runs separate prefill and decode workers with KVBM enabled on prefill and CPU cache storage. Decode receives the transferred KV state without enabling KVBM.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_kvbm.sh

View the launch script.

Scales the KVBM topology to two prefill and two decode workers behind the KV router. Distinct barrier identifiers keep the prefill workers’ offload coordination separate.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_kvbm_2p2d.sh

View the launch script.

Combines separate prefill and decode pools, KV-aware routing, and KVBM-backed offloading. Use it to test cache placement and offload behavior together.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_kvbm_router.sh

View the launch script.

Runs separate prefill and decode workers and enables LMCache on prefill. Use it to test LMCache-backed prompt-cache offloading in a disaggregated topology.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/disagg_lmcache.sh

View the launch script.

vLLM-Omni

Aggregated

Starts a vLLM-Omni worker for audio generation. Override the default model and generation settings through the script flags.

The default model is Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_omni_audio.sh

View the launch script.

Starts a vLLM-Omni image-to-video pipeline with a Wan model by default. Pass a different model or worker arguments through the script options.

The default model is Wan-AI/Wan2.2-TI2V-5B-Diffusers.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_omni_i2v.sh

View the launch script.

Starts a vLLM-Omni image-generation worker and exposes the generation endpoint. Use the script flags to select the model.

The default model is Qwen/Qwen-Image.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_omni_image.sh

View the launch script.

Starts a vLLM-Omni video-generation worker and exposes the generation endpoint. Use the script flags to select the model and generation settings.

The default model is Wan-AI/Wan2.1-T2V-1.3B-Diffusers.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_omni_video.sh

View the launch script.

Disaggregated

Splits GLM-Image across an autoregressive stage and a diffusion stage on separate GPUs. A router coordinates the intermediate token output and final image response.

The default model is zai-org/GLM-Image.

Configuration

VariableDefaultDescription
MODELzai-org/GLM-ImageModel identifier or local model path loaded by the worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_NAMESPACEEmptyDynamo discovery namespace used by the frontend and workers.

Command-line flags: --model sets the model identifier or path; --stage-configs-path sets the diffusion stage-configuration file.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/disagg_omni_glm_image.sh

View the launch script.

LoRA

Aggregated

Starts one vLLM worker with Low-Rank Adaptation (LoRA) support and loads adapters from the configured object store.

The script expects MinIO at http://localhost:9000 and loads adapters from the my-loras bucket. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
$./launch/lora/agg_lora.sh

View the launch script.

Starts LoRA-capable workers behind the KV router. Use it to test adapter selection together with prefix-cache-aware request placement.

The script expects MinIO at http://localhost:9000 and loads adapters from the my-loras bucket. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/lora/agg_lora_router.sh

View the launch script.

Serves a multimodal model with dynamically selected LoRA adapters. Use it for image-and-text requests that require adapter-specific behavior.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
DYN_MODEL_NAMEQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
DYN_LORA_PATH/tmp/dynamo_loras_multimodalDirectory containing LoRA adapter subdirectories.
DYN_MAX_LORA_RANK64Maximum LoRA rank accepted by the worker.
CUDA_VISIBLE_DEVICES0GPU devices visible to the launched worker processes.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/lora/multimodal/lora_agg.sh

View the launch script.

XPU

Aggregated

Runs LoRA-capable vLLM workers on Intel XPU devices behind the KV router.

The script expects MinIO at http://localhost:9000 and loads adapters from the my-loras bucket. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/lora/xpu/agg_lora_router_xpu.sh

View the launch script.

Runs one LoRA-capable vLLM worker on an Intel XPU device.

The script expects MinIO at http://localhost:9000 and loads adapters from the my-loras bucket. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.
$./launch/lora/xpu/agg_lora_xpu.sh

View the launch script.

Runs LMCache on Intel XPU while configuring the Prometheus multiprocess directory used by multi-process deployments.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
PROMETHEUS_MULTIPROC_DIR/tmp/prometheus_multiproc_$$_$RANDOMDirectory used by Prometheus multiprocess metrics.
ZE_AFFINITY_MASK0Intel XPU devices visible to the launched worker process.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.
$./launch/xpu/agg_lmcache_multiproc_xpu.sh

View the launch script.

Connects a vLLM worker on Intel XPU to LMCache for KV-cache offloading.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
ZE_AFFINITY_MASK0Intel XPU devices visible to the launched worker process.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.
$./launch/xpu/agg_lmcache_xpu.sh

View the launch script.

Uses vLLM’s Python chat processor on the frontend and routes multimodal requests to workers on Intel XPU devices. The script requires NATS and etcd to be running before it starts.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
NAMESPACEdynamoDynamo discovery namespace used by the frontend and workers.
HTTP_PORT${DYN_HTTP_PORT:-8000}Port used by the corresponding service or worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
BLOCK_SIZE16KV-cache block size; it must match the backend block size for KV routing.
GPU_MEMORY_UTILIZATION0.40Fraction of GPU memory available to each backend worker.
MAX_MODEL_LEN4096Maximum model context length.
NUM_WORKERS2Number of backend workers launched by the script.
SINGLE_GPUfalseWhen true, places all workers on GPU 0 for functional testing.
NUM_FRONTENDS1Number of frontend replicas launched by the script.
NATS_SERVERnats://127.0.0.1:4222NATS server URL used for Dynamo discovery and messaging.
ETCD_ENDPOINTShttp://127.0.0.1:2379etcd endpoint used for Dynamo discovery state.
VLLM_SYSTEM_PORT_BASE18081Starting system and metrics port for vLLM workers.
KV_EVENTS_PORT_BASE29080Starting port for per-worker KV event publishers.
DYN_MM_IMAGE_CACHE_SIZE32Frontend multimodal image-cache capacity.
DYNAMO_MM_TRANSFERshmTransport used to transfer preprocessed multimodal data.
VLLM_EXTRA_ARGSEmptyAdditional arguments appended to each dynamo.vllm worker command.
FRONTEND_EXTRA_ARGSEmptyAdditional arguments appended to each frontend command.
ZE_AFFINITY_MASKNoneComma-separated XPU assignments; required when NUM_WORKERS is greater than 1.
DYN_SYSTEM_PORT118081System and metrics port for the first worker.
FRONTEND_SYSTEM_PORT_BASE9080Starting system port for frontend replicas.

Command-line flags: --model sets the model identifier or path; --num-workers sets the worker count; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/xpu/agg_multimodal_router_chat_processor_xpu.sh

View the launch script.

Uses the Rust multimodal routing path to place image and text requests across vLLM workers on Intel XPU devices.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
NAMESPACElightseek-pocDynamo discovery namespace used by the frontend and workers.
HTTP_PORT${DYN_HTTP_PORT:-8000}Port used by the corresponding service or worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
BLOCK_SIZE16KV-cache block size; it must match the backend block size for KV routing.
GPU_MEMORY_UTILIZATION0.20Fraction of GPU memory available to each backend worker.
MAX_MODEL_LEN4096Maximum model context length.
NUM_WORKERS2Number of backend workers launched by the script.
SINGLE_GPUfalseWhen true, places all workers on GPU 0 for functional testing.
NATS_SERVERnats://127.0.0.1:4222NATS server URL used for Dynamo discovery and messaging.
ETCD_ENDPOINTShttp://127.0.0.1:2379etcd endpoint used for Dynamo discovery state.
VLLM_SYSTEM_PORT_BASE18081Starting system and metrics port for vLLM workers.
KV_EVENTS_PORT_BASE5557Starting port for per-worker KV event publishers.
DYN_LOGinfo,lightseek_mm=debug,dynamo_kv_router::scheduling=debugDynamo log filter used by the launched processes.
VLLM_EXTRA_ARGSEmptyAdditional arguments appended to each dynamo.vllm worker command.
ZE_AFFINITY_MASKNoneComma-separated XPU assignments; required when NUM_WORKERS is greater than 1.
DYN_SYSTEM_PORT118081System and metrics port for the first worker.

Command-line flags: --model sets the model identifier or path; --num-workers sets the worker count; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/xpu/agg_multimodal_router_xpu.sh

View the launch script.

Serves multimodal requests with frontend media preprocessing and one vLLM worker on an Intel XPU device.

The default model is Qwen/Qwen3-VL-8B-Instruct.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
ZE_AFFINITY_MASK0Intel XPU devices visible to the launched worker process.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/xpu/agg_multimodal_xpu.sh

View the launch script.

Runs the selected frontend-to-worker request-plane transport with a vLLM worker on Intel XPU.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
ZE_AFFINITY_MASK0Intel XPU devices visible to the launched worker process.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.

Command-line flags: --tcp uses the TCP request plane (the default); --nats uses the NATS request plane.

$./launch/xpu/agg_request_planes_xpu.sh

View the launch script.

Uses approximate cache tracking to route requests across vLLM workers on Intel XPU without backend KV events.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
$./launch/xpu/agg_router_approx_xpu.sh

View the launch script.

Runs vLLM workers on Intel XPU behind the KV router and publishes cache events for prefix-aware placement.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
$./launch/xpu/agg_router_xpu.sh

View the launch script.

Starts the frontend and one vLLM worker on an Intel XPU device. Use it as the baseline XPU text-generation deployment.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
ZE_AFFINITY_MASK0Intel XPU devices visible to the launched worker process.
MAX_MODEL_LEN4096Maximum model context length.
MAX_CONCURRENT_SEQS2Maximum number of sequences scheduled concurrently.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
BLOCK_SIZE64KV-cache block size; it must match the backend block size for KV routing.

Command-line flags: --model sets the model identifier or path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/xpu/agg_xpu.sh

View the launch script.

Disaggregated

Splits multimodal inference into encode, prefill, and decode workers on Intel XPU devices, with a single-device mode for functional testing.

The default model is llava-hf/llava-1.5-7b-hf.

Configuration

VariableDefaultDescription
DEVICE_PLATFORMcudaAccelerator platform used to choose the worker device mapping.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_ENCODE_WORKER_GPU0GPU device assigned to the corresponding worker.
DYN_PREFILL_WORKER_GPU1GPU device assigned to the corresponding worker.
DYN_DECODE_WORKER_GPU2GPU device assigned to the corresponding worker.
DYN_ENCODE_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_PREFILL_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_DECODE_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.

Command-line flags: --model sets the model identifier or path; --single-gpu places workers on one accelerator.

$./launch/xpu/disagg_multimodal_epd_xpu.sh

View the launch script.

Runs separate prefill and decode pools on Intel XPU with KV-aware routing and direct device-memory transfer between workers.

The default model is Qwen/Qwen3-0.6B.

Configuration: This script does not define user-facing environment-variable overrides.

$./launch/xpu/disagg_router_xpu_gdr.sh

View the launch script.

Source

Browse all vLLM launch scripts in the Dynamo repository.