CLI Templates

SGLang Local Deployment Examples

Copyable commands for running the SGLang launch scripts in the Dynamo repository.
View as Markdown

Clone the NVIDIA Dynamo repository before running an example. The launch scripts load supporting files through paths relative to examples/backends/sglang.

$git clone https://github.com/ai-dynamo/dynamo.git
$cd dynamo/examples/backends/sglang

Review the selected script and confirm its model, GPU, and environment requirements before running the command. Pass supported flags after the script name to override its defaults.

Aggregated

Starts the frontend and one SGLang worker that handles both prefill and decode. The default configuration uses one GPU and is the baseline local text-generation deployment.

The default model is Qwen/Qwen3-0.6B. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model-path sets the model path; --enable-otel enables OpenTelemetry tracing; --unified uses the unified backend entry point.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg.sh

View the launch script.

Starts a tensor-parallel worker behind session-aware routing with KV-event tracking, sticky sessions, reasoning parsing, and tool-call parsing. The default model uses two GPUs.

The default model is zai-org/GLM-4.7-Flash. The default topology uses two GPUs because the model runs with --tp 2.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model-path sets the model path; --tp configures the corresponding launch option.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_agent.sh

View the launch script.

Starts an SGLang embedding worker and exposes the OpenAI-compatible embeddings endpoint. Use it for vector search, retrieval, and other embedding workloads.

The default model is Qwen/Qwen3-Embedding-4B. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model-path sets the model path.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_embed.sh

View the launch script.

Routes multimodal requests by their text and media-derived cache keys so repeated images can reuse cached work. This example requires the mm-routing build feature and a compatible SGLang build.

The default model is Qwen/Qwen3-VL-2B-Instruct.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-VL-2B-InstructModel identifier or local model path loaded by the worker.
NAMESPACEsglang-mm-routerDynamo discovery namespace used by the frontend and workers.
HTTP_PORT${DYN_HTTP_PORT:-8000}Port used by the corresponding service or worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
BLOCK_SIZE16KV-cache block size; it must match the backend block size for KV routing.
MAX_MODEL_LEN4096Maximum model context length.
NUM_WORKERS2Number of backend workers launched by the script.
SINGLE_GPUfalseWhen true, places all workers on GPU 0 for functional testing.
NATS_SERVERnats://127.0.0.1:4222NATS server URL used for Dynamo discovery and messaging.
ETCD_ENDPOINTShttp://127.0.0.1:2379etcd endpoint used for Dynamo discovery state.
SGLANG_SYSTEM_PORT_BASE18091Starting system and metrics port for SGLang workers.
KV_EVENTS_PORT_BASE29090Starting port for per-worker KV event publishers.
DYN_LOGinfo,mm_routing=debug,dynamo_kv_router::scheduling=debug,dynamo_llm::kv_router=debugDynamo log filter used by the launched processes.
SGLANG_EXTRA_ARGSEmptyAdditional arguments appended to each dynamo.sglang worker command.

Command-line flags: --model sets the model identifier or path; --num-workers sets the worker count; --single-gpu places workers on one accelerator.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_multimodal_router.sh

View the launch script.

Starts two workers behind the KV router and publishes cache events for prefix-aware request placement. The default topology uses two GPUs.

The default model is Qwen/Qwen3-0.6B. The script declares a GPU requirement of 2.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT_WORKER18081System and metrics port for the corresponding worker.
DYN_SYSTEM_PORT_WORKER28082System and metrics port for the corresponding worker.

Command-line flags: --enable-otel enables OpenTelemetry tracing; --approx configures the corresponding launch option.

$./launch/agg_router.sh

View the launch script.

Serves a vision-language model for image and video prompts. Pass --frontend-decoding to decode media in the Dynamo frontend and transfer the result to SGLang.

The default model is Qwen/Qwen3-VL-2B-Instruct. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
NIXL_PLUGIN_DIR$NIXL_WHEEL_LIBS/pluginsDirectory containing NIXL plugins.
DYN_SYSTEM_PORT8081System and metrics port for the worker.

Command-line flags: --model-path sets the model path; --chat-template sets the backend chat template; --page-size sets the KV-cache page size; --enable-otel enables OpenTelemetry tracing; --frontend-decoding moves media preprocessing into the Dynamo frontend.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/agg_vision.sh

View the launch script.

Disaggregated

Runs prefill and decode on separate workers so each stage can be scaled and tuned independently. The default topology uses two GPUs and transfers KV cache between the workers.

The default model is Qwen/Qwen3-0.6B. The script declares a GPU requirement of 2.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_DISAGG_BOOTSTRAP_PORT12345Bootstrap port used to establish disaggregated worker connections.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --unified uses the unified backend entry point; --enable-otel enables OpenTelemetry tracing.

$./launch/disagg.sh

View the launch script.

Starts two prefill and two decode workers with KV-aware routing across both pools. Use this four-GPU example to test scaling and cache-aware placement.

The default model is Qwen/Qwen3-0.6B. The script declares a GPU requirement of 4.

Configuration

VariableDefaultDescription
OTEL_EXPORTER_OTLP_TRACES_ENDPOINThttp://localhost:4317OpenTelemetry Protocol endpoint that receives traces.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.
DYN_SYSTEM_PORT38083System and metrics port for the third worker.
DYN_SYSTEM_PORT48084System and metrics port for the fourth worker.

Command-line flags: --enable-otel enables OpenTelemetry tracing.

$./launch/disagg_router.sh

View the launch script.

Runs separate prefill and decode processes on the same GPU with reduced memory limits. Use it for functional testing of the disaggregated path without a multi-GPU host.

The default model is Qwen/Qwen3-0.6B.

Configuration

VariableDefaultDescription
CONTEXT_LENGTH4096Context length configured for the model.
MAX_RUNNING_REQUESTS2Maximum number of requests run concurrently.
MAX_TOTAL_TOKENS25000TensorRT-LLM KV-cache token capacity per worker.
CUDA_VISIBLE_DEVICES0GPU devices visible to the launched worker processes.
DYN_DISAGG_BOOTSTRAP_PORT12345Bootstrap port used to establish disaggregated worker connections.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_CHAT_PROCESSOREmptyChat processor used by the frontend for multimodal preprocessing.
DYN_ROUTER_MODEEmptyOptional router mode passed to the frontend for this topology.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path; --single-gpu places workers on one accelerator.

$./launch/disagg_same_gpu.sh

View the launch script.

Splits vision encoding, language-model prefill, and decode into three workers. The default layout uses three GPUs; --single-gpu co-locates them for functional testing.

The default model is Qwen/Qwen2.5-VL-7B-Instruct.

Configuration

VariableDefaultDescription
DYN_ENCODE_WORKER_GPU0GPU for the encode worker in the default three-GPU layout.
DYN_PREFILL_WORKER_GPU1GPU for the prefill worker; --single-gpu changes the default to 0.
DYN_DECODE_WORKER_GPU2GPU for the decode worker; --single-gpu changes the default to 0.
DYN_ENCODE_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_PREFILL_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_DECODE_GPU_MEM0.9GPU-memory fraction assigned to the corresponding worker.
DYN_DISAGG_BOOTSTRAP_PORT12345Bootstrap port used to establish disaggregated worker connections.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path; --served-model-name sets the model name exposed by the API; --chat-template sets the backend chat template; --single-gpu places workers on one accelerator.

$./launch/multimodal_disagg.sh

View the launch script.

Runs vision encoding separately from a worker that performs both prefill and decode. The default layout uses two GPUs, with a single-GPU mode available for functional testing.

The default model is Qwen/Qwen2.5-VL-7B-Instruct. The default topology uses two GPUs; --single-gpu co-locates both workers on one GPU.

Configuration

VariableDefaultDescription
DYN_ENCODE_WORKER_GPU0GPU device assigned to the corresponding worker.
DYN_WORKER_GPU1GPU for the combined prefill/decode worker; --single-gpu changes the default to 0.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
DYN_SYSTEM_PORT18081System and metrics port for the first worker.
DYN_SYSTEM_PORT28082System and metrics port for the second worker.

Command-line flags: --model sets the model identifier or path; --served-model-name sets the model name exposed by the API; --chat-template sets the backend chat template; --single-gpu places workers on one accelerator.

$./launch/multimodal_epd.sh

View the launch script.

Diffusion

Aggregated

Serves LLaDA-style language models that generate text through iterative refinement instead of autoregressive decoding. The default configuration uses one GPU.

The default model is inclusionAI/LLaDA2.0-mini-preview. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
DLLM_ALGORITHMLowConfidenceDiffusion language-model decoding algorithm.
DLLM_ALGORITHM_CONFIGEmptyConfiguration passed to the diffusion decoding algorithm.
NAMESPACEdynamoDynamo discovery namespace used by the frontend and workers.
COMPONENTbackendDynamo component name used in the worker endpoint URI.
ENDPOINTgenerateDynamo endpoint name registered by the diffusion worker.
HTTP_PORT8001Port used by the corresponding service or worker.
TP_SIZE1Tensor-parallel degree used by the backend.
$./launch/diffusion_llada.sh

View the launch script.

Starts an image-diffusion worker and serves the generation endpoint. The default FLUX model requires one GPU with enough memory for the model.

The default model is black-forest-labs/FLUX.1-dev. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT${HTTP_PORT:-8000}HTTP port exposed by the Dynamo frontend.
HTTP_PORT8000Port used by the corresponding service or worker.

Command-line flags: --model-path sets the model path; --fs-url sets the generated-media storage URL; --http-url sets an optional base URL for serving generated images; --http-port sets the frontend HTTP port.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/image_diffusion.sh

View the launch script.

Starts a video-generation worker for Wan models. Select the model size and generation settings with the script flags; the 14B option uses two GPUs.

The default model is Wan-AI/Wan2.1-T2V-1.3B-Diffusers. The --wan-size 1b mode uses one GPU, while --wan-size 14b uses two GPUs.

Configuration

VariableDefaultDescription
DYN_HTTP_PORT${HTTP_PORT:-8000}HTTP port exposed by the Dynamo frontend.
HTTP_PORT8000Port used by the corresponding service or worker.

Command-line flags: --wan-size selects the 1B or 14B Wan model; --fs-url sets the media-storage URL; --http-port sets the frontend HTTP port; --num-frames, --height, and --width set video dimensions; --num-inference-steps sets the denoising step count.

Pass-through arguments: Additional unrecognized arguments are forwarded to the backend worker command.

$./launch/text-to-video-diffusion.sh

View the launch script.

LoRA

The LoRA example also requires the setup described in the launch/lora README.

Aggregated

Starts one SGLang worker with Low-Rank Adaptation (LoRA) support and loads adapters from the configured object store.

The script expects MinIO at http://localhost:9000 and loads adapters from the my-loras bucket. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

The default model is Qwen/Qwen3-0.6B. The script declares a GPU requirement of 1.

Configuration

VariableDefaultDescription
MODELQwen/Qwen3-0.6BModel identifier or local model path loaded by the worker.
LORA_NAMEcodelion/Qwen3-0.6B-accuracy-recovery-loraHugging Face Low-Rank Adaptation (LoRA) adapter repository to load.
DYN_SYSTEM_PORT8081System and metrics port for the worker.
DYN_HTTP_PORT8000HTTP port exposed by the Dynamo frontend.
$./launch/lora/agg_lora.sh

View the launch script.

Source

Browse all SGLang launch scripts in the Dynamo repository.