vLLM Configuration (DynamoVllmConfig) | NVIDIA Dynamo Documentation

DynamoVllmConfig holds the Dynamo-specific configuration for the vLLM backend (python -m dynamo.vllm). Every field, type, default, and choice on this page comes from the DynamoVllmArgGroup and DynamoVllmConfig definitions. For features and operational details, see the vLLM Reference Guide.

These are only the Dynamo wrapper flags. The vLLM backend also accepts every native vLLM EngineArgs argument (--model, --tensor-parallel-size, --max-model-len, and so on) in the same command, plus the cross-cutting Dynamo Runtime flags (--namespace, --endpoint, and others). Except for the KV event interoperability note below, this page covers neither — only the vLLM-specific DYN_VLLM_* surface.

How the config is loaded

Each field is both a CLI flag and an environment variable. The CLI flag takes precedence; the environment variable is the fallback. Boolean fields are negatable — --headless sets it on, --no-headless sets it off.

Kubernetes

Local

Set flags in the worker container’s args and environment variables in its env, under the vLLM worker service of a DynamoGraphDeployment (DGD). The operator passes both through to the process without validation.

1 spec:
2   services:
3     VllmDecodeWorker:
4       extraPodSpec:
5         mainContainer:
6           command:
7             - python3
8             - -m
9             - dynamo.vllm
10           args:
11             - --model
12             - meta-llama/Llama-3.1-8B-Instruct
13             - --disaggregation-mode
14             - decode
15           env:
16             - name: DYN_VLLM_EMBEDDING_TRANSFER_MODE
17               value: nixl-read

Native KV event configuration

--kv-events-config and --enable-prefix-caching are native vLLM engine arguments rather than DynamoVllmConfig fields, but they determine whether a worker publishes the cache events used by event-driven KV-aware routing.

Starting the frontend with --router-mode kv does not configure event publishing on vLLM workers. Enable publishing explicitly on every aggregated or prefill worker whose cache state the router should track.

$ python -m dynamo.vllm \
>   --model Qwen/Qwen3-0.6B \
>   --enable-prefix-caching \
>   --kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'

The endpoint value is the base ZeroMQ port. vLLM assigns each data-parallel rank the port at base port + data-parallel rank. When multiple workers share a host or network namespace, reserve one port per rank and choose base ports whose resulting ranges do not overlap.

If workers do not publish KV events, configure the frontend with --no-router-kv-events for prediction-based KV routing or --load-aware for load-only routing.

Worker role and disaggregation

These flags control which role this worker plays in a disaggregated deployment. The default when no --disaggregation-mode is set is aggregated (agg).

--disaggregation-mode

stringDefaults to null

Worker disaggregation mode. agg (default when unset) runs a combined aggregated prefill+decode worker. pd is a legacy alias for agg. prefill and decode split the pipeline for prefill/decode disaggregation. encode starts a multimodal encode-only worker.

Allowed values: pd agg prefill decode encode

Environment variable: DYN_VLLM_DISAGGREGATION_MODE

--headless

booleanDefaults to false

Run in headless mode for multi-node tensor-parallel or pipeline-parallel deployments. Secondary nodes run vLLM workers only with no Dynamo endpoints. See the vLLM multi-node data parallel documentation for details.

Environment variable: DYN_VLLM_HEADLESS

Tokenizer and multimodal

--use-vllm-tokenizer

booleanDefaults to false

Use vLLM’s tokenizer for pre- and post-processing. This bypasses Dynamo’s preprocessor; only the /v1/chat/completions endpoint will be available through the Dynamo frontend.

Environment variable: DYN_VLLM_USE_TOKENIZER

--enable-multimodal

booleanDefaults to false

Enable multimodal processing. Must be set when using any multimodal worker role flag; without it, none of the multimodal components can be activated.

Environment variable: DYN_VLLM_ENABLE_MULTIMODAL

--route-to-encoder

booleanDefaults to false

Enable routing to separate encoder workers for multimodal processing.

Environment variable: DYN_VLLM_ROUTE_TO_ENCODER

--mm-prompt-template

stringDefaults to USER: <image>\n<prompt> ASSISTANT:

Prompt template used to construct the final multimodal prompt sent to the model. The literal <prompt> token is replaced with the user’s text at inference time; <image> marks where the image placeholder appears. Update this template to match your model’s expected format.

Environment variable: DYN_VLLM_MM_PROMPT_TEMPLATE

--frontend-decoding

booleanDefaults to false

Enable frontend decoding of multimodal images. Images are decoded in the Rust frontend and transferred to the backend via NIXL RDMA, bypassing in-engine HTTP fetch and decode.

Environment variable: DYN_VLLM_FRONTEND_DECODING

Embedding

--embedding-transfer-mode

stringDefaults to nixl-write

Embedding transfer mode used between encode and decode workers. local keeps embeddings on the local file system. nixl-write and nixl-read transfer them over NIXL RDMA (writer-initiated or reader-initiated, respectively).

Allowed values: local nixl-write nixl-read

Environment variable: DYN_VLLM_EMBEDDING_TRANSFER_MODE

--embedding-worker

booleanDefaults to false

Run as a text-embedding worker. The vLLM engine must be started with --runner pooling. KV-event publishing, KV router registration, and InstrumentedScheduler injection are all skipped, as they do not apply to pooling models.

Environment variable: DYN_VLLM_EMBEDDING_WORKER

Other

--enable-rl

booleanDefaults to false

Enable reinforcement-learning training support. Selects RL-friendly vLLM defaults for token-in/token-out (TITO) workloads and per-token logprob parity. Mirrors --enable-rl on the SGLang backend.

Environment variable: DYN_ENABLE_RL

--gms-shadow-mode

booleanDefaults to false

Enable GMS (GPU Memory Service) shadow/standby mode. Shadow engines skip KV cache allocation at startup, automatically pause after initialization, and resume on demand when the active engine dies. Requires --load-format=gms.

Environment variable: DYN_VLLM_GMS_SHADOW_MODE

Benchmarking

These flags control the self-benchmark sweep that runs on startup before the worker begins accepting production requests.

--benchmark-mode

stringDefaults to null

Run a self-benchmark on startup before accepting requests. Sweeps prefill input sequence lengths (ISLs) and/or decode (context_length × batch_size) operating points, collecting ForwardPassMetrics at each point.

Allowed values: prefill decode agg

Environment variable: DYN_BENCHMARK_MODE

--benchmark-prefill-granularity

integerDefaults to 16

Number of ISL sample points for the prefill sweep.

Environment variable: DYN_BENCHMARK_PREFILL_GRANULARITY

--benchmark-decode-length-granularity

integerDefaults to 6

Number of context length sample points for the decode sweep.

Environment variable: DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY

--benchmark-decode-batch-granularity

integerDefaults to 6

Number of batch size sample points per context length for the decode sweep.

Environment variable: DYN_BENCHMARK_DECODE_BATCH_GRANULARITY

--benchmark-warmup-iterations

integerDefaults to 5

Number of warmup iterations to run before benchmark measurement begins.

Environment variable: DYN_BENCHMARK_WARMUP_ITERATIONS

--benchmark-output-path

stringDefaults to /tmp/benchmark_results.json

File path where benchmark results are written in JSON format.

Environment variable: DYN_BENCHMARK_OUTPUT_PATH

--benchmark-timeout

integerDefaults to 300

Maximum seconds to wait for the benchmark to complete. Worker startup fails if this limit is exceeded.

Environment variable: DYN_BENCHMARK_TIMEOUT

Deprecated

These flags are retained for backward compatibility and will be removed in a future release. Each is mapped to its replacement at startup with a deprecation warning.

--is-prefill-worker