ComponentsFrontend

Frontend Configuration Reference

Complete reference for all frontend CLI arguments, environment variables, and HTTP endpoints
以 Markdown 格式查看

This page documents all configuration options for the Dynamo Frontend (python -m dynamo.frontend).

Every CLI argument has a corresponding environment variable. CLI arguments take precedence over environment variables.

HTTP & Networking

CLI ArgumentEnv VarDefaultDescription
--http-hostDYN_HTTP_HOST0.0.0.0HTTP listen address
--http-portDYN_HTTP_PORT8000HTTP listen port
--tls-cert-pathDYN_TLS_CERT_PATH—TLS certificate path (PEM). Must be paired with --tls-key-path
--tls-key-pathDYN_TLS_KEY_PATH—TLS private key path (PEM). Must be paired with --tls-cert-path

The Rust HTTP server also reads these environment variables (not exposed as CLI args):

Env VarDefaultDescription
DYN_HTTP_BODY_LIMIT_MB192Maximum request body size in MB
DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS5Graceful shutdown timeout in seconds

Router

This is the canonical CLI and environment-variable reference for the frontend’s embedded router. The Router Guide explains deployment modes and behavior, while Configuration and Tuning explains when to adjust these settings.

Routing and Readiness

CLI ArgumentEnv VarDefaultDescription
--router-modeDYN_ROUTER_MODEround-robinRouting strategy: round-robin, random, power-of-two, kv, direct, least-loaded, or device-aware-weighted. power-of-two samples two workers and selects the one with fewer in-flight requests
--router-min-initial-workersDYN_ROUTER_MIN_INITIAL_WORKERS0Minimum workers required before router startup continues. 0 disables the startup wait
--router-session-affinity-ttl-secsDYN_ROUTER_SESSION_AFFINITY_TTL_SECSunsetEnable session affinity and best-effort binding sync with this router-local idle TTL
--decode-fallback / --no-decode-fallbackDYN_DECODE_FALLBACKfalseFall back to aggregated mode when prefill workers are unavailable

KV Scoring and Cache Locality

CLI ArgumentEnv VarDefaultDescription
--load-aware / --no-load-awareDYN_ROUTER_LOAD_AWAREfalsePreset for KV load-aware routing without cache-reuse signals; implies --router-mode kv
--router-kv-overlap-score-creditDYN_ROUTER_KV_OVERLAP_SCORE_CREDIT1.0Credit multiplier for device-local prefix overlap. Must be finite and nonnegative; values greater than 1.0 give overlap extra credit and can make adjusted prefill cost negative
--router-kv-overlap-score-credit-decayDYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY0.0Decay rate for device-local overlap credit as active prefill load rises above the least-loaded eligible worker. 0 disables decay; 1 halves credit at one request-equivalent of excess load
--router-prefill-load-scaleDYN_ROUTER_PREFILL_LOAD_SCALE1.0Scale adjusted prompt-side prefill load after cache-hit credits are subtracted
--router-host-cache-hit-weightDYN_ROUTER_HOST_CACHE_HIT_WEIGHT0.75Credit multiplier from 0.0 to 1.0 for host-pinned cache hits
--router-disk-cache-hit-weightDYN_ROUTER_DISK_CACHE_HIT_WEIGHT0.25Credit multiplier from 0.0 to 1.0 for disk or other lower-tier cache hits
--shared-cache-multiplierDYN_SHARED_CACHE_MULTIPLIER0.5Experimental multiplier from 0.0 to 1.0 for external shared-cache hits
--shared-cache-typeDYN_SHARED_CACHE_TYPEnoneExperimental external shared cache: none or hicache
--router-temperatureDYN_ROUTER_TEMPERATURE0.0Softmax temperature for normalized worker sampling. 0 is deterministic

KV State and Indexers

CLI ArgumentEnv VarDefaultDescription
--router-kv-events / --no-router-kv-eventsDYN_ROUTER_USE_KV_EVENTStrueConsume worker KV cache events, or predict cache state from routing decisions when disabled
--router-ttl-secsDYN_ROUTER_TTL_SECS120.0Block TTL for prediction-based routing with --no-router-kv-events
--router-predicted-ttl-secsDYN_ROUTER_PREDICTED_TTL_SECSunsetEnable a local predict-on-route side indexer with this TTL. Requires KV events and is independent of --router-ttl-secs
--router-event-threadsDYN_ROUTER_EVENT_THREADS4KV indexer worker threads. Values greater than 1 use the concurrent radix tree, including with --no-router-kv-events
--use-remote-indexer / --no-use-remote-indexerDYN_USE_REMOTE_INDEXERfalseExperimental: query a remote indexer served on the worker component instead of maintaining a local primary indexer
--serve-indexer / --no-serve-indexerDYN_SERVE_INDEXERfalseServe this frontend’s local KV indexers over the request plane. Requires --router-mode kv with positive overlap credit and is mutually exclusive with --use-remote-indexer
--router-replica-sync / --no-router-replica-syncDYN_ROUTER_REPLICA_SYNCfalseBest-effort active-sequence synchronization through the Runtime event plane

Active Load and Queueing

CLI ArgumentEnv VarDefaultDescription
--router-track-active-blocks / --no-router-track-active-blocksDYN_ROUTER_TRACK_ACTIVE_BLOCKStrueTrack blocks used by in-progress requests for load balancing
--router-track-output-blocks / --no-router-track-output-blocksDYN_ROUTER_TRACK_OUTPUT_BLOCKSfalseTrack output blocks with fractional decay during generation
--router-assume-kv-reuse / --no-router-assume-kv-reuseDYN_ROUTER_ASSUME_KV_REUSEtrueAssume KV cache reuse when tracking active blocks
--router-track-prefill-tokens / --no-router-track-prefill-tokensDYN_ROUTER_TRACK_PREFILL_TOKENStrueInclude prompt-side prefill tokens in active-load accounting
--router-prefill-load-modelDYN_ROUTER_PREFILL_LOAD_MODELnonePrompt-side load model: none for static load or aic for oldest-prefill decay using an AIC prediction
--router-queue-thresholdDYN_ROUTER_QUEUE_THRESHOLDunsetQueue threshold fraction of prefill capacity. Setting a numeric value enables queueing; priority hints only affect requests waiting in this queue
--router-queue-policyDYN_ROUTER_QUEUE_POLICYfcfsQueue scheduling policy: fcfs for tail TTFT or wspt for average TTFT
--router-policy-configDYN_ROUTER_POLICY_CONFIGunsetStartup-only policy-family and cache-bucket YAML. When omitted, the threshold and queue policy define one synthetic policy class

AIC Prefill Load Model

These options are used only when --router-mode kv is combined with --router-prefill-load-model aic.

CLI ArgumentEnv VarDefaultDescription
--aic-backendDYN_AIC_BACKEND—Backend family to model in AIC, for example vllm or sglang
--aic-systemDYN_AIC_SYSTEM—AIC hardware/system identifier, for example h200_sxm
--aic-model-pathDYN_AIC_MODEL_PATH—Model path or model identifier used for AIC perf lookup
--aic-backend-versionDYN_AIC_BACKEND_VERSIONbackend-specificPinned AIC database version. If omitted, Dynamo uses the backend default
--aic-tp-sizeDYN_AIC_TP_SIZE1Tensor-parallel size to model in AIC
--aic-moe-tp-sizeDYN_AIC_MOE_TP_SIZE—MoE tensor-parallel size for models that require AIC MoE parallelism
--aic-moe-ep-sizeDYN_AIC_MOE_EP_SIZE—MoE expert-parallel size for models that require AIC MoE parallelism
--aic-attention-dp-sizeDYN_AIC_ATTENTION_DP_SIZE—Attention data-parallel size for models that require AIC MoE parallelism

When enabled, the frontend’s embedded KV router predicts one expected prefill duration per admitted request, using the selected worker’s overlap-derived cached prefix. The router then decays only the oldest active prefill request on each worker for prompt-side load accounting.

For MoE models, AIC requires aic_tp_size * aic_attention_dp_size == aic_moe_tp_size * aic_moe_ep_size. For Kimi-style TP-only MoE runs, set --aic-moe-tp-size to the same value as --aic-tp-size, with --aic-moe-ep-size 1 and --aic-attention-dp-size 1.

Fault Tolerance

CLI ArgumentEnv VarDefaultDescription
--migration-limitDYN_MIGRATION_LIMIT0Max request migrations per worker disconnect. 0 = disabled
--active-decode-blocks-thresholdDYN_ACTIVE_DECODE_BLOCKS_THRESHOLD—KV cache utilization fraction (0.0–1.0) for busy detection. Setting a value independently enables this rejection check
--active-prefill-tokens-thresholdDYN_ACTIVE_PREFILL_TOKENS_THRESHOLD—Absolute token count for prefill busy detection. Setting a value independently enables this rejection check
--active-prefill-tokens-threshold-fracDYN_ACTIVE_PREFILL_TOKENS_THRESHOLD_FRAC—Fraction of max_num_batched_tokens for prefill busy detection. Setting a value independently enables this rejection check and uses OR logic with the absolute threshold

The deprecated --admission-control and DYN_ADMISSION_CONTROL settings are accepted but ignored with a startup warning and no longer gate these thresholds. See Request Rejection for migration instructions.

Model Discovery

CLI ArgumentEnv VarDefaultDescription
--namespaceDYN_NAMESPACE—Exact namespace for model discovery scoping
--namespace-prefixDYN_NAMESPACE_PREFIX—Namespace prefix for discovery (e.g., ns matches ns, ns-abc123). Takes precedence over --namespace
--model-nameDYN_MODEL_NAME—Override model name string
--model-pathDYN_MODEL_PATH—Path to local model directory (for private/custom models)
--kv-cache-block-sizeDYN_KV_CACHE_BLOCK_SIZE—KV cache block size override

Infrastructure

CLI ArgumentEnv VarDefaultDescription
--discovery-backendDYN_DISCOVERY_BACKENDetcdService discovery: kubernetes, etcd, file, mem
--request-planeDYN_REQUEST_PLANEtcpRequest distribution: tcp (fastest), nats
--event-planeDYN_EVENT_PLANEzmqEvent publishing: nats or zmq. Defaults to zmq for every discovery backend

KServe gRPC

CLI ArgumentEnv VarDefaultDescription
--kserve-grpc-server / --no-kserve-grpc-serverDYN_KSERVE_GRPC_SERVERfalseStart KServe gRPC v2 server
--grpc-metrics-portDYN_GRPC_METRICS_PORT8788HTTP metrics port for gRPC service

See the Frontend Guide for KServe message formats and integration details.

Monitoring

CLI ArgumentEnv VarDefaultDescription
--metrics-prefixDYN_METRICS_PREFIXdynamo_frontendPrefix for frontend Prometheus metrics
--dump-config-toDYN_DUMP_CONFIG_TO—Dump resolved config to file path

Tokenizer

CLI ArgumentEnv VarDefaultDescription
--tokenizerDYN_TOKENIZERdefaultTokenizer: default (HuggingFace) or fastokens (high-performance Rust tokenizer). See Tokenizer

Experimental

CLI ArgumentEnv VarDefaultDescription
--enable-anthropic-apiDYN_ENABLE_ANTHROPIC_APIfalseEnable /v1/messages (Anthropic Messages API)
--dyn-chat-processorDYN_CHAT_PROCESSORdynamoChat processor: dynamo (default), vllm, or sglang. See Parser Configuration for how this combines with the parser flags.
--dyn-debug-perfDYN_DEBUG_PERFfalseLog per-function timing for preprocessing (vllm processor only)
--dyn-preprocess-workersDYN_PREPROCESS_WORKERS0Worker processes for CPU-bound preprocessing. 0 = main event loop (vllm processor only)
-i / --interactiveDYN_INTERACTIVEfalseInteractive text chat mode

HTTP Endpoints

The frontend exposes the following HTTP endpoints:

OpenAI-Compatible

MethodPathDescription
POST/v1/chat/completionsChat completions (streaming and non-streaming)
POST/v1/completionsText completions
POST/v1/embeddingsText embeddings
POST/v1/responsesResponses API
POST/v1/images/generationsImage generation
POST/v1/videos/generationsVideo generation
POST/v1/videos/generations/streamVideo generation (streaming)
GET/v1/modelsList available models

Anthropic (Experimental)

MethodPathDescription
POST/v1/messagesAnthropic Messages API (requires --enable-anthropic-api)
POST/v1/messages/count_tokensToken counting for Anthropic API

Infrastructure

MethodPathDescription
GET/healthHealth check
GET/liveLiveness check
GET/metricsPrometheus metrics
GET/openapi.jsonOpenAPI specification
GET/docsSwagger UI
POST/busy_thresholdSet busy thresholds (gated by DYN_DISABLE_FRONTEND_ADMIN_API, see below)
GET/busy_thresholdGet current busy thresholds (gated by DYN_DISABLE_FRONTEND_ADMIN_API, see below)

Frontend feature switches

Environment variables controlling frontend extensions. Extensions are enabled by default. When deploying, consider whether each is needed for your use case; if not, disable it to prevent accidental abuse.

Set an env value of 1 / true / yes / on (case-insensitive) to disable the extension.

Env VarDefaultBehavior when set (disabled)
DYN_DISABLE_FRONTEND_NVEXTunset (enabled)Frontend drops request.nvext at handler entry on /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, and /v1/messages; ignores Dynamo routing headers (x-dynamo-worker-instance-id, x-dynamo-prefill-instance-id, x-dynamo-dp-rank, x-dynamo-prefill-dp-rank, x-dynamo-request-priority, x-dynamo-request-strict-priority) and their compatibility aliases; silently ignores the response-side nvext.extra_fields opt-in. Note: disabling this breaks EPP / GAIE serving, Prime-RL-style training that uses nvext.cache_salt, multi-tenant agent platforms that forward nvext.agent_hints, and clients that opt into response disclosure via nvext.extra_fields.
DYN_DISABLE_FRONTEND_ADMIN_APIunset (enabled)GET /busy_threshold and POST /busy_threshold are not registered (404 instead of 503). Inference, metrics, models, health, and liveness routes are unaffected.

Endpoint Path Customization

All endpoint paths can be overridden via environment variables:

Env VarDefault Path
DYN_HTTP_SVC_CHAT_PATH_ENV/v1/chat/completions
DYN_HTTP_SVC_CMP_PATH_ENV/v1/completions
DYN_HTTP_SVC_EMB_PATH_ENV/v1/embeddings
DYN_HTTP_SVC_RESPONSES_PATH_ENV/v1/responses
DYN_HTTP_SVC_MODELS_PATH_ENV/v1/models
DYN_HTTP_SVC_ANTHROPIC_PATH_ENV/v1/messages
DYN_HTTP_SVC_HEALTH_PATH_ENV/health
DYN_HTTP_SVC_LIVE_PATH_ENV/live
DYN_HTTP_SVC_METRICS_PATH_ENV/metrics

See Also