Frontend Configuration Reference

Field reference for the Dynamo Frontend CLI arguments, environment variables, and HTTP endpoints.
View as Markdown

This page documents the configuration options for the Dynamo Frontend (python -m dynamo.frontend).

Every CLI argument has a corresponding environment variable. The CLI argument takes precedence; the environment variable is the fallback.

These are the frontend-specific flags. The frontend also parses the shared Dynamo runtime flags (namespace, discovery, planes, parsing) — see Runtime Configuration. For the router cost model and tuning guidance behind the router flags below, see Configuration and Tuning.

HTTP and networking

--http-host
stringDefaults to 0.0.0.0

HTTP listen address.

Environment variable: DYN_HTTP_HOST

--http-port
integerDefaults to 8000

HTTP listen port.

Environment variable: DYN_HTTP_PORT

--tls-cert-path
stringDefaults to null

TLS certificate path (PEM). Must be paired with --tls-key-path.

Environment variable: DYN_TLS_CERT_PATH

--tls-key-path
stringDefaults to null

TLS private key path (PEM). Must be paired with --tls-cert-path.

Environment variable: DYN_TLS_KEY_PATH

The Rust HTTP server also reads these environment variables, which are not exposed as CLI arguments:

DYN_HTTP_BODY_LIMIT_MB
integerDefaults to 192

Maximum request body size in MB.

DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS
integerDefaults to 5

Maximum time the Frontend waits for admitted HTTP response bodies and /v1/realtime WebSocket tasks to finish after shutdown begins. New inference requests are rejected while draining.

DYN_HTTP_OVERLOAD_STATUS_CODE
integerDefaults to 529

HTTP status returned for overload and admission-control rejection. Use 503 only for clients that cannot handle 529. Invalid or out-of-range values fall back to 529. The value is read once and cached when the HTTP service initializes.

Router

This section is the canonical CLI and environment-variable reference for the frontend’s embedded router. See Router Guide for deployment modes and Configuration and Tuning for behavior and tuning guidance.

Routing and readiness

--router-mode
stringDefaults to round-robin

Routing strategy. power-of-two samples two workers and selects the worker with fewer in-flight requests. In disaggregated prefill mode, power-of-two and least-loaded use the synchronous prefill fallback path.

Allowed values: round-robin random power-of-two kv direct least-loaded device-aware-weighted

Environment variable: DYN_ROUTER_MODE

--router-min-initial-workers
integerDefaults to 0

Minimum number of workers required before router startup continues. 0 disables the startup wait.

Environment variable: DYN_ROUTER_MIN_INITIAL_WORKERS

--router-session-affinity-ttl-secs
integerDefaults to null

Enable session affinity with this router-local idle TTL in seconds. Bindings synchronize across router replicas on a best-effort basis. Valid values are 1 through 31536000; omit the option to disable session affinity. See Session affinity.

Environment variable: DYN_ROUTER_SESSION_AFFINITY_TTL_SECS

--decode-fallback / --no-decode-fallback
booleanDefaults to false

Fall back to aggregated mode when prefill workers are unavailable.

Environment variable: DYN_DECODE_FALLBACK

KV scoring and cache locality

--load-aware / --no-load-aware
booleanDefaults to false

Preset for KV load-aware routing without cache-reuse signals; implies --router-mode kv.

Environment variable: DYN_ROUTER_LOAD_AWARE

--router-kv-overlap-score-credit
floatDefaults to 1.0

Credit multiplier for device-local prefix overlap. The value must be finite and nonnegative. Values greater than 1.0 give device overlap extra credit and can make adjusted prefill cost negative. See Configuration and Tuning.

Environment variable: DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT

--router-kv-overlap-score-credit-decay
floatDefaults to 0.0

Decay rate for device-local overlap credit as active prefill load rises above the least-loaded eligible worker. 0 disables decay; 1 halves the credit at one request-equivalent of excess active prefill load.

Environment variable: DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY

--router-prefill-load-scale
floatDefaults to 1.0

Scale applied to adjusted prompt-side prefill load after overlap and lower-tier cache-hit credits are subtracted. The minimum is 0.0; there is no hard maximum.

Environment variable: DYN_ROUTER_PREFILL_LOAD_SCALE

--router-host-cache-hit-weight
floatDefaults to 0.75

Credit multiplier from 0.0 through 1.0 for host-pinned, CPU-tier prefix-cache hits.

Environment variable: DYN_ROUTER_HOST_CACHE_HIT_WEIGHT

--router-disk-cache-hit-weight
floatDefaults to 0.25

Credit multiplier from 0.0 through 1.0 for disk or other lower-tier prefix-cache hits.

Environment variable: DYN_ROUTER_DISK_CACHE_HIT_WEIGHT

--shared-cache-multiplier
floatDefaults to 0.5

Experimental. Credit multiplier from 0.0 through 1.0 for external shared-cache hits.

Environment variable: DYN_SHARED_CACHE_MULTIPLIER

--shared-cache-type
stringDefaults to none

Experimental. External shared KV-cache implementation.

Allowed values: none hicache

Environment variable: DYN_SHARED_CACHE_TYPE

--router-temperature
floatDefaults to 0.0

Softmax temperature for normalized worker sampling. 0 selects the lowest-cost worker deterministically; higher values add randomness.

Environment variable: DYN_ROUTER_TEMPERATURE

KV state and indexers

--router-kv-events / --no-router-kv-events
booleanDefaults to true

Consume KV cache state events from workers. Disable this option to predict cache state from routing decisions instead.

Environment variable: DYN_ROUTER_USE_KV_EVENTS

--router-ttl-secs
floatDefaults to 120.0

Block TTL in seconds for prediction-based routing. Used only with --no-router-kv-events.

Environment variable: DYN_ROUTER_TTL_SECS

--router-predicted-ttl-secs
floatDefaults to null

Enable a local predict-on-route side indexer with this TTL in seconds. This option requires KV events and is independent of --router-ttl-secs, which configures pure approximate mode.

Environment variable: DYN_ROUTER_PREDICTED_TTL_SECS

--router-event-threads
integerDefaults to 4

KV indexer worker threads. Values greater than 1 use the concurrent radix tree, including with --no-router-kv-events.

Environment variable: DYN_ROUTER_EVENT_THREADS

--use-remote-indexer / --no-use-remote-indexer
booleanDefaults to false

Experimental. Query a remote KV indexer served by a worker component instead of maintaining a local primary indexer.

Environment variable: DYN_USE_REMOTE_INDEXER

--serve-indexer / --no-serve-indexer
booleanDefaults to false

Serve this frontend’s local KV indexers over the request plane. Requires --router-mode kv and is mutually exclusive with --use-remote-indexer.

Environment variable: DYN_SERVE_INDEXER

--router-replica-sync / --no-router-replica-sync
booleanDefaults to false

Enable best-effort active-sequence synchronization through the Runtime event plane.

Environment variable: DYN_ROUTER_REPLICA_SYNC

--router-tracking-hash
stringDefaults to public-xxh3-v1

Hash function for router-derived active-sequence identities. keyed-xxh3-v1 is experimental and requires both a tracking key file and key ID.

Allowed values: public-xxh3-v1 keyed-xxh3-v1

Environment variable: DYN_ROUTER_TRACKING_HASH

--router-tracking-key-file
stringDefaults to null

File containing the provider tracking key. Keyed tracking requires exactly 32 raw bytes.

Environment variable: DYN_ROUTER_TRACKING_KEY_FILE

--router-tracking-key-id
stringDefaults to null

Nonempty provider-managed key epoch identifier required for keyed tracking.

Environment variable: DYN_ROUTER_TRACKING_KEY_ID

Active load and queueing

--router-track-active-blocks / --no-router-track-active-blocks
booleanDefaults to true

Track blocks used by in-progress requests for load balancing.

Environment variable: DYN_ROUTER_TRACK_ACTIVE_BLOCKS

--router-assume-kv-reuse / --no-router-assume-kv-reuse
booleanDefaults to true

Assume KV cache reuse when tracking active blocks.

Environment variable: DYN_ROUTER_ASSUME_KV_REUSE

--router-track-output-blocks / --no-router-track-output-blocks
booleanDefaults to false

Track output blocks with fractional decay during generation.

Environment variable: DYN_ROUTER_TRACK_OUTPUT_BLOCKS

--router-decode-active-request-weight
floatDefaults to 0.0

Experimental. Finite, nonnegative block-equivalent decode cost added for each active request on a candidate worker. Tune this value only when decode step latency depends materially on active batch size.

Environment variable: DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT

--router-track-prefill-tokens / --no-router-track-prefill-tokens
booleanDefaults to true

Track prompt-side prefill tokens in worker load accounting.

Environment variable: DYN_ROUTER_TRACK_PREFILL_TOKENS

--router-prefill-load-model
stringDefaults to none

Prompt-side load model. none keeps static prompt load; aic decays the oldest active prefill request using an AIC prediction. See AIC prefill load model.

Allowed values: none aic

Environment variable: DYN_ROUTER_PREFILL_LOAD_MODEL

--router-queue-threshold
floatDefaults to null

Queue threshold fraction of prefill capacity. Setting a nonnegative numeric value enables queueing; priority hints affect only requests waiting in this queue.

Environment variable: DYN_ROUTER_QUEUE_THRESHOLD

--router-queue-policy
stringDefaults to fcfs

Queue scheduling policy. fcfs optimizes tail Time To First Token (TTFT); wspt optimizes average TTFT.

Allowed values: fcfs wspt

Environment variable: DYN_ROUTER_QUEUE_POLICY

--router-policy-config
stringDefaults to null

Startup-only policy-family and cache-bucket YAML. When omitted, --router-queue-threshold and --router-queue-policy define one synthetic policy class. Queueing remains disabled until a threshold is configured. See Policy-class queues.

Environment variable: DYN_ROUTER_POLICY_CONFIG

AIC prefill load model

These options apply only when --router-mode kv is combined with --router-prefill-load-model aic.

When enabled, the frontend’s embedded KV router predicts one expected prefill duration per admitted request, using the selected worker’s overlap-derived cached prefix, then decays only the oldest active prefill request on each worker for prompt-side load accounting.

--aic-backend
stringDefaults to null

Backend family to model in AIC, for example vllm or sglang.

Environment variable: DYN_AIC_BACKEND

--aic-system
stringDefaults to null

AIC hardware/system identifier, for example h200_sxm.

Environment variable: DYN_AIC_SYSTEM

--aic-model-path
stringDefaults to null

Model path or model identifier used for AIC perf lookup.

Environment variable: DYN_AIC_MODEL_PATH

--aic-backend-version
stringDefaults to backend-specific

Pinned AIC database version. If omitted, Dynamo uses the backend default.

Environment variable: DYN_AIC_BACKEND_VERSION

--aic-tp-size
integerDefaults to 1

Tensor-parallel size to model in AIC.

Environment variable: DYN_AIC_TP_SIZE

--aic-moe-tp-size
integerDefaults to null

MoE tensor-parallel size for models that require AIC MoE parallelism.

Environment variable: DYN_AIC_MOE_TP_SIZE

--aic-moe-ep-size
integerDefaults to null

MoE expert-parallel size for models that require AIC MoE parallelism.

Environment variable: DYN_AIC_MOE_EP_SIZE

--aic-attention-dp-size
integerDefaults to null

Attention data-parallel size for models that require AIC MoE parallelism.

Environment variable: DYN_AIC_ATTENTION_DP_SIZE

For MoE models, AIC requires aic_tp_size * aic_attention_dp_size == aic_moe_tp_size * aic_moe_ep_size. For Kimi-style TP-only MoE runs, set --aic-moe-tp-size to the same value as --aic-tp-size, with --aic-moe-ep-size 1 and --aic-attention-dp-size 1.

Fault tolerance

--migration-limit
integerDefaults to 0

Maximum request migrations per worker disconnect. 0 disables migration.

Environment variable: DYN_MIGRATION_LIMIT

--migration-max-seq-len
integerDefaults to null

Maximum prompt-plus-output sequence length that remains eligible for migration. When a request strictly exceeds the limit, migration and token tracking stop for that request. Must be from 1 through 4294967295. Unset means no sequence-length limit.

Environment variable: DYN_MIGRATION_MAX_SEQ_LEN

--active-decode-blocks-threshold
floatDefaults to null

KV cache utilization fraction from 0.0 through 1.0. A numeric value independently enables decode-block busy rejection. Decode-block telemetry requires --router-mode kv; use --router-track-output-blocks when generated tokens should contribute to the observed load.

Environment variable: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD

--active-prefill-tokens-threshold
integerDefaults to null

Absolute active-prefill-token threshold. A numeric value greater than or equal to 0 independently enables this check. Uses OR logic with the fractional prefill threshold.

Environment variable: DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD

--active-prefill-tokens-threshold-frac
floatDefaults to null

Non-negative fraction of max_num_batched_tokens used as the active-prefill-token threshold. A numeric value independently enables this check. Uses OR logic with the absolute prefill threshold.

Environment variable: DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD_FRAC

For guidance on choosing thresholds, see Request Rejection and Request Migration.

Model discovery

--namespace
stringDefaults to null

Exact namespace for model discovery scoping.

Environment variable: DYN_NAMESPACE

--namespace-prefix
stringDefaults to null

Namespace prefix for discovery (for example, ns matches ns, ns-abc123). Takes precedence over --namespace.

Environment variable: DYN_NAMESPACE_PREFIX

--model-name
stringDefaults to null

Override model name string.

Environment variable: DYN_MODEL_NAME

--model-path
stringDefaults to null

Path to a local model directory (for private/custom models).

Environment variable: DYN_MODEL_PATH

--kv-cache-block-size
integerDefaults to null

KV cache block size override.

Environment variable: DYN_KV_CACHE_BLOCK_SIZE

Infrastructure

--discovery-backend
stringDefaults to etcd

Service discovery backend.

Allowed values: kubernetes etcd file mem

Environment variable: DYN_DISCOVERY_BACKEND

--request-plane
stringDefaults to tcp

Request distribution transport. tcp is the fastest.

Allowed values: tcp nats

Environment variable: DYN_REQUEST_PLANE

--event-plane
stringDefaults to auto

Event publishing transport. Defaults to zmq for file/mem discovery and nats for etcd/kubernetes.

Allowed values: nats zmq

Environment variable: DYN_EVENT_PLANE

KServe gRPC

--kserve-grpc-server / --no-kserve-grpc-server
booleanDefaults to false

Start the KServe gRPC v2 server.

Environment variable: DYN_KSERVE_GRPC_SERVER

--grpc-metrics-port
integerDefaults to 8788

HTTP metrics port for the gRPC service.

Environment variable: DYN_GRPC_METRICS_PORT

The gRPC server also supports optional HTTP/2 flow-control tuning via environment variables:

DYN_GRPC_INITIAL_CONNECTION_WINDOW_SIZE
integerDefaults to 65536

HTTP/2 connection-level flow control window size in bytes. Default is the tonic default (64 KB).

DYN_GRPC_INITIAL_STREAM_WINDOW_SIZE
integerDefaults to 65536

HTTP/2 per-stream flow control window size in bytes. Default is the tonic default (64 KB).

See the Frontend Guide for KServe message formats, gRPC tuning examples, and integration details.

Monitoring

--metrics-prefix
stringDefaults to dynamo_frontend

Prefix for frontend Prometheus metrics.

Environment variable: DYN_METRICS_PREFIX

--dump-config-to
stringDefaults to null

Dump the resolved config to a file path.

Environment variable: DYN_DUMP_CONFIG_TO

Tokenizer

--tokenizer
stringDefaults to default

Tokenizer implementation. default uses HuggingFace; fastokens uses the high-performance Rust tokenizer. See Tokenizer.

Allowed values: default fastokens

Environment variable: DYN_TOKENIZER

Experimental

Python route extensions

--frontend-route-extension
stringDefaults to []

Load a trusted HTTP route provider by a name registered in the dynamo.frontend.routes entry-point group or by an importable module:function path. Repeat the option to load multiple providers. Providers can add static GET routes but cannot override built-in routes; invalid or duplicate routes fail startup. Extensions apply only to the HTTP frontend. See Python Route Extensions for a packaged and direct-load example.

Environment variable: DYN_FRONTEND_ROUTE_EXTENSIONS accepts whitespace-separated values.

Handler contract

  • Provider resolution: A registered dynamo.frontend.routes entry-point name takes precedence. If no registered name matches, a value containing : is resolved as module:function. Unknown names fail startup and report the available registered extensions.
  • Provider return: The provider callable returns one FrontendRoute or an iterable of routes.
  • Route shape: Extensions support static-path GET routes only. Path parameters, wildcards, other HTTP methods, asynchronous handlers, and duplicates of built-in routes are rejected.
  • Handler signature: A synchronous handler(ctx: FrontendExtensionContext) returns a JSON-serializable body for HTTP 200, or FrontendResponse(status_code, body) to override the status code.
  • Live state: FrontendExtensionContext exposes is_ready(), is_cancelled(), has_any_ready_model(), is_model_ready_to_serve(name), model_display_names(), and serving_ready_display_names().
  • Execution limits: Handlers run in a small dedicated thread pool. A handler that exceeds 30 seconds returns 503. A saturated pool rejects new extension requests with the configured overload status code, 529 by default.
  • Multiple providers: Repeating the CLI option or listing whitespace-separated environment values loads multiple providers. Duplicate provider names are de-duplicated.

Other experimental options

--enable-anthropic-api
booleanDefaults to false

Enable /v1/messages (the Anthropic Messages API).

Environment variable: DYN_ENABLE_ANTHROPIC_API

--dyn-chat-processor
stringDefaults to dynamo

Chat processor. See Chat Processors for how this combines with the parser flags.

Allowed values: dynamo vllm sglang

Environment variable: DYN_CHAT_PROCESSOR

--dyn-debug-perf
booleanDefaults to false

Log per-function timing for preprocessing (vllm processor only).

Environment variable: DYN_DEBUG_PERF

--dyn-preprocess-workers
integerDefaults to 0

Worker processes for CPU-bound preprocessing. 0 uses the main event loop (vllm processor only).

Environment variable: DYN_PREPROCESS_WORKERS

-i / --interactive
booleanDefaults to false

Interactive text chat mode.

Environment variable: DYN_INTERACTIVE

HTTP endpoints

The frontend exposes the following HTTP endpoints.

OpenAI-compatible

MethodPathDescription
POST/v1/chat/completionsChat completions (streaming and non-streaming)
POST/v1/completionsText completions
POST/v1/embeddingsText embeddings
POST/v1/responsesResponses API
POST/v1/images/generationsImage generation
POST/v1/videos/generationsVideo generation
POST/v1/videos/generations/streamVideo generation (streaming)
GET/v1/modelsList available models

Anthropic (Experimental)

MethodPathDescription
POST/v1/messagesAnthropic Messages API (requires --enable-anthropic-api)
POST/v1/messages/count_tokensToken counting for the Anthropic API

Infrastructure

MethodPathDescription
GET/healthHealth check
GET/liveLiveness check
GET/metricsPrometheus metrics
GET/openapi.jsonOpenAPI specification
GET/docsSwagger UI
POST/busy_thresholdSet independently enabled busy thresholds for a discovered model (disabled when DYN_DISABLE_FRONTEND_ADMIN_API is truthy)
GET/busy_thresholdGet stored busy thresholds (disabled when DYN_DISABLE_FRONTEND_ADMIN_API is truthy)

POST /busy_threshold accepts model plus any of active_decode_blocks_threshold, active_prefill_tokens_threshold, and active_prefill_tokens_threshold_frac. A numeric field enables only that check. The router reevaluates workers on the next worker-load or runtime-configuration update; the call does not synchronously recompute the busy set and does not change startup-only router options such as --router-mode kv or --router-track-output-blocks.

Frontend feature switches

Environment variables controlling frontend extensions. Extensions are enabled by default. Set a truthy value (1, true, yes, or on, case-insensitive) to disable the corresponding surface.

DYN_DISABLE_FRONTEND_NVEXT
booleanDefaults to false

When truthy, the frontend drops request.nvext on /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, and /v1/messages; ignores the x-dynamo-worker-instance-id, x-dynamo-prefill-instance-id, x-dynamo-dp-rank, x-dynamo-prefill-dp-rank, x-dynamo-request-priority, and x-dynamo-request-strict-priority routing headers and their compatibility aliases; and ignores the response-side nvext.extra_fields opt-in.

DYN_DISABLE_FRONTEND_ADMIN_API
booleanDefaults to false

When truthy, GET /busy_threshold and POST /busy_threshold are not registered and return 404. Inference, metrics, models, health, and liveness routes are unaffected.

Endpoint path customization

All endpoint paths can be overridden via environment variables:

Env VarDefault Path
DYN_HTTP_SVC_CHAT_PATH_ENV/v1/chat/completions
DYN_HTTP_SVC_CMP_PATH_ENV/v1/completions
DYN_HTTP_SVC_EMB_PATH_ENV/v1/embeddings
DYN_HTTP_SVC_RESPONSES_PATH_ENV/v1/responses
DYN_HTTP_SVC_MODELS_PATH_ENV/v1/models
DYN_HTTP_SVC_ANTHROPIC_PATH_ENV/v1/messages
DYN_HTTP_SVC_HEALTH_PATH_ENV/health
DYN_HTTP_SVC_LIVE_PATH_ENV/live
DYN_HTTP_SVC_METRICS_PATH_ENV/metrics

Deprecated

--admission-control
stringDefaults to nullDeprecated

Deprecated and ignored. Configure the three busy thresholds directly. The compatibility flag and DYN_ADMISSION_CONTROL are accepted only so older launch commands continue to start.

Environment variable: DYN_ADMISSION_CONTROL

Runtime Configuration

Shared Dynamo runtime flags parsed by the frontend and every backend.

Configuration and Tuning

Router cost model and tuning guidance behind the router flags on this page.

Frontend Guide

KServe gRPC configuration and integration details.

NVIDIA Request Extensions (nvext)

Custom per-request fields for routing, preprocessing, and response metadata.