Environment Variables
This page catalogs the environment variables that govern Dynamo observability. Set them on every frontend, router, or worker that should emit a signal. For local procedures, see Observe a Local Deployment and Observe a Local Deployment.
The canonical list of variable names lives in lib/runtime/src/config/environment_names.rs, which groups them by area (logging, OTLP, system server, canary, request trace). The logging and OTLP variables are read in lib/runtime/src/logging.rs setup_logging(); the DYN_SYSTEM_* and health variables are read by the system-status server. The backend log-level variables (VLLM_LOGGING_LEVEL, TLLM_LOG_LEVEL) are read by the respective engines, not the Dynamo runtime.
Every variable on this page is set the same way whether you run locally or on Kubernetes — the same name, only the value or convention differs. A few fields note an Operator preset: the value the Dynamo operator injects on Kubernetes, which your own configuration overrides. Kubernetes-only metrics enablement is controlled by CRD annotations and Helm values rather than environment variables — see Operator Metrics.
Prometheus metric families in Dynamo are registered lazily: each label set is created the first time it fires, so a freshly-started process shows empty metric families until the first relevant request. An idle cluster does not mean scraping is broken.
Setting these variables
Every Dynamo process reads its own environment at startup, so set these on each process you want to emit signals — the frontend, the router, and each worker. The tabs below cover the two ways to do that; everything inside applies to whichever environment you pick.
Local
Kubernetes
Export the variables in the shell before launching each process, then start the frontend and one or more workers:
Use python -m dynamo.sglang or python -m dynamo.trtllm for the other backends, and python -m dynamo.router for a standalone router.
System and metrics
Port for the backend component’s system-status server, which serves /metrics and the health endpoints. Disabled by default; local examples use 8081. Must be set explicitly for backend workers and the standalone router to expose metrics.
Operator preset: 9090 on worker, prefill, decode, and planner pods.
Frontend HTTP port, where dynamo_frontend_* metrics are served at /metrics. Also configurable via the --http-port flag.
Operator preset: 8000 on frontend pods.
Enables NIXL telemetry. NIXL metrics track KV cache and embedding data transfers and are populated only during disaggregated serving or multimodal embedding transfers.
Allowed values: y nOperator preset: n on worker, prefill, and decode pods.
NIXL telemetry exporter to use, e.g. prometheus.
Operator preset: prometheus on worker, prefill, and decode pods.
Port for NIXL’s Prometheus exporter — a separate port from the Dynamo metrics port. Each NIXL agent binds its own exporter, so every agent needs a port of its own; two agents pointed at the same port leave the second unable to bind and the process exits.
Give each worker a non-overlapping value when workers share a network namespace. Because each value is the base of an eight-port range (see below), 19090 for the prefill worker leaves 19098 as the next free base for the decode worker.
Within a single worker, treat this as the base of a range rather than one port. SGLang runs one scheduler process per node-local GPU and each builds its own NIXL agent, so Dynamo gives node-local rank i the port base + i and the operator declares the whole range on the pod as nixl, nixl-1, … so Prometheus scrapes every rank. Leave room for as many consecutive ports as the pod has GPUs (up to 8).
Set it to a literal port between 1 and 65535. The operator reads this value to declare the range as container ports, so while NIXL Prometheus telemetry is on it rejects a value taken from valueFrom, and one that is not a usable port, rather than let the exporters bind a range nothing scrapes.
Operator preset: 19090 on worker, prefill, and decode pods.
OpenTelemetry (traces and logs)
Master switch for OTLP export. Gates both traces and runtime logs. Request trace records require the separate otel request-trace sink.
Generic endpoint for traces and logs. For grpc, Dynamo uses it as-is. For http/protobuf, Dynamo appends /v1/traces or /v1/logs. When unset, each signal uses its protocol default.
Trace-only endpoint, used as-is. Falls back to OTEL_EXPORTER_OTLP_ENDPOINT, then http://localhost:4317 for grpc or http://localhost:4318/v1/traces for http/protobuf.
Log-only endpoint, used as-is. Falls back to OTEL_EXPORTER_OTLP_ENDPOINT, then http://localhost:4317 for grpc or http://localhost:4318/v1/logs for http/protobuf. It does not fall back to the traces endpoint.
Default OTLP transport for traces and logs.
Allowed values: grpc http/protobufTrace-only protocol override. Falls back to OTEL_EXPORTER_OTLP_PROTOCOL.
Log-only protocol override. Falls back to OTEL_EXPORTER_OTLP_PROTOCOL.
Head-sampling ratio in [0.0, 1.0]. Unset exports every trace. For example, 0.01 retains approximately one percent of traces.
Service name attached to exported telemetry. Use a per-component value such as dynamo-frontend to distinguish services.
Logging
Console output format. Supported values are readable and jsonl; surrounding whitespace is ignored. When unset or blank, Dynamo falls back to DYN_LOGGING_JSONL for backward compatibility. This setting does not control OTLP export.
Legacy JSONL switch used when DYN_LOGGING_CONSOLE_FORMAT is unset or blank. It also remains a local trace-context enablement signal for backward compatibility when the new setting explicitly selects readable output.
Emit span entry/close events (SPAN_FIRST_ENTRY, SPAN_CLOSED messages).
Log level, optionally per target: <default_level>,<module_path>=<level>,.... Example: info,dynamo_runtime::system_status_server:trace.
Use the local timezone for log timestamps. Default is UTC.
Path to a custom TOML logging configuration. Unset by default.
vLLM backend log level. Completely independent of DYN_LOG.
TensorRT-LLM backend log level. Independent of DYN_LOG and read once at import time — it must be set before the process starts.
Disable Dynamo’s SGLang log configuration so you can manage the SGLang engine’s logging independently (e.g. via the worker’s --log-level flag).
Health checks
Path of the frontend health endpoint.
Path of the frontend liveness endpoint.
Initial health status a component reports before its required endpoints are served.
Allowed values: ready notreadyPath of the health endpoint on the system-status server.
Path of the liveness endpoint on the system-status server.
Deprecated compatibility variable. The current runtime logs a warning and does not use this value to select endpoints for system health.
Operator preset: ["generate"] on worker, prefill, and decode pods.
Enables canary health checks, which actively probe worker endpoints during idle periods. Defaults to false. The shadow-engine failover path may re-enable it per engine. (The health-check guide describes canary checks as “automatically enabled” in Kubernetes — that reflects the intended failover behavior, not the base value the operator injects.)
Operator preset: false on worker, prefill, and decode pods.
Seconds of endpoint idle time before a canary health check is sent. Lower values check more frequently.
Maximum seconds to wait for a canary health-check response before marking the endpoint unhealthy.
Optional canary payload override for unified backends. Accepts a JSON object or
@/path/to/payload.json and takes precedence over the backend’s default payload.
Forward Pass Metrics tracing
Enables Forward Pass Metrics persistence. Equivalent to the worker --fpm-trace switch.
Output prefix for <prefix>.<producer-id>.<index>.jsonl.gz files.
Capture mode: sampled writes the latest changed value per worker and data-parallel rank; full writes every valid payload.
Sampling interval used by sampled mode.
Segment roll threshold in uncompressed JSONL bytes.
Number of segments retained independently for each producer.
See Forward Pass Metrics Trace Reference for topology support and file semantics.
FPM self-benchmark collection
These variables tune the vLLM worker’s self-benchmark collector (--benchmark-mode), which measures whole-model forward-pass latencies for the FPM performance database. They have no effect on serving.
KV warm-up master switch. When on, decode benchmark points on expert-parallel MoE models read KV produced by real prefill chains instead of self-injected synthetic KV. Set off to force the legacy synthetic-KV path. Ineligible configurations (dense models, no expert parallelism, prefix caching disabled, or a seeding dataset that is unavailable, empty, too shallow for the model length, or without a loadable tokenizer) skip the warm-up automatically and record the reason. Under attention data parallelism the decision is group-wide: if any rank is ineligible, every rank uses synthetic KV; if any rank’s warm-up stage fails to build or cannot reserve its shadow blocks, every rank falls back to synthetic KV for that stage.
Seeding-text dataset: a local path or an http(s) URL. Defaults to the ShareGPT V3 conversation dump; the default download is verified against a digest pinned in code.
Cache directory for a downloaded dataset. Defaults to fpm_datasets/ next to HF_HOME, falling back to /tmp/fpm_datasets.
Expected sha256 of the dataset file. Required to pin a custom DYN_BENCH_KV_WARMUP_DATASET download; overrides the built-in digest for the default dataset.
Total-KV-read threshold (tokens) above which a fake-injected decode point is measured with repeated steady steps. Points served from the real-KV warm-up chains always repeat. Set 0 to apply repeat protection to every point.
Steady-step repeat count for real-KV points and for fake-injected points above the threshold; the recorded latency is the median. The warm-up chains reserve this many steady writes per request.
Content source for synthetic benchmark prompts: unset keeps salt-seeded random token ids; sharegpt draws deterministic windows from a real-text token pool; sharegpt_chain reuses the KV warm-up chain texts so seeded and measured requests share one construction.
Real-KV seeding for prefill points that read past KV. Accepts on, 1, or true (case-insensitive); anything else keeps the synthetic prefix blocks that are registered in the prefix cache but never computed. When on, the seeded prefix is first computed by a real, unbooked prefill pass (staging; slots already deep enough are not re-run), an unbooked same-shape warm shot follows, and only then does the timed request hit the real KV in the prefix cache. The expected hit length is validated before admission; on a miss the chain is re-staged once and the point is skipped on a second miss (real_seed_injection_failed, real_seed_warm_injection_failed, real_seed_cache_validation_failed). All ranks of an attention-DP group must agree on the setting (it is part of the grid-invariants digest). Dense-attention models measure the same either way; sparse-attention and hybrid-KV models need it for deep past-KV points.
Optional tag mixed into the real-text pool draw so repeated collections sample different content windows under otherwise identical settings.
Every result and skipped-point entry in the artifact carries a kv_seed_regime field plus kvwarm and synthetic_prompts metadata blocks, so downstream consumers can filter by seeding provenance. Decode rows carry real_kv, fake_fallback, legacy (warm-up switched off), skip:<reason> (gate rejected the configuration), or unstamped (decode point that never reached injection). Prefill rows that read past KV carry real_prefix (real-KV seeding on) or fake_prefix (synthetic prefix blocks); other prefill rows are not_applicable.
Request tracing
Master switch. When enabled without an explicit record selection, emits request_end,tool.
Comma-separated record types: request_end, request_payload, and tool.
Comma-separated sinks: file, stderr, nats, otel, and s3.
Literal JSONL path or gzip segment prefix, depending on DYN_REQUEST_TRACE_FILE_FORMAT.
File encoding: jsonl or rotating jsonl_gz.
Best-effort in-process broadcast capacity.
Subject used by the nats sink.
Maximum serialized OTLP payload attribute size. Oversized payload rows emit an incomplete marker.
File batching threshold in bytes.
Periodic file flush interval in milliseconds.
Gzip roll threshold in uncompressed bytes.
Optional gzip roll threshold in records.
Destination bucket for the s3 sink. Required when DYN_REQUEST_TRACE_SINKS includes s3. When it is unset, the s3 sink fails to initialize: Dynamo logs a warning carrying the underlying error, starts no sink from that DYN_REQUEST_TRACE_SINKS list, and keeps serving. Startup does not fail. See Sink Behavior.
Region override for the s3 sink. When unset, the client uses AWS_REGION, then AWS_DEFAULT_REGION, then us-east-1.
Object key prefix for the s3 sink. When unset, records land at the bucket root. Keys are {prefix}/{yyyy}/{mm}/{dd}/{host}-{HHMMSS}-{run_id}-{seq}.jsonl.gz.
Batch roll threshold for the s3 sink in uncompressed bytes. When the pending batch reaches this size it is gzipped and uploaded as one object.
Periodic flush interval for the s3 sink in milliseconds. A partial batch is uploaded when this elapses so low-volume traces still reach S3.
Optional ZMQ PULL bind address for harness tool events. Configure it on only one process.
First-frame ZMQ topic filter.
Comma- or whitespace-separated allowlist of HTTP header names captured on request_payload rows. Values are not redacted.
See Request Trace Reference for compatibility aliases, sink behavior, and record semantics.
Related
- Metrics Catalog — the
dynamo_*metrics these variables expose. - Metric Labels — the dimensions attached to those metrics.
- Operator Metrics — Kubernetes-only metrics enablement knobs.
- Local Resource Monitor (Local) — host-side per-process exporter.