> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Environment Variables This page catalogs the environment variables that govern Dynamo observability. Set them on every frontend, router, or worker that should emit a signal. For local procedures, see [Observe a Local Deployment](/dynamo/dev/cli/operations/observability) and [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#check-deployment-health). The canonical list of variable names lives in [`lib/runtime/src/config/environment_names.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/config/environment_names.rs), which groups them by area (logging, OTLP, system server, canary, request trace). The logging and OTLP variables are read in [`lib/runtime/src/logging.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/logging.rs) `setup_logging()`; the `DYN_SYSTEM_*` and health variables are read by the system-status server. The backend log-level variables (`VLLM_LOGGING_LEVEL`, `TLLM_LOG_LEVEL`) are read by the respective engines, not the Dynamo runtime. Every variable on this page is set the same way whether you run locally or on Kubernetes — the same name, only the value or convention differs. A few fields note an **Operator preset**: the value the Dynamo operator injects on Kubernetes, which your own configuration overrides. Kubernetes-only *metrics enablement* is controlled by CRD annotations and Helm values rather than environment variables — see [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics#kubernetes-enablement-knobs). Prometheus metric families in Dynamo are registered lazily: each label set is created the first time it fires, so a freshly-started process shows empty metric families until the first relevant request. An idle cluster does not mean scraping is broken. ## Setting these variables Every Dynamo process reads its own environment at startup, so set these on each process you want to emit signals — the frontend, the router, and each worker. The tabs below cover the two ways to do that; everything inside applies to whichever environment you pick. #### Local Export the variables in the shell before launching each process, then start the frontend and one or more workers: ```bash export DYN_SYSTEM_PORT=8081 export DYN_LOGGING_CONSOLE_FORMAT=jsonl export DYN_LOG=info # Frontend (serves dynamo_frontend_* on DYN_HTTP_PORT, default 8000) python -m dynamo.frontend & # vLLM worker (serves dynamo_component_* on DYN_SYSTEM_PORT) python -m dynamo.vllm --model Qwen/Qwen3-0.6B & ``` Use `python -m dynamo.sglang` or `python -m dynamo.trtllm` for the other backends, and `python -m dynamo.router` for a standalone router. #### Kubernetes Set the variables on the [`DynamoGraphDeployment`](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment) (DGD) — at graph level in `spec.envs` (prepended to every component) or per-component under `services..extraPodSpec.mainContainer.env`. A component-level entry overrides a graph-level one of the same name. ```yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: name: vllm-agg-tracing spec: envs: # applied to every component - name: DYN_LOGGING_CONSOLE_FORMAT value: "jsonl" - name: OTEL_EXPORT_ENABLED value: "true" - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "http://tempo.observability.svc.cluster.local:4317" services: Frontend: componentType: frontend extraPodSpec: mainContainer: env: # applied to this component only - name: OTEL_SERVICE_NAME value: "dynamo-frontend" ``` A [`DynamoGraphDeploymentRequest`](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment-request) (DGDR) has no environment-variable fields of its own. To set these under a DGDR, put them in the embedded DGD it produces via `spec.overrides.dgd`. You never set these on the operator deployment itself — the operator only writes them into the workload pods it creates; it never reads them from its own environment. ## System and metrics **`DYN_SYSTEM_PORT`** `integer` — default: -1 (disabled) Port for the backend component's system-status server, which serves `/metrics` and the health endpoints. Disabled by default; local examples use `8081`. Must be set explicitly for backend workers and the standalone router to expose metrics. **Operator preset:** `9090` on worker, prefill, decode, and planner pods. --- **`DYN_HTTP_PORT`** `integer` — default: 8000 Frontend HTTP port, where `dynamo_frontend_*` metrics are served at `/metrics`. Also configurable via the `--http-port` flag. **Operator preset:** `8000` on frontend pods. --- **`NIXL_TELEMETRY_ENABLE`** `string` — default: n Enables [NIXL](https://github.com/ai-dynamo/nixl) telemetry. NIXL metrics track KV cache and embedding data transfers and are populated only during disaggregated serving or multimodal embedding transfers. Allowed values: y n **Operator preset:** `n` on worker, prefill, and decode pods. --- **`NIXL_TELEMETRY_EXPORTER`** `string` NIXL telemetry exporter to use, e.g. `prometheus`. **Operator preset:** `prometheus` on worker, prefill, and decode pods. --- **`NIXL_TELEMETRY_PROMETHEUS_PORT`** `integer` — default: 19090 Port for NIXL's Prometheus exporter — a separate port from the Dynamo metrics port. Use distinct values per worker (e.g. `19090` prefill, `19091` decode) to avoid collisions. **Operator preset:** `19090` on worker, prefill, and decode pods. --- ## OpenTelemetry (traces and logs) **`OTEL_EXPORT_ENABLED`** `boolean` — default: false Master switch for OTLP export. Gates both traces and runtime logs. Request trace records require the separate `otel` request-trace sink. --- **`OTEL_EXPORTER_OTLP_ENDPOINT`** `string` Generic endpoint for traces and logs. For `grpc`, Dynamo uses it as-is. For `http/protobuf`, Dynamo appends `/v1/traces` or `/v1/logs`. When unset, each signal uses its protocol default. --- **`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`** `string` Trace-only endpoint, used as-is. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317` for `grpc` or `http://localhost:4318/v1/traces` for `http/protobuf`. --- **`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`** `string` Log-only endpoint, used as-is. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317` for `grpc` or `http://localhost:4318/v1/logs` for `http/protobuf`. It does not fall back to the traces endpoint. --- **`OTEL_EXPORTER_OTLP_PROTOCOL`** `string` — default: grpc Default OTLP transport for traces and logs. Allowed values: grpc http/protobuf --- **`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`** `string` Trace-only protocol override. Falls back to `OTEL_EXPORTER_OTLP_PROTOCOL`. --- **`OTEL_EXPORTER_OTLP_LOGS_PROTOCOL`** `string` Log-only protocol override. Falls back to `OTEL_EXPORTER_OTLP_PROTOCOL`. --- **`OTEL_TRACES_SAMPLE_RATIO`** `number` Head-sampling ratio in `[0.0, 1.0]`. Unset exports every trace. For example, `0.01` retains approximately one percent of traces. --- **`OTEL_SERVICE_NAME`** `string` — default: dynamo Service name attached to exported telemetry. Use a per-component value such as `dynamo-frontend` to distinguish services. --- ## Logging **`DYN_LOGGING_CONSOLE_FORMAT`** `string` — default: readable Console output format. Supported values are `readable` and `jsonl`; surrounding whitespace is ignored. When unset or blank, Dynamo falls back to `DYN_LOGGING_JSONL` for backward compatibility. This setting does not control OTLP export. Allowed values: readable jsonl --- **`DYN_LOGGING_JSONL`** `boolean` — default: false Legacy JSONL switch used when `DYN_LOGGING_CONSOLE_FORMAT` is unset or blank. It also remains a local trace-context enablement signal for backward compatibility when the new setting explicitly selects readable output. --- **`DYN_LOGGING_SPAN_EVENTS`** `boolean` — default: false Emit span entry/close events (`SPAN_FIRST_ENTRY`, `SPAN_CLOSED` messages). --- **`DYN_LOG`** `string` — default: info Log level, optionally per target: `,=,...`. Example: `info,dynamo_runtime::system_status_server:trace`. --- **`DYN_LOG_USE_LOCAL_TZ`** `boolean` — default: false Use the local timezone for log timestamps. Default is UTC. --- **`DYN_LOGGING_CONFIG_PATH`** `string` Path to a custom TOML logging configuration. Unset by default. --- **`VLLM_LOGGING_LEVEL`** `string` — default: INFO vLLM backend log level. Completely independent of `DYN_LOG`. Allowed values: DEBUG INFO WARNING ERROR CRITICAL --- **`TLLM_LOG_LEVEL`** `string` — default: INFO TensorRT-LLM backend log level. Independent of `DYN_LOG` and read **once at import time** — it must be set before the process starts. Allowed values: TRACE DEBUG INFO WARNING ERROR INTERNAL\_ERROR --- **`DYN_SKIP_SGLANG_LOG_FORMATTING`** `boolean` — default: false Disable Dynamo's SGLang log configuration so you can manage the SGLang engine's logging independently (e.g. via the worker's `--log-level` flag). --- ## Health checks **`DYN_HTTP_SVC_HEALTH_PATH`** `string` — default: /health Path of the frontend health endpoint. --- **`DYN_HTTP_SVC_LIVE_PATH`** `string` — default: /live Path of the frontend liveness endpoint. --- **`DYN_SYSTEM_STARTING_HEALTH_STATUS`** `string` — default: notready Initial health status a component reports before its required endpoints are served. Allowed values: ready notready --- **`DYN_SYSTEM_HEALTH_PATH`** `string` — default: /health Path of the health endpoint on the system-status server. --- **`DYN_SYSTEM_LIVE_PATH`** `string` — default: /live Path of the liveness endpoint on the system-status server. --- **`DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS`** `string` Deprecated compatibility variable. The current runtime logs a warning and does not use this value to select endpoints for system health. **Operator preset:** `["generate"]` on worker, prefill, and decode pods. --- **`DYN_HEALTH_CHECK_ENABLED`** `boolean` — default: false Enables canary health checks, which actively probe worker endpoints during idle periods. Defaults to `false`. The shadow-engine failover path may re-enable it per engine. (The health-check *guide* describes canary checks as "automatically enabled" in Kubernetes — that reflects the intended failover behavior, not the base value the operator injects.) **Operator preset:** `false` on worker, prefill, and decode pods. --- **`DYN_CANARY_WAIT_TIME`** `integer` — default: 10 Seconds of endpoint idle time before a canary health check is sent. Lower values check more frequently. --- **`DYN_HEALTH_CHECK_REQUEST_TIMEOUT`** `integer` — default: 3 Maximum seconds to wait for a canary health-check response before marking the endpoint unhealthy. --- **`DYN_HEALTH_CHECK_PAYLOAD`** `string` Optional canary payload override for unified backends. Accepts a JSON object or `@/path/to/payload.json` and takes precedence over the backend's default payload. --- ## Forward Pass Metrics tracing **`DYN_FPM_TRACE`** `boolean` Enables Forward Pass Metrics persistence. Equivalent to the worker `--fpm-trace` switch. --- **`DYN_FPM_OUTPUT_PATH`** `string` — default: /tmp/dynamo-fpm Output prefix for `...jsonl.gz` files. --- **`DYN_FPM_MODE`** `string` — default: sampled Capture mode: `sampled` writes the latest changed value per worker and data-parallel rank; `full` writes every valid payload. --- **`DYN_FPM_SAMPLE_INTERVAL_MS`** `integer` — default: 5000 Sampling interval used by `sampled` mode. --- **`DYN_FPM_JSONL_GZ_ROLL_BYTES`** `integer` — default: 268435456 Segment roll threshold in uncompressed JSONL bytes. --- **`DYN_FPM_MAX_SEGMENTS`** `integer` — default: 4 Number of segments retained independently for each producer. --- See [Forward Pass Metrics Trace Reference](/dynamo/dev/reference/observability/forward-pass-metrics-traces) for topology support and file semantics. ## Request tracing **`DYN_REQUEST_TRACE`** `boolean` Master switch. When enabled without an explicit record selection, emits `request_end,tool`. --- **`DYN_REQUEST_TRACE_RECORDS`** `string` — default: request\_end,tool Comma-separated record types: `request_end`, `request_payload`, and `tool`. --- **`DYN_REQUEST_TRACE_SINKS`** `string` — default: file Comma-separated sinks: `file`, `stderr`, `nats`, `otel`, and `s3`. --- **`DYN_REQUEST_TRACE_FILE_PATH`** `string` — default: /tmp/dynamo-request-trace Literal JSONL path or gzip segment prefix, depending on `DYN_REQUEST_TRACE_FILE_FORMAT`. --- **`DYN_REQUEST_TRACE_FILE_FORMAT`** `string` — default: jsonl\_gz File encoding: `jsonl` or rotating `jsonl_gz`. --- **`DYN_REQUEST_TRACE_CAPACITY`** `integer` — default: 1024 Best-effort in-process broadcast capacity. --- **`DYN_REQUEST_TRACE_NATS_SUBJECT`** `string` — default: dynamo.request\_trace.v1 Subject used by the `nats` sink. --- **`DYN_REQUEST_TRACE_OTEL_MAX_PAYLOAD_BYTES`** `integer` — default: 4194304 Maximum serialized OTLP payload attribute size. Oversized payload rows emit an incomplete marker. --- **`DYN_REQUEST_TRACE_FILE_BUFFER_BYTES`** `integer` — default: 1048576 File batching threshold in bytes. --- **`DYN_REQUEST_TRACE_FILE_FLUSH_INTERVAL_MS`** `integer` — default: 1000 Periodic file flush interval in milliseconds. --- **`DYN_REQUEST_TRACE_FILE_ROLL_BYTES`** `integer` — default: 268435456 Gzip roll threshold in uncompressed bytes. --- **`DYN_REQUEST_TRACE_FILE_ROLL_LINES`** `integer` Optional gzip roll threshold in records. --- **`DYN_REQUEST_TRACE_S3_BUCKET`** `string` Destination bucket for the `s3` sink. Required when `DYN_REQUEST_TRACE_SINKS` includes `s3`; startup fails if unset. --- **`DYN_REQUEST_TRACE_S3_REGION`** `string` Region override for the `s3` sink. When unset, the client uses `AWS_REGION`, then `AWS_DEFAULT_REGION`, then `us-east-1`. --- **`DYN_REQUEST_TRACE_S3_PREFIX`** `string` Object key prefix for the `s3` sink. When unset, records land at the bucket root. Keys are `{prefix}/{yyyy}/{mm}/{dd}/{host}-{HHMMSS}-{run_id}-{seq}.jsonl.gz`. --- **`DYN_REQUEST_TRACE_S3_ROLL_UNCOMPRESSED_BYTES`** `integer` — default: 67108864 Batch roll threshold for the `s3` sink in uncompressed bytes. When the pending batch reaches this size it is gzipped and uploaded as one object. --- **`DYN_REQUEST_TRACE_S3_FLUSH_INTERVAL_MS`** `integer` — default: 10000 Periodic flush interval for the `s3` sink in milliseconds. A partial batch is uploaded when this elapses so low-volume traces still reach S3. --- **`DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT`** `string` Optional ZMQ PULL bind address for harness tool events. Configure it on only one process. --- **`DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_TOPIC`** `string` — default: agent-tool-events First-frame ZMQ topic filter. --- **`DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST`** `string` Comma- or whitespace-separated allowlist of HTTP header names captured on `request_payload` rows. Values are not redacted. --- See [Request Trace Reference](/dynamo/dev/reference/observability/request-traces) for compatibility aliases, sink behavior, and record semantics. ## Related * [Metrics Catalog](/dynamo/dev/reference/observability/metrics-catalog) — the `dynamo_*` metrics these variables expose. * [Metric Labels](/dynamo/dev/reference/observability/metric-labels) — the dimensions attached to those metrics. * [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) — Kubernetes-only metrics enablement knobs. * [Local Resource Monitor (Local)](/dynamo/dev/reference/observability/local-resource-monitor-local) — host-side per-process exporter.