Feature Guides

Observability (Local)

Monitor Dynamo deployments with metrics, logging, and tracing
以 Markdown 格式查看

Required environment variables

Set these on every Dynamo process (frontend, router, workers) for metrics, traces, and logs to flow:

VariablePurposeRequired
DYN_SYSTEM_PORT=8081Unified system port (metrics + health).Yes for metrics.
OTEL_EXPORT_ENABLED=trueEnable OpenTelemetry export. Without this, traces and logs never leave the process — Loki and Tempo will show nothing even if they are healthy.Yes for traces/logs.
OTEL_EXPORTER_OTLP_ENDPOINTGeneric OTLP endpoint for both traces and logs (e.g. http://otel-collector:4317). With http/protobuf, Dynamo appends the signal path (/v1/traces, /v1/logs).Yes, unless both signal-specific endpoints below are set.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTEndpoint for traces only (e.g. http://tempo:4317), used as-is — no path is appended. Falls back to OTEL_EXPORTER_OTLP_ENDPOINT, then the protocol default (http://localhost:4317 for grpc, http://localhost:4318/v1/traces for http/protobuf).Optional override.
OTEL_EXPORTER_OTLP_LOGS_ENDPOINTEndpoint for logs only (e.g. http://loki-otlp:4317), used as-is. Same fallback chain as the traces endpoint (the http/protobuf default is http://localhost:4318/v1/logs). Does not fall back to the traces endpoint (changed in v1.3.0) — if you set only the traces endpoint, logs go to the protocol default and are silently dropped.Optional override.
OTEL_EXPORTER_OTLP_PROTOCOLOTLP transport: grpc (default) or http/protobuf. Override per signal with OTEL_EXPORTER_OTLP_TRACES_PROTOCOL / OTEL_EXPORTER_OTLP_LOGS_PROTOCOL.Optional.
OTEL_TRACES_SAMPLE_RATIOTrace head-sampling ratio in [0.0, 1.0] (e.g. 0.01 keeps ~1% of traces). Unset exports every trace.Optional.
DYN_LOGGING_JSONL=trueStructured JSON log output (recommended for Loki).Optional.

Source of truth: lib/runtime/src/logging.rs setup_logging().

Passing --enable-metrics on an individual backend only exposes metrics per backend. The unified frontend metrics surface (scraped by Prometheus) requires DYN_SYSTEM_PORT to be set on the frontend process as well — setting it on workers alone is not enough.

Prometheus metric families in Dynamo are registered lazily: each label set is created the first time it fires, so a freshly-started process shows empty metric families until the first relevant request. This is expected — an idle cluster does not mean scraping is broken.

Getting Started Quickly

This is an example to get started quickly on a single machine.

Prerequisites

Install these on your machine:

Starting the Observability Stack

Dynamo provides a Docker Compose-based observability stack that includes Prometheus, Grafana, Tempo, Loki, an OpenTelemetry Collector, and various exporters for metrics, tracing, logging, and visualization.

From the Dynamo root directory:

$# Start infrastructure (NATS, etcd)
$docker compose -f dev/docker-compose.yml up -d
$
$# Start observability stack (Prometheus, Grafana, Tempo, DCGM GPU exporter, NATS exporter)
$docker compose -f dev/docker-observability.yml up -d

For detailed setup instructions and configuration, see Prometheus + Grafana Setup.

Observability Documentation

GuideDescriptionEnvironment Variables to Control
MetricsAvailable metrics referenceDYN_SYSTEM_PORT†
Operator Metrics (Kubernetes)Operator controller and webhook metrics for KubernetesN/A (configured via Helm)
Health ChecksComponent health monitoring and readiness probesDYN_SYSTEM_PORT†, DYN_SYSTEM_STARTING_HEALTH_STATUS, DYN_SYSTEM_HEALTH_PATH, DYN_SYSTEM_LIVE_PATH, DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS
TracingDistributed tracing with OpenTelemetry and TempoDYN_LOGGING_JSONL†, OTEL_EXPORT_ENABLED†, OTEL_EXPORTER_OTLP_ENDPOINT†, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT†, OTEL_EXPORTER_OTLP_PROTOCOL†, OTEL_TRACES_SAMPLE_RATIO, OTEL_SERVICE_NAME†
Forward Pass Metrics TracingBest-effort rotating gzip JSONL capture of backend forward pass metricsDYN_FPM_TRACE, DYN_FPM_OUTPUT_PATH, DYN_FPM_MODE, DYN_FPM_SAMPLE_INTERVAL_MS, DYN_FPM_JSONL_GZ_ROLL_BYTES, DYN_FPM_MAX_SEGMENTS
Request Replay TracingPer-request trace capture for direct DynoSim replay and optional payload loggingDYN_REQUEST_TRACE, DYN_REQUEST_TRACE_SINKS, DYN_REQUEST_TRACE_FILE_PATH, DYN_REQUEST_TRACE_RECORDS
LoggingStructured logging and OTLP log export to LokiDYN_LOGGING_JSONL†, DYN_LOG, DYN_LOG_USE_LOCAL_TZ, DYN_LOGGING_CONFIG_PATH, OTEL_SERVICE_NAME†, OTEL_EXPORT_ENABLED†, OTEL_EXPORTER_OTLP_ENDPOINT†, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT†, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT†, OTEL_EXPORTER_OTLP_PROTOCOL†
Request Payload LoggingPer-request chat-completion payload rows exported over OTLP logsDYN_REQUEST_TRACE_SINKS, DYN_REQUEST_TRACE_RECORDS, DYN_REQUEST_TRACE_OTEL_MAX_PAYLOAD_BYTES, DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT†, OTEL_EXPORTER_OTLP_LOGS_PROTOCOL†

Variables marked with † are shared across multiple observability systems.

Developer Guides

GuideDescriptionEnvironment Variables to Control
Metrics Developer GuideCreating custom metrics in Rust and PythonDYN_SYSTEM_PORT†
Local Resource MonitorPer-process VRAM / PCIe / CPU exporter for engine-startup profiling (200 ms scrape, profile-gated)N/A (host-side script)

Kubernetes

For Kubernetes-specific setup and configuration, see docs/kubernetes/observability/.

Operator Metrics: The Dynamo Operator running in Kubernetes exposes its own set of metrics for monitoring controller reconciliation, webhook validation, and resource inventory. See the Operator Metrics Guide.

Topology

This provides:

  • Prometheus on http://localhost:9090 - metrics collection and querying
  • Grafana on http://localhost:3000 - visualization dashboards (username: dynamo, password: dynamo)
  • Tempo on http://localhost:3200 - distributed tracing backend
  • Loki on http://localhost:3100 - log aggregation backend
  • OpenTelemetry Collector on http://localhost:4317 (gRPC) / http://localhost:4318 (HTTP) - receives OTLP signals and routes traces to Tempo and logs to Loki
  • DCGM Exporter on http://localhost:9401/metrics - GPU metrics
  • NATS Exporter on http://localhost:7777/metrics - NATS messaging metrics

Service Relationship Diagram

The dcgm-exporter service in the Docker Compose network is configured to use port 9401 instead of the default port 9400. This adjustment is made to avoid port conflicts with other dcgm-exporter instances that may be running simultaneously. Such a configuration is typical in distributed systems like SLURM.

Configuration Files

The following configuration files are located in the dev/observability/ directory: