Observability

This page covers the day-to-day observability actions for a running Dynamo deployment: turning each signal on or off per deployment, loading the Grafana dashboards, and querying the results. It assumes the monitoring stack (kube-prometheus-stack, DCGM exporter, and the Loki + Alloy logging stack) is already installed — see Observability in the Installation section for those one-time steps.

For what each metric, label, and variable means, see the reference: Metrics Catalog, Metric Labels, Environment Variables, and Operator Metrics.

Enable signals per deployment

Metrics

Metrics are on by default. The operator adds a PodMonitor and labels every pod nvidia.com/metrics-enabled: "true", so Prometheus discovers and scrapes frontend and worker /metrics endpoints with no deployment-level field required. Both components expose OpenMetrics-format metrics: the frontend on its HTTP port, workers on their system port.

To opt a deployment out of metrics collection, set the nvidia.com/enable-metrics: "false" annotation:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
  annotations:
6
    nvidia.com/enable-metrics: "false"
7
spec:
8
  # …

NIXL telemetry

To add NIXL transfer metrics (populated only during disaggregated serving or multimodal embedding transfers), set NIXL_TELEMETRY_ENABLE: "y" on the worker component. NIXL exposes its metrics on a separate port (NIXL_TELEMETRY_PROMETHEUS_PORT, default 19090). See Environment Variables for the full NIXL variable set.

1
spec:
2
  services:
3
    VllmDecodeWorker:
4
      componentType: worker
5
      extraPodSpec:
6
        mainContainer:
7
          env:
8
            - name: NIXL_TELEMETRY_ENABLE
9
              value: "y"

Logging

To emit structured JSONL logs (required for the Loki logging stack), set the logging variables on the DGD. Graph-level spec.envs applies to every component; a per-component env entry overrides it. See Environment Variables for the full set and their meanings.

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: vllm-agg-logging
5
spec:
6
  envs:                                # applied to every component
7
    - name: DYN_LOGGING_JSONL
8
      value: "true"
9
    - name: DYN_LOG
10
      value: "info"
11
  services:
12
    Frontend:
13
      componentType: frontend
14
      extraPodSpec:
15
        mainContainer:
16
          env:                         # applied to this component only
17
            - name: DYN_LOG
18
              value: "info,dynamo_runtime::system_status_server:trace"

Traces and logs export

To export traces to Tempo and logs to Loki via OTLP, set the OpenTelemetry variables on the DGD. OTEL_EXPORT_ENABLED is the master switch — without it, traces and logs never leave the process. See Environment Variables.

1
spec:
2
  envs:
3
    - name: OTEL_EXPORT_ENABLED
4
      value: "true"
5
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
6
      value: "http://otel-collector.observability.svc.cluster.local:4317"
7
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
8
      value: "grpc"

Use signal-specific endpoints only when traces and logs have different collectors. The logs endpoint does not fall back to the traces endpoint. For HTTP/protobuf, set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; Dynamo appends signal paths only to the generic endpoint.

To export chat payload records through the same collector, also set DYN_REQUEST_TRACE_RECORDS=request_payload and DYN_REQUEST_TRACE_SINKS=otel. See Request Trace Reference.

Forward Pass Metrics Trace Files

Forward Pass Metrics (FPM) tracing records backend scheduler telemetry to rotating gzip JSONL files. To retain the files across pod replacement, mount a persistent volume on each worker and set the output prefix inside that mount:

1
spec:
2
  pvcs:
3
    - create: false
4
      name: fpm-traces
5
  services:
6
    VllmWorker:
7
      volumeMounts:
8
        - name: fpm-traces
9
          mountPoint: /var/log/dynamo
10
      extraPodSpec:
11
        mainContainer:
12
          env:
13
            - name: DYN_FPM_TRACE
14
              value: "1"
15
            - name: DYN_FPM_OUTPUT_PATH
16
              value: /var/log/dynamo/fpm

Replace VllmWorker with each worker service that should write a trace. The PVC must already exist when create: false. Dynamo writes a separate producer-specific file sequence for every worker, so size retention for the number of replicas and old producer IDs that can remain on the volume.

Graceful shutdown flushes accepted records. SIGKILL, node loss, queue pressure, and upstream transport loss can still create gaps. Treat the files as observability data, not as a durable event plane or replay input. See Forward Pass Metrics Trace Reference for topology support, configuration, schema, and rotation behavior.

View dashboards

$
# Application dashboard (frontend, KV-router, worker metrics)
$
kubectl apply -n monitoring -f deploy/observability/grafana-dynamo-dashboard-configmap.yaml
$
$
# Operator dashboard (reconciliation, webhooks, resource inventory)
$
kubectl apply -f deploy/observability/grafana-operator-dashboard-configmap.yaml
$
$
# Logging dashboard + Loki datasource
$
kubectl apply -n monitoring -f deploy/observability/logging/grafana/loki-datasource.yaml
$
kubectl apply -n monitoring -f deploy/observability/logging/grafana/logging-dashboard.yaml

$
# Grafana credentials
$
export GRAFANA_USER=$(kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 --decode)
$
export GRAFANA_PASSWORD=$(kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode)
$
echo "Grafana user: $GRAFANA_USER / password: $GRAFANA_PASSWORD"
$
$
# Port-forward Grafana
$
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

$
kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring

Useful queries

Application metrics

# Total frontend requests
dynamo_frontend_requests_total
# Time-to-first-token distribution
dynamo_frontend_time_to_first_token_seconds_bucket

Derived signals

These cluster-wide totals combine per-stage frontend gauges into the signals operators most often want. dynamo_frontend_stage_requests has no model label, so these cannot be split by model; add by (pod) or by (instance) for per-pod visibility. See Stage values for what each stage covers.

# Requests waiting for a worker to start generating (the old "queued" semantic)
sum(dynamo_frontend_stage_requests{stage=~"preprocess|route|dispatch"})
# Requests currently being processed by a backend worker (authoritative worker-side gauge)
sum(dynamo_component_inflight_requests{dynamo_component="backend",dynamo_endpoint="generate"})
# Router saturation — spikes when workers can't be selected fast enough
sum(dynamo_frontend_stage_requests{stage="route"})
# Backend prefill saturation — spikes when the backend is slow to produce first tokens
sum(dynamo_frontend_stage_requests{stage="dispatch"})

Operator metrics

Operator metrics use the dynamo_operator_* prefix and are scraped via a ServiceMonitor (created by the Helm chart), separate from the application PodMonitor. See Operator Metrics for the full catalog and label sets.

# P95 reconciliation duration by resource type
histogram_quantile(0.95,
  sum by (resource_type, le) (
    rate(dynamo_operator_reconcile_duration_seconds_bucket[5m])
  )
)
# Reconciliation errors by type
sum by (resource_type, error_type) (
  rate(dynamo_operator_reconcile_errors_total[5m])
)
# Webhook denial rate
sum by (resource_type, operation, reason) (
  rate(dynamo_operator_webhook_denials_total[5m])
)
# Managed resources by type and state
sum by (resource_type, status) (
  dynamo_operator_resources_total
)

Troubleshooting

Metrics not appearing in Prometheus

1. Check the monitor exists — PodMonitor for application metrics, ServiceMonitor for operator metrics:

   $
   kubectl get podmonitor,servicemonitor -n dynamo-system

2. Confirm Prometheus discovered the target: in the Prometheus UI, go to Status → Targets and look for the Dynamo job in state UP.

3. Check the Prometheus selector configuration. The discovery of monitors outside the Prometheus release requires podMonitorSelectorNilUsesHelmValues=false (and the equivalent for ServiceMonitors), set during kube-prometheus-stack installation:

   $
   kubectl get prometheus -o yaml | grep -iE "podMonitorSelector|serviceMonitorSelector"

Dashboard not appearing in Grafana

1. Confirm the ConfigMap was created and carries the discovery label:

   $
   kubectl get configmap -n monitoring <dashboard-configmap-name> \
   >
     -o jsonpath='{.metadata.labels.grafana_dashboard}'   # should return "1"

2. Confirm the Grafana dashboard sidecar is watching for that label:

   $
   kubectl get deployment -n monitoring prometheus-grafana -o yaml | grep -A5 sidecar

3. Restart Grafana to force a refresh:

   $
   kubectl rollout restart deployment/prometheus-grafana -n monitoring