> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Observability

Dynamo emits three signals — **metrics** (Prometheus), **logs** (structured text or JSONL, exportable to Loki), and **traces** (OpenTelemetry, exportable to Tempo). This page installs the backends that collect them. Enabling each signal is a matter of setting a few environment variables per process or deployment.

## Signals at a glance

| Signal | Turn it on with | Collected by |
|--------|-----------------|--------------|
| Metrics | `DYN_SYSTEM_PORT` (workers/router); the frontend serves metrics on its HTTP port | Prometheus |
| Traces | `OTEL_EXPORT_ENABLED=true` + an OTLP endpoint | Tempo |
| Logs | `DYN_LOGGING_JSONL=true` (+ `OTEL_EXPORT_ENABLED` to export) | Loki |
| FPM traces | `DYN_FPM_TRACE=1` or `--fpm-trace` | Rotating gzip JSONL files |
| Health status | `/live` and `/health` endpoints | Supervisors or Kubernetes probes |
| Request traces | `DYN_REQUEST_TRACE=1` or `DYN_REQUEST_TRACE_RECORDS` | Files, NATS, stderr, or OTLP logs |

`OTEL_EXPORT_ENABLED` is the master switch for both traces and logs — without it, neither leaves the process even when Tempo and Loki are healthy. Dynamo supports OTLP over `grpc` and `http/protobuf`. Use `OTEL_EXPORTER_OTLP_ENDPOINT` for a shared collector or signal-specific endpoint variables for separate destinations. For the full variable catalog — defaults, per-signal grouping, and the Kubernetes operator presets — see [Environment Variables](/dynamo/dev/observability/environment-variables).

For scheduler telemetry, see [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#capture-forward-pass-metrics). For replay metadata or request payload capture, see [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#capture-and-replay-requests).

<Note>
Prometheus metric families are registered lazily: each label set is created the first time it fires, so a freshly-started process shows empty metric families until the first relevant request. An idle cluster does not mean scraping is broken.
</Note>

## Kubernetes stack

On Kubernetes, install the monitoring backends once, then enable signals per deployment — see [Observability](/dynamo/dev/kubernetes/operations/observability) under Operations for the per-deployment steps. The Dynamo operator wires metrics scraping automatically (it adds a `PodMonitor` to every managed pod), so the only one-time work is installing Prometheus, the exporters, and the logging stack.

### kube-prometheus-stack

Install Prometheus, Grafana, and the Prometheus Operator. The selector flags let Prometheus discover `PodMonitor` and `ServiceMonitor` resources created outside its own Helm release (such as the ones the Dynamo operator creates):

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \
  --set-json 'prometheus.prometheusSpec.probeNamespaceSelector={}'
```

<Note>
The Dynamo install command in the [Installation Guide](/dynamo/dev/kubernetes/installation/install-dynamo#kube-prometheus-stack) sets `dynamo-operator.dynamo.metrics.prometheusEndpoint` to point the operator at this Prometheus. Set it there, in one pass, rather than here.

An older form of the discovery flags used `--set ...podMonitorNamespaceSelector.matchLabels=null` (and the same for `probeNamespaceSelector`). The `--set-json '...={}'` form above is equivalent and preferred; use one form or the other consistently, not both.
</Note>

kube-prometheus-stack bundles **node-exporter** by default, which supplies the node CPU, system load, and container resource panels on the Dynamo dashboard. Verify it is running:

```bash
kubectl get daemonset -A | grep node-exporter
```

If you run a custom Prometheus instead of kube-prometheus-stack, deploy node-exporter separately as a DaemonSet.

### DCGM exporter (GPU metrics)

GPU-utilization panels are populated by [dcgm-exporter](https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html). Check whether it is already running (the NVIDIA GPU Operator installs it as part of its monitoring components):

```bash
kubectl get daemonset -A | grep dcgm-exporter
```

If the output is empty, install it via the GPU Operator or the standalone dcgm-exporter chart.

### Loki + Alloy (logging stack)

To collect pod logs into Grafana Loki, install Loki and the Grafana Alloy collector. This reference setup suits development and testing clusters (including Minikube and MicroK8s); use a high-availability configuration for production.

Set the namespaces once:

```bash
export MONITORING_NAMESPACE=monitoring    # where Loki is installed
export DYN_NAMESPACE=dynamo-system        # where Dynamo is installed
```

Install Loki in single-binary mode (local MinIO storage), then install Alloy pointed at it:

```bash
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Loki
helm install --values deploy/observability/logging/values/loki-values.yaml \
  loki grafana/loki -n $MONITORING_NAMESPACE

# Alloy collector (k8s-monitoring chart), templated with the namespaces above
envsubst < deploy/observability/logging/values/alloy-values.yaml > alloy-custom-values.yaml
helm install --values alloy-custom-values.yaml \
  alloy grafana/k8s-monitoring -n $MONITORING_NAMESPACE
```

The Alloy values file forwards logs to Loki, restricts collection to `$DYN_NAMESPACE`, and maps the `nvidia.com/dynamo-component-type` and `nvidia.com/dynamo-graph-deployment-name` pod labels into Loki labels so the logging dashboard can filter by deployment and component. Applying the Grafana datasource and logging dashboard, and enabling JSONL logging on a deployment, are covered in [Observability](/dynamo/dev/kubernetes/operations/observability#logging) under Operations.