Skip to navigation
System ComponentsMiddleware

Observability

View as Markdown

This page provides basic information for accessing the observability stack that gets deployed with the Blueprint. The stack includes Grafana, Prometheus, cAdvisor, DCGM Exporter, and Node Exporter.

Components Overview

ComponentWhat it does
GrafanaVisualization layer for metrics dashboards and alerts.
PrometheusMetrics collection and time-series storage for scraping exporters.
cAdvisorContainer-level metrics for CPU, memory, filesystem, and network.
DCGM ExporterGPU telemetry exporter for temperature, power, clocks, and utilization.
Node ExporterHost-level metrics exporter for CPU, memory, disk, and network.

Service Access Points

ServiceURL
Grafanahttp://<HOST_IP>:35000
Prometheushttp://<HOST_IP>:9090
cAdvisorhttp://<HOST_IP>:18080
DCGM Exporterhttp://<HOST_IP>:9400
Node Exporterhttp://<HOST_IP>:9100

Grafana Dashboards

The following dashboards are available by default in Grafana:

Default user and password are:

  • user: admin

  • password: admin

  • nvidia-dcgm-exporter-dashboard: GPU metrics such as temperature, power usage, SM clocks, and utilization.

  • Node Exporter: Host-level metrics for CPU, memory, disk, and network.

  • Docker Monitoring: Container-level metrics including running containers, total memory usage, CPU usage, and network RX.

Alerts Microservice Traces

The Alerts Microservice can export OpenTelemetry traces and metrics. This is off by default and uses two additional components, an OpenTelemetry Collector and Jaeger, that are behind the otel-collector and jaeger Compose profiles. When enabled, the collector receives OTLP on port 4318, forwards traces to Jaeger and Phoenix, and exposes the OpenTelemetry metrics in Prometheus format on port 8889 (bound to 127.0.0.1 by default).

ServiceURL
Jaeger UIhttp://<HOST_IP>:16686

Enablement steps, the span hierarchy, and the configuration reference are in OpenTelemetry Tracing in the Alerts Microservice page.