Skip to navigation
NVKE telemetry pipeline (NKTP)Management cluster telemetry

Management cluster telemetry: getting started

View as Markdown

NVKE telemetry pipeline (NKTP) collects the management cluster’s metrics, logs and traces and exports them over OTLP. It is installed by the deploy/nico deploy path, the same way as the rest of the management plane. There is nothing to install by hand.

This guide covers what NKTP gives the management cluster, what the deploy path needs from you, and how to take a first look. The other guides in this directory cover configuration, verification and troubleshooting.

What you get

The nktp item in deploy/nico installs the NKTP chart from the same checkout, into namespace nktp:

  • The OpenTelemetry Operator and its CRDs, plus the Prometheus monitoring CRDs (ServiceMonitor, PodMonitor and the rest).
  • Collectors, each an OpenTelemetryCollector the Operator runs:
    • one on every node, for pod logs, the systemd journal, host auth and audit logs, the kubelet and cAdvisor, and every monitor scheduled to its node
    • a gateway per signal (metrics, logs, traces), the OTLP entry points that export to the backend
    • a large-target pool for kube-state-metrics and the control-plane monitors
    • a Kubernetes Event reader
    • a watchdog that receives every Collector’s own telemetry
  • kube-state-metrics, node-exporter and metrics-server. metrics-server is the site’s only provider of metrics.k8s.io, which the Collector autoscalers read.
  • A development backend, lgtm: Grafana, Prometheus, Loki and Tempo in one Pod, with the NKTP dashboards. It keeps data in memory and has no authentication, so it is for evaluation. See Connecting a real backend.

The deploy path also wires the rest of the management plane into it:

ComponentSignalHow
NVKE controllermetrics, logs, tracesServiceMonitor, pod logs, OTLP traces to the node Collector; its metrics NetworkPolicy admits nktp
Temporalmetricsa ServiceMonitor per server role, from Temporal’s own chart
SPIRE server and agentmetricsPodMonitors, from SPIRE’s own chart
API server, etcd, CoreDNSmetricsthe chart’s control-plane monitors
API serveraudit logread from the node, classified k8s.api.audit
Every Podlogs, statepod logs from the node; state from kube-state-metrics
The nodemetrics, journalnode-exporter, kubelet and cAdvisor, the systemd journal

Everything exported carries k8s.cluster.name set to the site’s name.

Not scraped:

  • cert-manager, NodeLocal DNSCache, OVN-Kubernetes and CAPNICo. NKTP ships no monitors for other components: each service’s owner ships its own. These four don’t yet; see monitoring your service.
  • CAPI core and the kubeadm providers, which serve metrics only to a caller with a bearer token.
  • The scheduler and controller manager, which listen on 127.0.0.1 on NVKE clusters.
  • Postgres, which runs no metrics exporter.

The state of all their Pods still arrives through kube-state-metrics.

Before you deploy

The deploy path installs NKTP with the rest of the management plane, so its prerequisites are the path’s own (Before this runs), plus:

  • cert-manager. clusterctl init installs it. It issues the Operator’s webhook certificate and the mTLS certificates between each Collector and its Target Allocator.
  • ClusterName in the BootstrapConfig passed with --config. It becomes k8s.cluster.name on every signal. The install stops at the nvke-base-config item if it is missing: 
    { "ClusterName": "<site-name>", "Nico": { "...": "..." } }
  • Egress from the node running the install:
    • to open-telemetry.github.io, prometheus-community.github.io and kubernetes-sigs.github.io, because the install builds the chart’s dependencies from its Chart.lock before installing
    • to docker.io, ghcr.io, quay.io and registry.k8s.io, for the images
  • Room on the nodes. The site values size NKTP for a small management cluster. Its Pods request about 1.6 CPU and 3.9Gi of memory in total, plus about 125m and 320Mi for each node beyond the first: the node Collector and kube-state-metrics’ node Pod run on every node. The development backend sets no requests but used about 1.2Gi at idle.

Nothing else. The deploy path creates the nktp namespace with the privileged Pod Security level the node Collector needs, and every setting a management cluster needs is in nktp/values.yaml.

Install order

nktp comes straight after capnico and before everything that creates a monitor:

nvke-base-config -> spire-tpm-pubhashes -> spire-server-nodeport
  -> spire-oidc-provider-nodeport -> capnico -> nktp
  -> postgresql -> spire-crds -> temporal -> spire -> controller
  • Why this position: the chart installs the monitoring CRDs, which the temporal, spire and controller items need for their monitors.
  • A side effect: installing it first means NKTP also records the rest of the management plane coming up.
  • The cost: a failed nktp install stops the path there.

First look

On the management cluster, with its kubeconfig:

export KUBECONFIG=<site-kubeconfig>
kubectl -n nktp get pods                       # all Running
kubectl -n nktp get opentelemetrycollectors    # seven, each READY 1/1
kubectl -n nktp get certificates               # all Ready

To open Grafana, port-forward the backend. Bind it to localhost only: Grafana gives anonymous visitors admin rights.

kubectl -n nktp port-forward deploy/lgtm 3000:3000 --address 127.0.0.1

If you are working through a jump host, forward that port over SSH rather than binding it on the site network. The NKTP Health dashboard shows whether the pipeline itself is healthy.

NVKE reference dashboards

NVKE also ships reference Grafana dashboards as JSON files. These are starting points for investigating a site, not a complete monitoring or alerting setup. Review their Requirements and scope panels, check that your backend collects the required metrics, and adapt the dashboards and alerts to your operating needs. An empty optional panel can mean its metric is not scraped.

To use these dashboards, open Grafana Dashboards → New → Import, upload its JSON file (or paste the JSON), and select a Prometheus datasource when prompted. The files are not installed automatically on the management cluster. You can import them into the bundled lgtm Grafana while developing, but import them into your real telemetry backend for site operations: lgtm keeps data in memory and has no authentication. See the dashboard README for the full catalog and import details.

For the full checklist, see verification.

Next steps