> This page is for NVIDIA Kubernetes Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvke/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvke/_mcp/server.

# Management cluster telemetry: getting started

NVKE telemetry pipeline (NKTP) collects the management cluster's
metrics, logs and traces and exports them over OTLP. It is installed by the `deploy/nico` deploy path, the same way as the rest of the
management plane. There is nothing to install by hand.

This guide covers what NKTP gives the management cluster, what the deploy path needs
from you, and how to take a first look. The other guides in this directory
cover [configuration](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-configuration), [verification](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-verification) and
[troubleshooting](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-troubleshooting).

## What you get

The `nktp` item in [`deploy/nico`](https://github.com/dsx-ai-factory/nvke/blob/main/deploy/nico/README.md)
installs the [NKTP chart](https://github.com/dsx-ai-factory/nvke/blob/main/nktp/deploy/charts/nktp/README.md) from the
same checkout, into namespace `nktp`:

- **The OpenTelemetry Operator** and its CRDs, plus the Prometheus monitoring
  CRDs (`ServiceMonitor`, `PodMonitor` and the rest).
- **Collectors**, each an `OpenTelemetryCollector` the Operator runs:
  - one on every node, for pod logs, the systemd journal, host auth and audit
    logs, the kubelet and cAdvisor, and every monitor scheduled to its node
  - a gateway per signal (metrics, logs, traces), the OTLP entry points that
    export to the backend
  - a large-target pool for kube-state-metrics and the control-plane monitors
  - a Kubernetes Event reader
  - a watchdog that receives every Collector's own telemetry
- **kube-state-metrics, node-exporter and metrics-server.** metrics-server is
  the site's only provider of `metrics.k8s.io`, which the Collector
  autoscalers read.
- **A development backend,** `lgtm`: Grafana, Prometheus, Loki and Tempo in one
  Pod, with the NKTP dashboards. It keeps data in memory and has no
  authentication, so it is for evaluation. See
  [Connecting a real backend](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-configuration#connecting-a-real-backend).

The deploy path also wires the rest of the management plane into it:

| Component | Signal | How |
|---|---|---|
| NVKE controller | metrics, logs, traces | ServiceMonitor, pod logs, OTLP traces to the node Collector; its metrics NetworkPolicy admits `nktp` |
| Temporal | metrics | a ServiceMonitor per server role, from Temporal's own chart |
| SPIRE server and agent | metrics | PodMonitors, from SPIRE's own chart |
| API server, etcd, CoreDNS | metrics | the chart's control-plane monitors |
| API server | audit log | read from the node, classified `k8s.api.audit` |
| Every Pod | logs, state | pod logs from the node; state from kube-state-metrics |
| The node | metrics, journal | node-exporter, kubelet and cAdvisor, the systemd journal |

Everything exported carries `k8s.cluster.name` set to the site's name.

**Not scraped:**
- **cert-manager, NodeLocal DNSCache, OVN-Kubernetes and CAPNICo.** NKTP ships
  no monitors for other components: each service's owner ships its own. These
  four don't yet; see [monitoring your service](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-monitoring-your-service).
- **CAPI core and the kubeadm providers,** which serve metrics only to a
  caller with a bearer token.
- **The scheduler and controller manager,** which listen on `127.0.0.1` on
  NVKE clusters.
- **Postgres,** which runs no metrics exporter.

The state of all their Pods still arrives through kube-state-metrics.

## Before you deploy

The deploy path installs NKTP with the rest of the management plane, so its
prerequisites are the path's own
([Before this runs](https://github.com/dsx-ai-factory/nvke/blob/main/deploy/nico/README.md#before-this-runs)),
plus:

- **cert-manager.** `clusterctl init` installs it. It issues the Operator's
  webhook certificate and the mTLS certificates between each Collector and
  its Target Allocator.
- **`ClusterName` in the BootstrapConfig** passed with `--config`. It becomes
  `k8s.cluster.name` on every signal. The install stops at the `nvke-base-config`
  item if it is missing:
  ```json
  { "ClusterName": "<site-name>", "Nico": { "...": "..." } }
  ```
- **Egress from the node running the install:**
  - to `open-telemetry.github.io`, `prometheus-community.github.io` and
    `kubernetes-sigs.github.io`, because the install builds the
    chart's dependencies from its `Chart.lock` before installing
  - to `docker.io`, `ghcr.io`, `quay.io` and `registry.k8s.io`, for the images
- **Room on the nodes.** The site values size NKTP for a small management
  cluster. Its Pods request about 1.6 CPU and 3.9Gi of memory
  in total, plus about 125m and 320Mi for each node beyond the first: the node
  Collector and kube-state-metrics' node Pod run on every node. The development
  backend sets no requests but used about 1.2Gi at idle.

Nothing else. The deploy path creates the `nktp` namespace with the
`privileged` Pod Security level the node Collector needs, and every setting a
management cluster needs is in
[`nktp/values.yaml`](https://github.com/dsx-ai-factory/nvke/blob/main/deploy/nico/nktp/values.yaml).

## Install order

`nktp` comes straight after `capnico` and before everything that creates a
monitor:

```text
nvke-base-config -> spire-tpm-pubhashes -> spire-server-nodeport
  -> spire-oidc-provider-nodeport -> capnico -> nktp
  -> postgresql -> spire-crds -> temporal -> spire -> controller
```

- **Why this position:** the chart installs the monitoring CRDs, which the
  `temporal`, `spire` and `controller` items need for their monitors.
- **A side effect:** installing it first means NKTP also records the rest of
  the management plane coming up.
- **The cost:** a failed `nktp` install stops the path there.

## First look

On the management cluster, with its kubeconfig:

```bash
export KUBECONFIG=<site-kubeconfig>
kubectl -n nktp get pods                       # all Running
kubectl -n nktp get opentelemetrycollectors    # seven, each READY 1/1
kubectl -n nktp get certificates               # all Ready
```

To open Grafana, port-forward the backend. Bind it to localhost only: Grafana
gives anonymous visitors admin rights.

```bash
kubectl -n nktp port-forward deploy/lgtm 3000:3000 --address 127.0.0.1
```

If you are working through a jump host, forward that port over SSH rather than
binding it on the site network. The **NKTP Health** dashboard shows whether the
pipeline itself is healthy.

### NVKE reference dashboards

NVKE also ships [reference Grafana dashboards](https://github.com/dsx-ai-factory/nvke/blob/main/controller/deploy/dashboards/README.md)
as JSON files. These are starting points for investigating a site, not a
complete monitoring or alerting setup. Review their **Requirements and scope**
panels, check that your backend collects the required metrics, and adapt the
dashboards and alerts to your operating needs. An empty optional panel can mean
its metric is not scraped.

To use these dashboards, open Grafana **Dashboards → New → Import**, upload its
JSON file (or paste the JSON), and select a Prometheus datasource when prompted.
The files are not installed automatically on the management cluster. You can
import them into the bundled `lgtm` Grafana while developing, but import them
into your [real telemetry backend](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-configuration#connecting-a-real-backend)
for site operations: `lgtm` keeps data in memory and has no authentication.
See the [dashboard README](https://github.com/dsx-ai-factory/nvke/blob/main/controller/deploy/dashboards/README.md)
for the full catalog and import details.

For the full checklist, see [verification](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-verification).

## Next steps

- [Configuration](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-configuration): what the site values change, connecting
  a real backend, and overriding values safely.
- [Monitoring your service](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-monitoring-your-service): how a service gets its
  own metrics scraped.
- [Verification](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-verification): checking every signal end to end.
- [Troubleshooting](/nvke/documentation/nvke-telemetry-pipeline-nktp/management-cluster-telemetry/management-cluster-telemetry-troubleshooting): the failures seen on real management
  clusters, and their fixes.
- [Observe NVKE and its clusters](/nvke/documentation/operate/observe-nvke-and-its-clusters): health checks,
  controller logs and metrics, which NKTP complements rather than replaces.