Topology

NVKE telemetry pipeline (NKTP) deploys OpenTelemetry Collectors in two tiers. Collection-tier Collectors gather telemetry inside the cluster and forward each signal to the gateway for that signal. The gateways export to the backend. A separate Collector, the watchdog, carries the pipeline’s own telemetry.

The OpenTelemetry Operator, installed by the bundled opentelemetry-kube-stack chart, creates each Collector and Target Allocator from an OpenTelemetryCollector resource that the chart renders. Each Collector’s configuration is the native YAML under <component>.otelcol.config in values.yaml, rendered unchanged; see the design principles. The flows below are the ones those default configurations define.

Data path

Each Collector’s exporters name the next hop explicitly. For example, the node collector’s otlp/metrics exporter sends to nktp-gateway-metrics-collector:4317 over OTLP/gRPC. To change where a signal goes, change the exporter in that Collector’s configuration.

Metrics scraping

Three Target Allocators decide which Collector replica scrapes each target. The target allocation design explains why each scraping Collector requires one.

The bundled exporters are split between the first two:

• node-exporter and the kube-state-metrics DaemonSet are scraped by the node collector on the same node.
• The central kube-state-metrics Deployments and the Kubernetes control plane ServiceMonitors carry the large-target label and are scraped by the large-target pool.
• The kubelet and cAdvisor are not discovered through a Target Allocator. Each node collector scrapes its own node’s kubelet directly; see Configure kubelet metrics.

Each Collector reaches its Target Allocator over mTLS by default; see Configure cert-manager.

Self-telemetry

The watchdog receives the pipeline’s own telemetry at nktp-watchdog-collector:4317 over OTLP/gRPC with mTLS, except allocator metrics which use plaintext OTLP/gRPC on port 4320:

• Every Collector except the watchdog exports its own metrics, logs and traces through service.telemetry to the sidecar in its Pod, and the sidecar exports them there. See Self-telemetry sidecars.
• Every Target Allocator pushes its metrics to nktp-watchdog-collector:4320.
• The node collector reads Target Allocator and Operator container logs on its node and sends them there through its logs/targetallocator and logs/operator pipelines. Its pod-log reader skips these files and the NKTP Collectors’ own pod logs, so they are not also sent through the logs gateway.
• The watchdog scrapes the Operator’s metrics endpoint itself.

The watchdog forwards what it receives directly to the backend. Its own telemetry goes through its sidecar to the backend. Self-telemetry therefore does not pass through the gateways. See Backpressure for how self-telemetry is buffered separately from collection.

Supporting components

• OpenTelemetry Operator (nktp-operator) and the OpenTelemetry and Prometheus CRDs, from opentelemetry-kube-stack. See Managed dependencies.
• kube-state-metrics: a DaemonSet for Pods scheduled on each node, plus central Deployments grouped by resource type. See kube-state-metrics.
• node-exporter: host metrics from every node.
• metrics-server, off by default: the resource metrics API that the autoscalers read.
• otel-lgtm: the default backend, for evaluation only. See otel-lgtm.

Scaling

The gateways, the large-target pool, and the watchdog scale horizontally on CPU and memory utilization, which requires the Kubernetes resource metrics API. The node collector scales with the number of nodes. The cluster event reader runs two replicas, one active and one standby; more replicas do not increase watch throughput. See Size resources.