Management cluster telemetry: troubleshooting
Failures seen while bringing up NVKE telemetry pipeline (NKTP) on the management cluster, with their causes and fixes. For checks that tell whether anything is wrong, see verification.
The Operator crash-loops with “cannot set blockOwnerDeletion”
Symptom: nktp-operator restarts repeatedly, and helm install with
--wait never finishes. Its log shows that creating its NetworkPolicy was
forbidden:
cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on.
Cause:
- The Operator creates a NetworkPolicy for its own Pod, owned by its
Deployment, with
blockOwnerDeletion: true. - Every cluster NVKE builds runs the
OwnerReferencesPermissionEnforcementadmission plugin (defaultEnabledAdmissionPluginsincontroller/internal/kubeadmconfig). It allows that field only if the requester canupdatethe owner’sdeployments/finalizers. - The Operator’s own install manifests grant that permission. The OpenTelemetry Operator Helm chart, which NKTP bundles, does not.
Clusters without that plugin, Kind included, never show the problem.
Fix: the site values add the grant through
opentelemetry-kube-stack.opentelemetry-operator.extraObjects. If the
Operator crash-loops anyway, check that extraObjects still has both
entries. A values file that sets extraObjects replaces the whole list and
drops the grant.
The install stops at nvke-base-config: “{{ .ClusterName }} has no value”
Cause: the install has no site name. With --sitespec, the site spec’s clusterName is empty; without one, the BootstrapConfig passed with --config has no ClusterName.
The nktp-cluster ConfigMap is rendered from it, and the renderer refuses an
empty value rather than labeling every signal with nothing.
Fix: set clusterName: <site-name> in the site spec, or add "ClusterName": "<site-name>" to the BootstrapConfig.
The nktp item cannot download the chart
Cause: The install pulls the chart from https://helm.ngc.nvidia.com/nvidia,
which redirects the index and the archive to one of NGC’s download hosts,
currently xfiles.ngc.nvidia.com, and the node running it has no egress to one
of them.
Fix:
- Allow egress to
helm.ngc.nvidia.comand NGC’s download hostsfiles.ngc.nvidia.com,xfiles.ngc.nvidia.comandxlfiles.ngc.nvidia.com(NGC network protocols), or - Install from a release.
nvke-deploy-local.sh install --release …puts the packaged nktp chart the release carries undercharts/, and installs from that archive.
The watchdog or a gateway stays Pending
Cause: Their exporter queues are on PersistentVolumes. Either the cluster has no default StorageClass, so the claims are never bound, or the claim is bound to a node-local volume on a node that is gone.
Fix: check kubectl -n nktp get pvc. An unbound claim needs a default
StorageClass. A claim bound to a lost node holds queued telemetry that is
already unreachable. Delete the claim without waiting, then the Pod:
The claim stays until no Pod uses it, and a Pod on a lost node is never
confirmed stopped, so it has to be deleted with --force. Once it is gone, the
claim is removed and the StatefulSet recreates both on a node that is up.
Collectors don’t appear, or the Operator’s webhook fails
Cause: cert-manager isn’t serving yet, so the Operator’s webhook certificate or the Target Allocators’ mTLS certificates aren’t issued.
Fix: check kubectl -n nktp get certificates and cert-manager’s own
Pods. On the management cluster, clusterctl init installs cert-manager before
the deploy path runs; if it was skipped, run it.
Scrape targets are down
promql 'count by (job) (up == 0)', from
verification, lists the failing jobs.
The backend only receives what the Collectors push, so it can’t say why. The
Collector that scraped a target logs each failure, with the error and the
target’s labels:
No API audit logs
Cause:
- Wrong path: the receiver reads a path that doesn’t exist. NVKE writes the
audit log to
/var/log/kubernetes_audit/audit.log, and the site values pointfile_log/kube-api-auditthere. A cluster that writes it elsewhere needs the same override. - Logging off: API audit logging is off, so there is no file at all.
No journal logs
Cause: the node keeps its journal in /run/log/journal, which is
in-memory only, rather than /var/log/journal.
Fix: mount /run/log/journal and point the journald receiver’s
directory at it; see
configuration. Collection starts
with new entries, so an idle node can take a while to show any.
The autoscalers show no metrics
Cause: nothing serves metrics.k8s.io, or two providers are fighting over
it.
Fix: the site values turn metrics-server on because a management cluster has
no other provider. If something else later installs a provider, turn one of
the two off. The v1beta1.metrics.k8s.io APIService has a single owner, and
two providers make it flip between them.
Metrics are rejected by a real backend
Cause: Target Allocator 0.159.0 exports some counters as delta even though cumulative temporality is requested.
Fix: make the backend accept delta metrics, as Prometheus’s
otlp-deltatocumulative feature does for the development backend; see
configuration.
SPIRE OIDC discovery rejects requests by IP after a hand upgrade
Symptom:
https://<site-ip>:30443/.well-known/openid-configuration stops returning
200 after someone upgraded the spire release by hand.
Cause: the upgrade used a deploy/nico overlay with --reuse-values. The
overlay’s spiffe-oidc-discovery-provider.config.additionalDomains list
replaced the per-site one, which the deploy path had passed last with the
site IP in it.
Fix:
helm rollback spire <last good revision> -n spire-server- Apply only the change you meant to make, with
--set.
In general, change a site through the deploy path rather than release by release; it passes the per-site values last.
Chart upgrades don’t pick up new CRD fields
Cause: the chart ships the OpenTelemetry and Prometheus CRDs in crds/,
and Helm installs those once and never upgrades them. After a chart upgrade
that bumps the Operator, an existing site keeps the old CRDs. The API server
then silently drops fields they don’t know from new custom resources.
Fix: after such an upgrade, apply the CRDs from the new chart package yourself:
Some images still pull from public registries behind a mirror
Cause: global.imageRegistry moves the Collector and Target Allocator
images to the mirror, but three images don’t read it: metrics-server’s, the
OpenTelemetry Operator’s, and the kubectl image of opentelemetry-kube-stack’s
uninstall hook.
Fix: on an air-gapped site, point those three at the mirror through their own values:
The chart fails with image tag ... must be a quoted string
Cause: a numeric-looking tag such as tag: 1.0 was written unquoted, so
YAML reads it as a number.
Fix: quote it: tag: "1.0".