> This page is for NVIDIA Kubernetes Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvke/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvke/_mcp/server.

# Node collector

The node Collector (`daemonset`) runs on every node. It collects the node's logs
and metrics and forwards OTLP that workloads send to it.

## OTLP endpoint

The node Collector accepts OTLP logs, metrics, and traces at
`nktp-daemonset-collector:4317` over gRPC and `http://nktp-daemonset-collector:4318`
over HTTP. The Service routes traffic only to the Collector on the sender's node.
If no local Collector is ready, traffic fails rather than reaching another node.
The Collector sets `k8s.node.name` and forwards each signal to its corresponding
gateway. Qualify the Service name with the NVKE telemetry pipeline (NKTP) namespace when
sending from another namespace.

## Pod logs and metrics

The DaemonSet reads pod logs from `/var/log/pods` and sends them to the logs
gateway. A dedicated Target Allocator assigns monitored metric endpoints only
to collectors on the same node. Metrics go to
`nktp-gateway-metrics-collector:4317` over OTLP/gRPC, which requires the metrics gateway.
Per-process host metrics are not collected by default. Node exporter provides
host metrics, and the kubelet cAdvisor scrape provides per-container resource
use. See [Enable host process metrics](/nvke/documentation/nvke-telemetry-pipeline-nktp/guides/enable-host-process-metrics)
to enable the `hostmetrics` process scraper.

Each DaemonSet collector also scrapes the kubelet `/metrics` and
`/metrics/cadvisor` endpoints on its own node. See
[Configure kubelet metrics](/nvke/documentation/nvke-telemetry-pipeline-nktp/guides/configure-kubelet-metrics) to change these scrapes, verify
kubelet certificates, or run NKTP alongside kube-prometheus-stack.

## Journal

The DaemonSet also collects systemd journal entries at info priority and above
and forwards them to the logs gateway. Nodes must provide `/usr/bin/journalctl`
and `/etc/machine-id`, with journals in `/var/log/journal`. The Collector uses
selective read-only host mounts and
the `SYS_CHROOT` capability to run the host executable. The `/lib64` host volume
is also mounted at `/hostfs/usr/lib64` for Red Hat library paths. Collection
starts with new entries. Journal cursors are not persisted across Collector restarts.

Override `root_path`, `journalctl_path`, and `directory` under
`daemonset.otelcol.config.receivers.journald` for a different host layout.
Update `daemonset.kubernetes.volumes` and `volumeMounts` to match, retaining
the other mounts when replacing either list. For runtime journals, add a
`/run/log/journal` host volume mounted at `/hostfs/run/log/journal` and set the
receiver directory to `/run/log/journal`.

See [Log classification](#log-classification) for the default audit and
control plane log sources.

## Log classification

NKTP classifies logs by source using the `nvidia.telemetry.class` resource
attribute. All records from a classified source receive the same class,
regardless of their content. The class is forwarded to the logs gateway for
use in routing rules.

| Source | Classification | `service.name` | Recognition |
| --- | --- | --- | --- |
| sudo and SSH journal entries | `system.auth` | `journald` | Journal producer identifier is `sudo` or `sshd`. |
| Linux authentication files | `system.auth` | `auth` | All records from `/var/log/auth.log` or `/var/log/secure`. |
| Linux auditd | `system.audit` | `auditd` | All records from `/var/log/audit/audit.log`, including authentication records. |
| Kubernetes control plane logs | `k8s.control_plane` | Not set by NKTP | Pods in `kube-system` with matching pod-name prefixes and container names for `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, or `etcd`. |
| Kubernetes API audit logs | `k8s.api.audit` | `kube-apiserver` | Records from `/var/log/kubernetes/audit/audit.log`. API audit logging must be enabled on the API server. |

Other locally collected logs remain unclassified. All journal records use
`journald` as `service.name`. Shared authentication files use `auth`. Service
names reach the logs gateway alongside the classification.

## Self-telemetry

The collector's own metrics, logs and traces go through its
[sidecar](/nvke/documentation/nvke-telemetry-pipeline-nktp/architecture/self-telemetry-sidecars) to watchdog.
Target Allocator telemetry is sent to watchdog.
Each collector also reads Target Allocator stdout and stderr logs on its node
in the release namespace and sends them to watchdog through `logs/targetallocator`.
Operator manager container logs go to watchdog through `logs/operator`.

## Configuration

Use `daemonset.otelcol.config` for Collector configuration and
`daemonset.kubernetes` for Kubernetes workload configuration. The pod-log mount
is read-only, but host file permissions require the collector to run as root.
The file reader follows new records and does not persist offsets across restarts.
Retain `GOMEMLIMIT` and `GOGC` when replacing its environment list.
The chart adds `NODE_NAME`, `POD_NAMESPACE`, and `OTEL_K8S_NODE_IP` if the list
omits them.

## Target Allocator

Setting `daemonset.kubernetes.targetAllocator.enabled` to `false` will cause Helm
rendering to fail.

Monitoring CRs labeled `nktp.nvidia.io/large-target: "true"` are excluded.
Unlabeled monitors remain eligible. Targets without a matching collector node
remain unassigned.

The Target Allocator rejects ServiceMonitor and PodMonitor endpoints that reference
local bearer-token or TLS files in the Collector container.

### Monitor credentials

To grant the allocator access to Secrets and CA ConfigMaps referenced by monitors, list
the namespaces containing those monitors and their referenced objects:

```yaml
daemonset:
  kubernetes:
    targetAllocator:
      prometheusCR:
        secretNamespaces: [metrics-monitors]
```

`daemonset.rbac.create` defaults to `true`. For each explicitly
listed namespace, Helm creates a Role granting Secrets `get/list/watch` and
ConfigMaps `get`, bound to the allocator ServiceAccount in the release namespace.
The binding honors `daemonset.kubernetes.targetAllocator.serviceAccount`.
Omitting the list or setting it to `[]` creates no namespace RBAC. Set
`daemonset.rbac.create: false` to manage these grants externally.

Grants cover all Secrets in each listed namespace. Use namespaces with trusted
monitor authors. Secret and CA references resolve in the monitor's namespace.
The namespaces must already exist. `secretNamespaces` controls Secret watching.
Configure monitor and namespace selectors separately.