Monitoring & Observability

The NVIDIA Cluster Agent and Operator provide built-in monitoring through Prometheus metrics, structured logging, and OpenTelemetry tracing.

Monitoring Data

Metrics

Prerequisites

To use the PodMonitor and ServiceMonitor examples below, you must first install the Prometheus Operator. Follow the Prometheus Operator installation guide to set this up in your cluster.

The cluster agent and operator emit Prometheus-style metrics. The following metric labels are available by default. The full list of available metrics are updated regularly and therefore not listed.

Cluster maintainers can scrape the available metrics. See a full example of how to do this with an OpenTelemetry Collector in the cluster here.

Use the following examples of a PodMonitor for NVCA Operator and ServiceMonitor for NVCA for reference:

Sample NVCA Operator PodMonitor

1
  apiVersion: monitoring.coreos.com/v1
2
  kind: PodMonitor
3
  metadata:
4
      labels:
5
          app.kubernetes.io/component: metrics
6
          app.kubernetes.io/instance: prometheus-agent
7
          app.kubernetes.io/name: metrics-nvca-operator
8
          jobLabel: metrics-nvca-operator
9
          release: prometheus-agent
10
          prometheus.agent/podmonitor-discover: "true"
11
      name: metrics-nvca-operator
12
      namespace: monitoring
13
  spec:
14
      podMetricsEndpoints:
15
      - port: http
16
        scheme: http
17
        path: /metrics
18
      jobLabel: jobLabel
19
      selector:
20
          matchLabels:
21
              app.kubernetes.io/name: nvca-operator
22
      namespaceSelector:
23
          matchNames:
24
          - nvca-operator

Sample NVCA ServiceMonitor

1
  apiVersion: monitoring.coreos.com/v1
2
  kind: ServiceMonitor
3
  metadata:
4
      labels:
5
          app.kubernetes.io/component: metrics
6
          app.kubernetes.io/instance: prometheus-agent
7
          app.kubernetes.io/name: metrics-nvca
8
          jobLabel: metrics-nvca
9
          release: prometheus-agent
10
          prometheus.agent/servicemonitor-discover: "true"
11
      name: prometheus-agent-nvca
12
      namespace: monitoring
13
  spec:
14
      endpoints:
15
      - port: nvca
16
      jobLabel: jobLabel
17
      selector:
18
          matchLabels:
19
              app.kubernetes.io/name: nvca
20
      namespaceSelector:
21
          matchNames:
22
          - nvca-system

Logs

Both the Cluster Agent and Cluster Agent Operator emit logs locally by default.

Local logs for the NVIDIA Cluster Agent Operator can be obtained via kubectl:

$
  kubectl logs -l app.kubernetes.io/instance=nvca-operator -n nvca-operator --tail 20

Similarly, NVIDIA Cluster Agent logs can be obtained with the following command via kubectl:

$
  kubectl logs -l  app.kubernetes.io/instance=nvca -n nvca-system --tail 20

Tracing

The NVIDIA Cluster Agent provides OpenTelemetry integration for exporting traces and events to compatible collectors. As of agent version 2.0, the only supported collector receiver is Lightstep.

Enable Tracing with Lightstep

1. Get your Lightstep access token from the Lightstep UI and set to LS_ACCESS_TOKEN environment variable.
2. Get the NVCF cluster name:

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"

3. Apply the tracing configuration:

$
  kubectl patch nvcfbackends.nvcf.nvidia.io -n nvca-operator "$nvcf_cluster_name"  --type=merge --patch="{\"spec\":{\"overrides\":{\"featureGate\":{\"otelConfig\":{\"exporter\":\"lightstep\",\"serviceName\":\"nvcf-nvca\",\"accessToken\":\"${LS_ACCESS_TOKEN}\"}}}}}"