Adding Monitoring to UCS Tools Applications#

Prerequisites#

  • You must install the Prometheus Operator before deploying the application.

    • Install the Kube Prometheus Stack.

    • Add the --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false argument to the helm install command. This enables monitoring of all PodMonitor resources.

  • Your microservices must be instrumented and implement a Prometheus metrics endpoint.

  • Your microservices must include details in the metrics section. See Microservice Metrics.

PodMonitor Resources#

UCS Application building tools generate PodMonitor Custom Resources automatically. They use information from the metrics section of each microservice. The tools add these resources to the output application Helm chart.

These PodMonitor Custom Resources tell the Prometheus server to scrape the microservice pods. For more details, see the Prometheus Operator API Reference.

Run this command to view the generated PodMonitor resources:

$ helm install --dry-run test <application-helm-chart>
...
---
# Source: myservice-test-app/templates/podmonitors.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: 'myservice-test-app-myservice-metrics-podmonitor'
  labels:
    app.kubernetes.io/instance: 'test'
    app.kubernetes.io/name: 'myservice-test-app'
spec:
  selector:
    matchLabels:
      app: myservice-myservice-deployment
      app.kubernetes.io/name: podmonitor-test-ms1
      app.kubernetes.io/version: 0.0.1
  podMetricsEndpoints:
  - port: metrics
    path: /metrics

The Helm chart deployment automatically installs these PodMonitor resources with your application.

Custom Labels for PodMonitor Resources#

You can add custom labels to PodMonitor resources. The Prometheus Operator can then select these labels.

Set the podMonitorLabels value during Helm installation:

$ helm install <release-name> <application-helm-chart> --set "podMonitorLabels.<label>=<value>"

Verifying Pod Scraping#

Check that your application and monitoring work correctly. The pod targets should appear in the Prometheus Web UI. Forward the Prometheus service port to access it:

$ kubectl get svc -l "app=kube-prometheus-stack-prometheus"
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
prometheus-kube-prometheus-prometheus   ClusterIP   10.152.183.234   <none>        9090/TCP   2m

$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus --address 0.0.0.0 9090:9090
Forwarding from 0.0.0.0:9090 -> 9090

Open http://<NODE-IP>:9090/targets in your browser. Replace <NODE-IP> with the IP address of the machine running the kubectl port-forward command. You should see the microservice pods in the targets list.

Application Information - Metrics#

Building an application creates an application information file in the output directory. This file lists all metrics exported by the microservices in your application. For more details, see Application Info.