Title: Adding Monitoring to UCS Tools Applications#

URL Source: https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html

Published Time: Thu, 30 Oct 2025 07:23:03 GMT

Markdown Content:
Prerequisites[#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#prerequisites "Link to this heading")
----------------------------------------------------------------------------------------------------------------------

*   You must install the **Prometheus Operator** before deploying the application.

> *   Install the [Kube Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack).
> 
>     *   Add the `--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false` argument to the `helm install` command. This enables monitoring of all **PodMonitor** resources.

*   Your microservices must be instrumented and implement a Prometheus metrics endpoint.

*   Your microservices must include details in the `metrics` section. See [Microservice Metrics](https://docs.nvidia.com/ucf/2.10.0/text/UCS_MsBuilder_Input.html#ucf-msb-input-manifest-metrics).

PodMonitor Resources[#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#podmonitor-resources "Link to this heading")
------------------------------------------------------------------------------------------------------------------------------------

UCS Application building tools generate **PodMonitor** Custom Resources automatically. They use information from the `metrics` section of each microservice. The tools add these resources to the output application Helm chart.

These **PodMonitor** Custom Resources tell the Prometheus server to scrape the microservice pods. For more details, see the [Prometheus Operator API Reference](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMonitor).

Run this command to view the generated **PodMonitor** resources:

$ helm install --dry-run test <application-helm-chart>
...
---
# Source: myservice-test-app/templates/podmonitors.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: 'myservice-test-app-myservice-metrics-podmonitor'
  labels:
    app.kubernetes.io/instance: 'test'
    app.kubernetes.io/name: 'myservice-test-app'
spec:
  selector:
    matchLabels:
      app: myservice-myservice-deployment
      app.kubernetes.io/name: podmonitor-test-ms1
      app.kubernetes.io/version: 0.0.1
  podMetricsEndpoints:
  - port: metrics
    path: /metrics

The Helm chart deployment automatically installs these **PodMonitor** resources with your application.

Custom Labels for PodMonitor Resources[#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#custom-labels-for-podmonitor-resources "Link to this heading")
------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You can add custom labels to **PodMonitor** resources. The **Prometheus Operator** can then select these labels.

Set the `podMonitorLabels` value during Helm installation:

$ helm install <release-name> <application-helm-chart> --set "podMonitorLabels.<label>=<value>"

Verifying Pod Scraping[#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#verifying-pod-scraping "Link to this heading")
----------------------------------------------------------------------------------------------------------------------------------------

Check that your application and monitoring work correctly. The pod targets should appear in the Prometheus Web UI. Forward the Prometheus service port to access it:

$ kubectl get svc -l "app=kube-prometheus-stack-prometheus"
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus-kube-prometheus-prometheus ClusterIP 10.152.183.234 <none> 9090/TCP 2m

$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus --address 0.0.0.0 9090:9090
Forwarding from 0.0.0.0:9090 -> 9090

Open `http://<NODE-IP>:9090/targets` in your browser. Replace `<NODE-IP>` with the IP address of the machine running the `kubectl port-forward` command. You should see the microservice pods in the targets list.

Application Information - Metrics[#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#application-information-metrics "Link to this heading")
------------------------------------------------------------------------------------------------------------------------------------------------------------

Building an application creates an application information file in the output directory. This file lists all metrics exported by the microservices in your application. For more details, see [Application Info](https://docs.nvidia.com/ucf/2.10.0/text/UCS_AppDevelopment.html#app-info).

Links/Buttons:
- [#](https://docs.nvidia.com/ucf/2.10.0/text/UCS_app_monitoring.html#application-information-metrics)
- [Kube Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
- [Microservice Metrics](https://docs.nvidia.com/ucf/2.10.0/text/UCS_MsBuilder_Input.html#ucf-msb-input-manifest-metrics)
- [Prometheus Operator API Reference](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMonitor)
- [Application Info](https://docs.nvidia.com/ucf/2.10.0/text/UCS_AppDevelopment.html#app-info)