Prometheus Monitoring

Overview

dps-server exports Prometheus metrics through a dedicated HTTP metrics service. Use the chart-managed ServiceMonitor for an in-cluster Prometheus deployment. For an external Prometheus deployment, expose the metrics service through a customer-managed network and security layer.

This page covers DPS metrics discovery and scraping. It does not cover Prometheus installation, alerting, or dashboard configuration.

Metrics Endpoint

The default chart creates the following endpoint:

Property	Value
Service labels	app: dps-server and app.kubernetes.io/component: metrics
Service name	The chart-generated metrics Service, such as dps-server-metrics for a release named dps with default naming
Service port	dps-metrics, port 9090, configurable through dps.service.metrics.port
Container port	9090, configurable through dps.containerPorts.metrics.port
Path	/metrics
Scheme	http. DPS does not provide TLS on the metrics listener.
Authentication	Disabled by default. Protected mode requires an OpenID Connect (OIDC) Bearer token with the dps:metrics:read scope.

Use the Chart-Managed ServiceMonitor

The Prometheus Operator and the monitoring.coreos.com/v1 ServiceMonitor custom resource definition must exist in the cluster. Prometheus must discover the namespace that contains the ServiceMonitor, and the ServiceMonitor must select the DPS release namespace.

Enable the resource in your values overlay:

dps:
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s
    scheme: http
    additionalLabels:
      release: kube-prometheus-stack

Many Prometheus installations select ServiceMonitor resources by label. Check your Prometheus instance’s serviceMonitorSelector to determine which labels it requires, and set them through dps.serviceMonitor.additionalLabels. Replace the example release label with the labels required by your Prometheus instance, then apply the chart:

helm upgrade --install dps ngc/dps \
  --namespace dps \
  --values values.yaml

To create the ServiceMonitor in another namespace, set its namespace and the namespace that contains the metrics Service:

dps:
  serviceMonitor:
    enabled: true
    namespace: monitoring
    namespaceSelector:
      matchNames:
        - dps

The selected chart’s values.yaml is authoritative for the available dps.serviceMonitor settings. Use Inspect the Versioned Configuration to inspect the version you are deploying.

Protect the Metrics Endpoint

Set dps.metrics.authentication.enabled: true to require an OIDC Bearer token with the dps:metrics:read scope. Protected metrics require OIDC, cannot be combined with LDAP, and require an encrypted scrape path. When the chart-managed ServiceMonitor is enabled, configure exactly one credential profile. OAuth 2.0 client credentials are preferred because Prometheus refreshes the token.

Enabling dps.auth.oidc.enabled switches DPS API authentication to external OIDC and disables DPS /Token issuance. Treat this as an authentication mode change, not only a metrics setting. The ServiceMonitor OAuth 2.0 field requires Prometheus 2.27.0 or later.

Protected metrics with OAuth 2.0 client credentials

dps:
  auth:
    oidc:
      enabled: true
      issuer: "https://idp.example.com"
      jwksUrl: "https://idp.example.com/.well-known/jwks.json"
      audience: "dps-api"
  ldap:
    enabled: false
  metrics:
    authentication:
      enabled: true
  serviceMonitor:
    enabled: true
    oauth2:
      clientId:
        secret:
          name: "prometheus-oidc"
          key: "client-id"
      clientSecret:
        name: "prometheus-oidc"
        key: "client-secret"
      tokenUrl: "https://idp.example.com/oauth2/token"

Create prometheus-oidc through your approved Secret workflow. Create each referenced Secret in the namespace that contains the ServiceMonitor, and ensure that the Prometheus Operator can read it. The chart adds the dps:metrics:read scope to the OAuth 2.0 request.

As an alternative, replace the preceding oauth2 block with a reference to a customer-managed access token:

dps:
  serviceMonitor:
    authorization:
      type: Bearer
      credentials:
        name: "prometheus-token"
        key: "access-token"

Do not configure both credential profiles. Replace a customer-managed token before it expires.

The direct metrics listener remains HTTP in protected mode. Carry Bearer tokens only over an encrypted path supplied by a transparent service mesh, proxy, or other layer that fronts the selected Service. Set dps.serviceMonitor.scheme to https and configure tlsConfig only when that layer presents HTTPS at the endpoint selected by the ServiceMonitor.

Verify the ServiceMonitor

Confirm that the Service and ServiceMonitor exist. Adjust the namespaces when the ServiceMonitor is deployed separately:

kubectl get service --namespace dps \
  --selector app=dps-server,app.kubernetes.io/component=metrics
kubectl get servicemonitor --namespace dps
kubectl describe servicemonitor --namespace dps dps-server-metrics

In the Prometheus UI, open Status > Targets and confirm that the dps-server-metrics target is UP.

When metrics authentication is disabled, you can also test the direct endpoint from the cluster. The following service name assumes that the Helm release is named dps:

kubectl run --namespace dps --rm -it --restart=Never curl \
  --image=curlimages/curl -- \
  curl -sS http://dps-server-metrics:9090/metrics

For protected metrics, verify the scrape through Prometheus. Do not place an access token on a command line.

Use a Standalone ServiceMonitor

If another team manages ServiceMonitor objects, apply a resource that selects the chart-managed metrics Service. Adjust the chart name, release instance, and namespaces for your deployment.

Standalone ServiceMonitor example

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dps-server-metrics
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: dps-server
      app.kubernetes.io/name: dps
      app.kubernetes.io/instance: dps
      app.kubernetes.io/component: metrics
  namespaceSelector:
    matchNames:
      - dps
  endpoints:
    - port: dps-metrics
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s
      scheme: http

When metrics authentication is enabled, add exactly one credential profile under the endpoint. A standalone ServiceMonitor must request the metrics scope explicitly.

OAuth 2.0 endpoint configuration

oauth2:
  clientId:
    secret:
      name: prometheus-oidc
      key: client-id
  clientSecret:
    name: prometheus-oidc
    key: client-secret
  tokenUrl: https://idp.example.com/oauth2/token
  scopes:
    - dps:metrics:read

As an alternative, add a customer-managed Bearer token profile:

authorization:
  type: Bearer
  credentials:
    name: prometheus-token
    key: access-token

Create referenced Secrets in the ServiceMonitor namespace, and ensure that the Prometheus Operator can read them. Configure the DPS authentication settings described in Protect the Metrics Endpoint before you enable either profile.

Scrape Metrics From Outside the Cluster

The chart’s dps.ingress and dps.gateway resources route to the gRPC Service. They do not expose /metrics. The dedicated metrics Service is a ClusterIP Service by default and listens on HTTP port 9090.

Expose the metrics Service through a path appropriate for your environment:

• When metrics authentication is disabled, a customer-managed NodePort or LoadBalancer Service can select the dps-server pods and set targetPort to dps-metrics. Restrict access to Prometheus with firewall rules, NetworkPolicy, load balancer source ranges, or equivalent controls.

• Across routed or shared networks, use an Ingress, proxy, or service mesh that terminates TLS and restricts access. An encrypted path is required when metrics authentication is enabled.

The following Ingress example terminates TLS and limits source addresses:

Metrics Ingress example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dps-server-metrics
  namespace: dps
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "<prometheus-source-cidr>"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - dps-metrics.example.com
      secretName: dps-metrics-tls
  rules:
    - host: dps-metrics.example.com
      http:
        paths:
          - path: /metrics
            pathType: Prefix
            backend:
              service:
                name: dps-server-metrics
                port:
                  name: dps-metrics

Configure external Prometheus with the exposed endpoint and the trust material for the customer-managed TLS layer. The following example assumes that the Ingress certificate chains to a certificate authority that Prometheus already trusts. For a private certificate authority, configure tls_config.ca_file as described in the Prometheus configuration reference.

External Prometheus scrape example

scrape_configs:
  - job_name: dps-server
    metrics_path: /metrics
    scheme: http
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - dps-metrics.example.com
        labels:
          service: dps-server
          cluster: my-dps-cluster

When metrics authentication is enabled, configure the external Prometheus job to obtain or supply the required OIDC Bearer token. Prefer OAuth 2.0 client credentials so Prometheus refreshes the token automatically. Add the following configuration to the scrape job and mount the client secret file through your approved Secret workflow:

oauth2:
  client_id: "<prometheus-client-id>"
  client_secret_file: /etc/prometheus/secrets/dps/client-secret
  token_url: https://idp.example.com/oauth2/token
  scopes:
    - dps:metrics:read

Verify the unprotected Ingress endpoint from the Prometheus host:

curl -sS https://dps-metrics.example.com/metrics

For protected metrics, verify the scrape through Prometheus instead of placing an access token on a command line.

Troubleshooting

Symptom	Likely cause	Action
ServiceMonitor exists but Prometheus shows no target	Prometheus does not select its labels or namespace.	Match dps.serviceMonitor.additionalLabels and namespace settings to the Prometheus selectors.
Target is DOWN with a TLS error	The customer-managed proxy, Ingress, or mesh certificate cannot be validated.	Configure scheme and tlsConfig for that layer. dps.serverTLS does not configure metrics TLS.
Target is DOWN or returns 404	The scraper uses the UI Service, port 8000, or the wrong scheme.	Select the dedicated metrics Service and use dps-metrics over HTTP unless a customer TLS layer is present.
Scrape returns 401 Unauthorized	The Bearer token is missing, malformed, expired, or invalid.	Verify the configured credential profile, referenced Secret, and OIDC settings.
Scrape returns 403 Forbidden	The token does not include dps:metrics:read.	Issue a token with the required scope. The chart-managed OAuth 2.0 profile requests it automatically.
External Prometheus cannot connect	The network path, firewall, or external endpoint is unavailable.	Test the exact exposed endpoint and inspect the customer-managed network resource.

Related Pages

• Deployment and Configuration Guide – chart installation and external endpoint configuration.

• Telemetry and Observability – how DPS ingests live power data, which is separate from Prometheus metrics.