Telemetry and Observability

DPS publishes operational metrics in Prometheus text format for power, topology, allocation, device, and service state. Use the metrics to observe the control plane, investigate power behavior, and build dashboards and alerts.

This page covers the DPS metrics service and its metric reference. The DPS Power Steering Controller publishes a separate endpoint when it is deployed. Refer to DPS Sample Dashboards for ready-to-import dashboard examples.

Configure and Access Metrics

Use Prometheus Monitoring to expose and scrape the DPS metrics service. That guide owns Helm ServiceMonitor configuration, external Prometheus access, and metrics authentication.

The chart exposes /metrics through the dedicated <release>-server-metrics Service on port 9090 (service port name dps-metrics). The direct listener uses HTTP and is separate from the HTTP API Service. Inspect a release named dps locally:

kubectl -n dps port-forward service/dps-server-metrics 9090:9090
curl -s http://127.0.0.1:9090/metrics

Use the generated Service name when the Helm release has a different name. If metrics authentication is enabled, use an OIDC Bearer token with the dps:metrics:read scope as described in Prometheus Monitoring.

Metric labels can include operational identifiers such as hostnames, device IDs, and resource-group names. Restrict endpoint access to the monitoring identity.

Metric Conventions

The reference lists the metric family names exposed by Prometheus. Counter metric families end in _total, and histograms use _bucket, _sum, and _count. For example, query redfish_request_duration_milliseconds_bucket for a Redfish duration histogram. Dots in OpenTelemetry attribute names become underscores.

A series can be absent until its collector runs or its related operation occurs. Do not treat absent device or node telemetry as zero; it can indicate an inactive topology, a disabled collector, or a missing source reading.

Zapp-Backed Telemetry and Excursion Detection

When Zapp-backed telemetry is enabled and configured, DPS receives real-time power readings from Zapp for excursion detection and automated mitigation. Without Zapp-backed telemetry, DPS continues to operate, but features that require live power readings do not receive them.

Note

For setup details, refer to Configure Zapp Telemetry.

Zapp provides a sensor manifest. DPS binds manifest sensor IDs to the configured host set, subscribes to matching sensors, aggregates readings through the topology tree, and compares each device’s power to its configured operational limit.

DPS keeps Zapp-backed telemetry current as hosts and streams change. It resubscribes when the configured host set changes.

DPS Metric Reference

Most metric families appear only after DPS records related data.

Exporter Metadata

Metric family	Type	Unit	Attributes	Usage and availability
target_info	Gauge	None	service_name, telemetry_sdk_language, telemetry_sdk_name, telemetry_sdk_version	Shows the service and OpenTelemetry SDK that produced the metrics. The value is 1. A deployment can add more resource labels.

gRPC API

Metric family	Type	Unit	Attributes	Usage and availability
grpc_requests_total	Counter	None	component, operation, api_family, grpc_service, grpc_method, response_code, grpc_status_code, status, status_class, source_component (conditional), source_direction (conditional), source_confidence (conditional)	Shows the request rate and result for each API and method. Source attributes appear only when DPS can determine the request source.
grpc_duration_milliseconds	Histogram	ms	component, operation, api_family, grpc_service, grpc_method, response_code, grpc_status_code, status, status_class, source_component (conditional), source_direction (conditional), source_confidence (conditional)	Shows request latency for each API, method, and result. Source attributes appear only when DPS can determine the request source.

Topology and Resource Groups

API metrics appear after DPS runs an operation. The periodic topology_service_* metrics start after topology activation.

Metric family	Type	Unit	Attributes	Usage and availability
topology_api_activation_duration_milliseconds	Histogram	ms	topology_name	Shows how long each topology takes to activate.
topology_api_deactivation_duration_milliseconds	Histogram	ms	topology_name	Shows how long each topology takes to deactivate.
topology_entity_hostname	Gauge	None	entity_name, hostname	For an entity with a Redfish hostname, records 1 when created and 0 when deleted.
topology_coordination_duration_milliseconds	Histogram	ms	operation	Shows time spent preparing and validating topology activation before device work starts.
topology_deactivation_coordination_duration_milliseconds	Histogram	ms	operation	Shows time spent preparing topology deactivation before device work starts.
resource_group_coordination_duration_milliseconds	Histogram	ms	operation, strict_mode (conditional)	Shows the duration of resource-group work identified by operation. strict_mode appears only for activation.
resource_group_deactivation_coordination_duration_milliseconds	Histogram	ms	operation	Shows the total resource-group deactivation time, including device responses and power give-back.
resource_group_update_coordination_duration_milliseconds	Histogram	ms	operation	Shows the total resource-group update time, including device requests and responses.
resource_group_api_create_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long each resource group takes to create.
resource_group_api_add_resources_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long DPS takes to add devices to each resource group.
resource_group_api_activate_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long each resource group takes to activate.
resource_group_api_update_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long each resource-group update takes.
resource_group_api_remove_resources_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long DPS takes to remove devices from each resource group.
resource_group_api_delete_duration_milliseconds	Histogram	ms	resource_group_name	Shows how long each resource group takes to delete.
resource_group_count	Gauge	None	resource_group_name, policy_name, dpm_enable, prs_enabled, workload_profile_ids	Records a resource group when it is created (1) or deleted (0). Labels contain the settings recorded at that time.
resource_group_devices	Gauge	None	resource_group_name, node	For an inactive resource group, records 1 when a node is added and 0 when it is removed or the group is deleted.
topology_service_device_count	Gauge	None	device_type	Shows the number of configured devices of each type.
topology_service_active_device_count	Gauge	None	None	Shows the number of active devices.
topology_service_active_status	Gauge	None	topology_name	Shows whether each topology is active. A value of 1 means active, and 0 means inactive.
topology_service_resource_group_active_count	Gauge	None	None	Shows the number of active resource groups.
topology_service_resource_group_active_dpm_enabled_count	Gauge	None	None	Shows how many active resource groups have Dynamic Power Management enabled.
topology_service_resource_group_active_prs_enabled_count	Gauge	None	None	Shows how many active resource groups have Power Reservation Steering enabled.
topology_service_resource_group_device_count	Gauge	None	None	Shows the total number of device memberships across resource groups.
topology_service_policy_count	Gauge	None	None	Shows the number of policies available to the topology service.
topology_service_total_allocated_power_watts	Gauge	W	None	Shows how much power is allocated. Compare it with total capacity to find available power.
topology_service_total_capacity_watts	Gauge	W	None	Shows the total power capacity available for allocation.
topology_service_device_power_capacity_watts	Gauge	W	device_type, device_id	Shows the power capacity of each device.
topology_service_device_power_load_watts	Gauge	W	device_type, device_id	Shows the allocated power load of each device.
topology_service_device_instance_info	Gauge	None	device_id, device_type, device_model, is_active, constraint (conditional), power_factor (conditional)	Shows each device’s identity, activity, optional power constraint, and power factor. The value is 1.
topology_service_device_instance_power_capacity_watts	Gauge	W	device_id, device_type, device_model, is_active, constraint (conditional), power_factor (conditional), gpu_id (GPU series only), gpu_uuid (GPU series only)	Shows power capacity for each device and GPU when detailed data is available.
topology_service_device_instance_power_load_watts	Gauge	W	device_id, device_type, device_model, is_active, constraint (conditional), power_factor (conditional), gpu_id (GPU series only), gpu_uuid (GPU series only)	Without gpu_id and gpu_uuid, the metric reports the device’s provisioned load. With GPU labels, it reports each GPU’s minimum power; it does not report current or requested GPU power.
topology_service_active_resource_group_info	Gauge	None	resource_group_id, policy_name, dpm_enable, prs_enabled, workload_profile_ids	Shows policy and power-management settings for resource groups tracked by the topology service. The value is 1.
topology_service_active_resource_group_priority	Gauge	None	resource_group_id	Shows each active resource group’s priority. Lower values have higher importance.
topology_service_desired_allocation_info	Gauge	None	allocation_type, total_capacity, total_allocated	Shows the desired total capacity and allocated power. The value is 1.
topology_service_current_allocation_info	Gauge	None	allocation_type, total_capacity, total_allocated	Shows the current total capacity and allocated power. The value is 1.
topology_service_pending_tasks_count	Gauge	None	None	Shows the number of queued topology tasks.
topology_service_oldest_pending_task_age_seconds	Gauge	s	None	Shows how long the oldest queued topology task has waited.

Reconciler and Control-Loop Phases

Metric family	Type	Unit	Attributes	Usage and availability
reconciler_queue_size	Gauge	None	device_type	Records a reconciler’s queue size by device type. Reconcilers of the same type share one series.
reconciler_state	Gauge	None	device_type, state	Records a reconciler’s state by device type. The current state is 1; other states are 0. Reconcilers of the same type share one series.
reconciler_errors_total	Counter	None	kind, device_type	Counts errors by type for each device type.
reconciler_request_duration_seconds	Histogram	s	device_type, request_type	Shows how long each type of device request takes, including response waits and retries.
reconciler_events_total	Counter	None	event, request_type, device_type (conditional), reason (conditional)	Counts queued requests, sent requests, responses, retries, timeouts, removals, merges, and ignored responses. Ignored GPU policy responses use reason instead of device_type.
devctrl_phase_duration_ms_milliseconds	Histogram	ms	component, operation, phase, request_type (conditional), response_type (conditional), result (conditional), error_class (conditional), device_found (conditional), plugin_found (conditional), source (conditional), has_latest_ping_time (conditional), has_ping_error (conditional)	Shows device-controller request phase duration and result.
power_give_back_phase_duration_ms_milliseconds	Histogram	ms	component, operation, phase, result, error_class	Shows the total power give-back time and the result. The phase value is total.

Device Power and NvGrid

These metrics appear only when the node or NvGrid collector is enabled and data is available. Component labels depend on the data that each component reports. DPS removes component series for a node that stops returning metrics instead of preserving a stale sample.

Metric family	Type	Unit	Attributes	Usage and availability
device_total_power_watts	Gauge	W	device_id, device_type, topology_name, power_domain, device_parent	Shows measured power for each device and its location in the topology.
node_component_power_watts	Gauge	W	node, component, id, gpu_uuid (GPU series only)	Shows how much power each node component uses.
node_component_energy	Gauge	None	node, component, id, gpu_uuid (GPU series only), unit	Shows the cumulative energy reported by each node component and the reported unit.
node_component_temperature_C	Gauge	°C	node, component, id, gpu_uuid (GPU series only), unit (conditional)	Shows the temperature of each component. unit also appears when the same component reports an energy reading in that collection cycle.
node_component_power_limit_watts	Gauge	W	node, component, id, gpu_uuid (GPU series only), unit (conditional), limit_type	Shows the set, minimum, and maximum power limits for each component. unit also appears when the component reports an energy reading in the same collection cycle.
nvgrid_feed_schedule_load_target_watts	Gauge	W	feed_tag, start_time, end_time, load_unit, status	Shows load targets from the previous 24 hours through the next seven days.
nvgrid_system_status_load_target_watts	Gauge	W	feed_tag, load_unit	Shows the current scheduled or default load target for each feed.
nvgrid_system_status_calculated_load_watts	Gauge	W	feed_tag, load_unit	Shows the calculated load for each feed. Compare it with the current target.
nvgrid_system_status_in_flight	Gauge	None	feed_tag	Shows whether a power event is in progress for each feed. A value of 1 means in progress, and 0 means not in progress.

Redfish and Policy

Metric family	Type	Unit	Attributes	Usage and availability
redfish_request_duration_milliseconds	Histogram	ms	component, operation, redfish_method, redfish_route, redfish_status_class, error_class, redfish_retry_outcome	Shows request time by normalized Redfish route, HTTP status group, retry result, and error type.
redfish_open_connections	Gauge	None	bmc, pool, target_kind	Shows the current number of Redfish connections for each BMC, connection pool, and direct or proxy target.
redfish_open_connections_max	Gauge	None	bmc, pool, target_kind	Shows the highest observed open connection count. The value resets when DPS clears metrics for the target.
environment_metrics_set_power_limit_duration_milliseconds	Histogram	ms	node, device_count, status	Shows how long DPS takes to apply a power limit and whether the request succeeds, grouped by node and device count.
environment_metrics_get_power_defaults_duration_milliseconds	Histogram	ms	node, device_count	Shows how long DPS takes to get default power settings, grouped by node and device count.
environment_metrics_get_readings_duration_milliseconds	Histogram	ms	node, device_count	Shows how long DPS takes to get environment readings, grouped by node and device count.
environment_metrics_get_chassis_readings_duration_milliseconds	Histogram	ms	node	Shows how long DPS takes to get chassis readings from each node.
policy_apply_duration_milliseconds	Histogram	ms	node, operation, policy_count, domain_max_capability, domain_min_capability	Shows how long DPS takes to apply policies, grouped by node, policy count, and minimum and maximum domain capability.
policy_delete_duration_milliseconds	Histogram	ms	node, operation, target_domain_name	Shows how long DPS takes to remove policies from each target domain.
policy_domain_create_duration_milliseconds	Histogram	ms	node, operation, policy_count	Shows how long DPS takes to create a policy domain, grouped by node and policy count.
policy_domain_update_duration_milliseconds	Histogram	ms	node, operation, policy_count	Shows how long DPS takes to update a policy domain, grouped by node and policy count.

Authentication

These metrics never include usernames, tokens, or client addresses.

Metric family	Type	Unit	Attributes	Usage and availability
auth_failed_attempts_total	Counter	None	reason	Counts failed authentication attempts by reason.
auth_rate_limited_requests_total	Counter	None	limiter	Counts requests rejected by the IP or username rate limiter.
auth_progressive_delay_seconds	Histogram	s	limiter	Shows the delays added by the IP and username rate limiters.

Database Pool

These metrics appear when the database collector is enabled and a database pool is available.

Metric family	Type	Unit	Attributes	Usage and availability
db_pool_total_conns	Gauge	None	None	Shows the total number of open database connections.
db_pool_acquired_conns	Gauge	None	None	Shows the number of database connections currently in use.
db_pool_idle_conns	Gauge	None	None	Shows the number of database connections ready for use.
db_pool_constructing_conns	Gauge	None	None	Shows the number of database connections being opened.
db_pool_max_conns	Gauge	None	None	Shows the maximum number of database connections allowed.
db_pool_empty_acquire_count_total	Counter	None	None	Counts connection requests that found no idle database connection.

BMC Health

Metric family	Type	Unit	Attributes	Usage and availability
bmc_health_runs_total	Counter	1	status	Counts completed, failed, and canceled health checks.
bmc_health_run_duration_milliseconds	Histogram	ms	status	Shows how long each health check takes, grouped by result.
bmc_health_issues_total	Counter	1	code, severity	Counts health-check issues by code and severity.
bmc_health_node_probe_duration_milliseconds	Histogram	ms	phase	Shows how long each phase of a node probe takes.

Tracing

Metric family	Type	Unit	Attributes	Usage and availability
otel_trace_export_errors_total	Counter	None	error_class	Counts OpenTelemetry trace export failures by error type.

Common PromQL Queries

Adapt label values and time ranges to the deployment. These are investigation queries, not alert thresholds. In the Prometheus expression browser, replace $__rate_interval with a fixed range such as 5m.

BMC Request Latency

The redfish_request_duration_milliseconds histogram records the end-to-end duration of DPS requests to BMCs. Its _sum and _count series calculate the mean latency in milliseconds:

sum(rate(redfish_request_duration_milliseconds_sum[$__rate_interval]))
/
sum(rate(redfish_request_duration_milliseconds_count[$__rate_interval]))

Use histogram buckets to calculate the 95th-percentile latency across the selected BMC requests:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(redfish_request_duration_milliseconds_bucket[$__rate_interval])
  )
)

To keep a BMC or request route in the result, filter on labels that are present in the local /metrics output and retain those labels in the aggregation. For example, add the BMC and route labels to sum by (..., le) when calculating a per-BMC or per-route percentile.

Datacenter Topology Power Utilization

To chart aggregate observed power in watts by topology and PowerDomain:

sum by (topology_name, power_domain) (device_total_power_watts)

Calculate observed utilization of a selected PowerDomain as its aggregated power divided by physical capacity. Replace my-topology and PDU-A with the topology and PowerDomain to investigate.

100 *
sum(device_total_power_watts{
  topology_name="my-topology",
  device_type="PowerDomain",
  device_id="PDU-A"
})
/
sum(topology_service_device_power_capacity_watts{
  device_type="PowerDomain",
  device_id="PDU-A"
})

The result is a percentage of physical capacity. Query the PowerDomain rather than summing its parent and child device types; topology aggregation would otherwise count the same power more than once.

Provisioned-Load Coverage and Power Pressure

Provisioned-load coverage shows how much of the PowerDomain’s physical capacity is currently provisioned:

100 *
sum(topology_service_device_instance_power_load_watts{
  device_type="PowerDomain",
  device_id="PDU-A"
})
/
sum(topology_service_device_power_capacity_watts{
  device_type="PowerDomain",
  device_id="PDU-A"
})

Observed-versus-provisioned pressure shows whether current measured power is above or below the provisioned load. A value above 100 means observed power exceeds the provisioned load.

100 *
sum(device_total_power_watts{
  topology_name="my-topology",
  device_type="PowerDomain",
  device_id="PDU-A"
})
/
sum(topology_service_device_instance_power_load_watts{
  device_type="PowerDomain",
  device_id="PDU-A"
})

Reconciler Errors

Count reconciler errors over the last 15 minutes by error kind:

sum by (kind) (increase(reconciler_errors_total[15m]))

For a point-in-time BMC view, use dpsctl check metrics --nodes <node-name>. Its --effective option reads enforced limits directly from the BMC and adds one Redfish GET per device.

Related Information

• DPS Power Steering Controller

• DPS Sample Dashboards

• Prometheus Monitoring