Excursion Mitigation#

Scope: This document covers DPS-side behavior only. For configuration, refer to the Deployment Guide.

Overview#

This document describes how DPS receives power telemetry from Zapp, detects power excursions, and automatically mitigates them by reducing power to lower-priority workloads.

The simple strategy automatically reduces power to lower-priority resource groups (RGs) when an excursion is detected.

Simple Mitigation Algorithm#

Identify Targets → Reduce Power → Wait → Verify
       ↑                                    |
       |                                    |
       +--- still in excursion, retry ------+

Step 1: Identify Targets#

DPS determines which resource groups to take power from:

It identifies the highest-level devices in excursion (e.g., if both a PDU and one of its nodes are in excursion, only the PDU is considered to avoid double-counting).
It calculates the power deficit — how many watts over the limit the system is.
It sets a steal target of oversteal_factor x deficit (default: 2x the deficit) to account for the fact that actual power reduction may not exactly match estimates.
It finds resource groups that have an effective DPM policy with a specified strategy and that have devices under the excursion points. Groups are ordered by --priority band. Lower-importance bands (higher numbers) are targeted first. Refer to Resource Group Priority.
If no resource groups can be reduced, the mitigation fails and a critical alert is fired.

Step 2: Reduce Power#

DPS reduces power policies on the identified resource groups:

Starting with the least important --priority band, DPS lowers power toward the policy floor.
Groups that share a priority number are reduced together. A lower-importance band is exhausted before a higher-importance band is touched.
At each step, DPS estimates the watts saved based on the difference between the old and new limits.
The process stops when either the steal target is reached or all candidate groups have been lowered as far as possible.
Power policy changes are sent to the device plugins to take effect.

Step 3: Wait#

DPS waits for the configured interval (default 10 seconds) to allow the power changes to take effect and for fresh telemetry readings to arrive.

Step 4: Verify#

DPS re-evaluates the excursion using fresh telemetry data:

If all excursion points are now within their provisioning limits, the mitigation is considered successful. DPS then attempts to give back stolen power. Giveback first tries a complete restore of eligible members. Priority tiers apply only if that complete plan cannot be admitted.
If the excursion persists and the maximum number of iterations has not been reached, the process loops back to Step 1 with updated telemetry.
If the maximum number of iterations is reached without resolution, the mitigation fails and a ExcursionMitigationFailed alert is fired.

Power Give-Back#

After a successful mitigation, DPS automatically attempts to restore power to the resource groups that were reduced. This is a best-effort process — whether power can actually be restored depends on the real source of the excursion. If the underlying cause persists (e.g., sustained high workload), there may not be enough headroom to give power back, and the affected RGs will remain at their reduced policy levels. DPS will do its best to restore as much power as possible given the current conditions.

Alerts#

DPS generates two excursion-related alerts via Alertmanager:

PowerExcursion (warning)#

Fired for every device where aggregated power exceeds the operational limit.

alertname: PowerExcursion
severity: warning
labels: entity (device ID)
summary: Power excursion on <entity>: <measured>W measured exceeds <expected>W expected