DPS Power Steering Controller#

Overview#

The DPS Power Steering Controller is the current implementation of Power Reservation Steering (PRS) for DPS. It dynamically redistributes GPU power within the power budget that DPS assigns to an active resource group.

DPS remains the authority for topology, resource groups, power policies, and device safety constraints. The controller proposes per-GPU limits from live telemetry and workload allocation data. DPS validates each proposed limit against the applicable policy and applies only valid limits to the hardware.

Architecture and Control Cycle#

Power Steering runs as two services deployed by the power-steering Helm subchart:

Service	Port	Responsibility
Config Service	`8000`	Stores Power Steering domains, nodes, devices, and budgets.
Controller Service	`8001`	Runs the control loop that collects data, calculates limits, and requests their application.

When you use the DPS backend, the Controller runs once per configured interval. The Controller redistributes power only within the domain budget and GPU bounds that DPS supplies. It does not replace an effective power policy or create additional budget.

Redistribution Loop#

flowchart TD Start([Scheduled control cycle]) --> Config[Load domains, nodes, and devices] Config --> Collect[Collect power telemetry, workload allocations, and history] DPS["DPS"] -->|Telemetry and allocations| Collect Collect --> Analyze[Predict demand and detect under-allocation and idle devices] Analyze --> Allocate[Resolve target power and distribute the domain budget] Allocate --> Verify{Topology unchanged?} Verify -->|Yes| Apply[Request calculated GPU limits] Apply -->|Calculated GPU limits| DPS Apply --> Persist[Persist state and record metrics] Verify -->|No| Skip[Skip this cycle] Persist --> Next([Next scheduled interval]) Skip --> Next Next -.-> Start

Each control cycle performs the following steps:

Load the current configuration. The Controller reads the domain, node, and device configuration from the Config Service.
Collect the current state. It requests device power telemetry and workload allocations from DPS and loads the recent power history.
Estimate demand. The Controller predicts power demand, identifies under-allocated devices, and detects idle devices that can yield part of their request.
Resolve and distribute target power. It calculates a target for each GPU, then uses the waterfall allocator to satisfy device minimums before distributing the remaining domain budget across demand.
Verify and apply the result. Before applying limits, the Controller confirms that the topology did not change during the cycle. It then sends the calculated GPU limits to DPS, which validates and applies them in a compliance-safe order.
Persist and observe. The Controller records the cycle state and metrics for the next prediction window and for operational monitoring.

If a pipeline step fails, the Controller stops that cycle and tries again at the next scheduled interval. A GPU that has no current power reading retains its last confirmed limit instead of being treated as idle.

Resource Groups and Power Domains#

DPS creates one Power Steering domain for each active resource group with resolvable effective policies. The domain contains only that resource group’s nodes. When you deactivate the resource group, DPS removes that domain from Power Steering.

An active resource group needs applicable power policies that define node-level power limits for its members. DPS calculates the domain budget from these policies, reserved CPU and memory power, and the configured PRS headroom. If DPS cannot resolve an entity’s policy, it excludes that entity from the Controller configuration until the policy becomes available.

The Controller can redistribute only within the resulting budget. It does not override DPS policy or shared-GPU limits.

Deployment and Configuration#

Enable both the Power Steering subchart and the DPS integration. The selected DPS chart pins the Power Steering dependency. Use that dependency version when you inspect its database, object storage, and deployment requirements:

helm show readme ngc/power-steering --version "<power-steering-version>"
helm show values ngc/power-steering --version "<power-steering-version>"

Prepare PostgreSQL and S3-compatible object storage as described by that chart version.

For the DPS backend, the Controller must point to the release-specific DPS gRPC Service.

Refer to Deployment and Configuration Guide for the DPS security prerequisites and installation procedure.

dps:
  prs:
    enabled: true

power-steering:
  enabled: true
  postgresql:
    enabled: false
  controller:
    cluster:
      dps:
        host: dps-server-grpc

The following values control the DPS integration. Refer to the deployment values for the authentication, TLS, storage, and resource settings required by your environment.

Value	Description
`dps.prs.enabled`	Enables the DPS server integration with the Power Steering Config Service.
`dps.prs.hostPort`	Sets the Config Service URL. When empty, DPS derives the in-cluster service URL.
`dps.prs.controllerHostPort`	Sets the Controller Service URL. When empty, DPS derives the in-cluster service URL.
`dps.prs.headroomPercent`	Adjusts the GPU pool that DPS sends to a Power Steering domain. The supported range is `-50` through `100`; the default is `-15`.
`dps.prs.schedulerIntervalSeconds`	Changes the Controller schedule. Set `0` to retain the Power Steering default, or use a value of at least `5` seconds.
`dps.prs.historyWindowSize`	Sets the number of per-GPU samples kept for prediction. Set `0` to retain the Power Steering default.

After deployment, the DPS global prs_enabled setting controls whether DPS sends Power Steering updates. For example:

dpsctl settings update --set prs_enabled=true

Note

The legacy prs.* Helm values, the 8880 configuration port, the job-scheduler sidecar, and the staged configuration workflow do not apply to the DPS Power Steering Controller. Use the power-steering.* subchart values and dps.prs.* integration values instead.

Operational Checks#

Before you expect Power Steering to change GPU limits, confirm the following:

power-steering.enabled and dps.prs.enabled are enabled in the deployed Helm values.
The DPS global prs_enabled setting is enabled.
The resource group is active and has resolvable effective policies with node-level power limits for its managed entities.
The Config and Controller Services are ready. For the DPS backend, confirm that the Controller can reach DPS over gRPC.

Power Steering records control-cycle results with power_steering_ metrics. Use the Controller logs and these metrics to investigate topology, telemetry, allocation, or limit-application failures.